What will happen in Japan will depend on what Nintendo does for both, home consoles and handhelds. Xbox can be written off, Microsoft is not going to bother. Sony will maintain the status quo, even if that means that their home consoles will not sell more than 10m units again. So it's all up to Nintendo. I don't think they are happy with the current status despite a generation where they will sell over 20m handhelds in Japan alone, so they will change what they've been doing. What exactly their next home console and handheld is going to be, that's up in the air, but they aren't going to sit idly by and watch how the market for dedicated gaming hardware shrinks. If Nintendo is able to present a coherent vision, then I expect both segments to grow in the ninth generation.

My personal experience, having lived in Japan for nearly three years:

Previous generations were bigger because the number of children, teens and people in their 20s was much bigger. It's mostly just a demographics issue.

Unlike in Western countries, where adults seem to keep playing games, most Japanese adults seem to give up on it as soon as they find a job with a permanent contract. They simply have no time to play games anymore, because many people work insane hours and are 100% dedicated to their jobs, and the little time they get they often need to spend with their families. Hence handheld gaming grew exponentially as home consoles declined, those teens became adults and now played mostly on the train. Now that everyone has a smartphone, that is replacing handheld gaming too, because it's more convenient for most people. Japanese people are obsessed with "convenience". Seriously, you'll hear that word here all the time.

Other smaller factors: many adults seem content with replaying old games from the NES-PS2 era, and don't much care for better graphics. Retro gaming is really big here. Another thing I often get is people saying modern games are too "difficult", which is funny because most people would say otherwise, so I think they mean "complex".

As for your arguments on the Japanese industry, it might be true that Japanese output has declined, but that's also to do with there not being too big a market and games becoming expensive to make. However, I'd like to point out that many teens and kids are way more interested Western games than adults, Actually, most people I know with PS4s are fairly into Western games like GTA, CoD, Battlefield or Minecraft. Those are definitely the top 4, but there is more buzz for Western games in general among the core gamers left. Those people also seem more willing to buy PS4s right now.