Generation will more consoles has more sales than generation with fewer consoles. More at 11.
Joking aside, the problem with this assessment is you're not comparing apples to apples. Sure, let's say total Gen 9 consoles sales are less than Gen 8. But Gen 8 has more systems. This means someone could own a Wii U, a 3DS and a Vita in Gen 8 but they would just own a Switch in Gen 9. So there really wasn't a decline since that customer is still in the market and still buys systems and games. The best comparison would be to remove the Vita and Wii U since you can assume those customers should be absorbed into another system (or try and come up with a factor to reduce them). Or, as you mentioned, look at software as this should remain consistent (someone may buy multiple systems but probably wouldn't have bought multiple of the same title). Otherwise, right now it's a flawed analysis because we should expect the sales of 5 systems is greater than the sales of 3 systems.
However more consoles in the market means higher revenue from hardware units which means the market shrinking as far as hardware revenue is concerned
I questioned whether having less systems in the market will lead to more software spending. If software revenue growth is big enough to cover the hardware loss, then the market is flat or growing. If it's not, then the market is declining
Ultimately, I don't think we can separate home and handled console markets anymore. Nintendo is the only company releasing dedicated handhelds, and their best selling model is a hybrid. What can be said however is home-exclusive consoles are failing to meet their past popularity, with Japanese customers choosing portable or hybrid models instead