I wouldn't call their consoles niche. They were just unnapelling to anyone but Nintendo fans. Nintendo consoles audience were a mix of hardcore gaming enthusiasts who buys everything every gen and fans of Nintendo IPs
Lack of good games compared to Playstation is what made Nintendo sales decrease. Not saying there aren't good games on N64, GC and Wii U but their libraries pale compared to anything Sony had in the same generations. Wii somehow reversed this giving a reason for people to buy Nintendo, they could get something out of Wii they couldn't get with Playstation/Xbox. Wii U didn't have such lucky
Switch still lacks many Playstation, Xbox and PC titles but now we are at a point where the biggest console IPs tends to take many years to be released and sometimes we have only one game to spawn two generations (Hello GTA and Elders Scroll)
Nintendo been two generations behind can provide a software pipeline much faster than Sony or Xbox first parties and this make up their lack of 3rd party support. There is enough high quality games on Switch to completely offset the absence of some AAA IPs. Not to mention those IPs are completely irrelevant in Japan anyways.
This generations power leaps had been used almost only to increase resolution and FPS of last gen games, meaning there is no FOMO for mass market to rush into Xbox/Playstation/PC. Nintendo can now provide a good amount of AAA games that don't feel generations behind.
I don't know if Nintendo can replicate this again though. I believe Switch 2 will be fine and sell over 100 million units, but they will eventually reach a roadblock where they cannot spend 6 years to put out all their AAA games like other publishers do.
Mostly because Nintendo likely won't monetize their games to long range profit like GTA Online, if they do they are losing one of their main appeals which is distance themselves from AAA western publishers
I'm curious to see how Nintendo will manage to put out 4K development. Develop to system with PS4 level of graphics is not easy. Let's how they manage it