I think it all comes down to the kind of audience they target. Nintendo as a company have stayed focused either on the traditional younger gamers that like Mario or more niche, hardcore experiences like Xenoblade style RPGs, maybe some horror games here and there.
TBH Mario Kart still moves big numbers, though Nintendo could do with making new IPs that target the western 3rd party audience, since that's where most gamers are.
It's the lack of games that could interest 3rd party gamers, exclusive to a Nintendo platform that really prevents gamers from buying into a Nintendo system now.
Stuff like The Witcher, Gears, Uncharted, God of War. More shooters, genres that are a bit more grounded in reality, instead of fantasy would help to move platforms and that would have a knock on effect in how many people buy other Nintendo games.
Maybe Mario and co would sell even better if Nintendo had more new IPs that bring the western 3rd party gamers to their system, because those people would already own the system and may be interested to try something different.
