Here is what Nintendo's least successful consoles have in common: They are home consoles where most of the big first party games require the analogue stick. People do not like the analogue stick, at least not from Nintendo. Sony thrives on the analogue stick, but Nintendo wilts away.
8 and 16 bit games focus on the d-pad, while Wii games were about motion controls. People eat these games up like ice cream. But 3D Mario, especially, was never as popular as 2D Mario. That is because the gameplay changed going from 2D to 3D. Flagpole Mario is fundamentally different from starfinder Mario. Even BotW, while technically 3D, found success by going back to the 2D gameplay philosophy as much as possible.
What I find especially interesting is the best selling game on the Gamecube was Smash Bros: a 2D game. The best selling game on the Wii U was Mario Kart: a motion controls game. Even on the failed consoles, the games that stand out are the ones that don't follow the 3D/analogue stick trend of the rest of the console.