If you compare Nintendo's 3 least selling home consoles, N64, GameCube, and Wii U, you notice they all have one thing in common. The lack of first party variety. The Nintendo 64 had some of the greatest games of its generation from Super Mario 64, to The Legend of Zelda Ocarina of Time. But compared to other Nintendo consoles, it's first party lineup lacks both variety and quantity. The console would often go months without a single first party release, and Most of Nintendo's own games were either the tried and trues (Mario & Zelda), Sequels to SNES games (Star Fox and F-Zero), or riding off of popular sports trends at the time (1080 Snowboarding, NBA Courtside). Rare and their games helped make the N64's first party lineup look a lot better than it actually was, and even then, Rare actually self-published most of their N64 catalog. Honestly, it's a miracle that something as creatively risky and niche as Sin & Punishment, came out of an otherwise, mostly risk-averse first party roster.

The GameCube was better, but not by a whole lot. Nintendo started finding new western partners in the wake of Rare's sale to Microsoft such as Retro Studios, Silicon Knights, n-space, Next Level Games, and Kuju Entertainment, but none of them could quite fill the void. With Iwata becoming president, and sub-sequentially restructuring Nintendo's internal development, the console did start to get more unique and Mature first party games towards the end of its life. But most of its early years were filled with an eco friendly Mario, Cartoon Zelda, Ghost-busting Luigi, Animal Town simulator, and Gardening RTS. While not as risk-averse as the N64, Nintendo simply didn't have the variety or number of games that other publishers did. And unlike the N64, they seemed hell bent on not giving the GameCube the types of games it needed to survive.

Then there's the Wii U, which IMO, was the worst aspects of both the N64 and GameCube rolled into one. Not only was Nintendo giving the Wii U games it didn't need, but they also seemed hell bent on doing almost nothing new for most of the generation. We got lots of Wii, and 3DS sequels, software droughts out the ass, a seemingly endless stream of cute mascot platformers, and no real groundbreaking game, or new ideas for most of its life. Sure, Nintendo did try to turn ship around towards the end with Bayo 2, Splatoon, and Devil's Third (bad of a game as it may be). But by that point, it was too little, too late. This stuff should've arrived early in the Wii U's life, not when the writing was on the wall.

Compare this to Nintendo's more successful consoles, where there's more thematic, conceptual, and genre variety since with a large user-base, there's bound to be somebody out there that'll like different types of games. In my opinion, Nintendo's at its most banal, and risk-averse when their platforms aren't doing well in sales. Since Nintendo depends so heavily on its own first party properties to move systems, that means they have to double down on safe, guaranteed hits whenever one of their consoles isn't doing so hot.

That's a low blow. True but low.

Yet I rate the N64 and Gamecube as better consoles than the higher selling ones.

Yeah. I will prefer a Nintendo 64 over something like a Wii any day of the week.

WiiU is so damn cheap, it's a good console to collect for right now.

Bit of a stretch. There are too many other factors to consider.

Didn't the N64 have pilot wings, wave race and smash?  And Excite 64, 1080 Snowboarding..... Sin and Punishment... star fox, f zero and animal crossing.  Plus Rare and everything they released

I recall first party variety. What I don't recall is third party support. Games prices were also almost double compared to the ps1.  I do not agree with the OP.

-

1) Major aspects: Games.

N64, Gamecube and Wiiu have more Nintendo modern games than Nintendo classic/arcade games. 

Nintendo modern games focus in 3d, slow pace gameplay, and funky mechanics. Have great games, like Mario 64 an OoT, but don't have the broad appeal to booster the sale's console. Focus in Tear 1 consumers

Classic Nintendo is an arcade-style, easy to start hard to master, focus in the local coop, new IP try to emulate classic Games (Wii Sports, Arms), New Ip with new vision of classic Genre ( Splatoon), the major games try to emulate the classic Nintendo games( Odyssey is Mario 64, BoTW is Zelda 1) . Brings Tear 2 ( old consumers) and Tear 3 ( new consumers). 

All portables, minus 3DS, was majority classic Nintendo. NES, SNES, Wii, and Switch are more classic Nintendo style.  

2) Minor aspects: Tech, design choices.

Nintendo respect form follow functions. 

The modern Nintendo has a focus on 3d. All design choices reflect this. Your funky controller t ( n64 three hands controllers, Gamecube C button, Wiiu Gamepad).

Classic Nintendo focuses in many ways of playing. NES, SNES, Wii and Switch have many controllers options, and a multitude of different game styles. 

The modern Nintendo tends a more conservative side of the innovations.  Classic Nintendo tends a more disruptive innovation side. 

3) Major aspects: The other companies.

Modern Nintendo try to fight, red ocean style, with other companies. It starts with SNES x Megadrive ( but Sega is likewise Nintendo, a game focus company) and ends with Gamecube. Gamecube is the attempt to emulate core games, Nintendo style. Nintendo doesn't have the money or infrastructure to fight Sony and Microsoft when Nintendo do that, sales plummed. 

Classic Nintendo believes in more markets to invest, sometimes by Blue Ocean, sometimes disruptive techs, sometimes using the two ways.  

Wii U had one huge disadvantage over N64, GameCube, Wii, and Switch. It was their first HD console. Plus, they were still pushing 3ds games, so why would there be sequels to 3ds games on Wii U when they still wanted to sell 3ds? They wouldn't be unified until Switch. Aside from that, most people didn't know it was a new console.
Wii had basically no power jump over GameCube, so their software teams were already used to programming on that level. Same as Switch over Wii U. I can not really see any difference in mk8 delux over 8, or tropical freeze switch vs u, or any other re release. By the time Switch came out, they were used to U's capabilities. Wii U they were simply completely unprepared for and had a terrible name confusing the market.

Here is what Nintendo's least successful consoles have in common: They are home consoles where most of the big first party games require the analogue stick.  People do not like the analogue stick, at least not from Nintendo.  Sony thrives on the analogue stick, but Nintendo wilts away.

8 and 16 bit games focus on the d-pad, while Wii games were about motion controls.  People eat these games up like ice cream.  But 3D Mario, especially, was never as popular as 2D Mario.  That is because the gameplay changed going from 2D to 3D.  Flagpole Mario is fundamentally different from starfinder Mario.  Even BotW, while technically 3D, found success by going back to the 2D gameplay philosophy as much as possible.

What I find especially interesting is the best selling game on the Gamecube was Smash Bros: a 2D game.  The best selling game on the Wii U was Mario Kart: a motion controls game.  Even on the failed consoles, the games that stand out are the ones that don't follow the 3D/analogue stick trend of the rest of the console.

They also have another thing in common: Underrated amazing games that were better than their Mario counterparts:
Ogre Battle 64, Bayonetta 2 and Eternal Darkness.