Nintendo's made two main slip-ups and they're two slow moving (like the Titanic trying to avoid the ice berg) to recover from them.
The first was the decision to not use CD-ROM on N64. That basically handed Sony the market leader position without even firing a shot in the console wars. Nintendo shot themselves in the foot.
The second was launching the GameCube too late and making several other sloppy errors with it that gave Microsoft all the room they needed to get a foothold in the industry.
They should have come out swinging against MS like Sony has beaten MS like a red headed step child this gen. They needed to be sharp, focused, and on point in 2001/2002 and they weren't. They simply weren't.
But those two mistakes basically cost them the core market. Now they have no choice but to make unconventional consoles because Sony and MS occupy the market that used to be theirs and it's too hard to move them out.