Looking back and reflecting on it now, the decision to not use CD-ROM for the N64 was a killer and that really put Nintendo in a bad position, but with Sega imploding, Nintendo still had a reasonable chance to gain ground again with the GameCube.
This is a nice article on the story of the GameCube (initially codenamed Dolphin):
https://dromble.wordpress.com/2014/01/07/dolphin-tale-story-of-gamecube/
To me the thing that really killed the GameCube was not launching it in 2000. Nintendo initially had planned to release the system in 2000, but didn't have enough games. This gave Sony like over a full year head start, and also gave Microsoft time to get the XBox and Halo ready.
This is Yamauchi speaking around 1999:
“We would like to release the Dolphin closer to the PS2, but since we’re aiming for the Christmas 2000 shopping season, I don’t think the time difference is that big of a handicap,” said Yamauchi.
Of course, Nintendo being Nintendo they couldn't stick to this release date and had to delay to 2001:
“It’s always the case with Nintendo, the hardware is already completed, but the software is not,” said Imanishi.
I think this was the killer. Had they been able to launch in 2000, I think they would've been more competetive with the PS2 even though the PS2 would've won and they would've consolidated a lot of the split that the GameCube (22 million) and XBox (24 million) had into one, so I think GameCube would've sold 45-55 million if they could have launched it for fall 2000. XBox would've been way too late, and Sony didn't have a great first year of software either.
Yes they would have great success with the Wii a few years later, but in doing so they basically surrendered the entire traditional market to Sony/MS and they weren't able to even keep the casual audience, so they kind of locked themselves out of both markets.
If I could go back in time and tell them anything it would be to pull Zelda: Majora's Mask, Perfect Dark, and Conker from the N64 and put them on the GameCube as launch titles and launch in 2000 no matter what.