The storage medium dramatically damaged their ability to compete.
In the N64-PSX era, the N64 was the most powerful console but because cartridges held less the CDs, it required expensive cartridges to match the cheap CDs. Developers were getting used to with the PSX due to the fact that thanks to the advent of FMVs, more developers wanted the space for those and new more massive game features that weren't available before. This combined with bad 3rd party relations in the previous generations aided in the PSX victory despite the fact that the N64 at least in the western world was still very much a competitor thanks to Nintendo's software and its abundance of multilayer party games.
In the GC-PS2-XBOX era, once again Nintendo chose a different medium then its competitors with a propriety disc that while making the software more secure from piracy was smaller then the DVDs its competitors had as standard. To compound on this failing was Nintendo's decision to release the system with the initial case coloring of Purple and its normal control emphasis on the A button. The controller was meant as a way of simplifying gaming which Miyamoto and others at Nintendo thought might be going too complicated and the coloring of the casing was who knows fault. But these decisions allowed others to cement the idea that Nintendo and its console was meant for younger family oriented gaming which combined with the storage medium drove most 3rd party support from the system as it required extra costs compared to porting a PS2 game over to Xbox (2 compressed GC discs equally one PS2 or Xbox disc). Developers that received success during the N64 era sticking with Nintendo began to desert and it was looking very much like Nintendo outside of its own stable and portables would go the way of Sega
Technically their is little doubt that both the DS and Wii were incredibly lucky gambles that Nintendo struck big on and we all know how the casual market they had been trying to simplify gaming during the previous generation for finally aided them. Now with the Wii U they are prepping for another different type of gamble, time will tell how successful it is.







