They failed to keep its gaming audience as they reached their teens/early adulthood, and those gamers basically basically en mass moved to the PS1.
Both the N64 and GameCube were viewed as the "kiddie" consoles for their perspective generations. Not so much because of Nintendo's first-party offerings, but the lack of major-hitters from 3rd parties. If the N64 had gone with a disc-based system (or Nintendo's deal with Sony and the Play Station hadn't fallen through, for that matter) it would definitely have attracted the 3rd party games that went to the Play Station because of its extra space. Those 3rd parties offered the "mature" games that defined the PS brand.
Not to mention that Nintendo already had a bad relationship with 3rd parties from the NES and SNES era. in the 8-bit era 3rd parties had to jump through hoops to get their games published, and were limited to five games a year because Nintendo was gun-ho about quality control. The market was recovering from a crash, so Nintendo had good reason to do so. They didn't have to keep strict regulations in the SNES era, though, and that's when 3rd parties started to look for other options. The genesis offered that option to some extent, but when the PS1 came along they just left Nintendo behind and didn't look back.
TL; DR, Nintendo fell from grace because it sucked with 3rd parties.
Tag (Courtesy of Fkusumot) "If I'm posting in this thread then it's probally a spam thread."
