Nintendo's biggest mistake was how it treated third parties. They all abandoned the N64 for greener pastures on PS1. Things started to turn around when Iwata became the President around the time of the GameCube launch, and then Nintendo's second biggest mistake became more obvious: its image problem.

Nintendo had a lot of "family-friendly" appeal, and not much else, which is a nice way of saying parents knew Nintendo games were okay for their children. Of course, most Nintendo games are suitable for all ages, but once it had the reputation of being for kids, something you grow out of and move on to "big boy" consoles like PS2 or XBox, it was hard to shake.

Nintendo turned this image problem into a positive by not confronting it head-on. Instead of trying to do a 180 on its "kiddy" reputation and go head-to-head with Sony and Microsoft for the teenager market, it curved its image slightly. They made games truly "family-friendly" by making them fun for families to play together, taking the emphasis off of kids playing by themselves.

They've also gone back to the NES marketing strategy by sort of ignoring the fact that their products are video game consoles. Nintendo marketed the NES as an "entertainment system," a 'thing' that happens to play games. The DS and Wii got their popularity a similar way. People who bought those consoles for Brain Training and Wii Sports don't think of themselves as 'gamers' and don't want to be treated like gamers. They don't even consider Brain Training or Wii Sports video games. This is the meaning of Nintendo's new "I am not a gamer" ad campaign -- they know how this part of their audience thinks.

It's an odd kind of marketing, too be sure. Nintendo is basically saying one thing and doing the opposite. They're pretending to be a kind of service provider to the masses -- a company that makes interesting interactive products, hip new technology, not really video games. Meanwhile when you get to the heart of the company, they're really all about making games. They don't care about much else. Everything they do is about enhancing the gaming experience. It's really weird, but if people want to delude themselves into thinking they're doing something other than playing video games while they're rolling virtual bowling balls around, Nintendo seems to be fine with that.

Man, Spurge, you sure do know how to phrase a thread title to get people excited. I saw this thread and clicked so hard I probably crushed my mouse :)

Everyone has covered all the reasons pretty comprehensively: cartridges, the consequent drop-off in third party support, marketing, etc.

Still, it's not like the N64 and Gamecube were disasters. Even the Gamecube, which sold only 21 million units, was profitable for Nintendo. And let's not forget the games: Ocarina of Time, Majora's Mask, Super Mario 64, GoldenEye 007, Excitebike 64, Perfect Dark, Paper Mario, Conker's Bad Fur Day, Metroid Prime, The Wind Waker, Mario Kart: Double Dash!!, Pikmin, Resident Evil 4, plus a lot of great multiplatforms games on the Gamecube.

I would also argue that by ignoring third-party games and focusing on (and protecting) first-party games, Nintendo lost the battle but might win the war. As more and more games go multiplatform, exclusives will matter more and more.

Any real gamer would know why Nintendo broke it up with Sony, it's just like Viper1 said.

If Nintendo agreed to what Sony wanted we wouldn't have the Nintendo we have today.

The Sony deal was bad, but Nintendo had one with Philips too for CD drives.

It's not like Sony had some patent on CD-ROM technology and Nintendo couldn't have gotten that elsewhere.

Nintendo should've sat down with Squaresoft, Capcom, Enix, Konami, etc. and compromised. If Miyamoto wanted to make cartridge games, great, keep the cartridge slot, but a CD-ROM drive should've been added to the hardware.

They would've dominated that generation if that was the case IMO.

I honestly don't know if Sony would've even have made a Playstation 2 after that, Nintendo would probably have been in the clear to beat up on a finacially weak Sega.

This.  During the Nes, Super Nes, and Gameboy days, Nintendo was the biggest game in town.  As a result, you played by their rules or you didn't play at all.  As a result, Nintendo was able to charge high premiums to third parties.  They could limit the amount of games a third party could produce.  If Nintendo was releasing a game in a certain genre, they could force a company to postpone their title so that Nintendo's game had no competition.  They could force censorship.  In short, Nintendo of old was a tyrant company. 

So, in 1995, they decided to stubbornly stick to the outdated cartridge format and attempt to force companies to continue to buy expensive cartridges from Nintendo.  Unfortunately, for them, Sony was on the market and their Playstation console was easy and cheap to develop for.  As a result, Nintendo mainstays like Konami (Metal Gear, Castlevania), Capcom (Mega Man, Street Fighter), Square (Final Fantasy, Chrono Trigger), and tons and tons of companies that created classics for Nintendo consoles all jumped ship.  To be honest, I never felt that Sony knew what they were doing during the early days of the PS1.  The influx of AMAZING titles from third parties made the PS1 the place to be for all gamers. 

So, with newfound freedom, companies were able to release the kinds of games they wanted to release without fear of censorship, expensive license fees, release date control, etc.  And, of course, real gamers go where the games are.  Nintendo (who was considered a more family friendly option, even during the 16-Bit era compared to the Turbo-Grafx and Genesis) just couldn't compete with the "cooler" choice.  And, once Sony had a grip on the industry, there wasn't much Nintendo could do to take leadership back from them. 

Even the Wii had the potential to fail miserably had Sony not made many terrible business decisions at the start of the gen.  Nintendo's hubris brought about their dark times just like Sony's has now.  If Nintendo has managed to mend the damage caused during their Golden Age, and third parties are willing to support them in droves, the Wii U will be unstoppable.  If third parties are restricted by Nintendo, Nintendo is in for a rude awakening.  The competition is tougher than it's ever been and other options exist.

"What happened to Nintendo between the SNES and the Wii? What went wrong~"

Internet.

In their dominant years, they became really abusive with third-party publishers. When Sony entered the market, every publisher simply decided to support it. Two generations of really bad market performance have made a hit too. Even Xbox 1 had a better third-party support. This guys don't want that anyone dominates the console scene. That's a bad deal for them. They support PS1 because Nintendo was becoming abusive. They supported Xbox/X360 because Sony was becaming abusive. Now they have a pretty good position, Sony, MS and Ninty have big portion of the market but none of them is creating a monopoly situation (NES, PS1, PS2), and the game studios prefer that things remain this way.

Even with the Wii sales, they couldn't recover their support. This gen, MS got all the third-party support and Sony really had to struggle to get it again (what they have accomplished by now). So it's a situation that it's still problematic. Sony and MS have a strong third-party support and Ninty don't. If GTA V really don't come for Wii U, that'll be a really bad signal regarding future third-party support.

Which games have 'intellectual storytelling' in them?

What changed is that Sony and Microsoft were willing to lose billions of dollars on hardware, and spend millions of dollars to secure exclusive third party games for their system, and were trying to push all competition out of the market. Much like a small chain of grocery stores suddenly finding themselves in direct competition with Target and Walmart, Nintendo was struggling to compete on the same terms as Sony and Microsof.

With the DS and Wii Nintendo (essentially) decided that they weren't going to play by the same rules as Sony or Microsoft anymore; and as a result they have become much more competitive. Using the grocery store example, while Sony and Microsoft are like Walmart/Target, Nintendo became like Whole Foods and started targeting people based on "healthy eating" rather than on lowest price.

Like many said, the decision to ultimately stick to cartridges for the N64 is the start of it all.

You can't blame their extremely strict quality control on the NES and SNES for this. If it wasn't for that, gaming would be nowhere near what it is today. There was no control on Atari 2600 of any kind, anyone could make a game for it. Because this resulted in hundreds of extremely bad games, the market crashed. Had Nintendo never regulated what was on NES and SNES, gaming wouldn't have recovered from that crash so quickly.

Anyway, with the cartridge decision it is obvious what the main reason was for most third parties to migrate to the new competitor; money. In this regard Nintendo made a bad business decision. Damage was done and the GameCube being responsible for the 'kiddie-image' also didn't help.

From a gaming perspective though, I have to say Nintendo made the right decision. CD technology was in hindsight maybe just a year or two too young. Having played both consoles (N64 and PS1) when they were in their prime, N64 wiped the floor with PS1. The discs on PS1 were atrocious compared to the N64's cartridges. They were much less reliable, half the time games wouldn't load immediately. Loading itself was a pain, it took ages. On every game. Discs also made a lot of noise, the N64 cartridges do not have spinning parts which besides from making them way more durable, means they make no sound.
My N64 probably got ten times the amount of playtime compared to my PS1 because of issues like this.