Really, everyone knows about the N64 and its decision to stick with carts, but one seriously underrated factor was the delays that pushed it back well behind the releases of its competitors.
While people were anxious for an optical media console given what it promised, Nintendo would have done faaaar better that gen if they'd simply released their console on time.
The Gamecube was a huge blunder as well, mostly as they didn't want to rely on someone else's proprietary format and went with the small disks. It's often forgotten today, but the reason the PS2 set the all-time record for console sales was not due to gaming, but because everyone and their mother was buying them as the best option for a DVD player. Sure, they would have sold very well either way, but that pushed them through the stratosphere, as they were often the best deal available for playing DVDs.
Otherwise, the Gamecube actually maintained rather strong 3rd party support, had superior hardware to the PS2, and looked gorgeous (I was just watching someone playing Pikmin 1 last night and was amazed by how great it still looks). If they hadn't lost so many players the previous generation and actually provided a DVD player with the Gamecube they could have done much better than they did.
Still, the one thing that secured a place for Sony (many were skeptical when it was first announced) and won people over in droves to the Playstation was, to put it quite simply, Final Fantasy VII and the FF franchise overall. This defection (they'd only been on Nintendo consoles to that point) was due to Nintendo's decision to stick with carts, and people just had to have a Playstation when they saw those games (I personally made the jump for FF Tactics and played VII/VIII/and IX to death).
Overall, then, it's impossible to overlook the delayed release of the N64, and Nintendo's unwillingness to use someone else's proprietary format with the N64 and Gamecube really screwed them. If, for instance, the N64 had simply released on time and later the Gamecube also released on time with a DVD player, they'd have been in far, far better shape than how it turned out even without Final Fantasy.
On the bright side, we got two of the best multiplayer consoles of all time, and anyone who was around them has fond memories of them. :)
For the record, btw: Sony and Microsoft weren't going anywhere. This was the first time companies with almost unlimited piggy banks joined the fray, and they weren't afraid to absorb any necessary losses to the same extent that Nintendo was. Sega tried their approach and it sunk them. To this day, the Xbox brand hasn't turned a profit for Microsoft, but they clearly just like having a brand for a company that is otherwise mostly known for Operating Systems.