Xbox One was a Return to Status quo, and PS3 didn't do any lasting damage, just a temporary misstep.
By the time Saturn came out, Sega's reputation had already declined drastically. But Sega really only had a few years of being in the discussion as a market leader. The Saturn was a good console, but it went up against a superior console with a better price point. But it's worth noting that the Sega Saturn was Sega's fourth console. Their first was the SG-1000, second was the Master System, and third was the Mega Drive. I have all five of Sega's consoles.
The N64, on the other hand, ended a period where Nintendo was synonymous with video games that had lasted 12-13 years. Nintendo went from dominance to falling into a distant second before falling into third place the next generation where GC sold less than 1/6th of the PS2. The N64 also used cartridges, which were not only obsolete, but inferior and more expensive than CDs. Nintendo consoles went from being the home of RPGs to being more or less devoid of the genre.
A lot of this was because of the poor choices with the hardware, but the main problem was Nintendo's poor leadership prior to Iwata taking over - specifically the CEO (H. Yamauchi). Iwata's predecessor not only destroyed relationships between Nintendo and RPG developers, but decided it would be a good idea to publicly insult RPG fans. Instead of acknowledging his own incompetence when actually going up against some real competition, he made things much worse. His antics more or less lost all of Nintendo's major third party developers as exclusive developers or developers who primarily supported Nintendo consoles - this included (but wasn't limited to) Squaresoft, Enix, DMA (Rockstar North), Capcom, Konami, Koei, and Rare would make its exit shortly after. Some of these companies, such as Squaresoft, would release no games on N64. The damage to third party relations was profound, and lasts to this very day. Final Fantasy is still primarily a Playstation product.
Lastly, as a Nintendo fan, and an RPG fan, apart from the ridiculous statements from Nintendo's CEO, they also completely ripped off a contingent of Nintendo fans who supported them (during this generation and the next) with something like religious fervour as they would even buy Nintendo consoles and games out of duty, even when they sucked, even when the games were extremely overpriced. The Yamauchi era Nintendo took advantage of this.
That's why I say N64.
1. Nintendo should have easily won that generation.
2. Saturn wasn't really a bad console, just a commercially unsuccessful one, and also Sega's fourth console rather than their third. PS3 was a small stumble, and Xbox One was a return to Status Quo.
3. Permanent damage done between Nintendo and third parties.
4. Ended Nintendo's 12-13 year period of dominance.
5. It took around 10 years for Nintendo to recover.
6. Nintendo ripped off fans with the N64's software.
While the Wii and DS did the majority of the recovery, I'd say it took all the way until the Switch for Nintendo to regain the good will of the gaming audience and developers as a whole.
Currently Online 
