Really, the Saturn takes the cake on this hands down. It sealed the fate of Sega's hardware and allowed its competition to dominate handily. The real question is what's second.
XBOne has damaged the XBox brand in a similar way, but it hasn't threatened to derail the entirety of MS Gaming (MS gaming is too big for XBox's failure to really impact it).
The PS3 was a money sinkhole that burnt all previous PS profits and nearly bankrupt Sony, but afterwards the PS4 kept its dominance and they recovered their losses for the most part.
The N64's high price and cartriges allowed the PS to thrive, but Nintendo was very safe with the GameBoy's dominance, so it was really in no danger at all (a situation that continued with the GC-GBA).

All did some damage, of course.
Least damage: Nintendo 64. Yes, it was outsold globally by the PS1 by a little over 3 to 1. So, in that way it fumbled the generation for Nintendo and led to an even more severe gap in hardware sales in the sixth generation. It was profitable, and games like Super Mario 64, Zelda: OOT, GoldenEye 007, Super Smash Bros. and Animal Forest (JP only) and others changed the industry forever. Also, innovations like 4-players with no multi-tap needed, a Rumble Pak, and an analog stick.
From there, I can hardly pick between PS3 and Xbox One. PS3 lost Sony billions of dollars and required massive course correction both for PS3 itself and the planning of PS4. Xbox One had epic fails during its unveiling, many of which were course-corrected before launch and after launch. Xbox One ended up delivering some great experiences by moving away from Kinect and the "VCR" model, Game Pass, One X, etc. Still, Xbox One is to blame for many of the issues they still have with Series X/S which is now tracking below Xbox One largely because a new console is already confirmed to be in development.
Sega Saturn is the biggest failure. The add-ons of the Mega Drive (CD and 32X) in addition to the Saturn itself made Dreamcast pretty much dead on arrival even with its initial and short success. Dreamcast would've had to be a hit to reverse Sega's dying console business.

The Game Boy's "dominance" was a bit of a fluke as the Game Boy was in major decline by the mid-1990s. When people say "what the fuck was Nintendo thinking with the Virtual Boy" ... it was because Game Boy sales were sliding down the tubes and Yamauchi basically ordered Yokoi to make something else to sell. 

Pokemon came out of nowhere and gave the Game Boy a second lease on life, but this basically never happens in the industry. 

Nintendo's position would have been precarious without that, they didn't plan for the Game Boy getting that second wind at all. 

Saturn and N64 are 4th consoles and this was NEC's third console. Despite the name, it is a console. It was the successor to the PCEngine/TG16 and SuperGrafx (which was such a disaster it only has 5 games) PCFX launched the same year as Saturn/PS1 at a higher price tag and less powerful. It could not do 3D and the kicker is Saturn was better at 2D than PCFX which prided itself on 2D abilities that cost more. PC Engine was a big success in Japan and was Nintendo's main rival in that region for Famicom and early Super Famicom. NEC a big electronics company at the time (they still exist but not what they were then). This thing was such a disaster it never left Japan. Saturn at least was a success in Japan. This is NEC's last console with only 60 games and 300k systems sold.

Oh sure, Pokemon was a miracle for Nintendo at the moment. But the fact of the matter is that even without Pokemon, Nintendo still had no true competition over the handheld market, so the pressure was much less than for Sega, which had no real competitive alternative for either market, with both the Game Gear and the Saturn failing.

Well, I learned something new today!
I never knew the SG-1000 was its own thing; as someone not that familiar with Sega when I always thought it was kinda like the Famicom to the Master System's NES, a JP only version of the same console that looked different and released earlier. Cool stuff.

Although if we get technical we could also argue for stuff like the Colour Game TV for Nintendo which muddy the waters still further.

I'm gonna let the poll and such stay as is for now, so as not to wipe the input received so far. We can just consider it third major console as stuff like SG-1000 and Colour Game TV didn't see global release.

The thing is the Sega SG-1000 and Nintendo Famicom both launched on the exact same day, July 15th, 1983 in Japan. 

They were definitely head to head competitors. The Famicom was so much more advanced though, the SG-1000 was like an Atari 2600 on steroids whereas Famicom was capable of far more advanced games, even the first Super Mario Bros. was significantly beyond SG-1000 (let alone things that would come later like Super Mario Bros. 3). 

They released the Sega Master System (Sega Mark III) then in late 1985 in Japan I believe, and it was more powerful than the NES/Famicom, but not really a full generation ahead. 

And then Mega Drive/Genesis arrived in late 1988 (Japan), that would be a full generation ahead of the Famicom. 

Meanwhile Nintendo stuck with the Famicom from 1983 to 1990 as it was a monstrous success, though not by design. The Super Famicom was supposed to release in 1989 but fell behind schedule, this is why the Game Boy was actually allowed to release, Yamauchi had cancelled the Game Boy project, but with no Super Famicom for 1989, they quickly fast tracked the cancelled Game Boy project to market. 

Mega Drive never really took off in Japan where PC Engine did. PC Engine was the biggest competitor Famicom had in Japan by the late 80s in sales. PC Engine sold roughly 6 million in Japan vs the 3 and half of Mega Drive

If I had to rank from biggest failure to smallest failure, I would go Saturn, Xbox One, N64 & PS3.

Saturn is the biggest failure of the four because it basically led to Sega leaving the console race. None of the other three have led to that. Xbox One was a pretty big blow to Xbox though and they still haven't recovered. N64 is next because it was the first time Nintendo wasn't in first place in the console race and they lost big exclusives for a while. They were saved by the fact that N64 still had iconic releases. Also, they had their handheld market to fall back on and Pokemania. They also turned to a blue ocean strategy that was a mostly successful. I consider the PS3 to be the smallest failure of the three because even though Sony lost billions on it, it still sold the best of the 4 and Sony didn't have to completely change their strategies in order to course-correct.

Yeah, SG-1000 was very similar to ColecoVision, which, while as an actual start of 3rd generation in '82 was quite superior to Atari 2600, was not enough for SG-1000 to go head to head against Famicom in 83.

SEGA MarK III, aka Master System, 2 years later, is then in turn quite better system than NES - but Famicom/NES were quite cleverly designed - give consumers very solid base console and then have them pay really expensive cartridges that pack additional hardware as well for even better looking games.