Bigger sales failure than the N64, Nintendo repeated many of the same mistakes. The N64 was the most powerful console in it's gen, its only flaw/mistake was going Cartridge. It has some the greatest games ever made in its library.  For my money out of that list the Wii U is the big failure. 

I wanted to say Saturn, it was hard to develop for that system, sold terribly, but alot of it was Sega and not the system itself.

Personally I think the N64 was given a losing hand to begin with, following the best console in the history of gaming and being in a transitional point in technology, but still had iconic games that still to this day are talked about and influence the industry. Theres an argument to make that GC also has games like that, but for me,  Its a far smaller list of games than what the N64 had, and as such the GC gets my vote. 

It counts. I was in a rush and the OP was already pretty long--and the popnups on this site are so bad that even my adblocker is having poblems today.

I just tried to stick to home consoles. The VB was definitely a disaster of epic proportions. Not sure if it actually followed up the other portables, though. Could you even play it on the go?

As Rol already mentioned the Atari 5200 is the king of fails.

But from the ones listed in the OP I'll go with the Xbone (it's possible that the Saturn was worse but I admittedly don't know that much about it. The WiiU is also a tempting choice but you know, Nintendo bias). The 360 did really well, it sold great and had an excellent library. The Xbone ruined pretty much everything. The amount of exclusives has been abysmal compared to the 360 and while it do get the far majority of third-party games most of them are inferior to the PS4 experience while the 360 often got the best versions. Sales also haven't been too great.

It was okay to bypass the VB. It doesn't deserve to stand beside the rest of the consoles on this list. It was that bad. 

I don't think anything can ever top the failure that was the Wii U. It was literally a failure in all regards: Absolutely horrendous marketing, no clear message, the "gimick" of the gamepad which even Nintendo's own teams couldn't find a good use for in their own games that they built the system around, it was slow as shit, had nonexistent third party support, and droughts droughts droughts of no games. Remember when DKC: TF came out and the gamepad screen was just black? And then halfway through the gen they tried to start marketing it like the Wii again with things like Wii Sports Club. It really felt like Nintendo had no idea what the fuck they were doing with that system from the get go. I've never seen a system look so undesirable. Those were dark times, man.

It had some great software but not even that could have saved how badly the system was handled. 

From a business perspective, depending on how much PS3 cost Sony in the end, it could have been one of the worst.
From my personal point of view, each system I've had has brought me many hours of memorable entertainment.

If you didn't buy the Goerge Foreman Grill for $600, but instead a Slim for $120, then it felt like a great deal because it had all the games you'd expect.
For me there was no problem with it.

N64 alienated a lot of third parties. But as an N64 owner, while I missed a lot of the rpgs and fighting games that went to PS instead, I still had a lot of unforgettable experiences with it. How Mario 64 and OOT subverted my expectations (because few such games were on the market at the time) is something that very few games since have been able to replicate.

Dreamcast also had a lot of great games, even though it ended up making Sega exit the console business. I didn't own one personally, but played on my friend's Dreamcast frequently. Oh yeah, I even borrowed it one time to play Sonic Adventure. Good times.

So I don't know which one I'd consider the worst.
Virtual Boy maybe?

Yeah. I tried to find a silver lining.

While my Wii U is still connected and gets use to this day, the problems it had just can't be denied.  I don't regret owning one but it should have been at least a modest success. Like the Xbox One did with Kinect, it just relied on something that was popular once but those glory days were long gone.

the wii u, it destroyed the out of the box 4 player option

Does the Atari 5200 count? N64, Saturn, Vita and WiiU have a great library for all of their flaws, and VirtualBoy is not really a successor so much as an attempt to create a 3rd market. The 5200 was so pointless it succumbed while the 2600 was still in the market. Either that or the Jaguar.