Personally I think the N64 was given a losing hand to begin with, following the best console in the history of gaming and being in a transitional point in technology, but still had iconic games that still to this day are talked about and influence the industry. Theres an argument to make that GC also has games like that, but for me,  Its a far smaller list of games than what the N64 had, and as such the GC gets my vote. 

It counts. I was in a rush and the OP was already pretty long--and the popnups on this site are so bad that even my adblocker is having poblems today.

I just tried to stick to home consoles. The VB was definitely a disaster of epic proportions. Not sure if it actually followed up the other portables, though. Could you even play it on the go?

As Rol already mentioned the Atari 5200 is the king of fails.

But from the ones listed in the OP I'll go with the Xbone (it's possible that the Saturn was worse but I admittedly don't know that much about it. The WiiU is also a tempting choice but you know, Nintendo bias). The 360 did really well, it sold great and had an excellent library. The Xbone ruined pretty much everything. The amount of exclusives has been abysmal compared to the 360 and while it do get the far majority of third-party games most of them are inferior to the PS4 experience while the 360 often got the best versions. Sales also haven't been too great.

It was okay to bypass the VB. It doesn't deserve to stand beside the rest of the consoles on this list. It was that bad. 

I don't think anything can ever top the failure that was the Wii U. It was literally a failure in all regards: Absolutely horrendous marketing, no clear message, the "gimick" of the gamepad which even Nintendo's own teams couldn't find a good use for in their own games that they built the system around, it was slow as shit, had nonexistent third party support, and droughts droughts droughts of no games. Remember when DKC: TF came out and the gamepad screen was just black? And then halfway through the gen they tried to start marketing it like the Wii again with things like Wii Sports Club. It really felt like Nintendo had no idea what the fuck they were doing with that system from the get go. I've never seen a system look so undesirable. Those were dark times, man.

It had some great software but not even that could have saved how badly the system was handled. 

Yeah. I tried to find a silver lining.

While my Wii U is still connected and gets use to this day, the problems it had just can't be denied.  I don't regret owning one but it should have been at least a modest success. Like the Xbox One did with Kinect, it just relied on something that was popular once but those glory days were long gone.

the wii u, it destroyed the out of the box 4 player option

Does the Atari 5200 count? N64, Saturn, Vita and WiiU have a great library for all of their flaws, and VirtualBoy is not really a successor so much as an attempt to create a 3rd market. The 5200 was so pointless it succumbed while the 2600 was still in the market. Either that or the Jaguar.

In the end its about making money for companies and even with failed systems compared to past systems, Nintendo was making money on the N64 and Gamecube. I haven't owned a Sega system so I won't speak for the Saturn. When it comes to losing money the end of the Wii's lifecycle and the start of the 3DS and Wii U, Nintendo went on to lose money for the first time in 30 years and would lose money for a few years. The PS3 at one point had lost 5 Billion dollars. The answer to this question in my opinion is either the PS3, 3DS, or Wii U, but I don't have the time right now to research it to figure out my ultimate answer.

From a economic point of view probably the PS3, since it nearly bankrupted one of the biggest names in gaming. Most Nintendo consoles thst sold bad, simply didn't make all that much money, but it didn't lose buckets of it along the way.