The system is struggling now, what happens if it never really becomes a hit and goes the way of the GameCube? People said "wait for 3D Mario and Zelda and Metroid and Mario Kart!" there too but the system never really got going, and it even had Resident Evil exclusive for a while.

So this is a thread to discuss a possible Nintendo future in this scenario. So if your post is going to be "damage control about how the Wii U will pick up later on" please save it, there are 50 other threads you can post in. Here we want to discuss a Nintendo future in light of a Wii U that never really takes off. 

Does Nintendo re-focus and create a handheld/console tablet hybrid machine? Something you can play at home on the TV (wirelessly?) but then pick up and take on the road? 

Or would Nintendo try again in the console business with another machine around 2014/2015 that's speced comparably to the PS4/720? Maybe with VR tech or something? 

Well, I don't see them releasing a new console in 2014/2015 - they didn't do that whilst the GC was failing, and they're not gonna do that now. Earliest release date of a new console is 2016, but more likely in 2017.

But I think you said it with "Does Nintendo re-focus and create a handheld/console tablet hybrid machine". I've been thinking along side the same note; a home console you can pick up and play on the go - but nothing like the Vita, mind you. What a console like that would look like I have no idea, but I'm sure Nintendo would find a way to make it work. That would put them in a category for themselves; combating both the threat of smartphones and the other home consoles. We'll at least see much more integration between the home console and the handheld.

I think the WiiU was a year too late, but I'm sure Nintendo could release another system without too much worry. But still, Nintendo survived the N64 and GC, they'll survive the WiiU regardless of how well it sells.

They dig into their cash pile and they try again, in the absolute worst case scenario, presumably dropping the Wii brand in the process.

First off, it's much too early to be talking like this. I know it's difficult, but just remember that nearly every platform in history has trouble when it first launches. Launch libraries and the six-to-nine months that follow are always light on big software as things always get delayed and word spreads that the system actually exists. Wii U isn't exceptional in this case, it is merely normal.

Secondly, not taking off isn't an option. Wii U will come to represent about 50% of Nintendo's business in the next year or two, and it will stay that way for five or six years. So "not taking off" isn't an option for them, and believe me, Iwata won't let that happen. We're probably all in store for some megaton bombs at E3. Wait for it.

Worst case scenario, Nintendo repeats the GameCube era, sells 30 million Wii U's, tightens their belts, lives off of 3DS income, and releases a new system in 2017 or so.

After the success of the Wii/DS, Nintendo can survive a relative failure.

Releasing a new console before the five-year mark will damage their reputation and image in the long-term, as many will simply not buy hardware from them anymore if they think it's a possibility it will be replaced in three or four years.  It would cause more damage than merely having a five or six-year system lifespan with moderate success.

The GameCube had many "megatons" too.

Resident Evil full franchise exclusivity is a bigger megaton than anything I've seen from any console maker the last few years. Metal Gear Solid remake, Tales + Baten Kaitos, Sega + Namco TRIFORCE arcade board, GOTY Metroid Prime, etc.

The core hook of the Wii U just isn't appealling enough to bring in casuals which buoyed the Wii IMO, I think what's actually going to happen is as the months go on, this fact will become more evident.

Nintendo can announce all the Marios and Zeldas they want, Wii Sports/Fit is yesterday's news, and that's the main problem for the system. They bet the farm on that audience coming back and didn't make a machine appealling enough to get the 360/PS3 crowd either, so they're stuck up creek without a paddle IMO. 

They go back to the drawing board, to come up with something else.

They go back to the drawing board. I'm pretty sure that the system will do better than the Gamecube and probably be more of a N64 sales wise, but if it sells like the GC they will just get rid of the Wii name as was said above and try something completely different (like maybe selling toasters).