FIrst of all, what to do with the Wii U:
Release their next system right away. Just joking. In all seriousness, Nintendo has reached a point where it can not count on major 3rd party publishers on their platforms. They found reasons not to support the N64, GameCube, Wii, and now the Wii U, and one can argue that the Vita gets better 3rd party support than the 3DS despite the latter being a few times more popular. Indie developers seem to have largely gotten on board, but as the Vita can attest, we have not reached a point where those can push hardware.
Therefore, the mid-to-long term solution is to simply increase 1st and 2nd party output. In the latter area, Nintendo is actually doing well. The Wonderful 101, Bayonetta 2, Super Smash Bros 4, Hyrule Warriors, LEGO City, SMTxFE, and Ninja Gaiden 3: Razor's Edge are all example of 2nd party output. Nintendo has increased their software output on the Wii U by probably 25% to 50% this way. This can be improved, but the job so far has been well done enough.
Of course, the problem of unideally low first party output remains. There are several ways to work on this. One is to naturally let studios get experience with HD development. Inexperience is probably why it took Retro, for example, 3 years to make a 2.5D platformer with 40 levels. Studios will get better at this over time. However, the core problems remain.
Options include increasing the size of existing teams, creating new studios, or acquiring studios. The first is probably inevitable, but should be limited: one of Nintendo's strengths is relatively small yet experienced and skiiled teams, and simply giving EAD another 200 newbies and saying "go wild" will barely help. Creating new studios should be done to a greater extent, but the problem of inexperienced teams not creating quality work arises. These new teams could not immediately be assigned, say, the next Metroid game for the Wii U. At first, they'd only be trustworthy enough for smaller eShop projects and aiding other studios.
Acquiring existing teams thus seems like one of the better options available. These teams would probably have to be independent studios without a constant publisher that put out fairly decent work regularly. Preferably they'd be Western, but I have nothing against Japanese studios. These studios could then be given the task of making new series or handhling the development of series not currently underway by Nintendo's "core" teams, ala Next Level Games and Luigi's Mansion: Dark Moon.
The problem is, how many new studios can Nintendo afford? If they buy too many at once, losses will be great and shareholders will be furious. They'd have to limit themselves to a maximum of a couple of smaller or one big studio a year in order to pass judgement.
Thus, even if Nintendo goes all out with expanding both its 2nd and 1st parties, it likely won't be able to, say, double its output of roughly ten games of any sort a year by the Wii U's death. Considering how not all of those ten games are top tier, and many hardcore gamers like to buy 20+ games a year, then we still have a problem.
Another path that needs to be taken is the one taken by Super Smash Bros 4: combining Wii U and 3DS development. Two games for the price of 1.4, you could say. Many games for the 3DS could simply be upscaled for the Wii U, especially if they don't make extensive use of stereoscopic 3D.
All that said, the above steps, though probably necessary, are insufficent, especially in the short term. The Wii U needs to sell in 2014, not 2017. And for that, Nintendo has to do something other than becoming bigger.
Here's an idea: make the Virtual Console a selling point.
Imagine being able to play almost every Nintendo-developed game from 1983 to today all on one machine. I'm talking about the Wii's focus on the NES, SNES, and N64 combined with handheld games at least up through the GBA (a step already being takem) as well as GameCube and Wii games.
This would satisfy a lot of fanboy cravings, but how can Nintendo turn this into a selling point.
Redo the price structure.
Nintendo doesn't have to make Super Mario Bros a freemium game, or even 99 cents. But Wrecking Crew should be $1 or $2, not $5. GameCube and Wii games can cost as much as $30 to download, but Nintendo's retro library should extend as large a scale of prices as possible.
Nintendo can also make a loyalty program. It doesn't have to be PS Plus, though that's not a bad idea. They could have people vote on what games get released next and get rewarded with discounts if they vote for the winner. Nintendo could randomly give a choice of gifts every few months to all Wii U owners. They could bundle their big games with digital copies of retro predecessors. Imagine if MK8 came bundled for $300 with a 32 GB Wii U preloaded with the SNES, N64, GBA, GmaeCube, and Wii iterations of the series? It would not cost Nintendo very much, and create a reputation as a less expensive but still quality system compared to the other 8th gen consoles.
The Wii U, therefore, would rely on three pillars (Big Retail Nintendo Games, indie devs, and retro games) to support a console that sells hardware and software at a significantly lower price than the competition. It would not make a TON of money, but Nintendo could still make a profit on every system without too much trouble. Especially since they would dominate retail software more than ever before. I can see the Wii U selling 30 to 50 million units this way, including 8 to 12 times as many retail games and maybe a Billion indie and VC titles, including bundled games.
Now for part 2; the next Nintendo console. This should not come before 2017. 2016 would be acceptable, but 2017 is preferable. It would preferably be at least as powerful as the PS4 in every individual respect, if only to let the platform get 2+ years of ports more easily and have an easier time getting started, but otherwise, power should not be a focus.
What I think Nintendo should do is combine their handheld and console halves to make a cohesive whole. They could develop a single "super-handheld" with PS4-like power, they could make software for 4DS games compatible with the Wii3, they could have deals where you buy games for both systems at the low price of $70, but the 9th gen Nintendo platforms need to be closely connected. If Nintendo can only rely on themselves, then they have to get all their games on both platforms. This means Mario Kart 9 should be available on both platforms the same day, even if there are more features or a better resolution on the console.
tl;dr Nintendo needs to make a new niche for home consoles to save the Wii U, and needs to have both of their platforms be similar for cross-development next gen.
Essay Pro