It could work, but it couldn't be a "new console" in the sense that it replaces the Wii U. It would have to "expand" the Wii U rather and expectations would have to be managed. 

What it would be is this:

Wii U - Regular system, same as today, $199.99 or less by fall 2015.

Wii U "Pro" - Basically a Wii U chip + a high end AMD GPU in the box + 8GB RAM. $349.99. 

The GPU would be higher end than the PS4 or XB1, lets say 2.8 TFLOP processing power. 

The Wii U Pro plays the same Nintendo games as the Wii U, except when played on the Wii U Pro, they get a resolution boost to 1080p, 60 fps, anti-aliasing etc. Maybe even a bump to 2K or 4K resolution depending on the visual complexity of the game. 

It has a higher end OS maybe developed in conjunction with Google that has more set top box functionality. 

The main difference would be that third parties would be able to port their higher end PC engines, ones that may even have trouble running on the XB1 or PS4.

And the expectations would have to be realistic here, you're not going to sell 50 million Wii U Pros. Maybe by the end of the generation you have sold 22 million Wii U Pros + 25 million regular Wii Us ... that gives you 47 million systems to play Mario 3D World on (SNES size userbase). 

In that sense it *could* work, but you can't alienate existing Wii U owners by killing their console. They have to be getting the same Wii U Nintendo games they were going to get otherwise. 

It would be a bad thing because instead of having a big userbase you would have two small ones and each game released could do better and it wouldn`t because most people wouldn`t buy another console for a few games.
3rd parties would suffer more because they already sell the less on Nintendo`s HW. Now try imagining splitting it into two.

to cater to the ps/xbox audience? lol

awful idea

I would not split the userbase, I'd just make two iterations of the Wii U. The regular (budget/low cost) model and the higher end model that more or less plays the same Nintendo games. 

But I would invite 3rd parties to release some of their higher end PC ports of big-game franchises to the Wii U Pro model. 

Which I think is OK. These companies aren't going to be supporting the Wii U (some aren't already like EA) for much longer it doesn't look like anyway.

What harm is it if Nintendo creates a model that caters pretty much all to third parties but also plays their franchise Wii U titles as well.

You sell 22 mill of the "Pro" Wii U and 25 million of the regular Wii U ... that's not a bad userbase.

Truth be told, there is definite room for improvement over the PS4/XB1 anyway ... the XB1 can't even run most games in 1080p, the PS4 struggles with 1080p/60 fps.  

But that would split the Nintendo userbase in two.
If one console plays one type of games and the other, another type of games, you`d be splitting them in two. As an example, one would buy for the EA games and another wouldn`t. Instead of a userbase of two, you would have two userbases of one. Which would mean less money to be made of each console.

And why would gamers buy a new console that would have necessarily a low library of games compared to PS4 and XB1?

Not to mention that introducing a new console now would only cause more financial damage to Nintendo with none of the gains of a single console business model.

Restricting software on one platform as a measure to emphasize 'audience focus' is the worst idea ever and no one has ever done it before or ever will.

So to continue your story, Ultra launches and moves only 10 units in the holiday season and Nintendo resigns in the console race.

I was thinking of it as more of a model akin the iPad model ... most apps still work on the iPad 3rd generation onwards or whatever. Sure there may be some apps that run better or only on the newer iPad model, but does that really bother old iPad users? Not really I don't think. And there are plenty of new apps that work on all iPad models. 

I think the important thing would be that Nintendo's own games work on both models without any issue. Because right now, third parties are going to abandon the Wii U. It's coming, we might as well accept it, the numbers and sales data is that bad.

I don't think Call of Duty or Assassin's Creed or any further Batman games are coming to the Wii U next year, I think even Sega may bail out of that Sonic deal, as even Sonic can't sell on this thing and developers are already grumbling about how many systems they have to work on, Wii U is clearly looking like the odd man out. 

I think 8 months from now the Wii U is going to be basically a Nintendo only box with a few licensed kids property games and the annual edition of Just Dance as the only visible third party support. 

It would be a massive flop.

*facepalm*