I can't see how a more powerful Nintendo gaming console (NX) is going to sell more and be successful. I can almost guarantee you Nintendo will still get shafted by third party support. What is Nintendo going to do with this console that will make them successful? If they fail with the NX, I believe they need to leave the console industry and just make software from now on.

I speculate that it will try and be the Apple of gaming. It will try to revolutionize UI be making a digitally focused platform that prioritizes convenience and simplicity of use across multiple platforms. Nintendo has always tried to innovate software and hardware. This time, I think they are going for firmware.

They need to attract either the hardcore non-Ninty gamer or the casual, preferably both. For the former, they need 3rd party support (meaning easy to dev for / port to), the latter means having a certain something to capture the imagination of non-gamers. (That 'something' doesn't necessarily need to be a gimmick. It could be a new game that does interesting things.)

If they just want to be successful they could go 3rd party on consoles and keep doing handhelds. But at this point I think Nintendo measures success by staying in hardware, not necessarily profit.

I don't think that would be something that would fly off shelves. The phone/tablet market is different than the gaming market.

Have games that people want to buy so much that they're willing to pay $200-$300 for a console first.

They need to make it even more child friendly, pack it with a lot of kiddie games, toys and trinkets, and make a very cheap buy ($99-149) that has power as much as a good phone, as kids don't really care about graphics.
Market it a lot towards parents during Christmas, maybe with a gimmick toy like Furby or Tickle Me Mario and watch it fly off the shelves. They already have a lot of child-friendly mascots, they should focus on their strengths.

Something completely new (like the Wii was).

What I learned is that we don't always know what we want in the end, so I'm not making any bold predictions. However, I reckon that Nintendo is going to make their next console ambitious.

Though what would help is if they launch a 3D Mario game first month.

I'm 100% sure it would be. The markets are definitely different, but that doesn't mean they have absolutely nothing in common. A digital platform isn't a "phone/tablet" thing, it's a modern thing. Console gaming is way behind the times in that aspect. This concept is cutting edge. It obviously has to be done intelligently, but a unified platform that focuses on shared digital content across all hardware configurations and an intuitively modern UI experience, with the right software behind it, would be massive.

It's like when people said that no one would buy digital book readers like the Kindle Fire and iPad because people liked the "feel" of holding a book in their hands. Now those are massively successful.

Maybe market it and have a name that isn't dumb and confusing