I'm not really sure what the blue ocean is right now. I think the industry has shifted to this ...

Home Console - High budget, big graphic experiences and sport sims for big screen HDTV gaming. Emphasis on realistic/violent visuals.

Phone/Tablet Gaming - Easy to Play/Casual/Arcadey game experiences usually with cartoony graphics.

Traditional Handheld Gaming - Nintendo franchises + some Japanese IP.

Are the three pillars of the industry right now. What's not being offered really? Casuals are happy with phones/tablets and still have their Wii around for a game of Just Dance for family/friend gatherings once in a while.

Hardcore gamers have their dear PS4/X1/PS3/360 consoles.

Kids, Nintendo fanatics, and the Japanese market have the 3DS which is kind of a mix of a hardcore/old school platform + some casual stuff.

VR and sex gaming seem like untried blue oceans in this landscape, but Nintendo obviously won't touch adult/XXX games. VR? Maybe in a few years when its cheaper, but Sony seems to already be working on a VR helmet, so that might diminish Nintendo's interest in it since they wouldn't be first to market.

Augmented reality stuff has been done, and on PSP as well as DS/3DS. They might improve it and make it more social, but it's hardly disruptive when it's already out there and not that many people are hooked on it. Unless you think they can Apple-ise existing technology and somehow make people think it's sexy, cool and innovative.

VR is the next thing, and Google glasses is the basis for it to become cool and accessible not the clunky and dorky looking Occulus Rift. OR is the better tech, but a Google glasses-like product will be cooler.

Nintendo Fanbase is divided into Mario Bros Series, Mario Kart Series, Pokemon, Kirby, Donkey Kong Serires, Metroid Series, StarFox Series, Legend of Zelda, Yoshi, Wario Series, Pikmin Series, Mario Sports Series, F-Zero, Chibi Robo, Advance Wars, Xenoblade, Wii Series, and countless others. They just can't revive all the left out ones at once can they?

They don't need a new handheld, they need something to turnaround their struggling console. Why would they need a new handheld when the 3DS is still doing great? 

Yes augmented reality stuff has been done. But it's been done like shit!  And with no where near the level of social interaction I described in the OP.  No one has done anything close to that!

By the same arguent Nintendo shouldn't have done the Wii because motion controls were done before!

It's all in the excecution, and a really high quality, well executed social gaming focused ar prduct has not been explored. I personally think kids would love it!

None of those franchises are going to improve the fortunes of the Wii U.

They need new experiences. Not new experiences shoe-horned into existing franchises. New IP's. New ways to play. New messaging aimed at a wider market*.

* Targetting kids is a failed strategy. The sooner Nintendo realise young people don't want to play their games the better.

Contrary to the popular belief that the PS4/X1 crowds are full hardcore gamers, the majority of them are actually what you'd consider to be casuals. 

Nothing on the 6DS

Or how about the 0DS for those without much money

Or for the rich snobby kids that have everything that does absolutely nothing the 8DS

Blue ocean drove me to PS360. I don't like blue ocean.

My opinion is Nintendo's next handheld could run Android.

Nintendo should come up with a unique hardware concept that sets it above other Android handsets as far as gaming goes, while taking advantage of Android's existing features like browser, mail, photos, services, etc.

Note that some projections are that the Handheld gaming has peaked in 2013 and will decline rapidly in 2014, while Mobile gaming is growing rapidly.

http://www.gartner.com/newsroom/id/2614915

Table 1. Video Game Market Revenue, Worldwide, 2012-2015 (Millions of Dollars)

Console gaming is projected to increase here, but obviously Nintendo is not having much of it, and the decline in Handhelds is set to become an even bigger problem.  Since Nintendo dominates the Handheld market, that market decline is almost all Nintendo's to swallow.

In this picture can you see how Nintendo is becoming obsolete without moving into mobile space, and possibly going 3rd party on home consoles?