Just make a hybrid handheld/console.

Going at apple and sammy with tablets and all the others is suicide.

Seriously...Microsoft and Sony fans that keep suggesting Nintendo leave the market and go to mobile gaming or fully touch-based gaming sound either scared of Nintendo's impeccable ability to stay alive in this market (like Metroid in that series elimination game) or they truly don't understand Nintendo fans. We are gamers...with controllers. Just because it doesn't have a bazillion polygons, showcase blood and guts with photo-realistic accuracy, or play on your PS4 or XBoxOne doesn't mean we're not happy with it. The WiiU isn't a failure...and this generation will surely make all of us a bit more humble once the writing's on the wall.

Question: what kind of OS should it have?

Honestly, they should just make an Android phone/tablet that gives exclusive access to Nintendo games then make a beast, exclusive Android game.

Then make an analog controller adapter that fits comfortably around the phone/tablet that doubles as a battery backup.

Then watch Samsung, Nokia, Google and others fall over themselves for licensing.

Granted there's the risk of piracy -- but that's already happening anyway isn't it?

There's no reason why Nintendo couldn't be a relevant phone/tablet manufacturer of Android devices (or other mobile OS)

EDIT: The right haptic feedback implementation could suffice for not having a d-pad, buttons.

Sure, why not sacrifice portability and go touch-screen only?

So...the Nintendo Vita?

Why create a custom tablet when you could create a Nintendo App for existing Android and iOS tablets that allows you to purchase any game from Nintendo's vast catalog?

I could easily drop $100 purchasing old NES and SNES games to play on my tablet.

I can see something around the size of an iPad mini with a 5.5-6 inch display and physical controls on the outside.

That would allow for greater heat dissipation from the chipset too.

They could always make a smaller pocket version later on.

Nintendo's next handheld can be whatever it wants. It's not like their handheld is a problem because it's the only one with a bright future. All I want to see is to bring more variet and unique software to their console rather than always giving us innovation in software due to new hardware. Focus a little bit more on power and variety in games rather than going with innoative hardware and then sometimes find themselves with problems.

I don't like the idea. They're better off just making mobile games. That would be much more profitable.

Casuals won't buy a dedicated tablet for gaming. Casuals have Android and iOS tablets that satisfy their gaming needs.

I would love to read up on this. Do you have any links to additional info on it?