In this hypothetical scenario, let's say Nintendo hardware continues to decline, at what point is their install base too low to remain in the hardware business?

Currently Wii U+3DS are around 60 million, when all is said and done that number will likely increase to 80 million or so. With this size userbase, their franchises are able to sell strong numbers, Pokemon/Smash Bros/2D Mario/3D Mario/Animal Crossing/Mario Kart are all capable of selling over 10 million and other games like Zelda/Luigi's Mansion/Kirby/Metroid/Fire Emblem/Tomodachi/Donkey Kong/Yoshi/Mario RPGs/Pokemon spinoffs/etc are capable of selling from 1-5 million units.

The biggest problem Nintendo is facing is that they have been selling their consoles at a loss. After the 3DS price cut back in summer 2011, it took a full year for the hardware to become profitable again and Wii U was sold at a loss from day 1 and it's possible they are still selling it at a loss. Had they planned a bit better and had their hardware profitable this whole time, they would be fine with their current install base.

So how low would Nintendo hardware have to go before it became too small of a market for them to remain in? Obviously something llike 5 million consoles and 20 million handhelds would be too niche and wouldn't allow their franchises to sell strong enough but would something like 10 million consoles along with 40 million handhelds be enough assuming their hardware wasn't sold at a loss?

Honestly, only 10 million or so.

10 million between home consoles and handhelds?

30/40.

They will never ever pull that with a console. Next handheld might see a massive drop of to under 20m lt.

Nintendo needs to make profit to stay relevant. That's all

Exactly what I was thinking. With that said I'd say a combined 50 million would be needed to stay relevant.

As far as consoles, I think you need at least 15 million, don't forget that less hardware sold means no software sold. Some can argue as long as they are profitable, but no, it's not simply about making at least a dollar profit at the end of each generation, if the company isn't growing and gaining or maintaining marketshare it will be overshadowed by it's competition.

I think even 15 million may not be enough, and that goes for any console, if consoles have been gradually increasing each gen and like Nintendo you are losing sales each gen with the exception of the Wii, than their is a serious problem not only of consumer demand, but in the appeal of your software.

10 million for both devices combined or each? Do u really think that's big enough for their franchises to remain strong and be overall profitable?

As Eric said above, they just need to be profitable. If they are, viola. They won't go anywhere. Plus, they'd have to unprofitable for years on end before they decided to quit the console business.

If that's all they manage for both handheld and home console combined in the next generation then they won't be relevant or viable.

10 million home consoles and 30 million handhelds I think would be the absolute minimum, IMO. But that would be a rather uncomfortable position for them. 50 million per generation would be viable and fairly comfortable, and doable over multiple generations.