Why would the average consumer want to buy a Nintendo home console when, in the world you're describing, they have access to the majority of major 3rd party releases on stuff like their TV? Nintendo is already struggling to sell the WiiU on the back of just their own IP, and that's in a market where dedicated hardware of the kind we have right now is often still a necessity.
I think the handheld market somewhat highlights the above. Despite having larger control limitations, the mobile gaming market has had a massive impact on the success of dedicated handhelds, simply because it removes a lot of their necessity. A TV + a controller will function pretty much identically to a traditional consoles, so outside of being technically inferior (which you say won't matter), there will be even less incentive to invest in dedicated hardware. Especially when all you're gaining is 1 publisher's IP.
