I don't think they would fail. In fact, I think they would do nicely. However, I do agree that they would not come close to the numbers they see on their own console.
Part of why Nintendo games sell so many units is that they effectively have a captive audience. Third-party titles are few and far between on their home consoles. If you want to buy a new game for your Wii U then you'll probably be giving your coin to Nintendo. Also, being the console manufacturer, they can direct all the attention toward their own titles. This is something Nintendo does better than anyone and which I've long suspected many third-party publishers resent. Nintendo sells Nintendo products aggressively, while Microsoft and Sony spend a lot of time and energy selling the products of others.
Nintendo games on other systems would obviously not see the same level of advertising. They would have to depend more on outside media advertising and they would have to push each and every release. The lower level Nintendo games would likely perform poorly. It's possible we would see their output retract.
I'm also willing to bet that a lot of Nintendo-only gamers would see the scope of their interests expand if they had to buy a Sony or Microsoft console. There really is so much more content on those machines. I think quite a few of them would have to drop out of the "buys anything Nintendo makes" club, as some of their purchase slots would be occupied by other selections.
Of course, there would also be further opportunities for Nintendo. I'd be willing to bet that Metroid would sell better on Xbox/Playstation than only on a Nintendo console. Mario Kart and SmaBro would likely perform extremely well and benefit financially from the DLC culture.
Now, all of that said, I think Nintendo will be fine as a console manufacturer if they just lower their projections and modify their business model. They can do quite well for themselves even if that means "third place".
