A system of 50m people is useless if they're only going to buy casual games and Nintendo titles, if the system isn't designed for your audience, you don't expect your product to sell much on it regardless of its userbase. I couldn't speak on the many of the devs that lacked interest in Wii U, I still suspect a lot of it is due to power and percpection but regardless I think you speak in black and white terms regarding a systems potential to sell. The question isn't whether Nintendo was going to win 1st or second place or whether 100% of third party games would be there, but could they comfortably grow beyond the success we've seen on the Wii U.
It wouldn't have to outsell the Xbox One for it be more successful then the gamepad scheme. One which i don't think attracted any notable amount of the Wii audience, one which also generated confusion through marketing with emphasis on the controller, making it appear like an accessory. With a more powerful Wii U, being the least powerful isn't a huge problem as long as you're offering a generational leap. Would plenty of people hold back and wait for the PS4/X1? Sure! The question really is whether more people would more have adopted the Wii U? I think so, of course assuming this brought it better third party support (I'll agree to disagree on that). We may disagree here too but I don't think the power difference has too much to do with the PS4's success over the XB1. In the markets where X1 is strong (USA/UK), the difference between the platforms is negligable and can be drawn from the fact X1 was $100 more expensive at launch+bad press etc. Similarly, being weakest of the 3 certainly wouldn't help the 1tflop Wii U, but it doesn't mean it wouldn't find success. It'd be the cheapest of the lot whilst still being very capable system considered a leap over 360/PS3.
N64 and Gamecube may not have hit their sale goals but they were successful at being profitable and satisfying consumer demand. Essentially there is potential in the "me-too" approach, its how the Xbox brand grew. I believe the market is more receptive to it now then it was in the PS2 era where jaugernauts like Final Fantasy, GTA , Tekken and kingdom Hearts were treated like sony 1st party titles. The jaugernauts of today will be whored out to any system where they will make profit without too much effort
We were talking about Nintendo handhelds, so where does "only going to buy casual games and Nintendo titles" even come from? Stuff like JRPGs and other longliving series have always sold well enough. There wasn't any good reason for concern that games wouldn't sell. As for the other portion of that paragraph of yours, growing comfortably beyond the sales numbers of the Wii U with a me-too approach wasn't likely because the GC already struggled big time at lower price points. Additionally (and this goes for your second paragraph), a third place finish isn't good enough. Why should Nintendo be satisfied with such a result? Likewise, the goal for NX can't be to merely outsell the Wii U. NX (the home console variant alone) needs to sell multiple times more than the Wii U. Nintendo has to have the ambition to clearly outdo their results for the eighth generation.
The main reason why the Xbox brand grew last generation is that Sony messed up really hard. We are seeing very different results this generation where Microsoft is basically a non-factor outside of the USA and the UK, and even in their strongest markets they are forced to undercut the PS4 just to keep up. If success for Microsoft requires a Sony that messes up really hard, then this in turn means that success for Nintendo with a me-too approach would require both Sony and Microsoft to mess up; because if only one of them messes up, the other will be the first choice as alternative, so there won't be much left for Nintendo. Placing your bets on factors that are completely outside of your control is an asinine business decision. It's like building your entire strategy in a football match around the hope that the other team scores an own goal while you park the bus for the entire game; well, except that in football the chance for success would still be higher than in the video game business.