The controller is confusing people, but it's definitely not the leading issue by any means. We've already seen that the dual-screen concept works marvels for handhelds, and being a WiiU owner, I can speak without any doubt that the controller only contributes to gaming, not detracts, like many non-owner naysayers have tried to say.
The problem is that (1) Nintendo did a horrible job at marketing the system initially, and too many people even NOW are under the misunderstanding that it's just a peripheral for the Wii, (2) numerous third-party publishers have from the get-go written WiiU off, most of which never even gave it a chance. Your games won't sell if you don't give them effort, and the system won't sell if it doesn't have your games. The only reason Xbone and PS4 are better options for your games, Third Parties, is because you chose for them to be better options. It's simple as that. (3) It's not just third-parties' faults, though, as to why this relationship is bad. Nintendo lost a lot of third parties in the 5th generation and have just never been able to pull themselves out of that funk. They ruled gaming with an iron fist in the 3rd and 4th gens, and when Sony released the PS1 with a lot of the restrictions lifted - and especially as far ahead of the N64 as they did - third-parties moved en masse. And Nintendo has never been able to attract them back properly. The 5th gen resulted in Nintendo developing a kiddy image, what with Mario, Pokemon, Donkey Kong, and Banjo Kazooie, while PS1 had Twisted Metal, Tomb Raider, Resident Evil, and other flagship "mature" titles. Sure, both systems had their fair share of the other side, but that's not what either ended up being known for. And it's entirely the 5th gen's fault as to why Nintendo's in this home console slump today - they may get the titles, but they don't sell as well because the public's never been able to shrug off their perception of Nintendo as a kiddy system. So, next time around, they don't get the titles. And with no titles, people decide not to buy. It's a vicious cycle, and with the Wii/Mii brand, Nintendo did nothing but reinforce it.