Nintendo's main problem is that they don't cater to the market that buys 3rd party, by which I mean their system wasn't X86, it was also underpowered compared to PS4 & XBox One and it wasn't a significantly large enough step above PS3 and 360 to show current gen visuals off.
3rd party makes up the majority of console software sales, so to ignore that is to ignore the biggest potential, guaranteed sales base for your system.
Going after the casuals that bought the Wii was the biggest mistake Nintendo could have made, because those people were only hooked by the gimmick of motion controller gaming and they will probably be happy with the Wii if they still happen to play any video games. They were basically the one time buys, that bought the system, for Wii Sports and Mario Kart, hence the big sales of those games, in line with a huge chunk of the platform's sales.
Marketing is only part of why Wii U didn't take off. Obviously when you come out with a new platform you need to let people know that it's a new system, calling it Wii 2 would have been a better move, because at least the mass market consumers would have known it was the sequel and not just an add-on.
Really though the entire philosophy surrounding Wii U was a mistake for all but the die hard Nintendo gamers that come back every generation and a few others that were intrigued by the tablet control and some games they can't get on other consoles.
If Nintendo ever wants to become a big player in the home console market again they basically need a change of image, to broaden their own focus as far as the games they make, exclusives they get on the platform to appeal to the market that buys 3rd party, because if you can land those gamers then 3rd party will return.
They also basically need to take Sony's approach, building a system that comes out on top as far as console hardware performance, priced competitively, simple to develop games on, with hardware that will reduce in manufacturing costs quickly to allow you to make a profit, while still being able to keep a competitive price.
