The common trend as of late, especially after Zelda U's reveal, is to say that realistic Zelda games sell more than stylized ones, using Wind Waker and Skyward Sword as an example. But let's look at things closer.
Wind Waker
Wind Waker wasnt only "a cartoony Zelda game," it was the very first Zelda game on the Gamecube, and the very first console Zelda game after Ocarina of Time and Majora's Mask. This was BEFORE different art styles for thier games (and console games especially) were the standard. People were incredibly HYPED for a new Zelda game, and there was no reason to expect an art style change. In fact, the art style of Wind Waker was kept secret from Miyamoto until late in development, because Anouma and the other developers were afraid of what he would think.
There's also the fact that the tech demo for the Gamecube was this:
http://www.youtube.com/watch?v=wBtAvX4Pkyk
So obviously, after seeing the art style of Wind Waker, many people felt betrayed and this would have possibly had an effect on sales for this game. There's also the install base of the Gamecube being less than the N64.
So yes, it seling less was partly because of the art style, but that was far more relevant to the time rather than overall.
Skyward Sword
Skyward Sword also wasnt just a "cartoony Zelda game." It was a Zelda game who's primary input method included Motion Controls. You know, the thing gamers have a HUGE bias against? The thing everyone thiks is a fad and wish would die? It's strange people think the art style had a bigger effect on sales than the motion controls. A lot of reviews even said the motion controls were the biggest and in some cases, ONLY problem with the game. So there's that.
Secondly, it came near the end of the Wii's life cycle. How much people realise that Majora's Mask sold less than Skyward Sword? you know what they have in common? Coming at the near end of a Nintendo console's lifespan. Gettig overshadowed by upcoming consoles and player fatigue setting in will definitely have an impact. Ever heard of Kirby: Mass Attack? No? Well there you go.
Zelda U
Zelda U is suffering from neither of those problembs. Art style change is now an established aspect of Zelda games, and it's coming in the middle of the Wii U's lifespan. Not to mention it will have a much more open world, something Zelda fans have been praying for for many years. This is something people want.
So no, I dont think the sales of the previous "cartoony" console Zeldas mean much at all.