I think it would have been about a 3 way tie, although I'll give the edge to Sony, since that is what actually happened between Sony vs. Microsoft.  Here are some factors to consider:

-As others have said, Nintendo does not get most of its new customer base that it would have gotten from Wii. 
-I agree with Bofferbrauer2 that the Virtual Console was something of a selling point, and "Wii HD" would still have brought in some old school gamers like myself.  
-Nintendo launches at a $300/$350 price tag means it is the cheapest HD console on the market.  That gives it something of an advantage as well.
-Third party companies were gradually transitioning to making their games multiplatform.  Microsoft was the main company championing this, but Nintendo would get some of the benefit.  This means that the "Wii HD" would have had a better third party library than the Gamecube, but not as good as PS3 & XBox360.  (Take the average of the Gamecube and PS3 library sizes and that is a decent estimate of the "Wii HD" library, and then add in Virtual Console games.)

In the end I think Nintendo steals a very large chunk of the XBox360 audience in NA, and it also steals a decent amount from Sony in Europe in Japan.  The "Wii HD" sells to 3 groups: 1) Gamecube fans who stay loyal to Nintendo, 2) People who just want the cheapest HD console (a huge selling point for the XBox360 originally), 3) old school gamers who return for the Virtual Console.  Put that all together and I think Nintendo sells about as much as XBox360 and PS3.  Perhaps they all sell about 60-65m each, and again I give Sony the edge.

Also, given the lack of a $250 console that appeals to new gamers, I'd say DS sales go up about 10m, which would end up putting the DS above the PS2 in lifetime sales.   

NP, honestly, I was assuming you meant different timeline to start with, but I personally find what happens further in N64-CD timeline much more interesting for speculations (and to be fair, timeline I would much more prefer), since it means fairly different Nintendo and, if not fully absent, than limited MS influence.

The Xbox 360 had a $300 USD SKU from launch, so Nintendo wouldn't have had any price advantage there.

Well, that thread remains open and you are more than welcome to use it to speculate about the long term ramifications beyond Gen 5. ;)

I think it's fair to say that by the time we got to Gen 7 in that timeline the industry would likely look totally unlike what we actually got.

Why wouldn't a Wii HD appeal to lost Playstation customers?  It's a cheaper HD console.  Why do they automatically have to go to XBox360?  The whole premise is that Nintendo hardware is no longer a generation behind.  We are still assuming PS3 was a big fumble at launch right?  PS3 would still be $500/$600 at launch right?  Those lost customers would get XBox360 if they prefer FPS games like Halo and they'd go to Wii HD if they preferred adventure games like Mario Galaxy and Twilight Princess.

Also, here is why Generation 6 is different from Generation 7.  The PS2 had almost all of the games.  After a quick internet search I found these results for the physical game libraries.

PS2 had far more games than XBox or Gamecube.  This is why PS2 can sell far better than either one, even if Gamecube is a cheaper price.  Price still matters but game library matters more.  Now look at Generation 7.  PS3 and XBox360 have close to the same number of titles.  PS3 actually does have about 250-ish more titles, but XBox360 was usually cheaper to purchase.  When the libraries are that close then price matters a lot.

Since XBox360 was getting a similar number of titles to PS3, then why wouldn't a Wii HD as well?  Making multiplatform games became the norm in generation 7.  If there is a Nintendo console with similar HD specs, then why wouldn't it get a lot of these ports too?  A Wii HD would be very similar to the XBox360 in that it would be a lot cheaper than the PS3 and have a comparable number of games, because of multiplats.  Now add in the Virtual Console and Wii HD becomes a decently viable platform.  It would be competitive to both the PS3 and XBox360.

And considering Nintendo don't release consoles to lose money, they would either be much weaker than X360 or cost 100 USD more for the same performance. So no price advantage at all.

360 would appeal more to PS gamers by maintaining more of the "hardcore"/"grownup"/"cool" image they sought. Coming off the Gamecube, Nintendo's image in this department was poor. And again, you could get a 360 for the same price as a Gamecube 2.

Hardware sales lead to more games; they feed into each other but the former must come first, and I doubt Gamecube 2 would sell enough to get even close to as many games as PS3 or 360.

Yeah that is a fair point, though I think you get probably get 360 level performance at a profit for $300-$350 by end of 2006. I kinda feel it'd be unfair to change the OP at this point.

A cool image does not sell hardware.  Cool games do, but a cool image does not.  The Genesis had the coolest image of any console and it still lost to the SNES, which had a kiddie image.

At this point we are talking in circles about game library though.  I think the Wii HD would have gotten most of the multiplats that the PS3 and XBox360 had.  You don't.  It's kind of pointless arguing beyond that, because this is a hypothetical scenario and we'll never find out who is right.

No problem as I don't think WiiHD costing 50 or even 100 more than X360 would be the key point of it doing worse or not than what you predicted anyway. Just pointed out because Liquid was so damn sure WiiHD would have price advantage against it. MS is much more willing to eat loss on HW than Nintendo.

I would probably prefer a WiiHD than what we had, but for Nintendo I believe Wii was a much better ideia, only if WiiU had done a better job =[

I think "Gamecube 2" is a better description of this theoretical console than "Wii HD". The Wii after all was largely defined by its motion controls and blue ocean approach whereas what we're talking about here, a standard HD console from Nintendo, would be more like a traditional successor to Gamecube.

Fair enough. GC2 or Super GC then.

For your tastes would you have liked more the GC2 than Wii?

I think the Wii probably would've more to my tastes. While it would've been cool seeing Nintendo games in HD a gen earlier, I expect we would've had to suffer through a lot of the delays and droughts that made the Wii U so painful as Nintendo like a lot of devs had trouble transitioning to HD development. I'm also one of those who quite liked motion controls, when done right.