If we assume the HD Wii would lose all of its casual titles, then we have to assume it would get some major HD games. Perhaps it would get a Zelda Breath of the Wild, or a Splatoon. The Wii got a lot of quality games in its first years, so the HD Wii would have gotten even more. Consoles sell due to games, but the HD Wii would have lost the blue ocean crowd. So I will call it a narrow loss, but still satisfactory performance. Overall 7th gen sales would be much lower, and the PS2 would have gained a bit by being the cheap popular casual console. Due to the PS2 continuing to do well and the PS3 being the most expensive out of 3 expensive consoles it would have lost even more. 360 would be the eventual winner with kinect and more momentum early without a cheap Wii.

So without further ado here is my estimates.

PS2- 172m
360- 85m
PS3- 73m
Wii HD- 67m

The reason I decided to use HD Wii instead of Gamecube 2 is because I am assuming Nintendo's marketing would be on point for this theoretical gen unlike in the Gamecube era. Naming it Gamecube 2 would have been a disaster, due to the perception of Nintendo at the time. Of course this theoretical console would have had it's own brand, but I choose to use the stronger brand to better represent the superior marketing Nintendo had at the time.

I didn't mean it'd literally be called Gamecube 2, more that it would be a continuation of that system's philosophy of playing the power game extended into the HD era. The name "Wii" on the other hand was chosen to reflect the blue ocean nature of the system, something this theoretical console won't be embracing as described in the OP.

Again, it wasn't like 3rd parties were going all in on making multiplats for the first couple of years of the generation. Bioshock 1 was exclusive to XBox. So was Mass Effect 1. Assassin's Creed only went multiplat because of the 360's success in its first year. It took several years for most of the big third parties to settle on making everything multiplat. A Gamecube 2 with hardware comparable to the 360 would get COD 4, but it would still miss out on many of the biggest franchises of the generation, and it would need to sell in that crucial first 2 years when 3rd party support would be all but guaranteed to be abysmal and Nintendo itself would be struggling with HD development. Maybe the sequels like Mass Effect 2, Bioshock 2, Assassin's Creed 2, would come over. But the first games would almost certainly not come to Nintendo and by the time any of the sequels came out the system would be dead in the water and possibly not worth bringing anything over to.

Considering:

- Nintendo was in a disadvantaged postinion in terms of image compared to the PS brand at its peak and the growing Xbox brand.
-The slow momentum of the Gamecube in 2004/2005
- Their smaller avaliablity of marketing resources compared to the other 2 larger companies
- Their lack of alternative business braches to mitigate hardware losses
- Their lack of an online infrastructure to compete with Xbox live
-Increased costs and production time of their 1st party content (due to increased production values and)

It would have been a bloodbath for them.

I propose another alternate story:
Wii U released 2010 or 2011, without the gamepad.
2 version:
1-white, with wiimote+ inside and wii sports club(just the wii version in HD);
2-black, with pro controller.
250$~300$.
In 2012/13 release the gamepad to be bought separately, along with games that use it(all adapted also to be used without it).
Also a wii U version with gamepad included and a game.

Any chance to outsell at least N64?

Could outsell N64 but I don't think so. It would be at least 100USD to expensive in comparison with WiiU original timeline.

Why would anyone pay 300 USD for the WiiU with wiimote and low performance?

In my mind I am trying to do as minimum of a change as possible.  All of the "Wii ____" games just disappear, like Wii Sports, Wii Fit, etc... as well as any game that relies highly on motion controls like Wario Ware and Fling Smash.  These get replaced with nothing to reflect the higher development costs of HD.  On the other hand a lot of other games that have motion controls just get conventional controls: Mario Kart, Wario Shake It, New Super Mario Bros Wii, etc....  Twilight Princess is their big launch title, and all of their games get HD graphics.  Visually they would probably look a lot like the Wii U versions.

Given all of that, their first party has some heavy hitters that the Gamecube did not have.  NSMBW has sold 30m+ copies and Mario Kart has sold more than that.  Those games would still be popular if you took away motion controls.  The Virtual Console also would be a big hit with old schoolers.  Basically if you take out motion controls they lose the new motion control crowd, but they would still get a lot of old school gamers with both the VC and games like 2D Mario, Donkey Kong Country, Mario Kart, etc....  It would not be a Gamecube 2.  And because it would have good sales, then it would get a lot of HD multiplat games.  I stand by my assertion that the generation would be close to a 3 way tie.

I do like your sales assessment including that the PS2 would go up in sales.  The lack of cheap console would help the PS2 even more (and maybe the DS too).  I was thinking more like an extra 5-10m though.

Under the circumstances, I think going the Wii route as they historically did was clearly the right move for Nintendo. They needed a refresh, the same old approach just wasn't working for them anymore.

And indeed, it paid off handsomely, with Wii going on to not only win the generation but become one of the highest selling consoles of all time.

Of course, they then took a bunch of stupid pills in 2011 that took until 2017 to wear off, but hey, every one of the big three have shit the bed at some point, doesn't negate the times they hit home runs.

A HD Gamecube 2.0 would've been far less successful than the actual Wii both in sales and profitability, and I think ultimately PS3 would've come out on top, with 360 in second place and Nintendo in last.