I’m going to go against the stream, and vote HD.

So, I’ll make it abundantly clear I’m speaking for myself because my opinion is far from what most fans think 😀

I didn’t really start to enjoy Nintendo’s 3D efforts until the Wii. Yeah, Super Mario 64 impressed me, so did Wave Race 64, but I found that wore off quickly. I found myself primarily playing SNES (as far as Nintendo games went) during the N64 era.

Rare, on the other hand: IMO they stole the show on the N64. But Nintendo lost them in the end.

I did enjoy Mario Kart 64 and Lylat Wars, more than the 2D era, but that was it. I didn’t really enjoy the Super Mario series in 3D until Galaxy, and Zelda didn’t happen until the HD era with Breath of the Wild.

On the downside of HD, Nintendo flubbed the Wii U. But not on the software side, what they needed was a great console for that software, and we got it with the Switch. During that generation they developed Xenoblade Chronicles X and Breath of the Wild, which remain two of my favourite games of all time. Mario Kart 8 Deluxe somehow became my most played Mario Kart game of all time on the Switch.

So, IMO, neither transition was perfect. But, speaking for myself, HD is a clear winner.

Honestly, Nintendo & Rare were among the few major developers to do a good job in the transition to 3D. The industry writ large decided to almost fully abandon 2D games with sprite-based graphics in Gen 5 (Capcom with Mega Man & Street Fighter and Enix with Dragon Quest being exceptions). Instead of easing themselves in, they dived head-first into this new dimension despite few of them having any experience developing 3D games. It was a new and novel thing, after all. The end result was countless games with poor controls and bad cameras.

A lot of this was because of the controllers. The original PS1 gamepad was essentially just a modified SNES controller, so devs had to figure out how to create control schemes using just a D-pad and the face & shoulder buttons. That's how we ended up with cumbersome schemes like tank controls (which were never good, I don't care how many overly-nostalgic people say otherwise).

Nintendo built the N64 controller from the ground up to work for Super Mario 64 and 3D platformers in general. Having an analog stick as the standard was incredibly important, and was utilized in an intuitive manner. You press the stick in one direction, and Mario moves in that direction. The camera was intuitive to control with the C-buttons ("C" for "camera," obviously) and therefore fixed or preset camera angles weren't necessary anymore. It set the groundwork for what would become industry standards. While Nintendo abandoned C-buttons for a right stick as dual analog quickly became the norm, it's long since been the standard for control schemes that "left stick moves, right stick looks/moves the camera." It's what allowed 3D games to eventually progress pass the awful clunky controls that were so commonplace in the 90s.

Interesting to hear from the HD side of the discussion, cheers.

Wii U's first party output did impress me as a whole, even if some stuff like NSMBU or Nintendoland felt a bit too conservative, relative to how ambitious their N64 output was. BOTW and Mario Kart 8 though were top notch, Tropical Freeze while not a technical showpiece was one of the best platformers ever made, Splatoon was a fresh and exciting new IP, and Xenoblade X while it had its flaws delivered one of the most striking open worlds in gaming.

On the N64 side of things, Mario 64, Ocarina of Time, and Lylat Wars blew my mind back in the day, more than anything on Wii U sans BOTW and XCX. F-Zero X also deserves a mention for pushing an insanely fast racer at 60fps.

If we're talking commercial - it's easier to show because it's more based on data
The transition to 3D was rough, and not as commercially successful as their past 2D efforts on the NES and SNES, nor was it as successful as their present 2D efforts on Gameboy and GBA - so the decline became so bad that the home console front represented just a small fraction of Nintendo's revenue, despite being its biggest cost - and was the cause of Nintendo's first financial quarter to dip into the red for the first time in 50 years. While Wii U in isolation was a larger failure than the N64 (landing Nintendo in the red for the full year), it was also their shortest generation. And (unlike the N64) did not mark a true collapse, as the Switch was a swift and massive recovery. But not only did the N64 end Nintendo's decade of dominance and forever ruined their brand power (before being synonymous with video games), but Nintendo continued to struggle and decline for another five years on the home console front. Nintendo was 2 for 2 in the 2D era on home console, and 2 for 2 on handhelds. It really took Nintendo 10 years before they found commercial winner since the transition to 3D, as both N64 and Gamecube were losers. Nintendo had a winner in its second generation of HD.

As far as software goes
Nintendo has 21 games on Switch that have exceeded 10 million sales - and that number will continue to grow. Mario Kart 8 Deluxe and Animal Crossing: NH have a combined sales of over 110 million units.
Nintendo only has 10 games above 10 million for the entire 3 generations of the 3D era, and only one of those games (Super Mario 64) is not a Wii game.


Then there is the handheld front
Nintendo managed a hybrid console that had HD graphics within 4 years. That hybrid console is close to becoming the best selling video game console in history.
It took Nintendo twice as long, 8 and a half years, before 3D was a thing on handheld - and while it was a great success (far more successful than the N64), it was an additional 2 years, 10 years total, before they got hot. But Switch was hot out the gate. While the 3DS was a winner, it was also Nintendo's least successful handheld console.

Let me clear, my personal opinion is that the 3DS was a great handheld, but it had a lot going against it: anti-3D hysteria (like weird press freaking out), and mobile gaming coming out strong around this time, Nintendo also miscalculated the launch (a lot because they had their eyes off the negative 3D press and outside industry competition). But numbers are numbers.

They handled 3d very well. HD very poorly. UhD non existent anf VR... c'mon. Do it already... again. Just try ye, oh innovators of innovation.

To us hardcore gamers, Nintendo Switch outpacing PS2 as a console will never count. You can't combine handheld with console and expect to say you neat the competition. Ps2 and Ps3 needs to be combined with PSP for comparison. DS + WIi kills all on those terms still but I'll never accept the Switch as a home console.

I think that's where we differ, that for me the N64 lustre wore off quickly. And I found the games somewhat lacking compared to their 2D counterparts. But the HD games I find to be big improvements over their 3D era counterparts - well, except Mario Odyssey, which I like considerably better than Mario 64 and Sunshine 3D, but not nearly as much as Galaxy 1 and 2.

I liked 3D era Zelda at first. But then it came to represent everything I disliked about 3D era games. And while I did like Ocarina of Time during the N64 era, I was still disappointed with it for two reasons: first, I didn't enjoy it as much as Link to the Past; second, it fell very short of my expectations - Breath of the Wild exceeded my already crazy high expectations. Lylat Wars was fantastic - but it might be the only N64/Gamecube game by Nintendo I'd consider playing again. I can scarcely imagine I won't want to play many games from the Wii U/Switch era again in 10-20 years time.

That's as flawed as saying PS2 sales shouldn't count because it doubled as a DVD player. It's not a valid argument to disqualify a console's sales numbers based on a key feature that contributes to its success. Being "hardcore" is irrelevant.

HD.
Mario Kart 8 as a "HD" game has sold more copies than Nintendo 64 and Gamecube console sales combined.

Ok, I have time now for a more thorough reply.  I just want to remind everyone that the question is not "N64 vs. Wii U, which is better?"  The question is "which transition did Nintendo handle better 2D -> 3D or SD -> HD?"  If you actually look at this question with the benefit of hindsight, the answer is unquestionably     SD -> HD.  They actually handled this transition better than any other company.  However, when going from 2D -> 3D they were in the bottom half of major companies (although Sega was definitely even worse).  And yes, I am talking about software quality, both from my own personal taste and how their software was received by gamers in general.

2D -> 3D Transition
One reason why Nintendo handled this transition so poorly was that they were the king of 2D software.  Here were the major 2D software makers that transitioned to the 3D era: Nintendo, Sega, Konami, Capcom, Enix, Squaresoft, and Rareware (maybe a couple of others I'm forgetting).  The PC/home computer as a platform was also significant going from 2D -> 3D.  Out of these companies, which made the best 2D games?  Nintendo, unquestionably.  Not only were they my favorite software developer at this time, but they dominated the marketplace, and when I was conducting my "Greatest Games of All Time" series of threads, people voted for Nintendo games as the best of almost every year during the 8-Bit and 16-Bit era.  Nintendo was the king of 2D software.

The PC is interesting during this time period, because the PC was always ahead of consoles in terms of graphics (by a very noticeable amount), and yet PC software was completely overshadowed by consoles.  Consoles just made better action games than PCs did, and Nintendo was the absolute king of console software.  Then PCs made a major transition to 3D software in the early 90's.  This really started with Myst.  Since 3D gaming can create these huge immersive worlds, non-action games like Adventure games and RPGs greatly benefit from going 3D.  FPS games also became significant since this genre was created from the ground up for 3D.

On the console side Sony enters the fray, with their major software being Gran Turismo, a racing game.  Racing games also greatly benefit from transitioning to 3D.  Final Fantasy (a non-action RPG) greatly benefits from 3D.  Rockstar goes from being this little known studio to a huge overnight success because they transitioned from 2D to 3D, showing that open-world action games are hugely popular in 3D.  Blizzard, ironically, becomes a major studio during this time period by focusing on 2D games like Warcraft, Starcraft, and most importantly, Diablo.  During the 8/16-bit eras PCs could not compete at all with console action games, but when consoles go 3D, millions head to the PC to play Diablo, a 2D action game.  This is because action games are not as fun in 3D outside of a few genres (like FPS, Racing, open world).  Precise controls, timing, reflexes, twitchiness, etc... tend to be more important in most 2D action games and that is harder to replicate in a 3D action game.

This brings us to Nintendo and Sega, the two best developers of 2D action games.  Sega really crashes and burns transitioning to 3D.  If you look at the list of "must play Saturn games" the top one is usually Nights into Dreams which is a 2D game.  Another big one is Panzer Dragoon Saga, which is an RPG (a non-action game).  Most of the skills Sega developers had in 2D game development did not transfer to 3D game development, especially with action games.  All of this can also be said about Nintendo, since Nintendo and Sega were very similar companies.  Nintendo had to develop all new skills to develop 3D action games, and this basically leveled the playing field between them and other developers.  Leveling the playing field was a bad move for Nintendo, because they were #1 for 2D game development.  They went from king to just another developer.  Platforming went from the #1 genre in the 2D era to a low to mid tier genre in the 3D era.  Outside of 3D Mario, what 3D platformers are popular?  Like none.  (Maybe Astrobot which just released, lol.)  Outside of 2D Mario, what 2D platformers are popular?  Sonic, DKC, Castlevania, Contra, Mega Man, Cuphead, etc....  Platforming is inherently more fun in 2D.  Most action games are inherently more fun in 2D.  

I personally do not like most 3D Nintendo games.  However, even for people who do like these games, you have to realize that Nintendo was clearly #1 for 2D software development, while in the 3D era most people preferred some other developer over Nintendo: Squaresoft, Rockstar, id Software, etc....

SD -> HD Transition
IMO, Nintendo handled this transition better than any other major developer.  This is because they developed a very different strategy during this transition.  They delayed.  All other major companies transitioned as fast as they could, and this was the wrong move, because the transition from SD to HD is about development times slowing down.

A key part of this transition is the Wii.  While everyone else was getting their footing going to HD, Nintendo just output a ton of SD games on the Wii and seemingly beat the entire gaming industry single-handedly.  This allowed them to stockpile cash.  At the same time other software companies release schedules slowed down a ton, because that is what happens going to HD.  Nintendo had to take this hit too with the Wii U, but it was a hit that every other major company also took.  They were smart in that they took it at a different time.  Then they could use all of this surplus cash in HD software development.

There are several reasons why the Wii U failed, but it should be fairly clear that software quality is not one of them.  (Quantity? yes.  Quality? no.)  BotW is far better than any previous 3D Zelda.  3D World is probably my favorite 3D Mario.  I personally liked both Hyrule Warriors and Mario Maker a lot.  I was critical of Mario Kart 8 at first, but now it has so much content that I'd have to say its the best Mario Kart so far.  There are other games I haven't played yet, but they are their most successful version is their first HD version like Fire Emblem, Luigi's Mansion, and Animal Crossing.

Part of the SD -> HD transition is in making their handheld franchises now available in HD on the Switch.  Combining their two platforms into one might be the smartest thing Nintendo has ever done.  So even their franchises which were normally handheld like Pokemon and Animal Crossing are now available in HD like a home console.  And they are sitting on so much cash from the Wii years (and now the Switch years too), that they can just keep expanding software development at a rate they are comfortable with.

Basically when transitioning from SD -> HD, Nintendo did better than any other company.  They output lots of really popular software now, and no other company has nearly as many high quality franchises (not even close).  Basically, they have the same status in the game industry that they had when they were the #1 2D software company.  They lost this status transitioning to 3D, because other software developers were seen as equal or better than Nintendo during this time period, but now in the HD era, Nintendo is the king of gaming software again.