In terms of software quality, which transition did Nintendo handle better, 3D or HD?

[Shadow1980]

Honestly, Nintendo & Rare were among the few major developers to do a good job in the transition to 3D. The industry writ large decided to almost fully abandon 2D games with sprite-based graphics in Gen 5 (Capcom with Mega Man & Street Fighter and Enix with Dragon Quest being exceptions). Instead of easing themselves in, they dived head-first into this new dimension despite few of them having any experience developing 3D games. It was a new and novel thing, after all. The end result was countless games with poor controls and bad cameras.

A lot of this was because of the controllers. The original PS1 gamepad was essentially just a modified SNES controller, so devs had to figure out how to create control schemes using just a D-pad and the face & shoulder buttons. That's how we ended up with cumbersome schemes like tank controls (which were never good, I don't care how many overly-nostalgic people say otherwise).

Nintendo built the N64 controller from the ground up to work for Super Mario 64 and 3D platformers in general. Having an analog stick as the standard was incredibly important, and was utilized in an intuitive manner. You press the stick in one direction, and Mario moves in that direction. The camera was intuitive to control with the C-buttons ("C" for "camera," obviously) and therefore fixed or preset camera angles weren't necessary anymore. It set the groundwork for what would become industry standards. While Nintendo abandoned C-buttons for a right stick as dual analog quickly became the norm, it's long since been the standard for control schemes that "left stick moves, right stick looks/moves the camera." It's what allowed 3D games to eventually progress pass the awful clunky controls that were so commonplace in the 90s.