3D by far. One of the many reasons the WiiU did not succeed was the long waits between 1st party titles.

The transition to 3D was screwed over by cartridges scaring off third-party publishers. Nintendo's software itself was not disastrous. 

....In other words, Wii U v N64? I feel this question has an obvious answer Lol. Yeah, you can say Nintendo made some screw ups with cartidges and allowing Sony to run away with PlayStation, but Wii U had the gamepad (devs did not like it), weaker hardware (devs did not like it), far more disasterous commercial failure, lacked much first-party support (only had like ten notable releases: Splatoon, Mario Maker, Mario 3D World, Captain Toad, Smash Bros 4, Pikmin 3, MarioKart 8, Bayonetta 2, Xenoblade X, and prolly missing smth), and Animal Crossing Amiibo Festival + Mario Tennis Ultra Smash + Star Fox Zero + NSMBU + massive droughts during 2013 & 2016, etc etc etc.

I dont really feel these are comparable, its clearly 3D the one

What makes you say they did a bad job with 3D? 

Well, some of those first 3D games are still among my favourites of all time, so I guess they did something right there. Kind of a tough question though.

Considering that the N64 games are so legendary, it's gotta be 3D... easily.

Why do you want to argue with me on this?  Are you old enough to remember when Super Mario World was a brand new game or at least Donkey Kong Country?  You have to have at least played the SNES when it was a current console to get how bad the N64 really was.

Ran through the old reviews with a few series, and in both generational transitions, series that were represented on the consoles in question had slightly worse reception on average than their predecessors. However, the SNES to N64 transition scored worse overall, with critical declines being most noticeable with Yoshi and Kirby.

How is there even a debate ?

Nintendo is probably the company that managed the transition from 2d to 3d (software) the best in the entire World.
Mario 64 ? Revolutionised its franchise. Instant classic that defined the 3d platforming genre for years.
Zelda OOT ? Revolutionised its franchise. Instant Classic that defined action adventure 3d games for years. Still considered by many the single greatest game of all time to this day.
Metroid Prime ? Revolutionised its franchise. Instant classic that pretty much created its own genre. Level/world design so influential and ahead of its time it's frequently cited by the most critically acclaimed game designers nowadays.

Meanwhile most companies where struggling and failing hard at the same task with their most famous franchises.