There is nothing that indicates that the demand for 2D games lessened with the arrival of 3D games. Only the amount of 2D games.

Kirby 64 sold well for a Kirby game.
Mega Man X4, if the VGC numbers are to be trusted, sold alright.
Symphony of the Night became the best-selling game in the franchise, which is something.
Rayman sold really well for a new unproven IP without a massive marketing campaign revolving around it. (Looking at you, Sonic)
And Yoshi's Story sold well too, for a shitty spin-off with narcotic overtones.

And there's pretty much every serious attempt at a 2D platforming, action, adventure or action-adventure game I can think of that sold well. (Didn't include fighters because I don't feel like doing research on the quadrillion fucking Street Fighter spin-offs released 5th gen) Every other game either had a publisher who failed to adequately market or distribute the game for one reason or another, or just... well, sucked. A Super Mario Bros. game published by Nintendo wouldn't have had either problem.

I think that it would have sold much more poorly than most here expect. At the time, 3D was the selling point for the N64 and if the game wasn't in 3D people would have asked why the lazy people at Nintendo weren't trying to put out a "modern" platformer like Crash Bandicoot.

usrevenge has a point. Numbers don't lie, but the interpretations some attach to them do.

He wasn't looking at it from a fanboy PoV from what I understand from his post. What he means is that some things can sell very well it doesn't make them good. For example, tamagochi sold very very well some 10-15 years ago, but it wasn't anything to die for, and has faded into memory lane. Other games may sell a little less well, but are true classics. Mario 64 is one of those imho.

What he means by most talked about is that people actually respect the experience the game brought to them. One such example is the Final Fanatasy series. True it doesn't have the sales of games like Super Mario Brothers, but the fans of the series are very vocal about their love for the game. Which is better? it's hard to tell, but sales are by no means an indicator of anything else than popularity. And popularity =/= better, sorry. It can be true, but it isn't necessarily true.

Anyways, back to the actual topic------------------

Had Nintendo released a 2D Mario on the N64, I can guarantee you the architecture of both the controller and the machine would have accomodated it, so would the artistic composition of the teams. Nintendo would have had a hybrid 2D/3D strategy in its software and HW R&D to bring a NSBM64 to success. History shows that things didn't work that way. Also, SMB at the time of the N64 had not yet entered the realm of unbound Mario which mostly started as of Super Mario Bros Advance (Mario Bros 2 on GBA). That's when the world of Mario started really experimenting with a much more seamless casual experience. So even then I don't think 2D Mario was yet ready for a revival. The likes of Paper Mario and Kirby 64 was only the begining of what would become Nintendo's future into 2D success. And these games are arguably interesting. They both were good games, but had nowhere near the appeal of games like SMB Advanced, NSMB or DKR. As I said, it was just the beginning. Hence why I believe NSMB64 would not have occured at the right time in Nintendo's evolution.

The N64 maybe could have pushed 37-38m sold, but 2D Mario was not the trend of the time, would not have done as well as it would have on NGC.

At launch? Hard to say.

Christmas 1997? Probably would have done pretty well.