Many franchises did not make a very good jump from 2D to 3D, and it ended several of the franchises. Fortunately for Nintendo, they released Super Mario 64 with universal acclaim. It was seen as a very successful move, and Nintendo would follow it successfully with other franchises such as Zelda and Metroid in the future.

However, what if Mario 64 wasn't such as a great game, and it was terrible? What if it was like the new Sonic games, where would the Mario franchise be now?

It would have just stayed 2D, most likely. They probably would have followed it up with a 2.5D title and went from there. Maybe down the line they would have still tried a 3D Land/World thing where it's 3D but no free-roaming. This would all obviously be a disappointing thing, seeing as the 3D games are awesome, but at the same time it would probably mean more innovation for the 2D games over the years. Because in my opinion, every NSMB game has played it too safe and relied too much on nostalgia. If there had never been that break while 3D was king, they would have just kept advancing.

A casual glance at the N64 controller shows they were already hedging their bets. I assume the failure of the game would have simply made them revert back to largely 2D titles for the series instead, although that's not completely clear seeing as how the 2D games continue(d) to outperform the 3D ones, but 2D disappeared for over a decade.

PlayStation would have outsold N64.



Also, Super AIDS.

Well, in that case, they would just continue with 2D Mario and focus less on 3D

I wonder what ramifications it would have on gaming as a whole. 3D gaming could be way different.

Mario is too big for a single failure as has been shown. They'd try it again and eventually succeed like they did. Might actually be better. 

yeah, video games without open environments and the camera would be static and fixed.

oh whait... 

It would have been terrible for the N64, but maybe better for Nintendo as a whole because it likely would have meant the N64 would have died earlier, thus they would have been forced to released a optical disc format console like several years earlier and may have been able to stem their losses.