It probably would have done very poorly. With Mario 64, this was the first adventure into a full 3D experience (Other 3D games before generally were on rails or had the user automatically navigated along the path). If that game came out, it would have been in the shadow of other very fresh feeling 3D games, and even in the Shadow of Rare's Donkey Kong Country games, and games like Kirby Superstar.
On the otherhand, when New Super Mario Brothers came out, it was in a time when 2D platformers were a large rarity. Although, people who think these sold systems need to look at the stats - for DS, sales were actually down in 2 of the 3 markets the week that NSB launched; and heavy sales were largely due to pack-ins. People mostly buy 2D Mario because they already have the system, or are just buying the system and want a recognizable Nintendo franchise to play when they get it - they probably don't buy a system for 2D Mario. On the otherhand, people bought N64 for Mario 64, and it was almost a year before sales picked up again (Around GE007, Diddy Kong Racing, and Mario Kart 64) after that initial launch period; and by then it was way too late.
I describe myself as a little dose of toxic masculinity.
