Galaxy 3.

And it would deserve it.

smg3

I think Galaxy 3 would have done better, but let's not forget that Galaxy had solid marketing as part of the Wii Would Like To Play campaign while 3D World got...basically nothing until the Christmas "Pitch" ad campaign that was so abysmal it probably scared people away. So while I think it would have sold better, if ALL you did was swap the titles, no advertising changes or anything, it you would likely see a more modest advantage the many assume.

Galaxy 3 much hyped!

Super Mario Universe

I think sales-wise the way they chose was the worst one, I think both a Galaxy 3 or a Mario 64-type game would have sold more. Not in extreme amounts, but probably more. Needless to say though, 3D World hasn't stopped selling yet, and we should wait for Nintendo's figures this month (if they update this game).

I think a Galaxy 3 would suffer from a 'been there, done that' reception, but it would be considered a 'real' 3D Mario. 3D World lacks that perception, on top of it being viewed as 'just an upscaled 3DS game'. A 64-style game, of course a modern, innovative take on it, would be a refreshing change of pace for Mario because the last time they done the franchise that way was Sunshine on GameCube, which didn't really work out the way it should. It would probably also be the most welcomed style by the mainstream, with it being semi-open world and as such much grander in scope. The current N64 nostalgia in general would also help it.

Galaxy 3.

There's no way to know of course, but one thing's for certain, had they done a Galaxy 3 my Wii U would be 3D Marioless.

64-2.
Everyone was expecting an ambitious game like that.

3D World would've sold like 15+ million copies on the original Wii, maybe 20+ million.