That's strange. My experience with the 3D All-Stars version of the game was the opposite of yours. I came out of it amazed at how well they nailed the controls on their first try. Mario has an impressive set of moves and they all work flawlessly, crouch jump, long jump, triple jump, dives, ground pound, everything flows really well.
Then maybe my controller just isn't working or something (It does get drift but I didn't factor that into my opinion), because even something as simple as crouching for a backflip required like a full second's worth of waiting and holding the button. This doesn't sound like much but when you are in the middle of a fight with an enemy and need it, it's excruciatingly long.
I rarely seemed to go in the proper direction I was aiming (I'd press directly straight up and it'd veer to the left or right, etc), the camera made it nearly impossible to line up many of my jumps, even the slightest touch on angled surfaces resulted in me sliding off, which was compounded by the whole 'character tends to go in whatever direction they want rather than the direction I instruct them to' problems. I just found every element of controlling Mario in this game to be a frustrating chore, something you fight with as opposed to it coming naturally or him being an extention of the player.
I love Galaxy and Galaxy 2. I was astounded at how much I LOVED Odyssey. I even really liked Sunshine a lot and Mario 3D Land/World remain some of my favourite mario games. But Mario 64? I don't think it aged well at all.
Keep in mind I feel that way about pretty much everything on the N64/PS1 era, though. This isn't some 'I hate Nintendo' thing, it's just me saying that, during the 5th generation of consoles I felt we had the technology and skill to make 3D Games, but lacked the skill and technology needed to make it work well. I also heavily dislike Zelda Ocarina of Time/Majora's mask, as well. I can't even fathom playing Crash Bandicoot on its original platform. Spyro, too. Platformers on the 5th generation were all nightmares. I only like Final Fantasy VII-IX because they're turn based. if they were action/adventure games I'd likely hate them.
My Console Library:
PS4, PS3, PS2, PS1, WiiU, Wii, GCN, N64 SNES, XBO, 360
3DS, DS, GBA, Vita, PSP, Android
Top 6 this generation:
Bloodborne, Sekiro: Shadows Die Twice, God of War, The Legend of Zelda: Breath of the Wild, Dark Souls III, Red Dead Redemption II, Rock Band 4