The N64 had so many incredibly good games that is really hard to choose: Super Mario 64, Ocarina of Time, Majora's Mask, Perfect Dark, Jet Force Gemini, Conker's Bad Fur Day, Paper Mario... Even the "less ambitious" ones were peak in their genre at the time: Wave Race, F-Zero, Smash Bros...
But, if I had to choose one, It would be Majora's Mask. It's world, and the deep of his story and characters were (and I think still are) unparalelled. No game has provided the kind of catching stories tied to daily/hourly routines and the 3 day cicle like it did. I still wish someday we'll get a game with a 5 or 7 day cicle where you can explore not only the world but it's inhabitants routines for each day.
Aside of it, JFG and Conker were also very dense, complete and original on their proposal, but the charm, mystery and character of Majora's is of a diferent kind, really special. Plus, it's temples were awesome, even more complex and with more interesting mechanics than the ones in Ocarina. And Link's transformations, and enemies, and horse and goron and even dog races, and... God, there were so many awesome things on it.
I recognice the importance of SM64 and OoT as pioners, genre and mechanic definers, but MM took story, mechanical and gameplay depth one steep beyond.