It's a very tricky question to answer.
The N64 had more RAM, more raw processing power, and generally a much more 3D-suited architecture. It did texture filtering and texture crossfading, and it didn't do all the miscalculations of 3D geometry the PS1 was infamous for, which lead to the charming "popping" you'd see in many of the games.
It just didn't push as many polygons on screen as the PS1, though, and didn't allow for any single texture file to be any larger than 4 kilobyte. This meant a lot of low-color low-res textures in N64 games, and a lot of games hitting draw call limits early, leading to stuttering frame rates.
The whole thing meant that games on PS1 often had smoother framerates and bigger, prettier textures than the N64 counterparts, despite them jittering a bit and not being filtered.
The N64 -could- run textures unfiltered, which could mean better frame rates for the titles that did exactly need it. But Nintendo weren't at all keen with allowing it. Games on the system had to flaunt the feature.
Lastly is Cartridges vs. CDs. Obviously Nintendo's choice meant voice acting for the most part was axed, music was usually MIDI (towards the end they did get MP3s running on the thing) and FMV was an incredibly rare thing to see. This did mean very short load times for everyone, however, but it wasn't a trade-off many developers appreciated, what with having to amputate their games AND their budgets, do to the relatively sky-high price of production for cartridges compared with CDs.
I guess in the end they were quite different systems. From a purely processing numbers perspective, the N64 is strongest, but quite many hardware design choices cause bottlenecks that held it back gravely.












