Of all the generation match-ups I think this one is the most difficult to decide. It's really close.
Both have great libraries, with some strengths and some weaknesses. N64 was relatively weak in terms of RPGs and fighting games, but was really strong with platformers, shooters, and action-adventure games. Conversely, PS1 was strong with RPGs, fighters, and racing games, but relatively weak with platformers and shooters.
Overall PS1 has a bigger and probably more diverse library, but the top-tier games on N64 are so damn good that I have to give it to Nintendo's fifth generation console.