Between Zelda and TR, Zelda definitely looked better. Probably the average N64 game, especially the average EAD or Rare game, held up better than the average PS1 game. Mario and Banjo probably hold up the best because of the cartoonishness of their visuals.
Final Fantasy still holds up pretty solidly, though, especially FF9, and the battle sequences to all three games actually don't look too bad today. Final Fantasy and Resident Evil had those pre-rendered backgrounds, and a lot of those vary in quality.
A lot of it had to do with the artistry of the designers as much as the hardware.. Rare made Banjo-Kazooie look really good, but the human models in Goldeneye and Perfect Dark were as blocky as the models in Virtua Fighter 1. I liked the human models in Tekken a lot better. A lot of N64 games looked had blurry, faded texture-mapping. The PS1 had sharper colors.