If we're just comparing the ability to do 3D engines then I think N64 was much better, despite the blurry graphics. Like many have stated, the N64 was designed to do convincing 3D better. For example, the N64 hardware had a z-buffer which allowed for 3D objects' images to remain stationary like they would in reality. For example, when you approach a statue in Tomb Raider and then proceed to walk around it to view it from various angles, the front of the image follows you as if the statue was rotating as you circle it.
Here's an excerpt from a dev interview than explains some other tech details which are beyond me:
"The N64 hardware has something called a Z-Buffer, and thanks to that, we were able to design the terrain and visuals however we wanted.
This was a huge advantage for them. In contrast, for Crash Bandicoot -- which came out for the PS1 at the same time -- we had to use over an hour of pre-computation distributed across a dozen SGI workstations for each level to get a high poly count on hardware lacking a Z-buffer.
A Z-buffer is critical, because sorting polygons is O(n^2), not O(n lg n). This is because cyclic overlap breaks the transitive property required for an N lg N sorting algorithm.
The PS2 got Sony to parity; at that point both Nintendo and Sony had shipped hardware with Z-buffers."
https://news.ycombinator.com/item?id=12558589
