Saw this one posed on gaf and wondered what the VGChartz community thought; if the PS1 doesn't enter the 5th gen hardware race, who would have won the 5th generation of consoles, and what's your reasoning for why?

I'm guessing the N64 wins.

Without the threat of PlayStation, the Saturn would likely only have a single SH-2 chip instead of two. This would have decreased it capabilities, especially for 3D games, but would've likely lowered the manufacturing price by a bit (perhaps $20). From there, we can assume that the Saturn and N64 are launched at similar times as in OTL, save for an actual September launch for the Saturn in America.

Without the second SH-2 chip, the Saturn would've done a much worse job running 3D games. This wouldn't have been too problematic early in its life, since many early games didn't even use both chips, but even by late 1995, games like Virtua Fighter 2 and Sega Rally would not look or run as well as IRL. Considering how these were some of the Saturn's biggest games, this is a big problem.

These publishers would likely support Sega more

• Capcom - Capcom's big games in this era were pretty evenly split between their 2D efforts, especially their fighting games (Capcom released 15 or so fighters on Saturn IRL), and their 3D efforts like Resident Evil, which even IRL saw a Saturn release. Resident Evil's 3D nature might seem like a better fit for N64, but given the game's development started in 1993, Capcom would likely not want to endure the development hell of the N64 dev cycle. Not to mention that the FMV and voice acting wouldn't fit on early N64 cartridges.
• Eidos - Tomb Raider actually was released on Saturn first IRL. That series alone would give Saturn some momentum.
• Electronic Arts - EA made a lot of money on Genesis, and would likely see the Saturn as the easier transition compared to N64. EA released a couple of Saturn games in 1995, and published around 10% of the Saturn's US library IRL before giving up by the end of 1997.

These publishers would likely support Nintendo more

• Activision - Activision only started becoming big when Kotick tok control in the 90's, and was PC-focused early in Gen 5. From about 1998 onwards, their Spider-Man and Tony Hawk's Pro Skater games were big hits, and these games would work better on the N64 than the nerfed Saturn.
• Enix - Enix was behind Nintendo IRL as late as 1997, when they declared plans to publish games for both PlayStation and N64. Dragon Quest VII was in development for the N64 until 1997 IRL, meaning it would likely have released sooner in this scenario. Note that the devs wanted to focus less on storytelling and cinematic presentation, which would otherwise push them to the CD-based Saturn.
• Namco - Sega had two nemeses: Nintendo at home and Namco in the arcade. Out of the 1000 or so Saturn games released in Japan IRL, NONE of them were published by Namco. Any Namco games that went to home consoles in this timeline would be on Nintendo. This includes Tekken, Ace Combat, Ridge Racer, Time Crisis, Soul Edge, and Tales of.
• THQ - WWF and licensed children's games were big during this era, and seeing how both found a sizable home on N64 IRL...

Up in the Air Publishers

• Konami - Games like Metal Gear Solid and Castlevania: Symphony of the Night would almost certainly go to Saturn. But even IRL, Konami put out a number of big N64 games.
• Square - Square's devs were attracted to the CD-ROM's capabilities, and basically switched from SNES to PS1 within a space of a few months in 1996. Saturn would offer the better platform for more cinematic stories, but Square would be sticking with Nintendo rather than Saturn through 1996 anyways...

Conclusion:

• In real life, in the second half of the 90's, about 37 known games were released that eventually sold 5+ million copies (including on PC and handhelds, with some sales totals extending past 1999). Of these, 14 were Nintendo-published and 0 were Sega-published. In a theoretical sorting of the PS1's greatest hits, the N64 would receive the Tekken series and most of the big RPG's. Saturn would receive the Resident Evil and Tomb Raider series. Games like Gran Turismo would never exist.
• In this scenario, Nintendo still releases a similar lineup to OTL. In contrast, Sega's output is held back by their weaker hardware compared to OTL. So many of the biggest Saturn games IRL, which were sold in large part by their technical prowess? They're going to fare worse. This includes Virtua Fighter 2 and other 3D fighters, Sega Rally, Virtua Cop, and presumably Sonic X-treme.

Prediction: Saturn does better in the West, since the N64's newfound JRPG stranglehold is less important, but N64 outdoes it in all major regions.

People forget that Saturn outsold N64 in Japan. There would be a possibility of capturing even more Japanese support - solidifying it's position in the land of the rising sun. In turn this would affect its global position. I think it would still be a N64 victory but a much closer race.

Really I think it depends on how much of the PS1 library would still exist and be transferred to the Saturn.

Would the likes of Crash, Tekken, Ridge Racer or FF7 have been there? Would they be the primary platform for stuff like Resident Evil instead of being a late port? Would Gran Turismo exist still and be a Saturn title?

Without a PS1 how many units would the Saturn have sold without competition for 2 years before the N64 launch?

It also still had the advantage of games only costing around $50 while N64 titles could go up to $80.

Launching the same year and being undercut by $100 I believe the PS1 did far more damage to the Saturn than the N64.

Sega Saturn would take all third party games of Playstation due to the cd-rom capability.

So Sega Saturn is the winner.