Saw this one posed on gaf and wondered what the VGChartz community thought; if the PS1 doesn't enter the 5th gen hardware race, who would have won the 5th generation of consoles, and what's your reasoning for why?

I'm guessing the N64 wins.

Without the threat of PlayStation, the Saturn would likely only have a single SH-2 chip instead of two. This would have decreased it capabilities, especially for 3D games, but would've likely lowered the manufacturing price by a bit (perhaps $20). From there, we can assume that the Saturn and N64 are launched at similar times as in OTL, save for an actual September launch for the Saturn in America.

Without the second SH-2 chip, the Saturn would've done a much worse job running 3D games. This wouldn't have been too problematic early in its life, since many early games didn't even use both chips, but even by late 1995, games like Virtua Fighter 2 and Sega Rally would not look or run as well as IRL. Considering how these were some of the Saturn's biggest games, this is a big problem.

These publishers would likely support Sega more

• Capcom - Capcom's big games in this era were pretty evenly split between their 2D efforts, especially their fighting games (Capcom released 15 or so fighters on Saturn IRL), and their 3D efforts like Resident Evil, which even IRL saw a Saturn release. Resident Evil's 3D nature might seem like a better fit for N64, but given the game's development started in 1993, Capcom would likely not want to endure the development hell of the N64 dev cycle. Not to mention that the FMV and voice acting wouldn't fit on early N64 cartridges.
• Eidos - Tomb Raider actually was released on Saturn first IRL. That series alone would give Saturn some momentum.
• Electronic Arts - EA made a lot of money on Genesis, and would likely see the Saturn as the easier transition compared to N64. EA released a couple of Saturn games in 1995, and published around 10% of the Saturn's US library IRL before giving up by the end of 1997.

These publishers would likely support Nintendo more

• Activision - Activision only started becoming big when Kotick tok control in the 90's, and was PC-focused early in Gen 5. From about 1998 onwards, their Spider-Man and Tony Hawk's Pro Skater games were big hits, and these games would work better on the N64 than the nerfed Saturn.
• Enix - Enix was behind Nintendo IRL as late as 1997, when they declared plans to publish games for both PlayStation and N64. Dragon Quest VII was in development for the N64 until 1997 IRL, meaning it would likely have released sooner in this scenario. Note that the devs wanted to focus less on storytelling and cinematic presentation, which would otherwise push them to the CD-based Saturn.
• Namco - Sega had two nemeses: Nintendo at home and Namco in the arcade. Out of the 1000 or so Saturn games released in Japan IRL, NONE of them were published by Namco. Any Namco games that went to home consoles in this timeline would be on Nintendo. This includes Tekken, Ace Combat, Ridge Racer, Time Crisis, Soul Edge, and Tales of.
• THQ - WWF and licensed children's games were big during this era, and seeing how both found a sizable home on N64 IRL...

Up in the Air Publishers

• Konami - Games like Metal Gear Solid and Castlevania: Symphony of the Night would almost certainly go to Saturn. But even IRL, Konami put out a number of big N64 games.
• Square - Square's devs were attracted to the CD-ROM's capabilities, and basically switched from SNES to PS1 within a space of a few months in 1996. Saturn would offer the better platform for more cinematic stories, but Square would be sticking with Nintendo rather than Saturn through 1996 anyways...

Conclusion:

• In real life, in the second half of the 90's, about 37 known games were released that eventually sold 5+ million copies (including on PC and handhelds, with some sales totals extending past 1999). Of these, 14 were Nintendo-published and 0 were Sega-published. In a theoretical sorting of the PS1's greatest hits, the N64 would receive the Tekken series and most of the big RPG's. Saturn would receive the Resident Evil and Tomb Raider series. Games like Gran Turismo would never exist.
• In this scenario, Nintendo still releases a similar lineup to OTL. In contrast, Sega's output is held back by their weaker hardware compared to OTL. So many of the biggest Saturn games IRL, which were sold in large part by their technical prowess? They're going to fare worse. This includes Virtua Fighter 2 and other 3D fighters, Sega Rally, Virtua Cop, and presumably Sonic X-treme.

Prediction: Saturn does better in the West, since the N64's newfound JRPG stranglehold is less important, but N64 outdoes it in all major regions.

People forget that Saturn outsold N64 in Japan. There would be a possibility of capturing even more Japanese support - solidifying it's position in the land of the rising sun. In turn this would affect its global position. I think it would still be a N64 victory but a much closer race.

Really I think it depends on how much of the PS1 library would still exist and be transferred to the Saturn.

Would the likes of Crash, Tekken, Ridge Racer or FF7 have been there? Would they be the primary platform for stuff like Resident Evil instead of being a late port? Would Gran Turismo exist still and be a Saturn title?

Without a PS1 how many units would the Saturn have sold without competition for 2 years before the N64 launch?

It also still had the advantage of games only costing around $50 while N64 titles could go up to $80.

Launching the same year and being undercut by $100 I believe the PS1 did far more damage to the Saturn than the N64.

Sega Saturn would take all third party games of Playstation due to the cd-rom capability.

So Sega Saturn is the winner.

That is a tough one. Saturn had CDs but a considerably worse first party output compared to the Genesis and Dreamcast. It was expensive, without the price drops that made Sega bleed money, and very hard to develop for.

The N64 had a very limited storage for games and each cartridge was very expensive to make, which made each game more expensive for the consumer while way less profitable for the publisher. Would the third party go all out on Saturn and learn how to use the machine to create well optimized games? If so, ths Saturn would take it.

Definitely Saturn.
Biggest reason devs did not develop for N64 was the high cartridge cost, and this wouldn't change. So devs would definitely have flocked to the Saturn and its low price, high capacity drive, instead of paying outrageous fees with high risk of failure.

Saturn would have got all the JRPGs probably including the Squaresoft ones, and other big adventures like RE, Tomb Raider sequels...

Saturn. But we certainly wouldn't have gotten the kinds of games we got on PS1 due to its difficult architecture. And it wouldn't have won by near the margins that PS1 did.

I think the fact that Saturn was easier to develop some games for than N64 would've kept it competitive. We'd probably see a closer global race than Mega Drive vs. SNES. There's even a chance Saturn outsells Mega Drive. N64 would likely sell over 40 million units even if it falls obviously below SNES.
Saturn would be helped immensely by Final Fantasy VII and would likely be successful enough to get its own 3D Sonic game that would predate Sonic Adventure.
Nintendo would likely still win due to Mario and Zelda.

The reason the Sega Saturn bombed had a lot to do with its price, it cost nearly double the N64.
But the N64 had cartridges which were much more expensive than CD games, and held significantly less data.

I'm honestly not sure. But I do like the idea of analyzing the major companies and seeing which direction they'd go. I felt Sega Saturn had the best initial marketing, but then Sony really nailed it by bragging about the volume of games... especially in light with how few there were on the competing consoles.