I think Nintendo would have won, and handily so. People tend to forget that the PS1 was not an instant hit, and took a while to become truly successful. In all of 1995, it sold only 1.37M units in Japan, a very low number by the standards of the time. Shipment data suggests that it was likely outsold by the Super Famicom and may have done only about on par with or slightly less than the Saturn. It did improve considerably in 1996, selling nearly 2.7M units, likely due to a combination of two separate price cuts that dropped it down to ¥20k by mid year and an improving library, including several high-profile releases like Tekken, Resident Evil, and Arc the Lad II (sales data for 1996 isn't granular enough to tell what did what exactly in terms of pushing hardware, though). But it wasn't until 1997 that it really took off, and it was the release of Final Fantasy VII that appears to be the initial trigger, pushing the system to nearly 180k units the week of its release.

In the U.S., the PS1 had a very weak launch period, selling only about about 600k units from Sept. to Dec. 1995, and it sold just short of 2 million units in all of 1996 even after a major price cut, a rather paltry number and one of the worst first full years of any major system (only the Genesis, Saturn, and Wii U did worse). In fact, though the N64 was only available for the last 14 weeks of the year, it sold almost as many units as the PS1 did that year. While the PS1's sales did improve after some price cuts, it continued to be outsold by the N64 for the first two-thirds of 1997. But then FFVII was released in September that year, and it blew the doors off the PS1. It sold approximately 3.76M for the last four months of 1997, compared to about 3.86M for the previous two years before. That's how much the PS1 improved in the U.S. thanks to FFVII.

Europe is harder to gauge without any actual sales data, but shipment data suggests that the PS1 struggled early on as well. By March 31, 1997, 18 months after launch, Sony had shipped only 3 million units, about par for course in a region that never really had gotten into console gaming before. However, another 3.1 million units were shipped over just the next six months, though whether it was in anticipation of FFVII's release in November or just a general increase in demand is unknown. By June 30, 1998, shipments had nearly doubled yet again, putting cumulative LTD shipments at over 11.7M (already better than lifetime sales of any prior console in the region). By that point, it was clear that Europe had chosen the PS1 as the first console it would adopt en masse. Unfortunately, the absence of any data beyond shipment figures makes it impossible to tell the exact factors that led to growth of the PS1's sales in Europe.

So, if the PS1 started off slowly and didn't start to really grow until certain factors propelled it to mainstream status, would those factors still have existed had the N64 been disc-based?

Well, Japanese third parties likely would have remained with Nintendo for the most part. Namco was Sony's earliest and biggest third-party supporter early that generation, and it's possible that might not have changed if the N64 was CD-based. Capcom and Konami may have supported both the PS1 and N64 just as they supported the Mega Drive and SFC in the previous generation; they showed minimal interest in the PS1 at first, but that started to change as the generation progressed, with Capcom having some high-profile PS1 releases in 1996 and Konami accelerating PS1 support in 1997. Given their prior support for Nintendo, I think most of their games (including notable titles like Resident Evil and Metal Gear Solid) would have been multiplatform instead of PS1 exclusive if format had never been a concern, giving Sony no real advantage. Square and Enix exclusively supported Nintendo in the previous two generations, meaning Final Fantasy and Dragon Quest would likely have remained exclusive (in fact, FFVII was originally in development for the N64 before Square jumped ship to Sony to take advantage of the PS1's disc format). Without the massive system-selling success of FFVII to propel it forward, the PS1 almost certainly would have had a far worse 1997, and likely would have continued trailing the N64 in the U.S.

As for Western developers, the N64 did get quite a bit of support from them, including quite a few titles from big names like Activision, Acclaim, THQ, and LucasArts (though EA primarily supported the PS1). While Sony did nab Crash Bandicoot as a high-profile third-party exclusive in 1996, most of the PS1's best-selling Western titles released in 1995 & 1996 were not exclusive, and some did come to the N64, or likely would have had it been CD-based (e.g., Tomb Raider was released on the Saturn as well). In fact, it wasn't until 1997 and especially 1998 that we really saw major Western third-party releases as PS1 exclusives, and even then such support was primarily from the likes of EA, Eidos, and Insomniac.

So, had the N64 been CD-based, given what we saw earlier in Gen 5 we can plausibly assume a scenario where the PS1's biggest supporters were reduced to just EA, Namco, and Naughty Dog. Without any sort of leverage originating from a dominant market share as well as known support patterns from older third parties, multiplatform titles would have been more common, with Capcom, Konami, and every major Western third-party sans EA supporting the PS1 and N64 about equally. Meanwhile, Nintendo would have likely retained Square and Enix as exclusive partners. In terms of first-party content, Sony was largely relegated to Gran Turismo and the output of 989 Studios (e.g., Twisted Metal, Syphon Filter), while Nintendo already had well-established brands. Aside from Gran Turismo, the PS1's biggest exclusives (either first- or third-party) would have been Tekken, Namco Museum, and Crash Bandicoot, and possibly not much else (EA's franchises were nowhere near as big then as they are today, even with the PS1's huge install base).

With the N64 retaining the support of the largest third parties of the previous generation (all Japanese) and most publishers that started to really take off that generation (many of which were Western companies), and retaining JRPG juggernauts like Final Fantasy and Dragon Quest, I think it would have easily won the U.S. and Japan, no contest. While the PS1 likely would have done well for itself in the U.S., it would at best had been the company that filled the vacuum left by Sega's failures, while Nintendo would likely continue strong off their prior momentum. In Japan, the PS1 probably would have done better than Sega ever did, but it probably would have run a distant second without the support of Square and Enix and a lack of major exclusives from Capcom and Konami. Europe is a tougher call due to the lack of sales data for the region. The PS1 not having the sheer library size advantage, including Final Fantasy being N64 exclusive and other big-name third-party titles like Tomb Raider, MGS, and Resident Evil, almost certainly would have impacted its market share considerably. European gamers were almost certainly going to adopt consoles en masse after the collapse of the home computer market in the mid 90s, and it's entirely possible that they could have simply split between the PS1 and N64 instead of becoming "PlayStation Country" like it's been these past four generations. Then again, Nintendo had no real presence in Europe before Gen 5, and it's possible that they might not have seen the marketing opportunities that Sony still might have, putting the PS1 ahead (though I doubt by much).

If I had to guess regional market share under this alternate timeline, then assuming the most optimal circumstances for Nintendo I would assume the N64/PS1 split would have been 60/40 in the U.S. and 75/25 in Japan, with Europe being a toss-up. Globally, that could have put the N64 at around 70-75M and the PS1 at around 45-50M. Despite coming in second place, Sony would still have a brand with significant potential. Nintendo would still likely have continued making very conventional consoles after seeing a CD-based, still very conventional N64 winning the day, meaning we would never had have a GameCube, much less a Wii, Wii U, or Switch. This leaves Xbox in a nebulous place. Would MS have been as emboldened to enter the console market had Sony not dominated like they did, or would they still see potential? If so, then what? Could they have succeeded? Would the OXbox's market share have been better or worse? And if there was no Xbox, then the face of online gaming on consoles would likely have turned out very differently (and may have never caught on, at least not until much later), not to mention there never would have been franchises like Halo and Gears of War.

Conclusion: Had the N64 utilized CDs instead of cartridges, the Fifth Generation likely would have ended in Nintendo's favor but with Sony still in a good position. This would have resulted in a video market that would already be very different in Gen 6 and utterly alien to us by Gen 8.

Square and Enix were the ones that mattered the most by far, and they were pretty Nintendo-loyal.

Namco had always had a bad relationship with Nintendo ever since their dispute over re-negotiating Namco's licensing terms for the NES, so that wouldn't have changed. But Tekken on its own wasn't enough to turn the tide for Sony. 

Capcom and Konami would have been multiplatform. Konami was Nintendo's biggest Japanese supporter on N64. Sony might have tried to get MGS as a PS1 exclusive anyway, but MGS wasn't as huge of a series sales-wise as its fans like to make it out to be. 

Square and Enix were the ones that mattered the most by far, and they were pretty Nintendo-loyal.

Namco had always had a bad relationship with Nintendo ever since their dispute over re-negotiating Namco's licensing terms for the NES, so that wouldn't have changed. But Tekken on its own wasn't enough to turn the tide for Sony. 

Capcom and Konami would have been multiplatform. Konami was Nintendo's biggest Japanese supporter on N64. Sony might have tried to get MGS as a PS1 exclusive anyway, but MGS wasn't as huge of a series sales-wise as its fans like to make it out to be.