Sega had already dug their own grave as a console maker. Their shift to a third-party was mostly their own fault, not because of the actions of their competitors. They already were making bad decisions towards the tail end of the 16-bit era, most notably the ill-fated 32X.
But what really got them was the Saturn. They absolutely bungled the launch of the system in the West, their much-hyped "Saturnday" release of Saturday, Sept. 2, 1995 was cancelled in May 1995 at the very first E3, with Sega announcing a surprise release date of "right now." Retailers were caught off guard, and only certain ones carried it. The whole thing was a disaster. It doesn't help that they priced themselves out of competition by launching it for $400, which was an unreasonably high price point in the mid 90s (that's $745 adjusted). As a result, the Saturn had absolutely atrocious sales, pulling in only a bit over 30,000 units in the U.S. in its launch month and about 780k for the entirety of 1995. While it did do better in Japan, that's not saying much.
Not only did the Saturn stall right of the game, the system being difficult to program for arguably made it to where a lot of companies were even less incentivized to make game for the struggling system. While in retrospect the Saturn does get some praise for its games library, it was sorely lacking in big marquee titles that could have drawn significant interest towards the system. Speaking of which, another reason cited in the Saturn's struggles was the cancellation of the first fully 3D Sonic title "Sonic X-treme," leaving the system with no mainline Sonic game, which was a huge loss considering Sonic was b far their biggest franchise. The Blue Blur's only notable presence on the system being a port of 3D Blast, Sonic Jam (a compilation of the Genesis side-scrollers), and the racing game Sonic R.
The Saturn was dead in the water from launch, this despite the PS1 having some struggles itself early on. Nobody was in a rush to buy either system. But while the Saturn never could recover and ended up being discontinued in North America and Europe in 1998, the PS1 quickly built up steam in 1997, but Sony really only filled a vacuum left by the competition. Between Sega's bungling and Nintendo deciding to stick with expensive, low-capacity cartridges, resulting in many devs looking elsewhere to publish their biggest and best games, Sony succeeded simply by not making any huge mistakes. The PS1 was well-marketed and attractive to developers, and as it amassed an ever-larger library of notable titles, especially Final Fantasy VII, more and more eyes were on the rookie, which eventually became the best-selling system of that generation (and of all-time globally at the time, thanks largely to it being the first console adopted en masse in Europe).
By time the Dreamcast was released, the damage had already been done. Despite a good initial reception, a decent launch performance, and some solid games, its sales faltered early despite being the only Gen 6 console on the market until the PS2 was released. The market was already burned big time by the Saturn, and few people outside of dedicated Sega fans and a few other enthusiasts were going to bite. The Saturn's failure all but ensured the Dreamcast would not be Sega's big comeback, and in March 2001 Sega would discontinue the system, leaving the PS2 as the sole Gen 6 system for the next eight months. The Dreamcast might have been a good system, but it was too little, too late.
In conclusion, Sega's demise as a console maker is a story of self-inflicted wounds. Had they made better decisions back in the mid 90s, especially in regards to the Saturn, they likely would have retained much of the good will and momentum they had with the Genesis. It's hard to say how well they could have done had things gone better, but at the very least they might still be making consoles to this day if it weren't for the catastrophe that was the Saturn.