It was Sony, combined with bad leadership at Sega of Japan. The pressure from Playstation caused them to make some foolish mistakes which really hurt them in gen 5. There were also more direct attacks from Sony such as moneyhatting Tomb Raider 2 off of Saturn after the first one released on Saturn. But the main cause of Sega's demise was Sega themselves, specifically Sega of Japan.
The heads of Sega of America and Sega of Europe during gen 4 told them they were making mistakes with Saturn and 32X and such, but Sega of Japan was arrogant and thought they knew best, even though Genesis/Mega Drive was far more successful in America and Europe than in Japan, their foolish decisions hurt Sega so much financially during gen 5 that even though Dreamcast was selling well in gen 6, it just wasn't selling well enough to make up for the losses incurred the previous generation. PS2's hugely successful launch was the final nail in the coffin.
Ps2 was a beast and could be all it took to do them in. A part of me started wondering recently though if Sega could have gained enough investors to stay in the race if it weren’t for the behemoth Microsoft making it impossible to find a niche.