Back when Sega left the console business, it really looked like Sony caused their downfall. Now that I've looked into it more I'd say the following companies contributed to Sega's demise in order of greatest to least:
1. Sega (by far the most)
2. Nintendo
3. Sony
4. Microsoft (i.e. not at all, but they are in the poll)
As many posters have said, Sega mostly screwed themselves. Tom Kalinske was great as the head of Sega's American division, but the heads of Sega in Japan really doomed the company. The heads of Sega did a lot of things that sabotaged their presence in the American market, and I personally think they were kind of envious of Kalinske's success during the 16-bit years. The Genesis (Mega Drive) did worse in Japan than the PC-Engine, while the Genesis was extremely competitive with the SNES in the US. The Saturn actually got some early success in Japan, but it did terribly everywhere else. The Saturn lost Sega a ton of money, and the Sega-CD and 32X were not executed well either.
However, another factor in Sega's financial problems was the decline and eventual demise of the arcades in the US. Arcades in the US dried up right around the end of the 20th century. The Saturn actually lost Sega more money than the Dreamcast did, however Sega's arcade business helped offset a good amount of their losses during the Saturn years. By the time that the Dreamcast launched, their arcade business couldn't help them much anymore. In fact, the Dreamcast actually did fairly well for the two years it was on the market, and probably could have done better than the Gamecube and OG XBox if given a normal amount of time for a console's life. However, it's normal for a console to lose some money the first year, and Sega was not in any position where it could continue to lose money. Ironically, it seemed like they panicked the last couple of years, and they were doing some desperate things that they normally would not have to do (like drastically cut the price of the Dreamcast). But anyway, it was a combination of the Saturn, overall mismanagement, and the arcades drying up that doomed Sega.
So, if you want to trace back what lead to the demise of the arcades, it was essentially actions by Nintendo in the NES era. Games like the first Legend of Zelda could not be made for either the arcade or home computers, and it kind of inspired a whole string of longer action games made for home play. At the same time it was easier for third party companies to make money on consoles than in the arcades, and the variety of games in the arcades started drying up in the NES era. (Fighting games made most of the money in the arcades in the 90's.) So, the decline in the arcades started during the NES era, and it eventually dried up in the US entirely around the time the Dreamcast launched. That is why I'd say Nintendo contributed to Sega's demise second most after Sega themselves.
So, if I had to pick a competitor that doomed Sega it was Nintendo, but that was due to trends Nintendo started in the NES era, and not due to, say, the performance of the N64 or Gamecube. Obviously, getting outcompeted by other consoles is a factor too, but if Sony wasn't around, I think Sega would have just gotten clobbered by Nintendo instead. The biggest factor in Sega's demise was the mismanagement of the company by the heads in Japan. They made a whole series of bad decisions, but the worst was probably not listening to or backing Tom Kalinske who was responsible for the success of Genesis in America.