Like many said, the decision to ultimately stick to cartridges for the N64 is the start of it all.
You can't blame their extremely strict quality control on the NES and SNES for this. If it wasn't for that, gaming would be nowhere near what it is today. There was no control on Atari 2600 of any kind, anyone could make a game for it. Because this resulted in hundreds of extremely bad games, the market crashed. Had Nintendo never regulated what was on NES and SNES, gaming wouldn't have recovered from that crash so quickly.
Anyway, with the cartridge decision it is obvious what the main reason was for most third parties to migrate to the new competitor; money. In this regard Nintendo made a bad business decision. Damage was done and the GameCube being responsible for the 'kiddie-image' also didn't help.
From a gaming perspective though, I have to say Nintendo made the right decision. CD technology was in hindsight maybe just a year or two too young. Having played both consoles (N64 and PS1) when they were in their prime, N64 wiped the floor with PS1. The discs on PS1 were atrocious compared to the N64's cartridges. They were much less reliable, half the time games wouldn't load immediately. Loading itself was a pain, it took ages. On every game. Discs also made a lot of noise, the N64 cartridges do not have spinning parts which besides from making them way more durable, means they make no sound.
My N64 probably got ten times the amount of playtime compared to my PS1 because of issues like this.







