I see a lot of people who are using the N64 cartridge as the problem with the N64's support but that (really) wasn't what hurt them in that generation ...
After the SNES's (and Gameboy's) success Nintendo was very arogant and thought that people would buy any system that had 'Nintendo' written on it. At the same time Nintendo talked (openly) about how they wanted to produce a system that only had the highest quality games on it; there are lots of rumors of Nintendo's behind doors meeting with third party developers and Nintendo's attempts to accomplish this goal.
Ultimately, these discussions offended a lot of third party publishers (and to a certain exted scared them) which resulted in a very strained relationship.
Had Nintendo maintained better third party relations durring the N64 generation they probably would have had better third party support, and many of the "classic" third party games people believed couldn't be done on the N64 would have been produced with realtime cut scenes rather than pre-rendered FMV.
Basically, the mistake was hubris and arogance which is true for most of the biggest mistakes inside the gaming market.
Edit: There were many more problems in third party relations durring the N64 generation, and it was the combination of all these problems that killed support.