In my opinion the fact that GameCube did so well in competition with X-Box is great.
I think honestly Microsoft had the name going for it. PC's dominated the industry and when Microsoft entered the industry alot of gamers expected an amazing experiance. Halo was definatly the biggest factor in Microsoft's success but it also had several big PC games exclusively on the X-Box like MechAssualt for example.
X-Box had online and GameCube didn't. X-Box Live was probably the X-Box's biggest selling feature. Developers like Lucas Arts actually refused to develop games on GameCube because Nintendo refused to give them online support. Nintendo fell out with several developers because of this and thus lost alot of mutliplatform and exclusive games.
Loss of Rare. Believe it or not RareWare was one of the biggest driving forces behind the N64. Its franchises like Conker, Banjo and Donkey Kong. Their last title for GameCube StarFox:Adventures was considered a huge failure and hurt Nintendo's reputation. Their move to Microsoft definatly hurt Nintendo, maybe not being a deciding factor in X-Box's success but it definatly did contribute to it.
The lack of classic franchises and Gameplay. Nintendo decided to launch the GameCube with Luigi's Mansion instead of a proper Mario game, Nintendo also decided early on to bring new franchises like Pikmin, Animal Crossing and CustomRobo to the market. Then instead of giving people the photo-realistic Zelda of their dreams they brough WindWaker. The core franchises were forgotten as Nintendo chose innovation over peddaling GameCube to the former Nintendo hardcore. Even Donkey Kong saw Donkey Konga games and an innovative Jungle Beat instead of classic gameplay. As Wii has proven if Nintendo had stuck to the classic gameplay that made their past consoles successful they may have sold more.
The lack of a killer App. Nintendo had several great first party titles. Many of which I prefer to anything on X-Box or PS2. But I am a Nintendo fan who loves Nintendo's games. Fact is Nintendo didn't have a single killer app. Mario Sunshine failed to become a killer ap, LOZ:WW failed to become a killer app. Even SmashBros:Melee failed to be considered a killer app. Halo was huge for Microsoft that single title helped shift millions of hardware units, but Nintendo didn't have a single title that made people say "I must have this". Nintendo had alot of AAA games but nothing that caught on with the general public.
Third party support, like the lack of classic franchises and killer apps. Third parties abandoned the GameCube early on. I remember early on Capcom commited to releasing five titles for the GameCube. However when GameCube sales wained they took for ever to release these five titles and ended up porting most of them to PS2 in the long run. Similiarly EAGames, LucasArts, Atari, Activision, RockStar all of them prefered X-Box and PS2, Nintendo did a horrible job at attracting third party support. UbiSoft and Sega were pretty much the only publishers with some loyalty to Nintendo.
In the end those alone were several factors which lead to GameCube failing to beat Microsoft's X-Box. I agree it was a far superior console with way better first party games. But in the end Nintendo made several errors which lead to its ultimate demise!