A multitude of reasons.
1. The consoles exterior design: The primary color for the Gamecube was purple. A good deal of the general consumer base is shallow.
2. The N64 "hangover": Nintendo rubbed most third party competitors period and the N64 was the last straw. Eventhough the GC was far cheaper to developer for, had lower licensee fees, and a much cheaper to purchase storage medium. 3rd Party's were reticent to deal with Nintendo when they have two other viable options.
3. The PS2 head start: By the time the Gamecube and Xbox launched, the race was over. The PS2's lead was huge. With Sega being in massive debt (2 Billion dollars) before the DC even launched. They did have enough ammo to slow down the PS2 sales train. The PS2's huge lead combined with prior success of the PS1 was too much to overcome.
4. The DVD factor: The PS2 was the cheapest DVD player available at it's launch and for significant time after. This greatly boosted PS2 sales, particularly in Japan.
5. The Xbox: Simply put, Nintendo wasn't going pay same type of incentives that MS was for third party support. This exacerbated the GC's poor third party support.
6. Hiroshi Yamauchi: Although he's hardline tactics were the saving grace of the home console market during the 80s, he'd be the anchor dragging Nintendo down from the 90s on. While Iwata took Nintendo over shortly after the Gamecube's launch. The GC was designed under Yamauchi's watch.
7. The GC Controller: While it was comfortable to hold despite it's diminutive size. The GC controller had issues. It had less buttons than the N64 controller amd it's competitors. It's odd A button centric configuration also led to problems. The GC controller was not ideal for fighting games or any games that relied on the higher button count of the PS2 and Xbox.
8. Memory Cards: The initial GC memory card were 1/8 the size of the PS2 and Xbox memory cards. This caused issues, particularly with sports games. While this problem was rectified eventually the damage was done. The memory card issues combined with the lack of buttons and poor online support doomed the GC to third place in North America since it effectively turned away sport gamers.
9. Poor online support: While Nintendo's philsophy on online gaming was essentially the same as Sony, their execution was horrid. Phantasy Star Online was delayed because of lack of online adapters. Nintendo was slow to get online support in their development kits. By the time they did resolve their online support issues, third party developers had written off the feature.
