The failings of N64 and GameCube were easy to see.

N64 used cartridges which jacked up the price so that the games were twice the price as PSX. Also when N64 had 3 games available, PSX had over 200. When N64 finally got up to around 25 games, PSX had over 500 spanning over more interesting genres like Survival and Horror, and RPGs. It all boiled down to Nintendo using cartridges.

GameCube was a purple lunch box with a fisher-price controller populated with games that didn't really appeal to the 100 million or so Nintendo fans who bought handhelds. Every console featured some sort of major interface innovation except GameCube.
NES - d-pad
SNES - diamond face button configuration, shoulder buttons
N64 - analog stick
Wii - Motion controls
Wii U - touch screen.

All GameCube did was bring a deformed Dual-shock controller which wasn't very usable due to a tiny d-pad, a terrible face button configuration with three small buttons and one giant button, and three very awkward shoulder buttons. Possibly the worst controller of a major console ever.

Gamecube is also the only Nintendo console that doesn't have a smash hit game:

NES - Mario, Mario 3

SNES - Donkey Kong Country trilogy

N64 - Goldeneye 007, Ocarina of Time

Wii - Wii Sports, Mario Galaxy, Just Dance, Wii Fit, New Super Mario Bros, Mario Kart

In short: N64 failed due to cartridges, GameCube because it had nothing going for it and lots of things wrong with it.

Haha, I'm sorry, I was writing about the gc, and for some reason I must have wrote gbc :p But yeah, I meat gamecube, not game boy colo(u)r