The design (awkward size, awkward color and it has a handle sticking out), lack of DVD & 3rd party support, and ultimately weak 1st party games in general. They should have known when they release a Luigi game before Mario one. And, PS2 rode the PSX success, and Xbox was just that new thing that.
Another opinion - they just got lucky with Wii. Nintendo could have upgraded the hardware as they did with DS, updated the Wii Menu
Edit - Another awkward thing I forgot that someone has mentioned - Mini disc!