The 64s solid 1st and 2nd party games kept it running neck and neck with the PSX for most of its first year. It was only after it became clear that they couldn't keep those titles coming on a regular basis, and that with FF7 the PSX had titles worth having, that the N64 dropped into its distant 2nd place finish.
With the PS3, the hope of big, proven franchise titles like MGS4 and FF13 will be marquee titles that people will want (along with top tier non-exclusives like GTA4, UT3) and get them interested in the PS3, while "second tier" non-franchise titles like Lair, Heavenly Sword and the rest (which may end up being top tier after being released) will prove a healthy enough library to finally justify investing in the system.
That's the plan, anyway. Not sure if it'll work.
As for the Gamecube, thanks to the PSX a new generation of gamers learned that they could live without Mario and friends.
It didn't help that Nintendo played into the hands of "It's for teh kiddez" arguments with a purple lunchpail, cartoon Zelda, and this Mario campaign http://youtube.com/watch?v=0WPteMFkI2k
It didn't help that established franchises were often being asked to not just match but better their predecessors, many of which are considered the best games yet released in their respective series.
Finally the Nintendo games released in the first year largely were not as good as the should have been - Luigi's Mansion, Pikmin (good for a first outing, but not brilliant), and SMS (a very good platformer, but lacking the polish we come to expect from Nintendo). That left SSBM to carry the water, and that didn't have a huge fanbase yet.
