I really don't think Nintendo got much audience from the PS2/XBox crowd. I think it was more like
Nintendo's core base (20 million) + the Soccer mom/Wii Fit/Sports/elderly/lapsed gamer crowd + properly utilizing Mario nostalgia. That was the Wii "trinity" for their userbase.
I think Nintendo's "core" base actually shrunk with the Wii even if the overall userbase went up. They had big trouble selling stuff like Xenoblade, Sin & Punishment 2, Excitebots, etc. on Wii, whereas even on the GameCube games like Viewtiful Joe and Pikmin and Tales of Symphonia and Soul Calibur 2 were able to find reasonable success.
Mario and DKC did well on Wii for Nintendo, but a lot of other IP fell by the wayside, to the point where NOA couldn't even bother with IP like The Last Story by the end of the gen. That's the type of game that probably would've done OK on the GameCube and would've killed it on the N64 (barring cartridge restrictions).
Nintendo is really not competing against MS/Sony right now (maybe that will change), the Wii was the trendy/hip thing for casual consumers to buy circa 2006-2009. What they're really fighting against is their brand running out of steam/no longer being the cool trend. Without a new hook as trendy as the Wiimote, that's the real challenge for Nintendo, it's very hard to replicate the same formula, without the same hook.
