Back that time Nintendo home console market was essentially USA and Japan only. PS1 made console gaming popular worldwide and those markets were way more ready to buy a PS2 than a console from the company know for producing Game Boy
People overlook how much the lack of a brand affected Nintendo here. Wii was pretty much their first home console to sell a decent amount of copies in Europe, their old efforts didn't manage to sell even 8 million there
Elsewhere their best seller was N64 with not even a million copies sold
Meanwhile PS1 sold an outstanding 40 million copies outside North America and Japan. PS2 market was already 40 million potential consumers ahead of Nintendo, and devs knew that so why put their odds on Nintendo in first place ?
In Japan, GC did lack of appealing titles for console gamers (mostly RPGs) all on PS2 thanks for an awful relationship Nintendo build with third party developers during N64 era. JP support was cut down to favor GBA instead. Japan customers were happy playing their favorite Nintendo franchises on portables, this never changed until today and Japan is still growing more and more a handheld-heavy market
In USA Nintendo did quite well during N64 era, all droughts considerate, thanks to their very outstanding branding (same brand they never had in Europe), but by the end of N64 life it was clear Sony was giving a much better support so why get a Nintendo console again? Specially if that said console have nothing to stand out by itself, either on hardware or software side. XBOX entering in the market only help to dig the grave of Game Cube, as then even people who doesn't want a PS2 have no reason to buy a GameCube







