Nintendo lost at the Red Ocean battle with the N64, I think that is clear enough for all of us at this point.
Their failure is due to:
- launch timing,
- console cost and bundling,
- HW design choices (that alienated 3rd parties),
- poor 3rd party relations,
- weaker marketing and mainstream penetration than Sony,
- Improperly managed console image.
The reasons have been discussed ad nauseam, and it is really quite clear at this point that Mario had little space to counter the circumstance.
The Gamecube rectified the HW design choices, but still failed at everything else, even Mario (I'm looking at Sunshine).
--------------------
Having said that, having gone blue ocean with the N64 (SW-wise), I doubt they could have countered the Playstation's incredible lead, at least not in places other than NA. All the other circumstancial factors were there to make it fail anyways.







