Another mistake was not making the GameCube different enough from PS2.  With the N64 at least you had some positive new differences along with the limiting cartridges.  You had a analog stick, rumble, z trigger, four controller ports etc.  When you think about it the N64 and PSX  had very distinct console styles with what the hardware allowed the software to become.  Also because of mentioned features different game types gravitated towards the N64 (for example multiplayer and FPS games were bigger on the system)  The Gamecube didn't really have any game types gravitate towards it.

I can't really think of what positive (as in additional) features the GameCube had from the PS2 other than it's streamlined hardware that made it easy to develop for.  The Gamecube was supposed to be the Rebirth of Nintendo and that didn't happen.  But what it did do is teach Nintendo some lessons on how the industry had changed, and the SNES era is long gone so now they need to do things another way.

We all know nintendo's biggest mistake was showing off their new console in 2000 and teasing us with what zelda could look like on it with superior nex gen graphics then giving us Wind Waker. But we all know nintendo would never tease us again with a zelda for thier new console and raise our ex...pec...ta........wait a minute

I think Nintendo biggest mistake was trying to keep a clean image with past consoles, limiting 3rd party devs on what to bring to their console. I can remember a developer saying they had a negative experience with nintendo in the past. I can't remember if that was Rare who said that. Also I think Rockstar had a run in with Nintendo.

During the SNES era they did do things like no blood in Mortal Kombat but they realized that sales suffered vs the unaltered version.  Then the ESRB was created so game ratings took over the responsibility of content.  Rare was also given some freedom on what they could create on the N64.  Just look at Conker's Bad Fur Day on the N64 and it's allowed adult language and content vs Conker on the Xbox which was censored.  

The thing you're thinking of with Rockstar is actually their pre GTA days when they were known as DMA design.  Nintendo forced them to make some changes to one of their games, but not for mature content.

"DMA was asked by Nintendo to join their "Dream Team" of developers for the upcoming Ultra 64 system (later renamed Nintendo 64), alongside such other developers as Rare, Paradigm, Acclaim, Midway Games, and LucasArts.

Under this arrangement, DMA would produce an N64-exclusive title that Nintendo would publish. The result of this collaboration was Body Harvest, a third-person 3D vehicular action game with a storyline about aliens arriving on Earth to harvest humans for food. Nintendo requested a number of major overhauls, such as the addition of puzzle and role-playing elements, to make the game more appealing to the Japanese market. The game underwent numerous delays, and Nintendo finally decided to drop their publishing plans. Midway picked up the rights and finally released it in 1998, almost three years after the game was first shown. Reaction was mostly favourable, in particular for the game's innovation and free-roaming gameplay, although a few gamers criticized the graphics." - source