Nintendo does have high quality 2nd party devs, in Retro, in Monolith Soft, in (for now effectively) Next Level, HAL Laboratories, Intelligent Systems, Game Freak, etc. etc.
However, in each of those cases, almost all of those studios only are equipped to make one game at a time. There have been rumors for some time now that Retro has been hiring up people, and are in a position to be working on at least two games at once now, which is very likely happening as we speak.
But the fact remains, Rare kind of ballooned during the N64 era, to the point that they were able to make, at least, three games at once. None of Nintendo's other current studios have that capacity.
And besides that, while MOST of Rare's N64 output was great, not ALL of it was. Jet Force Gemini, for example, was okay, but that's about it. And Banjo Tooie, at least to this guy, was one of the worst, most disappointing sequels to a great game I've ever played. It wasn't outright BAD. But it was incredibly underwhelming, and not what a Banjo 2 should have been.