No, I don't think they could make a better Zelda than Nintendo.
I love Rockstar, I buy all their games, but if they were to make a Zelda it would probably be a buggy mess.
Especially if they were limited to using Ninendo's hardware. I think they would get half way through, quit and say the hardware was too weak.
If they were both making a Zelda on more powerful hardware, Nintendo would still win.