I honestly think it's better that it isn't any quicker than 4 years. Main Zelda games should be rare treats - something to be vastly anticipated. Each one should be its own individual gem, unique from any other in the series. Should they ideally speed up the development a little bit? Perhaps, but I'd rather they delay and release another gem than rush it out.
In reality we've had 1 new Zelda game, 2 remakes and a spin-off since Skyward Sword so its not like we've been starved of Zelda in the meantime.