If it's a good game why break what's broken?
Yeah, your rant makes since on a high level but when you get bogged down in it, you see why publishers do what they do.
Making a AAA games is a huge risk that has a very high possibility of blowing up in your face. If you have something that you know works, you keep it. Even worse, it's a huge risk that doesn't even always pay out huge when it works!
You do everything you can to keep risk as low as possible so that means you keep the same engine, you make only basic tweaks, and you just improve upon what you already have.
Since AAA games cost a stupid amount of money and there's already a stupid high risk, it doesn't make since to take 5 years to make a game with a new engine and completely new unproven play mechanics.
