Of all those best sellers, only one franchise was not a sequel or preexisting property...Horizon.
I agree that gamers want all those things mentioned by the OP, but I'd argue that making those games is no simple task.
So even if publishers are aware of this "winning formula" (making a complete game that keeps the gamer coming back), making a good game or one that is even profitable isn't necessarily going to happen. These huge experiences for the most part still piggyback on the success of the IP.
So if a developer isn't up to the task of delivering the type of quality that Nintendo or Rockstar are going to deliver, of course it's going to be easier to nickel and dime their customers using the popularity of a preexisting property.
It's not a good thing, but in every industry, there are stinkers and there are winners. It's not reasonable to expect all games to be made at the calibre that these winners are, because resources (including talent) are finite.
