The open world thing has already lost some steam. There's definitely plenty out of there and more coming, but people are supporting all kinds of smaller, indie style games.
The truth is that an open world game is very time consuming and must be stuffed with content, meaning they're very expensive to produce and require a ton of sales to recoup that investment. There's only ever going to be so many open world games that can compete for the limited time and attention of we rabid consumers, so those other genres will continue to get attention.
As for modernizing them, that's really already underway and, for the most part, it's hardly necessary. People still enjoy most of those old genres, the only one that really seemed to fade off that I can think of being the "point and click", but we seem to be seeing elements of that in VR games these days.
There will always be a tendency among some reviewers to dock certain genres points even when they're the top of their field, no "modernizing" will change that. I think you're attempting to identify a problem that doesn't really exist.
What about the retro-style games' shrinking audience?
In what way is it shrinking? They're still selling quite well... heck, I was just playing Dragon Boy, and a couple days ago I was playing the massive hit Stardew Valley.
I think the pixelated visuals probably don't wow people anymore given how common they've become (I like Dragonboy's approach of offering both), but the old genres (2/2.5D shooters, Platformers, Farming etc) are definitely alive and well. Steam and youtube are still loaded with them, and they're selling great even years after releasing.