How so? Gameboy era was all about exploring pokemon world and legends by your own ways, starting with a single basic creature. The pokedex entries always tell bizarre behaviors which are basically unused when we get to the game.
By that time you would be sure that was hardware limitations that stopped us from truly immerse on that world. But as of now they are just creating pointless gimmicks in battle and pokemons are nothing but 3d models placed randomly on a regular landscape. What's even the point to make the game 3d open world if you don't make an appealing world?
You talk about the right direction for a business corporation standingpoint, even that, it won't be good if you have the potential to reach an even wider audience but you remain with a part, even if its big in absolute numbers.
The brand is huge no doubts, but talking about the gaming aspects, it's just don't match.
Pokemon has always been about one thing collecting Pokemon and the player's interactions with them (battling and trading etc...) everything else was chopped and changed around this aspect to suit the times for the target audience of the era, this is apparent in that Pokemon only exists because of the GB transfer cable. What you think Pokemon is about was merely the team at the time applying one man's idea to the Earthbound template after all Creatures were formerly Ape the the Earthbound team, the goal was never to go down the path you're suggesting it was to always build a subculture around the collection and player interactions of Pokemon this is why more and more features over each gen focused more on interacting with Pokemon rather than what you mistook the series for this is even seen from as early as G/S onward and why newer games will even employ gimmicks.
The franchise never aimed to pushed hardware because unlike most series it always had a clear focus on what it wanted to do and who it's audience was always going to be and this continues even today hence why older fans are constantly highlighting what they don't like but the games keep doing better and better, here's something you're mistaken on Pokemon is reaching the wider market it always has been it's you who is not part of it hence why they're going the right direction on all fronts not just from a corporate view even if it means leaving some old fans behind. It may not please some to hear that but believe me that's the reality of things, brand power alone doesn't maintain the level of success Pokemon has consistently commanded since it first released.
While I agree with your perception about Pokemons games framework, I think you're exageritng a bit the mission/goal behind pokemon games
The focus and vision about Pokemon is to sell toys and merchandise and it's arguably a more successful interprise than selling video games. That's why Pokemon has a separate company to deal with marketing and products. Pokemon already reached the wider market way back in the 90s with anime and movies, everybody knows Pokemon and Pokemon merchan is a easy gift even for people who doesn't care at all for either gaming or animes.
The games weren't doing any better either. They were doing about the same from the third to seventh generation, with Switch generation been the outlier because of the Switch software boom coming from people who haven't owned Nintendo handheld consoles, but this success was given to more core gamers than supposed wider audiences
Wider audiences though are more present in other Games, like Pokemon Go and Pokemon Unite. We can see Pokemon company is succeeding bringing in casuals to platforms they don't need to pay for.
The reason why they keep releasing generational games is to release more Pokemons (i.e. more merchan to sell) and use this new generation to keep releasing new anime, movies and having source material to keep upgrading the host of their side games
In this sense, they can drastically change the gameplay of a mainline game and sales are very likely to not suffer much, and even if they suffer most of their source of money (merchan) will be kept intact for as long the brand still fresh and light, and Pokemon Company is pretty good are keeping Pokemon brand clean and soft.
So I'd say the reason why they never tried to changed much their formula is precisely because gaming is already secondary for them. They have the same devs from the 90s working in their host, they have no reason to pass trough the hassle and challenge of changing much when their framework keep providing content to release their animes, cards and movies.
I will be even more blunt and say if it wasn't by Nintendo sharing the IP I could see Game Freak changing their approach to focus only on mobile games and forgetting their mainline efforts completely, but alas it's a shared IP so it won't happen.