Honestly, Game difficulties shouldn't exist to begin with, for some of the reason you said.
Most games just deal with different difficulties as if its a slider that turn up or down and adjust enemy health/damage/defense, which in turn utterly unbalances most games, making them stupid easy(you can withstand most damage and not worry about dying) or unfairly hard( you die super fast, and enemies become hp sponges because the game needed them to ebcome harder without actually working on their moves, behavior, etc)
In order for games to actually have difficulties that actually represent their difficulty without making them cheap think Halo hardest difficulties in whic most enemies 1 hit kill you or cl;ose to it), you would have to rework so much of the game, like its level design, enemy behavior, attack patters, etc. that you would be basically putting in the same ammount of work and time as if you were doing a new game.
Maybe AI will be good enough in the future that this will not be a problem, but for now AI cant do all these stuff I just said. Even if they were to only handle enemy attack patterns to change betwenn difficulties, I would imagine that it would be alot of work(And I do mean alot) for something that, at the end of the day, its meaningless.
Carefully ahndcrafted experiences that doesn't compromise the vision is one of the reason that games like Dark Sould ands Cuphead became as famous as they did.
I talk more about this topic in this thread:
I mean, I see it as a later reply, but this is why Forza uses learning AI. It's not quite full human mimicry, but it has different behaviors based on what it recognizes the player doing often. Ergo, more aggressive routines are executed if the human player does more aggressive things like ramming other cars, so the truth is that the technology exists.
In fact, in some ways, the AI already exists, too: enemies already have complex path routines and behaviors, so while it IS definitely more work, I wouldn't say it's meaningless if someone takes the time to tweak and re-adjust timing and patterns based on difficulty levels. That in and of itself would be a start! The machine learning can come later after difficulty becomes a behavioral adjustment and not a metadata one.
Here is a decent article on a related topic.
Why Video Game AI does not Use Machine Learning
Forza uses machine learning, which is why its AI is decent.
But the problem is that it is expensive and difficult to make machine-learning based AI that is near-human in capabilities without making it too good, where very few humans will win against it. It's only been recently, and costs a decent amount of money (through labor costs) to get the right mix of algorithmic (via a conditional statements) AI and machine learning AI in games. I suspect that we'll see more of this by the 2030's, but right now publishers aren't seeing the advantages. It'll take a few innovative publishers/developers or some very ground-breaking middleware to push the boundaries and set the standards for everyone else to the point where they are willing to staff machine learning engineers (who make in the 100-150k USD on average and therefore aren't cheap) or not need to staff machine learning engineers because the middle-ware does the task for them.
Of course you can make game difficulty responsive to traditional AI methods, but again, that is costly as it requires more game testing.
The funny part about reading that article is that it touches upon the challenges but doesn't seem to really "believe" in the solutions from a business standpoint (cheaper and easier to just do it "this" way). The fact that Forza exists and also as mentioned below (I'll reply to the comment specifically) amiibo CPU players in Smash prove that while machine learning is a bit too complex and expensive, we can still utilize "tiered" or "locked" AI routines that adjust difficulty WITHOUT using the cheap-out metadata method.
The fact that Nintendo, the often lowest-budget of the three, has employed a non-machine learning but adaptable AI routine for its Smash CPU amiibo players proves that what I'm saying is quite possible as a "cheaper" method until machine learning becomes more affordable... OR that a major company creates what you might call a "third party" AI application. Think Unreal Engine, but for AI. Thus, devs can choose to use this "AI Engine" to control their enemy patterns and behaviors moreso than develop it themselves.
I like the difficulty systems in Supergiant games a lot. They basically give you toggles which act as challenges which make the game harder, so you can turn them on to increase your rewards. In Pyre, there is an option to make enemy AI better, in addition to options which increase the enemy's stats and change various other toggles. To that extent, I don't really agree with you when you say that increasing numbers isn't the same as increasing difficulty. While it can make the game feel bad (bullet sponges are one of my least favorite things in games), it is something that can be done well. By increasing these various variables, it forces you to play with less room for error, which often requires greater skill or a change in tactics to react to.
So yeah, improving AI with difficulty could provide benefits, but it is far from the only good option when it comes to difficulty increases.
I mean, you're proving the problem exists, and I applaud Supergiant for simply giving you, the player, control over the metadata adjustment. It's like they realized, "welp, we can't actually make the game more difficult, we can simply adjust the metadata, but we won't know what the player actually wants in their comfort zone, SO HERE YOU GO DO IT YOURSELF" lol
Maybe I'm asking for Skynet, maybe I'm not. All I know is that enemies just keep coming at me no matter what the game difficulty is set to, and that needs to change!
Aaaand I'm off to a client site, but I hope to have time this weekend to come back and reply to any replies, if there are any this time :P
Considering what you're probably asking for, I'm afraid I have to do this now. *winds up M16 and eviscerates Zyro with bullets* Anyone who cares about the future would've done the same!!
In all seriousness, it would be interesting to see video game enemy AI try machine learning, but I don't know how it would catch on. A colleague once told me that Amiibo have a sort of machine learning system for Super Smash Bros., but I don't know if he was right or just being a cocky little douche.
Whoa there, you got the wrong guy! lmao
BUT, I'm SUPER glad you mentioned the Smash amiibo and now I feel like a MORON for not mentioning that! I even have a video of me fighting a max level amiibo CPU I trained myself! That's actually a prime example of how the AI adapted WITHOUT any change to its metadata: they simply locked reaction times and move routines behind "levels", and that could easily be like difficulty levels! Shit, now I feel like deleting the video and re-doing the whole thing because Smash amiibo CPUs are a PERFECT example where it can be done WITHOUT the use of AI/machine learning! Damn it, damn it all... This is what happens when I pull back from the channel, I think less and take less time on videos and now there's big holes in it...