I think that is largely a different thing. What you are talking about is invisibly tweaking variables based on how the player is doing (tweaking variables being the exact thing the OP is complaining about). What the OP is talking about is having enemy intelligence change based on selected difficulty. 

Dynamic Difficulty Assistance is stupid and I hate it. Changing AI would be a good improvement, but it takes a lot of resources to get right. 

What you are describing is basically DDA(Dynamic Difficult Assitance) RE4 has it https://www.youtube.com/watch?v=zFv6KAdQ5SE

But it kind of done by plenty of games already. Crash 2 (and later games) had it aswell. Game devs don't tell us that they do it and us gamers don't notice it.

We probably all had experienced a game that had a hard part/chapter/level and you then notice an health kit/ammo box/checkpoint or even an enemy that isn't their like the last time. We all probably just thought it was our imagination/bad memory etc.

I think that is largely a different thing. What you are talking about is invisibly tweaking variables based on how the player is doing (tweaking variables being the exact thing the OP is complaining about). What the OP is talking about is having enemy intelligence change based on selected difficulty. 

Dynamic Difficulty Assistance is stupid and I hate it. Changing AI would be a good improvement, but it takes a lot of resources to get right. 

It's more of a problem that games have to cater to the male power fantasy. Intelligent enemies sound fun, but not everyone likes multiplayer. People want a single player campaign where they can feel superior to the AI and mow down a thousand enemies or more in a play through.

Even hard games like souls games work because it's all set in stone. Enemy attack patterns, positions, always the same. Suppose they actually had some intelligence and outsmarted the player at every turn, who would still play the game... It's not hard to make AI that can beat the player every time. The hard part is making the player feel good about his actions without being to obvious in holding back. Basically the roll of a good DM. Yet there is also the ego part, I beat the game on hard would become meaningless with self adjusting difficulty / AI. Heck see the resistance to an easier mode on souls games, even though those games are nothing but rote learning. NG+just ramps up the hit points, still same patterns, same positions, all there for the male power fantasy.

It would be nice if games would analyze the 'fun' level players are having with different sections of the game. I was hopeful something like that was going to happen when Kinect 2.0 was supposed to read player's emotions while playing. Bored, tone down on those sections or spice them up. Frustrated with failure over and over, lower the difficulty / provide assistance. Quickened pulse, more or those sections, increase the difficulty. Yet without any feedback, there is no way to know if a player is engaged in doing the same thing over and over (trying to beat a difficult section) or getting utterly frustrated and ready to delete the game. It's just as frustrating when you almost get it, then the game decides to make it easier...

Erhm, this is a two-edged sword you're describing.  The challenge IS often learning: learning the weak direction of Souls bosses, learning the attack patterns and range of a monster in Monster Hunter.  Once learned, then it becomes a rinse and repeat.  The feel good is gone once the human has learned the AI's patterns.  Keep in mind, I literally mention I don't want my enemy AI to be inhuman (and show an example of a button read in MK11 even though I still won the battle) as I'm keenly aware a CPU can respond faster than a human.  Humans have an average of 10-20ms response times, but thanks to modern computing, AI can respond in as little as 2-3ms (or shorter depending on the instruction set).

Thus, I'm not asking for an unbeatable AI, I'm asking for an AI that provides the very challenge that makes Souls and MH games so alluring: the learning process to defeat what was at first a tough fight.  If AI becomes adaptable with greater/deeper routines on subsequent play cycles in Souls, then it actually reinvigorates what you're saying: each playthrough is a much more "thrilling and new" experience because suddenly that boss is using moves you never saw the first playthrough that you took 10 times learning, or it's stringing together attacks with new patterns that require completely different responses.  Hell, imagine if they took the time to completely change the weapon for the next cycle, so all of a sudden the same boss went from slashing sword attacks to Stabby McStabberson with a spear!  I genuinely think AI is a weakness in our current pursuit of graphical prowess, and it needs improvement or the uncanny valley strikes beyond just appearances.

All you're describing is not AI learning, you're asking for the same polished fights yet changed up for higher difficulty, Basically that's what DS2 is, and then DS3. Same stuff basically, different attack patterns and weapons (and a new lick of paint of course) It's all still set patterns, the enemy you're fighting isn't learning. You're just learning its patterns and tells.

Can an AI basically make a new souls game. Probably. You can have AI generate attack patterns, probably even fine tune them after analyzing real player behavior. Then you can have the game analyze you as you play and change up attack patterns to those that best fit your style. Would people be happy with that? It seems that beating a difficult game only has meaning when everyone plays through the same thing. That's where this unwavering stance of don't touch the difficulty comes from with souls games. Plus what would content creators do? :p Guides on how to beat bosses would become meaningless!

One of the earlier examples of 'AI' adapting difficulty/behavior in games is rubber banding in racing games. First priority is to give the player a close race, stay with him, bump back, then back off near the finish in you're ahead of the 'set' time. It was painfully obvious in DriveClub. All the opponents had a target time to finish, but would stay near you during the race, being more aggressive if you did better early and being more lenient if you struggle first. GT Sport the same, go fast early, AI gives chase, go slow early, AI holds up. And everyone hates those systems. In Gt Sport it's like herding cats. Pass one, they suddenly wake up and give chase, even passing the car that was in front of you that they were dozing behind until you show up from the rear.

Whooooaaa there, I'm addressing Souls in particular and how previous posts are mentioning the complexity and cost, or in particular, the poster that's mentioning how it can be done without using AI learning.  I, myself in the video, am clearly talking about AI adaptability in which it DOES have to learn to some extent, but here on the forum I'm also addressing the current state of games like Souls and Monster Hunter and how simple changes can be highly effective since both rely heavily on cycles (read: grinding).  Once again, not looking for "human" AI, but am also discussing how simply tiered skill difficulty AI routines can be applied to currently considering "difficult" games, ergo Souls. Once again, amiibo CPU fighters are a surprisingly good example of how it can be applied without AI machine learning until said AI machine learning becomes more accessible/widespread/widely adopted.

You are right that "game guides" on how to beat bosses would become meaningless, but that's a GOOD thing.  Granted, on easier modes, the vast majority of players don't need guides.  For people looking for a challenge, the idea that they would then also look up how to beat the boss seems counterintuitive: the skilled players are NOT looking at game guides for how to beat bosses lol... I think they would welcome the change-up!

Rubberbanding, however, is not adaptability.  That is, in and of itself, simply metadata adjustments (seriously, making a car go faster, or slower, or selecting its place in the pack, that's all just conditional changes similar to the previous poster mentioning ammo/resource adjustments based on player performance).  This is why I kind of chuckle at Xbox still being the overall most hated of the three as they have done backend things that most "gamers" would completely fail to appreciate, and one of those is Drivatars.  Sure, there are still programmed confines to Drivatars, but I can tell you that it's implemented better than people imagine it is, and it uses simple AI learning processes where a human player is present to learn data from and apply to their digital offline AI racer.  Otherwise, they kind of randomize the behaviors where no data is present.  Is there still a target difficulty that the AI drivers try to achieve based on the selected difficulty by the player like what you described? Yes, I've tested it.  But it's still a step forward in the right direction when cars will brake early or brake check me, or side swipe me or pit me at all what is seemingly completely random and is what keeps even a simple offline race potentially "dynamically difficult".  When the AI behaves out of expectation, it's what challenges the human lmfao

Honestly, Game difficulties shouldn't exist to begin with, for some of the reason you said.

Most games just deal with different difficulties as if its a slider that turn up or down and adjust enemy health/damage/defense, which in turn utterly unbalances most games, making them stupid easy(you can withstand most damage and not worry about dying) or unfairly hard( you die super fast, and enemies become hp sponges because the game needed them to ebcome harder without actually working on their moves, behavior, etc)

In order for games to actually have difficulties that actually represent their difficulty without making them cheap think Halo hardest difficulties in whic most enemies 1 hit kill you or cl;ose to it), you would have to rework so much of the game, like its level design, enemy behavior, attack patters, etc. that you would be basically putting in the same ammount of work and time as if you were doing a new game.

Maybe AI will be good enough in the future that this will not be a problem, but for now AI cant do all these stuff I just said. Even if they were to only handle enemy attack patterns to change betwenn difficulties, I would imagine that it would be alot of work(And I do mean alot) for something that, at the end of the day, its meaningless.

Carefully ahndcrafted experiences that doesn't compromise the vision is one of the reason that games like Dark Sould ands Cuphead became as famous as they did.

I talk more about this topic in this thread:

https://gamrconnect.vgchartz.com/thread.php?id=241866&page=1

Doesn't FH4 use Drivatars, cause the 'AI' in that was horrible. So bad I set it to the easiest difficulty so I could just do time attacks. The 'AI' was ruining the game for me. That's a fundamental problem with racing games trying to simulate human behavior. The average player can't race for shit. Sure, they can use their car as a weapon and knock you off the road, but if you want a tactical close race, you need to get in the top tiers in GT Sport or iRacing. I honestly prefer DriveClub's sometimes murderous AI over FH4.

The funny thing about AI is, once you have it, it's not really AI anymore. Just another set of routines. I studied AI in university and all the stuff we learned there is now common stuff, simple routines, from language parsing (Is Alexa AI?) to route planning or path finding in games.

Anyway my point was that adaptive gameplay would undermine the Souls 'ego' boost. The whole point of those games is one fixed difficulty to master. But maybe there is another niche that would enjoy enemies becoming smarter and mixing things up to counter your play style. As it is now, Souls games are like a dance routine. You die, start back at the bonfire and it's basically a rhythm game to get back where you were to learn the next sequence. Now if the enemies get smarter and suddenly feign one attack to do another to catch you off guard (like skilled human players in racing games), dying becomes 10x more tedious!

The problem with game difficulties not existing is it excludes millions of people with disabilities or who just aren't blessed with the best reflexes or fine motor control skills.

The only reason I can enjoy most of the games that I do is because they have an easy mode that allows me to relax and enjoy them without becoming stressed and unhappy because my impaired motor skills keep getting me killed, or just plain being locked out the whole experience because I'd come up against a section that was impossible for me to beat.

Never once in all my time as a gamer have I experienced a game's design feeling compromised by it offering multiple difficulty options.

There are multiple ways to increase difficulty meaningfully without anything so sophisticated as improving AI.

Touhou 11:
In TH11 (a Shmup), you have two playable characters each with 3 selectable setups. I noticed that Satori (Stage 4 Boss) used different spell cards depending on the character I was playing. Later I watched videos on Youtube where Satori casts different spells from either of my playthroughs. It turns out that the "setups" was what triggers the change in her spell cards. On top of that, the 4 difficulty settings have an affect on the number, speed, and patterns of projectiles. The variety is just sweet.

Resident Evil Remake:
You had a mode where each Safe Room was assigned with its own item box rather than one box shared between all rooms. This made inventory management a lot more challenging, and I for one loved that. Then there is also Invisible Mode and the epic Bomb Zombie :D

Devil May Cry:
In the highest difficulty setting, enemies begin to transform if you don't kill them fast enough which doesn't only increase their stats (boring) but also increases their resistance to launchers and being staggered or knockdowned. It just changes the combat dynamics and sense of emergency.

Onimusha 3:
I don't know if Devilish difficulty had an impact on enemy behavior, but Critical Mode (an unlockable toggle separate from difficulty) made the 2nd walkthrough feel quite fresh and challenging on Hard.

Parasite Eve:
I'll never forget the mini hearattack I had when an invisible GOLEM (mid-late enemy) grabbed me by the neck at the beginning of PE2's Bounty Mode. Every mode you unlock makes exciting changes that encourage replaying. Facing highlevel enemies at earlier stages felt amazing. Unfortunately, school and time constraints meant that I didn't get around to playing "the highest" difficulty settings. But I liked what each had to offer.

NieR Automata:
Finally, a modern game with an excellent difficulty implementation that resembles an AI improvement! It's immediately noticeable upon a 2nd walkthrough that the hardest difficulty impacts the BEHAVIOR of enemies itself! It literally took seconds for me to notice it "Those motherfuckers shouldn't be dodging me like that! What's going on?! xD" Sounds amazing, right? Well... it isn't. Because the same difficulty setting force disables lock-on, which feels like an artificial increase in difficulty that compromises control and the system the game is designed around. It should have had the option to change enemy behavior without forcibly disabling lock-on.

NieR Automata was a frustrating experience. Normal difficulty was too easy, the hardest difficult is nice but enabling/disabling lock-on should have been a separate toggle. If they did that, and assigned a dedicated dodge button separate from the dashing/running button (so you wouldn't just button mash your way through), it would have been one hell of a game. Simple changes can make a huge difference.

From my limited experience, new games utterly fail at implementing a rewarding "hard mode". I'd much rather have game lengths shortened by 10%-20% if it means 2nd playthroughs and higher difficulty settings are more meaningful and inviting. A shame even From Software fails to recreate this feeling.