Well, open world games this gen do tend to run a lot better than they did last gen. It's mostly a developer convenience thing; devs no longer have to do insane memory management hijinks to work around harsh memory restrictions, which makes their job easier. Nowadays they may focus mostly on memory-hungry textures, but that's only cos now they have the luxury of doing so; back then it was a struggle just to get the fundamental design of an open world game to run decently, before even worrying about textures.
By say, 2008, PS3/360 were still fairly capable in the CPU/GPU department, (far from cutting edge, but decent) but even by then 512MB was just miniscule. Their GPUs continued to pump out great looking games throughout their lifespan, while their RAM limitations became quite evident long before their replacement.
That said, I definitely agree devs did amazing things on last gen; there are still PS3/360 games that I can boot up today and be like "holy shit, how the hell does this run on less than 500MB of RAM and a GPU from 2005?" Just this year I replayed Gears 3 and Halo 4 expecting them to have aged poorly and was taken aback by how good they still look.
Everything tends to run better than last gen because is a greater focus on stable performance. Later in the generation we saw greatly improved performance, but you still saw technically impressive stuff like Sleeping Dogs or Far Cry 3 running pretty bad. Meanwhile, GTAV was stunning and maintain fairly stable performance.
You keep saying bringing open world games to last gen was a struggle, but it was done many time over. It also depends on what the game was attempting to do. For example, we've seen Bethesda games struggle when there are too many changes in the world, which is saved in some sort of memory. But the average open world game isn't doing that. Maybe the way Bethesda games did it was also inefficient, their engine(s) aren't exactly praised. The way you speak though games like Assassin's Creed should never have existed on last gen or even improved with each release.
I started playing Gears 3 recently and it especially looks fantastic on the X1X. Halo 4 is the more technically impressive game of the two, because its a larger scale action. The campaign maps are like small open world with lots of enemies, vehicles, etc. It even does that in split screen which is like an anomaly in modern games. Even multiplayer can have up to 16 players on a large map with vehicles, some of the most fun I had last gen. I'm sure it took effort to make all of this work on ~500MB of RAM, but I feel the average game was struggling more with the GPU limitations and were more often pushing the limits of that.
Gears 5 for X1X (5/5) - Mortal Kombat 11 for X1X (5/5) - Doom 64 for N64 (emulator) (3/5) - Crackdown 3 for X1S/X1X (4/5) - Infinity Blade III - for iPad 4 (3/5) - Infinity Blade II - for iPad 4 (4/5) - Infinity Blade - for iPad 4 (4/5) - Wolfenstein: The Old Blood for X1 (3/5) - Assassin's Creed: Origins for X1 (3/5) - Uncharted: Lost Legacy for PS4 (4/5) - EA UFC 3 for X1 (4/5) - Doom for X1 (4/5) - Titanfall 2 for X1 (4/5) - Super Mario 3D World for Wii U (4/5) - South Park: The Stick of Truth for X1 BC (4/5) - Call of Duty: WWII for X1 (4/5) -Wolfenstein II for X1 - (4/5) - Dead or Alive: Dimensions for 3DS (4/5) - Marvel vs Capcom: Infinite for X1 (3/5) - Halo Wars 2 for X1/PC (4/5) - Halo Wars: DE for X1 (4/5) - Tekken 7 for X1 (4/5) - Injustice 2 for X1 (4/5) - Yakuza 5 for PS3 (3/5) - Battlefield 1 (Campaign) for X1 (3/5) - Assassin's Creed: Syndicate for X1 (4/5) - Call of Duty: Infinite Warfare for X1 (4/5) - Call of Duty: MW Remastered for X1 (4/5) - Donkey Kong Country Returns for 3DS (4/5) - Forza Horizon 3 for X1 (5/5)