The bad part would be all games needing downloads to have the better visual on XSX.
Not all. Some like Cyberpunk won't have their next-gen patches release until after the game itself releases, so obviously on both PS5 and XSX people have to download the Cyberpunk next-gen version even if they bought the disc. But on games that already have their next-gen versions ready on release, there could be enough disc space left for the next-gen assets, just depends on the size of the game.
But honestly, with disc sizes being what they are (100 GB max), and some current gen games already being over 100 GB, I honestly think we're going to see alot of physical games that still require a download, regardless of platform. Better compression will help some, but still I'm expecting to see lots of 100 GB+ open world games. Publishers have proven they are too cheap to put 2 discs in one case these days, so if a game goes over 100 GB you're going to be forced to download part of it.