The bad part would be all games needing downloads to have the better visual on XSX.
Not all. Some like Cyberpunk won't have their next-gen patches release until after the game itself releases, so obviously on both PS5 and XSX people have to download the Cyberpunk next-gen version even if they bought the disc. But on games that already have their next-gen versions ready on release, there could be enough disc space left for the next-gen assets, just depends on the size of the game.
But honestly, with disc sizes being what they are (100 GB max), and some current gen games already being over 100 GB, I honestly think we're going to see alot of physical games that still require a download, regardless of platform. Better compression will help some, but still I'm expecting to see lots of 100 GB+ open world games. Publishers have proven they are too cheap to put 2 discs in one case these days, so if a game goes over 100 GB you're going to be forced to download part of it.
From what you said I misunderstood that all games with crossgen (not the only X1 that works because of BC, but the ones using this label) would need the download. But still even before the console launches they could already have to code to be on disc.
The two lastest physical games I bought had 2 discs, FFVIIR and TLOU2, I would bet additional 0.5 or 1.0 for the media is possibly less expensive than rates they possibly pay for the server to MS or Sony.
Like several of my android games, they have a download or update on the story of less than 200 Mb but then when you enter the game there is a 2Gb download (probably they have to pay more for the size on the store).