The majority of that extra RAM in XSX is for getting the games running in 4K. You can see this already in PC - take a PC game, max out the settings and run it in 1080p. Maybe it will get to 4-5GB of the GPU's RAM usage. Now, using the same settings, run it in 4K - watch the RAM go up to more 7-9GB of usage.
There has already been a great article showing how a 4TF GPU could deliver the exact same games as a 12TF one, only running them in 1080p versus 4K. You really need that much more power in order to push that many pixels.