Okay I'm now convinced ARM is the way to go and Xbox should go with ARM after what I've been told
The only issue is Backwards Compatibility but without that, they will really hurt Game Pass by killing a large chunk of peoples libraries so if they went with ARM, I assume they would come up with some sort of emulation solution for Xbox Series X/S titles.