You aren't getting enough power into a 500 - 1000 USD box to perform real time ray tracing at an acceptable quality level.
Not this generation, not for the next gen.
Go look at the real time ray tracing demos from GDC.
If 3000USD+ GPUs can only barely do it, on today's games, how much compute power, memory, memory bandwidth, ect, is is going to take to do it on games that are vastly more graphically complex.
1080p30, with a still scene and 8th gen level visuals (with ray tracing on top).
They are using a hybrid approach (raster/ray trace), and they have thousands of dollars in GPU at their disposal.
I don't think Microsoft/Xbox could afford to build and sell a box capable of running tomorrow's games with ray tracing properly utilised.
Maybe for some effects like screen space reflections, that have issues normally, they could use ray tracing to help fill in the gaps.
You're not getting the full ray traced experience.
Not a full resolution, and/or 60 FPS.
Not in a game changing manner, not on consoles.
"Single ray per pixel on 1080p is roughly 5ms"
That's on a powerful (or multiple powerful Volta GPUs).
This is only one portion of the rendering, so no, you're not getting this at 1080p or 4k next gen.
Ray tracing will be for high end systems, like high end PCs, with one or more high performance GPUs.
An FPGA or ASIC designed for ray tracing at this kind of quality, would be very expensive, and power hungry, and memory hungry, ect.
Next gen games are going to be more complex than this, and adding ray tracing on top, is going to be extremely difficult for hardware to do, for the next 5 - 10 years.