The graphics rendering isnt actually what adds Latency.... its the "other stuff".
Your "idea" would lower the amount of bandwidth needed to stream, but the Graphics part of a console is what makes up most of its cost.... so you ll basically be paying almost the same as a complete console, for a lesser experiance? why would anyone do that?

So your thinking is kinda backwards.... also why Microsoft is rumored to do it the otherway around, of what your saying.
Ms plan was to keep the input and cpu calculations in a box you buy, and have them do the rendering server side and stream that to you.
That way you get a streaming box, that has less latency.

He basically got it backwards :)
The complete oppersite of what MS was rumored to be doing, to help reduce latency when streaming.

His idea would reduce the amount of bandwidth needed to stream greatly, but it would basically require a total console.
The "price" saved aspect would be minimal to the point where, you might as well just have a real console then.

Like why would anyone want to be forced to stream, if they for the same money could get a actual real console, that gives a better experiance?

It is what ms promised with the xbox one

Not quite the same. AFAIK it was mostly about particle effects and such, which could be introduced as some sort of filter over the original, console-generated image, so even if there was latency, it wouldn't affect the game. No explanation was ever given, however, as to how the streaming service would interact with the GPU frame buffer etc. and actually do what it promised to do. I suppose there was a way, and it was actually a thing at some point, before being shelved, so who knows.

Edit - an alternative could be, say, the local hardware doing a 1080p image with 8 TFLOPS and the streaming service using ~ 2 TFLOPS or so to upscale the image to a very respectable 4K with checkerboarding. But again, the latency would have to be frame buffer-sized or it would just be as bad or worse than standard streaming.

Your Analogy is actually spot on.

The codecs are already in AMD's video decoder though, albeit... It's fallen severely behind nVidia in recent times. (The exception being the Ryzen notebook APU's.)
H.266 should provide significant reductions in bandwidth and enable 8k.

There are so many ways to skin a cat.

In my opinion... If latency is the biggest issue, then you want as much work performed on the host hardware as possible... But then offload whatever is not latency sensitive to the Cloud... Such as weather effects.