| Zkuq said:
Is this more of a hobby project, or do you perhaps have some grander plans for this tool in the future? |
It started as an experiment because I wanted to see if I could use feature embeddings that capture mathematical/statistical surprise in an image to detect regions of artifacts in the image, and then use those artifact-detections to help guide a relatively lightweight CNN decoder to clean up the image, mostly by masking regions of non-correction before applying corrections. My hypothesis was that areas of high surprise would be flagged as "artifacts"/"anomalous" and the decoder - given high quality examples (in this case the same image with DLAA applied rather than say FSR Performance), would adjust the "artifact" regions to more closely match the higher quality output through masking, slight color value corrections, contrast corrections, etc. The artifact-detector features would help prevent global corrections that add additional artifacts.
The models were small, so they could train on any GPU with at least 12GB of VRAM within half a day before the networks got saturated.
Then it pretty much evolved from there. I needed a way to test it in-game after the model was trained, since it was hard to tell subjective image quality from stills and short clips alone, so I had to basically recreate how Lossless Scaling does it using the WGC API. This was the hardest part of the work for me, because I don't have much knowledge on Windows systems programming. But eventually I was able to get it to work with CUDA and TensorRT on a cloned game Window. I also tried DirectML to support non-Nvidia GPUs but it isn't performant enough to make it worth it, at least without more effort. Will have to look at better ways to accelerate this on AMD/Intel GPUs in Windows. Maybe will try Vulkan. Also the capture doesn't work right with HDR on, so I have to figure that out.
Currently experimenting with optical flow models to see if they'll help with temporal artifact clean-up, because I am pretty happy with the spatial clean-up, but temporal artifacts look identical to the lower-quality upscale, if not slightly worse.
It might end up being something I open-source. If I did that, I'd definitely want to figure out the AMD/Intel hardware acceleration problem because people with those GPUs are probably who this would benefit the most. Might even look into seeing if I can get it working on NPUs that are being paired with most new systems.
