The idea is using convolutional neural networks for resolving low resolution image output from gaming machines to a high resolution one. why it is still not happening?
First let me put some concepts:
-super resolution (SR) is obtaining a high resolution image/video from a low resolution one. Yes, it seems like a magic that generate missing information. the simpler methods are using interpolation (upscaling), that are used in every tv, console, HDMI cable, etc.
-Neural Networks are a machine learning technique that using layers of 'neurons' in order to learn something. Given tons of classifyed training data, the training process embed in the network the weigths that generate the trained data. Is important to note that training takes very long time, while classifyng data with already trained net is very fast. It can be very specifyed depending on training data, in such way it can increase the performance of input data if we already know it could be next to the trained data.
- In the SR application, the input is a low res image, and the output is a high resolution one. (Convolutional neural networks (CNN) allows to work with big chunks of data instead of just classes). The training process uses millions of very diverse low res images and the corresponding high res image by a given factor. After the net is trained, you put on the net your low res image (even if it does not is part of training data) and it generates the correspondent high resolution image. This process is very fast, and the quality of generated image is superior that other SR methods.
Well, the idea is to use this powerful weapon in gaming, adapting it to the specific application:
-In a powerful hardware, they generate the 2 versions of the rendered image: one with the full resolution and AA, and other in low resolution/no AA that would be the result on the power limited gaming machine.
-Train a CNN with the low and high res images (millions of them) to get a good model that generate high res images from low res images. As they know exactly the kind of images they will get as input (the net will only be used on the images of the game), the net will be very accurate and the results tend to be even better than the regular SR CNN.
Justify/advantages:
-Better Image quality when compared to regular upscaling and other methods. In a world where the resolution is increasing, and, as consequence, the perceptual difference of native full resolution to upscaled one is decreasing, the SR CNN can be undistinglishable from native full resolution.
- Developers could develop games focused more in performance, effects and details, not exactly in resolution/AA. as they know the result of the upscaler could bring the quality of image up to full resolution levels with a perfect AA, other aspects can be optimized.
-Less powerful machines would be able to run high resolution games. Render natively with high resolution is very costly in terms of computing power and power consumption. SR CNN is very efficient and has low demand for hardware.
-Can be embedded on the consoles. additionally, it can provide a high resolution mode for it (-example 1, that would be the only way to have a power up dock on switch. The dock would still receive only the low res images, and would output the high resolution ones passed by the SR CNN. -exemple 2: PS4 slim/one S could match resolution of PS4 pro/ one X just by adding an extra simple cheap hardware for the SR CNN), and that could also cheaper the consoles, and let the cheaper ones with a high-end aspect.
-Developers that do not trained for their games could use already trained profiles of networks. For example, Sony could train nets for realistic games, cartunish games, and a lot of visual styles. Developers choose one for their game, and that does not bring any work overload for developers. (despite the results in image quality wouldnt be maximized specificaly for their game.)
Concluding, I cant see why not applying that. Anyway, maybe im too optimistic. I am just starting to read things about CNNs and got facinated on the results(i still did not get on practice yet). If I commited some mistake, or to know why it wouldnt be viable, I would like to know in the comments. And general thoutghs too. Thank you.
