That's insanely impressive, but at the same time, the end result looks very different and generic. If you ask me, clearly WoW is not where DLSS 5 should be applied, at least like this, but it's still a nice showcase of what the tech is capable of.

I agree. It is brilliant as in it makes every game look brighter.

WoW video just removed any of the style and becomes just some shiny character unity game

And all that while dropping from 300+ to 30-40fps.

Honestly, I think DLSS 5 is a dead end, given how little it actually knows about the frame and what happens outside of it, and approach where neural rendering is incorporated deeper into the rendering pipeline is a way forward.

DLSS5 actually has quite a bit of input information, according to their SIGGRAPH presentation. 

It has albedo buffers, surface normal buffers, miscellaneous lighting information, motion vectors, and of course engine-level masks. 

Most of the differences so far look on-par with something like RT Remix or adding realistic lighting to older titles. That is obviously going to significantly change how things look. Especially the chiaroscuro effect. 

It will be interesting to see how games with it as part of the development turn out. 

As for performance, model distillation will almost certainly solve that. 

I do think small online NN's are the future though. My guess is that we need some sort of generalized meta-optimization model to coordinate thousands of small (hundreds to thousands of parameters) online models/neural shaders through selecting optimal hyper-parameters. It would probably be smaller than DLSS 5's model, on the order of a few million parameters rather than I am guessing 50-400M params for DLSS5.

It is also important to remember many of these mods are using inferred rather than engine-level direct inputs. This will have both a performance and quality impact. Like any game that doesn't have motion vector support is using optical flow models to infer them. Nvidia mentioned direct engine-level motion vector inputs in their SIGGRAPH presentation.

@sc94957

IIRC, DLSS 5 is not aware of depth buffers, geometric coordinates, material properties and off screen light directions.

What I'm saying is that future of neural rendering shouldn't be post process thing, but directly inserted into pipeline (like what AMD's "Temporally stable generative illumination with a one-step diffusion model" paper describes). DLSS 5 feels like "be amazed, buy our GPUs" band-aid approach, not a proper future solution for high cost of intensive path tracing.

It definitely is aware of at least some material properties. Albedos are a material property, as an example, and Nvidia mentioned them as an input. 

https://youtu.be/8X0uDU6Ru3w?is=D_LQqfy97JuFmwxE

"We used some of the renderer's internal buffers such as albedos, surface normals, and some of the lighting information to make sure the model can understand what it has to stay consistent with." 

Depth buffers are also an input, as all of these unofficial mods pass that data into the model as an input. 

This is a compatibility matrix for Easy AIO. Notice that when a game doesn't have DLSS available it requires a "feeder." The role of this feeder is to act as a virtual config which provides data for the DLSS5 model when a game doesn't already support DLSS. 

There is an HLSL shader called DLSSFeed.fx that does this. Below is the comment in that file.

The minimal inputs to get the model to even work are motion vectors, depth buffers, color buffers, and validation masks. 

You also notice that for games that don't have DLSS, motion vectors have to be inferred using very basic optical flow methods like VORT or Launchpad.

Comment from DLSSFeed.fx

"It turns what ReShade already has into the guide textures DLSS needs, in the exact layout the add-on expects:

    DLSS5_MV     RG16F   motion vectors in PIXELS, pointing from the current pixel to where it was in the previous frame (DLSS convention). Vectors that fail validation (below) are zeroed.

      DLSS5_Depth  R32F    the game's raw hardware depth (not linearised), sampled at backbuffer size, with ReShade's RESHADE_DEPTH_INPUT_* orientation fixes applied.

      DLSS5_Mask   R8      "bias current colour" mask for DLSS: 1 where the motion vector could be trusted, so DLSS leans on the current frame there instead history in. "

----

"MOTION VECTOR PROVIDER -- set the DLSS5_MV_PROVIDER preprocessor definition (ReShade overlay:
this effect's "Preprocessor definitions", or the global list) and enable that provider's
technique ABOVE this one in the effect list:

0 texMotionVectors the community-standard shared texture: qUINT_motionvectors,
dh_uber_motion, ReshadeMotionEstimation (DRME -- NOTE: DRME does not
compile on ReShade 6.8, "cannot sample from texture that is also used
as render target"; it then silently writes nothing) [default]
1 Launchpad iMMERSE Launchpad (MartysMods_LAUNCHPAD.fx): Deferred::MotionVectorsTex.
Launchpad only runs its optical flow when asked to, so this mode also
files that per-frame request (Launchpad's IPC buffer, see below).
2 VORT vort_Motion.fx (MIT): MotVectTexVort -- the recommended provider
3 LumeniteFX Kernel lumenite_Kernel.fx ("LUMENITE: Kernel"): Kernel::tFlow -- pyramidal
optical flow with per-level median + a-trous filtering and previous-
frame seeding. 1/8 resolution, upsampled here. Needs no depth buffer.
4 LumeniteFX QuantMotion
lumenite_QuantMotion.fx: QuantMotion::tFlow -- the light cut of 3.

This is the same mechanism dh_uber_rt (USE_MARTY_LAUNCHPAD_MOTION / USE_VORT_MOTION) and
vort (V_MV_MODE) use: the selected provider's OUTPUT texture is declared here exactly as the
provider declares it, so ReShade binds the same resource, and only that one is allocated.
Every provider above hands out delta UV with prev_uv = uv + mv. Nothing of any provider is
included or bundled: this file contains no third-party code and includes no third-party
files beyond ReShade's own headers."

Edit: 

And here is the C++ code that implements the feed. 

https://github.com/jlrouzies-fr/DLSS5-Feeder/blob/main/src/dlss5-feed.cpp