| HoloDust said: @sc94597 |
Ah, that paper was actually helpful. So the g-buffers are used in the training objective with derived features (to better meet the objective) doing the heavy work to maintain semantic consistency in inference.
This is what they say about the topic.
| Artistic intent preservation. The output preserves what the artists authored: geometric structure, object and character identity, material identity, composition, and the organization of light and shadow. Nothing enforces this at runtime — the supervision that encourages it acts during training, and every generated change carries some risk of drifting from the authored scene. Section 2.1 describes the supervision, and Section 4 measures the preservation. |
| Conditioning on rendered RGB alone, however, provides a very strict constraint on the output that introduces challenges in achieving high realism. RGB entangles scene content with the appearance already produced by the renderer. If the model follows the input too literally, it tends to reproduce the same edges, colors, textures, and baked shading, leaving too little freedom to improve the final appearance. If the model is given too much freedom, it can drift from the authored content. We resolve this challenge during training rather than at the runtime interface. During training, we use renderer-derived albedo, surface normal, and lighting information to provide attribute-specific consistency supervision. Albedo provides illumination-independent chromatic information, surface normal provides local orientation information, and the lighting-derived signals supervise the authored illumination structure. These signals are used only during training and are not inputs to the shipped model. Artistic intent that is not already expressed in the rendered frame enters through the artistic-direction control of Section 3. |
| Existing image generative models commonly rely on text embeddings, exemplar images, or spatial control fields such as depth, edges, segmentation, and pose (Esser et al., 2024; Imagen 3 Team, 2024; Black Forest Labs et al., 2025; Wu et al., 2025; Li et al., 2024). These conditions are effective for general-purpose generation and editing, but they do not uniquely determine the object identities, materials, visibility relationships, lighting decisions, and pixel-aligned detail contained in an engine-rendered frame. DLSS 5 is therefore conditioned on the rendered frame itself. |
The immediate advantage I would see in providing the actual inputs would be performance. You can have a smaller, more efficient model. They probably will do that in a future iteration of it. Maybe they wanted to start out simple, release something, and then iterate. But I do think DLSS5 is a stop-gap before Nvidia moves to online learners (which they and AMD have already implemented into their PT pipelines, to an extent.)
