Neural Supersampling for Real-Time Rendering(research.fb.com)
research.fb.com
Neural Supersampling for Real-Time Rendering
https://research.fb.com/blog/2020/07/introducing-neural-supersampling-for-real-time-rendering/
47 comments
Nvidia recently demoed a similar technique running in real time: https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-your-q...
More than demoed, they've shipped DLSS in quite a few games now. The 1.0 version was underwhelming but the 2.0 version works extremely well in practice.
However Nvidia are treating DLSS as their secret sauce and not publishing any details, so Facebook's more open research is interesting even if it's not as refined yet.
However Nvidia are treating DLSS as their secret sauce and not publishing any details, so Facebook's more open research is interesting even if it's not as refined yet.
> However Nvidia are treating DLSS as their secret sauce and not publishing any details
https://developer.nvidia.com/gtc/2020/video/s22698
https://developer.nvidia.com/gtc/2020/video/s22698
There are no real details in that presentation, they explain what input data DLSS needs, and demonstrate the outputs it can produce, but gloss over how it actually gets there.
From the inputs you can glean that DLSS is using temporal integration but that's hardly a new idea, the novel part is in how it performs the integration and Nvidia isn't sharing that.
From the inputs you can glean that DLSS is using temporal integration but that's hardly a new idea, the novel part is in how it performs the integration and Nvidia isn't sharing that.
It is using TAA-upsampling and I think the only DL part is replacing the neighborhood clamping filter with a trained function.
You can look at the tradeoffs of neighborhood clamping/clipping here:
https://de45xmedrsdbp.cloudfront.net/Resources/files/Tempora...
Those slides were before TAA upsampling was added, but the issues are similar once it is in place too. Unreal Engine already has TAA upsampling, but has to fight against the issues mentioned in the Nvidia presentation.
Nvidia trained it for specific upsampling multiples, so the main functional difference with existing TAA upsampling is it can't do continuous base-resolution changes.
You can look at the tradeoffs of neighborhood clamping/clipping here:
https://de45xmedrsdbp.cloudfront.net/Resources/files/Tempora...
Those slides were before TAA upsampling was added, but the issues are similar once it is in place too. Unreal Engine already has TAA upsampling, but has to fight against the issues mentioned in the Nvidia presentation.
Nvidia trained it for specific upsampling multiples, so the main functional difference with existing TAA upsampling is it can't do continuous base-resolution changes.
[deleted]
It's basically just a good TAA and the tensor cores are basically used as a poor man's Lanczos. The tensor cores are used for obfuscation, and to make sure it doesn't run on AMD. Very little about DLSS 2.0 is "DL".
This is inaccurate.
[deleted]
Please be aware that the parent is Brian Catanzaro, VP of Applied Deep Learning Research at NVidia, who overviews the development of DLSS among many other things.
If he says that the GP is inaccurate, trust him.
Hi, Brian!
If he says that the GP is inaccurate, trust him.
Hi, Brian!
This appears to be more accurate than DLSS according to Fig. 11 in the paper, and has the 16x mode as well as 4x.
That was DLSS 1.0 they compared with.
Meanwhile DLSS is real time and this takes 100ms a frame.
DLSS is possibly baked into the graphics cards. It is what the RT cores are for. Facebook is just releasing a POC. I doubt the code is handwritten CUDA optimized for performance.
On Nvidia cards the RT cores are the ones handling BVH traversal and ray-triangle intersection operations.
The part of the hardware that is running the DLSS ML model are the tensor cores. But the algorithm and the model itself is not baked in, it is provided in the driver and/or game
The part of the hardware that is running the DLSS ML model are the tensor cores. But the algorithm and the model itself is not baked in, it is provided in the driver and/or game
Relevant performance details:
A Titan V GPU, using the 4x4 upsampling, at a target resolution of 1080p takes 24.42ms or 18.25ms for "fast" mode. This blows out the 11ms budget you have to render at 90hz (6.9ms for 144hz), and it doesn't appear to include rendering costs at all...that time is purely in upsampling.
Cool tech but a ways to go in order to make it useful for VR.
A Titan V GPU, using the 4x4 upsampling, at a target resolution of 1080p takes 24.42ms or 18.25ms for "fast" mode. This blows out the 11ms budget you have to render at 90hz (6.9ms for 144hz), and it doesn't appear to include rendering costs at all...that time is purely in upsampling.
Cool tech but a ways to go in order to make it useful for VR.
it doesn't appear to include rendering costs at all...that time is purely in upsampling
That part wouldn't be an issue if the plan is to render low resolution images in the cloud and stream them to a device that can upsample them locally. There wouldn't be any local rendering costs.
That part wouldn't be an issue if the plan is to render low resolution images in the cloud and stream them to a device that can upsample them locally. There wouldn't be any local rendering costs.
I'd be very surprised if this is what it's to be used for. The technique requires color, depth and motion vectors. That's three separate video channels, and two of them contain data that isn't usually stuffed into videos.
Any compression artifacts are going to stick out like a sore thumb, so you'll need to stream very high quality, and you're going to have weird interactions between different layers being compressed differently.
Any compression artifacts are going to stick out like a sore thumb, so you'll need to stream very high quality, and you're going to have weird interactions between different layers being compressed differently.
That might be the plan, but it seems redundant to require users to have high end gaming hardware just to stream games.
How close are we to working Foveal rendering with eye tracking so that upsampling only needs to be done for a small area of the screen?
Spending precious milliseconds perfecting the corners of the image for VR seems like a complete waste.
Spending precious milliseconds perfecting the corners of the image for VR seems like a complete waste.
Foveal rendering is in a weird spot. The software seems to be there, but mostly only in academia. The hardware is almost nowhere to be found, because it is another expense. People prefer spending that extra money on a better computer, since that improves every VR experience, not just the possibility of (part of) future experiences.
FVR needs a hook: what can it do that "dumb" VR headsets don't?
FVR needs a hook: what can it do that "dumb" VR headsets don't?
Do you need every single frame to be perfectly upsampled? Maybe there's a proportion of frames that could be rendered faster but with a less accurate method?
Oh interesting. I wonder if you could combine temporal antialiasing techniques with this to get a pseudo upsampling by only upsampling portions of the screen. Maybe focus on edges every other frame, and do different flat surfaces every few frames in between the edge passes. Then use TAA concepts to blend the pixels over frames.
You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.
Indeed, but you only need the part you are actively looking at in high resolution.
the bit of your eye that need high resolution is surprisingly small.
the bit of your eye that need high resolution is surprisingly small.
Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).
Add an additional 100ms of latency and people are going to throw up. Pipelining can be used to some degree, but definitely not what you are suggesting.
This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR.
ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable.
Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms.
Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver.
This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.
ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable.
Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms.
Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver.
This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.
If a post processing filter takes 18ms to complete, uncontested, you can't "pipeline the shit" out of it. You can get a frame at least every 18ms. Adding more load won't magically transform throughput into latency, it will just make your throughput worse.
Right. If it's 18ms before you get any result you can't do anything & that's the lower bound on the FPS you can display. The problem is with respect to the pipeline granularity you're thinking about. I'm not suggesting creating a deeper pipeline. I'm saying make the granularity smaller. Can the app simulate the portion of the world that's actually needed for display? Can the rendering engine get you the result for the first 1/4 of a frame in 1/4 of the time it would take to do the whole screen? Can the display pipeline understand portions of the same frame being submitted at different times? Ultimately can you drive different parts of the display at different refresh rates based on when those pieces arrive.
I'm not trying to make this seem like a trivial thing. The single global view is easy to reason about. We need fundamental research into this space on how to build physics & rendering engines that can unlock this kind of granularity. There's massive data dependencies that are really challenging to untangle. Each layer in the pipe, from physics engines & rendering engines, to display pipeline APIs, to GPUs likes to treat everything else like an opaque system with each providing a complete result to the next. There's an enormous amount of existing architecture that would have to be rewritten. I'm also 100% clear that it's entirely possible that parts of the pipeline can't be segmented in this way to get good looking result or it might make the SW burden on content creators to costly. I still believe it's the right direction though & is the only way you're going to ever see 8k displays @ 120hz or above. Well, that's not strictly true. The other trick will be to use motion estimation & neural nets to provide the interpolation when you drive it higher than the content can support - you can see that with all the motion estimation & supersampling improvements that various vendors are producing.
I'm not trying to make this seem like a trivial thing. The single global view is easy to reason about. We need fundamental research into this space on how to build physics & rendering engines that can unlock this kind of granularity. There's massive data dependencies that are really challenging to untangle. Each layer in the pipe, from physics engines & rendering engines, to display pipeline APIs, to GPUs likes to treat everything else like an opaque system with each providing a complete result to the next. There's an enormous amount of existing architecture that would have to be rewritten. I'm also 100% clear that it's entirely possible that parts of the pipeline can't be segmented in this way to get good looking result or it might make the SW burden on content creators to costly. I still believe it's the right direction though & is the only way you're going to ever see 8k displays @ 120hz or above. Well, that's not strictly true. The other trick will be to use motion estimation & neural nets to provide the interpolation when you drive it higher than the content can support - you can see that with all the motion estimation & supersampling improvements that various vendors are producing.
I work in graphics and this is complete news to me - not pipelining, of course, but the idea that frames can take this long to render and still reach the user at a steady rate. Do you have sources that describe what you mean in more detail?
I don't mean to say that a single frame can take 100ms to render & that you can somehow add more pipeline stages. That's silly because the slowest single stage of your pipeline is the bottleneck.
I was instead saying that if you could re-architecture the entire system end-to-end so that you could generate 1/10th of the frame in 1/10th the time, & there wasn't a resource contention between frame (i.e. you would start processing the next frame as soon as you're finished with the current one), then even though it took 100ms for the entire frame to scan out, each individual section is still updating at 100 fps. Similarly, if you could get update the display in pieces rather than the one-shot serial operation as it is now, then you'd see some serious changes to our understanding of latency & pipelining.
This is all easier said than done & was a total strawman taken to an absolute extreme. However, if the SW could get that pipelined, HW vendors would have a much easier time making their end of it more efficient.
I was instead saying that if you could re-architecture the entire system end-to-end so that you could generate 1/10th of the frame in 1/10th the time, & there wasn't a resource contention between frame (i.e. you would start processing the next frame as soon as you're finished with the current one), then even though it took 100ms for the entire frame to scan out, each individual section is still updating at 100 fps. Similarly, if you could get update the display in pieces rather than the one-shot serial operation as it is now, then you'd see some serious changes to our understanding of latency & pipelining.
This is all easier said than done & was a total strawman taken to an absolute extreme. However, if the SW could get that pipelined, HW vendors would have a much easier time making their end of it more efficient.
It's an interesting idea, but how would updating individual tiles at different rates appear to the user? Obviously if some parts of the frame remain static you could delay updating those, but how would it look in cases where the entire frame has changed such as during a camera pan? And how would it work with post effects like motion blur or DOF?
Maybe it would look good at very high frame rates, but would the individual tiles updates become visible if your app encounters slowdown and FPS drops below 30?
It's an interesting idea, and thanks for providing more info. However, I'd need to see a proof of concept, including post effects, before I considered it further.
Not in VR. You do that and you’ll induce vertigo.
My thoughts exactly, I didn't see anything on the paper addressing that.
I suppose it's assumed that with the contributions of this one, future work can be done to make it faster.
I suppose it's assumed that with the contributions of this one, future work can be done to make it faster.
Yeah, I wonder if they plan to fab an asic (FAIR already has their own accelerator asic) to run this on the next Oculus.
scanning through it for the clause that gives away what slight of hand was used to correctly get BERLIN from nothing. Suspects
> and combines the additional auxiliary information
> multiple framesNo idea what "some of the input" means, or why you thought "Low Resolution Input" is disingenuous?
It uses color, depth and subpixel motion vectors of 1-4 previous frames. All things that modern game engines can easily calculate. You didn't even need to read the paper to get this info, it's literally in a picture on the blog post.
It uses color, depth and subpixel motion vectors of 1-4 previous frames. All things that modern game engines can easily calculate. You didn't even need to read the paper to get this info, it's literally in a picture on the blog post.
Right - so a single low-res image should not be paired with the high-res one and labelled as input and output, because that implies the algorithm turned the one data into the other, which it did not do.
This isn't about upsampling low-res bitmaps. It is a technique for upsampling the output of a game engine.
The low-res image is itself output generated from a lot of other input the game engine generates. That same input that is already being generated anyway can also be fed into this to improve the post-processing. Finding ways to productively reuse already existing/generated data is the hallmark of any top (graphically) game.
I read a detailed write-up on the graphics pipeline for GTA:V on Xbox 360. It blew my mind how many different ways they reused every single bit that ever hit the RAM. Which explains how they pulled off those graphics on a system with half as much RAM as an Apple Watch.
The low-res image is itself output generated from a lot of other input the game engine generates. That same input that is already being generated anyway can also be fed into this to improve the post-processing. Finding ways to productively reuse already existing/generated data is the hallmark of any top (graphically) game.
I read a detailed write-up on the graphics pipeline for GTA:V on Xbox 360. It blew my mind how many different ways they reused every single bit that ever hit the RAM. Which explains how they pulled off those graphics on a system with half as much RAM as an Apple Watch.
Isn't this literally NVIDIA's dlss? And it has already been productized.
Except Nvidia has published pretty much nothing about their method.
They did publish a video on how it works, but I can't find it right now.
The inputs are similar:
https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/n...
In contrast to DLSS1, the output of the NN is not color values, but sampling locations and weights, to look up the color values from the previous low-resolution frames.
The inputs are similar:
https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/n...
In contrast to DLSS1, the output of the NN is not color values, but sampling locations and weights, to look up the color values from the previous low-resolution frames.
Would love to add this to my game engine.
Did you read the paper? Or the benchmarks at least?
In its fastest mode, it takes like 18ms. Not even usable in real time if you target 30fps.
Great start but definitely needs additional work to be usable in games.
Great start but definitely needs additional work to be usable in games.
There is a big difference between latency and throughput. FPS is throughput. If you assume the entire system is producing only the current frame then those numbers are directly correlated. But most systems, especially game engines/hardware, always have multiple things going in parallel simultaneously.
The H.264 encoder on my CPU introduces >16.7ms of latency into a video stream, but it can encode hundreds of frames per second of SD video all day. Adding ~1 more frame of latency may be worth a quadrupling in image quality/resolution in most circumstances.
The H.264 encoder on my CPU introduces >16.7ms of latency into a video stream, but it can encode hundreds of frames per second of SD video all day. Adding ~1 more frame of latency may be worth a quadrupling in image quality/resolution in most circumstances.