Enhancing Photorealism Enhancement(intel-isl.github.io)
intel-isl.github.io
Enhancing Photorealism Enhancement
https://intel-isl.github.io/PhotorealismEnhancement/
153 comments
Maching learning is often oversold as a shortcut solution for some very hard problems and this is a perfect example.
What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.
What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.
Actually, I'm pretty amazed by the results. This is indeed a giant leap. Also it's coming from Intel instead of AMD or Nvidia, which makes me hopeful about its wide availability.
The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.
Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.
The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.
Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.
> The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.
RDR2 is based on further development of the GTA5 engine and is arguably the best looking open world game at this time.
RDR2 is based on further development of the GTA5 engine and is arguably the best looking open world game at this time.
>They've probably grafted a lot of tapping points to video driver to be able to implement this.
They didn't mention using anything else than output frames and G-buffers.
They didn't mention using anything else than output frames and G-buffers.
I'm not an expert in graphics programming but I don't think that accessing G-Buffers of an application and grafting your ML based post processor into the pipeline is exactly trivial. From my point of view it's only possible by either engine provided tapping points, or you tap the processing pipeline at the driver level directly.
I'd love to be corrected if I'm wrong, BTW.
I'd love to be corrected if I'm wrong, BTW.
You need to hook into game process, but it's nothing too hard. Either patching game binary in right place to funnel pointer to needed buffers to your DLL, or hooking something like D3D11 OMSetRenderTargets and doing some clever guessing which one of buffers this is. Doing this at driver side wouldn't make any difference, as driver doesn't have any extra semantic info what the buffer is used for anyway.
> You don't want your games to look realistic?
The creators aren’t necessarily aiming for photorealism. They want their games to look good enough and be entertaining. Adding more realism could make it look worse. Obviously if AI could solve the hard problems like global illumination that’s great. And I’m sure it’ll be possible to e.g easily add more variation to textures by using larger datasets. But changing the tonemapping to a colder palette, or just replacing the asphalt textures with smoother ones?
The creators of GTA didn’t use the sunset tones in the game by some creative mistake or lack of computing power. They obviously went for non-photorealism deliberately! It’s a stylized view of California. It’s like the burger in the commercial doesn’t look like the real one - it looks a bit unrealistically perfect because that’s actually more attractive than the real thing. Realistic and photorealistic are tangentially related.
The comparison here should perhaps be between frames of the game that already attempted to look like the cityscapes dataset.
GTA obviously doesn’t have a ground texture that isn’t smooth because it wouldn’t be possible, or “too warm” palette because it’s too computationally expensive to have the bleak cityscapes tone mapping.
A GTA frame rendered at its current frame cost, with textures the same size (but different) and a customized tone mapping, would look more photorealistic and make the difference to the cityscapes dataset much much smaller.
The authors could likely have done this comparison since they probably have the ability to modify textures and shaders to set tonemapping. That would have separated out the differences that are only artistic from those that are lacking photorealism because of computational expense.
The creators aren’t necessarily aiming for photorealism. They want their games to look good enough and be entertaining. Adding more realism could make it look worse. Obviously if AI could solve the hard problems like global illumination that’s great. And I’m sure it’ll be possible to e.g easily add more variation to textures by using larger datasets. But changing the tonemapping to a colder palette, or just replacing the asphalt textures with smoother ones?
The creators of GTA didn’t use the sunset tones in the game by some creative mistake or lack of computing power. They obviously went for non-photorealism deliberately! It’s a stylized view of California. It’s like the burger in the commercial doesn’t look like the real one - it looks a bit unrealistically perfect because that’s actually more attractive than the real thing. Realistic and photorealistic are tangentially related.
The comparison here should perhaps be between frames of the game that already attempted to look like the cityscapes dataset.
GTA obviously doesn’t have a ground texture that isn’t smooth because it wouldn’t be possible, or “too warm” palette because it’s too computationally expensive to have the bleak cityscapes tone mapping.
A GTA frame rendered at its current frame cost, with textures the same size (but different) and a customized tone mapping, would look more photorealistic and make the difference to the cityscapes dataset much much smaller.
The authors could likely have done this comparison since they probably have the ability to modify textures and shaders to set tonemapping. That would have separated out the differences that are only artistic from those that are lacking photorealism because of computational expense.
I am not very interested in whether or not the dataset that these authors chose is a pleasing dataset to use as a basis for a video game aesthetic. I am interested in in the technical accomplishments that this model was able to achieve in regards to preserving image features of semantic importance (e.g. writing, symbols, logos), not "hallucinating" new image features, and maintaining frame-to-frame stability in the output video product. These are very notable accomplishments for an adversarial model in this space, and the techniques they used to accomplish them are interesting. That Rockstar could have made their game more Cityscapes-like without this tech is missing the forest for the trees.
Honestly my main take here is just how much better the foliage can look. Grass/trees in real time graphics just aren't even close.
[deleted]
I completely agree with you. Also those people who complain about the "dull european" look, did you scroll down and look at the images altered using the Mapillary Vistas set?
The cool thing is that this gives game-designers an extremely powerful additional tool. You could for example easily go for a more retro-look by training with kodakcolor images ...
And I don't buy the argument that GTA doesn't look more realistic because that's what the designers were going for. If this was a game with a clear "comic appearance" that might be true, however I would argue it looks like it does, because the designers did the maximum that they could achieve with their tools and within a certain budget.
The cool thing is that this gives game-designers an extremely powerful additional tool. You could for example easily go for a more retro-look by training with kodakcolor images ...
And I don't buy the argument that GTA doesn't look more realistic because that's what the designers were going for. If this was a game with a clear "comic appearance" that might be true, however I would argue it looks like it does, because the designers did the maximum that they could achieve with their tools and within a certain budget.
I almost clicked away from the the article because those Cityscapes transformations looked worse in my opinion - just washed out and dull, so I can understand why people didn't like the look. The Mapillary Vistas processing definitely was nicer, although it looked uneven to me - the road still looked washed out, which created an almost disturbing level of contrast with the greenery.
What they've done is interesting, and I think with further refinement it is certainly a technique to consider.
What they've done is interesting, and I think with further refinement it is certainly a technique to consider.
I think expectations influenced by culture play a role here. Since I live in a Brazilian metropolis (4mi population), the Cityscapes transformations looked more real. That is probably because they look more like what I'm used to see when I walk the streets. People from rich neighborhoods in the US will likely find the Mapillary Vistas more realistic since that is what they used to see.
Impressive result nevertheless. Improving this with adversarial networks will probably generate images which are indistinguishable from reality in just a few years.
Impressive result nevertheless. Improving this with adversarial networks will probably generate images which are indistinguishable from reality in just a few years.
The color grade in GTA is of course an artistic choice and rendering the entire game with foggy overcast light and a strong green tinge (which btw. is not how German cities look, we aren't living in a bad color transfer of The Matrix) makes it look bad at the macro level. The details might be enhanced by this method, but honestly it's kind of hard to tell from a heavily compressed 720p Youtube video. What looks okay or even better in a video doesn't have to look better interactively, uncompressed, in native resolution.
Hot take: The overall tone of the style produced using this dataset comes from capture artifacts.
Hot take: The overall tone of the style produced using this dataset comes from capture artifacts.
the only thing i could get out of the demo videos is that their fps is shit.
fps > beautiful images until you get at least 60... which this wasn't.
im pretty bearish on this kind of innovation in games tbh, its never going to be low latency enough to be viable, even if they'd improve their framerate - i dont see it going beyond academic interest because of that
definitely interesting to read about though, even if i doubt its viability in gaming
fps > beautiful images until you get at least 60... which this wasn't.
im pretty bearish on this kind of innovation in games tbh, its never going to be low latency enough to be viable, even if they'd improve their framerate - i dont see it going beyond academic interest because of that
definitely interesting to read about though, even if i doubt its viability in gaming
>its never going to be low latency enough to be viable
Why is that? Just because it's expensive in terms of computation? I can't see why latency wouldn't improve with framerate.
It couldn't because of the nature of this technology.
Their post processing can only start after the full image has been rendered, so whatever photorealism filter they apply will necessarily delay the image until the process is finished.
Their post processing can only start after the full image has been rendered, so whatever photorealism filter they apply will necessarily delay the image until the process is finished.
This is not a fundamental obstacle for the technology. Lots of stuff is happening in post-processing already.
The Cityscapes transformations have a photorealism advantage because they're blurrier and more washed out. It's easier to match less detailed photos.
The blurriness, green tinge, and washed out colours are qualities photos sometimes have, but it's not what we see with our eyes. But they do help hide details that would give the image away as a computer rendering.
The blurriness, green tinge, and washed out colours are qualities photos sometimes have, but it's not what we see with our eyes. But they do help hide details that would give the image away as a computer rendering.
Definitely; I mean if they had a dataset of the cities and locations GTAV was based on I think it might look uncanny.
> You don't want your games to look realistic? Go play pacman or outrun.
That's a silly dichotomy. We don't even want films to look completely realistic, that's why CGI exists in the first place. Imagine that you ran this over something like an Avengers movie, or even better, Blade Runner 2049. The AI would remove all the stylized/unrealistic details, and someone here would claim that this is clearly an improvement, and any criticism must come from luddites!
That's a silly dichotomy. We don't even want films to look completely realistic, that's why CGI exists in the first place. Imagine that you ran this over something like an Avengers movie, or even better, Blade Runner 2049. The AI would remove all the stylized/unrealistic details, and someone here would claim that this is clearly an improvement, and any criticism must come from luddites!
The primary reason why CGI is used is to save money. The secondary is to ease production of movies, which at the end saves money and time (which is money).
Styling as done in some of them is an optional bonus if creators strive for some additional artistic message, but otherwise its all about money, money, money.
You even contradict yourself - they tried to look Avengers as realistically as possible within the realm of comic book fairy tale, that's why they changed character of Thanos significantly well after introducing him (arguably a good choice). From cartoonish styling to realism. Nobody would take the epic battles so seriously if they all looked like Ready Player One.
Styling as done in some of them is an optional bonus if creators strive for some additional artistic message, but otherwise its all about money, money, money.
You even contradict yourself - they tried to look Avengers as realistically as possible within the realm of comic book fairy tale, that's why they changed character of Thanos significantly well after introducing him (arguably a good choice). From cartoonish styling to realism. Nobody would take the epic battles so seriously if they all looked like Ready Player One.
If I want to film a dragon, I'm probably going to choose CGI because it gets me the best dragon. It is coincidentally cheaper than building a puppet, or genetically engineering a lizard to look like a dragon, but cost isn't the motivating factor here.
What? That's not at all why CGI exists. It's to save money and increase flexibility, and sometimes to do shots that would otherwise be impossible.
Modern CGI in films is already basically indistinguishability from reality. You only notice it these days when it has to be CGI (e.g. space ships) or is badly done (which leads to the impression that CGI is still bad like it was in the 90s).
Modern CGI in films is already basically indistinguishability from reality. You only notice it these days when it has to be CGI (e.g. space ships) or is badly done (which leads to the impression that CGI is still bad like it was in the 90s).
The only director who laboriously follows through on 'I want it to not look like CGI' is Christopher Nolan. Others seem to favor stylized beauty over absolute CGI realism.
Uh, no, you just don't notice the absolute gobs of CGI used in tons of films because it is meant to look hyperrealistic.
There are several, several youtube videos showcasing the incredible work of studios like Industrial Light and Magic creating most of the backdrops you thought were shot on location.
There are several, several youtube videos showcasing the incredible work of studios like Industrial Light and Magic creating most of the backdrops you thought were shot on location.
This is an enormous leap. Texturing and lighting are two very hard problems in CGI. Realistic lighting using raytracing requires a lot of radiosity passes, which means adjusting lighting of each surface to account for each ray of light bouncing from surface to surface to surface to surface like a 3D billiard ball, more bounces computed yields better results but there’s computational limits to the number of reflections per ray, and many parameters and formulas to choose from each with its own trade offs. Texturing and shader pipelines is another matrix of trade offs to get a good-enough result. Instead of all of that complexity, applying ML this way cleverly hops over the most difficult parts to a really impressive result.
Some don’t want games to look this way. Nonetheless this technique can be applied to give games and movies a different look that would very hard to acheive by standard rendering pipelines, and it doesn’t have to look washed out like this example, that’s all based on the training data set and how much this ML lighting and texturing is blended into the scene, and that’s another creative choice. I can imagine filmmakers wanting to adjust CG lighting and texturing of a scene using a training data set from old TV shows to make a convincing reboot of a classic TV series, or new series done in that style.
Some don’t want games to look this way. Nonetheless this technique can be applied to give games and movies a different look that would very hard to acheive by standard rendering pipelines, and it doesn’t have to look washed out like this example, that’s all based on the training data set and how much this ML lighting and texturing is blended into the scene, and that’s another creative choice. I can imagine filmmakers wanting to adjust CG lighting and texturing of a scene using a training data set from old TV shows to make a convincing reboot of a classic TV series, or new series done in that style.
Photorealism shouldn't be the end goal. Watch a B/W television for a week and you won't be missing the colors. Photorealism is much like sugar in that it is addictive, but you won't miss the sugar if you didn't eat it for a while.
It's not the end goal, but it's definitely an aesthetic direction that a lot of games want to go towards. And not just video games, but films too. Most films already have photorealistic CGI, sometimes applying it without you even noticing it (e.g. backgrounds).
Better, play IF games/roguelikes such as Slashem/DCSS, Gearhead, Anchorhead, Vicyous Cycles, Slouch over Bedlam and All Thing Devours.
Then, every AAA game looks dumb, non-interactive and limited even with HQ graphics.
Then, every AAA game looks dumb, non-interactive and limited even with HQ graphics.
And yet, the vast majority of DCSS players choose tiles over ASCII. Liking games without nice visuals doesn't mean people don't like nice visuals.
Could they do this process at video game speeds though? (25-60 times a second)
I think this technique has a lot of potential, but will probably need a lot of work before it can be applied to video games. It might first show up as a filter for 'photo mode' in some games though, that would be interesting.
I think this technique has a lot of potential, but will probably need a lot of work before it can be applied to video games. It might first show up as a filter for 'photo mode' in some games though, that would be interesting.
The first video on that page says they can run it at interactive framerates.
>These images are within a hair's breadth of being indistinguishable from reality.
This. I thought those were photos they took somewhere that looks exactly the same as GTA V, but then when I watch the video explaining it I was like holy crap. No Ray Tracing or whatever tech demo was ever this close to photo realistic.
>You have no idea how much cheaper this technique is.
Could someone explain this in greater details? I suppose you will still need to create some decent base image or graphics for the enhancement to work to its maximum?
This. I thought those were photos they took somewhere that looks exactly the same as GTA V, but then when I watch the video explaining it I was like holy crap. No Ray Tracing or whatever tech demo was ever this close to photo realistic.
>You have no idea how much cheaper this technique is.
Could someone explain this in greater details? I suppose you will still need to create some decent base image or graphics for the enhancement to work to its maximum?
I think the source game could be lower polygon count and texture resolution, but the neural network might be quite heavy...
While I agree that the quality of this method is impressive, there are two caveats:
1. We do not know the set of pictures that the authors did not show us. So maybe this technique works really well only for a small fraction of cases.
2. Games (especially games that involve moving cars) tend to simplify mechanics so much that the moving end result will never look realistic, regardless how perfect the still picture is.
So yes, I expect great outcomes from such a method, but I don't think its going to be a revolutionary silver bullet.
1. We do not know the set of pictures that the authors did not show us. So maybe this technique works really well only for a small fraction of cases.
2. Games (especially games that involve moving cars) tend to simplify mechanics so much that the moving end result will never look realistic, regardless how perfect the still picture is.
So yes, I expect great outcomes from such a method, but I don't think its going to be a revolutionary silver bullet.
I thought the video was from real life at first until I listened to the explanation. It's amazing that this is possible in real time.
Whenever there is a sanctimonious comment like this at the top complaining about all the negative posts, I usually agree with the other posts. As in this case.
I actually don’t think this looks any less “like a video game”, but its color is more washed-out, and darker. It seems like a net loss to me, sorry. Not for fancy theoretical reasons, I just think the GTA looks better in all the examples.
I actually don’t think this looks any less “like a video game”, but its color is more washed-out, and darker. It seems like a net loss to me, sorry. Not for fancy theoretical reasons, I just think the GTA looks better in all the examples.
I think photo-realism in games is not merely a black and white, positive or negative. There are trade-offs, and I think it has to do with perception around aesthetics.
(also, this is simply one guy's opinion, but I am curious if anyone else feels the same)
For instance, in my view, I tend to prefer a stylized less hyper-realism in games. I think this is because the enhanced color, light, and brilliance is more aesthetically pleasant to look at, even though it is certainly more uncommon in the real world.
I don't believe this is because saturated color, and brilliant light doesn't exist in reality, but these occurrences happen more rarely and in specific, often planned, situations and settings. And some people (trained or gifted photographers, for instance), have learned how to capture this look, or to take something rather ordinary, and capture it to appear extraordinary, much like the difference between your average person snapping a photo with a disposable camera, and Ansel Adams. The former will produce a photo that is far more common in appearance, and thus...boring, or uninteresting. So when it comes to games, if you simulate the "average look of reality" the game certainly appears more photo-realistic, but may also be less aesthetically interesting (to me, at least).
This says absolutely nothing about the technology developed here, which is incredibly impressive. Nor do I think there is a right or wrong way create game visuals. I think that misses the point.
However, I feel I'm not the only person who can be blown away with how realistic something can be made to look, yet ...also find it more aesthetically drab and boring to look at. And in the case of games, as I mentioned above, I tend to prefer to see things that are rare, interesting, and extra-ordinary, and thus less "realistic".
Anyone else feel the same?
(also, this is simply one guy's opinion, but I am curious if anyone else feels the same)
For instance, in my view, I tend to prefer a stylized less hyper-realism in games. I think this is because the enhanced color, light, and brilliance is more aesthetically pleasant to look at, even though it is certainly more uncommon in the real world.
I don't believe this is because saturated color, and brilliant light doesn't exist in reality, but these occurrences happen more rarely and in specific, often planned, situations and settings. And some people (trained or gifted photographers, for instance), have learned how to capture this look, or to take something rather ordinary, and capture it to appear extraordinary, much like the difference between your average person snapping a photo with a disposable camera, and Ansel Adams. The former will produce a photo that is far more common in appearance, and thus...boring, or uninteresting. So when it comes to games, if you simulate the "average look of reality" the game certainly appears more photo-realistic, but may also be less aesthetically interesting (to me, at least).
This says absolutely nothing about the technology developed here, which is incredibly impressive. Nor do I think there is a right or wrong way create game visuals. I think that misses the point.
However, I feel I'm not the only person who can be blown away with how realistic something can be made to look, yet ...also find it more aesthetically drab and boring to look at. And in the case of games, as I mentioned above, I tend to prefer to see things that are rare, interesting, and extra-ordinary, and thus less "realistic".
Anyone else feel the same?
> You don't want your games to look realistic? Go play pacman or outrun.
I can't just play GTA 5 as is?
I can't just play GTA 5 as is?
Yeah the foliage is remarkably life-like and that is so hard and labor (art) intensive to do well. This is an amazing tool for indies without tons of resources.
> You don't want your games to look realistic? Go play pacman or outrun.
I want my games to look cool. I dont wanna them to look like streetview or geoguessr.
The result is impressive, but I'm not sure it would be good for video games. Maybe with some added filters it would be a lot better.
I want my games to look cool. I dont wanna them to look like streetview or geoguessr.
The result is impressive, but I'm not sure it would be good for video games. Maybe with some added filters it would be a lot better.
> These images are within a hair's breadth of being indistinguishable from reality
Without a comparison to the original pictures, they do look very good, the roads in particular look more realistic, as though taken from a dashcam or medium quality phone camera. However the foliage just looks blurred.
A lot of (interactive) CGI already looks more realistic than this. Take for example, this Australian scene made in Unreal Engine 4 - https://www.youtube.com/watch?v=Pg75bfkegtU
And that's not just pre-rendered, here's someone exploring the same scene interactively - https://www.youtube.com/watch?v=y87QL7IPE34
Without a comparison to the original pictures, they do look very good, the roads in particular look more realistic, as though taken from a dashcam or medium quality phone camera. However the foliage just looks blurred.
A lot of (interactive) CGI already looks more realistic than this. Take for example, this Australian scene made in Unreal Engine 4 - https://www.youtube.com/watch?v=Pg75bfkegtU
And that's not just pre-rendered, here's someone exploring the same scene interactively - https://www.youtube.com/watch?v=y87QL7IPE34
This is because its from scanned assets, I.e. there's no artist trying to imitate reality. The neural network above can take your artist created graphics and make them appear like they were scanned in. Both techniques achieve the same thing.
I think you’re absolutely right. The innovations in deep learning make it such an exciting time to be an indie game developer. What used to take armies of content creators can now be a tiny team. Unity should be all over this kind of tech.
Not only am I extremely impressed with the current realism they were able to accomplish, I suspect that others will figure out what it is about the final result and that will feed back in to even further realism in games (just like the breakthrough of subsurface scattering in digital skin).
As well, I’m very excited for the scrappy devs who will turn this technique in to something wildly unexpected.
As well, I’m very excited for the scrappy devs who will turn this technique in to something wildly unexpected.
I am especially impressed by how much more realistic the lighting ends up looking despite the network probably having less information to work with than the engine! Though it ends up turning the lighting very flat. I guess that is an effect of the source images they were using.
As others have said, using this directly in the engine (tuned to work with the intended art style etc) could probably produce almost miraculous results, if it can be made to work at a reasonable frame rate.
That would also allow the developers to use high-quality rendered images instead of these green-tinted, low-contrast "automotive grade" camera images as source of truth (there are good reasons these images look like that ... but they don't look pleasant).
As others have said, using this directly in the engine (tuned to work with the intended art style etc) could probably produce almost miraculous results, if it can be made to work at a reasonable frame rate.
That would also allow the developers to use high-quality rendered images instead of these green-tinted, low-contrast "automotive grade" camera images as source of truth (there are good reasons these images look like that ... but they don't look pleasant).
[deleted]
Impressive, but I think that photorealism is not necessarily what you would want to strive for in games, as it is not true realism. It emulates the deficiencies of the camera that we perceive it as realistic, because we are trained on thousands of photo images and video footage. I prefer visionrealism instead of photorealism in my games and don't really find that lens effects like flares, chromatic aberration, and depth of field enhances the experience of realism. It feels like I'm looking through a camera rather than being there myself. For me it breaks immersion instead of enhancing it.
I found their approach to be more like the difference between a digitally filmed movie (that can look too crisp) and one shot on cellulose that has a kind of softness to it.
Where it noticeably fell short was displaying red lights (tail lights, traffic lights, etc) that lost their brightness.
Where it noticeably fell short was displaying red lights (tail lights, traffic lights, etc) that lost their brightness.
yeah, games visuals are more about art direction than search of realism
this would work great in a flight sim tho, albeit blur and out of focus effects are terrible for visual clarity
however, there's one small bit that has not been given enough attention imho, which is on the latter set, under "removes distant haze": this would be a great technique to complement flat LOD reduction of open world games while keeping up the crispness and contrast of distant items.
this would work great in a flight sim tho, albeit blur and out of focus effects are terrible for visual clarity
however, there's one small bit that has not been given enough attention imho, which is on the latter set, under "removes distant haze": this would be a great technique to complement flat LOD reduction of open world games while keeping up the crispness and contrast of distant items.
That's mostly a problem with the input data.
A gamestudio of the size of rockstar could afford to acquire their own custom fit dataset.
Get some good cameras and lenses and capture footage of wherever they want to set their next game. Then they can apply some motion picture color grading to that and use that as the input data for the NN.
Get some good cameras and lenses and capture footage of wherever they want to set their next game. Then they can apply some motion picture color grading to that and use that as the input data for the NN.
I think photorealism has the advantage of maintaining a cinematic feel. You can do a lot of 'realistic' things withs light in a game, that aren't really realistic (you'd never see them in real life), but are the type of thing you'd see in a photo or movie. This allows for a more tightly controlled experience.
Of course, this all depends on what the intention behind the game is. If you do want maximum immmersion, visionrealism would be the way to go.
Of course, this all depends on what the intention behind the game is. If you do want maximum immmersion, visionrealism would be the way to go.
I don't think you could be any more wrong.
Enhancements like this are going to be the thing that brings VR fully into the mainstream. It will be industry-changing.
This tech is in its infancy. I promise you, the final result is going to be indistinguishable from human vison.
Enhancements like this are going to be the thing that brings VR fully into the mainstream. It will be industry-changing.
This tech is in its infancy. I promise you, the final result is going to be indistinguishable from human vison.
Looking like a photo is quite distinguishable from human vision.
This could extend beyond games easily. I can see adoption from animation studios for eg.
At first I was impressed, but then I realize they also work on (raw) G-buffers, depth maps, normals, etc. grabbed directly from the engine. I mean if we're already doing all of that, why not just pass it through 3DSMax and get an actual photorealistic rendering? Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed.
There might also be a misconception that GTAV couldn't make their game more murky/realistic-looking, but more often than not, saturation and contrast is purposefully cranked up in games (particularly driving ones). I will say that the lush background mountains look cool though.
There might also be a misconception that GTAV couldn't make their game more murky/realistic-looking, but more often than not, saturation and contrast is purposefully cranked up in games (particularly driving ones). I will say that the lush background mountains look cool though.
Realistic renders with raytracing assumes accurate models. In the scene, the grass can be a rectangular block with a grass texture applied to it for instance, NN will transform it into an actual grass patch (see example images) similar to how it would look like in real life (not constrained by your 3D geometry / scene setup). So realistic lighting is just one part of the problem space here. A road with an asphalt texture applied on it will need a lot of manual tweaks / material design to look realistic under raytracing regime, NN can look at it and implicitly say "hey, this is a road, and I know what a road looks like under this lighting, so I'm going to render it like this" and it automatically becomes realistic.
Almost sounds like one could grossly placeholder-mark-paint surfaces/areas by type and the NN stuff will paint over it proper content and consistent details by dreaming them from these semantic hints.
You can probably have the game output custom-made data to serve as an input for the subsequent NN and skip a whole bunch of lighting calculations.
> [...] Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed.
Define 'photorealism'. You limit yourself to the rendering aspect and ignore the content/asset problem and thereby economic factors.
This technique is probably one or two orders of magnitude cheaper than generating geometry and shaders that have the required level of detail in any traditional way.
Either by an artist or programmatic/procedural (Someone has to write that code too/set up that node graph in Houdini or the like). Yes, you can also just 3D scan stuff (see Quixel, etc.) but that has limits too.
More specifically for achieving photorealism by 'traditional means': consider scales.
An asset, however produced in finite time, will only hold up to some range of scales. The technique in the paper allows to get as close or far away as you want and everything will still look real. No level of detail work needed, just a few more training data.
Traditionally, if you use textures, you will reach the limit of the texture resolution. Even if you use procedural techniques – not everything is a fractal. A close up of a brick looks very different from a wall of bricks etc. Again, who will write that code?
My guess is rather that we will see a hybrid of more photorealism by traditional means and more 'icing on the cake' by methods like this.
Define 'photorealism'. You limit yourself to the rendering aspect and ignore the content/asset problem and thereby economic factors.
This technique is probably one or two orders of magnitude cheaper than generating geometry and shaders that have the required level of detail in any traditional way.
Either by an artist or programmatic/procedural (Someone has to write that code too/set up that node graph in Houdini or the like). Yes, you can also just 3D scan stuff (see Quixel, etc.) but that has limits too.
More specifically for achieving photorealism by 'traditional means': consider scales.
An asset, however produced in finite time, will only hold up to some range of scales. The technique in the paper allows to get as close or far away as you want and everything will still look real. No level of detail work needed, just a few more training data.
Traditionally, if you use textures, you will reach the limit of the texture resolution. Even if you use procedural techniques – not everything is a fractal. A close up of a brick looks very different from a wall of bricks etc. Again, who will write that code?
My guess is rather that we will see a hybrid of more photorealism by traditional means and more 'icing on the cake' by methods like this.
I think it's worth exploring multiple approaches to a problem. Maybe this neural network approach can be improved upon and it will also generate photorealistic looking images but have other advantages such as less effort on behalf of the game developer or less compute.
The Cityscapes dataset is a bit murky looking; try scrolling down to the Mapillary Vista dataset instead. Much better colors. They should have led with that one instead.
Not disagreeing, but the Cityscapes samples looked 100% photorealistic to me, whereas the Mapillary samples just looked different.
Disclaimer: in the UK, so grey and murky == realistic.
Disclaimer: in the UK, so grey and murky == realistic.
>Disclaimer: in the UK, so grey and murky == realistic.
Ditto in Northern Spain.
The sunny dataset is more akin to Mediterranean landscapes. To me the shinny/sunny/reflective stuff it looked unreal.
Ditto in Northern Spain.
The sunny dataset is more akin to Mediterranean landscapes. To me the shinny/sunny/reflective stuff it looked unreal.
The MV examples generally look better, but seem to have one characteristic weakness: street signs. Many of them seem unrealistically saturated and glow-y with blue fringes around letters/symbols. Kind of odd, considering that everything else looks so good.
Good ray tracing is expensive and still requires a lot of asset preparation. Difference between what RTX can do realtime right now and production quality final renders is massive. I think there will ultimately be quality wall for realtime tracing that target ms/frame versus mins/frame for HQ renders. After which hacks like this must be used to further close distance by adding layers of additional details that really sell photorealism. TBH I doubt RTX could reproduce even this quality for many years, if ever.
It's not just about murkiness and lighting quality, this is akin to adding the last 10% of polish that usually takes 90% of the time. It's also the difference between amateur vs much greater proficiency level. Having access to the assets of scene in 3D max, someone with a basic training can replicate base GTA scene, the enhancements shown here takes years more experience. The amount accessibility and saved labour is tremendous. It's why even VHS video is still more photo realistic than 4K RTX. Real life has a lot of minutiae and interacting details beyond resolution and good lighting models.
It's not just about murkiness and lighting quality, this is akin to adding the last 10% of polish that usually takes 90% of the time. It's also the difference between amateur vs much greater proficiency level. Having access to the assets of scene in 3D max, someone with a basic training can replicate base GTA scene, the enhancements shown here takes years more experience. The amount accessibility and saved labour is tremendous. It's why even VHS video is still more photo realistic than 4K RTX. Real life has a lot of minutiae and interacting details beyond resolution and good lighting models.
I watched their video, and I tend to agree. With the amount of computing they use, ray-tracing is probably going to be just as good and probably higher frame-rate. But maybe this technique, in due time, will result in an alternative class of rendering pipelines with less focus in shading and materials.
You no longer need 3dsmax for raytracing. But what could be really interesting is to train your ml model against path traced images and apply the outputs to selective areas of your screem where the g buffer provives insufficient information density (i. i.e.foliage)
I don't think the goal of this is to make games look better.
Especially the GTAV dataset is often used as a synthetic dataset for research. The appeal is that you can extract normal/material and other semantic information from the game engine for a driving scene.
Basically avoid expensive segmentation labels on real data by working the other way around. Things like this synthetic to real translation can then be used for other downstream tasks (semantic segmentation, distance/normal prediction).
Especially the GTAV dataset is often used as a synthetic dataset for research. The appeal is that you can extract normal/material and other semantic information from the game engine for a driving scene.
Basically avoid expensive segmentation labels on real data by working the other way around. Things like this synthetic to real translation can then be used for other downstream tasks (semantic segmentation, distance/normal prediction).
I share your disappointment. There was recently a thread on HN on how good metadata always beats machine learning.
I wonder how much complexity could be saved by letting the application properly annotate objects: "This is a tree located at coordinates X,Y,Z in the scene. Please NN take over and project a tree appropriate for the light conditions and geographical location."
I wonder how much complexity could be saved by letting the application properly annotate objects: "This is a tree located at coordinates X,Y,Z in the scene. Please NN take over and project a tree appropriate for the light conditions and geographical location."
Good point. But I wouldn't rule out neural networks. They could still be used to derive efficient shader models that can be run in real time. https://arxiv.org/abs/1603.06078
For an example of photorealistic rendering approach coupled to a scenario generator, see https://synscapes.on.liu.se/index.html
So, can we gradually get to a point where the engine itself only renders simplified abstractions (i.e. not the current state-of-the-art graphics) and feeds them to a neural network that produces the output?
This is what has been called Deep Shading: https://arxiv.org/abs/1603.06078
[deleted]
Amazing how this is all done in real-time, and incredible stability of image.
Further, I’d love to also see a tech demo using a Californian dataset as the input: To my originally European eyes, it’s simply making GTA look more like video shot in Germany vs more realistic (there’s a difference!), though I get that any data set can be fed into this :)
Further, I’d love to also see a tech demo using a Californian dataset as the input: To my originally European eyes, it’s simply making GTA look more like video shot in Germany vs more realistic (there’s a difference!), though I get that any data set can be fed into this :)
There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( https://youtu.be/P1IcaBn3ej0?t=462 )
To me that version is much more reflective of the actual potential of this method than "GTA Düsseldorf dash cam". When watching the images produced with that one I get the sense that the sense of realism is enhanced by the dashcam-style image degradations and could be largely emulated in-engine (green LUT, blur + glow filters to emulate lens issues).
Is it real-time rather than post processed?
From the "Enhancing Photorealism Enhancement" video [1]: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.".
[1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
[1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
From the paper:
"Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU."
That means 1 or 2 frames per second, so interactive but pretty much unplayable, which is to be expected as it's just research, but considering that they have DLSS working in realtime with tensor cores perhaps something like this will also be achievable very soon.
Edit: Out of curiosity I looked up how long DLSS takes to process and it's less than 2ms, so you'd probably need to speed this up by two orders of magnitude to run it a game.
"Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU."
That means 1 or 2 frames per second, so interactive but pretty much unplayable, which is to be expected as it's just research, but considering that they have DLSS working in realtime with tensor cores perhaps something like this will also be achievable very soon.
Edit: Out of curiosity I looked up how long DLSS takes to process and it's less than 2ms, so you'd probably need to speed this up by two orders of magnitude to run it a game.
Great point I don’t know
They mention temporal stability across the footage, but what about the entire game? It would be unfortunate to have the same car to turn from Audi to BMW under entirely different lighting and angles, or a palm tree into a pine. (the difference might be much more subtle but annoying nonetheless)
Part of the contribution of this paper is about preventing that specific problem.
“We also seek to eliminate artifacts that can be seen in the results of prior deep-learning approaches, which often hallucinate objects. To this end, we analyze the datasets that are commonly used for photorealism enhancement. Our analysis reveals that their scene layouts differ in ways that can explain artifacts commonly seen in prior work. To better align the datasets and alleviate the artifacts, we propose a new strategy for sampling image patches during training. We further design a new adversarial training objective that facilitates enhancements that are geometrically and semantically consistent with the content of the input image.”
“We also seek to eliminate artifacts that can be seen in the results of prior deep-learning approaches, which often hallucinate objects. To this end, we analyze the datasets that are commonly used for photorealism enhancement. Our analysis reveals that their scene layouts differ in ways that can explain artifacts commonly seen in prior work. To better align the datasets and alleviate the artifacts, we propose a new strategy for sampling image patches during training. We further design a new adversarial training objective that facilitates enhancements that are geometrically and semantically consistent with the content of the input image.”
Why not let it output the gbuffer images instead, or perhaps even let it operate on the artwork instead? That way you could bake the changes into the game without having to run the NN in real time.
From what I can tell, the whole reason the road "looks like a video game" is that it's using a source texture and then stretching it along the road's geometry, "diluting" the level of detail as seen from the camera viewpoint.
The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are.
Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road.
If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.
The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are.
Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road.
If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.
The part about ridiculous resolutions for texture detail like leaves isn't necessarily true. There are very old techniques for scattering small decals densely across surfaces. The basic trick is to use coarse textures that encode the positioning of decals within each texel footprint.
Sure, but no matter what the geometry/bump maps/etc. are on the road, they're not going to render out as anything but an aliased, badly-lit mess unless you're heavily supersampling them (and then doing ray-tracing to the supersampled image.) We're talking about "objects" where the crucial detail-features of their meshes for getting correct color math out (e.g. the normal+reflectance maps) would be, on average, scaled to be smaller than a pixel from the viewing distance+angle, with potentially many objects contributing to the final color of each pixel.
So, rather than rendering the road at 8x-16x and then downsampling it, in order to get a pile of leaves to appear to have ~1.5x LoD at the projection plane, you can just achieve that 1.5x LoD pixel by pixel at the projection plane.
Or, to put that another way: when a digital artist is painting a landscape, they don't have to zoom in by 8x, paint every individual blade of grass, and then zoom out. They can just use an artistic technique at 1x zoom that approximates the texture that would have been created by downsampling "photographic" detail painted in at 8x to your screen resolution.
Here, the ML model is playing the role of the digital artist, "painting" on the projection plane.
So, rather than rendering the road at 8x-16x and then downsampling it, in order to get a pile of leaves to appear to have ~1.5x LoD at the projection plane, you can just achieve that 1.5x LoD pixel by pixel at the projection plane.
Or, to put that another way: when a digital artist is painting a landscape, they don't have to zoom in by 8x, paint every individual blade of grass, and then zoom out. They can just use an artistic technique at 1x zoom that approximates the texture that would have been created by downsampling "photographic" detail painted in at 8x to your screen resolution.
Here, the ML model is playing the role of the digital artist, "painting" on the projection plane.
The problem is not texture antialiasing (which is essentially solved). The problem is shape antialiasing, whether many distant small shapes are so close together that become a sort of translucent buzz, which is very computationally heavy to compute.
REYES renderers had a pretty nifty solution for shape antialiasing with their slicing and dicing of geometry into sub-pixel sized primitives. That was in the 80s, just for reference. Unfortunately, this method has a pretty complex implementation and just doesn't play all that well with hardware rasterization or path tracing and has since fallen out of favor.
I don't think so. I think it's because the texture is already photo-like, and then there is a normal map applied on top of that. It basically creates two layers of shading that don't fit together. Few games manage to avoid that problem.
Interesting hypothesis. I think this would predict that you’d get more realistic results if you were using “bare, unlit” textures (e.g. texture images + normal maps created by using 3D scanning or even computational tomography, as you would if you wanted to 3D-print a copy of the object) and then applying regular normal maps to them. Anyone have screenshots of 3D game objects that were explicitly mentioned as being authored this way?
This is a good question - it certainly is possible to run a NN on the artwork or pre-g-buffer, and you could imagine having something bake the changes.
To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild).
So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer.
Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why.
That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.
To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild).
So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer.
Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why.
That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.
Wow, that is pretty cool.
Maybe GTA 6 or 7 will include no textures and lower poly models, and instead spend a large budget driving around LA and just film stuff to train neural render.
Maybe GTA 6 or 7 will include no textures and lower poly models, and instead spend a large budget driving around LA and just film stuff to train neural render.
Really impressive and striking to see the stylistic effects of different training sets (Cityscapes vs Mapillary Vistas).
Obviously the GTA V designers applied a particular style and colour scheme which gets lost, but I'm sure something like StyleGAN could address that.
Obviously the GTA V designers applied a particular style and colour scheme which gets lost, but I'm sure something like StyleGAN could address that.
The results dont look that great to me. The original looks better for a game. The "enhanced" version is so unsaturated it could be bordering on black & white.
That's because of the dataset. Later in the video they show results using another dataset with more vibrant colour, that looks a lot better IMO.
I just watched the video to the end - definitely an improvement, but it still looks "faker" than the original to me. Maybe some uncanny valley effect is going on with me.
You’re just used to the style. Putting a photo of a road next to it will show you how far from reality GTA rendering is.
The model trained on TSIT gives nicer
results.
is the road-smoothing an artifact from the fact that the camera input sources lack the ability to resolve a detailed picture of the road at-speed?
Probably yes. Alternatively, it shows that the road textures used in GTA V were not very apt for their job. I do think the smoother ones look considerably more realistic.
Many of the GTA V pictures look more realistic to me. I think some of this is due to location; in the GTA V pics, the brownish hue, the haze and washed-out colors, the patchy grass, and the grainy roads seem to me to really capture the actual look of typical southern California urban sprawl. The enhancement makes things greener, less hazy, more grassy, and smooths roads, which is more typical for many areas of the United States, but not for southern California.
In addition, the enhancement blurs a bit, which imo is not good; the blur adds to photorealism by hiding mistakes, but on the other hand it makes it more obvious that you are looking at a picture because you can't see details.
In addition, the enhancement blurs a bit, which imo is not good; the blur adds to photorealism by hiding mistakes, but on the other hand it makes it more obvious that you are looking at a picture because you can't see details.
For me the most unrealistic part of the GTA renders was the trees, and the post processing significantly improved that. Blur is also an important part of real photos, it's unusual for everything to be in perfect focus.
I would argue that these kinds of renders are getting good enough that we should be more precise about what is meant by photorealism. Most pictures i take with my phone have alot of blur, don't capture colors well, and are frankly quite crappy. I feel like people expect photorealism to be equivalent to a really well-planned photo by a professional.
I would argue that these kinds of renders are getting good enough that we should be more precise about what is meant by photorealism. Most pictures i take with my phone have alot of blur, don't capture colors well, and are frankly quite crappy. I feel like people expect photorealism to be equivalent to a really well-planned photo by a professional.
This is pretty impressive, I wonder if you could build really cheaply textured maps and then upgrade them with this method as a post production touch up. Essentially removing most of the human work from texture libraries.
"return of GTA IV"
[deleted]
I love how the license plate in the first image gains a blue strip common in european license plates due to the dataset coming from germany.
Oh this is really impressive.
I've always wondered how games were going to solve the last mile of photorealism, and an additional color-grading filter seems like the perfect step. Really impressive work.
I think titling it "photorealistic enhancement" is a bit of a whiff, but "photorealistic real-time color grading / filtering" is bad-ass.
I've always wondered how games were going to solve the last mile of photorealism, and an additional color-grading filter seems like the perfect step. Really impressive work.
I think titling it "photorealistic enhancement" is a bit of a whiff, but "photorealistic real-time color grading / filtering" is bad-ass.
Uninformed take: this looks to me like an example of ‘style transfer’. If so, it doesn’t seem as compelling as other examples I’ve seen before, eg making a photo look like a Van Gogh using a single painting as input, which has been around for a few years. Can someone with better knowledge in this area explain if I’m missing something?
Style transfer commonly hallucinates objects or adds artifacts. Making something look like a painting is actually much easier than making it look realistic. In paintings, artifacts are often tolerated as 'artistic'. In a photo, it just pops out. Also, style transfer approaches are not temporally stable most of the time. Have a look at the comparisons with state-of-the-art image-to-image translation (CUT, TSIT) and photo style transfer (WCT2). In the case of GTA/Cityscapes shown here, most of these methods put trees in the sky and/or flicker. Also the use of G-buffers allows a much deeper and more robust translation than just using images as style-transfer approaches.
Thank you, this is really interesting.
[deleted]
Nice! So a next gen game could be made with a simple renderer, painting areas of the screen with a tag saying "grass" or "window", and a neural network trained on high quality renders in the desired style (not necessarily photorealistic). The same technique would probably work for character animation as well.
Looks really nice. It is however funny to see what it does to road signs (black arrows suddenly turn blue or the orange "don't walk" hand of pedestrian traffic lights seems to disappear). I wonder if this has to do that they use image data from Germany (having slightly different signs than the US).
It's also giving the license plates the blue stripe.
Very impressive. It will hard to balance realism and readability though.
GTA uses some surrealist contrasts because it makes the games easier to play, especially for long hours, or when you go fast.
But the tech is so new I'm not going to complain, it's already very good, especially given what it eats in resources.
GTA uses some surrealist contrasts because it makes the games easier to play, especially for long hours, or when you go fast.
But the tech is so new I'm not going to complain, it's already very good, especially given what it eats in resources.
It makes it clear there's way too much contrast in the road textures in the actual game.
How does this compare to Physical Based Rendering [0]?
[0]: https://en.m.wikipedia.org/wiki/Physically_based_rendering
[0]: https://en.m.wikipedia.org/wiki/Physically_based_rendering
FWIW, I think it doesn’t compare to PBR - these are two completely different things, and there’s no reason you can’t do both. Maybe as a simple example of that, it’s common today to run a NN based denoiser on top of a physically based path traced rendering.
This neural network operates on images - input is image and output is image. Note that it cannot synthesize an image from scratch, it can only take an image and make it look more like the lighting in it’s training set.
PBR is a technique for taking a 3d scene description and synthesizing an image, so for PBR you need models and lighting and materials, and even with physically based techniques it might still come out not completely realistic looking. There’s always room to emulate realistic subtleties of texture, lighting, camera lenses, video/film response, outdoor colors, atmosphere, etc.
This neural network operates on images - input is image and output is image. Note that it cannot synthesize an image from scratch, it can only take an image and make it look more like the lighting in it’s training set.
PBR is a technique for taking a 3d scene description and synthesizing an image, so for PBR you need models and lighting and materials, and even with physically based techniques it might still come out not completely realistic looking. There’s always room to emulate realistic subtleties of texture, lighting, camera lenses, video/film response, outdoor colors, atmosphere, etc.
For one, it will be a lot faster to compute. For two, it is a lot less laborious work to set up, which does not scale with the size of the world like PBR, but only with performance.
So where previously, you needed a lot of designers to make a big world look realistic with PBR, now you need to collect a big dataset with the style you want to have, and apply it to your game. I suspect that will scale differently.
So where previously, you needed a lot of designers to make a big world look realistic with PBR, now you need to collect a big dataset with the style you want to have, and apply it to your game. I suspect that will scale differently.
I noticed some strange (unrealistic) effects on the back of the red car at the 8:19 mark in this video:
https://youtu.be/P1IcaBn3ej0?t=499
https://youtu.be/P1IcaBn3ej0?t=499
You mean the shadow of the tree?
That's really cool!
But why does every car look like it's been freshly waxed? The newspaper boxes have a bit of rust and grime on them. Those look really real.
But why does every car look like it's been freshly waxed? The newspaper boxes have a bit of rust and grime on them. Those look really real.
I would love to see this on NBA2K21 ... seems like you have an ideal training set with recorded basketball games.
Kinda looks like Google StreetView.
Looks great in overcast settings but has some difficulties with sunny whether.
Pretty impressive.
Looks great in overcast settings but has some difficulties with sunny whether.
Pretty impressive.
Is there a cert issue? I can't access the page. A cache anywhere, or youtube link perhaps?
Photo-realistic burgers
http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food...
They made the game look bad with shitty fps.
If GTA developers wanted they could make the game look photo-realistic but that was never the intention.
http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food...
They made the game look bad with shitty fps.
If GTA developers wanted they could make the game look photo-realistic but that was never the intention.
Does any one here know about some existing self-driving(self-crashing) technology that uses two cameras in unison on the sides of the car like the eyes of a equide providing a nearly 360 angle image only readable by the computer due to 2D perspective constraints.
It seems to be completely removing the atmospheric haze, unfortunately.
One day there's going to be a gan where the input is a random number and the output is the source code of an entire triple A game.
I mean it just looks like they're simulating the windscreen glass. I guess it's more realistic but I somewhat prefer the original.
Is the windshield glass really the only change you see, when grass becomes green, reflections are added, the entire photo has a higher color temperature, etc?
Looking at the last comparison on the page, the grass isn't just becoming green, it's being "puffed" out from its original geometry, as if the grass geometry is being vertex-shaded with small spiky-ball-like objects. (A bit like the look of "static grass" in model making.)
Most of it is making light more intense, and most of everything else could be emulated with a blur filter and a green tint.
It's impressive for what it represents technically, but it's visually subtle.
It's impressive for what it represents technically, but it's visually subtle.
I went through and looked at each of the examples and gave my impression. That's why I say 'looks like...' as opposed to doing a critique of their method.
I think the Cityscapes dataset pictures were taken from the car's hood; the Mercedes emblem is up close. The camera position in game also seems to match this.
Impressively more google streetview / Russian dashboard camera like. Not a knock, quality improvement noticeable. Base GTA feels like intern 3D rendering work, after looks produced by someone fairly talented / competent. I can see this saving and incredible amount of labour.
> GTA feels like intern 3D rendering work
Lol, I guess you’re new? Or maybe you don’t know the difference between 3D rendering software and game engines?
GTA V came out in 2013 and the fact that it is still used as a reference for graphics quality tells you everything you need to know.
Lol, I guess you’re new? Or maybe you don’t know the difference between 3D rendering software and game engines?
GTA V came out in 2013 and the fact that it is still used as a reference for graphics quality tells you everything you need to know.
My old dayjob was photorealistic architectural renderings.
The point is GTAV, which was/is exemplar for game engine graphics, and even current gen RTX game engine rendering all look "bad" due to constraints and budgets of real time rendering. Bad in the sense that they're necessarily rudimentary - actual photo realism requires many layers of expensive details. This technique fills in many of those kind of details. The improvement in photo realism demonstrated here is significant and a huge leap in terms of image quality.
The point is GTAV, which was/is exemplar for game engine graphics, and even current gen RTX game engine rendering all look "bad" due to constraints and budgets of real time rendering. Bad in the sense that they're necessarily rudimentary - actual photo realism requires many layers of expensive details. This technique fills in many of those kind of details. The improvement in photo realism demonstrated here is significant and a huge leap in terms of image quality.
These images are within a hair's breadth of being indistinguishable from reality. It's an enormous leap. An enormous leap from the previous techniques and an enormous leap towards photorealistic realtime graphics.
You don't want your games to look realistic? Go play pacman or outrun.
You want to use path tracing instead? You have no idea how much cheaper this technique is. And on top of that the neural network is also fixing up the unrealistic texture work on all kinds of things in the scene.
The first games to use this are going to be from small game studios with less investment in the status quo and they are going to blow the competition out of the water.