Using GANs to Create Fantastical Creatures(ai.googleblog.com)
ai.googleblog.com
Using GANs to Create Fantastical Creatures
https://ai.googleblog.com/2020/11/using-gans-to-create-fantastical.html
80 comments
Actually, not bad for a monster,it looks a lot like the "Oogie Boogie" character from "The Nightmare Before Christmas" [1].
[1] https://i.imgur.com/md9P1lk.png
[1] https://i.imgur.com/md9P1lk.png
nailed it
I wonder if you enjoy The Bikini Bottom Horror fanfic
"Using GANs to create textures for already drawn fantastical creatures" would be a more accurate title. It's impressive, but not nearly as impressive as the title would suggest.
I feel like I just get the same texture stretched over whatever figures I draw.
"Using GANs to apply textures to color-coded hand-drawn figures". That doesn't catch headlines though.
That's actually a pretty awesome thing to have for concept art, although it seems like all of the textures seem very similar in style (and not to mention low-res)
Surely I can't be the only one that tried to paint a penis to see how it went.
This is the first time I'm happy that up and downs votes are ANONYMOUS.
Doesn't seem to be very well suited to that particular task. ;)
eg: https://ded.ninja/gan-monster1.png
eg: https://ded.ninja/gan-monster1.png
I think mine came out pretty great: https://imgur.com/kNlSAbX
Yep, yours definitely wins. :)
Truly a man of culture.
Reminds me of this:
Here are the color names most disproportionately popular among men:
* Penis
* Gay
* WTF
* Dunno
* Baige
https://blog.xkcd.com/2010/05/03/color-survey-results/
Here are the color names most disproportionately popular among men:
* Penis
* Gay
* WTF
* Dunno
* Baige
https://blog.xkcd.com/2010/05/03/color-survey-results/
Fascinating! I was suprised to see that 6% of users had CRTs and promptly noticed "2010" in the URL. It seems to me his style of writing has changed in a decade - can't point out exactly how though.
it was the first thing I did
I love that this conversation can still happen on HN without triggering some type of culture police
Does anyone know of work towards automatic 3D model generation and, ideally, rigging for animation? It seems like the future, but it also seems like it’s not getting the attention it should.
There are a lot of moving parts. Sculpting, UV Mapping, Rigging, Skinning, Animation, LODs, Baking, Texturing, engine import, lighting, materials.
A lot of things can be done procedurally, but going from 2d concept to actual working 3D Character in video game is a long jump.
On top of that every game will have very different asset and gameplay requirements.
A lot of things can be done procedurally, but going from 2d concept to actual working 3D Character in video game is a long jump.
On top of that every game will have very different asset and gameplay requirements.
I think it might take the form of artist augmentation rather than replacement. Think of it like a new sort of power tool. And it's true there are a lot of steps for triple-A engines, but most gamedevs (like myself, once) simply don't have any art assets, and need them greatly; a tool similar to MSPaint for models might enable the next Notch to emerge more easily.
On the other side of it, from the triple-A point of view, automating even one of the steps you mention will be a huge time saver. Artists tend to worry about machines replacing them, but I think it will only unlock more, better art. You'll be able to do so much. So there is a big financial incentive to get this right; the first one to capture the market will be in a strong position.
On the other side of it, from the triple-A point of view, automating even one of the steps you mention will be a huge time saver. Artists tend to worry about machines replacing them, but I think it will only unlock more, better art. You'll be able to do so much. So there is a big financial incentive to get this right; the first one to capture the market will be in a strong position.
> [...] most gamedevs (like myself, once) simply don't have any art assets, and need them greatly;
But isn't that exactly why market places like the Unity Asset Store, TurboSquid, and UE Marketplace exist?
The asset stores for Unity and UE in particular offer assets that are ready for use with these frameworks and can also be easily modified if need be.
I rather think the problem is the expectation to get high quality assets for free, which just isn't a realistic assumption, unless you're willing to team up with an artist who is willing to volunteer their time.
It's also funny you mentioned Notch, as Minecraft and its aesthetic are a direct result of lack of artistic talent, proving that you don't need great looking 3d models at all to create the best selling video game in history...
I agree, though that ML/AI tooling for asset generation would be a major improvement for the gaming (and film!) industry.
But isn't that exactly why market places like the Unity Asset Store, TurboSquid, and UE Marketplace exist?
The asset stores for Unity and UE in particular offer assets that are ready for use with these frameworks and can also be easily modified if need be.
I rather think the problem is the expectation to get high quality assets for free, which just isn't a realistic assumption, unless you're willing to team up with an artist who is willing to volunteer their time.
It's also funny you mentioned Notch, as Minecraft and its aesthetic are a direct result of lack of artistic talent, proving that you don't need great looking 3d models at all to create the best selling video game in history...
I agree, though that ML/AI tooling for asset generation would be a major improvement for the gaming (and film!) industry.
I've been looking for that for the last couple of days and couldn't find much. Great resources linked as comments.
I had found that one myself: https://github.com/mishig25/3d-posenet
This almost touches the field of A-Life, rather than neural methods for 3D geometry. There's OpenAI's humanoid walker, that taught itself bi-pedal motion. But as far as I know there is no MeshNet or ShaderNet, analogous to Imagenet. The research data sets like ShapeNet consist of CAD models for object detection and scene understanding.
Chimera Painter demo itself is unbelievably fun. When AI Dungeon and GPT-3 launched the observation was made that we are all "prompt engineers" now. It's the quality of the input that differentiates final results ;)
Chimera Painter demo itself is unbelievably fun. When AI Dungeon and GPT-3 launched the observation was made that we are all "prompt engineers" now. It's the quality of the input that differentiates final results ;)
This might interest you: https://dcgi.fel.cvut.cz/home/sykorad/monster_mash
Since your comment became a list of AI links, here is one from me. I'm not an expert in this area, but it looked like a very interesting project in the space of artificial worlds creation: https://www.prometheanai.com/
Related video from 2018: https://www.youtube.com/watch?v=FlgLxSLsYWQ
Can we look forward to a Netrunner card generator? :)
GAN-generated art, name and flavor text, a perhaps more structured rules engine for the card type and behavior, and AlphaZero to play the card in a deck against itself a few thousand times to tune the balance.
GAN-generated art, name and flavor text, a perhaps more structured rules engine for the card type and behavior, and AlphaZero to play the card in a deck against itself a few thousand times to tune the balance.
Man, I was really hoping to be the first to pull the trigger on that. But if I'm being honest with myself: I ain't gonna be putting in the time to learn to tensorflow.
Tensorflow (or PyTorch, or any other framework) aren't the problem, though.
It's choosing a suitable net architecture, loss function, and training methodology all of which require in-depth knowledge and lots of experimentation.
Even after that you're only half way there, because now the fun and exploration is over and the pain begins: finding (and worse even - annotating!) and validating training data.
Once this herculean task has been finished, it's time to get your credit card ready and shell out hundreds of dollars/pounds/euros for days of GPU time to train the basic model.
With that out of the way, it's time for a little fun to return and finetune your model(s), which usually can be done using even a mediocre desktop PC or laptop.
There's a reason you can make serious money with even a comparatively simple and straight forward idea that uses ML and AI to get quality results. It's much, much more involved than just "learning some tensorflow".
It's choosing a suitable net architecture, loss function, and training methodology all of which require in-depth knowledge and lots of experimentation.
Even after that you're only half way there, because now the fun and exploration is over and the pain begins: finding (and worse even - annotating!) and validating training data.
Once this herculean task has been finished, it's time to get your credit card ready and shell out hundreds of dollars/pounds/euros for days of GPU time to train the basic model.
With that out of the way, it's time for a little fun to return and finetune your model(s), which usually can be done using even a mediocre desktop PC or laptop.
There's a reason you can make serious money with even a comparatively simple and straight forward idea that uses ML and AI to get quality results. It's much, much more involved than just "learning some tensorflow".
> It's choosing a suitable net architecture, loss function, and training methodology all of which require in-depth knowledge and lots of experimentation.
Yes, things have progressed since my time in undergrad studying NNs and such. Presumably the work done on MtG transfers.
> Even after that you're only half way there, because now the fun and exploration is over and the pain begins: finding (and worse even - annotating!) and validating training data.
Well, for A:NR there's already multiple databases and a thousand cards. That's your training set. Throw in the online league cards if you require more.
> It's much, much more involved "learning some tensorflow".
I already work in SRE for a huge AI you've heard of, but thanks for the lecture.
Yes, things have progressed since my time in undergrad studying NNs and such. Presumably the work done on MtG transfers.
> Even after that you're only half way there, because now the fun and exploration is over and the pain begins: finding (and worse even - annotating!) and validating training data.
Well, for A:NR there's already multiple databases and a thousand cards. That's your training set. Throw in the online league cards if you require more.
> It's much, much more involved "learning some tensorflow".
I already work in SRE for a huge AI you've heard of, but thanks for the lecture.
A thousand cards is not much of a training set so I'm not really sure deep learnign would get very far.
A few years ago there was a community project training RNNs to generate M:tG cards [1], that if I'm not mistaken became Roborosewater [2]. M:tG already had around 12-15k cards by the time and still the majority of generated cards did not make sense. In the mtgsalvation thread in [1], most of the time people crack out with the nonsense that's generated by the RNNs people train [3]. There are also plenty of cards that make sense and are interesting and even usable, but they had to be hand-picked out of buckets of nonsense. A lot of curation and probably editing generated cards by hand would be needed, which somewhat defeats the purpose of the whole endeavour.
To generate new cards it would make much more sense to use good old procedural generation starting with a hand-crafted grammar of the rules text. This is certainly doable for a game of the size of M:tG, let alone Netrunner. For instance, M:tG Arena, the online p2p version of the game, runs on a game engine with a hand-crafted parser, the Game Rules Parser, that essentially resolves spells like an interpreter executing a script [4][5].
I mean to say, sometimes jumping feet-first into learning how to use a new set of complex tools is not necessary. Simpler tools that should already be in every programmer's toolbox can do the job fine, sometimes, even when it seems easier to just get a bunch of data and put it through a machine learning meat grinder. "Easier" might turn out to mean you need to do a lot more work before and after you can use the "easier" method (e.g. labelling, training, curation, etc) and not even get very good results in the end.
___________
[1] https://www.mtgsalvation.com/forums/magic-fundamentals/custo...
[2] https://twitter.com/RoboRosewater
[3] The project was started by the user Talcos, then more people started training their own nets using Talcos' code.
[4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren...
[5] To tout my own horn a bit, I did the same thing in my degree as a final year project, but I didn't have enough time to get full coverage of the entire card corpus at the time - still, I was just one undergrad student and I did manage to get a big chunk of the game working with a rules parser. The parser was written in Prolog meaning I could "run it backwards" as a generator so the project had an M:tG ability text generator that spat out grammatically correct, if not always particularly useful, text. I'm not linking to the project because it was 9 years ago and it's painfully embarrassing looking at it now.
A few years ago there was a community project training RNNs to generate M:tG cards [1], that if I'm not mistaken became Roborosewater [2]. M:tG already had around 12-15k cards by the time and still the majority of generated cards did not make sense. In the mtgsalvation thread in [1], most of the time people crack out with the nonsense that's generated by the RNNs people train [3]. There are also plenty of cards that make sense and are interesting and even usable, but they had to be hand-picked out of buckets of nonsense. A lot of curation and probably editing generated cards by hand would be needed, which somewhat defeats the purpose of the whole endeavour.
To generate new cards it would make much more sense to use good old procedural generation starting with a hand-crafted grammar of the rules text. This is certainly doable for a game of the size of M:tG, let alone Netrunner. For instance, M:tG Arena, the online p2p version of the game, runs on a game engine with a hand-crafted parser, the Game Rules Parser, that essentially resolves spells like an interpreter executing a script [4][5].
I mean to say, sometimes jumping feet-first into learning how to use a new set of complex tools is not necessary. Simpler tools that should already be in every programmer's toolbox can do the job fine, sometimes, even when it seems easier to just get a bunch of data and put it through a machine learning meat grinder. "Easier" might turn out to mean you need to do a lot more work before and after you can use the "easier" method (e.g. labelling, training, curation, etc) and not even get very good results in the end.
___________
[1] https://www.mtgsalvation.com/forums/magic-fundamentals/custo...
[2] https://twitter.com/RoboRosewater
[3] The project was started by the user Talcos, then more people started training their own nets using Talcos' code.
[4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren...
[5] To tout my own horn a bit, I did the same thing in my degree as a final year project, but I didn't have enough time to get full coverage of the entire card corpus at the time - still, I was just one undergrad student and I did manage to get a big chunk of the game working with a rules parser. The parser was written in Prolog meaning I could "run it backwards" as a generator so the project had an M:tG ability text generator that spat out grammatically correct, if not always particularly useful, text. I'm not linking to the project because it was 9 years ago and it's painfully embarrassing looking at it now.
This reminds me of Matthew Rayfield's GPT work to generate Pokémon:
https://matthewrayfield.com/articles/ai-generated-pokemon-sp...
Discussion: https://news.ycombinator.com/item?id=25152764
Discussion: https://news.ycombinator.com/item?id=25152764
Here's the URL to test it : https://storage.googleapis.com/chimera-painter/index.html
It is very much not working for me. I get a mostly-transparent creature. The opaque parts seem to follow the brushstrokes of my drawing. Maybe it is because it was trained on finely detailed material, whereas my drawing was a coarse doodle.
You have to draw solid regions, not an outline.
I did draw solid regions.
I did that and also got inexplicable holes:
https://i.paste.pics/8bed0cd17629f0e9c852a24162bf381e.png
Otherwise my clumsy, misshapen caricature turned out surprisingly nice. (I mean, relative to how nonsensical its anatomy is.) The shapes are followed very precisely, so yes, blobby input begets blobby output.
https://i.paste.pics/8bed0cd17629f0e9c852a24162bf381e.png
Otherwise my clumsy, misshapen caricature turned out surprisingly nice. (I mean, relative to how nonsensical its anatomy is.) The shapes are followed very precisely, so yes, blobby input begets blobby output.
I get the same issue. I know I'm not even close to a decent artist, but come on, why are my legs transparent:
https://imgur.com/a/kb0gWI2
https://imgur.com/a/kb0gWI2
I think I drew Satan
https://i.imgur.com/FHrP8La.jpg
https://i.imgur.com/FHrP8La.jpg
I’m not sure which picture is more scary...
In the future GANs will replace all of human creativity so much so that humans will spend all of their creative outlets on training GANs and using GANs to do the creation for them.
A lot of AI/ML technology is (inherently?) self-reinforcing (think product recommendations), so your prediction is not wrong.
GANs and computers are mirrors of the engineer's mind. You can only get out some combination of the trained data set and user input.
Huamns can produce novel artworks that a computer cannot, usually with more control over the intagibles like creativity and the fundamentals like drawing and perceiving the world.
Huamns can produce novel artworks that a computer cannot, usually with more control over the intagibles like creativity and the fundamentals like drawing and perceiving the world.
> You can only get out some combination of the trained data set and user input.
The same could be said of humans. A lot of humans would say a random piece of quality GAN-generated art is a creative work. They may not say so if they first looked at every piece of training data that went into it. But then, they might not do the same if they looked at every piece of art produced and viewed by a given human artist previously.
As a thought experiment, if the line in the sand is whether a computer can generate a piece of art that people think is creative, even with full knowledge of every piece of training data- I think that's not at all difficult to overcome long term. Possibly even with current models.
The same could be said of humans. A lot of humans would say a random piece of quality GAN-generated art is a creative work. They may not say so if they first looked at every piece of training data that went into it. But then, they might not do the same if they looked at every piece of art produced and viewed by a given human artist previously.
As a thought experiment, if the line in the sand is whether a computer can generate a piece of art that people think is creative, even with full knowledge of every piece of training data- I think that's not at all difficult to overcome long term. Possibly even with current models.
Yes, in some point in the future though the amount of output data from GANs trained on Human Data will surpass the amount of output data from humans directly.
When that happens people will start training GANs with GAN data.
It will become a self feeding loop with humans only curating data.
Of course even the curating of the data is ripe for automation via ML.
When that happens people will start training GANs with GAN data.
It will become a self feeding loop with humans only curating data.
Of course even the curating of the data is ripe for automation via ML.
You talk as if a combination of the training space and user input can't be novel. There are a finite number of colors and textures, so if your data set has good coverage (and your model splits hue and luminosity) you could conceivably cover nearly the entire space. That would leave the shape as the differentiator of novelty.
Yes. Every kind of creative activities will be done using GANs. From writing lyrics, designing architecture, producing music to painting.
Ah, they figured out how to brand the nightmare generator!
Example context: "Fear and Loathing in Las Vegas run through Google's Deep Dream neural network is pure nightmare fuel" https://www.independent.co.uk/arts-entertainment/films/news/...
Example context: "Fear and Loathing in Las Vegas run through Google's Deep Dream neural network is pure nightmare fuel" https://www.independent.co.uk/arts-entertainment/films/news/...
It seems like the drawing has to be very well drawn in order to get decent results.
I agree, my attempt did not yield results anywhere near the examples: https://i.imgur.com/cZUBWzH.png
Developments in this area will help (indy) game developers to prototype more quickly. I like it.
Seems to be a couple of iterations from being usable but it'll get there.
I'm also envisioning a Ready Player One type of online escape world where content is generated from social media images/films.
I'm also envisioning a Ready Player One type of online escape world where content is generated from social media images/films.
Has anyone managed to get good results from this thing? I can only get wood-textured junk.
the 'clear' button doesn't work on firefox :(
None of the brushes seem to work in Firefox either.
Chrome 86 on Ubuntu. Not working either. :(
See also the old GauGAN: http://nvidia-research-mingyuliu.com/gaugan/
In that one you couldn't get much control like drawing letter clouds in the sky
In that one you couldn't get much control like drawing letter clouds in the sky
Reminds me of the RTS game "Impossible Creatures"
https://www.youtube.com/watch?v=G_qjgr-JM9U
https://www.youtube.com/watch?v=G_qjgr-JM9U
[deleted]
The future is truly now [1]. The detail on this.
[1]: https://imgur.com/a/VwuLdbi
[1]: https://imgur.com/a/VwuLdbi
We got a Cthulhu image generator
https://i.imgur.com/F95uksf.png
https://i.imgur.com/F95uksf.png
The depth in your generated image is very interesting.
Do "Manny Manimal, Man of Many Animals!"
Why is everything brown-ish?
It just keeps putting the same texture over all the drawings. Maybe, in the color space, it has collapsed or just overfitted.
Started as a human, but i got lost along the way https://i.imgur.com/xEgmY0y.png
[deleted]
So not only will we not have any meaningful local hardware in the future, but games themselves will simply be custom-generated to satisfy the player’s exact desires?
Sounds like it would become pretty dull. Art and experiences are interesting because they are unexpected. Experiencing someone else’s unique and surprising artistic voice is what I love most about good games / art experiences.
Sounds like it would become pretty dull. Art and experiences are interesting because they are unexpected. Experiencing someone else’s unique and surprising artistic voice is what I love most about good games / art experiences.
This seems like a similar comment that would have been made about drum machines. Creative people will always find a way to push the limits of a new technology that makes something difficult suddenly trivial.
> This seems like a similar comment that would have been made about drum machines
I doubt that. Simplified and (semi-)automated instruments have been around for a long time since there is legitimate interest in such tools from musicians.
The difference here is that people have a tendency to regard machine generated output as more "correct" (whatever that means). The same psychological mechanisms that make Level 3 autonomy in cars and aeroplanes so dangerous can lead to a future where a closed loop automated content generation (e.g. train the next gen GAN on output from the previous gen GAN) kills all creativity and conditions us to prefer the generated aesthetic.
I doubt that. Simplified and (semi-)automated instruments have been around for a long time since there is legitimate interest in such tools from musicians.
The difference here is that people have a tendency to regard machine generated output as more "correct" (whatever that means). The same psychological mechanisms that make Level 3 autonomy in cars and aeroplanes so dangerous can lead to a future where a closed loop automated content generation (e.g. train the next gen GAN on output from the previous gen GAN) kills all creativity and conditions us to prefer the generated aesthetic.
I’m not sure why you’re being downvoted, it’s a fair concern. I would say any sufficiently advanced system that you’re describing would detect your desire for something unexpected and create something that fit that for you. Ironically you wouldn’t know that it was tailored specifically to you, because it was so precise in evaluating your tastes.
No, games and art will be custom-generated to satisfy the designer's exact desires as they do their work, allowing them to more rapidly explore the space of possible expression.
Don't worry, it's still dystopian -- games will become exceedingly good and compelling; you won't want to do anything else.
Don't worry, it's still dystopian -- games will become exceedingly good and compelling; you won't want to do anything else.
Good and compelling games do not even make my top100 when thinking about dystopia. Or is this some kind of "amusing ourselves to death while the world burns around us" comment?
One way to look at it is that only humans can have a unique and surprising artistic voice, and leave it at that. However, some games can have _extremely_ interesting emerging content, e.g. as seen in Dwarf Fortress and RimWorld. Since much of the content is procedurally generated, at the moment we are left with text descriptions of things. I can easily see a future where the game worlds are automatically fleshed using these types of techniques, and it would be for the better.
Also, on a purely economic scale, automating some aspects of art asset creation for games would be a complete game changer. It would probably up the art capabilities of indie game developers quite a bit, although it's hard to predict these types of things (in the sense that it might benefit AAA developers even more - who knows).
Also, on a purely economic scale, automating some aspects of art asset creation for games would be a complete game changer. It would probably up the art capabilities of indie game developers quite a bit, although it's hard to predict these types of things (in the sense that it might benefit AAA developers even more - who knows).
[1] https://i.imgur.com/yBDW2gd.png