Find images from movies based on what you draw(huggingface.co)
huggingface.co
Find images from movies based on what you draw
https://huggingface.co/spaces/osanseviero/draw_to_search
33 コメント
Looks like it's just guessing the object and then finding images tagged with that object. I was hoping for some sort of scene detection to find specific scenes from films.
Ah shucks! Well, you saved me a click.
Was hoping for some AI visual processing thing and was doing the mental gymnastics of how to store all that data.
Was hoping for some AI visual processing thing and was doing the mental gymnastics of how to store all that data.
It somewhat decently identifies the object drawn, but fails badly to actually find the identified object.
I tried to draw a stick figure and it didn’t work for me.
my mona lisa drawing is apparently a tree
Given the tiny drawing area and the insanely thick brush, you're lucky you got something as coherent as 'tree'
Here’s a demo of it in action: https://twitter.com/osanseviero/status/1447830148225589251?s...
[It takes a long time to load, so be patient.]
Nice idea, but I was never able to find the movie I was thinking of. Perhaps I'm too bad drawing.
Nice idea, but I was never able to find the movie I was thinking of. Perhaps I'm too bad drawing.
Are there details on how it works? How is CLIP being used here?
The code is available in the repo: https://huggingface.co/spaces/osanseviero/draw_to_search/blo...
It first uses a simple CNN to get one of the 100 classes. It then uses that class to retrieve images using CLIP. The original CLIP Space that helped for inspiration is https://huggingface.co/spaces/vivien/clip.
It first uses a simple CNN to get one of the 100 classes. It then uses that class to retrieve images using CLIP. The original CLIP Space that helped for inspiration is https://huggingface.co/spaces/vivien/clip.
Oof. That 100 class limit is less practical than just asking the user for arbitrary text input and using CLIP as a zero-shot classifier. You would have gotten better responses if the limitations and motivations were explained on the page.
Thanks for the feedback! This was just a demo I shared in Twitter and didn't expect it to be shared so widely by others. We're training a new CNN for more classes and we might switch to another dataset for the image retrieval part. It's really a very small pet project :D
I’m trying to draw the silence of lamb poster with Jodie foster but it keeps returning pictures of bearded men. Lol. Let me know if anyone is successful with that poster.
I had to come back to comment - I tried to draw the Batman symbol and the first thing that came back was The Silence of the Lambs poster.
Tried to draw sheep, but it sees cat...
It seems to keep suggesting the same few movies no matter what I draw
[deleted]
Literally anything -> syringe
Current state of A.I. in a few words.
I drew a closed fist, it says moon / alarm clock.
I drew a closed fist, it says moon / alarm clock.
Doesn’t work when I try to use it, always returns the same thing, and it seems like everyone else is having the same problem. Sounds cool though and I’ll be excited about the day we progress enough where something like this works! GANs are very cool.
I don't experience your problems. It returns new stuff for me.
[deleted]
Try sketching one of these:
- bench (my favorite)
- drums
- airplane
- basketball
[deleted]
From what I gather this does object detection on the picture and then from those detected items compares to movie stills that have been similarly labelled. On paper sounds like it could work but in practice .. results are a bit underwhelming when I tried
It doesn't seem to work for me - you draw a shape and then it returns some completely unrelated images. You then pick one of those images and try to draw it as closely as possible and then it returns a completely unrelated image again.
I thought the weird thing was that with that second step, if I drew a broom exactly for example, and it guessed “broom” correctly, it returns images with no brooms in them at all?
Tried to draw the Deathly Hallows symbol with no luck :( Cool idea though
First of all, most results are anime. I tried a few easy-to-draw, pretty popular symbols like Harry Potter's forehead scar, The Incredibles logo but got nothing close to them. It's a very cool idea but still needs a lot of work.
Drew Batman in the Animated Series style. Kept getting a cat and Deadpool in the Flashdance pose. Cool, but still has a ways to go.
This is not ready for prime time. This is not even ready to go up against infomercials at 4am on a Sunday morning.