Comparison of the Best NSFW Image Moderation APIs 2018(adityaananthram.com)
adityaananthram.com
Comparison of the Best NSFW Image Moderation APIs 2018
https://www.adityaananthram.com/comparison-of-the-best-nsfw-image-moderation-apis-2018/
75 comments
That’s absolutely amazing. Reminds me of Francis Bacon (the painter) and his flesh / gore themed works.
This is brilliant, especially the cross-categorised images.
I had thought these were talking about 'pornography', but this makes it clear that a very large amount of its training set must consist of dick pics.
I had thought these were talking about 'pornography', but this makes it clear that a very large amount of its training set must consist of dick pics.
Not necessarily, the NN may just concentrate on them in the porn it's trained on, because it's a common factor that's in porn, but not in beach pics for example. (whereas in both you have almost naked people).
I'm not so sure about that - I see a lot of labia here.
Agreed. It's fascinating, really. To me it seems that the algorithm is generating both male/female genitalia, but it's sort of "choosing" based on what "looks best".
That is to say, certain features that in their SFW form already look vaguely like a penis seem to be depicted as a penis (the statues, volcano, etc), whereas other features like the beaches were depicted as vulvas. And the slight red tinge present in some of the canyon/beach images also elicited the vulva, even if the shape of the surface wasn't something "obvious".
Man, ironically this stuff to me is true abstract art. To use a somewhat trite cliche, the [extremely subtle] erotic nature of these environments that are clearly not actually genitalia gives insight into how the human mind works just as well as it does into (convolutional?) neural networks themselves
That is to say, certain features that in their SFW form already look vaguely like a penis seem to be depicted as a penis (the statues, volcano, etc), whereas other features like the beaches were depicted as vulvas. And the slight red tinge present in some of the canyon/beach images also elicited the vulva, even if the shape of the surface wasn't something "obvious".
Man, ironically this stuff to me is true abstract art. To use a somewhat trite cliche, the [extremely subtle] erotic nature of these environments that are clearly not actually genitalia gives insight into how the human mind works just as well as it does into (convolutional?) neural networks themselves
This is very cool comparison. I would really like folks at Netflix/Amazon to take this and build a feature that auto-skips through non-PG rated content so you can watch some of otherwise very great movie/TV content with very young kids. For example, StarTrek. They can call this feature "PGify" :).
Better yet you could wait until the kids are mature enough, watch the content, and then discuss it with them.
That assumes the stripped content is important in some way.
How about when the extra scenes are literally useless to the point of the content and are just "wedged in" as fanservice to increase viewership, in a way where the content is better off without them? This describes nearly every sex scene in a Hollywood movie that I can remember.
Though also, separately: I'm an adult, but I'm imagining going home for the holidays to my parents, sitting around at night, picking some movie to watch on Netflix, and it ends up having a sex scene in it. I don't want that to happen! It's not a matter of maturity—I just don't want to share the experience of watching a sex scene with my parents. I'd rather be able to tell Netflix/the streaming box/the TV, to skip those scenes.
How about when the extra scenes are literally useless to the point of the content and are just "wedged in" as fanservice to increase viewership, in a way where the content is better off without them? This describes nearly every sex scene in a Hollywood movie that I can remember.
Though also, separately: I'm an adult, but I'm imagining going home for the holidays to my parents, sitting around at night, picking some movie to watch on Netflix, and it ends up having a sex scene in it. I don't want that to happen! It's not a matter of maturity—I just don't want to share the experience of watching a sex scene with my parents. I'd rather be able to tell Netflix/the streaming box/the TV, to skip those scenes.
So the scenes where they kill each other are okay but God forbid a nipple? That's sick...
This is a really good idea.
It makes me smile to think that my 15-year-old self would have loved a feature that did the same but in reverse. That was, of course, a pre-internet, almost porn-free world.
It makes me smile to think that my 15-year-old self would have loved a feature that did the same but in reverse. That was, of course, a pre-internet, almost porn-free world.
There is (at least) one company providing this kind of services. They tag in the movie all the content with timestamps, and can provide it to a player to skip or not depending on the tag (nudity, violence, etc.).
As far as I know, it was manual, but that was already a few years ago.
I don't remember the name of the company, but surely it can be found and there are probably multiple such providers. The service was provided to TV operators (satellite, cable or pay-TV) to include in their set-top boxes.
PS: of course, as sjcsjc mentioned in a sibling comment, it immediately gives the idea to many people to only show those parts :)
As far as I know, it was manual, but that was already a few years ago.
I don't remember the name of the company, but surely it can be found and there are probably multiple such providers. The service was provided to TV operators (satellite, cable or pay-TV) to include in their set-top boxes.
PS: of course, as sjcsjc mentioned in a sibling comment, it immediately gives the idea to many people to only show those parts :)
There was. The most well known was CleanFlicks. They were all sued into oblivion.
https://en.m.wikipedia.org/wiki/CleanFlicks
There's a documentary about it: https://m.imdb.com/title/tt1007026/
https://en.m.wikipedia.org/wiki/CleanFlicks
There's a documentary about it: https://m.imdb.com/title/tt1007026/
Not all; ClearPlay won: https://en.wikipedia.org/wiki/ClearPlay
More recently: VidAngel
Hadn't seen that. Predictably, also being sued by multiple entities: http://blog.vidangel.com/category/legal/
Studios have the airplane edits of many movies.. not sure why they won't let you select that on the Netflixes.
My 14 yo would be very happy to select those versions.
My 14 yo would be very happy to select those versions.
This idea was extensively discussed here in 2016: https://news.ycombinator.com/item?id=11354021
If the nsfw images were from google search (or are images that could easily be indexed by google), it's not surprising that google is performing well on them. Presumably any low confidence images would've been manually tagged and re-added to their training set already.
Hey zawerf, your right. 2 notes.
1) it's very difficult to find nsfw images especially a particular kind like gore or suggestive nudity unless you Google things (which indicates a bigger problem). Maybe the solution is to use Bing (maybe this would cause the same issue in compariosn) or DuckDuckGo. But honestly I think if DuckDuckGo indexed a page, I'm pretty sure Google did as well. You would probably need something off of non indexed website which makes the job significantly harder.
2) even though google has all the images it's still not the best performing NSFW Detector Nanonets is.
1) it's very difficult to find nsfw images especially a particular kind like gore or suggestive nudity unless you Google things (which indicates a bigger problem). Maybe the solution is to use Bing (maybe this would cause the same issue in compariosn) or DuckDuckGo. But honestly I think if DuckDuckGo indexed a page, I'm pretty sure Google did as well. You would probably need something off of non indexed website which makes the job significantly harder.
2) even though google has all the images it's still not the best performing NSFW Detector Nanonets is.
If you're struggling on (1), Reddit has all kind of bizarre subreddits which cater to all kinds of images. It's also conveniently (is that fortunate?) well categorised. There are definitely subs for gory photos, non-nudity NSFW images and so on. Reddit is also a great resource for categorised SFW, since there are so many subreddits with active and strict moderation.
It's reasonably easy to scrape: https://github.com/jveitchmichaelis/redditdownloader
They may have changed the API since I wrote that though.
It's reasonably easy to scrape: https://github.com/jveitchmichaelis/redditdownloader
They may have changed the API since I wrote that though.
So one thing I did for a while was to take content from porn sites, extract key frames and any available images, and look for SFW images. Hard problems include detecting NSFW stuff that includes no nudity / genitalia (bodily fluids and solids) and correcting skin tone (most models being trained on primarily Caucasian and Asian performer data had trouble with darker skin tones). Some previous research showed that the trained CNNs were looking very hard for lipstick, so adding in samples from performers with less contrast on the lips was also important for training purposes. I didn’t notice anything terribly different when training with transgender performers (hotdog / not hot dog is very easy from an object detection basis) but I had to be sure that there wasn’t confusion that a human could have that would introduce bias into the model. Another big plus with porn sites is that your data is already tagged by its users and they are checked aggressively for accuracy.
My point is really that image searches can only get you so far and that biases are abound in casual NSFW searches to the extent you may need to curate your own data sets that look like they could be on a random porn site in ANY section. Finding an appropriate training set almost reminds me of jury selection processes.
My point is really that image searches can only get you so far and that biases are abound in casual NSFW searches to the extent you may need to curate your own data sets that look like they could be on a random porn site in ANY section. Finding an appropriate training set almost reminds me of jury selection processes.
[deleted]
Skimmed through. Article says "Yahoo is the only one that is completely free to use but is self hosted hence not included in this"
Is the set that yahoo released long ago the only self hosted option available?
I have several projects that might end up using some of the yahoo release in order to stay self hosted. I can't imagine telling all users that we share all their pics with these third parties to check up on them.
Is the set that yahoo released long ago the only self hosted option available?
I have several projects that might end up using some of the yahoo release in order to stay self hosted. I can't imagine telling all users that we share all their pics with these third parties to check up on them.
stevenicr your'e right in that regard. Yahoo is the only one who has provided an on premises solution which is really sad. Even they haven't released the dataset of their images just the model. If more of these companies released open source models on a frequent basis we would all have less objectionable content on the internet.
I guess the reason companies haven't done that and the reason Yahoo isn't really good is because it's difficult to constantly keep updating a model that's already been "released".
I guess the reason companies haven't done that and the reason Yahoo isn't really good is because it's difficult to constantly keep updating a model that's already been "released".
I look forward to learning more about "updating a model that's already been "released"." - If it could be as simple as wordpress updates (along with multiple ways to be notified similarly) - that would be optimal imho.
I believe many of these companies are keeping the models to themselves in an effort to stave off competition and try to be monopolies that can hope to get all the government cheese.
Unfortunately keeping the data to themselves in this regard is a disservice to the worldwide community in many unforeseen ways. Certainly they could keep contracts with big players just by providing the cloud computing powers and ease of setup, while offering updated models to others to use.
Maybe offer 99% accuracy to small site operators and 99.5% to contract API accounts or something.
It's more than just objectionable content for the net, it's other things too. Many apps and online connected services could benefit from running pics uploaded through a system that checks not only for nudity but also for age of people.
I could see adding some server space and cpu power to check images transferred for important issues like these - however I can't see sending all that data to third parties, sacrificing privacy of users while at the same time helping to make their proprietary model better without a stake of ownership in such.
Seriously this reminds me of facebook's "upload your nude pics to our system so we can notify you if someone tries to revenge porn share you nudes with third parties" - except that is people doing something consciously knowing they are sharing with fbook, it's employees and it's computer systems...
with these API calls, I would guess that most people who have their images run through them have no idea their pics are being shared with third parties, who are sometimes also putting the images in front of human moderators for further scrutiny.
I guess this kind of IP thing is happening with voice recognition stuff as well in many areas. At least there is more open source in that domain I believe.
I believe many of these companies are keeping the models to themselves in an effort to stave off competition and try to be monopolies that can hope to get all the government cheese.
Unfortunately keeping the data to themselves in this regard is a disservice to the worldwide community in many unforeseen ways. Certainly they could keep contracts with big players just by providing the cloud computing powers and ease of setup, while offering updated models to others to use.
Maybe offer 99% accuracy to small site operators and 99.5% to contract API accounts or something.
It's more than just objectionable content for the net, it's other things too. Many apps and online connected services could benefit from running pics uploaded through a system that checks not only for nudity but also for age of people.
I could see adding some server space and cpu power to check images transferred for important issues like these - however I can't see sending all that data to third parties, sacrificing privacy of users while at the same time helping to make their proprietary model better without a stake of ownership in such.
Seriously this reminds me of facebook's "upload your nude pics to our system so we can notify you if someone tries to revenge porn share you nudes with third parties" - except that is people doing something consciously knowing they are sharing with fbook, it's employees and it's computer systems...
with these API calls, I would guess that most people who have their images run through them have no idea their pics are being shared with third parties, who are sometimes also putting the images in front of human moderators for further scrutiny.
I guess this kind of IP thing is happening with voice recognition stuff as well in many areas. At least there is more open source in that domain I believe.
Great write up. As a photographer, the more companies are relying on this the more frustrating the mis classification of SFW as NSFW is getting. I’ve copped a few Facebook bans from automated detection (I assume given how quickly the report comes in) for images totally within their community guidelines. A number of my photographer friends are in the same boat also. I counted at least 5 of us on 30 day bans in October alone.
I had a friend pick up a presumably automated ban from Facebook this month for posting a picture of a soldier from a WWII re-enactment he went to. No gore, no nudity, just a half body shot of a German soldier in uniform standing in the camp, smiling at the camera. No idea what triggered it, but it is scary.
"WWII.. German soldier..."
"No idea what triggered it..."
Really? I don't agree with it, but it is almost certainly just "hey that's a nazi!"
"No idea what triggered it..."
Really? I don't agree with it, but it is almost certainly just "hey that's a nazi!"
In the past, Facebook's moderation (outside of user flagging) has involved some automated Facebook processes flagging the photo for review and then a human looking at the photo and decided to issue a account ban.
My friend lives in the mostly drama-free midwest, has a mostly drama-free facebook life, and participates in WWII reenactments seemingly a couple times a month. It seems very unlikely that this photo was flagged by a friend of his.
So it would appear that Facebook's automated tools are now sometimes thinking that WWII German soldiers are Nazi's. And either Facebook human thought the photo was worth a temp ban, or an automated tool is now banning anyone it thinks is a "Nazi". These are new things that haven't been happening in the past.
My friend lives in the mostly drama-free midwest, has a mostly drama-free facebook life, and participates in WWII reenactments seemingly a couple times a month. It seems very unlikely that this photo was flagged by a friend of his.
So it would appear that Facebook's automated tools are now sometimes thinking that WWII German soldiers are Nazi's. And either Facebook human thought the photo was worth a temp ban, or an automated tool is now banning anyone it thinks is a "Nazi". These are new things that haven't been happening in the past.
Ah, I didn't realize there were humans in the loop. Could also be that they are overworked and have just a few seconds to decide. Or a friend was being silly, or accidentally pushed the flag button.
[deleted]
Facebook uses human filters since their policy is not a simple "no nudity" one. For example, breastfeeding pictures are allowed.
https://www.wnycstudios.org/story/post-no-evil
https://www.wnycstudios.org/story/post-no-evil
I know that Facebook uses human content filters - A friend of a friend is one - but something has changed with Facebook moderation in the last few months, and I have a strong suspicion that Facebook has started rolling out algorithmic moderation without humans in the loop.
Given I've had (SFW) images reported within seconds of posting them to a private group, I think they've gone back on their word of that.
The Yahoo NSFW classifier that is mentioned is available here: https://github.com/yahoo/open_nsfw
I wonder if there's a way to ship this model and run it in an end-user app, in a way that does not require additional setup. I could use that.
I wonder if there's a way to ship this model and run it in an end-user app, in a way that does not require additional setup. I could use that.
Ooh, I wonder if you could make a "family friendly" browser extension or firewall from that.
I think it's possible to convert the caffe model and use something like Firebase MLKit or Core ML:
https://firebase.google.com/docs/ml-kit/use-custom-models
https://developer.apple.com/documentation/coreml
https://firebase.google.com/docs/ml-kit/use-custom-models
https://developer.apple.com/documentation/coreml
The Gore10 image isn't gore, unless there's something I don't know about going on.
It appears to be a toy cartoon pet rat situated on top of some glossy rubbery plastic red toy prop slime guts. Whatever it is, or is supposed to be, it isn't an explicit or graphic depiction of real violence.
Does that matter? I think so. Maybe it's not something to show a small child, and at a glance, one might develop the wrong impression, but it's obviously phoney nonsense, and as harmless as a red halloween Jell-o dish with rubber eyeballs suspended in the mold.
It appears to be a toy cartoon pet rat situated on top of some glossy rubbery plastic red toy prop slime guts. Whatever it is, or is supposed to be, it isn't an explicit or graphic depiction of real violence.
Does that matter? I think so. Maybe it's not something to show a small child, and at a glance, one might develop the wrong impression, but it's obviously phoney nonsense, and as harmless as a red halloween Jell-o dish with rubber eyeballs suspended in the mold.
Hey miss_classified I was pretty disturbed by most of these images. That images was no exception. Honestly till you pointed it out I had no clue. Anyway if it's safe for work or not that's debatable. Would I want my co-workers seeing this on my screen probably not. As I mentioned in the blog right at the end I'm more convinced after going through this exercise that it's difficult if not impossible for us to agree on what is and isn't safe for work.
This image is definitely safe for work:
https://drive.google.com/file/d/1J2YwtYzDUyK9ESdyxjaWLArws6U...
No one would get fired for that image. I could imagine it being sold as a Halloween product on Ebay or Amazon, and it should appear in normal Safe Search search results, since it is certainly a toy. It's a scary monster toy, but a toy nonetheless.
Is it safe for every context? Maybe not. There are definitely more scenarios than just being "Safe For Work" when it comes to rating content. Work is ostensibly filled with adults.
Would this image be safe for all age groups? Maybe not. Is it possible you might encounter a product like this in a department store or seasonal store? Yes. Depending on labeling, you might see it in either Spencer Gifts or Target. So, this is where parental controls come in, and MPAA or ESRB ratings are probably a better guide to moderation.
Is it rated "G" or "E for Everyone"? I'd still argue yes, and that red plastic isn't worthy of restriction unless context is explicitly demonstrating that it's an animal eating human remains somehow. If it were labeled as "Rat Eating Zombie Guts" it might be labeled "PG" or "T for Teens" but completely unlabeled, and stripped of context, or other conceptual cues lent by well-known notoriety/infamy, it's clearly just painted rubber, and a curiosity, not a psychologically damaging image.
https://drive.google.com/file/d/1J2YwtYzDUyK9ESdyxjaWLArws6U...
No one would get fired for that image. I could imagine it being sold as a Halloween product on Ebay or Amazon, and it should appear in normal Safe Search search results, since it is certainly a toy. It's a scary monster toy, but a toy nonetheless.
Is it safe for every context? Maybe not. There are definitely more scenarios than just being "Safe For Work" when it comes to rating content. Work is ostensibly filled with adults.
Would this image be safe for all age groups? Maybe not. Is it possible you might encounter a product like this in a department store or seasonal store? Yes. Depending on labeling, you might see it in either Spencer Gifts or Target. So, this is where parental controls come in, and MPAA or ESRB ratings are probably a better guide to moderation.
Is it rated "G" or "E for Everyone"? I'd still argue yes, and that red plastic isn't worthy of restriction unless context is explicitly demonstrating that it's an animal eating human remains somehow. If it were labeled as "Rat Eating Zombie Guts" it might be labeled "PG" or "T for Teens" but completely unlabeled, and stripped of context, or other conceptual cues lent by well-known notoriety/infamy, it's clearly just painted rubber, and a curiosity, not a psychologically damaging image.
This comparison is a nice snapshot in time. Could this be developed towards the direction of Virustotal[0] where an image can be uploaded at any and all services would be compared. Combined with a 'reference' set of images that are periodically ran through all the services as to see how they improve/change in their capabilities over time.
[0] https://www.virustotal.com/
[0] https://www.virustotal.com/
[deleted]
This is a pretty comprehensive comparison given the number of APIs included there. Based on my experience, different APIs return different content, could you share how do you normalize them (the logic behind each API response to your NSFW/SFW labels) ? Thanks!
"This diagram gives us a sense of the biases of each of the providers and how sensitive they are. Higher values for TP and TN are better. Values for FN and FN should be as small as possible."
I suppose this should read "FP and FN" instead of "FN and FN".
I suppose this should read "FP and FN" instead of "FN and FN".
Great comparison, and thanks for the review, happy being in the TOP-5 :).
It seems, though, Sightengine's scores on the SFW category were swapped. The detailed google sheet lists the NSFW "raw" scores, not the "safe" scores.
It seems, though, Sightengine's scores on the SFW category were swapped. The detailed google sheet lists the NSFW "raw" scores, not the "safe" scores.
looking into it, I will update the results if I find an error!
Also, the 3rd chart in "Overall Comparison" has a #REF! label
Since medium won't let me make edits. Fixed this and posted here:
https://www.adityaananthram.com/comparison-of-the-best-nsfw-...
Since medium won't let me make edits. Fixed this and posted here:
https://www.adityaananthram.com/comparison-of-the-best-nsfw-...
I wrote a comparison of all popular content moderation APIs on the Internet. My basic understanding from doing this is that it is far from a solved problem and there is no 1 size fits all.
What would be the best way to moderate user generated video content / live streams?
Take one frame per second and check it using these services? But that would get super expensive real quick ..
Take one frame per second and check it using these services? But that would get super expensive real quick ..
I'm already imagining someone reverse engineering your one frame per second solution and uploading porn where every 24th frame has been replaced with the Windows XP wallpaper.
Many of the named providers offer video APIs for detecting this content. Will work out much cheaper than throwing every frame at the image APIs.
Check out the "links to original images" sections as they are incorrect ~50% of the time. The name above the blurred image does not correspond to the link name.
I'm aware of the issue, and trying to fix it. But medium won't let me make any edits. Since I originally published the blog have been in constant touch with their support but they all seem to be away for thanksgiving. Just keep getting the error "Oops! Something happened to Medium. We’ll fix it as soon as we can."
Since medium won't let me make edits. Fixed this and posted here:
https://www.adityaananthram.com/comparison-of-the-best-nsfw-...
Not on-topic: A recent post here about blogging on medium.
https://news.ycombinator.com/item?id=18440756
https://news.ycombinator.com/item?id=18440756
Also be careful with opening many of these images, as they open in Google Drive. Which will display all images viewed on this blog the next time you open it.
Well, they're getting flagged now, how ironic:
> We're sorry. You can't access this item because it is in violation of our Terms of Service.
Tried to view a subset, because as this comment pointed out, NSFW is subjective: https://news.ycombinator.com/item?id=18526170
And if NSFW is subjective, the most offendable person gets to dictate what is NSFW (as we are already seeing happening in other areas).
> We're sorry. You can't access this item because it is in violation of our Terms of Service.
Tried to view a subset, because as this comment pointed out, NSFW is subjective: https://news.ycombinator.com/item?id=18526170
And if NSFW is subjective, the most offendable person gets to dictate what is NSFW (as we are already seeing happening in other areas).
How these images were collected? My worry is that Google and Bing could already have some of them indexed and marked depending on page safety score.
Is there a mathematically sound way to combined API results to obtain a higher confidence? Applying Bayes training on top perhaps?
Great article
The blog had a ton of errors I couldn't fix since medium's edit feature is down for me.
I posted the blog here again with fixes https://www.adityaananthram.com/comparison-of-the-best-nsfw-...
I posted the blog here again with fixes https://www.adityaananthram.com/comparison-of-the-best-nsfw-...
Ok, we've switched to that from https://towardsdatascience.com/comparison-of-the-best-nsfw-i....
Thanks a lot dang!
We use this in our adtech campaign moderation.
Just think, someone had to train these models. There are also models trained specifically for child pornography. Would you like to know these people?
My brother is a retired Inspector who did many child pornography and abuse cases. His IT guy did many searches and indexes of computers full of child porn. I’m happy to know both of these people.
It's fairly NSFW: https://open_nsfw.gitlab.io