The trauma of Facebook’s content moderators(restofworld.org)
restofworld.org
The trauma of Facebook’s content moderators
https://restofworld.org/2020/facebook-international-content-moderators/
267 comments
It would be great if there were a way to make content moderation more humane, but perhaps this is like saying that it would be great if we could make hospital emergency room work more humane. It is unavoidably traumatic, and ER staff are known to detach and use dark humor as a coping mechanism. Content moderation is a similarly noble and difficult profession.
ER staff are paid a lot more, are sometimes even in a union, and also get the benefit of helping people and seeing the effect of their help.
Ever hang out with an ER staffer?
Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers.
And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.
Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers.
And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.
I would imagine that cases should follow a bell curve - sure, the exceptionally good cases where you feel you saved someone's life are rare, but so are the terrible cases that haunt you at night; and most would fall somewhere in the middle.
That said, I would also wager that the curve can be skewed towards one side or the other depending on the location, timing of your shift, or any other factors.
That said, I would also wager that the curve can be skewed towards one side or the other depending on the location, timing of your shift, or any other factors.
Having worked as a psych tech and later as an EMT in a major public hospital in Houston, you’d be surprised about how not-well paid many allied health professons are. And the janitors get to clean up the blood and hear the exact same screams or being on the receiving end of drug fueled rage attacks — and they aren’t getting paid much. If you sign up to work in what is effectively a battlefield hospital, you are exposed to the vile underbelly of society on a daily basis. And how does a union matter? It doesn’t shield you from getting attacked or vomited in by violent drunks or attacked by hysterical family members.
Let’s say a content moderator got paid $200k per year, that doesn’t change the images they see. This isn’t about money as much as it’s about the requirements and preparation required for the job.
Let’s say a content moderator got paid $200k per year, that doesn’t change the images they see. This isn’t about money as much as it’s about the requirements and preparation required for the job.
I think the diffrence is that ER personal are prepare for that kind of situations and they more or less know what to expect.
It would be great if there were a way to make content moderation more humane,
Oh I'm sure FB could think of a few ways.
A decent compensation structure might be a good place to start.
Oh I'm sure FB could think of a few ways.
A decent compensation structure might be a good place to start.
> I'm sure FB could think of
I still maintain expecting companies to do this on their own is a category error.
Companies think moderation is a cost center. Even if the moderators do the best possible job, they aren't adding value, only removing liabilities. The company is the wrong entity we should be asking for solutions to this problem. They are going to minimize costs and shift liabilities to 3rd parties (which will in turn hire independent contractors and play shell games in case they are sued).
We have to demand adversarial outside actors (government, labor unions, lawyers, non-profits, mental health professionals, etc) to start shaping the direction of the solution.
I still maintain expecting companies to do this on their own is a category error.
Companies think moderation is a cost center. Even if the moderators do the best possible job, they aren't adding value, only removing liabilities. The company is the wrong entity we should be asking for solutions to this problem. They are going to minimize costs and shift liabilities to 3rd parties (which will in turn hire independent contractors and play shell games in case they are sued).
We have to demand adversarial outside actors (government, labor unions, lawyers, non-profits, mental health professionals, etc) to start shaping the direction of the solution.
I think that you are wrong in general, but that parallel is right on point.
The root of the problem, which is rarely mentioned in these discussions, is that many people enjoying comfortable living conditions believe themselves to be “pure-blooded” and unable to deal with “unclean” things because of the possible “traumas”. (What is an ideal person free of “traumas”, anyway? A vegetable stoned on drugs?) You can easily see here the resemblance of a old time countess complaining about her “nerves” while her serfs do all the dirty work. Is there a natural law that says a poor grunt is bound to clean the junk from what you see on the website? Do it yourself.
The same happens with medical, law enforcement, and other professions. They are considered “special” jobs for “special” people (which is not true, the one who defines it like that is “special”). Then, any worries and doubts are neutralized by convenient “but it's their profession, they chose it, they are paid for it, etc.” This creates a stupid rift between “us” and “them”, and affects how both sides reason about themselves.
The resulting imbalance is routinely abused. Social networks are competing in presenting a rose-colored walled fantasy to their human cattle to increase yield — everyone can see they use “The Matrix” as a business plan. This creates a lot of shitty jobs where people waste their emotional capacity to make others pleased, not unlike in prostitution. On the other hand, media is hooking the softened consumer with the sensationalist outrage bait about maniacs, wars, corruption, presumed orgies, etc. To makes things worse, the limits to all of that are set not by some moral standards or concern about someone's well-being, as it is often implied, but by advertisement contracts. The bureaucratic machine decides what is “suitable” for the visitors. For example, this makes most of the talk around “Free The Nipple” moot: there is no undercover agents of patriarchy to battle, it's advertisers who bring money and judge what is unwanted “sexual content” (some might say it's a perfect example of capitalist oppression — cue “L'Internationale”).
Even this discussion is full of overreacting people. No, the video on your screen is not as “horrible, horrible” as experiencing something firsthand, nor the pain of the tortured can be compared to the pain of the random witnesses on the internet (if we don't count their own wild fantasies). If it happens, then it's no good to pretend it's not. Who knows, it may even make you think. All in all, humans don't show a lot of originality in what they produce, so the number of things you can expect to see is quite limited. I can only conclude by reiterating that knowledge is power, ignorance is bliss, and there's a lot of people who'd like to benefit from others' ignorance in many ways.
The root of the problem, which is rarely mentioned in these discussions, is that many people enjoying comfortable living conditions believe themselves to be “pure-blooded” and unable to deal with “unclean” things because of the possible “traumas”. (What is an ideal person free of “traumas”, anyway? A vegetable stoned on drugs?) You can easily see here the resemblance of a old time countess complaining about her “nerves” while her serfs do all the dirty work. Is there a natural law that says a poor grunt is bound to clean the junk from what you see on the website? Do it yourself.
The same happens with medical, law enforcement, and other professions. They are considered “special” jobs for “special” people (which is not true, the one who defines it like that is “special”). Then, any worries and doubts are neutralized by convenient “but it's their profession, they chose it, they are paid for it, etc.” This creates a stupid rift between “us” and “them”, and affects how both sides reason about themselves.
The resulting imbalance is routinely abused. Social networks are competing in presenting a rose-colored walled fantasy to their human cattle to increase yield — everyone can see they use “The Matrix” as a business plan. This creates a lot of shitty jobs where people waste their emotional capacity to make others pleased, not unlike in prostitution. On the other hand, media is hooking the softened consumer with the sensationalist outrage bait about maniacs, wars, corruption, presumed orgies, etc. To makes things worse, the limits to all of that are set not by some moral standards or concern about someone's well-being, as it is often implied, but by advertisement contracts. The bureaucratic machine decides what is “suitable” for the visitors. For example, this makes most of the talk around “Free The Nipple” moot: there is no undercover agents of patriarchy to battle, it's advertisers who bring money and judge what is unwanted “sexual content” (some might say it's a perfect example of capitalist oppression — cue “L'Internationale”).
Even this discussion is full of overreacting people. No, the video on your screen is not as “horrible, horrible” as experiencing something firsthand, nor the pain of the tortured can be compared to the pain of the random witnesses on the internet (if we don't count their own wild fantasies). If it happens, then it's no good to pretend it's not. Who knows, it may even make you think. All in all, humans don't show a lot of originality in what they produce, so the number of things you can expect to see is quite limited. I can only conclude by reiterating that knowledge is power, ignorance is bliss, and there's a lot of people who'd like to benefit from others' ignorance in many ways.
The case of law enforcement in US is unusual in that claims to being "special" come largely from the officers themselves - that's the whole point of the sheep/wolf/sheepdog theory that much law enforcement training is centered around these days.
Citation please? This seems like mind reading.
If you search online for "I am the sheepdog", you'll find plenty:
https://www.google.com/search?q=i+am+the+sheepdog&tbm=isch
(Note how much the "thin blue line" imagery is used in conjunction with this - that's how you know who's the target audience.)
It all ultimately comes from Dave Grossman, who's one of the most worshipped figures in modern American policing: https://www.policeone.com/police-products/training-products/...
This guy now runs the "Grossman Academy", which offers police training. It's popular enough that they have train-your-entire-PD discounts.
(Note how much the "thin blue line" imagery is used in conjunction with this - that's how you know who's the target audience.)
It all ultimately comes from Dave Grossman, who's one of the most worshipped figures in modern American policing: https://www.policeone.com/police-products/training-products/...
This guy now runs the "Grossman Academy", which offers police training. It's popular enough that they have train-your-entire-PD discounts.
Whoa, thank you
> ... make content moderation more humane.
Modest proposal: Find people who enjoy horrific content, and have them rate it. Then black-hole whatever they rate as most extreme.
I was is a dark place after 9/11, both from the trauma of that, and some personal issues. So I spent considerable time on Rotten, Something Awful, and worse. Based on that experience, many would likely do content moderation for free.
Modest proposal: Find people who enjoy horrific content, and have them rate it. Then black-hole whatever they rate as most extreme.
I was is a dark place after 9/11, both from the trauma of that, and some personal issues. So I spent considerable time on Rotten, Something Awful, and worse. Based on that experience, many would likely do content moderation for free.
Maybe not enjoy, but there are certainly people who are much less affected, or not affected at all, by it.
These media stories always focus on those who picked the wrong job for their caracter, and then make a big drama of it.
I mean, there are people cutting open other humans daily. We call them surgeons. I would puke right there and could never do that. Different people have different abilities. Simple.
These media stories always focus on those who picked the wrong job for their caracter, and then make a big drama of it.
I mean, there are people cutting open other humans daily. We call them surgeons. I would puke right there and could never do that. Different people have different abilities. Simple.
It'd be very cool if someone who thinks that this is a horrible idea (and there must be some) would explain why. Just sayin'.
That sounds like a great idea, not only to just black hole the content, but to have people who are less prone to have adverse side effects from watching such material be moderators.
Thanks for sharing that part about being in that kind of place, I have a feeling many could relate.
Thanks for sharing that part about being in that kind of place, I have a feeling many could relate.
Find people who enjoy horrific content,
Yup - these are exactly the kind of people you want to rub shoulders, and stock your company's ranks with.
Yup - these are exactly the kind of people you want to rub shoulders, and stock your company's ranks with.
Yes, I get it. Those are not nice people. Few of us would want to be friends with them. Or even work in the same building. So have them work from home, or for a standalone subcontractor.
I also get the downside of employing them. But people need to look at that stuff, even if software does most of the work. So it might as well be people who can tolerate it, and perhaps even enjoy it.
I also get the downside of employing them. But people need to look at that stuff, even if software does most of the work. So it might as well be people who can tolerate it, and perhaps even enjoy it.
Sounds like /b/ during the gore era.
They should recruit people on Reddit for this. A large number of them seem perfectly fine with this type of content.
There are dedicated communities to all these things.
I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.
There are dedicated communities to all these things.
I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.
I spent my youth on 4chan. I saw everything that's mentioned in the article and worse.
Desensitization is certainly a thing. The first few times seeing those things were shocking and disturbing. Then it got exciting. Then it got boring. I don't actively look for that content, but never having left 4chan, I still sometimes stumble upon it and it causes no big reaction. I think I could totally do that content moderation job, I have to admit some morbid curiosity about what I would stumble upon exists in me. The only content that never stopped to disturb me was violence against animals.
Talking with my "normie" friends about this I realized I am the odd one out. I wonder just how much consuming that content in my youth changed me.
Desensitization is certainly a thing. The first few times seeing those things were shocking and disturbing. Then it got exciting. Then it got boring. I don't actively look for that content, but never having left 4chan, I still sometimes stumble upon it and it causes no big reaction. I think I could totally do that content moderation job, I have to admit some morbid curiosity about what I would stumble upon exists in me. The only content that never stopped to disturb me was violence against animals.
Talking with my "normie" friends about this I realized I am the odd one out. I wonder just how much consuming that content in my youth changed me.
But are people like that people you want as your co-workers?
Serious question, I don't mean this facetiously. I think is an extremely complex question and I wonder if introducing someone who is openly comfortable with gore is dangerous to the psychological health of the company.
For example, at a previous company a co-worker publicly shared, without a trigger warning and in great detail, a very gorey thing he enjoyed watching to relax. Lots of people were extremely disturbed by this -- not disturbed by him, per se, but disturbed that he shared this without any kind of warning. I don't even resent him despite myself being pretty disturbed because what one considers normal is subjective, and if you regularly relax to this content you probably don't realise that this might not be anyone else's cup of tea. But it really makes me wonder -- what if the guy who sits next to me starts telling me that ISIS beheadings are relaxing to him?
Serious question, I don't mean this facetiously. I think is an extremely complex question and I wonder if introducing someone who is openly comfortable with gore is dangerous to the psychological health of the company.
For example, at a previous company a co-worker publicly shared, without a trigger warning and in great detail, a very gorey thing he enjoyed watching to relax. Lots of people were extremely disturbed by this -- not disturbed by him, per se, but disturbed that he shared this without any kind of warning. I don't even resent him despite myself being pretty disturbed because what one considers normal is subjective, and if you regularly relax to this content you probably don't realise that this might not be anyone else's cup of tea. But it really makes me wonder -- what if the guy who sits next to me starts telling me that ISIS beheadings are relaxing to him?
Well, I probably wouldn't invite him to my game night, but if it's his job to watch ISIS videos, it seems like he'd be well-suited, no?
It seems like if you acknowledge that: a) Terrible job exists b) Terrible job is necessary
You should also allow for a person to do terrible job without thinking that they must be a terrible person by extension. Otherwise you continue the well-trodden history of certain professions like hide-makers, executioners, or coroners being ostracized from society for doing the job we told them to do.
It seems like if you acknowledge that: a) Terrible job exists b) Terrible job is necessary
You should also allow for a person to do terrible job without thinking that they must be a terrible person by extension. Otherwise you continue the well-trodden history of certain professions like hide-makers, executioners, or coroners being ostracized from society for doing the job we told them to do.
I'm sorry if I implied that I think that people who would enjoy this work are bad people. I certainly don't feel this way; I mean, I myself have worked on a platform that plenty of Nazis use and i don't consider myself a Nazi, so it would be hypocritical of me to say as such.
I agree that I wouldn't personally spend time with said person, but my question was less about personal social responsibilities outside of work. My example I pointed out happened at work, during normal work hours, in a social space where people have inherently less control over who they surround themselves with. This is the part of it all that is fuzzy to me, and I don't have a clear answer for.
I agree that I wouldn't personally spend time with said person, but my question was less about personal social responsibilities outside of work. My example I pointed out happened at work, during normal work hours, in a social space where people have inherently less control over who they surround themselves with. This is the part of it all that is fuzzy to me, and I don't have a clear answer for.
Would they need to be co-workers beyond corporate title? Content moderation is sitting at a desk clicking through things for hours a day.
Certainly in our own, private social spaces we are free to choose whomever we socialize with, but at work that option is less available. The example I gave happened during work hours over work communication, so there is always going to be social overlap that you can't simply opt out of (the age old 'what is culture fit' question applies here, and every co-worker contributes to company culture in some way).
And while a lot of content moderators do focus purely on clicking thru things, a lot of other ones still regularly interface with the company. I used to work at Discord, where content moderation of this level (and traumatic nature) was a constant concern. As engineers you may be staffed to work on a tool to help with moderation. Or you may be working on some fancy AI to help sort and tag unsafe content. And this is only from the product engineering perspective; content moderators will have to work with customer success to create content policy, or enforce bans for unsavory behavior. So I don't think they really are siloable away from the rest of the company, and I'm not even certain that that is a humane way to treat them.
And while a lot of content moderators do focus purely on clicking thru things, a lot of other ones still regularly interface with the company. I used to work at Discord, where content moderation of this level (and traumatic nature) was a constant concern. As engineers you may be staffed to work on a tool to help with moderation. Or you may be working on some fancy AI to help sort and tag unsafe content. And this is only from the product engineering perspective; content moderators will have to work with customer success to create content policy, or enforce bans for unsavory behavior. So I don't think they really are siloable away from the rest of the company, and I'm not even certain that that is a humane way to treat them.
These niche demographics are typically into it for the wrong reasons though, which includes fantasizing about the material (murder, rape, child pornography, etc.).
There are not hordes of people on Reddit watching intensely disturbing child pornography. The gore stuff maybe (although r/wpd was banned), but still I think any normal person, even the person who was occasionally on r/wpd, would be extremely traumatized by watching anything involving children. Which to be honest is probably a lot of the content discussed here, as sick as it is.
You want the subscribers to r/wpd. The people who mod it or those who come and visit everyday.
The entire point of this is to find those who deviate enough from normal to not be as impacted.
The entire point of this is to find those who deviate enough from normal to not be as impacted.
I think there's a good chance that if someone seems perfectly fine with this content, they've already got mental health issues.
I was thinking more of 4chan moderators.
But we want these people to actually do their job
Then let them do it for free.
They might be too young for the office...
Perhaps. I would argue that there is some difference between seeing someone fall off a cliff and watching someone being tortured to death.
Then they should run sidebar ads on the gore sites. The point is that there appears to be a subset of the population that is not traumatized by this stuff. We may as well give them an opportunity to put that difference to use.
Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
There is a subset of people who seek this stuff out, yes. Whether or not it is traumatic for them does not strictly follow from that.
People come to their choices with all manner of existing problems and conflicts, and they don't always do things in their own best interest.
People come to their choices with all manner of existing problems and conflicts, and they don't always do things in their own best interest.
How do we know they're not traumatized by it?
That they seek it out doesn't mean they're not messed up.
That they seek it out doesn't mean they're not messed up.
^this. I am not arguing one way or another, because I don't see strong evidence making me lean predominantly one way. But imo, it is at least just as possible that they are seeking out that kind of content voluntarily because they are already traumatized in the first place, so this is a coping mechanism.
Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows.
But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims.
tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".
Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows.
But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims.
tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".
>Then they should run sidebar ads on the gore sites.
That would do wonders for the Facebook brand.
That would do wonders for the Facebook brand.
Have a subcontracting company do the moderation and have them run the sidebar ads.
Which is still linked to the brand - "Facebook pays $10 M to company that runs ads on gore and rape sites"
Look at how much trouble they got in when it was revealed they were using Onavo (even before acquisition).
Look at how much trouble they got in when it was revealed they were using Onavo (even before acquisition).
Meh, if that’s the worst PR they can get they’d be massively ahead of where they are now. Especially considering how massively the press narrative has turned against them overall.
But that's my point - Facebook has a massive brand problem right now, and this would just be throwing more gas into the fire.
At some point that stops being a factor. If the haters are going to hate no matter what, you actually get impunity because your actual actions stop making a difference either way.
Are moderators really meant to watch the whole video in cases like that? It seems to me like they could stop once the content of the video became apparent. Even just a fraction of a second of that sort of content could be scaring, but I think probably less scaring than watching several minutes of it...
My basis for thinking this is I've seen videos like that before and clicked away quickly. It's disturbing, but I think I'm okay.
My basis for thinking this is I've seen videos like that before and clicked away quickly. It's disturbing, but I think I'm okay.
There were pockets where the video of the Jordanian pilot being burned to death by ISIS was popular. Same with the raw video of the New Zealand shooting.
Obviously not everyone, but they are there.
Obviously not everyone, but they are there.
Most importantly, there's a big difference between seeing someone being tortured to death and frequently seeing people being tortured to death every weekday.
I'd wonder if frequents in these subreddits are typically professional and healthy people to have on a team, though (honestly no idea, never been).
It's a strange conundrum because ideally you want good people who are comfortable watching things good people rarely want to watch.
Don't have much experience with this demographic -- maybe there are more sane, emotionally healthy people watching snuff films for fun than I instinctively expect.
It's a strange conundrum because ideally you want good people who are comfortable watching things good people rarely want to watch.
Don't have much experience with this demographic -- maybe there are more sane, emotionally healthy people watching snuff films for fun than I instinctively expect.
For me it was more like watching nature documentaries, morbid curiosity and all that. There wasn't really anything I'd consider snuff on that subreddit, it was natural death, murder, and accidents, all of which are part of natural life.
I found most of the videos less offensive than extremely graphic horror movies, which ironically I have less stomach for. If anything that subreddit made me generally more aware of my surroundings and safer in my daily life.
I found most of the videos less offensive than extremely graphic horror movies, which ironically I have less stomach for. If anything that subreddit made me generally more aware of my surroundings and safer in my daily life.
As long as they have a pulse and can sit in the chair long enough to do the job they'll likely be fine. Content moderation farms, call centers and the like tend to optimize their internal processes such that they don't need to restrict their hiring to "professional and healthy people". There are plenty of people who are insane and unhealthy that can hack this sort of work.
Do you want "insane and unhealthy" people who voluntarily look at horrifying stuff deciding whether that stuff should be allowed on the platform, though?
It'd be like having a jury pool made up of serial killers.
It'd be like having a jury pool made up of serial killers.
>Do you want "insane and unhealthy" people who voluntarily look at horrifying stuff deciding whether that stuff should be allowed on the platform, though?
These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done. That's just the nature of doing an unskilled, non-laborious job for a BigCo. There will be automatic spot checks, audits, reviews, etc. It is no different than the cube farm on the next floor where they review images of documents that AI systems have flagged as possibly fraudulent. Such systems are likely in place already for their current moderators.
These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done. That's just the nature of doing an unskilled, non-laborious job for a BigCo. There will be automatic spot checks, audits, reviews, etc. It is no different than the cube farm on the next floor where they review images of documents that AI systems have flagged as possibly fraudulent. Such systems are likely in place already for their current moderators.
> These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done.
Yes, but their hiring pool tends to be "everyone".
Those procedures might not work as well if your hiring pool is explicitly confined to "the people who watch ISIS torture videos for fun". I suspect standard policies and procedures include "fire the people who get off on this stuff".
Yes, but their hiring pool tends to be "everyone".
Those procedures might not work as well if your hiring pool is explicitly confined to "the people who watch ISIS torture videos for fun". I suspect standard policies and procedures include "fire the people who get off on this stuff".
Why not? They don't benefit from it being public. They do benefit from not being fired and presumably there would be audits.
This is usually a tempting fate question but what is the worst that could happen?
They have little power to abuse and obvious accountability. If they block kitten and baby pics and allow people being tortured to death they'll get fired quickly after the inevitable shitstorm.
Insane and unhealthy or creepy doesn't neccessarily mean dangerous. Even if they got twisted pleasure from it they would be gross but not inflicting actual harm. It isn't like people are being brutally murdered to give them kicks - they aren't a source of demand.
They have little power to abuse and obvious accountability. If they block kitten and baby pics and allow people being tortured to death they'll get fired quickly after the inevitable shitstorm.
Insane and unhealthy or creepy doesn't neccessarily mean dangerous. Even if they got twisted pleasure from it they would be gross but not inflicting actual harm. It isn't like people are being brutally murdered to give them kicks - they aren't a source of demand.
> maybe there are more sane, emotionally healthy people watching snuff films for fun than I instinctively expect.
Probably not, since there are no documented instances of any snuff films ever found.
Probably not, since there are no documented instances of any snuff films ever found.
https://consortiumnews.com/2019/07/11/the-revelations-of-wik...
Thanks to Wikileaks publishing the Dutroux Dossier in 2008, we know hundreds of snuff films were recovered from the notorious pedo-rapist Marc Dutroux.
Just because the police keep those films as sealed evidence that are never published does NOT mean snuff films are an Urban Legend. I can see why the police always keep snuff films secret. Imagine what would happen if snuff films were uploaded all over the Internet? There would be mobs of angry villagers armed with torches and pitchforks descending on jails and prisons to lynch pedos. It would make harassment from QANONs look like school yard bullying.
Thanks to Wikileaks publishing the Dutroux Dossier in 2008, we know hundreds of snuff films were recovered from the notorious pedo-rapist Marc Dutroux.
Just because the police keep those films as sealed evidence that are never published does NOT mean snuff films are an Urban Legend. I can see why the police always keep snuff films secret. Imagine what would happen if snuff films were uploaded all over the Internet? There would be mobs of angry villagers armed with torches and pitchforks descending on jails and prisons to lynch pedos. It would make harassment from QANONs look like school yard bullying.
Snuff films are, fortunately, still only a subject of urban legends, since those films you mention do not meet the definition. Were those films made in order to entertain other people than the person who made them? Were the films ever distributed to any of those people? Was anyone ever actually killed in any of these movies? If the answer to any one of those three questions is “no”, then the movies, horrible as they may be, were not “snuff” films, as commonly defined. From what I can see in your reference, the first two of those things are decidedly not true, and the reference is unclear regarding the third.
(The films were, according to your reference, made only for blackmail purposes, and therefore certainly not made for enjoyment of any viewer, nor were they ever distributed to other people for their enjoyment. The reference does claim that people were killed in a specific place which was also filmed, but does not, what I can see, explicitly state that any murders were actually filmed, other than incorrectly calling all the films “snuff” films.)
(The films were, according to your reference, made only for blackmail purposes, and therefore certainly not made for enjoyment of any viewer, nor were they ever distributed to other people for their enjoyment. The reference does claim that people were killed in a specific place which was also filmed, but does not, what I can see, explicitly state that any murders were actually filmed, other than incorrectly calling all the films “snuff” films.)
I think that some of you should take a look at news websites about the drug and kidnapping cartels down in Latin America... They regularly post their own snuff films online and I can guarantee you that they're not only very real but also extraordinarily brutal. No urban legend about it.
A snuff film is a video in which someone actually IRL dies. There are about a zillion of them.
A snuff film is defined by most as at least having been made for the purpose of entertainment of others. Some include the proviso that it has to have been made for profit, and yet others demand that it has at least been sold. But regardless, not even the first level of these criteria have ever been fulfulled.
https://www.snopes.com/fact-check/a-pinch-of-snuff/
https://www.snopes.com/fact-check/a-pinch-of-snuff/
That article was last updated 2006 and does not take into account ISIS or Mexican drug cartel videos, which arguably meet your first definition and many would consider snuff.
One could hardly make the argument that those videos were made to entertain anybody for their enjoyment. Terrorists make videos to terrorize, i.e. scare, people, not for people to enjoy.
Making the argument that is at least partially made for the entertainment of other terrorists doesn't strike me as outlandish. Personally I wouldn't make that argument though.
Regardless my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
Regardless my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
> my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
And those people would simply be wrong.
I mean, let’s back up a bit. Why is it important whether or not “snuff” films exist? Because many people want to believe the common urban legends about it – i.e. that there exists an (be it ever so small) underground production and market for it, even though, by all accounts, this does not exist, and has never existed. Believing such urban legends helps people feel justifiably horrified and helps validate their cynicism; basically the same reason why many people believe many other urban legends. However, no movies fitting that particular model have ever been found to exist.
If anyone wants to to be picky about the definition and claim that some movie or other fits the definition of “snuff”, or is “considered” to fit even though it doesn’t(!), this is only because they want to be able to say “see, snuff films exist!”, and thereby validate their preexisting belief in the urban legend. But this is a motte-and-bailey fallacy; it tries to defend the urban legend (which is hard) by trying to defend a technical definition of a word to be slightly wider than it really is (which is comparitively easier). Luckily, even though it is easier to defend, this latter position is still demonstrably wrong.
And those people would simply be wrong.
I mean, let’s back up a bit. Why is it important whether or not “snuff” films exist? Because many people want to believe the common urban legends about it – i.e. that there exists an (be it ever so small) underground production and market for it, even though, by all accounts, this does not exist, and has never existed. Believing such urban legends helps people feel justifiably horrified and helps validate their cynicism; basically the same reason why many people believe many other urban legends. However, no movies fitting that particular model have ever been found to exist.
If anyone wants to to be picky about the definition and claim that some movie or other fits the definition of “snuff”, or is “considered” to fit even though it doesn’t(!), this is only because they want to be able to say “see, snuff films exist!”, and thereby validate their preexisting belief in the urban legend. But this is a motte-and-bailey fallacy; it tries to defend the urban legend (which is hard) by trying to defend a technical definition of a word to be slightly wider than it really is (which is comparitively easier). Luckily, even though it is easier to defend, this latter position is still demonstrably wrong.
I take it you haven't seen the "ISIS burns people in a cage, dance dance revolution edit/remix" video.
Those people weren’t killed for the purpose of entertainment. The initial videos were made by terrorists, to terrorize, not to entertain. Remixes aren’t “snuff” films, otherwise any compilation video on /r/watchpeopledie would qualify.
I remember talking to an FBI agent who had to watch a lot of terrible stuff. He said that at first it was very distressing and sickening, but after awhile he became desensitized to it and could eat lunch with no issues while continuing to watch it. I wonder if some people have the psychological profile that allows them to become desensitized while other people never can. Though, it probably isn't good that people have to become desensitized.
If one can become desensitized to loading people into ovens, they can certainly to pictures of the same thing.
Perhaps we could use DL to "dehumanize" the content.
For example, instead of showing one person torturing another person, a DL filter could turn the scene into two teddy bears torturing eachother.
For example, instead of showing one person torturing another person, a DL filter could turn the scene into two teddy bears torturing eachother.
> It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it.
I wouldn't assume "watches this stuff" and "is psychologically capable of tolerating this stuff" are one and the same. Those people might be actively harming themselves by watching the content. That's one thing when they're volunteering to do so (though that's an interesting conversation in and of itself) but it's quite another when Facebook is paying them to do it.
I wouldn't assume "watches this stuff" and "is psychologically capable of tolerating this stuff" are one and the same. Those people might be actively harming themselves by watching the content. That's one thing when they're volunteering to do so (though that's an interesting conversation in and of itself) but it's quite another when Facebook is paying them to do it.
Honestly, this post speaks a lot of truth, at least in my experience. I use to be subbed to /r/watchpeopledie, really just out of morbid curiosity.
One day I had an anxiety attack because I had a dream that I had a heart attack and suddenly I had a deep fear of randomly dying.
All the random deaths on that sub started to really get to me, I even started to get depressed. It became harder and harder to watch content on it. Eventually I unsubbed from it and gradually bounced back to normal.
I don't really regret it because it gave me a new perspective on life and how valuable it is. But I would be lying if I said it didn't fuck me up in a way.
One day I had an anxiety attack because I had a dream that I had a heart attack and suddenly I had a deep fear of randomly dying.
All the random deaths on that sub started to really get to me, I even started to get depressed. It became harder and harder to watch content on it. Eventually I unsubbed from it and gradually bounced back to normal.
I don't really regret it because it gave me a new perspective on life and how valuable it is. But I would be lying if I said it didn't fuck me up in a way.
I think the best counter to becoming desensitized or depressed is having an over arching sense of mission and identifying your work with a cause for good that is greater than yourself.
take ER doctors. they daily witness human trauma far worse than viewing horror show videos and images. do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering? Doctors have a personal sense of doing good in the world through healing. They witness so much trauma that they become so desensitized that non doctors often criticize them for seeming so detached and for treating human bodies like mechanistic processes. But doctors have to be somewhat detached in order to do their jobs well.
take ER doctors. they daily witness human trauma far worse than viewing horror show videos and images. do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering? Doctors have a personal sense of doing good in the world through healing. They witness so much trauma that they become so desensitized that non doctors often criticize them for seeming so detached and for treating human bodies like mechanistic processes. But doctors have to be somewhat detached in order to do their jobs well.
Quick googling yields this - https://www.webmd.com/mental-health/news/20180508/doctors-su...
It's not specifically about ER doctors. Anecdotally, I'm a lawyer, and they warned us in law school that lawyers have abnormally high rates of substance abuse and depression, and constant jokes to the contrary not withstanding, most lawyers believe they do good. Emotionally hard jobs are hard, even if they are indisputably good.
It's not specifically about ER doctors. Anecdotally, I'm a lawyer, and they warned us in law school that lawyers have abnormally high rates of substance abuse and depression, and constant jokes to the contrary not withstanding, most lawyers believe they do good. Emotionally hard jobs are hard, even if they are indisputably good.
> do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering?
Yes you do! But I would imagine it is at least somewhat offset by the fact that they are doing a lot of good: they are saving people from death while also witnessing death. I think a content moderator would struggle to feel such a strong sense of purpose.
Yes you do! But I would imagine it is at least somewhat offset by the fact that they are doing a lot of good: they are saving people from death while also witnessing death. I think a content moderator would struggle to feel such a strong sense of purpose.
I'm surprised there hasn't been a revenge movie where one of these moderators use their access to find out where these people live and start hunting them down like some sort of batman-like person. I could see a movie like this being a means of making the more general population aware of the terrible stuff content moderators have to watch.
Guns Akimbo is kind of similar but with a twist.
I did not like the movie much to be honest.
I did not like the movie much to be honest.
This is totally not what you are talking about, but the closest thing I have seen to that is the Netflix docuseries Don't Fuck With Cats. It hits on a lot of the same topics. The primary differences are that the hunters are Facebook users and not moderators and they are hunting the person to get justice and not violent revenge.
There's a show called Darknet https://en.wikipedia.org/wiki/Darknet_(TV_series) that's somewhat along these lines too, but it's very much in the horror genre.
I cant help wonder about the economic(choices) side of this around 1) Should this victimization be taken into account when punishing offenders eventually tried and 2) Is the damage done to the content moderators marginally worth keeping the marginally caught content off the platform (recall there already is automation doing the bulk of the work)...
I do think we need to think about the side effects of laws where some US citizen demands that Facebook must employ content moderations else be liable for hosting it, it doesnt result in Facebook executives, engineers et al. bearing the burden. It results in some poor person in a 3rd world country bearing the burden.
Kind of like the environmental aspect of things I would think the best investment is in total prevention of the heinous acts in the first place. ie, it's much cheaper to prevent CO2 than it is to clean it up after the fact.
I do think we need to think about the side effects of laws where some US citizen demands that Facebook must employ content moderations else be liable for hosting it, it doesnt result in Facebook executives, engineers et al. bearing the burden. It results in some poor person in a 3rd world country bearing the burden.
Kind of like the environmental aspect of things I would think the best investment is in total prevention of the heinous acts in the first place. ie, it's much cheaper to prevent CO2 than it is to clean it up after the fact.
Is this a case of trying to solve social problems with technical solutions? Maybe a platform like FB that is not run by the community that uses it simply cannot regulate users behaviour like a real social community can.
About 9 years ago, I found myself at a party consisting mostly of content moderators for The Cheezburger Network.
I walked out of there shellshocked.
I walked out of there shellshocked.
Only a human should be able to decide what other humans can and cannot see. A future where computers are responsible for "deciding" what information we consume about the world is not one I want to live in. Computers are already being used as tools of censorship, we don't need it to be expanded further.
If computers can help with spam, they can certainly help with other well-defined and well-audited types of illegal and misinformed content.
It would be worse to not invest in more automation, even malpractice.
It would be worse to not invest in more automation, even malpractice.
I hope they are tagging these videos and someone somewhere is working on a neural network to classify the new ones.
Funny how they can automatically flag copyright for videos now even when x mirrored but they can't identify these videos.
I can't believe there's so much original content for this. Maybe they could share a db of hashes of known bad videos across sites and government agencies.
I can't believe there's so much original content for this. Maybe they could share a db of hashes of known bad videos across sites and government agencies.
This story is an interesting contrast to the story of Twitter dropping users for spreading the QAnon conspiracy.
Clearly, there's near-universal agreement that some moderation is fine; only a handful of (downvoted) comments on this thread saying "The easiest way to fix this problem would be for Facebook to stop censoring posts and let users block what they don't want to see." But we clearly don't think that's a good solution in this case.
The question of whether QAnon should also be blocked is one of degree, not quality.
Clearly, there's near-universal agreement that some moderation is fine; only a handful of (downvoted) comments on this thread saying "The easiest way to fix this problem would be for Facebook to stop censoring posts and let users block what they don't want to see." But we clearly don't think that's a good solution in this case.
The question of whether QAnon should also be blocked is one of degree, not quality.
Some great undercover video of what goes on at Cognizant here, as far as bias in moderating actions is concerned:
https://www.politicalite.com/latest/facebook-employee-if-som...
In this video it is shown that regarding beheadings, Facebook made it a point to send their moderation contractors a memo that the image that was frequently reported of President Trump's face with a knife at the neck where he is being beheaded was an exception to the content policy on beheadings/violence/incitement because it was considered to be "art" by some museum somewhere in Portland - which from what I can see on TV - appears to be an extremely volatile and violent place in some parts.
There is a reference to a cartoon post of Elmer Fudd shooting a cartoon gun at Beto O'Rourke with the text "I'm here for your guns" at the top - which was not given any similar exception.
I don't think Facebook or their partners are serious about this kind of thing at all since they seem ok with advocating violence and gore when it resonates with some personal opinions.
Also noteworthy is that the name of this whistleblower garners fewer search results on Google, than the name of another whistleblower Eric Ciaramella who, if you uttered a-la Voldemort, would get you banned instantly.
Broadly, content policy at these websites seems to be a lord of the flies, unprincipled hackjob of whatever the political machinery at the organization deems worthy of steering society in the direction of their whims. Given the influence and power of Facebook and other tech platforms, this should worry people. I'm less inclined to be sympathetic to these folks and the difficulty of their jobs, given the apparent willingness to become social constructionists with the power that they wield so irresponsibly.
https://www.politicalite.com/latest/facebook-employee-if-som...
In this video it is shown that regarding beheadings, Facebook made it a point to send their moderation contractors a memo that the image that was frequently reported of President Trump's face with a knife at the neck where he is being beheaded was an exception to the content policy on beheadings/violence/incitement because it was considered to be "art" by some museum somewhere in Portland - which from what I can see on TV - appears to be an extremely volatile and violent place in some parts.
There is a reference to a cartoon post of Elmer Fudd shooting a cartoon gun at Beto O'Rourke with the text "I'm here for your guns" at the top - which was not given any similar exception.
I don't think Facebook or their partners are serious about this kind of thing at all since they seem ok with advocating violence and gore when it resonates with some personal opinions.
Also noteworthy is that the name of this whistleblower garners fewer search results on Google, than the name of another whistleblower Eric Ciaramella who, if you uttered a-la Voldemort, would get you banned instantly.
Broadly, content policy at these websites seems to be a lord of the flies, unprincipled hackjob of whatever the political machinery at the organization deems worthy of steering society in the direction of their whims. Given the influence and power of Facebook and other tech platforms, this should worry people. I'm less inclined to be sympathetic to these folks and the difficulty of their jobs, given the apparent willingness to become social constructionists with the power that they wield so irresponsibly.
Can we build better AI off of the data these moderators have generated to make their lives easier?
Can you crowdsource the moderation task to double check what the ai is flagging?
Can you crowdsource the moderation task to double check what the ai is flagging?
> Can you crowdsource the moderation task[...]?
I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
> putting that on a public platform, especially associated with the name of Facebook
Is that not what the person who originally posted the content being moderated is doing?
Is that not what the person who originally posted the content being moderated is doing?
> Is that not what the person who originally posted the content being moderated is doing?
The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
It still would open up a gore/scare platform (think LiveLeak), since only the bad content is reported. Sure, most might be tame, but it wouldn't take long for gore seekers to come.
>Can you crowdsource the moderation task to double check what the ai is flagging?
This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario.
Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts.
Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario.
Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts.
Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
Speaking as someone who has to deal with supervised learning models, human beings falsely tag positive examples as negative all the time. Merely having "average" human beings tag examples doesn't guarantee perfect recall.
Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
> Can you crowdsource the moderation task to double check what the ai is flagging?
"Facebook sent me child porn" is probably not a headline Facebook wants out there.
"Facebook sent me child porn" is probably not a headline Facebook wants out there.
> Can we build better AI
> Can you crowdsource the moderation task
When the goal is to prevent users from seeing this content, crowd-sourcing moderation defeats the purpose.
Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
> they want to see zero of it, ever
> When the goal is to prevent users from
> seeing this content, crowd-sourcing moderation
> defeats the purpose.
No system can ever stop 100% of users from seeing 100% of the material they find objectionable. Every user has their own idea of what material is beyond the pale, and even if we invent AI that is competent at blocking content, and capable of tailoring its filters per-user, it will still fail some of the time, because users are people, and people's tastes change over time.
Aside from that, the way Youtube, Reddit, Twitter et al, currently operate, a user who reports objectionable content typically waits days, weeks, or years (or forever) for the company to take action. If you give a user a little real agency, it goes a long way to mitigate their displeasure over occasional exposure to unwanted content.
What real agency does the system you propose offer? Because one down-moderation from a volunteer moderator is a drop in the bucket.
Meanwhile, if your system requires users to view beheading videos regularly, people will just migrate to a site that doesn't require that.
Meanwhile, if your system requires users to view beheading videos regularly, people will just migrate to a site that doesn't require that.
If I had to come up with a system, it would be some sort of reputation hierarchy, with a small number of paid employees at the top. Each level is responsible for auditing the levels under them.
> What real agency does the system you propose offer?
> if your system requires users to view beheading videos regularly
> people will just migrate to a site that doesn't require that.
> occasionally
Precisely. How often? Close to never.
Precisely. How often? Close to never.
Give or take a live-streamed Christchurch massacre?
Yes. One disastrous livestream in the history of the feature, with further controls added almost immediately as a result.
Neither Youtube nor Facebook currently are able to filter all these incidents. This happened after Christchurch:
https://www.bangkokpost.com/thailand/general/1853804/mass-sh...
I could swear there was also a Christchurch copy-cat attack in Europe, but I can't remember sufficient details about it to find an article. Perhaps that one wasn't streamed on FB.
I don't have encyclopedic knowledge of Facebook atrocities (I closed my account nearly a decade ago), but without vetting content before it goes live, I don't see how they will entirely prevent these videos from reaching some users.
https://www.bangkokpost.com/thailand/general/1853804/mass-sh...
I could swear there was also a Christchurch copy-cat attack in Europe, but I can't remember sufficient details about it to find an article. Perhaps that one wasn't streamed on FB.
I don't have encyclopedic knowledge of Facebook atrocities (I closed my account nearly a decade ago), but without vetting content before it goes live, I don't see how they will entirely prevent these videos from reaching some users.
I don't know that all is realistic. It's the desired unattainable goal. Almost all is the current status quo. If a service like FB were to adopt a volunteer moderator model, that track record would crash by definition because the moderators would be seeing the garbage.
(That's before we factor in unintended consequences such as the risk that a critical mass of moderators decide the garbage is signal and start passing it. It's more risk than FB wants to take on for a problem they already solve via paid employees).
(That's before we factor in unintended consequences such as the risk that a critical mass of moderators decide the garbage is signal and start passing it. It's more risk than FB wants to take on for a problem they already solve via paid employees).
I don’t think Gollum had a Hebrew letter on his back...
Right you are. It's been a while since I last read The King of the Rings.
Easier said than done
https://time.com/5739688/facebook-hate-speech-languages/
https://time.com/5739688/facebook-hate-speech-languages/
I also made a comment before I saw yours about creating a machine learning based system to automate this process, I think it must be fairly easy given the amount of data already available.
I can't help but wonder what the numbers look like on this content. One of the things Facebook should have enough data to know is how prevalent this sort of thing is.
Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?
Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?
Agreed. I would definitely love to see some sort of a basic data analysis blog post on this topic from FB, similar to how match.com used to publish data analysis posts on their blog back in the day.
I don't understand what's done with the contractors in the US after the successful lawsuit, surely all those jobs were just moved offshore? There's no mention of this at all in the article....
econcon(2)
The issue is as real as western countries dumping garbage in eastern countries just because they can pay them off.
The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.
The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.
I think in regards to India, the issue is a little more complex, and intertwined with social mores. This one sentence from the article is really important:
> According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment.
From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States.
The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish.
-----------
[1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
> According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment.
From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States.
The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish.
-----------
[1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
The awkward ethical question is would they be better off without "exploitation"? Part of the reason companies are even willing to go through the overhead is because the costs are lower.
Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".
Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".
I recall a statistic from about 10 years ago that computer forensic investigators in law enforcement burn out after two years due to the trauma of the images they are exposed to.
There's likely some confounding factors. Pressing "go" on the overpriced software tools and then entering into evidence what you find is the lowest level of work in that field so the churn is going to naturally be very high as people move up or out. The pay also isn't that great.
No, that is not the issue. Rather the issue is that even the hardest stuff on Facebook isn't remotely comparable to stuff of actual criminals, and the effort is wildly different:
- Facebook: it's violating rules? Delete, next.
- Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
- Facebook: it's violating rules? Delete, next.
- Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
This sounds very grevious, it's sad that a human being has to do this work
Radical idea: if your service requires subjecting lots and lots of workers to scarring images, the correct solution is for that service not to exist.
Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes.
“Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.
Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes.
“Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.
Any place anyone can add any user content has this problem.
Anything with an input box and a file upload would qualify as your not existing solution.
1) that seems basically fine, but also 2) non-commercial efforts run by hobbyists and gated from the general public ought to be Ok. If you want to run a PHPBB site and subject yourself to harmful garbage by letting randos write to your server, well, go nuts.
[edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
[edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
I think there’s a difference that depends on scale and privacy. If your network is private (or something more akin to p2p / messaging) this is much less of an issue, especially if you have strong blocking controls. Users moderate themselves.
I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.
I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.
It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around?
It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.
It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.
>at least in this case people are paid to do this.
I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population).
The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.
I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population).
The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.
This seems like something specific to the individual. Maybe this isn't the job for you, but there's probably enough people "made of different stuff" that could handle it? I feel like there's a subtext in this discussion that "nobody can handle it" or "you have to be a monster to work this job" but that's just not true.
Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones.
I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff.
I've talked to people who make the same claim about computer programming. They'd rather starve.
Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.
Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones.
I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff.
I've talked to people who make the same claim about computer programming. They'd rather starve.
Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.
Have you ever had a job where you have to look at pictures of mutilated children all day long?
I don't think many people are immune to PTSD, but I suppose sociopaths need jobs too. But I'm not sure you want to hire a department of sociopaths.
I don't think many people are immune to PTSD, but I suppose sociopaths need jobs too. But I'm not sure you want to hire a department of sociopaths.
Is this not the same type of thinking that’s convinced people who do all sorts of dirty but necessary jobs are “real sickos”? Jobs like homicide/sex crime investigation, fixing the sewers, exterminating pests, and so on. I would probably get PTSD if I had to investigate bleeding anuses digitally but my doctor seems to manage and I doubt she’s a sociopath.
My point is, some people simply aren’t traumatized by photos/videos, either via talent or training, and because of this they can process it enough to categorize it and keep their emotional/social/intellectual distance. That doesn’t make them sociopaths and it certainly doesn’t make them bad people.
My point is, some people simply aren’t traumatized by photos/videos, either via talent or training, and because of this they can process it enough to categorize it and keep their emotional/social/intellectual distance. That doesn’t make them sociopaths and it certainly doesn’t make them bad people.
> I would probably get PTSD if I had to investigate bleeding anuses digitally but my doctor seems to manage
A bleeding anus is just a thing that happens to people. It's a bit gross but I don't know why anyone should be traumatized by it.
The kind of content that these moderators have to deal with goes well beyond the mere abject.
A bleeding anus is just a thing that happens to people. It's a bit gross but I don't know why anyone should be traumatized by it.
The kind of content that these moderators have to deal with goes well beyond the mere abject.
You seem to be confusing any exposure with constant exposure, and also type of gross/messed up. Nothing you listed compares to content moderation in severity and volume.
Fixing a sewer is gross. Extermination is gross, not traumatic, unless you have extreme empathy for small rodents and insects. Doctors deal with very few remotely bleeding anuses on the order of one a week/month maybe versus one a minute. None of these should be called real sickos, but none would be expected to develop PTSD either.
A significantly bigger portion of the population would be traumatized and develop PTSD by the level of exposure of content moderation. While not nearly all are sociopaths, they 100% will be represented higher even without factoring in selection bias. Hiring a department of them would not be inconceivable, putting the validity of that aside.
You also make the jump from sociopath to bad person, which many I'm sure would have some qualms with as a categorical.
Fixing a sewer is gross. Extermination is gross, not traumatic, unless you have extreme empathy for small rodents and insects. Doctors deal with very few remotely bleeding anuses on the order of one a week/month maybe versus one a minute. None of these should be called real sickos, but none would be expected to develop PTSD either.
A significantly bigger portion of the population would be traumatized and develop PTSD by the level of exposure of content moderation. While not nearly all are sociopaths, they 100% will be represented higher even without factoring in selection bias. Hiring a department of them would not be inconceivable, putting the validity of that aside.
You also make the jump from sociopath to bad person, which many I'm sure would have some qualms with as a categorical.
> I don't think many people are immune to PTSD
PTSD is real but lots of people still go to war and not everyone comes back a basketcase. Most don't, in fact.
No, I've never had a job watching pictures of mutilated children all day. But like a lot of naturally curious people, I've searched out and watched bloody videos of beheadings and car accidents and whatnot. It wasn't pleasant, but it wasn't really traumatic either. If I didn't already have a much more lucrative career, I don't think I'd suffer too much as a Facebook censor.
Does that make me a sociopath? My wife and son don't think so, nor do my close friends. I'm not sure if I should feel a little offended here.
On the other hand, I may still have PTSD from those days of cold calling as a junior software engineer 25 years ago. I hate phones. Maybe people are just wired differently.
PTSD is real but lots of people still go to war and not everyone comes back a basketcase. Most don't, in fact.
No, I've never had a job watching pictures of mutilated children all day. But like a lot of naturally curious people, I've searched out and watched bloody videos of beheadings and car accidents and whatnot. It wasn't pleasant, but it wasn't really traumatic either. If I didn't already have a much more lucrative career, I don't think I'd suffer too much as a Facebook censor.
Does that make me a sociopath? My wife and son don't think so, nor do my close friends. I'm not sure if I should feel a little offended here.
On the other hand, I may still have PTSD from those days of cold calling as a junior software engineer 25 years ago. I hate phones. Maybe people are just wired differently.
> But I'm not sure you want to hire a department of sociopaths.
Why not, if they do the work well?
And anyway, they'd be working from home, these days.
Why not, if they do the work well?
And anyway, they'd be working from home, these days.
I get what you're saying, but you would have to look for traits and develop processes to make an environment that people could do well in.
Big companies like Facebook (although they are not alone) don't do that. That costs money. They use disposable workers, like on-shore or off-shore contractors, and do industrial-style quotas/piecework methods to maximize productivity.
When you hire like that, you'll have a normal distribution of people. On the left there will be pre-broken people who get pleasure from watching decapitation videos and nasty shit, on the right you'll have people who are traumatized and broken by the work, and in the middle you have people who will cope for some period of time, and burn out.
Big companies like Facebook (although they are not alone) don't do that. That costs money. They use disposable workers, like on-shore or off-shore contractors, and do industrial-style quotas/piecework methods to maximize productivity.
When you hire like that, you'll have a normal distribution of people. On the left there will be pre-broken people who get pleasure from watching decapitation videos and nasty shit, on the right you'll have people who are traumatized and broken by the work, and in the middle you have people who will cope for some period of time, and burn out.
It doesn't seem to be specific to an individual when there are class actions lawsuits and many people reporting to suffer from it.
There exist jobs, like slaughterhouse knockers, which are linked to causing PTSD and increased criminal behavior. It's not far fetched to believe these moderator jobs do too.
There exist jobs, like slaughterhouse knockers, which are linked to causing PTSD and increased criminal behavior. It's not far fetched to believe these moderator jobs do too.
Yeah, and they made an interesting point about how these problems are compounded by the lack of mental health awareness and the difficulties of defending mental health trauma in a court of law in countries like India and the United States. I didn't think of that before.
Before I accept this is a serious problem, I'd need to see some statistics. The article is just a bunch of anecdotes. If this job is hazardous, quantify it?
It wasn't that long ago HN was abuzz about large numbers of suicides at Foxconn... until someone realized that the suicide rate was actually slightly below the community average. I simply don't trust arguments without statistics anymore.
It wasn't that long ago HN was abuzz about large numbers of suicides at Foxconn... until someone realized that the suicide rate was actually slightly below the community average. I simply don't trust arguments without statistics anymore.
> wish there was a brain cleaning solution
'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
I wonder if showing a video/gif from reddit.com/r/eyebleach between viewings of disturbing content could reduce the toll it takes.
It sounds too simple to work, but worth a shot.
It sounds too simple to work, but worth a shot.
MySpace had to deal with this. They also had content moderators who were kept in a separate room at HQ who had to view all manner of flagged videos/photos. The turnover was really high even though at the time, MySpace had the kind of cool factor that developers would pick up women at bars with the promise of taking them into the offices after hours (which is a whole other weirdness and the corporate higher ups had to ban the practice after a drunk couple were injured after falling off a table one Saturday night). I don't know that there was anything acting at this kind of scale before MySpace.
Facebook was a zoo in its early years. Basically a frat house. Google was also a complete boys club and full of misconduct. But you have to be a complete nerd to take your date to your office, even if it’s a nice office.
It really wasn't a nice office. My wife worked there from 2005–2008 and I'd occasionally pick her up after work. I think the big thing was just that it was MySpace and at the time, it was the epitome of cool. They even had a record label https://en.wikipedia.org/wiki/MySpace_Records Unlike Facebook, MySpace was heavily tied into the entertainment industry, especially music, but to a lesser extent film and stand-up comedy which probably contributed to the cool factor. The engineering culture was pretty heavily bro-grammer in a way I've never seen anywhere else.
I think it's fair to say the sheer quantity has changed since Facebook first came around (FB predates smartphones for one!)
I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
FB does not predate smartphones, I was using a Treo 180 with a browser and gprs data service in 2002. Email client, ssh, lots of stuff.
> I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
That's an interesting point, and has me wondering something.
If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level?
What about a US/Europe based agency that did this thing onshore as a contract deal?
That's an interesting point, and has me wondering something.
If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level?
What about a US/Europe based agency that did this thing onshore as a contract deal?
YouTube tried doing something like this with YouTube Heroes and got tons of backlash for it. Then YouTube tried automating everything away and I'm guessing now people wish they had implemented something like YouTube Heroes instead.
For a different reason. The community was generally upset about Heroes for the potential of large groups to mass flag videos they disagreed with politically, something that was already a growing problem at the time.
Disclosure: I work for but don't represent Google. This is my own opinion.
Disclosure: I work for but don't represent Google. This is my own opinion.
Facebook already hires moderators inside the US via agencies, and they have their own problems:
https://www.theverge.com/2019/2/25/18229714/cognizant-facebo...
As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
https://www.theverge.com/2019/2/25/18229714/cognizant-facebo...
As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
I'd consider volunteering to moderate a particular geographical area, when I was a student Facebook constantly took down the student confessions page I eventually ended up helping to run for the vaguest reasons, while letting actual hate posters run amok. I know I would say that, but we would genuinely get taken down for jokes that wouldn't even make sense out of the tiny university I attended. It was immensely frustrating having to build up the userbase again and again because the moderators of Facebook just weren't/aren't terribly good at their jobs. I'd happily have taken on actual moderation duties rather than had to constantly appeal and get a blank, inhuman "declined" message.
I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.
I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.
I have moderated several forums over the course of 15+ years, and I have never come across this kind of content. But then again, all of them were very specialized in their topics. I still am convinced that we would be better off if we went back to small forums for people sharing common interests rather than having a huge platform. People can police their small group of peers much more easily. It works.
Communities grow - I think this kind of strategy only works on a small scale. If it was adopted en masse, it becomes self defeating. Suggesting everyone change groups the moment their current group reaches a certain size ignores the greater point, which is that these large groups are going to exist/emerge regardless.
It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around?
I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered.
https://www.pewresearch.org/internet/2015/10/08/social-netwo...
I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered.
https://www.pewresearch.org/internet/2015/10/08/social-netwo...
Right but the difference is the sheer amount of money involved and the politicization that comes with that. If Facebook was worth 100m nobody would care. If facebook only has 10m users nobody would care. It's the scale that completely breaks the legal and political frameworks the our society leans on.
I'd say another difference is the percentage of your attention is being devoted to this kind of thing. Presumably reddit or phpbb mods aren't spending 40 hr/wk looking at gore.
Wouldn't that suggest then that the "ideal" way to do it is a very part time job then with high volume? Like say 5 hours a week. It would be a scheduling and logistical nightmare of course but it would give more "dispersion time" for trauma.
I'd be interested to see if the tooling around moderation could also be improved. For example, could the images start heavily blurred, with a circle of clarity that opens when you click. Or something like that, which would harm throughput, but be more humane.
This is a very interesting idea. It could reveal only a portion and see how often that was enough for the moderator to pass judgement.
tbh I've been a moderator on various old school forums for over a decade, and I've never had to deal with anyone posting any gore at all. I think they're usually too tight knit and reputation based for anyone to feel like they could get away with it. it's not like people posting gore on facebook are doing it to mess with the moderators, that's just a side effect
From my time observing volunteer groups, it's also about leverage and force multipliers.
I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members.
It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members.
It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
Do you know what community organizing is? How movements accomplish change? Everything does not have some easy optimization method, only in silicon valley does everyone think it does.
edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
I think the point is that, if I as an activist am aiming to maximize the number of content moderators treated well, the highest leverage way to push for that is getting people mad at Facebook. There may be a preexisting system of unpaid content moderators who are supported even less, but that's much harder for anyone to address.
I worked with a group that got 100 volunteers at each event who did the work while the regular members provided coordination and resources. I am trying to do something similar on my own now and learning in a very visceral way just how important spreading that load out is.
But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all.
I think I would say that both are necessary but insufficient.
But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all.
I think I would say that both are necessary but insufficient.
Another factor is the working conditions at Cognizant. The difference between Reddit moderators and Cognizant employees is that the former are allowed to have bathroom breaks and individual agency. The latter not only have to deal with fairly oppressive workplace policies, but they also rely on this work to live.
Yeah, monopolies can do that.
edit: https://qz.com/1704143/the-antitrust-case-against-facebook/
edit: https://qz.com/1704143/the-antitrust-case-against-facebook/
I suspect there is a certain interest group that has emerged and grown around this subject that has been elevating and promoting this issue in the public's mind. It would not even surprise me one bit that especially overseas contractors, because there are likely fewer penalties/prohibitions, are paying people to flood, e.g., fb with things like beheading videos in order to produce a cause for their services. It may be cynical, but like I said, it would not surprise me one bit.
Very few forums had even a fraction of the number of members as Facebook has users and the amount of content is increased again. Also the type of content they're required to moderate has changed most forums before facebook were largely text and video is a big part of facebook. It's easy to see how video and photo content is more traumatizing than the largely text content of forums.
Given auto translate works reasonably well these days, how much commonality does Negative Content over different regions and languages have?
Anyone know? Is it like 10%? 80%?
There must be, by now, a whole lot of data on whats getting flagged region/language wise.
Maybe we need a Cloudflare for Content.
Anyone know? Is it like 10%? 80%?
There must be, by now, a whole lot of data on whats getting flagged region/language wise.
Maybe we need a Cloudflare for Content.
Depends on the language. In my experience Vietnamese -> English is horrid and comes out as mostly garbage. Interestingly, the reverse has been better from what my Vietnamese friends tell me.
I can't even barely read this article before becoming extremely sad for these people and the individuals (especially children) who were abused in these videos. I hope everyone involve gets the healing they need.
It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.
It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.
In my opinion this is an argument in favour of shutting Facebook (and every other large social network) down. This sort of work is damaging and abusive, and nobody should have to endure it just so we can have social networks.
I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.
I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.
That’s a non-solution.
What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible.
You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma.
In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand.
A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields.
Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible.
You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma.
In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand.
A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields.
Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
I’m actually not proposing shutting down any website that allows uploaded content, just large / public sites that require this sort of moderation. Not every site gets this stuff uploaded to it. The more private the network, the less need for this kind of company-led moderation.
As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job.
There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.
As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job.
There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.
Think like a legislator. How do you write this regulation?
> [shut down] just large / public sites that require this sort of moderation
Let’s say I start a restaurant review website that allows comments and photos to be uploaded. It does modest business for a while, I now have 50 employees. I’m following the law because my site isn’t big enough to violate this “no user content for big prominent websites” law.
Soon, it becomes big, like a major competitor to Yelp, and I’ve got 1,000 employees. But suddenly, this new law kicks in that says that I have to stop accepting uploads because my site is too high profile. Now, I lay everyone off and go out of business.
This just isn’t a workable solution, at least not in the particular way you’re proposing it be constructed.
And really, you’re asking the second largest advertiser on the web (Facebook), a Fortune 50 company, to just pack up its bags and shut down.
It’s not like I love Facebook or anything, but I’m sure their 45,000 employees wouldn’t be happy about that.
> [shut down] just large / public sites that require this sort of moderation
Let’s say I start a restaurant review website that allows comments and photos to be uploaded. It does modest business for a while, I now have 50 employees. I’m following the law because my site isn’t big enough to violate this “no user content for big prominent websites” law.
Soon, it becomes big, like a major competitor to Yelp, and I’ve got 1,000 employees. But suddenly, this new law kicks in that says that I have to stop accepting uploads because my site is too high profile. Now, I lay everyone off and go out of business.
This just isn’t a workable solution, at least not in the particular way you’re proposing it be constructed.
And really, you’re asking the second largest advertiser on the web (Facebook), a Fortune 50 company, to just pack up its bags and shut down.
It’s not like I love Facebook or anything, but I’m sure their 45,000 employees wouldn’t be happy about that.
This is not just a facebook problem though. Taking your proposal to its logical conclusion suggests that the internet should have no user created content on it - or am I missing some possible middle ground?
How on earth does the size make any difference to the supposed problem? Even if we somehow had a stable and viable size capping system to whatever nebulous concept of "too big" there would still be people exposed to the same content. Not to mention in the US said restrictions would get a "haha no" from the courts on First Ammendment grounds against arbitrary limitations on speech by source.
Why not just require social networks to verify identity and verify users are 18+ before allowing them to make an account? Furthermore, maybe even introduce a direct, easy-to-understand, hard-to-misuse law, enforceable locally (e.g. in the US it be a state law) that says committing fraud to join a social network is a misdemeanor with a $1,000 fine. Facebook can then report fraudsters to local police.
People would be less inclined to post this type of content to Facebook if it their account was very connected to their real identity, if they were 18+, and if there was meaningful punishment for making fake accounts or accounts with stolen identities.
People would be less inclined to post this type of content to Facebook if it their account was very connected to their real identity, if they were 18+, and if there was meaningful punishment for making fake accounts or accounts with stolen identities.
This hasn't worked the previous times it has been tried. https://anildash.com/2020/07/21/a-federal-blue-checkmark-won...
I wonder if prosecuting people who post horrible stuff would have a deterrence effect, though.
I wonder if prosecuting people who post horrible stuff would have a deterrence effect, though.
Every time you use Facebook, see an advertisement there, or click it, every time you share content on Facebook and get others to engage with that platform, you are contributing to a platform that is directly responsible for human psychological harm, in many different ways.
Same for Twitter, and for Reddit, and Instagram... and probably TikTok.
I don’t believe the use of these platforms can be considered ethical.
Same for Twitter, and for Reddit, and Instagram... and probably TikTok.
I don’t believe the use of these platforms can be considered ethical.
Does this logic also apply to commenting on hacker news? Or is it specifically the platforms you mentioned?
What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
> What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers.
Here’s a few solutions:
Limit the number of people that a person can be friends with to 150, the Dunbar number.
Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post.
Charge a fee for access.
These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers.
Here’s a few solutions:
Limit the number of people that a person can be friends with to 150, the Dunbar number.
Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post.
Charge a fee for access.
Why is size the determining factor, though?
Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
Size is not the determining factor. It's a single factor that highly influences the value in posting damaging content. It's clear from the rise of influencers on YT, IG, Twitter, etc -- as well as the prolific use of these platforms by terrorists and other malefactors -- that being able to reach many people with a single post is a significant driver of people to post content.
My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required. There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both. Of course, these are two suggestions which are antithetical to the ad-riddle, growth-driven social network model, so there is no way they would ever be implemented.
I should add another solution to my wish-list: remove ads entirely.
My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required. There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both. Of course, these are two suggestions which are antithetical to the ad-riddle, growth-driven social network model, so there is no way they would ever be implemented.
I should add another solution to my wish-list: remove ads entirely.
> My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required.
But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
> There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both.
Your suggestions can be implemented on smaller websites as well as larger ones.
It seems like your issue is with the ease of access to social media. Thus your solution is to limit access. However, I don't think there's a reasonable implementation of your idea that could work. Nor do I think any kind of social media is going to want to go that route. It doesn't matter if the site is funded by ads or is trying to grow. The point of these sites is to share content. People don't want to limit that.
Also, breaking large social media sites into smaller ones makes it much more difficult to deal with troublesome individuals. Right now, it only takes a few bans before you're blocked from most mainstream sites. If we had, say, 10× the number of sites, that's 10× the number of bans required to get these individuals out of the system and 10× the number of moderators who had to look at their content before banning them.
I'm not saying there's no solution here, but I think people are misidentifying the problem. We all want simple answers and simple solutions even when there aren't any.
But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
> There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both.
Your suggestions can be implemented on smaller websites as well as larger ones.
It seems like your issue is with the ease of access to social media. Thus your solution is to limit access. However, I don't think there's a reasonable implementation of your idea that could work. Nor do I think any kind of social media is going to want to go that route. It doesn't matter if the site is funded by ads or is trying to grow. The point of these sites is to share content. People don't want to limit that.
Also, breaking large social media sites into smaller ones makes it much more difficult to deal with troublesome individuals. Right now, it only takes a few bans before you're blocked from most mainstream sites. If we had, say, 10× the number of sites, that's 10× the number of bans required to get these individuals out of the system and 10× the number of moderators who had to look at their content before banning them.
I'm not saying there's no solution here, but I think people are misidentifying the problem. We all want simple answers and simple solutions even when there aren't any.
> But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
I’m not sure I follow. A site like Facebook already has mechanisms for detecting multiple accounts, and in any case, multiple accounts don’t mean that there are more “places” to post. It’s still Facebook.
I don’t propose breaking Facebook into smaller websites. I propose limiting the reach of a single individual on those sites. Basically, normalizing the localness of social media to be more meaningful.
As well, I think Facebook and other platforms need to reckon with the consent when it comes to posting images of people, in general. That’s why I suggested limiting the posts to only those of friends who consent.
I agree that no one wants this.
I’m not sure I follow. A site like Facebook already has mechanisms for detecting multiple accounts, and in any case, multiple accounts don’t mean that there are more “places” to post. It’s still Facebook.
I don’t propose breaking Facebook into smaller websites. I propose limiting the reach of a single individual on those sites. Basically, normalizing the localness of social media to be more meaningful.
As well, I think Facebook and other platforms need to reckon with the consent when it comes to posting images of people, in general. That’s why I suggested limiting the posts to only those of friends who consent.
I agree that no one wants this.
And again, you run into the problem where nobody wants their website to work like that. In fact, limiting the person's audience really only works with facebook's system where people post content specifically to their friends. Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content. You're specifically posting in a public space that everyone can see. That's not a social media thing. That's just an internet thing where most things are visible to everyone.
And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
>And again, you run into the problem where nobody wants their website to work like that.
Cancer patients don't want chemo, but it's better than dying, some might say.
>Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content.
Right, which is why those sites (besides HackerNews) would require slightly different solutions.
Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes. Verify accounts. Limit posting.
YouTube: Not sure. This one is video-forward, probably the most difficult problem in terms of bad content. Definitely remove the algorithm for targeting people based on interest, though.
>That's not a social media thing. That's just an internet thing where most things are visible to everyone.
This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information. Pre-social media, to get your idea out you had to build a website. There was friction. The earl y web had little moderation because you really had to go searching for bad stuff.
>And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Dividing the problem up is a strategy that I propose lessens the impact to both the users (because content can't spread as fast) and the moderators (because there will be less content to moderate).
Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
Cancer patients don't want chemo, but it's better than dying, some might say.
>Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content.
Right, which is why those sites (besides HackerNews) would require slightly different solutions.
Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes. Verify accounts. Limit posting.
YouTube: Not sure. This one is video-forward, probably the most difficult problem in terms of bad content. Definitely remove the algorithm for targeting people based on interest, though.
>That's not a social media thing. That's just an internet thing where most things are visible to everyone.
This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information. Pre-social media, to get your idea out you had to build a website. There was friction. The earl y web had little moderation because you really had to go searching for bad stuff.
>And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Dividing the problem up is a strategy that I propose lessens the impact to both the users (because content can't spread as fast) and the moderators (because there will be less content to moderate).
Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
> Cancer patients don't want chemo, but it's better than dying, some might say.
What good is chemo if your patients refuse it?
> Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes.
This doesn't stop people from posting disturbing content, and it destroys reddit's entire model. This solution doesn't work.
> Verify accounts.
In what way? Do you want to abolish anonymity? Because I can tell you right now that most of us aren't interested in such a solution.
> Limit posting.
In what way? Throttling posting speed? Sure if you just lessen the amount of content overall there will also be less disturbing content... but there's still disturbing content that needs to be moderated.
> This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information.
The default state of the internet is publicly viewable information. Thus it's an inherent feature that you can access almost anything out there.
> Pre-social media
As far as I'm aware, there was no such time. There have almost always been bulletin boards, forums, chat rooms, image/file sharing sites, and other forms of social media.
> there will be less content to moderate
Dividing up the content doesn't reduce the amount of it that needs to be moderated any more than cutting a cake reduces the amount of cake.
> Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
This just shuts people out according to income bracket. It reduces the total population, but I doubt it helps reduce the fraction of that population that are interested in posting and sharing disturbing content. It's just throwing the baby out with the bath water.
What good is chemo if your patients refuse it?
> Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes.
This doesn't stop people from posting disturbing content, and it destroys reddit's entire model. This solution doesn't work.
> Verify accounts.
In what way? Do you want to abolish anonymity? Because I can tell you right now that most of us aren't interested in such a solution.
> Limit posting.
In what way? Throttling posting speed? Sure if you just lessen the amount of content overall there will also be less disturbing content... but there's still disturbing content that needs to be moderated.
> This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information.
The default state of the internet is publicly viewable information. Thus it's an inherent feature that you can access almost anything out there.
> Pre-social media
As far as I'm aware, there was no such time. There have almost always been bulletin boards, forums, chat rooms, image/file sharing sites, and other forms of social media.
> there will be less content to moderate
Dividing up the content doesn't reduce the amount of it that needs to be moderated any more than cutting a cake reduces the amount of cake.
> Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
This just shuts people out according to income bracket. It reduces the total population, but I doubt it helps reduce the fraction of that population that are interested in posting and sharing disturbing content. It's just throwing the baby out with the bath water.
I think this is pushing it a bit. By similar logic, paying taxes to, or simply being a member of the US (or almost any country) is contributing to a platform that is directly responsible for human harm.
>paying taxes to, or simply being a member of the US (or almost any country) is contributing to a platform that is directly responsible for human harm.
I agree with that. But I also assert that people have an ethical obligation to opt-out of systems that cause harm when / if they are able, or at least advocate to change them.
I don't have the ability to change Facebook. But I have the ability to opt out of it, though my actions on the Internet, and the ability to advocate for others to do the same. So, here I am.
I agree with that. But I also assert that people have an ethical obligation to opt-out of systems that cause harm when / if they are able, or at least advocate to change them.
I don't have the ability to change Facebook. But I have the ability to opt out of it, though my actions on the Internet, and the ability to advocate for others to do the same. So, here I am.
You are commenting on one of those platforms right now.
HackerNews is a limited-focus link aggregation and comment platform. It doesn't require the kind of moderation that larger scale, broad-focus social media platforms do.
I don't envy the work that Dan and Scott have to do in the slightest, but I don't think they'll end up with PTSD from it. At least, that's what I gathered when I read "The Lonely Work of Moderating Hacker News"[0], especially this description of it: "Pressed to describe Hacker News, they do so by means of extravagant, sometimes tender metaphors: the site is a “social ecosystem,” a “hall of mirrors,” a “public park or garden,” a “fractal tree."
[0]http://archive.is/bbzan
I don't envy the work that Dan and Scott have to do in the slightest, but I don't think they'll end up with PTSD from it. At least, that's what I gathered when I read "The Lonely Work of Moderating Hacker News"[0], especially this description of it: "Pressed to describe Hacker News, they do so by means of extravagant, sometimes tender metaphors: the site is a “social ecosystem,” a “hall of mirrors,” a “public park or garden,” a “fractal tree."
[0]http://archive.is/bbzan
So you think the only ethical sites are the ones that operate a small scale?
I can't say for sure, since I don't know every site. But I would posit that ethical operation is inversely correlated with user base size.
[deleted]
This argument applies to any platform that lets users upload videos. There is nothing wrong with that kind of site in itself.
social media is a mirror--you see what you want in it. for every horrifying or hyperviolent act or outrageous scandal, you can find the most joyous and beautiful depiction.
have you ever seen foxes laughing? they sound like babies. without social media, i never would have known that, nor experienced one minute of happiness while watching it.
https://twitter.com/mollyfprince/status/1277601602866761729
have you ever seen foxes laughing? they sound like babies. without social media, i never would have known that, nor experienced one minute of happiness while watching it.
https://twitter.com/mollyfprince/status/1277601602866761729
What I am wondering, and this is probably a dumb question, is why this has not been automatized? Can't content moderation be done with modern and strong machine learning based systems? There must be plenty of training data on this, and just like a spam filter which does not require humans in most cases, this should also be automatable. Why is it not?
It has been. Most of these sites catch a ton of images and video automatically when they are similar enough to prior known content.
That doesn't matter from a staff point of view though. You have a queue to work through. You'll be putting in an 8 hour day dealing with the stuff the system doesn't catch. The automation just means they don't need as much staff.
That doesn't matter from a staff point of view though. You have a queue to work through. You'll be putting in an 8 hour day dealing with the stuff the system doesn't catch. The automation just means they don't need as much staff.
Yep. One way to view automation with machine vision / perception is that it can cover ~85-95% of the true positives.
You're still going to get false positives and false negatives that need human review, and at a scale of Facebook, that's a lot of humans.
You're still going to get false positives and false negatives that need human review, and at a scale of Facebook, that's a lot of humans.
I am just hoping that those currently uncovered cases will be "milder or more nuanced" cases that will be less damaging to the psyche of human moderators and will, once labeled correctly, improve the coverage rate of automated moderators.
The issue is that this is adversarial in nature. Certainly there are ML systems catching content before moderators.. however since it is adversarial people are pivoting to circumvent the automated systems.
>this is probably a dumb question
Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again.
If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again.
If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
I am guessing that by now plenty of data has already been generated which could be used to train future systems..
I'm guessing you don't actually understand the complexity involved.
How does it differentiate between normal ranting and hate speech? How is hate speech classified in the US vs Saudi Arabia? How does it tell the difference between someone asking a child innocent questions and asking them sexually-related questions on camera? Does the algorithm get trained to flag videos about depression that might lead to suicide, or does it say they're supporting getting help FOR depression and leave the video?
What about subjects that aren't already in the corpus of "flag these naughty things"? You still have to get a human to look at those; most likely a data scientist who knows what the algorithm is doing and what needs to be done to correct the training. Machine learning as-is will not get us there, so in the meantime the only option is moderation by people described in the article. It can be outsourced elsewhere, but it's just shifting the responsibility to a different subset of people.
How does it differentiate between normal ranting and hate speech? How is hate speech classified in the US vs Saudi Arabia? How does it tell the difference between someone asking a child innocent questions and asking them sexually-related questions on camera? Does the algorithm get trained to flag videos about depression that might lead to suicide, or does it say they're supporting getting help FOR depression and leave the video?
What about subjects that aren't already in the corpus of "flag these naughty things"? You still have to get a human to look at those; most likely a data scientist who knows what the algorithm is doing and what needs to be done to correct the training. Machine learning as-is will not get us there, so in the meantime the only option is moderation by people described in the article. It can be outsourced elsewhere, but it's just shifting the responsibility to a different subset of people.
Certainly there is more complexity than spam filters, and cultural differences, subtle nuances which are hard to train a system to auto-classify; things that need to be generalized to new cases that arises- that may be wrongly classified in the beginning, etc. Nevertheless such systems can be built from simple cases to more difficult ones step by step and gradually reduce the demand on humans to perform these tasks. Based on some other comments on this thread, I realize FB is already using these methods mostly in languages with plenty of data, and someone also in this thread has made an interesting comment about using machine translation to apply these systems to other languages. I think this will be less of a problem in the near future. (edited)
It's a really hard problem to automate. You can scan photos against a list of hashes to sort out known bad / illegal content, but new content is generated every single day. Also YouTube has increased the amount of auto-moderation but that leads to many legitimate videos being taken down, with little recourse for those legitimate users. Some examples:
https://news.ycombinator.com/item?id=22487403
https://news.ycombinator.com/item?id=5068626
https://news.ycombinator.com/item?id=18742640
https://news.ycombinator.com/item?id=19065859
https://news.ycombinator.com/item?id=22487403
https://news.ycombinator.com/item?id=5068626
https://news.ycombinator.com/item?id=18742640
https://news.ycombinator.com/item?id=19065859
What everyone else said, but also the presence of false positives means there'll be appeals. So you have a choice or ignoring the appeals and being accused of censoring people, or processing them - and that tends to need a person.
Not to mention the fact, that we'd be subjecting AIs to the worse that humanity has to offer. This is the kind of stuff that creates Ultrons or Skynets who decide that humanity is trash and that it ought to be wiped out. Lol.
It's not automatable at all, for example Facebook's guidelines change by the day and sometimes by the hour, and many cases are not able to be analyzed by machine as the error rate is too high.
Perhaps because the training of these systems requires human supervisors?
well, just as in spam filters, some supervisors would be necessary to automate things in the beginning, but eventually this should be less needed?
Yes, one would hope so. My guess is that we're still in the data-collection phase. It's probably a tough problem because you want the number of false-negatives to be extremely small.
It also sounds (from other articles) like people are seeing specific clips multiple times. Surely a bit of semi-AI filtering should be able to blacklist the regular stuff
Perhaps it's a solution to use brainwave headsets. So whenever the viewer would get too much negative stimulation, the video would stop and the viewer could take a break.
The headset could also put a mark into their performance review for failing to develop a sufficient coping strategy.
Is Facebook worth it? Is it worth the trauma?
if it wasn't worth it, people would not do such jobs and people would not use Facebook, so apparently for a lot of people it is worth it.
This is also relevant: https://www.theguardian.com/technology/2017/jan/11/microsoft...
I read somewhere long ago that there have been incidents of FBI employees tasked with reviewing child pornography [1] who saw so much that they became desensitized or even aroused by it. Does anyone know if this is well-documented, or was this an exception rather than a common occurence?
[1] https://www.fbi.gov/history/famous-cases/operation-innocent-...
[1] https://www.fbi.gov/history/famous-cases/operation-innocent-...
There are some things I will explicitly avoid in projects I'm working on. One of them is allowing picture or video uploads to users if the service is free. There's no low-risk, cost effective solution for moderating it, and in my country we also don't have safe-harbour type laws, so any content on the platform is your responsibility.
I had considered using services like Google Cloud Vision or other services for explicit image detection anyway.
Most websites it seems when you upload an image it's immediately available, I always wonder if it goes through some basic moderation system or just waits to get reported.
I've done tagging jobs on MTurk before it's weird... seeing random people's images.
Most websites it seems when you upload an image it's immediately available, I always wonder if it goes through some basic moderation system or just waits to get reported.
I've done tagging jobs on MTurk before it's weird... seeing random people's images.
I would wager most wait for things to get reported rather than be pre-emptive about it. I wonder if MTurk could ever be fast enough for an approval process on something like Instagram.
I think vetting users rather than content is probably the most efficient method of community curation though. If you have an approval process or other ways to validate that users aren't nefarious then you would reduce your workload by a bunch. Most bad images seem to come from burner accounts.
Unfortunately the only thing that comes to mind is a social credit as a service style system and maybe that's not where we want to steer this ship...
I think vetting users rather than content is probably the most efficient method of community curation though. If you have an approval process or other ways to validate that users aren't nefarious then you would reduce your workload by a bunch. Most bad images seem to come from burner accounts.
Unfortunately the only thing that comes to mind is a social credit as a service style system and maybe that's not where we want to steer this ship...
Yeah the tagging jobs I did were weird like "find the baby" haha... And other obvious tagging videos.
I can get from the user side about the burner accounts. I feel a physical response when I get denied to post a first post say in a sub and I'm like "Excuse me..." But yeah that's definitely discouraging/limiting to stick around before contributing.
I can get from the user side about the burner accounts. I feel a physical response when I get denied to post a first post say in a sub and I'm like "Excuse me..." But yeah that's definitely discouraging/limiting to stick around before contributing.
how do you become a Facebook moderator?
If you haven’t seen The Cleaners, I highly recommend it.
https://thoughtmaybe.com/the-cleaners/
https://thoughtmaybe.com/the-cleaners/
This quote from the article
“It gets to a point where you can eat your lunch while watching a video of someone dying. … But at the end of the day, you still have to be human.”
reminded me of a similar article published by WIRED six years ago [1].
Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue.
“Oh fuck! I've got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos. “I didn't want to look back and say I became so blasé to watching people have these really horrible things happen to them that I'm ironic or jokey about it.”
[1] https://www.wired.com/2014/10/content-moderation/