A Facebook content moderator’s resignation note(twitter.com)
twitter.com
A Facebook content moderator’s resignation note
https://twitter.com/rmac18/status/1382366931307565057
98 comments
Agreed - Massive human cost. The sanitation and police workers of the internet being directly exposed to radiation.
There is not free lunch.
Side note = firms increasingly see their policy divisions as responding to government and political pressure, not content moderation.
The next couple of years in the policy space are going to such for employees, as govts demand more power and oversight from SM firms.
There is not free lunch.
Side note = firms increasingly see their policy divisions as responding to government and political pressure, not content moderation.
The next couple of years in the policy space are going to such for employees, as govts demand more power and oversight from SM firms.
> and A.I. is not yet able to do the job.
AI won't be able to do the job until we manage to digitize brains. Stuff like sarcasm, irony, fair use or "dog whistling" (e.g. using "globalists" as a replacement for "jews") is something current AI's can't grasp at all, and even a digitized brain will struggle with these concepts - not to mention the ethical concerns that would come with digitizing a brain. Is it morally acceptable to torture a virtual brain that, in contrast to a human, can't even leave? To reset it to default state when it's too damaged?
AI won't be able to do the job until we manage to digitize brains. Stuff like sarcasm, irony, fair use or "dog whistling" (e.g. using "globalists" as a replacement for "jews") is something current AI's can't grasp at all, and even a digitized brain will struggle with these concepts - not to mention the ethical concerns that would come with digitizing a brain. Is it morally acceptable to torture a virtual brain that, in contrast to a human, can't even leave? To reset it to default state when it's too damaged?
Hell, humans don't even understand sarcasm, irony, fair use, or dog whistling much of the time. Try submitting a dog whistle to Facebook's human moderators and it'll almost certainly get declined as within their standards.
As a concrete example: "Congressman Falls for The Onion's Planned Parenthood 'Abortionplex' Story" https://www.theatlantic.com/national/archive/2012/02/congres...
As a concrete example: "Congressman Falls for The Onion's Planned Parenthood 'Abortionplex' Story" https://www.theatlantic.com/national/archive/2012/02/congres...
In the pursuit of creating a process that would be ethically challenging even for computers, we risk creating (have risked creating? have created?) processes that impose the same concerns on real people. Ostensibly so that we won't have to any more.
> Content from a site like Facebook cannot be moderated without causing harm to human moderators, and A.I. is not yet able to do the job.
It absolutely can - all you have to do is to discourage posting bad content in the first place by attaching actual consequences to breaking the rules.
Things like restrictions on new accounts such as limits on posting links, media, number of posts, etc until the account slowly levels up its trust level. This would protect both against spam from new accounts (as they aren't able to do any significant damage) as well as incentivizes people to not break the rules as to not lose their "leveled up" account. This worked very well for a decade on various forums often run by volunteers with near-zero budget.
The real problem is that unlike the forums of the good old days, Facebook's business model is not to provide a safe environment for quality discourse - instead it's to keep as many people "engaged" as possible - in this case banning rule-breakers is actually counter-productive, and is only done as a last-resort when their misdeeds are serious enough to become a PR risk for Facebook.
Their complaints about moderation being impossible is a sleazy attempt to make people accept this and for bad content to eventually stop being a PR risk, thus theoretically allowing them to completely do away with moderation and still earn ad revenue from bad/criminal content.
It absolutely can - all you have to do is to discourage posting bad content in the first place by attaching actual consequences to breaking the rules.
Things like restrictions on new accounts such as limits on posting links, media, number of posts, etc until the account slowly levels up its trust level. This would protect both against spam from new accounts (as they aren't able to do any significant damage) as well as incentivizes people to not break the rules as to not lose their "leveled up" account. This worked very well for a decade on various forums often run by volunteers with near-zero budget.
The real problem is that unlike the forums of the good old days, Facebook's business model is not to provide a safe environment for quality discourse - instead it's to keep as many people "engaged" as possible - in this case banning rule-breakers is actually counter-productive, and is only done as a last-resort when their misdeeds are serious enough to become a PR risk for Facebook.
Their complaints about moderation being impossible is a sleazy attempt to make people accept this and for bad content to eventually stop being a PR risk, thus theoretically allowing them to completely do away with moderation and still earn ad revenue from bad/criminal content.
Facebook absolutely does all the things you listed, and more; much more. While I agree that their incentives are often at odds with content enforcement, it is not a "just so" problem to be fixed with a better methodology or algorithm, and if you can think of it in an HN post I assure you they have already tried it in some fashion.
P.S. Former Facebook employee
P.S. Former Facebook employee
ML-backed automatic classification cannot solve 100% of problem, but it can do 90%-95% and left only tenative cases to humans. This combined with an easy human-reviewed appeal system and internal carma mechanics (even in the easiest form of reports/strikes/appeals/lost appeals stats) will reduce the problem 10-20-fold.
[deleted]
Why is more moderation asked for? The 1st is restricting the government to do it, so they are pushing private companies to do it? To what end goal? It does not make sense to me, but I don't know the political aspects of this.
It's worth reading but it doesn't seem very "blistering", it's very calmly written.
But hey if you work for BuzzFeed...
But hey if you work for BuzzFeed...
It looks like moderation is one of the biggest problem we're facing in tech, in different ways:
- fake reviews which make the ratings useless
- scam apps that riddle the various app stores
- abusive / illegal content on platforms that can't afford better moderation
- moderators who struggle with mental health issues, as per this tweet
- footballers receiving racist comments on their socials
Is there space for a startup to solve this problem with AI? And should it be a B2B or B2C scenario, as in working with platforms to moderate in the backend or being a user app / extension that does it in the frontend?
For example, Fakespot works great with Amazon but it's an extension you have to install, so it's not available on Safari / on Mobile. So it's still quite unknown and I'm not sure how much it can grow
- fake reviews which make the ratings useless
- scam apps that riddle the various app stores
- abusive / illegal content on platforms that can't afford better moderation
- moderators who struggle with mental health issues, as per this tweet
- footballers receiving racist comments on their socials
Is there space for a startup to solve this problem with AI? And should it be a B2B or B2C scenario, as in working with platforms to moderate in the backend or being a user app / extension that does it in the frontend?
For example, Fakespot works great with Amazon but it's an extension you have to install, so it's not available on Safari / on Mobile. So it's still quite unknown and I'm not sure how much it can grow
The essential problem is that excellent moderation can be counter to the platform incentives. Fake reviews can be bad for the customer, but potentially neutral or good for the platform. Same with fake engagement.
Doing a really good job with moderation will materially and negatively impact KPIs in the short term.
Doing a really good job with moderation will materially and negatively impact KPIs in the short term.
Why would AI solve this problem in the first place?
We barely understand how biased it is, and even if/why it's necessarily biased, not by design, but by nature. As in, biased, because the reference you use is subjective to the time and society and culture it is built in. Which is different from the whole context 30 years ago, and will be different in 30 years.
Why, also, would tech solve a problem tech did not create, but amplified?
We barely understand how biased it is, and even if/why it's necessarily biased, not by design, but by nature. As in, biased, because the reference you use is subjective to the time and society and culture it is built in. Which is different from the whole context 30 years ago, and will be different in 30 years.
Why, also, would tech solve a problem tech did not create, but amplified?
Consider the economics a bit. The point of moderation is to sustain the business model of the platforms that need it most. It's necessarily going to be a cost centre that will only get optimized down to where it's a kind of impoverished hellscape to work in.
The reason it's a cost centre is because it's trivial to stand up a new platform with a higher bar to entry that precludes almost all evil content, with the limiting factor that it won't have exposure to the total size of the legacy platforms that have no bar to entry. So for FB, better moderation is not a differentiator with an ROI, it's a risk management cost. This puts a ceiling on what they'll pay for it.
The ideal solution for an FB-like platform that is already committed to open doors user policy means that the cost model/value prop is that the solution must be able to show it's doing something above all.
If you sold them an AI solution that "just worked" for 98% of material and had a tiny relative cost, FB's real risk from the 2% that still got through doesn't change, because when FB gets asked by regulators and others what they are doing about it, your incredibly cost-effective AI solution just shows FB isn't committing resources to the problem.
When FB can show rooms full of workers at screens on 24/7 shifts responding to the problem (at some tiny foreign hourly wage that their employer with the moderation contract is probably making a 300%+ margin on) that is the solution. These peoples' job is to essentially be psychological human sacrifices to regulators. This also describes most of the economics of the managed security services SOC business model.
If you want to get into horror, find out if there is a correlation between the production of evil content and the regional presence of a moderation subcontractor. It's like anti-virus and security companies releasing viruses and exploits, but next level worse. If there is $100m/year market for moderation services, that literally creates the perverse incentive to ensure a flow of evil content.
Could there be a CrowdStrike of content moderation? Maybe. Especially in markets like China, Singapore, and increasingly Britain, where policing content could be creating the same regulatory environment that hackers created for security.
The reason it's a cost centre is because it's trivial to stand up a new platform with a higher bar to entry that precludes almost all evil content, with the limiting factor that it won't have exposure to the total size of the legacy platforms that have no bar to entry. So for FB, better moderation is not a differentiator with an ROI, it's a risk management cost. This puts a ceiling on what they'll pay for it.
The ideal solution for an FB-like platform that is already committed to open doors user policy means that the cost model/value prop is that the solution must be able to show it's doing something above all.
If you sold them an AI solution that "just worked" for 98% of material and had a tiny relative cost, FB's real risk from the 2% that still got through doesn't change, because when FB gets asked by regulators and others what they are doing about it, your incredibly cost-effective AI solution just shows FB isn't committing resources to the problem.
When FB can show rooms full of workers at screens on 24/7 shifts responding to the problem (at some tiny foreign hourly wage that their employer with the moderation contract is probably making a 300%+ margin on) that is the solution. These peoples' job is to essentially be psychological human sacrifices to regulators. This also describes most of the economics of the managed security services SOC business model.
If you want to get into horror, find out if there is a correlation between the production of evil content and the regional presence of a moderation subcontractor. It's like anti-virus and security companies releasing viruses and exploits, but next level worse. If there is $100m/year market for moderation services, that literally creates the perverse incentive to ensure a flow of evil content.
Could there be a CrowdStrike of content moderation? Maybe. Especially in markets like China, Singapore, and increasingly Britain, where policing content could be creating the same regulatory environment that hackers created for security.
> These peoples' job is to essentially be psychological human sacrifices to regulators.
Darkly true. I think this shows that the incentives are not aligned.
Outsourcing makes this a “solved” problem. Content moderation which ends up reducing time on site but saving people will not see the light of day. So the job is to flag and send content down the moderation pipeline.
For now the frontier is dealing with the politics, not improving the lot of content moderators.
Policy people will get hired because of how well they know/navigate the politics of their region, not because of their ability to design good moderation policy.
Darkly true. I think this shows that the incentives are not aligned.
Outsourcing makes this a “solved” problem. Content moderation which ends up reducing time on site but saving people will not see the light of day. So the job is to flag and send content down the moderation pipeline.
For now the frontier is dealing with the politics, not improving the lot of content moderators.
Policy people will get hired because of how well they know/navigate the politics of their region, not because of their ability to design good moderation policy.
Traubenfuchs(4)
People who comment on these moderator issues always overlook the huge huge mental red flag that these moderators always discuss: images of children being molested and tortured.
Endless images of young children. Being sexually assaulted, penetrated, made to do degrading things.
Think about how that statement makes you feel, and then imagine you’ve been tricked into looking at this stuff for money as a “content moderator.” It’s a form of cruel economic punishment.
Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. They are the ones who should be forced to pay for the long term mental health of these workers.
California ballot initiative, anyone?
Endless images of young children. Being sexually assaulted, penetrated, made to do degrading things.
Think about how that statement makes you feel, and then imagine you’ve been tricked into looking at this stuff for money as a “content moderator.” It’s a form of cruel economic punishment.
Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. They are the ones who should be forced to pay for the long term mental health of these workers.
California ballot initiative, anyone?
I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health. I'm sure numbers are hard to come by, but I'd have guessed that there wouldn't be more than a couple of these cases per month - the problem can't be that widespread, or you'd assume it would spill over into other areas as well.
There doesn't have to be that much of it created, if it can be copied for free, and someone is willing to automate posting it. Cynical-me totally believes there are enough people doing that to traumatize a human content-sorter.
YouTubes contentID system is quite good at detecting just a couple seconds of sound to determine if something is copyrighted or not. It should be quite easy to match copies of content and just get rid of it straight away, meaning they don't have to see duplicates. I would assume they already do this. Which leads me to believe there's actually that much content being produced, I imagine most of it is not produced in the West though.
There is a national database of child porn content that most big companies that host video and photos use (Facebook, Twitter, YouTube, et al). It allows them to filter the known content. I don’t know how fast that gets updated though. The larger issue is the new content. Facebook makes it very easy to take and share video, and makes producing and sharing this content easier.
It’s way, way more common than you’d ever want to believe.
Here’s a starting point, from one very repressed culture where the act of going to the police puts great shame on your family and turns you into an outcast:
> There seems to be no sign of decline in the number of sexual abuse cases against children. According to the Ministry of Health, Labor and Welfare, child consultation centers nationwide responded to 2,077 cases of child sexual abuse in fiscal 2019, up 2.75 times from fiscal 2000, when the Act on the Prevention, etc. of Child Abuse went into force. In 1,056 of these cases, the perpetrators were biological fathers.
https://mainichi.jp/english/articles/20210322/p2a/00m/0na/01...
Please, listen to children and believe their stories.
Here’s a starting point, from one very repressed culture where the act of going to the police puts great shame on your family and turns you into an outcast:
> There seems to be no sign of decline in the number of sexual abuse cases against children. According to the Ministry of Health, Labor and Welfare, child consultation centers nationwide responded to 2,077 cases of child sexual abuse in fiscal 2019, up 2.75 times from fiscal 2000, when the Act on the Prevention, etc. of Child Abuse went into force. In 1,056 of these cases, the perpetrators were biological fathers.
https://mainichi.jp/english/articles/20210322/p2a/00m/0na/01...
Please, listen to children and believe their stories.
It feels like you are suggesting that under a certain amount looking at images of children being sexually abused or tortured shouldn't have an impact on mental health? For me personally I think even a single image like that would haunt me.
Seeing one image once will have an impact but probably allow you to still lead a normal life after that. Spending all day looking at that sort of thing over an extended period of time is much more likely to have serious long-lasting effects.
I don’t think there’s a magical line where it suddenly has an impact on mental health, and the details will vary from person to person, but there is most certainly a point where you go from functioning adult to justifying a formal diagnosis of PTSD or something similar.
I don’t think there’s a magical line where it suddenly has an impact on mental health, and the details will vary from person to person, but there is most certainly a point where you go from functioning adult to justifying a formal diagnosis of PTSD or something similar.
Are you kidding me???
When I was at FB ~2012+ -- I commuted in with one of the content mods on the CP/abuse -- and shit I heard from him was HORRIFIC. and there is TONS of it.
And with all the pedo-network shit coming to light these days.. child abuse is WAY WAY WAY worse than anyone can imagine on this planet.
Ill take your downvotes without comment as a tacit acquiescence to child abuse.
When I was at FB ~2012+ -- I commuted in with one of the content mods on the CP/abuse -- and shit I heard from him was HORRIFIC. and there is TONS of it.
And with all the pedo-network shit coming to light these days.. child abuse is WAY WAY WAY worse than anyone can imagine on this planet.
Ill take your downvotes without comment as a tacit acquiescence to child abuse.
I am really glad you posted this, as I’ve heard similar from people within the org. It becomes hard to justify working on such products when you know you’re perpetuating so much harm, with zero interest from leadership in resolving the issues.
You might be getting downvoted by your former colleagues. Would you be surprised?
You might be getting downvoted by your former colleagues. Would you be surprised?
That's surprising to me. I believe you, but I'm surprised. I would have assumed that this information would have been turned over to the police and the perpetrators whittled out of the general population in short order.
They do turn stuff over, I have no idea of what % is though...
They should be transparent about it, IMO - as we really need a growing outcry on child safety rights.
We constantly talk about the rights and protections for all these various groups, yet, in really really dire situations like this, sex trafficking etc, by comparison is way more important than kvetching over identity pronouns and such.
We need a global anti-pedo-abuse force.
They should be transparent about it, IMO - as we really need a growing outcry on child safety rights.
We constantly talk about the rights and protections for all these various groups, yet, in really really dire situations like this, sex trafficking etc, by comparison is way more important than kvetching over identity pronouns and such.
We need a global anti-pedo-abuse force.
I don’t know about you, but I’ve accidentally came across a few pics on an old hard drive I was given second hand. I couldn’t even use my computer for a couple of weeks and I can still feel the revulsion decades later.
I'll burn some karma here: I know one is not supposed to question down votes, but here it goes. Why would someone down vote my comment? If someone doesn't think it adds to the narrative, skip the vote. If you feel it actively detracts or presents a false narrative, down vote.
> I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health
And I reply that just a few images had an outsized impact on me. And down voted. Normally, anecdotal evidence is replied to with the general response that "the plural of anecdote is not data."
Maybe the down note is that I'm not referencing mental health? Am I supposed to connect all the dots in my reply and go from "if a few images can have a visceral impact on me, that it is within reason that much more exposure especially over time could lead to mental health issues"?
> I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health
And I reply that just a few images had an outsized impact on me. And down voted. Normally, anecdotal evidence is replied to with the general response that "the plural of anecdote is not data."
Maybe the down note is that I'm not referencing mental health? Am I supposed to connect all the dots in my reply and go from "if a few images can have a visceral impact on me, that it is within reason that much more exposure especially over time could lead to mental health issues"?
It seems like some of your sibling comments have similarly been down-voted. God knows why. I’ve upvoted yours and theirs because I consider personal anecdotes to be useful contributions to discussion. If discussions were to be based solely on peer-reviewed meta-analyses of empirical data, they would be poorer for missing the colour and detail – and far, far drier.
Low frequency outcomes over large number of events can produce surprising numbers. 0.01% of 2B people is still 200k people. If each of them posts a single image once a year you’re still going to get around 600 pictures of child abuse per day.
Here's a currently open job role for a content assessor at the Internet Watch Foundation:
https://www.iwf.org.uk/what-we-do/who-we-are/work-us
The stats there say they've dealt with 1.2m reports over 23 years, with 560,000 takedowns. That's certainly more than a couple a month.
The stats there say they've dealt with 1.2m reports over 23 years, with 560,000 takedowns. That's certainly more than a couple a month.
I don't understand why on earth anyone would use Facebook as a platform for sharing this kind of content, they know It'll be removed and that Facebook is quite good at tracking you, so unless you're using a VM connected to a VPN with a browser soley for this content you'll have the police called on you?
It just doesn't make sense to me. Also, why is it that facebooks content moderators have to look at everything while Google just removed content and says nothing more about it? Does Google have more confidence in their AI while Facebook doesn't and therefore need people to manually check it? Or is it that there might be more content on Facebook uploaded that an AI would misinterpret as child abuse (baby photos and the likes that people seem so desperate to share on Facebook these days).
It just doesn't make sense to me. Also, why is it that facebooks content moderators have to look at everything while Google just removed content and says nothing more about it? Does Google have more confidence in their AI while Facebook doesn't and therefore need people to manually check it? Or is it that there might be more content on Facebook uploaded that an AI would misinterpret as child abuse (baby photos and the likes that people seem so desperate to share on Facebook these days).
This podcast from last year answers all these questions. Although the answers are not encouraging...
Very eye-opening stuff: https://youtu.be/qv_hokG2oSo
Very eye-opening stuff: https://youtu.be/qv_hokG2oSo
They use a number of platforms for this but FB is quite aggressive about cracking down on it, unlike AWS among others who mostly turn a blind eye.
> They are the ones who should be forced to pay for the long term mental health of these workers.
I don't disagree... However, this doesn't do anything to address the issue that someone still has to sift through this disturbing content, and society as a whole probably still pays the price.
I don't think any sort of compensation or mental help would prevent the kinds of side effects this sort of role is exposed to.
I don't disagree... However, this doesn't do anything to address the issue that someone still has to sift through this disturbing content, and society as a whole probably still pays the price.
I don't think any sort of compensation or mental help would prevent the kinds of side effects this sort of role is exposed to.
I don’t know why FB believes it should aim to be the repository for all of everyone’s media.
Most photos and videos are crappy. They could throttle it to 10 per person per day and increase quality. I’m sure people would complain and FB would think they lose out to a competitor with unlimited uploads. However, a reduced load means easier job to address the bad stuff.
It’s probably unworkable, but making people more discerning rather than less isn’t a bad thing.
Most photos and videos are crappy. They could throttle it to 10 per person per day and increase quality. I’m sure people would complain and FB would think they lose out to a competitor with unlimited uploads. However, a reduced load means easier job to address the bad stuff.
It’s probably unworkable, but making people more discerning rather than less isn’t a bad thing.
You know who else has to deal with these images - and much worse, real-world living cases? Law enforcement. Lawyers. Juries.
Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life.
See also: Soldiers. Firefighters. Emergency medical workers.
Personally, I'd rather moderate gross things than, say, work in a coal mine. But that's me. I don't appreciate the insinuation that this makes me some sort of monster.
Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life.
See also: Soldiers. Firefighters. Emergency medical workers.
Personally, I'd rather moderate gross things than, say, work in a coal mine. But that's me. I don't appreciate the insinuation that this makes me some sort of monster.
> We hire people with strong stomachs and a strong sense of duty.
It's difficult to have a strong sense of duty for the megacorp that doesn't even bother to employ you directly, but pays another megacorp which then pays you peanuts. When you don't even get to have the context of what you're doing, don't get to follow up on whether what you did was meaningful, whether the perp was caught and tried, whether the child was saved.
It's difficult to have a strong stomach if you don't see this as an occiasional element of your generally tame and meaningful work, but see hours upon hours, days upon days of some of the worst shit that the human race is capable of.
It's difficult to have a strong sense of duty for the megacorp that doesn't even bother to employ you directly, but pays another megacorp which then pays you peanuts. When you don't even get to have the context of what you're doing, don't get to follow up on whether what you did was meaningful, whether the perp was caught and tried, whether the child was saved.
It's difficult to have a strong stomach if you don't see this as an occiasional element of your generally tame and meaningful work, but see hours upon hours, days upon days of some of the worst shit that the human race is capable of.
Good point. If someone joins the Marines, they are not paid much but I assume they have a sense of pride in being a Marine. Contractors working for the military are not officially employed and often they're put to work on projects we do not want our military employees doing, but they do it for much higher pay. Law enforcement officials generally have a similar level of pride in their service and are recognized with a badge, uniform, respect, etc. If the contractors that Facebook is putting to work on content moderation don't really get to be part of the Facebook employee roster but are having to do difficult work on par with what many law enforcement personnel have to handle, should they perhaps be compensated far more than Facebook employees just like how a black ops contractor is paid far more than a uniformed Marine? If that's too expensive for Facebook, then perhaps Facebook is not running a profitable business and we should just let it fail.
> If someone joins the Marines, they are not paid much but I assume they have a sense of pride in being a Marine. Contractors working for the military are not officially employed and often they're put to work on projects we do not want our military employees doing, but they do it for much higher pay.
I know this isn't what you're doing, but this sentiment does come awful close to, "they know what they signed up for" logic. Most Marines know before they sign up that they are signing up for a high risk job, mentally and physically. In bootcamp you go through fairly extreme (to an every day person) conditioning and you are regularly conditioned to stress throughout your tenure. It takes an insane amount of work to make a person disregard concern for their own safety and run towards the sound of gun fire. Even then, this conditioning is problematic when people get out, due to numerous reasons, which is why (imo) you see so much disarray in veteran communities. Conditioning people to be to handle extraordinary circumstances is necessary, imo, but you have to find a way to decondition them.
I know this isn't what you're doing, but this sentiment does come awful close to, "they know what they signed up for" logic. Most Marines know before they sign up that they are signing up for a high risk job, mentally and physically. In bootcamp you go through fairly extreme (to an every day person) conditioning and you are regularly conditioned to stress throughout your tenure. It takes an insane amount of work to make a person disregard concern for their own safety and run towards the sound of gun fire. Even then, this conditioning is problematic when people get out, due to numerous reasons, which is why (imo) you see so much disarray in veteran communities. Conditioning people to be to handle extraordinary circumstances is necessary, imo, but you have to find a way to decondition them.
"We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life."
https://www.theatlantic.com/national/archive/2014/09/police-...
> Two studies have found that at least 40 percent of police officer families experience domestic violence, in contrast to 10 percent of families in the general population. A third study of older and more experienced officers found a rate of 24 percent, indicating that domestic violence is two to four times more common among police families than American families in general.
https://www.theatlantic.com/national/archive/2014/09/police-...
> Two studies have found that at least 40 percent of police officer families experience domestic violence, in contrast to 10 percent of families in the general population. A third study of older and more experienced officers found a rate of 24 percent, indicating that domestic violence is two to four times more common among police families than American families in general.
Those studies have been thoroughly discredited. Not only do they qualify things like "shouting" and "loss of temper" as domestic violence (which of course do not meet the legal threshold for that term), they have never been able to be reproduced. When the same exact researchers attempted to reproduce, they got a wildly different rate of 24% using the exact same methodology. Other researchers using better methodology and appropriate definitions of "domestic violence" have found numbers generally between 7% and 11%.
The 40% study itself even says that the victims reported a 10% rate of physical domestic violence from their partner, but specifically does not indicate who the aggressor is - the officer or their spouse. The study is also 30+ years old at this point. [0]
More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1]
Other studies [2] [3] report rates between about 2% and 11% for mixtures of one time and recurring domestic violence, which are in line with population averages.
[0] https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c
[1] http://scholarcommons.usf.edu/etd/1862
[2] http://webapp1.dlib.indiana.edu/virtual_disk_library/index.c...
[3] https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...
The 40% study itself even says that the victims reported a 10% rate of physical domestic violence from their partner, but specifically does not indicate who the aggressor is - the officer or their spouse. The study is also 30+ years old at this point. [0]
More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1]
Other studies [2] [3] report rates between about 2% and 11% for mixtures of one time and recurring domestic violence, which are in line with population averages.
[0] https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c
[1] http://scholarcommons.usf.edu/etd/1862
[2] http://webapp1.dlib.indiana.edu/virtual_disk_library/index.c...
[3] https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...
> More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1]
I'm sorry, are we reading the same thing?
"Using a survey created from existing scales, 250 officers were contacted within several departments in Central Florida, of these, 90 officers responded."
Asking cops "do you do crimes" is an interesting methodology, for sure.
I'm sorry, are we reading the same thing?
"Using a survey created from existing scales, 250 officers were contacted within several departments in Central Florida, of these, 90 officers responded."
Asking cops "do you do crimes" is an interesting methodology, for sure.
> Asking cops "do you do crimes" is an interesting methodology, for sure.
How do you think the population wide numbers are gathered?
The core point is that the 40% number is exaggerated BS, if for no other reason than it literally counts "shouting" and "loss of temper". There's no reliable evidence indicating the number for police officers is any different than the general population.
How do you think the population wide numbers are gathered?
The core point is that the 40% number is exaggerated BS, if for no other reason than it literally counts "shouting" and "loss of temper". There's no reliable evidence indicating the number for police officers is any different than the general population.
> How do you think the population wide numbers are gathered?
Not merely by asking the perpetrators.
13% of active cops admitting in a survey to domestic violence certainly means the actual number is higher than that.
Not merely by asking the perpetrators.
13% of active cops admitting in a survey to domestic violence certainly means the actual number is higher than that.
> More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime"
Which means 12-13% _admitted_ committing physical domestic violence, what has to be a significant undercount.
Which means 12-13% _admitted_ committing physical domestic violence, what has to be a significant undercount.
> See also: Soldiers. Firefighters. Emergency medical workers.
As a firefighter and paramedic, I wouldn't even begin to compare my job, even dealing with the occasional horrible things, in the real world, with a job that consisted of hours on end of viewing things that would have been deleted from rotten.com for being too problematic, and the effects that that would have on my mental health.
As a firefighter and paramedic, I wouldn't even begin to compare my job, even dealing with the occasional horrible things, in the real world, with a job that consisted of hours on end of viewing things that would have been deleted from rotten.com for being too problematic, and the effects that that would have on my mental health.
I mean, content moderation would be comparable after it had 500 years to evolve human centric best practices.
Even police forces have developed unions and employee care and training practices to keep them sane and healthy.
In contrast, for CM you have the magic of the market outsourcing the work to replaceable humans.
To be honest this assertion of “strong stomachs” is a unsubstantiated and unhealthy fantasy of how people in stressful jobs handle that stress.
Even police forces have developed unions and employee care and training practices to keep them sane and healthy.
In contrast, for CM you have the magic of the market outsourcing the work to replaceable humans.
To be honest this assertion of “strong stomachs” is a unsubstantiated and unhealthy fantasy of how people in stressful jobs handle that stress.
What 'best practices' developed over 500 years are you referring?
I have numerous friends who are police.
They had three months training, then were sent out to see the worst humanity has to offer.
One of my police friends has a coffee table book of interesting dead bodies from crime scenes.
I'm not sure what 500 years of best practice you're referring.
I have numerous friends who are police.
They had three months training, then were sent out to see the worst humanity has to offer.
One of my police friends has a coffee table book of interesting dead bodies from crime scenes.
I'm not sure what 500 years of best practice you're referring.
From crime scene handling, to forensics, to specialists like SWAT, all the way to non gun carrying police (uk/colonies) vs America’s gun carrying cops to access to medical and mental care and even labor unions, these are modern inventions.
Compare to things like Spanish brotherhoods or the military handling policing in Rome.
Compare to things like Spanish brotherhoods or the military handling policing in Rome.
It's a stretch
Even someone on a child porn task force won’t be looking at these images in equivalent quantity, frequency, or duration of a content moderator. Many of the things this former content moderator is suggesting are already available to the groups you mention and many people, myself included, don’t think we take good enough care of those groups. It’s also important to mention that stopping child porn is a more important goal by any sane metric than removing child porn from a website so the website can make money tracking people for advertising dollars.
You need to have the stomach to do some jobs. I remember the time when many of my friends in the medical school asked me to go with them to the dissection lab in the first year of college; I was doing CS at that time, but the professors did not know the students yet, so they gave me a lab coat and went with the group. 10 minutes later, a few students fainted, some were puking in the corners, the professor asked me to put the liver in the body on the table because I looked the least affected. I did it, so he asked me for the name to give one extra point at the exam for the semester (I gave a friend's name that was puking).
Most of these people grew up in cities, I grew up in the country side cutting chicken or pigs for food; this makes you less sensitive to some extent.
My friends that are now MDs told me that year was the worst experience of their lives. Being a Facebook moderator can be worse than that, so it can be really bad for some of the people doing this job.
Most of these people grew up in cities, I grew up in the country side cutting chicken or pigs for food; this makes you less sensitive to some extent.
My friends that are now MDs told me that year was the worst experience of their lives. Being a Facebook moderator can be worse than that, so it can be really bad for some of the people doing this job.
> My friends that are now MDs told me that year was the worst experience of their lives.
This is just incredibly telling how shielded these people are.
This is just incredibly telling how shielded these people are.
I think that selecting the "right" people may be key here.
Some time back, the child of some friends died, rather badly. This was a child I knew. They asked me to go through the police report because they could not bear to do so, just to see if anything was missed. In the interests of shielding them, I did. I had to detach in a peculiar way, and that may be while I have been able to attend autopsies and such.
Could I do it forty hours a week? I don't know, I won't claim to it, but I think looking for some kind of quality like that would be where to go until AI takes this over -- and frankly I think that will be a long time coming.
Some time back, the child of some friends died, rather badly. This was a child I knew. They asked me to go through the police report because they could not bear to do so, just to see if anything was missed. In the interests of shielding them, I did. I had to detach in a peculiar way, and that may be while I have been able to attend autopsies and such.
Could I do it forty hours a week? I don't know, I won't claim to it, but I think looking for some kind of quality like that would be where to go until AI takes this over -- and frankly I think that will be a long time coming.
> Not everyone who sees this stuff is scarred for life.
Psychological trauama and PTSD appear fairly prevalent for first-responders (worldwide)[1].
And first-responders tend to receive a lot of community support, respect and hazard pay in return for what they do.
[1] - https://en.wikipedia.org/wiki/Trauma_and_first_responders
Psychological trauama and PTSD appear fairly prevalent for first-responders (worldwide)[1].
And first-responders tend to receive a lot of community support, respect and hazard pay in return for what they do.
[1] - https://en.wikipedia.org/wiki/Trauma_and_first_responders
> Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life.
I happen to know cops. The suicide and dropout rate for those in IT evidence and sexual crimes departments is enormous.
I happen to know cops. The suicide and dropout rate for those in IT evidence and sexual crimes departments is enormous.
[deleted]
Why not hire the sickos who like these things to do the filtering?
> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible.
Is there any venture capitalist that would generally believe "there is value in not trying to connect everyone on Earth together?"
It feels like when you market your hot startup idea, connecting the world or enacting change is merely a cool new thing that carries no long-term downside. Find a market, connect with a target audience, grow. Maybe that's how Facebook started (or not).
I feel that, because of human nature, if it weren't Facebook then some other company would have inevitably taken its place. Being able to connect to someone faster and easier than ever sounds like a tangible gain for average people. The idea, with no strings attached, was too tempting to pass up for someone that truly wants to change the world and be a hero of some kind. It worked for a single university, after all. It's the sense of starting a technology company when the future is completely unknown and the potential for change is exciting. I'm not sure there was anyone that would have understood the consequences of taking such an idea to its logical conclusion, and now Facebook has become irreversibly baked into human culture.
The more people that want to connect, the more moderation staff is needed to match. Unfortunately, the desire to connect with others is wired into pretty much everyone on Earth at the deepest level (with exceptions, of course), and so your potential market is... pretty much everyone on Earth.
Is there any venture capitalist that would generally believe "there is value in not trying to connect everyone on Earth together?"
It feels like when you market your hot startup idea, connecting the world or enacting change is merely a cool new thing that carries no long-term downside. Find a market, connect with a target audience, grow. Maybe that's how Facebook started (or not).
I feel that, because of human nature, if it weren't Facebook then some other company would have inevitably taken its place. Being able to connect to someone faster and easier than ever sounds like a tangible gain for average people. The idea, with no strings attached, was too tempting to pass up for someone that truly wants to change the world and be a hero of some kind. It worked for a single university, after all. It's the sense of starting a technology company when the future is completely unknown and the potential for change is exciting. I'm not sure there was anyone that would have understood the consequences of taking such an idea to its logical conclusion, and now Facebook has become irreversibly baked into human culture.
The more people that want to connect, the more moderation staff is needed to match. Unfortunately, the desire to connect with others is wired into pretty much everyone on Earth at the deepest level (with exceptions, of course), and so your potential market is... pretty much everyone on Earth.
> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible.
Even small sites have people upload child porn and other stuff into them, and some IT guy has to go handle it.
The only thing that can really reduce the scale of this stuff is for the people doing it to suffer a consequence like jail time. That's clearly not happening in most cases, despite Facebook and others sharing information with the authorities.
The same is true for other issues like people being hounded by death/rape threats and the like online. The police mostly ignore it as an issue, so moderators have to read a bunch of it, and the people doing it just get a suspension or ban from the one platform.
Even small sites have people upload child porn and other stuff into them, and some IT guy has to go handle it.
The only thing that can really reduce the scale of this stuff is for the people doing it to suffer a consequence like jail time. That's clearly not happening in most cases, despite Facebook and others sharing information with the authorities.
The same is true for other issues like people being hounded by death/rape threats and the like online. The police mostly ignore it as an issue, so moderators have to read a bunch of it, and the people doing it just get a suspension or ban from the one platform.
Some things are not meant to operate at scale. Putting your mouth on the firehose of real human nature can’t be healthy for anyone.
I don't know that it's accurate to describe beheadings and child porn as "real human nature". The scum of nature, perhaps.
You live in the ivory tower of civilization. Do you really think you're that different from your ancestors? It's society that has evolved, not the component human parts of it.
Who decides where the line is drawn between scum and not scum? Alas, that is why this is human nature. It is all encompassing, you cannot pick and choose.
None of us are perfect.
None of us are perfect.
They should try showing the moderator a blurred version of the image first, and allow the moderator to slowly unblur it if they need further confirmation.
Or use ML to auto blur specific parts that are likely to be gore etc, and allow moderator discretion to remove the occlusion.
I bet some pretty basic tricks like this would help a lot. A high resolution image of all the gore (or whatever) inside it is probably not needed to come to a decision in most cases.
Or use ML to auto blur specific parts that are likely to be gore etc, and allow moderator discretion to remove the occlusion.
I bet some pretty basic tricks like this would help a lot. A high resolution image of all the gore (or whatever) inside it is probably not needed to come to a decision in most cases.
Do you really think that this sort of 'basic tricks' is something that didn't occur to the thousands of managers, moderators and engineers working on this?
Occur to? Sure.
Actively push to get on the roadmap and implemented, when the moderators are in another state employed via a third party contractor with no direct contact to said managers/engineers? Less likely.
Actively push to get on the roadmap and implemented, when the moderators are in another state employed via a third party contractor with no direct contact to said managers/engineers? Less likely.
Front line moderators are seen, treated and paid like expendable cannon fodder. No one cares about them, because if they drop out there's a steady stream of replacements for them.
It's similar to Amazon's warehouse pickers, callcenter agents and farm laborers. The less skills your job requires the more abuse employers can get away with.
It's similar to Amazon's warehouse pickers, callcenter agents and farm laborers. The less skills your job requires the more abuse employers can get away with.
I would add that Amazon is extremely innovative in ways to make warehouse employees more productive. They are physically limited in how any employees they can fit into a warehouse, and what geographic region those employees are hired from, so they have to make sure these employees are productive because they can't just hire twice as many to get through peaks if they aren't productive enough. At the same time, these employees are treated poorly and are peeing in bottles while standing at their station because Amazon didn't account for the physical need to urinate.
Facebook has none of these restrictions, and these employees don't even work for Amazon, and they are behaving the way you would expect a company to behave with these incentives.
Facebook has none of these restrictions, and these employees don't even work for Amazon, and they are behaving the way you would expect a company to behave with these incentives.
It's really surprising to me that people are modding parent down to oblivion. If you are going to vote it down, could you reply and say why? I'm legitimately curious.
> At the same time, these employees are treated poorly and are peeing in bottles while standing at their station because Amazon didn't account for the physical need to urinate.
Isn't it more likely that they did, but some highly paid employee or consultant said "you know what, just let them wear diapers or pee in bottles" and Bezos applauded such a brave and innovative approach?
Isn't it more likely that they did, but some highly paid employee or consultant said "you know what, just let them wear diapers or pee in bottles" and Bezos applauded such a brave and innovative approach?
Have you ever had a job?
Its quite common for companies to store all of their data in excel files on a share drive with no backups. Why would companies do that when there are so many better ideas?
The letter makes it clear that these contractors have basically no interaction with FB proper, and are treated as disposable. Is that the sort of environment that leads to tools improved in creative ways just to make it less painful to work there?
Its quite common for companies to store all of their data in excel files on a share drive with no backups. Why would companies do that when there are so many better ideas?
The letter makes it clear that these contractors have basically no interaction with FB proper, and are treated as disposable. Is that the sort of environment that leads to tools improved in creative ways just to make it less painful to work there?
It remains a major issue without end because there are no punitive consequences.
Facebook could out, shame, and be a significant aid to prosecution of the individuals who post illegal and obscene content.
But there are no regulations compelling them to do so. There's no business incentive to do it themselves.
And maybe most important, it's very clear that Facebook has no ethics in their own internal culture to take a stand on principle.
Facebook could out, shame, and be a significant aid to prosecution of the individuals who post illegal and obscene content.
But there are no regulations compelling them to do so. There's no business incentive to do it themselves.
And maybe most important, it's very clear that Facebook has no ethics in their own internal culture to take a stand on principle.
Heh - there can’t be punitive consequences, since FB has offloaded the manpower intensive work to outsourcing firms.
[deleted]
It's really sickening how much nsfl content is being uploaded and shared on facebook. Who are these people and what is wrong with them.
Have you met a human before?
[deleted]
The reason for the NDA is obvious: Facebook doesn't want you to know that there are people doing this job. Facebook doesn't want you to even realize that this job needs doing, they want you to think that the platform is just clean and nice without any of this icky stuff.
It's part of what makes Facebook enjoyable for end users, and that they've managed to do: keep the platform clean of most awful things, but it comes at a heavy cost. People don't notice this cost, and that's by design.
It's part of what makes Facebook enjoyable for end users, and that they've managed to do: keep the platform clean of most awful things, but it comes at a heavy cost. People don't notice this cost, and that's by design.
It's Vegas, right (or at least the image it wants to project when I visit as a tourist)? Clean streets, clean casinos, trouble infrequent and quickly dealt with, nice smells, free drinks, happy faces. Everything to keep you putting money on the tables and in the slots -- because showing anything nasty is terrible for business.
Is it me or is the current state of the internet akin to physical society that lacks trust systems and law enforcement?
In this analogy, are moderators of a given platform equivalent to mall cops/neighborhood watch?
In this analogy, are moderators of a given platform equivalent to mall cops/neighborhood watch?
I wonder how effective a law would be that simply said: "you can't outsource content moderation, and all employees, directors, etc. of any company that does moderation must perform at least X hours of moderation shifts per year."
If the Facebook execs had to do a day of content moderation every quarter, I imagine that meaningful change would happen a lot sooner.
If the Facebook execs had to do a day of content moderation every quarter, I imagine that meaningful change would happen a lot sooner.
Let's face it, nothing will change until US lawmakers start forcing change. Unfortunately, US lawmakers by and large don't understand social media (CEO testimonies prove this), and nothing will really change for everyone outside the US until US lawmakers force changes (GDPR still isn't really adhered to).
I wonder how much of the disturbing content is spam from botnets or nefarious groups.
That's a massive issue as lawmakers are pushing for more moderation. It's my belief that Facebook, and others, have sold our politicians on the idea that moderation can be done, at scale, with AI. Those of us who followed along knows that the human moderators have complained that at least Facebooks A.I. simply isn't up to the task.
I don't think Facebook and similar sites can exist, not at that scale, without massive human cost.