Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots(aclu.org)
aclu.org
Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots
https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-face-recognition-falsely-matched-28
113 コメント
I think a fun thought exercise is finding the fine line between tech and guns/alcohol/cars. You cannot sue a gun/car/alcohol manufacturer if their product is used to injure someone because it functioned as designed but was used maliciously. How does that legal precedent work when extrapolated to tech and something like facial recognition? If it worked exactly as designed and we know it has a margin of error (or can be used improperly and have disastrous results, like a car or gun), could Amazon or a tech administering it be liable for someone falsely imprisoned?
It's still ethically questionable. In fact I'm struggling to come up with a better example than facial recognition tech (except other mass surveillance). Maybe cutting corners while developing driverless cars that results in the death of a pedestrian.
Almost every engineering discipline has a code of ethics [0][1][2][3]. It's time software "engineering" grew up and did the same.
I rarely see ethics mentioned on HN, and granted, people's view differ. But it's weird we're not having that conversation at all.
[0] https://www.raeng.org.uk/policy/engineering-ethics/ethics
[1] https://www.ieee.org/about/corporate/governance/p7-8.html
[2] https://www.nspe.org/resources/ethics/code-ethics
[3] http://www.asce.org/code-of-ethics/
Almost every engineering discipline has a code of ethics [0][1][2][3]. It's time software "engineering" grew up and did the same.
I rarely see ethics mentioned on HN, and granted, people's view differ. But it's weird we're not having that conversation at all.
[0] https://www.raeng.org.uk/policy/engineering-ethics/ethics
[1] https://www.ieee.org/about/corporate/governance/p7-8.html
[2] https://www.nspe.org/resources/ethics/code-ethics
[3] http://www.asce.org/code-of-ethics/
You're right! This is a critically important conversation that we absolutely need to have within our profession. It's very often ignored and there's no support system for people who take ethical stands.
So. Let's talk about ethics. I, personally, subscribe to the ACM code of ethics.
I think Rekognition, as built and presented, falls fully within that strict ethical code. It can be put to uses that are unethical, but that does not fall upon the people who made it. Certainly, an engineer creating a system such as the one the ACLU created would be acting unethically.
So. Let's talk about ethics. I, personally, subscribe to the ACM code of ethics.
I think Rekognition, as built and presented, falls fully within that strict ethical code. It can be put to uses that are unethical, but that does not fall upon the people who made it. Certainly, an engineer creating a system such as the one the ACLU created would be acting unethically.
I think a better analogy would be a bridge or building- they are regulated by a rigorous code that will exonerate the engineers behind the implementation only if it was followed to the letter.
Will that slow things down and make them more expensive? Absolutely, but that’s the cost of safety.
In this case though what is probably most lacking is public knowledge about the shortcomings of such systems. The public needs to understand that 90% accuracy, while it sounds high, mean it’s wrong a lot, and the chances of it being wrong if it was continuously run all the time are actually quite high.
Will that slow things down and make them more expensive? Absolutely, but that’s the cost of safety.
In this case though what is probably most lacking is public knowledge about the shortcomings of such systems. The public needs to understand that 90% accuracy, while it sounds high, mean it’s wrong a lot, and the chances of it being wrong if it was continuously run all the time are actually quite high.
If firearms manufacturers produced guns that fired straight over ninety percent of the time, but no guarantee of bullet direction beyond that, they'd probably have a lot of lawsuits on their hands.
Would a camera manufacturer be liable if the camera's image processor happened to cause one person's photograph to look like another, which then caused a human looking at that photograph to misidentify someone?
Malice is beside the point here is the it? The concern isn't an evil contractor, so much as an incompetent one.
Car manufacturers are sued for incompetent use of their products.
Car manufacturers are sued for incompetent use of their products.
> could Amazon or a tech administering it be liable for someone falsely imprisoned?
I would wager "absolutely not" for the exact same issue as firearms or cars. The person who mis-applied the technology or a middle man vendor though? Unless they fall under qualified immunity, there's your fall guy.
I would wager "absolutely not" for the exact same issue as firearms or cars. The person who mis-applied the technology or a middle man vendor though? Unless they fall under qualified immunity, there's your fall guy.
Things I want to know:
- Show us the side-by-side images of the false-positives. Are the matches plausible?
- What is the demographic distribution of the mugshot database? If the data is disproportionately biased, then that bias would be reflected in the false-positives. A casual skimming of some mughot websites shows a potentially significant racial bias.
- Show us the side-by-side images of the false-positives. Are the matches plausible?
- What is the demographic distribution of the mugshot database? If the data is disproportionately biased, then that bias would be reflected in the false-positives. A casual skimming of some mughot websites shows a potentially significant racial bias.
I have a ton of quibbles with the article and methodology, honestly. I suspect it's just straight-up so flawed as to be irrelevant to the discussion.
At the same time, they do still have the correct final conclusion. Dragnet facial recognition is a bad idea that will produce too many false positives to be of use. Or at the very least, dragnet facial recognition used by people who put too much faith in it is a bad idea. It's at most useful to produce flags for slightly more attention; on its own, I would not consider it anywhere close to enough to arrest someone on.
And it can serve as a demonstration of how a clumsily-put-together system can be used to produce bad results, so, if we correctly assume that there may be people bodging together a system in much the same way the ACLU did, well, hey, it's a valid result then!
At the same time, they do still have the correct final conclusion. Dragnet facial recognition is a bad idea that will produce too many false positives to be of use. Or at the very least, dragnet facial recognition used by people who put too much faith in it is a bad idea. It's at most useful to produce flags for slightly more attention; on its own, I would not consider it anywhere close to enough to arrest someone on.
And it can serve as a demonstration of how a clumsily-put-together system can be used to produce bad results, so, if we correctly assume that there may be people bodging together a system in much the same way the ACLU did, well, hey, it's a valid result then!
I really want to see their methodology. I'd love to replicate it and play with certainty levels to see what happens.
Whether the matches are plausible is completely besides the point. This isn't an evaluation of Amazon's recognition quality, it's a demonstration that the system is fallible and notes on the consequences.
Same thing re data setup. Not the point again. Even if law enforcement hires the world's best computer vision experts (fat chance, it'll go to cheapest contractor) this is a note that there can be mistakes that get people killed.
Same thing re data setup. Not the point again. Even if law enforcement hires the world's best computer vision experts (fat chance, it'll go to cheapest contractor) this is a note that there can be mistakes that get people killed.
What are the consequences?? How is this any different than the suspect sketches?
This is a far worse situation and still quite minor
https://www.cbc.ca/news/canada/british-columbia/bc-mother-ch...
Because most people don't view sketches with the sort of religious reverence they do technology.
One major difference is that non-technical people tend to trust what a computer says far more than they should. People are familiar with sketches and can make judgments on how clearly they match a face. They have no similar frame of reference for computer face recognition.
But people are familiar with photographs.
There is no way the software wouldn't show the picture that's been matched before an action (arrest...) is taken
What does the link have to do with sketches? It's about name similarity on the no fly list.
Regarding sketches, they're actually interesting because they force a human to exercise judgement, being ambiguous by nature. This is fine - any law enforcement officer stopping someone who looks like a sketch is going to give them the benefit of the doubt. On the other hand, if a computer tells an officer a person ahead is an 89% match with a known killer, the conversation will start with guns drawn, and that will greatly increase the likelyhood of things going south.
You really don't want to be scanned by a trigger happy cop having a bad day. God knows who you happen to look like from that angle.
Regarding sketches, they're actually interesting because they force a human to exercise judgement, being ambiguous by nature. This is fine - any law enforcement officer stopping someone who looks like a sketch is going to give them the benefit of the doubt. On the other hand, if a computer tells an officer a person ahead is an 89% match with a known killer, the conversation will start with guns drawn, and that will greatly increase the likelyhood of things going south.
You really don't want to be scanned by a trigger happy cop having a bad day. God knows who you happen to look like from that angle.
"[..]conversation will start with guns drawn, and that will greatly increase the likelyhood of things going south."
That is where the two sides of this argument diverge. First: That things "will start with guns drawn". And Second: That things are more likely to "go south" for an innocent individual incorrectly blamed by this.
That is where the two sides of this argument diverge. First: That things "will start with guns drawn". And Second: That things are more likely to "go south" for an innocent individual incorrectly blamed by this.
Police have a poor history in the US of exercising trigger discipline. A fair amount of this is due to overly aggressive training that is designed to keep the officers as safe as possible, sacrificing safety of anyone perceived as a threat (real or not), but another large portion is caused by the culture of dominance and entitlement that is endemic to our law enforcement.
If police engage a target with a weapon already drawn, there is a long list of things that person can do that will get them shot, and a very short (and often unclear) list of actions that will keep them safe.
Source: 10 years PMO in the Marine Corps, 1 month SFPD (then quit because their training was so shitty)
If police engage a target with a weapon already drawn, there is a long list of things that person can do that will get them shot, and a very short (and often unclear) list of actions that will keep them safe.
Source: 10 years PMO in the Marine Corps, 1 month SFPD (then quit because their training was so shitty)
Regarding point two, I can't really see any argument that an encounter with a police officer is not more likely to go south if the officer is under the misapprehension that one is a potentially dangerous criminal; Police shootings are already not infrequent in the US, and adding the tension of greater perceived danger makes an already volatile situation worse - a sudden movement that might otherwise pass unnoticed is much more threatening from someone who has been misidentified as an armed gunman, for example, and is far more likely to elicit lethal response.
[deleted]
I don't really understand the sides and arguments you are implying here. Could you spell it out please? (I'm not from the US, and have no relevant experience with cops, guns.)
I don't see how this is a problem, unless the police are feeding it blurry surveillance camera footage then claiming "success, it matched person X", which I doubt, as it would fall apart at the smallest of questioning.
For the most part, this sort of article makes things seem like there will be some sort of racial-based, dystopian police future where automated algorithms will target people and have them put into jail. I don't see that happening in the near future at all. The police departments' personnel currently using this probably LOOK at the matches and only then make a further call. Just like I would assume AFIS fingerprint hit results are double-checked by skilled operators after matching to reduce false-positives.
For the most part, this sort of article makes things seem like there will be some sort of racial-based, dystopian police future where automated algorithms will target people and have them put into jail. I don't see that happening in the near future at all. The police departments' personnel currently using this probably LOOK at the matches and only then make a further call. Just like I would assume AFIS fingerprint hit results are double-checked by skilled operators after matching to reduce false-positives.
Depending on the technologi behind Amazons facial recognitions, there's very real chance that the software simply associated some black facial features with "being criminal". If that's the case we're back to craniometry in the 1910-1940.
Perhaps lawmakers should consider implementing rules to ban facial recognition software until the developers can prove some like 99.999% accuracy. Otherwise we end up harassing a large number of people, and the credibility of the technology will stuff. As it stand any lawyer will be able to argue that facial recognition isn't admisible as evidence, do to the high margin of error.
Perhaps lawmakers should consider implementing rules to ban facial recognition software until the developers can prove some like 99.999% accuracy. Otherwise we end up harassing a large number of people, and the credibility of the technology will stuff. As it stand any lawyer will be able to argue that facial recognition isn't admisible as evidence, do to the high margin of error.
They fed it a large number of mugshots. They didn't ask "is this person a criminal". They asked it "does this face match one of the faces you've seen before".
You're right that systems have previously learned to infer black -> criminal. It's absolutley possible that could have happened here! Given that not every one flagged is a person of color, there's clearly more at work.
Systems like Rekognition let a user specify what level of certainty the system has that it has found a match. In this instance, the system was told to use 80%. Rekognition can be told to use any number from 0 to 100. Your number of 99.999% could be used without any difficulty whatsoever! Since the ACLU doesn't seem to be sharing enough to reproduce the test, we can't tell what would happen.
Regulation based on accuracy is a wonderful idea! One well worth pursuing. With that said, "accuracy" is perhaps difficult to define here. How would you determine that? How accuracy is measured is highly sensitive to what you use as a dataset, and different test datasets can easily produce very different impressions of how common false positives and false negatives are.
You're right that systems have previously learned to infer black -> criminal. It's absolutley possible that could have happened here! Given that not every one flagged is a person of color, there's clearly more at work.
Systems like Rekognition let a user specify what level of certainty the system has that it has found a match. In this instance, the system was told to use 80%. Rekognition can be told to use any number from 0 to 100. Your number of 99.999% could be used without any difficulty whatsoever! Since the ACLU doesn't seem to be sharing enough to reproduce the test, we can't tell what would happen.
Regulation based on accuracy is a wonderful idea! One well worth pursuing. With that said, "accuracy" is perhaps difficult to define here. How would you determine that? How accuracy is measured is highly sensitive to what you use as a dataset, and different test datasets can easily produce very different impressions of how common false positives and false negatives are.
They didn't tell it to use 80%, 80% is the default setting. Anyone who thinks that many law enforcement agencies aren't going to use the default setting ought to go look at their routers and other hardware and see how they are configured.
I understand where you're coming from. They didn't choose that number! It was chosen for them by Amazon! It's on Amazon for not actively forcing users to consider what level of confidence they want to use. Amazon can, should, and I think is perhaps even morally obligated to make customers actively choose a confidence level.
Yet, might there not be other factors to consider? Bear in mind that this isn't pushbutton easy off-the-shelf. Some actual work is required here. There's probably a batch of Python scripts sitting around that did the work. The most charitable assumption is that whoever did this at the ACLU didn't look at how the tool under examination should be used at all and missed everything about confidence levels in whatever few docs they did read. Under the ACM code of ethics, I am comfortable calling that an unethical degree of incompetence and carelessness.
A less charitable interpretation is that they knew and didn't care because it produced the result they wanted. This is similarly unethical, in my mind on a level with the sort of p-hacking prevalent in social sciences.
You're completely right! They just used the default. It's just perhaps possible that there might be other factors worth considering.
Yet, might there not be other factors to consider? Bear in mind that this isn't pushbutton easy off-the-shelf. Some actual work is required here. There's probably a batch of Python scripts sitting around that did the work. The most charitable assumption is that whoever did this at the ACLU didn't look at how the tool under examination should be used at all and missed everything about confidence levels in whatever few docs they did read. Under the ACM code of ethics, I am comfortable calling that an unethical degree of incompetence and carelessness.
A less charitable interpretation is that they knew and didn't care because it produced the result they wanted. This is similarly unethical, in my mind on a level with the sort of p-hacking prevalent in social sciences.
You're completely right! They just used the default. It's just perhaps possible that there might be other factors worth considering.
When facial recognition fails, it usually looks pretty weird. Like it will identify faces in the knots and grain patterns of a wooden wall behind somebody. It's really a very ineffective technology and it's disturbing that it gets sold as something you could use to justify locking someone in a cage over. Machines with pareidolia are not good tools.
When people speak of "facial recognition", they speak of faces being compared to other faces. What you're mentioning is face-detection. Are you honestly saying that police departments will have a "criminal database" filled with wooden walls and furniture?
I'm saying that police departments DO have databases filled with images that have wooden walls and such. Not furniture usually, but photos are often taken in sub-optimal circumstances even when the police or others are trained to take the photos properly. I know this is the case because I've worked on biometric systems for the past 18 years, and I have personally worked on a system which was identifying almost every face to a mugshot taken against a wall with wooden paneling and the matching algorithm was keying in on it as a face rather than the human. I'm not speaking theoretically, this is how these systems fail.
They have Montana Rep. Greg Gianforte in the list of false positives. Did they call it a false positive because they are sure he wasn't included in the 25,000 mugs they loaded? His mug is out there.
https://www.google.com/amp/s/amp.usatoday.com/amp/756343001
https://www.google.com/amp/s/amp.usatoday.com/amp/756343001
https://www.theverge.com/2018/7/26/17615634/amazon-rekogniti...
>Reached by The Verge, an Amazon spokesperson attributed the results to poor calibration. The ACLU’s tests were performed using Rekognition’s default confidence threshold of 80 percent — but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.
Presented without real comment on my part.
>Reached by The Verge, an Amazon spokesperson attributed the results to poor calibration. The ACLU’s tests were performed using Rekognition’s default confidence threshold of 80 percent — but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.
Presented without real comment on my part.
Would that mean 5% could be false positives? That would be a lot.
From the AWS docs:
> A confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct. In the tropical beach example, if the object and scene detection process returns a confidence score of 99 for the label ‘Water’ and 35 for the label ‘Palm Tree’, then it is more likely that the image contains water but not a palm tree.
> Applications that are very sensitive to detection errors (false positives) should discard results associated with confidence scores below a certain threshold. The optimum threshold depends on the application. In many cases, you will get the best user experience by setting minimum confidence values higher than the default value.
Which I am interpreting as similar to your question, but perhaps should not be understood as a statistical probability.
> A confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct. In the tropical beach example, if the object and scene detection process returns a confidence score of 99 for the label ‘Water’ and 35 for the label ‘Palm Tree’, then it is more likely that the image contains water but not a palm tree.
> Applications that are very sensitive to detection errors (false positives) should discard results associated with confidence scores below a certain threshold. The optimum threshold depends on the application. In many cases, you will get the best user experience by setting minimum confidence values higher than the default value.
Which I am interpreting as similar to your question, but perhaps should not be understood as a statistical probability.
You think Police Departments won't knock down the threshold just for justification to knock down some perp's door? They've done far worse to get the results they want.
Thinking of an Onion headline: "This tech sucks" says person trying to use niche technology for the first time without knowing its specificities and limitations.
I guess the question is: do we trust any operator to know the specificities and limitations? I for one wouldn't trust Joe Cop to use something like this with the degree of discretion or understanding of statistical nuance called for here.
Wow, if that's the case then this article is quite unfair.
They should rerun this at 95% confidence and see how many mismatches they get, I suspect it will be much lower than the 6.4% they have at 80% confidence, thought it might cost more than the $12.33 they were willing to invest in this hitpiece against rekognition.
They should rerun this at 95% confidence and see how many mismatches they get, I suspect it will be much lower than the 6.4% they have at 80% confidence, thought it might cost more than the $12.33 they were willing to invest in this hitpiece against rekognition.
In my opinion, this is where the crux of the misinformed population epidemic resides. Sensational headline that causes mass amounts of doubt, panic, uncertainty, and outrage will get thousands and thousands of retweets and shares, yet the very valid response to it from the accused will not get any traction.
> but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.
That's not in https://aws.amazon.com/rekognition/faqs/ - while that threshold may minimize false positives, I'm curious if it was in any documentation that the ACLU saw. Or, to the bigger point, is it in any public documentation?
That's not in https://aws.amazon.com/rekognition/faqs/ - while that threshold may minimize false positives, I'm curious if it was in any documentation that the ACLU saw. Or, to the bigger point, is it in any public documentation?
Threshold is a configuration property no different than a dynamo read capacity. Misconfigured properties don’t work the way you expect. I don’t see how documentation can be expected to give black and white guidance to what this threshold should be set to without having more understanding of your use case.
> I don’t see how documentation can be expected to give black and white guidance to what this threshold should be set to without having more understanding of your use case.
It can say exactly what the Amazon response to the Verge dismissing the ACLU test should, and, in fact, it should say that if Amazon really does have such critical, specific domain-specific recommended settings, rather than it being something made up off the cuff to mitigate bad press.
It can say exactly what the Amazon response to the Verge dismissing the ACLU test should, and, in fact, it should say that if Amazon really does have such critical, specific domain-specific recommended settings, rather than it being something made up off the cuff to mitigate bad press.
You're unquestionably correct. Amazon should spell out in clear, simple, easily accessible language what level of certainty is appropriate for what type of application, giving concrete examples at every point.
With that said, it's possible that no amount of clear guidance would have mattered here. The ACLU isn't exactly asking for better docs.
With that said, it's possible that no amount of clear guidance would have mattered here. The ACLU isn't exactly asking for better docs.
If you're someone with minimal understanding of the use and deployment of technologies, who knows only of this technology as something used to detect faces, then it's a glaring oversight.
If you're a researcher making a deliberate choice to use the defaults to simulate a random police officer with no real understanding of their tools, then say as much. And compare with more cautious results.
If you're a researcher making a deliberate choice to use the defaults to simulate a random police officer with no real understanding of their tools, then say as much. And compare with more cautious results.
I would be shocked if LE started using this tech, imprisoned a bunch of people, and used the excuse "Well, we followed the FAQS." I would imagine they would work very closely with Amazon and run a ton of calibration. For the ACLU to say that we used the default settings and it doesn't work correctly is disingenuous.
I would not be shocked at all if some LE somewhere used the default settings. It seems absurd to assume otherwise. It's not as though LE has a great track record.
https://en.wikipedia.org/wiki/No_Fly_List#False_positives
https://en.wikipedia.org/wiki/No_Fly_List#False_positives
Assuming that law enforcement is going to spend any amount of time being careful about enforcing correct use of forensic tools is hard to reconcile with the history of the American judicial system. For example, there is essentially no scientific evidence that a polygraph test measures anything at all, but polygraph evidence is considered admissible in many courts and used to be admitted in many more. Moreover, rigorous studies of fingerprint evidence have found them to have false positive rates as high as 1 in 18, but they are still often treated as close to infallible in court. I am not sure why you think facial recognition is going to be different.
LE doesn't have the best track record following best practices. For example, they still rely heavily on notoriously unreliable eye-witness accounts and polygraph tests, two things science has largely debunked.
They were slow to let go of phrenology as well.
They were slow to let go of phrenology as well.
You are going to be very very shocked my friend!
[deleted]
On the other hand if you want to judge whether it would work on 300 million people with 95% confidence then running on 535 members of congress with 80% confidence might be a good approximation
(AWS employee)
That’s not how these numbers fit together. Using rekognition, you would create a collection of faces based off your image corpus and then ask the service if a provided image has a match against anything in your collection. Lowering your confidence threshold, invariant of your sample size, is by definition going to increase your false positive rate while lowering your false negative rate.
I think it would be worthwhile to compare a mixed test set and see false positives/negatives at various confidence thresholds, because the fear (IMO) is that law enforcement will minimize false negatives at the expense of a huge increase in false positives.
That’s not how these numbers fit together. Using rekognition, you would create a collection of faces based off your image corpus and then ask the service if a provided image has a match against anything in your collection. Lowering your confidence threshold, invariant of your sample size, is by definition going to increase your false positive rate while lowering your false negative rate.
I think it would be worthwhile to compare a mixed test set and see false positives/negatives at various confidence thresholds, because the fear (IMO) is that law enforcement will minimize false negatives at the expense of a huge increase in false positives.
I don't see why this is a bad thing.
Falsely arrest 28 members of Congress due to poor face recognition, and the problem of facial recognition in law enforcement is resolved the next day.
Falsely arrest 28 members of Congress due to poor face recognition, and the problem of facial recognition in law enforcement is resolved the next day.
Congress doesn't stick up for each other. As soon as those 28-Congressmen are gone, they'll start passing legislation that those 28-people didn't want.
Congress is a battleground. Not a club or secret society. You send representitives from your state to try and get a slice of the pie, and I send my reps from my area to try and get a slice for me.
Congress is a battleground. Not a club or secret society. You send representitives from your state to try and get a slice of the pie, and I send my reps from my area to try and get a slice for me.
[deleted]
> People of color were disproportionately falsely matched in our test.
They are trying to make this racially charged without giving enough information to verify their claims. If you use a dataset of mugshots, that's statistically going to have more data on people of color. If you have more data on people of color, it is more likely to match people of color. Claiming the algorithm is racist because your data is racist is inflammatory bullshit.
They are trying to make this racially charged without giving enough information to verify their claims. If you use a dataset of mugshots, that's statistically going to have more data on people of color. If you have more data on people of color, it is more likely to match people of color. Claiming the algorithm is racist because your data is racist is inflammatory bullshit.
Pretty lousy argument. Firstly, the majority of inmates in the US are white (58%). I'd assume mug shot stats are similar, or at the very least not mostly black people.
Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm
Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm
> Firstly, the majority of inmates in the US are white (58%). I'd assume mug shot stats are similar, or at the very least not mostly black people.
I actually did not know this. Thank you. I had conflated incarceration rate with inmate population. If anyone is curious for a source, see here: https://www.bop.gov/about/statistics/statistics_inmate_race....
> Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm
I agree with this 100%. But they haven't provided enough information about their dataset to make this conclusion. If they provided enough information for someone to independently analyze the data and reproduce the experiment, I would flip immediately.
I actually did not know this. Thank you. I had conflated incarceration rate with inmate population. If anyone is curious for a source, see here: https://www.bop.gov/about/statistics/statistics_inmate_race....
> Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm
I agree with this 100%. But they haven't provided enough information about their dataset to make this conclusion. If they provided enough information for someone to independently analyze the data and reproduce the experiment, I would flip immediately.
> Pretty lousy argument. Firstly, the majority of inmates in the US are white (58%). I'd assume mug shot stats are similar, or at the very least not mostly black people.
Personally, I would assume the mugshots are a random sample of a public corpus, and thus likely 35-40% people of color. Which is definitely skewed from the background population. I'm assuming the similarity of 40% is a coincidence.
> Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm
Is it? It sounds like it might be working exactly as it was set up and specified to work. I would call that the fault of the designers and specifiers, rather than the algorithm. It's a good algorithm. It fits its purpose well. Its purposes just happen to be completely evil.
Personally, I would assume the mugshots are a random sample of a public corpus, and thus likely 35-40% people of color. Which is definitely skewed from the background population. I'm assuming the similarity of 40% is a coincidence.
> Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a bad algorithm
Is it? It sounds like it might be working exactly as it was set up and specified to work. I would call that the fault of the designers and specifiers, rather than the algorithm. It's a good algorithm. It fits its purpose well. Its purposes just happen to be completely evil.
And the data is "racist" as well because...?
This whole topic is racially charged because incarceration rates are a race issue and you are being disingenuous with your comment pretending the problem with facial recognition can somehow exist and be fixed in isolation.
This whole topic is racially charged because incarceration rates are a race issue and you are being disingenuous with your comment pretending the problem with facial recognition can somehow exist and be fixed in isolation.
If your data is disproportionate then you're more likely to get matches in the larger portion. The ACLU is claiming that Rekognition is racially biased, but their experiment could be setting them to get the result they want because their dataset is already biased.
I'm not saying the topic isn't a race issue, I'm saying the ACLU isn't providing the data for someone to verify their results and is possibly making false claims that no one can verify.
I'm not saying the topic isn't a race issue, I'm saying the ACLU isn't providing the data for someone to verify their results and is possibly making false claims that no one can verify.
Claiming the algorithm is racist because your data is racist is inflammatory bullshit.
Unless the designers of the system are aware of and account for biases in the training data, the system will be biased and will be biased due to the actions of its designers.
I don't see what's so hard to accept about that. It's literally the oldest problem in computing (going back to the "Pray Mr. Babbage, if we put into your machine wrong figures, will the right answers come out" days).
Unless the designers of the system are aware of and account for biases in the training data, the system will be biased and will be biased due to the actions of its designers.
I don't see what's so hard to accept about that. It's literally the oldest problem in computing (going back to the "Pray Mr. Babbage, if we put into your machine wrong figures, will the right answers come out" days).
Is the claim that it is up to the system to detect bias in training data it is given?
My personal claim is it's up to the people creating and training the system to be aware of biases and to account for that in how they create and train the system.
The problem, of course, is how woefully underprepared the average tech person is for realizing the existence of even extremely obvious biases, let alone more subtle and pernicious ones, and a tech culture in which people are lauded for being knee-jerk contrarians on topics like "does our society have a history of systemic racism".
The problem, of course, is how woefully underprepared the average tech person is for realizing the existence of even extremely obvious biases, let alone more subtle and pernicious ones, and a tech culture in which people are lauded for being knee-jerk contrarians on topics like "does our society have a history of systemic racism".
How is stating a fact- it disproportionately matched people of color- somehow the same as claiming racism?
The fact is irrelevant to the problem and there is no reason to mention it, if they want the debate to stay professional and not emotional.
The whole racism angle of the article seems way too loaded for me.
The whole racism angle of the article seems way too loaded for me.
I don't see any mention of them verifying those members of congress are not in fact the same people as the mugshots (/s)
I admit that was my first interpretation before I remembered the issues with accuracy.
Well, it's not like police are using Amazon's face recognition technology in dragnets.
Oh, wait: https://www.usatoday.com/story/tech/talkingtech/2018/06/29/c...
Oh, wait: https://www.usatoday.com/story/tech/talkingtech/2018/06/29/c...
People have similar faces. Who knew?
The technology is not perfect. And should never be used as evidence of a crime, just as an indication that two images might be of the same person and to have humans look at them.
And then those humans will also make mistakes. People do look alike. And there are twins.
I think a jury should require more evidence than just similar appearance. But such a match is a strong indication of where to look.
The technology is not perfect. And should never be used as evidence of a crime, just as an indication that two images might be of the same person and to have humans look at them.
And then those humans will also make mistakes. People do look alike. And there are twins.
I think a jury should require more evidence than just similar appearance. But such a match is a strong indication of where to look.
I don't think that's the point -- the problem isn't that "oh there are other processes that [...]," it's that this wave of technology is misused, is bias, and care must be taken when any of the signals we create.
This is especially important as automation enters the equation.
This is especially important as automation enters the equation.
The scary part is that people believe "science" because it is created by people who they think are "clever" and thefore not likely to be wrong.
How many people have been falsely convicted because "DNA"? 1B-1 odds of a match, "and sir, yet you claim you were not even in the area?".
There should be a way of recognising the parts of the science that are basically correct and the parts that are either less reliable, open to bias or could simply be broken due to incorrect process/mistake in the lab.
How many people have been falsely convicted because "DNA"? 1B-1 odds of a match, "and sir, yet you claim you were not even in the area?".
There should be a way of recognising the parts of the science that are basically correct and the parts that are either less reliable, open to bias or could simply be broken due to incorrect process/mistake in the lab.
This is a long, horrific read about exactly that, resulting in the execution of an almost certainly innocent man.
https://www.newyorker.com/magazine/2009/09/07/trial-by-fire
> Many arson investigators, it turned out, had only a high-school education. In most states, in order to be certified, investigators had to take a forty-hour course on fire investigation, and pass a written exam. Often, the bulk of an investigator’s training came on the job, learning from “old-timers” in the field, who passed down a body of wisdom about the telltale signs of arson, even though a study in 1977 warned that there was nothing in “the scientific literature to substantiate their validity.”
> In 1992, the National Fire Protection Association, which promotes fire prevention and safety, published its first scientifically based guidelines to arson investigation. Still, many arson investigators believed that what they did was more an art than a science—a blend of experience and intuition. In 1997, the International Association of Arson Investigators filed a legal brief arguing that arson sleuths should not be bound by a 1993 Supreme Court decision requiring experts who testified at trials to adhere to the scientific method. What arson sleuths did, the brief claimed, was “less scientific.” By 2000, after the courts had rejected such claims, arson investigators increasingly recognized the scientific method, but there remained great variance in the field, with many practitioners still relying on the unverified techniques that had been used for generations. “People investigated fire largely with a flat-earth approach,” Hurst told me. “It looks like arson—therefore, it’s arson.” He went on, “My view is you have to have a scientific basis. Otherwise, it’s no different than witch-hunting.”
https://www.newyorker.com/magazine/2009/09/07/trial-by-fire
> Many arson investigators, it turned out, had only a high-school education. In most states, in order to be certified, investigators had to take a forty-hour course on fire investigation, and pass a written exam. Often, the bulk of an investigator’s training came on the job, learning from “old-timers” in the field, who passed down a body of wisdom about the telltale signs of arson, even though a study in 1977 warned that there was nothing in “the scientific literature to substantiate their validity.”
> In 1992, the National Fire Protection Association, which promotes fire prevention and safety, published its first scientifically based guidelines to arson investigation. Still, many arson investigators believed that what they did was more an art than a science—a blend of experience and intuition. In 1997, the International Association of Arson Investigators filed a legal brief arguing that arson sleuths should not be bound by a 1993 Supreme Court decision requiring experts who testified at trials to adhere to the scientific method. What arson sleuths did, the brief claimed, was “less scientific.” By 2000, after the courts had rejected such claims, arson investigators increasingly recognized the scientific method, but there remained great variance in the field, with many practitioners still relying on the unverified techniques that had been used for generations. “People investigated fire largely with a flat-earth approach,” Hurst told me. “It looks like arson—therefore, it’s arson.” He went on, “My view is you have to have a scientific basis. Otherwise, it’s no different than witch-hunting.”
At the risk of being down voted....it recognizes 28 members of congress as criminals....i’d Call that a good start.
[deleted]
I work in the industry and obviously I can't say for certain, but I don't think Amazon has anywhere near the most accurate face recognition software available. If you want to see where the tech is in terms of performance, the various NIST tests are highly informative.
Amazon really has to take this back to the drawing board. ~5% accuracy is pretty terrible...
Could this be down to the people training the algorithm being predominantly middle class white people? It's a pretty well known phenomenon that people are bad at recognising features of people of other races.
>middle class
? That doesn't add up to the rest of your comment.
? That doesn't add up to the rest of your comment.
Racial minorities are gong to be greatly overrepresented in a collection of mugshots, so it's almost certainly just the case that it's easier to make mistakes in that direction.
If the algorithm is, say, 99.99% accurate on faces that are the same race, gender, and roughly age, and there's 100 faces in there that match those, it'll correctly say no match .9999^100 = 99% of the time. If there's 5000, it's only going to find no match .9999^5000 = 60% of the time.
If the algorithm is, say, 99.99% accurate on faces that are the same race, gender, and roughly age, and there's 100 faces in there that match those, it'll correctly say no match .9999^100 = 99% of the time. If there's 5000, it's only going to find no match .9999^5000 = 60% of the time.
In other news, Amazons "Freudian Slip" Machine is 100% accurate.
America is in too much danger to take any chances. The 28 positives should be held indefinitely until they can prove they're innocent.
Presumption of Guilt is the new normal, isn't it?
Presumption of Guilt is the new normal, isn't it?
There are definitely concerns about false positives, but it has to be compared to the current system. Are the results, effectiveness, better?
When the gov steers opinion, we call it manufactured consent, when public advocacy organizations engage in sloppy methodology to further a cause, I propose calling it manufactured outrage.
When the gov steers opinion, we call it manufactured consent, when public advocacy organizations engage in sloppy methodology to further a cause, I propose calling it manufactured outrage.
The algo missed 507 of those crooks!
Are they sure this was a mistake? I wouldn't be terribly surprised to find out if that many had mugshots.
Sure. "Falsely" :-)
maybe we should let the cops have this...
What percentage of people in the openly available mugshot database were people of colour, and why should that not be relevant?
It's relevant, but not as an excuse to dismiss the problem.
One of the perennial problems with machine learning is that it has a tendency to intensify pre-existing biases in the system. It's not just that the algorithm tends to reflect biases in the source data, it's that that reflection tends to encourage the people using the system, who generally have some role in creating that source data in the first place, to become even more biased.
One of the perennial problems with machine learning is that it has a tendency to intensify pre-existing biases in the system. It's not just that the algorithm tends to reflect biases in the source data, it's that that reflection tends to encourage the people using the system, who generally have some role in creating that source data in the first place, to become even more biased.
biasception.
This can be the title of a new film where ai-powered echo chambers produce progressively more polarized societies until civil war and anarchy destroy all of humankind.
Oh wait, this might just be reality already.
We've banned this account for repeatedly breaking the site guidelines. If you don't want to be banned, you're welcome to email [email protected] and give us reason to believe that you'll follow the rules in the future.
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
Society's honestly not becoming particularly polarized, on anything remotely like a historical scale. To take US history, for example, I can think of several times in the past century alone that the public sphere was more acrimonious.
I haven't made much of a study of primary sources, but I wouldn't be surprised if a common thing that happened every one of those times was that people would start trying to figure out how to turn "appeal to acrimony" into a convincing debate tactic.
I haven't made much of a study of primary sources, but I wouldn't be surprised if a common thing that happened every one of those times was that people would start trying to figure out how to turn "appeal to acrimony" into a convincing debate tactic.
predictive analytics..?
So phrenology again? IIRC, that idea didn't exactly hold up to scrutiny.
I think it was humor and made me laugh a bit.. phrenology cannot be taken seriously
[deleted]
Internet used to be delivered through unreliable 28.8kbps modems. I didn't think for a second that we should cease using the Internet.
Bone marrow transplants for cancer treatment used to have a 20% failure rate. I didn't think for a second that we should cease using bone marrow transplants.
And yet why does the ACLU think we should cease using a technology to deliver potential location hits on wanted criminals because its not 100% perfect?
Bone marrow transplants for cancer treatment used to have a 20% failure rate. I didn't think for a second that we should cease using bone marrow transplants.
And yet why does the ACLU think we should cease using a technology to deliver potential location hits on wanted criminals because its not 100% perfect?
The problem is that it's only 95% perfect in this case (28 out of 534, being identified as criminals). That a pretty big margin of error when the result is possible arrest. The scope of its use is also in question, sure if you looking a someone who kidnapped a child, 95% is good enough. If you use it for any minor violation you risk harassing a large percentage of the population, who then will need to prove that it wasn't them. Some will flat out deny being the guilty party, and there's a 5% margin of error, so they may very well be right. Then what will you do, drop the charge? In that case what's the point. Or will you spend police time finding evidence that some random person stole $10 worth of good at the super market?
The cost associated with being just 5% wrong is huge, and that's not including the emotional damage done to falsely accused citizens, or the decrease level of trust in law enforcement.
The cost associated with being just 5% wrong is huge, and that's not including the emotional damage done to falsely accused citizens, or the decrease level of trust in law enforcement.
They weren't "identified as criminals." They were identified as possibly being criminals. (And as other posters have shown, they were identified with only 80% confidence as being a criminal.) And an officer can review the evidence presented by the report and make a human determination to follow up with a physical arrest.
Perhaps more of an issue in a country where being arrested can involve being murdered by the police.
> They were identified as possibly being criminals.
And you've highlighted the _exact_ problem here. That is not at all what has happened, they have been identified as "possibly having a similar appearance to one or more particular images of a suspected criminal."
> And an officer can review the evidence presented by the report and make a human determination to follow up with a physical arrest.
Which already presumes that the system will only have false positives. When the system produces false negatives, then this mode of investigation is entirely flawed.
And you've highlighted the _exact_ problem here. That is not at all what has happened, they have been identified as "possibly having a similar appearance to one or more particular images of a suspected criminal."
> And an officer can review the evidence presented by the report and make a human determination to follow up with a physical arrest.
Which already presumes that the system will only have false positives. When the system produces false negatives, then this mode of investigation is entirely flawed.
I’m pretty certain they take issue with law enforcement using the technology that’s not 100% and without oversight because they have the ability to seriously alter someone’s life.
[deleted]
Only 28? Should be a complete 1:1, meaning they all seem crooked in one way or another...
It also possibly highlights the fact that algorithms are not immune to bias when they are designed by humans. Obviously I don’t think this bias is intentional, but so much of it isn’t and happens anyway.