UK passport photo checker shows bias against dark-skinned women(bbc.co.uk)
bbc.co.uk
UK passport photo checker shows bias against dark-skinned women
https://www.bbc.co.uk/news/technology-54349538
48 comments
This is an awesome feature. Of your face, I mean, not of the recognition software.
At this points these complains are bullshit in my opinion. It shifts the debate about if we want to accept facial recognition to the side. It is like a free gift to those that sell these systems.
So if facial recognition is actually bad for minorities, people bringing up these topics should not be their friends. My opinion, I know it is a strong one, but it is also more correct.
Is a passport checker even needed for security would be a better discussion topic.
At this points these complains are bullshit in my opinion. It shifts the debate about if we want to accept facial recognition to the side. It is like a free gift to those that sell these systems.
So if facial recognition is actually bad for minorities, people bringing up these topics should not be their friends. My opinion, I know it is a strong one, but it is also more correct.
Is a passport checker even needed for security would be a better discussion topic.
While this may represent a failure of those systems to adequately cope with data outside of typical ranges, your anecdote is orthogonal to the notion that the system linked in this thread did not appear to place dark-skinned women firmly within those ranges, and thus represents a form of institutional bias, experienced as racism.
Your inconvenient experience differs at a material level, because as a white-skinned man, many other systems are optimised for you.
Your inconvenient experience differs at a material level, because as a white-skinned man, many other systems are optimised for you.
This looks oddly familiar to the medical field. There they often only test on men (1). Do here we see the classic example of not having tested the product on all possible clients.
Also, if they had known about this flaw, they probably had been able to build a submit for manual review option.
(1) https://www.theguardian.com/lifeandstyle/2015/apr/30/fda-cli...
(1) https://www.theguardian.com/lifeandstyle/2015/apr/30/fda-cli...
As we turn more of our lives over to the results of these machine learning algorithms, we need to keep reminding developers that any system that is not designed specifically to be anti-racist can still end up racist due to poor training data. There was a similar issue just a couple weeks back with Twitter. [1] We can't just ignore these issues and assume that an algorithm is inherently unbiased.
[1] - https://news.ycombinator.com/item?id=24530968
[1] - https://news.ycombinator.com/item?id=24530968
I think calling it racist is making a wrong connotation. It is introducing a bias. Racism is something else.
Racism is discrimination based on skin colour, which is literally what this system is doing.
It's a pity they didn't question the Home Office more on what they have done since 2019 to fix the bug since they knew existed from launch — e.g. how many times has the algorithm been retrained since then with good passport photos (since the issue still seems unresolved)
EDIT: Turns out from the New Scientist article linked that it actually launched in 2016, misread from the BBC one.
It's a pity they didn't question the Home Office more on what they have done since 2019 to fix the bug since they knew existed from launch — e.g. how many times has the algorithm been retrained since then with good passport photos (since the issue still seems unresolved)
EDIT: Turns out from the New Scientist article linked that it actually launched in 2016, misread from the BBC one.
I’m not racist, I’m just biased against people with dark skin?
Are you trained with poor training data? My comment was on parent's statement. I did not say racism does not exist, but it is not the only word for situations where there is a bias involving pepole's color.
This is bias against people with darker skin that disadvantages their lives in a way that people with lighter skin do not experience. How is that not an example of racism?
Equalling racism with anything that disadvantages pepole of a skin color, regardless of cause, devalues the word. Racism used to mean that there was a belief that certain races were superior to others, or a belief of that nature involved. It is simply not accurate to apply it to any circumstance regarding discrimination of skin color. You need to look at if there was an intent or a conscious prejudice.
You are talking about racism at the individual level. That compounds itself into institutional or systemic racism of which this is an example. This is a government system that puts more obstacles in the way of darker skinned individuals because during the development of the system, the developers probably never even considered the importance of this system working for anyone but lighter skinned people. Lighter skinned people were important and darker skinned people weren't. That is institutional racism.
Also I'm not aware of any credible authority on the subject that would say "intent or a conscious prejudice" is any type of requirement for something to be considered racist.
Also I'm not aware of any credible authority on the subject that would say "intent or a conscious prejudice" is any type of requirement for something to be considered racist.
You just need to look in a thesaurus or dictionary to find a description of the word along the lines of "the belief that some races are innately superior to others because of hereditary characteristics" (Collins), "the belief that certain races of people are by birth and nature superior to others" (Merriam-Webster) or "Racism is the belief that groups of humans possess different behavioral traits corresponding to physical appearance and can be divided based on the superiority of one race over another." (first sentence on Wikipedia). Core to this, it includes a human judgment. Describing all kinds of discrimination on color to racism is similar to calling involuntary mistakes, or even unanticipated effects of other actions, that breaks things "sabotage".
It sounds like you stopped reading all those sources as soon as you got a definition you liked and didn't bother scrolling to get a complete picture of the term.
From Collins[1]
>any program or practice of racial discrimination, segregation, etc., specif., such a program or practice that upholds the political or economic domination of one race over another or others
From Merriam-Webster[2]:
>the systemic oppression of a racial group to the social, economic, and political advantage of another
From Wikipedia[3]
>Institutional racism (also known as structural racism, state racism or systemic racism) is racial discrimination by governments, corporations, religions, or educational institutions or other large organizations with the power to influence the lives of many individuals. Stokely Carmichael is credited for coining the phrase institutional racism in the late 1960s. He defined the term as "the collective failure of an organization to provide an appropriate and professional service to people because of their colour, culture or ethnic origin".
Racism isn't purely about the belief that one race is superior to another. A system that oppresses one race in favor of another is racist. This example meets the definitions of all three sources.
[1] - https://www.collinsdictionary.com/us/dictionary/english/raci...
[2] - https://www.merriam-webster.com/dictionary/racism
[3] - https://en.wikipedia.org/wiki/Racism#Institutional
From Collins[1]
>any program or practice of racial discrimination, segregation, etc., specif., such a program or practice that upholds the political or economic domination of one race over another or others
From Merriam-Webster[2]:
>the systemic oppression of a racial group to the social, economic, and political advantage of another
From Wikipedia[3]
>Institutional racism (also known as structural racism, state racism or systemic racism) is racial discrimination by governments, corporations, religions, or educational institutions or other large organizations with the power to influence the lives of many individuals. Stokely Carmichael is credited for coining the phrase institutional racism in the late 1960s. He defined the term as "the collective failure of an organization to provide an appropriate and professional service to people because of their colour, culture or ethnic origin".
Racism isn't purely about the belief that one race is superior to another. A system that oppresses one race in favor of another is racist. This example meets the definitions of all three sources.
[1] - https://www.collinsdictionary.com/us/dictionary/english/raci...
[2] - https://www.merriam-webster.com/dictionary/racism
[3] - https://en.wikipedia.org/wiki/Racism#Institutional
No it doesn't. It just widens the discussion to also include what oppression means.
Your argument puts "poor training data", which I originally commented on, in the same bin as for example apartheid. Widen it to that use devalues the word.
I have made my point and have no interest to continue this discussion further.
Your argument puts "poor training data", which I originally commented on, in the same bin as for example apartheid. Widen it to that use devalues the word.
I have made my point and have no interest to continue this discussion further.
>It just widens the discussion to also include what oppression means.
Those first two quotes are literally dictionary definitions of racism. You are free to disagree, but you aren't just disagreeing with me. You are disagreeing with the dictionary.
Those first two quotes are literally dictionary definitions of racism. You are free to disagree, but you aren't just disagreeing with me. You are disagreeing with the dictionary.
[deleted]
> Racism used to mean that there was a belief [...]
Yes, the key word is "used". And the point why it isn't is because we pay now more attention to the target of racism than the perpetrator.
> You need to look at if there was an intent or a conscious prejudice.
Or alternatively, you need to look at what happens to the people being subjected to racism. Practically no one wants to be seen as a racist anymore. Intent and conscious prejudice is now hidden.
In my eyes, that is progress. We move forward from assigning guilt to trying to improve the situation for people subjected to racism.
I kind of doubt, that it will mean, more people would be happy or neutral being called a racists.
Yes, the key word is "used". And the point why it isn't is because we pay now more attention to the target of racism than the perpetrator.
> You need to look at if there was an intent or a conscious prejudice.
Or alternatively, you need to look at what happens to the people being subjected to racism. Practically no one wants to be seen as a racist anymore. Intent and conscious prejudice is now hidden.
In my eyes, that is progress. We move forward from assigning guilt to trying to improve the situation for people subjected to racism.
I kind of doubt, that it will mean, more people would be happy or neutral being called a racists.
Doorways in Japan tend to be biased against 6'2" people. Is that height discrimination?
That story was flagged, what a sad state of the industry. It’s no wonder we have terrible racist tools when we aren’t even able to acknowledge it.
And this story was the number 1 story on HN when I posted that comment 10 minutes ago and it has now been flagged down to number 37 and off the front page. I will repeat what I said in that thread. HN doesn't like talking about race and racism.
Probably because many people don't see this as racism.
The same app couldn't detect my babies head. He's white. Even with two sets of image correction in the photo shop, we couldn't get it work.
At what point can you say it's not racist, just not very good?
At what point can you say it's not racist, just not very good?
They seem to simply take it on faith that it's impossible for darker-skinned people to just have a higher chance of taking poor photos. This could be mediated by culture of income or a hundred other things.
They offer no way to evaluate the statistical validity of the conclusions drawn from "BBC Research". They looked at 1000 politicians, but what was the color breakdown of those? What was the actual sample size of dark-skinned people? Who coded the photos' skin colors and how? This is just utter garbage without any rigor at all.
Their "BBC Research" indicates that male faces read much better than female, but they don't comment on this at all. Why did they choose the race angle and not the sex angle?
Moreover there are real reasons why dark-skinned features are harder to read than light-skinned. It's the same reason things are hard to make out in the dark, or you can't tell the features of an object covered in vantablack. It's physics. Less light means less information.
This whole article is the perfect distillation of low-effort motivated race baiting. To an astute reader it proves the opposite of its bias - if this is what is considered racial greivance worthy of breathless BBC articles, there must really be nothing significant going on to complain about.
There is zero chance that anything like this would ever have been published if the results went the other way, no matter to what degree.
They offer no way to evaluate the statistical validity of the conclusions drawn from "BBC Research". They looked at 1000 politicians, but what was the color breakdown of those? What was the actual sample size of dark-skinned people? Who coded the photos' skin colors and how? This is just utter garbage without any rigor at all.
Their "BBC Research" indicates that male faces read much better than female, but they don't comment on this at all. Why did they choose the race angle and not the sex angle?
Moreover there are real reasons why dark-skinned features are harder to read than light-skinned. It's the same reason things are hard to make out in the dark, or you can't tell the features of an object covered in vantablack. It's physics. Less light means less information.
This whole article is the perfect distillation of low-effort motivated race baiting. To an astute reader it proves the opposite of its bias - if this is what is considered racial greivance worthy of breathless BBC articles, there must really be nothing significant going on to complain about.
There is zero chance that anything like this would ever have been published if the results went the other way, no matter to what degree.
Definitely agree the level of detail in the article wouldn't get published in an academic journal, though they do mention:
"Kirstie Whitaker, programme lead for tools, practices and systems at the Alan Turing Institute, reviewed our code, independently reproduced our results, and provided input on the validity of the reported outcomes."
Would be interesting to see a proper write-up of the same study.
That said, let's say that it is taken as given that it's harder for a ML set-up to determine good photos for dark skin. Should it be used at all then? Is it acceptable that it's harder for the ~15% of the population to get a passport than the rest of the population because of an algorithm in the loop?
I don't think you can put it down to other factors like socio-economics as the Home Office reproduced this in their user testing before launch and acknowledged as a problem of the system (Vs the photos)
"Kirstie Whitaker, programme lead for tools, practices and systems at the Alan Turing Institute, reviewed our code, independently reproduced our results, and provided input on the validity of the reported outcomes."
Would be interesting to see a proper write-up of the same study.
That said, let's say that it is taken as given that it's harder for a ML set-up to determine good photos for dark skin. Should it be used at all then? Is it acceptable that it's harder for the ~15% of the population to get a passport than the rest of the population because of an algorithm in the loop?
I don't think you can put it down to other factors like socio-economics as the Home Office reproduced this in their user testing before launch and acknowledged as a problem of the system (Vs the photos)
Almost all passport photos in the UK are taken via approved automatic booths.
Hardly anyone takes their own passport photos. Only the very rich would get a professional photographer.
Hardly anyone takes their own passport photos. Only the very rich would get a professional photographer.
[deleted]
Is this really a bias? Or maybe they are less facial features and less contrast, that make detection harder? For example there is only one eye color and hair color.. And it was fixed with simple appeal.
Rant start >
I think this just derails discussion about how system is really broken. My friend applied for UK residency around Christmas. She matches all criteria, her husband studied and now works in UK, she is EU resident. Border service holds her passport for 11 months now!!! Only info we can get is automated paid call, for 15 euro they tell us it is in progress...
Meanwhile she got pregnant and is about to deliver. All faster before border service can even answer single question... She is about to deliver, 2000 km from her husband, and can not even leave this country, because UK is holding her passport. And yes, she happens to be black as well...
...rant end
Rant start >
I think this just derails discussion about how system is really broken. My friend applied for UK residency around Christmas. She matches all criteria, her husband studied and now works in UK, she is EU resident. Border service holds her passport for 11 months now!!! Only info we can get is automated paid call, for 15 euro they tell us it is in progress...
Meanwhile she got pregnant and is about to deliver. All faster before border service can even answer single question... She is about to deliver, 2000 km from her husband, and can not even leave this country, because UK is holding her passport. And yes, she happens to be black as well...
...rant end
Hear hear, this is what racism is, not an algorithm that underperforms in certain situations.
For any algo, I'm sure you'll find features where it underperforms, and since, well, photos are 90% race-determined, of course there will be racial differences.
Bureaucracy doing its best to not deliver what the law says it should is, meanwhile, blatant racism. Just sounds less exciting than a Skynet-style racist operation of white men forcing Asian women to join a different queue at the airport.
For any algo, I'm sure you'll find features where it underperforms, and since, well, photos are 90% race-determined, of course there will be racial differences.
Bureaucracy doing its best to not deliver what the law says it should is, meanwhile, blatant racism. Just sounds less exciting than a Skynet-style racist operation of white men forcing Asian women to join a different queue at the airport.
> Is this really a bias?
Not making your software function properly for all the legitimate input it has to deal with is a bias. Not working to eliminate that bias because it affects a group you don't happen to be part of could even be called racist.
Not making your software function properly for all the legitimate input it has to deal with is a bias. Not working to eliminate that bias because it affects a group you don't happen to be part of could even be called racist.
The vast majority of software has bugs especially around edge cases that aren't frequently encountered. Sometimes it isn't even worth fixing somethings as they are expected so infrequently, it's easier to fix things by hand or via some other hack.
Are you asserting that people with dark skin need photo ID more “infrequently” than others? That it “isn't even worth fixing”? Then that's how the bias in your thinking translates directly into bad and racist software.
You are arguing for the developer's convenience. Good software is about providing functionality to users, not developer's convenience.
You are arguing for the developer's convenience. Good software is about providing functionality to users, not developer's convenience.
That's straw-manning my argument, I didn't say any of that. But as you bring it up, so far this case has come up once, so it does seem to be a rare edge case.
Perfection is the enemy of good and many bugs get relabelled features (and no I am not doing that in this case, just pointing out that software development always involves trade offs).
And stop calling it racist. Racist used to be something pretty bad, now it's being applied to everything.
Perfection is the enemy of good and many bugs get relabelled features (and no I am not doing that in this case, just pointing out that software development always involves trade offs).
And stop calling it racist. Racist used to be something pretty bad, now it's being applied to everything.
> Dark-skinned women are told their photos are poor quality 22% of the time, while the figure for light-skinned women is 14%
That doesn't surprise me even a tiny bit, given that the problem will already present itself in the camera - exposure settings are calibrated for white-ish people, and automated photo booths don't have a human to correct it or offer multiple different exposures to the user. Scanning the photo has the same problem.
That doesn't surprise me even a tiny bit, given that the problem will already present itself in the camera - exposure settings are calibrated for white-ish people, and automated photo booths don't have a human to correct it or offer multiple different exposures to the user. Scanning the photo has the same problem.
There is definitely something to be said about the quality of pictures taken in photo booths, but the study specifically used pictures that were all of sufficiently high quality:
> Photos of politicians were collected from parliaments around the world. The gender and skin tone were recorded for each photo using the Fitzpatrick skin-tone scale. Each image met the Home Office standards for a passport photo.
That definitely suggests the software used for automatic image quality check simply isn't up to par.
> Photos of politicians were collected from parliaments around the world. The gender and skin tone were recorded for each photo using the Fitzpatrick skin-tone scale. Each image met the Home Office standards for a passport photo.
That definitely suggests the software used for automatic image quality check simply isn't up to par.
While this sounds plausible - yet simply shifts the blame for the bias elsewhere in the technology supply chain - it does not adequately explain the experience of Elaine Owusu cited in the article.
Her photo shows a clear-as-day mouth closed, which was repeatedly interpreted by the system as mouth open.
No exposure settings would change that outcome.
Her photo shows a clear-as-day mouth closed, which was repeatedly interpreted by the system as mouth open.
No exposure settings would change that outcome.
[deleted]
Can someone with facial recognition experience chime in?
I always thought that it's just harder to extract facial features from an objectively darker image. Or is it real bias in the algo or the training set of photos?
I always thought that it's just harder to extract facial features from an objectively darker image. Or is it real bias in the algo or the training set of photos?
> I always thought that it's just harder to extract facial features from an objectively darker image.
Amateur photographer here. The key thing is to ask why the image of PoC is "objectively darker", and the answer to that is because exposure meters/auto programs in cameras are calibrated for correctly exposing white people.
Amateur photographer here. The key thing is to ask why the image of PoC is "objectively darker", and the answer to that is because exposure meters/auto programs in cameras are calibrated for correctly exposing white people.
Um no? For more colour correction, you can use the vectorscope to target the universal skin tone line which fixes contrast for all skin tones.
I'm talking about these "photo booths". A professional photographer of course can do a color correction in Photoshop, but there is no human involved in these.
Or maybe it's harder to find contrast in a darker skinned face in bad lighting.
either in-competent programmers that coded the system or bad training data that only had non-dark skinned pictures for training the system
solution is clear: do not use white racist computers to evaluate passport photos.
joking aside - better testing might help
joking aside - better testing might help
Clear but not easy. I’d be unpleasantly surprised if this is caused by someone having a line saying “if black & rand()<0.15 { reject() }”… although given the recent thing with the 2^16-1 Excel limit for Covid reporting, it certainly isn’t impossible for a U.K. system to be that badly wrong.
As a Swedish/Finnish man (and a somewhat stereotypical nerd at that), I'm paler than probably 95% of the world population.
Although facial recognition features (in the sense of trained AI models) often work fine on me given a good quality underlying photo, what often fails instead are preprocessing steps or color correction, especially in real-time systems. This can be the camera itself or software handling the image later.
For example if I'm in a Google Hangout in a normally lit room and then I move into slight sunlight or a lamp, my face will overexpose into a blob that looks like a low quality photo of the sun. It renders white, like #ffffff proper white. It's not actually overexposed at the hardware/camera level, but the software "corrects" it that way.
This typically never happens to other people.