Bias detectives: the researchers striving to make algorithms fair(nature.com)
nature.com
Bias detectives: the researchers striving to make algorithms fair
https://www.nature.com/articles/d41586-018-05469-3
50 comments
ProPublica's work on algorithmic bias has all seemed well below their usual standards. I haven't followed this rebuttal, but their work showing racism in car insurance pricing was heavily criticized, and while the authors defended the work it looked to me like they picked only the weakest criticisms to respond to.
(In the insurance case, ProPublica attempted to compare areas with comparable crash frequencies and show that rates were higher in poor and minority areas. But the data they had was moving accidents, especially with injuries, and the data they didn't have was stuff like "rate of car break-ins" and "odds of being hit by an uninsured driver". Which you would obviously expect to vary by region even when serious-injury accidents don't.)
(In the insurance case, ProPublica attempted to compare areas with comparable crash frequencies and show that rates were higher in poor and minority areas. But the data they had was moving accidents, especially with injuries, and the data they didn't have was stuff like "rate of car break-ins" and "odds of being hit by an uninsured driver". Which you would obviously expect to vary by region even when serious-injury accidents don't.)
> racist AI" is the kind of headline that's just too golden to abandon.
TBF there is at least one leading ML scientist that has made a huge narrative on discriminating AI, particularly against women.
TBF there is at least one leading ML scientist that has made a huge narrative on discriminating AI, particularly against women.
The purpose of Machine Learning is to generalize on a large scale. I like to think of it as the equivalent of someone who has years of experience in a particular area. Someone who has been doing something for years has seen so much, that they can know in an instant what a situation is based on clues and generalizations. It wouldn't be fast if it wasn't generalizing.
If you want to claim you know what fair is in any given situation, then go and hardcode all your own fair rules, because you aren't going to find "fairness" in machine learning.
If you want to claim you know what fair is in any given situation, then go and hardcode all your own fair rules, because you aren't going to find "fairness" in machine learning.
[deleted]
> If you want to claim you know what fair is in any given situation, then go and hardcode all your own fair rules
This strikes me as eerily similar to the argument that type systems are impossible because of the halting problem. It's sort of true in some sense, but not in an even remotely useful way. So it mostly functions as a way of derailing the conversation away of the more subtle distinctions that do matter (e.g., could we design an easy-to-use type system that rules out this particular type of non-termination/other class of bugs).
There's a large middle-ground between "hard code all your own rules" and "completely unconstrained learning". Learning under constraints is not a new idea.
A classical programming analogy to your argument might be "well the halting problem is undecidable so ignore all this high-level language stuff and just go code up your own turing machine; it's the best you'll ever be able to do".
> because you aren't going to find "fairness" in machine learning.
Why not? The human notion of fairness is fuzzy, which is why researchers have provided various formal notions of fairness in machine learning tasks. Obviously, these formal definitions may or may not correspond to your own gut instinct about what is "fair". And there might be friction between different notions of fairness. None of that should be surprising; otherwise, fairness wouldn't be something that philosophers continue to bleed ink about.
But it is equally obvious that for some notions of fairness, there will exist machine learning algorithms that learn well under the given constraint.
This strikes me as eerily similar to the argument that type systems are impossible because of the halting problem. It's sort of true in some sense, but not in an even remotely useful way. So it mostly functions as a way of derailing the conversation away of the more subtle distinctions that do matter (e.g., could we design an easy-to-use type system that rules out this particular type of non-termination/other class of bugs).
There's a large middle-ground between "hard code all your own rules" and "completely unconstrained learning". Learning under constraints is not a new idea.
A classical programming analogy to your argument might be "well the halting problem is undecidable so ignore all this high-level language stuff and just go code up your own turing machine; it's the best you'll ever be able to do".
> because you aren't going to find "fairness" in machine learning.
Why not? The human notion of fairness is fuzzy, which is why researchers have provided various formal notions of fairness in machine learning tasks. Obviously, these formal definitions may or may not correspond to your own gut instinct about what is "fair". And there might be friction between different notions of fairness. None of that should be surprising; otherwise, fairness wouldn't be something that philosophers continue to bleed ink about.
But it is equally obvious that for some notions of fairness, there will exist machine learning algorithms that learn well under the given constraint.
I agree with you when you say fairness is fuzzy. Though this implies that there are no set of rules that define it, that also means there is no way to train on it. By choosing the right features and utilizing it in the right way I believe you can avoid putting people in bad situations for bad reasons. I'm saying, if you train an algorithm to guess if someone is involved with crime, it's going to be incredibly stereotypical with it's answers. Same as if you rely completely on statistics.
I'm not actually saying hardcore all your rules. I was making an example that if you really know exactly what fairness is, then program it. But we both don't know there are no definite rules. So why hardcode at all?
I think it is never the ML that is "unfair", it's the one who made it who is responsible. If you're finding yourself running into issues in your ML with unfairness, I think you're just using it wrong.
EDIT: Rewording last sentence
I'm not actually saying hardcore all your rules. I was making an example that if you really know exactly what fairness is, then program it. But we both don't know there are no definite rules. So why hardcode at all?
I think it is never the ML that is "unfair", it's the one who made it who is responsible. If you're finding yourself running into issues in your ML with unfairness, I think you're just using it wrong.
EDIT: Rewording last sentence
> The human notion of fairness is fuzzy
... to the point that every attempt to define "fairness" in one (arguable) aspect leads to (codified, inescapable) unfairness in another.
... to the point that every attempt to define "fairness" in one (arguable) aspect leads to (codified, inescapable) unfairness in another.
The problem here is that modern machine learning is rarely more than a computerized way of building statistical models based on observed data. And the bias mitigation techniques I've read about all revolve around (indirectly) manipulating those models based on what results they produce. This can lead to various problems in the long run.
When I was reading through some algorithmic fairness literature some time ago, I came back a bit frustrated because as the article mentions, the fairness definitions are mutually incompatible (though some seem more plausible than others) and it's not really a problem that can be fully solved on a technical level. The only flicker of hope was that a perfect classifier can, by some definitions, be considered fair, so at least you have something to work with - if your classifier discriminates by gender or other attributes, you should at least make it good enough to back up its bias by delivering perfect accuracy (at which point you can investigate why inherit differences between groups seem to exist).
It's good that some Computer Science researchers are ready to work in such politicized fields though, it's definitely necessary. I find it admirable because I personally wouldn't enjoy those discussions.
It's good that some Computer Science researchers are ready to work in such politicized fields though, it's definitely necessary. I find it admirable because I personally wouldn't enjoy those discussions.
If unconstrained learning emits biased result it was given biased samples. That or the bias is in the dataset itself, which could still be fixed by removing and randomizing traits but at which point your alghorithm is learning a representation if reality which usefulness depend on the realm if application, say, great for university admittance and not so great for medical insurance purposes.
You assume that the underlying data cannot be correct and not fair.
uh, no?
> [then] it was given biased samples
> [then] it was given biased samples
Well, whether or not (and if so, to what extent) unconscious bias taints engineering or scientific research is an interesting, and potentially important question to answer, it seems to me that the only people who are addressing the question are people who are overtly, consciously biased themselves - to the point where they actively exclude anybody who doesn't share their own set of biases.
In summary: “You can’t have it all. If you want to be fair in one way, you might necessarily be unfair in another definition that also sounds reasonable.”
It reminds me of Arrow’s Impossibility Theorem [0].
David Deutsch had an anecdote in his book The Beginning of Infinity about explaining Arrow’s theorem to a US congressman and eventually getting him to the point of understanding how it applies to the electoral college in principle and there is no simple legislative change that could mollify it (I think the discussion was about how preferential voting would improve upon FPTP voting).
The congressman replied something like “this is lamentable” and Deutsch wrote a big passage about whether it makes sense that anyone should ever find a mathematical fact to be “lamentable.”
I do think there will be pragmatic varieties of this sort of impossibility theorem for machine learning fairness and that society in a broad sense will have a hard time processing it, and the possible legislative reactions might be totally unreasonable, even unintentionally harmful.
[0]: < https://en.m.wikipedia.org/wiki/Arrow's_impossibility_theore... >
David Deutsch had an anecdote in his book The Beginning of Infinity about explaining Arrow’s theorem to a US congressman and eventually getting him to the point of understanding how it applies to the electoral college in principle and there is no simple legislative change that could mollify it (I think the discussion was about how preferential voting would improve upon FPTP voting).
The congressman replied something like “this is lamentable” and Deutsch wrote a big passage about whether it makes sense that anyone should ever find a mathematical fact to be “lamentable.”
I do think there will be pragmatic varieties of this sort of impossibility theorem for machine learning fairness and that society in a broad sense will have a hard time processing it, and the possible legislative reactions might be totally unreasonable, even unintentionally harmful.
[0]: < https://en.m.wikipedia.org/wiki/Arrow's_impossibility_theore... >
[deleted]
Note that there's not only a potential selection bias problem, but a feedback issue as well. For instance, if the algorithm is biased toward assigning higher criminal activity risk to black people, the black people will be more likely to be checked and, as a consequence, the future versions of such algorithms will be even more biased in the same direction. Debiasing in such situations is a very tough endeavour.
A lot hinges on how you define fairness. Is it unfair if a higher percentage of <racial group> is flagged as “criminal activity risks”, to use your example, than of <other racial group>? What if it aligns with reality?
No matter how accurate your metric is—nay, because of its accuracy—people will be mad at you because it reflects an underlying reality they are in denial of or believe is itself unfair, and want to rectify by requiring fictions elsewhere.
I don’t envy people working on this because you by definition can’t win. Some of the powerful political forces, on the one hand, and the demands for accuracy and effectiveness, on the other, are irreconcilable.
No matter how accurate your metric is—nay, because of its accuracy—people will be mad at you because it reflects an underlying reality they are in denial of or believe is itself unfair, and want to rectify by requiring fictions elsewhere.
I don’t envy people working on this because you by definition can’t win. Some of the powerful political forces, on the one hand, and the demands for accuracy and effectiveness, on the other, are irreconcilable.
That's why the grandparent mentioned feedback. Even if a disparity "aligns with reality", if the algorithm is used in a way that reinforces the disparity, then it's discriminatory. When designing algorithms as parts of systems, we need to be careful to not ossify statistics we're ostensibly just reflecting.
> Is it unfair if a higher percentage of <racial group> is flagged as “criminal activity risks”, to use your example, than of <other racial group>? What if it aligns with reality?
The article is about the unfairness of a higher percentage of <racial group> getting incorrectly flagged as “criminal activity risks”. The fairness of the expected output is essentially out of scope for this kind of research, it's all about the distribution of correct vs. erroneous decisions. The underlying reality is an imbalance in data quality or similar, not whatever you were thinking about.
The article is about the unfairness of a higher percentage of <racial group> getting incorrectly flagged as “criminal activity risks”. The fairness of the expected output is essentially out of scope for this kind of research, it's all about the distribution of correct vs. erroneous decisions. The underlying reality is an imbalance in data quality or similar, not whatever you were thinking about.
This means it's a lot more dangerous to use machine learning for crimes like jaywalking where punishment is discretionary than crimes like murder or jumping bail where the crime will pretty much always be noted and investigated.
Seeing it applied to the chance of violating probation or reoffending after a minor crime alarms me for exactly this reason - lots of people commit minor probation violations (e.g. associating with criminals) or crimes (e.g. marijuana possession) and are never caught or charged.
Even if this is a case of "the data is biased, not the algorithm", it's important to remember that the data is "who was caught doing this" rather than "who did this". Concluding that people who live in stop-and-frisk areas are more likely to get drug convictions isn't actually wrong, but it's still unhelpful.
Even if this is a case of "the data is biased, not the algorithm", it's important to remember that the data is "who was caught doing this" rather than "who did this". Concluding that people who live in stop-and-frisk areas are more likely to get drug convictions isn't actually wrong, but it's still unhelpful.
Isn't the whole point of machine algorithms to find the best spot between variance and bias? In which case, every algorithms will have some bias.
IMO, the focus should instead be on not overselling algorithms as being infallible rather something which will have some bias and needs overriding time to time. If a system is fully automated without checks and balances we might have serious problems. A good non-ML example was discussed couple of days ago on HN where a person was terminated by a machine without much oversight:
https://idiallo.com/blog/when-a-machine-fired-me
IMO, the focus should instead be on not overselling algorithms as being infallible rather something which will have some bias and needs overriding time to time. If a system is fully automated without checks and balances we might have serious problems. A good non-ML example was discussed couple of days ago on HN where a person was terminated by a machine without much oversight:
https://idiallo.com/blog/when-a-machine-fired-me
>Isn't the whole point of machine algorithms to find the best spot between variance and bias? In which case, every algorithms will have some bias.
This references two different, almost opposite, meanings of the word bias. The meaning in TFA is being individually subject to judgements about groups (eg an individual AA prisoner being denied parole because AAs as a population have higher rates of recidivism) whereas in the context of ML it refers to a model ignoring important features of a dataset.
This references two different, almost opposite, meanings of the word bias. The meaning in TFA is being individually subject to judgements about groups (eg an individual AA prisoner being denied parole because AAs as a population have higher rates of recidivism) whereas in the context of ML it refers to a model ignoring important features of a dataset.
Aren't the algorithms already reasonably fair? The researchers are just trying to get them to be politically correct?
My understanding is that it isn’t the algorithm that’s biased per se, it’s the training data. If, for example, in the United States a black person were more likely to be investigated and prosecuted than a person of another race, then the crime statistics would be biased. If these statistics were then fed into a machine learning model, that model would actually be predicting an individual’s likelihood of conviction within the current system, rather than the likelihood of their actually committing a crime. I’m not even making the claim here that this is the case, but it is a reasonable example.
Garbage in, garbage out.
Garbage in, garbage out.
Before throwing around accusations of political correctness you should consider that questions like that are not new, they have just received more attention now that algorithms decide much more things in life than in the past.
For instance, US law has "disparate impact" provisions at least since the civil rights act (a time which was probably not dominated by political correctness), which requires outcomes to be not too different between races or other groups.
(Though disparate impact is not a particularly good metric of fairness and doesn't seem to be used so much in algorithmic fairness nowadays.)
For instance, US law has "disparate impact" provisions at least since the civil rights act (a time which was probably not dominated by political correctness), which requires outcomes to be not too different between races or other groups.
(Though disparate impact is not a particularly good metric of fairness and doesn't seem to be used so much in algorithmic fairness nowadays.)
> For instance, US law has "disparate impact" provisions at least since the civil rights act (a time which was probably not dominated by political correctness), which requires outcomes to be not too different between races or other groups.
That's not what “disparate impact” requires; it requires that acts producing differential outcomes for protected classes are demonstrably tightly aligned with a legitimate interest (e.g., in hiring, criteria with such an impact must be demonstrably tightly tied to job performance.)
Differences, even extremely wide differences, in outcomes associated with protected class are still legal, if they have a demonstrably legitimate basis.
That's not what “disparate impact” requires; it requires that acts producing differential outcomes for protected classes are demonstrably tightly aligned with a legitimate interest (e.g., in hiring, criteria with such an impact must be demonstrably tightly tied to job performance.)
Differences, even extremely wide differences, in outcomes associated with protected class are still legal, if they have a demonstrably legitimate basis.
The disparate impact doctrine is quite controversial, especially the ever more aggressive applications of it.
It has led to some unfortunate rules. For example, it’s (generally and presumptively) illegal to hire based on intelligence tests, but seems to be okay (in practice) to hire only from elite universities that select students largely on the basis of SAT scores, which correlate very strongly with IQ.
It has led to some unfortunate rules. For example, it’s (generally and presumptively) illegal to hire based on intelligence tests, but seems to be okay (in practice) to hire only from elite universities that select students largely on the basis of SAT scores, which correlate very strongly with IQ.
> The disparate impact doctrine is extremely controversial, especially the ever more aggressive applications of it.
What's mostly controversial is fictitious applications of it.
> For example, it’s (generally and presumptively) illegal to hire based on intelligence tests, but totally okay to hire only from elite universities that select students largely on the basis of SAT scores, which correlate very strongly with IQ.
No, it's not. Both policies have differential impact, and both have the same requirement of a tight connection to job performance under disparate impact analysis. The main relevant differences are:
(1) There is an widespread cargo cult belief, including among many people involved in hiring, that IQ tests are categorically illegal in hiring, so people avoid them in part based on a prohibition that does not actually exist, and
(2) The same false belief above also is common among workers, so they are particularly likely to seek legal counsel if they experience an adverse hiring decision and were subjected to an IQ test; at which point they are likely to discover the actual disparate impact rule. People who have adverse results and were, even overtly, subjected to another criteria which might invoke disparate impact analysis are less likely to seek a remedy.
(3) IQ tests are overt and the failed applicant is unlikely to be ignorant of the standard used; places that filter strictly by elite universities rarely disclose that, they just require a resume and apply opaque criteria to it. Even if those criteria (such as an elite university filter) would provoke disparate impact analysis, it is difficult for an injured party to know what criteria were used (and to prove what criteria were used even if they know or suspect that one was used that would be subject to disparate impact analysis.)
What's mostly controversial is fictitious applications of it.
> For example, it’s (generally and presumptively) illegal to hire based on intelligence tests, but totally okay to hire only from elite universities that select students largely on the basis of SAT scores, which correlate very strongly with IQ.
No, it's not. Both policies have differential impact, and both have the same requirement of a tight connection to job performance under disparate impact analysis. The main relevant differences are:
(1) There is an widespread cargo cult belief, including among many people involved in hiring, that IQ tests are categorically illegal in hiring, so people avoid them in part based on a prohibition that does not actually exist, and
(2) The same false belief above also is common among workers, so they are particularly likely to seek legal counsel if they experience an adverse hiring decision and were subjected to an IQ test; at which point they are likely to discover the actual disparate impact rule. People who have adverse results and were, even overtly, subjected to another criteria which might invoke disparate impact analysis are less likely to seek a remedy.
(3) IQ tests are overt and the failed applicant is unlikely to be ignorant of the standard used; places that filter strictly by elite universities rarely disclose that, they just require a resume and apply opaque criteria to it. Even if those criteria (such as an elite university filter) would provoke disparate impact analysis, it is difficult for an injured party to know what criteria were used (and to prove what criteria were used even if they know or suspect that one was used that would be subject to disparate impact analysis.)
Disparate impact applies to actions which are unjustified. For instance imagine I decide to never hire somebody who likes rap music. That would have a disparate impact on a certain group of people, yet since it probably has nothing to do with the job I'm hiring for, it would be unjustified and could be argued to be a form of disparate impact discrimination.
By contrast imagine for a labor job I decide to never hire anybody who can't left and carry at least 120 pounds. That would also have a disparate effect negatively impacting a protected class, but it would be justified. Machine learning takes all data and draws conclusions that map strongly against the data and ideally generalize to new scenarios. So long as the behavior trying to be predicted for was relevant to the task at hand, using the recommendations of such algorithms would certainly be justified and thus not fall under disparate impact.
In a nutshell disparate impact is not to ensure equality of result, but to prevent discrimination by proxy. And discrimination not being of the form 'I'm not going to hire women because they can't lift as much as men' but of the form 'I'm not going to hire women because I don't like working with women.'
By contrast imagine for a labor job I decide to never hire anybody who can't left and carry at least 120 pounds. That would also have a disparate effect negatively impacting a protected class, but it would be justified. Machine learning takes all data and draws conclusions that map strongly against the data and ideally generalize to new scenarios. So long as the behavior trying to be predicted for was relevant to the task at hand, using the recommendations of such algorithms would certainly be justified and thus not fall under disparate impact.
In a nutshell disparate impact is not to ensure equality of result, but to prevent discrimination by proxy. And discrimination not being of the form 'I'm not going to hire women because they can't lift as much as men' but of the form 'I'm not going to hire women because I don't like working with women.'
No. Take the example of the algorithm designed to tell whether people charged with a crime would commit another crime within two years:
> Comparing black and white defendants, the journalists found that a disproportionate number of black defendants were ‘false positives’: they were classified by COMPAS as high risk but subsequently not charged with another crime.
It goes on to note that there are other potential measures of bias, such as false negatives, and that external factors such as different arrest rates may bias the data. But it's clear that this issue is more complex than "researchers are trying to get the algorithms to be politically correct".
> Comparing black and white defendants, the journalists found that a disproportionate number of black defendants were ‘false positives’: they were classified by COMPAS as high risk but subsequently not charged with another crime.
It goes on to note that there are other potential measures of bias, such as false negatives, and that external factors such as different arrest rates may bias the data. But it's clear that this issue is more complex than "researchers are trying to get the algorithms to be politically correct".
The COMPAS analysis they refer to is exactly one of those cases where there is no bias whatsoever and they're trying to force the algorithm to conform to PC instead of reality.
What the ProPublica analysis showed was that a black person and a white person with the same risk score have the same probability of recidivism. There is no racial bias.
Black people have higher rates of recidivism, which means that they will also have a higher rate of false positives. This is mathematically unavoidable. I recommend creating a toy example in excel and playing around with some numbers to get an intuition for it.
Also check out this analysis: https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
What the ProPublica analysis showed was that a black person and a white person with the same risk score have the same probability of recidivism. There is no racial bias.
Black people have higher rates of recidivism, which means that they will also have a higher rate of false positives. This is mathematically unavoidable. I recommend creating a toy example in excel and playing around with some numbers to get an intuition for it.
Also check out this analysis: https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
If an algorithm matches or models an inherently unfair real world situation, it doesn't make the algorithm fair.
> ‘false positives’: [...] classified by COMPAS as high risk but subsequently not charged with another crime
It doesn't make sense to say that someone classified as "high risk" which is not charged again was misclassfied. Unless "high risk" is supposed to mean "100% probability of being charged with another crime in the next two years".
We wouldn't say that patients with "high risk" of a heart attack were misclassified because they didn't have a heart attack. If we applied a similar reasoning, we would conclude that cardiology guidelines are biased against obese people.
It doesn't make sense to say that someone classified as "high risk" which is not charged again was misclassfied. Unless "high risk" is supposed to mean "100% probability of being charged with another crime in the next two years".
We wouldn't say that patients with "high risk" of a heart attack were misclassified because they didn't have a heart attack. If we applied a similar reasoning, we would conclude that cardiology guidelines are biased against obese people.
That is not correct; risk is quantifiable. If an algorithm defines "high risk" as an 80% chance of having a heart attack, and only 60% of individuals classified as such end up having one, then we can say that 25% of whom the algorithm classifies as "high risk" are misclassified.
Another issue is that the classification can change future behavior. For example, if a new system says you're at high risk of a heart attack then you're likely to want to take medication, change your diet, etc. From that point of view the algorithm makes itself invalid, but it is still useful.
There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly.
I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS.
For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compa...
There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly.
I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS.
For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compa...
That’s different: you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm. And I can agree with that, but the argument in the article is not about miscalibration.
Yes it is? The article is about bias in algorithms, which skews results. In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set. Nowhere is it argued that individual samples can be explained probabilistically.
I'm sorry, I don't follow you.
> In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set.
https://www.propublica.org/article/how-we-analyzed-the-compa...
"black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)."
If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should.
Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision.
If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10.
> Nowhere is it argued that individual samples can be explained probabilistically.
I don't understand what you mean.
> In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set.
https://www.propublica.org/article/how-we-analyzed-the-compa...
"black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)."
If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should.
Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision.
If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10.
> Nowhere is it argued that individual samples can be explained probabilistically.
I don't understand what you mean.
You said
> you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm
But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way.
> They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision.
They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below:
> Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent).
In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones.
Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk.
(I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.)
I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
> you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm
But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way.
> They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision.
They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below:
> Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent).
In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones.
Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk.
(I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.)
I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
Consider the following situation:
There are two groups, A and B. For both groups, when a person is labeled high risk they have a 75% risk of reoffending (ie no bias).
10% of Group A people are labeled high risk. Which means that (1 - 75%) * 10% = 2.5% are misclassified.
20% of Group B people are labeled high risk. Which means that (1 - 75%) * 20% = 5% are misclassified.
As you can see, this is simply an artifact of the higher reoffending rate of Group B. If 1) the two groups have different reoffending rates, and 2) the algorithm for labeling people high risk is unbiased, it necessarily follows that the two groups will have different misclassification rates.
The entire ProPublica piece (and your arguments) is based on misunderstanding this. Part of the issue is that extremely misleading table titled "Prediction Fails Differently for Black Defendants", where "Labeled Higher Risk, But Didn’t Re-Offend" actually refers to P(labeled higher risk | didn't re-offend).
There are two groups, A and B. For both groups, when a person is labeled high risk they have a 75% risk of reoffending (ie no bias).
10% of Group A people are labeled high risk. Which means that (1 - 75%) * 10% = 2.5% are misclassified.
20% of Group B people are labeled high risk. Which means that (1 - 75%) * 20% = 5% are misclassified.
As you can see, this is simply an artifact of the higher reoffending rate of Group B. If 1) the two groups have different reoffending rates, and 2) the algorithm for labeling people high risk is unbiased, it necessarily follows that the two groups will have different misclassification rates.
The entire ProPublica piece (and your arguments) is based on misunderstanding this. Part of the issue is that extremely misleading table titled "Prediction Fails Differently for Black Defendants", where "Labeled Higher Risk, But Didn’t Re-Offend" actually refers to P(labeled higher risk | didn't re-offend).
Sure, I'm not disputing that. What I am trying to dispute here is kgwgk's assertion that there is no such thing as a false positive because even "high risk" people may end up not recidivating.
That’s not what I said.
> But no one is arguing that a particular person was misclassified;
The article claims that each and every particular person that was classified as high risk but didn't commit a new offence was "misidentified as high risk".
The ProPublica article and analysis claim that each and every particular person that was classified as high risk but didn't commit a new offence was "misclassified" and "falsely flagged as a future criminal".
The article claims that each and every particular person that was classified as high risk but didn't commit a new offence was "misidentified as high risk".
The ProPublica article and analysis claim that each and every particular person that was classified as high risk but didn't commit a new offence was "misclassified" and "falsely flagged as a future criminal".
[deleted]
This will be interesting. Most actions we people and ALL actions corporations take are optimised for maximal self / personal gain and not for justice (which is hard or impossible to define). This is the basis of neoliberalism.
It will be interesting to see where pressure points will emerge and who / how negotiations will progress.
http://www.uscourts.gov/sites/default/files/80_2_6_0.pdf