Justice.exe: Bias in Algorithmic sentencing(justiceexe.com)
justiceexe.com
Justice.exe: Bias in Algorithmic sentencing
http://justiceexe.com/
53 comments
> They try to make the most accurate predictions possible given the data.
So what you're saying is that a bias in the data could result in a biased outcome.
So what you're saying is that a bias in the data could result in a biased outcome.
There will be bias in the data, but that can be looked at in one place, and corrected once. At which point everyone gets the benefit. This means errors are found fixed and "released" broadly and quickly. This is much better than trying to disseminate training and information to judges who may or may not actually absorb it. For example new information on bias could be encoded in an algo but is possibly not teachable...because people are biased.
Sure. But if the data isn't biased then the algorithm won't be biased either. And in many cases we have perfect ground truth.
Thank you, you did a much better job than I did of explaining what I've been trying to say all along in my other posts.
Small bit of feedback on the game: not being a law student, it took me a minute to figure out what I was supposed to do. From the mention of "Risk Assessment Program", I infer that I'm supposed to assume that everyone was found guilty, and guesstimate whether they are likely to commit more offenses in the future (and choose a sentence based on that). Is that correct?
There has been some interesting discussion on this thread already.
However, what most seems to fail to see is that an algorithm compared to a human justice system, could inherently be less biased.
Having no bias at all is hard but only having less bias is totally fine IMO.
When writing the algorithm it is of course impossible to not instill some bias into the datasets, the code itself should be bias-free if written correctly. The end result is a bias-free judgement algorithm based on biased data.
Is this worse than what we have now?
However, what most seems to fail to see is that an algorithm compared to a human justice system, could inherently be less biased.
Having no bias at all is hard but only having less bias is totally fine IMO.
When writing the algorithm it is of course impossible to not instill some bias into the datasets, the code itself should be bias-free if written correctly. The end result is a bias-free judgement algorithm based on biased data.
Is this worse than what we have now?
How about the danger of HUMAN sentencing, talk about lack of training, transparency and consistency.
One of the things that computers really have going for in the case of "algorithmic sentencing" is that they are NOT biased; they are just machines that repeatedly and reliably run an ordered set of instructions X, their program, free of any human emotions that can override such logic.
Algorithmic sentencing, when implemented correctly (i.e. the X that it runs is not biased), have the potential to be vastly more "fair" and "just" than any human jury could be.
For example, from a widely circulated online video of a joint talk given by a lawyer and a cop at a law school:
"the [Jury's] perception is that if you're sitting next to a defense attorney, YOU have to prove you're innocent" [1]
Juries are comprised of people off the street who have no expertise in determining things like innocence. The average person is subject to all sorts of biases, most of which they aren't even consciously aware they have.
Knowing this, if I ever had to stand for trial in front of a jury of my peers as a defendant, I'd be scared shitless, _especially_ if I was innocent.
As you can see from the linked video, machines actually have a pretty low bar to live up to in order to outperform humans.
[1] https://www.youtube.com/watch?v=08fZQWjDVKE&feature=youtu.be...
Algorithmic sentencing, when implemented correctly (i.e. the X that it runs is not biased), have the potential to be vastly more "fair" and "just" than any human jury could be.
For example, from a widely circulated online video of a joint talk given by a lawyer and a cop at a law school:
"the [Jury's] perception is that if you're sitting next to a defense attorney, YOU have to prove you're innocent" [1]
Juries are comprised of people off the street who have no expertise in determining things like innocence. The average person is subject to all sorts of biases, most of which they aren't even consciously aware they have.
Knowing this, if I ever had to stand for trial in front of a jury of my peers as a defendant, I'd be scared shitless, _especially_ if I was innocent.
As you can see from the linked video, machines actually have a pretty low bar to live up to in order to outperform humans.
[1] https://www.youtube.com/watch?v=08fZQWjDVKE&feature=youtu.be...
People will write that software, select which software to purchase, write the rules for its use, and implement them. Powerful people aren't going to sit back and let those things happen in ways that they don't like, which are threats to them, or which reduce their influence.
Then we need the software to be implemented as a decentralized smart contract.
How would that help?
More specifically, how would implementing this as a smart contract prevent bias from making its way into the algorithms or the learning datasets?
Remember that bias can be inserted accidentally, by well-meaning programmers who don't realize that the data they are using is biased or has other methodological problems.
More specifically, how would implementing this as a smart contract prevent bias from making its way into the algorithms or the learning datasets?
Remember that bias can be inserted accidentally, by well-meaning programmers who don't realize that the data they are using is biased or has other methodological problems.
To me, it just seems like the problem is solvable by having really clean data (Maybe single data point has to be verified by a DNA test? There's plenty of ideas here) -- much more solvable than taking a random group of (probably) unqualified people off the street, calling them a "jury", and then having them make potentially life-or-death decisions regarding the fate of their peers. Right now, the existing system we have is so stacked in favor towards the wealthy and Caucasian that I can't possibly see a well-intentioned algorithm doing any _worse_ than what we have now (a study even successfully correlated good looks with less-strict judicial outcomes; it's pretty easy to design an algorithm that doesn't give an extremely lenient sentence to a Stanford swimmer), but at least, it has a fighting chance of doing _better_.
Try reading anything at all about the jury selection process, and you'll be shocked at how abysmal our existing system is.
People tend to dismiss machines as being cold, calculated, and emotionless, but that's exactly what we WANT if we're aiming for equal, fair justice for all.
Try reading anything at all about the jury selection process, and you'll be shocked at how abysmal our existing system is.
People tend to dismiss machines as being cold, calculated, and emotionless, but that's exactly what we WANT if we're aiming for equal, fair justice for all.
Clean data is basically impossible to get. You won't find very many experts who agree what it would even mean for data about these topics to be clean. There are tons of methodological disagreements, not to mention the fact that even if we started collecting new "clean" data, the historical data could never be used.
The law is a human construct, not a computer program. It is precisely because of issues of bias, fairness, methodology, etc. that the law is complicated and needs to be administered by people who can understand the complexities and use their education and initiative to ensure fair outcomes.
It's not perfect, and it never will be. But the legal system was designed with the assumption that judges and lawyers would be able to use discretion. To apply rigid computer logic on top of a system that was intentionally designed to be subject to human discretion would be a disaster.
The law is a human construct, not a computer program. It is precisely because of issues of bias, fairness, methodology, etc. that the law is complicated and needs to be administered by people who can understand the complexities and use their education and initiative to ensure fair outcomes.
It's not perfect, and it never will be. But the legal system was designed with the assumption that judges and lawyers would be able to use discretion. To apply rigid computer logic on top of a system that was intentionally designed to be subject to human discretion would be a disaster.
I'm a bit more optimistic. For example, for all intents and purposes, I would consider a positive DNA test result to be overwhelmingly damning proof of guilt.
Indeed, although unpopular, technological advances in things like data logging and DNA testing are able to establish both guilt and innocence in a much more scientific manner than a judge crippled by such things as their current mood (maybe they had an argument with their spouse before hearing the case and now they're feeling particularly vindictive) and whether they "like" the defendant as a person (maybe the defendant looks like their high school bully).
The Innocence Project has successfully exonerated many convicted "criminals" solely on the basis of DNA evidence, some of whom have been behind bars for many years. If that doesn't point to an extremely biased and flawed existing judicial system, I don't know what does.
When I think of bias in the judicial system, I don't immediately point the finger at machine learning algorithms and say "if we implemented these types of systems, then we'd be biased" -- instead, I look at peer juries and law enforcement officers as the source of bias, things like "Stop and Frisk" or "driving while black." You have very faulty systems in place like plea bargaining which effectively nullify due process and encourage false confessions in order to obtain a guaranteed lighter sentence. You have prosecutors with their own illicit motives like appearing in public to be "tough on crime" resulting in such atrocities as Aaron Swartz -- this very concept of "making an example out of somebody" is a biased one.
Someone brought up the point that "bias can be inserted accidentally, by well-meaning programmers" in an unintentional matter. On the other hand, right now, instead of employing some well-meaning programmer who is trying their very best to make sure things are fair, you have policies in place like racial profiling which are INTENTIONALLY biased. Which is worse?
In my mind, ML has the potential of creating a bias-free judicial system more than we could ever hope for by continuing to rely on fickle, highly error-prone, and even outright malicious humans. People just need to get past the fact that a "cold-hearted," emotionless computer is in the judge's chair because that's EXACTLY what we want.
Indeed, although unpopular, technological advances in things like data logging and DNA testing are able to establish both guilt and innocence in a much more scientific manner than a judge crippled by such things as their current mood (maybe they had an argument with their spouse before hearing the case and now they're feeling particularly vindictive) and whether they "like" the defendant as a person (maybe the defendant looks like their high school bully).
The Innocence Project has successfully exonerated many convicted "criminals" solely on the basis of DNA evidence, some of whom have been behind bars for many years. If that doesn't point to an extremely biased and flawed existing judicial system, I don't know what does.
When I think of bias in the judicial system, I don't immediately point the finger at machine learning algorithms and say "if we implemented these types of systems, then we'd be biased" -- instead, I look at peer juries and law enforcement officers as the source of bias, things like "Stop and Frisk" or "driving while black." You have very faulty systems in place like plea bargaining which effectively nullify due process and encourage false confessions in order to obtain a guaranteed lighter sentence. You have prosecutors with their own illicit motives like appearing in public to be "tough on crime" resulting in such atrocities as Aaron Swartz -- this very concept of "making an example out of somebody" is a biased one.
Someone brought up the point that "bias can be inserted accidentally, by well-meaning programmers" in an unintentional matter. On the other hand, right now, instead of employing some well-meaning programmer who is trying their very best to make sure things are fair, you have policies in place like racial profiling which are INTENTIONALLY biased. Which is worse?
In my mind, ML has the potential of creating a bias-free judicial system more than we could ever hope for by continuing to rely on fickle, highly error-prone, and even outright malicious humans. People just need to get past the fact that a "cold-hearted," emotionless computer is in the judge's chair because that's EXACTLY what we want.
> People just need to get past the fact that a "cold-hearted," emotionless computer is in the judge's chair because that's EXACTLY what we want.
That is not what I want. You should not want that either. If our current laws were implemented strictly and impartially, without reference to mitigating circumstances, the results would often be unfair, unjust, and morally wrong.
An emotionless computer would not consider any mitigating or unusual circumstances unless it was programmed to do so in advance -- but not all of the the circumstances that might come up in court are going to be obvious to the programmers.
Even foreseeable mitigating circumstances would be very difficult to program correctly, e.g. the difference between valid self-defense and manslaughter depends heavily on the state of mind of the defendant. Good luck teaching a computer how to deal with such a problem in a fair and just way, unless you can manage to develop the first artificial general intelligence.
That is not what I want. You should not want that either. If our current laws were implemented strictly and impartially, without reference to mitigating circumstances, the results would often be unfair, unjust, and morally wrong.
An emotionless computer would not consider any mitigating or unusual circumstances unless it was programmed to do so in advance -- but not all of the the circumstances that might come up in court are going to be obvious to the programmers.
Even foreseeable mitigating circumstances would be very difficult to program correctly, e.g. the difference between valid self-defense and manslaughter depends heavily on the state of mind of the defendant. Good luck teaching a computer how to deal with such a problem in a fair and just way, unless you can manage to develop the first artificial general intelligence.
"Good luck teaching a computer how to deal with such a problem in a fair and just way"
Good luck getting a HUMAN to deal with such a problem in a fair and just way!
Good luck getting a HUMAN to deal with such a problem in a fair and just way!
Humans do just that on a regular basis.
[deleted]
I don't know if I can believe that, given they also say that obviously-guilty people with "affluenza" should get off scot-free from murder charges.
So, for you, one bad outcome negates a preponderance of correct outcomes?
Even in the best legal system in the world, you'll be able to find a rapist or murderer who got away with it. That's not a valid counterargument.
Even in the best legal system in the world, you'll be able to find a rapist or murderer who got away with it. That's not a valid counterargument.
Forgive my ignorance since I don't have a CS degree, but don't all algorithms used for selecting have a human behind the logic. So wouldn't an algorithm always have some bias based on the weights given by a human. Kind of like OKCupid saying you are a 75% match with someone. You may not be compatible with that person, but whoever designed the algorithm declared it to be true with their design.
So wouldn't any algorithm always have some biased in it?
Sorry if this comes off as a stupid question or is unclear.
So wouldn't any algorithm always have some biased in it?
Sorry if this comes off as a stupid question or is unclear.
It's not a stupid question at all, and it cuts to the core of a lot of problems with AI and algorithmic-based anything.
Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group.
Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group.
> Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group.
Can you break this down more? It doesn't make sense to me. If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program.
Unless the programmers are changing the datasets then I don't see how this makes sense.
Can you break this down more? It doesn't make sense to me. If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program.
Unless the programmers are changing the datasets then I don't see how this makes sense.
In the case you presented, the biases are in the dataset itself.
Arrest statistics in the US are heavily skewed by race. If you were to take a dataset of all US arrests between 1900 and 2000 and ask which populations are most likely to "commit crimes" (i.e., be arrested), you'd get racially biased model without recording why it's biased (discrimination, enduring poverty, minimum sentencing, &c).
Arrest statistics in the US are heavily skewed by race. If you were to take a dataset of all US arrests between 1900 and 2000 and ask which populations are most likely to "commit crimes" (i.e., be arrested), you'd get racially biased model without recording why it's biased (discrimination, enduring poverty, minimum sentencing, &c).
Dead sibling comment asks a common questions I hear: why can't we just not tell the algorithm race? And the answer is that, with enough data points, if there is a statistical bias in the dataset, the algorithm will likely learn a substitute for race in the other features. For the sake of argument, perhaps an interaction between home address, annual income and model of car driven is predictive of race, and race is predictive of recidivism in the dataset- then the algorithm will learn this cross of features is predictive of recividism, even though we would like all of them to ideally be irrelevant to sentencing.
> home address, annual income and model of car driven is predictive
Why the hell would those be factors? The factors of the case should be things that actually matter in the case.
A rich person who murders someone should get the same sentence as a poor person who murders someone.
A black person who murders someone should get the same sentence as a white person who murders someone.
Adding those factors would be insane in the first place. If you're adding crazy things like that you might as well add factors like "Can Juggle" and "Can Burp the Alphabet" because things like that should have just as much to do with sentencing as what kind of car you drive or where you live.
Why the hell would those be factors? The factors of the case should be things that actually matter in the case.
A rich person who murders someone should get the same sentence as a poor person who murders someone.
A black person who murders someone should get the same sentence as a white person who murders someone.
Adding those factors would be insane in the first place. If you're adding crazy things like that you might as well add factors like "Can Juggle" and "Can Burp the Alphabet" because things like that should have just as much to do with sentencing as what kind of car you drive or where you live.
One example that's going to come up is "Has the person previously been convicted of a crime?" That question is strongly correlated with race, and yet is strongly arguable to be relevant.
Model of car is hard to say that it's useful for predictive value, but home address (=which neighborhood did you grow up in, e.g., are you in Cabrini-Green or are you Gold Coast?) and annual income (poor people are more likely to commit crimes than rich people) definitely are going to be queried inputs. Particularly if the question is things like "what should I set bail at?"
Model of car is hard to say that it's useful for predictive value, but home address (=which neighborhood did you grow up in, e.g., are you in Cabrini-Green or are you Gold Coast?) and annual income (poor people are more likely to commit crimes than rich people) definitely are going to be queried inputs. Particularly if the question is things like "what should I set bail at?"
> "Has the person previously been convicted of a crime?" That question is strongly correlated with race, and yet is strongly arguable to be relevant.
Past behavior is a good predictor of future behavior in my experience. That's more than doubled when it's recurring behavior.
> home address (=which neighborhood did you grow up in, e.g., are you in Cabrini-Green or are you Gold Coast?) and annual income (poor people are more likely to commit crimes than rich people) definitely are going to be queried inputs
Why?
> Particularly if the question is things like "what should I set bail at?"
That's still the job of the judge and there is law at hand for how those values are calculated. The computers won't be involved with that (I'd assume) just sentencing based on previous case decisions.
Past behavior is a good predictor of future behavior in my experience. That's more than doubled when it's recurring behavior.
> home address (=which neighborhood did you grow up in, e.g., are you in Cabrini-Green or are you Gold Coast?) and annual income (poor people are more likely to commit crimes than rich people) definitely are going to be queried inputs
Why?
> Particularly if the question is things like "what should I set bail at?"
That's still the job of the judge and there is law at hand for how those values are calculated. The computers won't be involved with that (I'd assume) just sentencing based on previous case decisions.
> Past behavior is a good predictor of future behavior in my experience.
That question includes the past behavior of a lot more people than just the person with a criminal conviction.
> recurring behavior
Institutional racism has recurred a lot in the justice system.
That question includes the past behavior of a lot more people than just the person with a criminal conviction.
> recurring behavior
Institutional racism has recurred a lot in the justice system.
I agree they shouldn't be factors, and my example is a bit contrived, but many do include socioeconomic and location based factors, like this reply by the authors of a paper on one recidivism calculator (who argue here why socioeconomic factors are included) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5047350/
Of course the factors chosen are just another place to introduce bias.
>In the case you presented, the biases are in the dataset itself.
The code that is going to handle the data is also biased in some direction. Its actions is value-based, coming from that of the creators of the algorithm. This is why all code must be open and publicly auditable if used in law enforcement, health care and insurance, etc.
The code that is going to handle the data is also biased in some direction. Its actions is value-based, coming from that of the creators of the algorithm. This is why all code must be open and publicly auditable if used in law enforcement, health care and insurance, etc.
Can't that problem be solved by simply not telling the machine the race of the person who's being judged?
[deleted]
As with anything, it's hard to generalise without oversimplifying, but here goes. You don't generally just have a data set and a machine learning algorithm that somehow magics outputs from a data set. Usually decisions have to be made by people either in training the model, selecting variables that are included in a model, etc.
Here's a simple example. Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know). We have to tell the algorithm whether a given article in the training set is fake or legitimate, otherwise how would it know? Clearly this will reflect the views of whoever is tagging the articles. When we run the model on a training set we need to score how well it did, again this will reflect the opinion of the person doing the scoring.
For a real example: https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-....
Here's a simple example. Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know). We have to tell the algorithm whether a given article in the training set is fake or legitimate, otherwise how would it know? Clearly this will reflect the views of whoever is tagging the articles. When we run the model on a training set we need to score how well it did, again this will reflect the opinion of the person doing the scoring.
For a real example: https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-....
I've done some machine learning in the past. I get how you tackle machine learning problems and the first thing that I would say that your proposed topic is not currently possible.
That's not a good facsimile for deciding sentencing. Sentencing is more like a linear regression classification. You have a history of previous cases where the defendant was found guilty. You then have a pile of factors that played into the judge's decision for sentencing. For example:
Deciding the validity of claims is not within the scope of modern day machine learning (as of 2017). Classification engines are very much in the scope of machine learning of today.
I don't see how case factors could be biased. I don't see how historical cases (when stripped of all identifying information) could be biased. I don't see why a system like this would be bad.
All treatment of everyone would converge into a uniform handling of cases.
"*Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news"
(topical, I know).*"
That's not a good facsimile for deciding sentencing. Sentencing is more like a linear regression classification. You have a history of previous cases where the defendant was found guilty. You then have a pile of factors that played into the judge's decision for sentencing. For example:
* If they meant to do it
* If they feel bad about doing it
* If they did do it (Beyond a reasonable doubt)
* If they have done it before
* What severity this crime is
* ... etc
* Time served
* Fines
* Privileges revoked
* Community Service
Deciding the validity of claims is not within the scope of modern day machine learning (as of 2017). Classification engines are very much in the scope of machine learning of today.
I don't see how case factors could be biased. I don't see how historical cases (when stripped of all identifying information) could be biased. I don't see why a system like this would be bad.
All treatment of everyone would converge into a uniform handling of cases.
> That's not a good facsimile for deciding sentencing
Sure. I'm not saying my example is the smartest (although there are people trying to use ML to do this to be fair - http://www.fakenewschallenge.org/) and I was unaware of your experience with ML, so I was working under the assumption you had no experience with it and trying to pick a simple example (even if it's dumb) to explain how bias can sneak into models in general, rather than in the specific case of sentencing criminals.
Let's loop back to what you originally said:
> If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program
You then go on to describe a number of factors that you think should go into sentencing models that leave plenty of scope for bias:
* "If they meant to do it" - this is a judgement made by a person and clearly reflects the view of the person making the decision.
* If they feel bad about doing it - again, someone has to judge whether someone is legitimately remorseful or is trying to pretend they are to get themselves a lighter sentence.
* "If they have done it before" - This will reflect things like policing tactics. For instance poorer areas might be subject to higher rates of policing especially in areas adopting the broken windows theory of policing (https://en.wikipedia.org/wiki/Broken_windows_theory#New_York...) and often in these areas petty crimes are cracked down on more frequently. This means that people are more likely to have run ins with the law, meaning they're less likely to get jobs due to convictions showing up in background checks, which in turn increases their likelihood to reoffend.
* What severity this crime is - I'm not sure what you mean by this. Do you mean e.g. murder being more severe than petty theft, or things like how severe an assault was? I'm assuming the latter since the former is often just covered by things like sentencing guidelines anyway. If someone commits an assault, then how do you rate this in a way that a model can understand? How do you ensure consistency across different cases and judges?
At any rate, the point of these models is usually to remove the biases that judges might have about people of certain backgrounds from sentencing guidelines and produce a score that informs the likelihood of the convict reoffending (I believe), so your proposal isn't how this works in practice. In practice they're trying to avoid exactly these kinds of subjective assessments you proposed and replace them with supposedly objective predictors for the likelihood of the person in question to reoffend. From the article linked from the post: https://www.propublica.org/article/machine-bias-risk-assessm...
> Northpointe’s software is among the most widely used assessment tools in the country. The company does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. (On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary.)
> Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. The survey asks defendants such things as: “Was one of your parents ever sent to jail or prison?” “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights while at school?” The questionnaire also asks people to agree or disagree with statements such as “A hungry person has a right to steal” and “If people make me angry or lose my temper, I can be dangerous.”
Given that independent research seems to confirm that the company's model seems to favour higher sentences for people of colour, it's pretty clear from that description where biases could sneak in to the model, I hope?
Sure. I'm not saying my example is the smartest (although there are people trying to use ML to do this to be fair - http://www.fakenewschallenge.org/) and I was unaware of your experience with ML, so I was working under the assumption you had no experience with it and trying to pick a simple example (even if it's dumb) to explain how bias can sneak into models in general, rather than in the specific case of sentencing criminals.
Let's loop back to what you originally said:
> If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program
You then go on to describe a number of factors that you think should go into sentencing models that leave plenty of scope for bias:
* "If they meant to do it" - this is a judgement made by a person and clearly reflects the view of the person making the decision.
* If they feel bad about doing it - again, someone has to judge whether someone is legitimately remorseful or is trying to pretend they are to get themselves a lighter sentence.
* "If they have done it before" - This will reflect things like policing tactics. For instance poorer areas might be subject to higher rates of policing especially in areas adopting the broken windows theory of policing (https://en.wikipedia.org/wiki/Broken_windows_theory#New_York...) and often in these areas petty crimes are cracked down on more frequently. This means that people are more likely to have run ins with the law, meaning they're less likely to get jobs due to convictions showing up in background checks, which in turn increases their likelihood to reoffend.
* What severity this crime is - I'm not sure what you mean by this. Do you mean e.g. murder being more severe than petty theft, or things like how severe an assault was? I'm assuming the latter since the former is often just covered by things like sentencing guidelines anyway. If someone commits an assault, then how do you rate this in a way that a model can understand? How do you ensure consistency across different cases and judges?
At any rate, the point of these models is usually to remove the biases that judges might have about people of certain backgrounds from sentencing guidelines and produce a score that informs the likelihood of the convict reoffending (I believe), so your proposal isn't how this works in practice. In practice they're trying to avoid exactly these kinds of subjective assessments you proposed and replace them with supposedly objective predictors for the likelihood of the person in question to reoffend. From the article linked from the post: https://www.propublica.org/article/machine-bias-risk-assessm...
> Northpointe’s software is among the most widely used assessment tools in the country. The company does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. (On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary.)
> Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. The survey asks defendants such things as: “Was one of your parents ever sent to jail or prison?” “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights while at school?” The questionnaire also asks people to agree or disagree with statements such as “A hungry person has a right to steal” and “If people make me angry or lose my temper, I can be dangerous.”
Given that independent research seems to confirm that the company's model seems to favour higher sentences for people of colour, it's pretty clear from that description where biases could sneak in to the model, I hope?
The "Machine Bias" report that ProPublica published about the Northpointe software (and that is cited in OP) has been shown to be wrong.
Source: Flores, Bechtel, Lowencamp; Federal Probation Journal, September 2016, "False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks.”", URL http://www.uscourts.gov/statistics-reports/publications/fede...
In fact the ProPublica analysis was so poorly done that the authors of the above study wrote in the conclusion:
> "It is noteworthy that the ProPublica code of ethics advises investigative journalists that "when in doubt, ask" numerous times. We feel that Larson et al.'s (2016) omissions and mistakes could have been avoided had they just asked. Perhaps they might have even asked...a criminologist? We certainly respect the mission of ProPublica, which is to "practice and promote investigative journalism in the public interest." However, we also feel that the journalists at ProPublica strayed from their own code of ethics in that they did not present the facts accurately, their presentation of the existing literature was incomplete, and they failed to "ask." While we aren’t inferring that they had an agenda in writing their story, we believe that they are better equipped to report the research news, rather than attempt to make the research news."
What I find remarkable is that in the ongoing coverage ProPublica has published on this subject in December 2016 they interviewed a bunch of more people, but none of the folks that have criticized their analysis (published in September). Make of that what you will.
Source: Flores, Bechtel, Lowencamp; Federal Probation Journal, September 2016, "False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks.”", URL http://www.uscourts.gov/statistics-reports/publications/fede...
In fact the ProPublica analysis was so poorly done that the authors of the above study wrote in the conclusion:
> "It is noteworthy that the ProPublica code of ethics advises investigative journalists that "when in doubt, ask" numerous times. We feel that Larson et al.'s (2016) omissions and mistakes could have been avoided had they just asked. Perhaps they might have even asked...a criminologist? We certainly respect the mission of ProPublica, which is to "practice and promote investigative journalism in the public interest." However, we also feel that the journalists at ProPublica strayed from their own code of ethics in that they did not present the facts accurately, their presentation of the existing literature was incomplete, and they failed to "ask." While we aren’t inferring that they had an agenda in writing their story, we believe that they are better equipped to report the research news, rather than attempt to make the research news."
What I find remarkable is that in the ongoing coverage ProPublica has published on this subject in December 2016 they interviewed a bunch of more people, but none of the folks that have criticized their analysis (published in September). Make of that what you will.
I cannot believe that the paragraph you cite actually appears in a journal article. It's really rather unprofessional and silly.
I don't want to end up defending the methods ProPublica have used as I am certainly not qualified to do that and have no skin in this game anyway. I posted here initially in response to a very general question about bias in models, and I'd rather not be drawn into a lengthy discussion about this specific piece.
However, I do have one or two issues with the conclusions you seem to be drawing in your comment:
> What I find remarkable is that in the ongoing coverage ProPublica has published on this subject in December 2016 they interviewed a bunch of more people, but none of the folks that have criticized their analysis (published in September). Make of that what you will
I'm not sure it's possible to conclude anything from that actually. There could be plenty of non-nefarious reasons for the omission. For instance, one individual they cite in the follow up review [1] has written a paper citing the paper you linked to showing that "the differences in false positive and false negative rates cited as evidence of racial bias in the ProPublica article are a direct consequence of applying an instrument that is free from predictive bias to a population in which recidivism prevalence differs across groups". [2] The Flores et al paper seems to claim that showing that predictive bias does not exist is enough, which it would seem is not the case. If racial bias might appear anyway in the situations in which the model is often applied in reality, then perhaps the ProPublica authors felt that the paper cited below [2] adequately addressed the criticism of the paper you cited and decided not to reference the FPJ article for reasons of clarity in their follow up? I think discounting their work because of a single omission would be throwing the baby out with the bathwater.
The ProPublica authors cite plenty of other research in the area in their follow ups. Sure, this is all largely in agreement with their conclusions or go further, but does this matter unless those publications are incorrect? The ProPublica authors are writing for a news publication not an academic journal and are therefore not obligated to cite every relevant publication when they're publishing. So long as they can do this without forcing a conclusion then I don't see the problem. Perhaps they deliberately ignored the paper. Who knows?
[1] https://www.propublica.org/article/bias-in-criminal-risk-sco...
[2] https://arxiv.org/abs/1610.07524
I don't want to end up defending the methods ProPublica have used as I am certainly not qualified to do that and have no skin in this game anyway. I posted here initially in response to a very general question about bias in models, and I'd rather not be drawn into a lengthy discussion about this specific piece.
However, I do have one or two issues with the conclusions you seem to be drawing in your comment:
> What I find remarkable is that in the ongoing coverage ProPublica has published on this subject in December 2016 they interviewed a bunch of more people, but none of the folks that have criticized their analysis (published in September). Make of that what you will
I'm not sure it's possible to conclude anything from that actually. There could be plenty of non-nefarious reasons for the omission. For instance, one individual they cite in the follow up review [1] has written a paper citing the paper you linked to showing that "the differences in false positive and false negative rates cited as evidence of racial bias in the ProPublica article are a direct consequence of applying an instrument that is free from predictive bias to a population in which recidivism prevalence differs across groups". [2] The Flores et al paper seems to claim that showing that predictive bias does not exist is enough, which it would seem is not the case. If racial bias might appear anyway in the situations in which the model is often applied in reality, then perhaps the ProPublica authors felt that the paper cited below [2] adequately addressed the criticism of the paper you cited and decided not to reference the FPJ article for reasons of clarity in their follow up? I think discounting their work because of a single omission would be throwing the baby out with the bathwater.
The ProPublica authors cite plenty of other research in the area in their follow ups. Sure, this is all largely in agreement with their conclusions or go further, but does this matter unless those publications are incorrect? The ProPublica authors are writing for a news publication not an academic journal and are therefore not obligated to cite every relevant publication when they're publishing. So long as they can do this without forcing a conclusion then I don't see the problem. Perhaps they deliberately ignored the paper. Who knows?
[1] https://www.propublica.org/article/bias-in-criminal-risk-sco...
[2] https://arxiv.org/abs/1610.07524
Well... why can't the dataset be biased?
How was the dataset produced?
Not only the algorithm can be biased but also the underlying dataset could be biased. See
http://www.sciencemag.org/news/2017/04/even-artificial-intel... And the paper http://opus.bath.ac.uk/55288/
http://www.sciencemag.org/news/2017/04/even-artificial-intel... And the paper http://opus.bath.ac.uk/55288/
I can see how the dataset can be biased, but I'm struggling to find any biases in the algorithms themselves.
Assuming I'm using some actual machine learning model and not my own hand-coded finite state machine, I just don't see how, say, k-means clustering could be biased (unless there was a bug in the programmer's implementation?).
Assuming I'm using some actual machine learning model and not my own hand-coded finite state machine, I just don't see how, say, k-means clustering could be biased (unless there was a bug in the programmer's implementation?).
There are two algorithms.
Machine learning generates models based off of input data. Machine learning, principally, will not be [biased or] exhibit any direct bias. [It's math, on its own it's not the problem.]
The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased.
[EDIT: Forgot two words, added a sentence]
Machine learning generates models based off of input data. Machine learning, principally, will not be [biased or] exhibit any direct bias. [It's math, on its own it's not the problem.]
The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased.
[EDIT: Forgot two words, added a sentence]
"There are two algorithms."
In the case of machine learning techniques that employ a learning algorithm (e.g. SGD) and a separate classification algorithm (e.g. forward prop), the situation is no different. Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" -- neither does, forward propagation.
These algorithms are merely functions that are _parametrized_ by a potentially biased learned model (which really entirely amounts to having biased data).
I recommend taking a look at Stanford's Intro to Machine Learning CS 229 course notes for a brief overview of these concepts. I can't speak anymore highly of the material in this course; it's by far the best out there, and it's how I learned ML.
"The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased."
Right, the "final model" (which comes directly from the data) will be biased, but not the ALGORITHM (the "math," as you call it).
For example, I can't make an unbiased data set "biased" by using linear regression instead of logistic regression. It would be quite a tremendous discovery if you somehow figured out a way SGD in itself was biased, but the burden of proof on such an extraordinary claim would be on you.
So like I said in my parent post, I can see how the data can be biased, but I'm struggling to find any biases in the algorithms themselves (assuming they are properly implemented and not an Underhanded C Contest [1] entry).
Granted, if we're not actually using ML, and we're using something like a hardcoded decision tree or FSM instead (it was pretty strongly implied that we weren't though because we were talking about data-driven algorithms), then sure, that could very easily be a biased algorithm, but I've maintained that all along.
[1] http://www.underhanded-c.org/
In the case of machine learning techniques that employ a learning algorithm (e.g. SGD) and a separate classification algorithm (e.g. forward prop), the situation is no different. Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" -- neither does, forward propagation.
These algorithms are merely functions that are _parametrized_ by a potentially biased learned model (which really entirely amounts to having biased data).
I recommend taking a look at Stanford's Intro to Machine Learning CS 229 course notes for a brief overview of these concepts. I can't speak anymore highly of the material in this course; it's by far the best out there, and it's how I learned ML.
"The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased."
Right, the "final model" (which comes directly from the data) will be biased, but not the ALGORITHM (the "math," as you call it).
For example, I can't make an unbiased data set "biased" by using linear regression instead of logistic regression. It would be quite a tremendous discovery if you somehow figured out a way SGD in itself was biased, but the burden of proof on such an extraordinary claim would be on you.
So like I said in my parent post, I can see how the data can be biased, but I'm struggling to find any biases in the algorithms themselves (assuming they are properly implemented and not an Underhanded C Contest [1] entry).
Granted, if we're not actually using ML, and we're using something like a hardcoded decision tree or FSM instead (it was pretty strongly implied that we weren't though because we were talking about data-driven algorithms), then sure, that could very easily be a biased algorithm, but I've maintained that all along.
[1] http://www.underhanded-c.org/
> Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias"
That's correct, but the decision to use gradient descent at all is a huge bias. The mean, median, and mode all describe a dataset's central tendency, but their values can be very different.
That's correct, but the decision to use gradient descent at all is a huge bias. The mean, median, and mode all describe a dataset's central tendency, but their values can be very different.
I fully understand what you are trying to say, but I don't think the analogy given in the second sentence is enough. Let's assume that the hypothetical well-intentioned but biased programmer developing our sentencing algorithm is competent enough to only use a basic Gaussian model like you are suggesting if it REALLY fits the data well (yes, a model of central tendency is about as rudimentary as it gets, whereas SGD is used in most state-of-the-art ML research -- the stuff that is popularly known as "deep learning").
Here's a thought experiment for you then: Can you give an example of how one's decision to use gradient descent might impact a data-driven sentencing algorithm?
Or just in general? In contemporary machine learning, it is true that choosing "hyperparameters" is still pretty intuition-guided and based on trial-and-error (even "guessing" if you will).
I wouldn't think those choices could be made maliciously, or even in a "biased" way (okay, _maybe_ I could choose comically out-of-range, nonsensical values -- but that's really a stretch).
If anything, in a hypothetical world, I would feel comfortable letting the accused choose the provably-convergent machine learning algorithm and hyperparameter values of their wishes, and then that would be the sentencing model that is used to determine their fate. As you can see, I claim that their choices wouldn't be able to "bias" anything at all.
As has been true all along, I can certainly see how the _dataset_ could be biased, but I claim that the choice of hyperparameters (and even the algorithm itself) is quite a bit more "immune" to human biases. For one thing, we aren't even capable of fully understanding the learned models produced by state-of-the-art ML algorithms; they are effectively "black boxes."
Despite all of the downvotes, I am genuinely interested in continuing this discussion, and I am curious what you have to say so I will continue to spend more of my karma expressing and arguing for my (unpopular) views.
Here's a thought experiment for you then: Can you give an example of how one's decision to use gradient descent might impact a data-driven sentencing algorithm?
Or just in general? In contemporary machine learning, it is true that choosing "hyperparameters" is still pretty intuition-guided and based on trial-and-error (even "guessing" if you will).
I wouldn't think those choices could be made maliciously, or even in a "biased" way (okay, _maybe_ I could choose comically out-of-range, nonsensical values -- but that's really a stretch).
If anything, in a hypothetical world, I would feel comfortable letting the accused choose the provably-convergent machine learning algorithm and hyperparameter values of their wishes, and then that would be the sentencing model that is used to determine their fate. As you can see, I claim that their choices wouldn't be able to "bias" anything at all.
As has been true all along, I can certainly see how the _dataset_ could be biased, but I claim that the choice of hyperparameters (and even the algorithm itself) is quite a bit more "immune" to human biases. For one thing, we aren't even capable of fully understanding the learned models produced by state-of-the-art ML algorithms; they are effectively "black boxes."
Despite all of the downvotes, I am genuinely interested in continuing this discussion, and I am curious what you have to say so I will continue to spend more of my karma expressing and arguing for my (unpopular) views.
It will be intentionally biased; that I'm confident of.
https://news.ycombinator.com/item?id=14288232
https://news.ycombinator.com/item?id=14288232
Ultimately, everything is politics, and I predict that if these algorithms become widely used, politics eventually will be the primary determining factor in their calculations. Why?
If these algorithms become widely used and thus influence important outcomes, then people who like to control power will seek to obtain control of the algorithms, either to protect their interests or to grow their own influence.
As possible examples, politicians who see an algorithm apolitically jail a powerful person might pass a law saying that public service should be weighed more heavily. Law enforcement might want exceptions or special weightings for police officer defendants, or for police officer-given testimony. Powerful social activists will buy the algorithm developers (do you know who is writing these algorithms even now? I don't) as a way of influencing society - the Koch brothers, for example, invest in other areas, such as academia and down-ballot elections (e.g., secretary of state in U.S. states) for society-changing purposes. Special code might even generate special outcomes for particular individuals - who will ever see the code? Who will even ask this question after the sentencing?
If you think such corruption couldn't happen, look around. Brazen corruption happens all the time, especially in state and local government. This will be harder to detect, and will be covered by the excuse that non-technical people will believe: I didn't do it; it was the algorithm! It must be objective!
I think the algorithms ultimately will be a step backward, reducing transparency by hiding the bias in a software development process, in the back room of a software firm, and in a mountain of code. At least legislation is published and voted on in the open.
If these algorithms become widely used and thus influence important outcomes, then people who like to control power will seek to obtain control of the algorithms, either to protect their interests or to grow their own influence.
As possible examples, politicians who see an algorithm apolitically jail a powerful person might pass a law saying that public service should be weighed more heavily. Law enforcement might want exceptions or special weightings for police officer defendants, or for police officer-given testimony. Powerful social activists will buy the algorithm developers (do you know who is writing these algorithms even now? I don't) as a way of influencing society - the Koch brothers, for example, invest in other areas, such as academia and down-ballot elections (e.g., secretary of state in U.S. states) for society-changing purposes. Special code might even generate special outcomes for particular individuals - who will ever see the code? Who will even ask this question after the sentencing?
If you think such corruption couldn't happen, look around. Brazen corruption happens all the time, especially in state and local government. This will be harder to detect, and will be covered by the excuse that non-technical people will believe: I didn't do it; it was the algorithm! It must be objective!
I think the algorithms ultimately will be a step backward, reducing transparency by hiding the bias in a software development process, in the back room of a software firm, and in a mountain of code. At least legislation is published and voted on in the open.
Humans should never ever be in charge of sentencing decisions, or anything else you want to be "fair". There's nothing remotely fair about human judgement.
One particularly weird bias humans have, is we strongly distrust machines. Psychologists have studied this and named it "Algorithm Aversion": http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... Even when we know machines do better than humans, we are much more distrustful of them. We can watch humans make mistakes and an algorithm make many fewer mistakes, and still end up trusting the human far more.
And it's weird, because in almost every domain that's been studied, algorithms do vastly better than humans. For any statistical problem where there are a few features available to predict the outcome, even simple linear regression typically beats human experts. We suck at probabilistic and statistical problems. Often the predictions of experts are barely better than chance. And this has been known for a long time. The earliest reference I found was from 1928! Where an incredibly crude statistical model (hand trained!) beat a group of prison psychologists, at predicting recidivism of convicts.
Even the sources that inspired this game have been shown to be fraudulent. That is the propublica study they cite. See https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Propublica claimed their results were "almost statistically significant." I.e., they not statistically significant. And why would it be so? Algorithms have no particular reason to hate a minority group. They try to make the most accurate predictions possible given the data.