Awful-AI: A curated list to track current scary usages of AI(github.com)
github.com
Awful-AI: A curated list to track current scary usages of AI
https://github.com/daviddao/awful-ai
47 comments
Ehh.. the discrimination part sounds a lot like: "we had an AI look at statistics and it reported the patterns we choose to ignore".
Ah, yes, but that's not really the point, is it?
COMPAS points out an uncomfortable trend ("violent crime is more prevalent in black communities...") but applies it in a way that is bad ("...and therefore black people should get longer criminal sentences for the same crime").
We both agree, I'm sure, that socioeconomics and race shouldn't meaningfully impact sentence length. We may also agree that it often does. What a lot of these models do is encode that outcome (that bias, some might say) in seemingly 'objective' systems. That reproduction of bias is why they're discriminatory - not because they look at crime stats, but because they apply them in a way that most people wouldn't.
COMPAS points out an uncomfortable trend ("violent crime is more prevalent in black communities...") but applies it in a way that is bad ("...and therefore black people should get longer criminal sentences for the same crime").
We both agree, I'm sure, that socioeconomics and race shouldn't meaningfully impact sentence length. We may also agree that it often does. What a lot of these models do is encode that outcome (that bias, some might say) in seemingly 'objective' systems. That reproduction of bias is why they're discriminatory - not because they look at crime stats, but because they apply them in a way that most people wouldn't.
You are deliberately twisting it. The crime stats says "violent crime is more prevalent in black communities...", but no AI/ML will produce the conclusion "...and therefore black people should get longer criminal sentences for the same crime". They are completely irrelevant. The AI model will simply conclude that race would be a factor when you are looking at crime rate, or posterior probability when a violent crime happens. And that is exactly accurate. What's wrong with that? That's a fact. Math doesn't lie nor does it have any agenda. There is nothing the AI is making up or adding up to. It is pure statistics and probability. It is the human beings that can't take the truth and the consequences.
You're right, of course: "human beings cant take the truth and consequences." "That is exactly accurate [...] That's a fact" - both true. I think the issue is less truth and more action, as in "I believe it to be immoral to act upon that fact in manner X."
I can come up with a dozen logical statements that are equally correct and immoral to act upon in certain ways. If an ML algorithm had come up with them, they would be no less correct and no less immoral. For example, "If a woman applicant is pregnant, she will take maternity leave and be less productive year 1. Therefore, we should not hire her." <- likely correct, definitely immoral (imo).
COMPAS drew a number of objective conclusions and ties them to specific recommendations in its intended use: (1) if you're poor, you're more likely to recidivate. Therefore, you should get a longer sentence. (2) if you live in a black community, you're more likely to recidivate. Therefore, you should get a longer sentence (3) if you're related to a gang member (not just one yourself - even if your estranged brother joined a gang a decade ago), you're more likely to recidivate. Therefore, you should get a longer sentence.
Notice: the "you should get a longer sentence" bit is actually part of the tool. That was the intervention it was designed to suggest. There's a number of critiques of that connection (mathematical, legal, moral), but it's not mine, nor is it the "AI's"; it's Northpointe's. And ultimately that's the scandal & intrigue: the math is right logically, but even so, the tool's recommendation is wrong ethically.
I can come up with a dozen logical statements that are equally correct and immoral to act upon in certain ways. If an ML algorithm had come up with them, they would be no less correct and no less immoral. For example, "If a woman applicant is pregnant, she will take maternity leave and be less productive year 1. Therefore, we should not hire her." <- likely correct, definitely immoral (imo).
COMPAS drew a number of objective conclusions and ties them to specific recommendations in its intended use: (1) if you're poor, you're more likely to recidivate. Therefore, you should get a longer sentence. (2) if you live in a black community, you're more likely to recidivate. Therefore, you should get a longer sentence (3) if you're related to a gang member (not just one yourself - even if your estranged brother joined a gang a decade ago), you're more likely to recidivate. Therefore, you should get a longer sentence.
Notice: the "you should get a longer sentence" bit is actually part of the tool. That was the intervention it was designed to suggest. There's a number of critiques of that connection (mathematical, legal, moral), but it's not mine, nor is it the "AI's"; it's Northpointe's. And ultimately that's the scandal & intrigue: the math is right logically, but even so, the tool's recommendation is wrong ethically.
The problem is that enforcement begets enforcement. Once one is entangled in the legal system, they are more likely to be called in for questioning, have their fingerprints possibly match a partial set left at a crime, have a possible DNA match turn into a warrant that leads to arrest on an unrelated crime. This is true because the web of laws, in the US at least, means that everyone breaks dozens of laws a day. So AI becomes a self-licking ice cream cone. Even worse, it is one that comes with a plausible argument of simply being an unbiased mathematical conclusion.
Of course math doesn’t have an agenda, but we aren’t talking about the actual algorithm having a human bias. We are talking about the application of math by humans to human problems using data sets curated by humans. Using math doesn’t purge the biases of all those humans from the system.
Of course math doesn’t have an agenda, but we aren’t talking about the actual algorithm having a human bias. We are talking about the application of math by humans to human problems using data sets curated by humans. Using math doesn’t purge the biases of all those humans from the system.
AIs produce correlation which gets sold as causation.
Which is then justified as causation we chose to ignore for political correctness.
A lot of the patterns these things pick up on are causal and we just don't like the fact that there are real causal networks that conflict with our religious/idealogical beliefs.
The issue is using the correlation to make predictions. You're just perpetuating the trend instead of trying to change it.
It has nothing to do with AI/ML. If you see someone repeatedly stealing from your yard, you can't predict he would do it again with high probability, because it would perpetuate the trend instead of trying to change it? Why would the responsibility of change lie on the observers instead of the perpetrators? If he cares about his image, he can stop stealing and with time, people would conclude so.
This list presents an incredibly pessimistic and alarmist view on these technologies, assuming only worst case abuses. And I don't know what solutions are advocated, so it feels like it is just against these technologies in general. And it is not fair to blame these mostly helpful tools for the awful things some people may use them for.
For example, they consider deep fakes in general to be awful, because of the potential for abuse. Should we just throw out deep fakes? Should we perhaps restrict their usage to licensed parties?
They also list Tay the chat bot as being awful because some trolls temporally hijacked it and got it to repeat back some racist stuff for a few weeks before it was re-calibrated. What does the author suggest? That Microsoft should never have made Tay public until they could 100% guarantee that no trolls would ever abuse it, even for a little bit?
Gender inference tools are written off as discrimination, because they could be used to discriminate against people. But they could just as well be used against AGAINST discrimination, by helping detect and document systemic discrimination. I think especially tools for analyzing populations present incredible benefits to help people, more so than potentials for abuse.
AI that can be used to detect genetic disease can save countless lives and reduce suffering. But it is bad because potential employers could maybe abuse it??
And a lot of these awful policies don't use the AI directly, they merely use the AI as a justification for what they were going to do already anyway. Indeed, you could go back 50 years and find "AI" being used to justify problematic policies. I'm sure that Oil executives, criminal justice lobbyists, and climate change deniers have all been able to algorithm based evidence for their agendas. Don't blame the algorithms, blame the people.
For example, they consider deep fakes in general to be awful, because of the potential for abuse. Should we just throw out deep fakes? Should we perhaps restrict their usage to licensed parties?
They also list Tay the chat bot as being awful because some trolls temporally hijacked it and got it to repeat back some racist stuff for a few weeks before it was re-calibrated. What does the author suggest? That Microsoft should never have made Tay public until they could 100% guarantee that no trolls would ever abuse it, even for a little bit?
Gender inference tools are written off as discrimination, because they could be used to discriminate against people. But they could just as well be used against AGAINST discrimination, by helping detect and document systemic discrimination. I think especially tools for analyzing populations present incredible benefits to help people, more so than potentials for abuse.
AI that can be used to detect genetic disease can save countless lives and reduce suffering. But it is bad because potential employers could maybe abuse it??
And a lot of these awful policies don't use the AI directly, they merely use the AI as a justification for what they were going to do already anyway. Indeed, you could go back 50 years and find "AI" being used to justify problematic policies. I'm sure that Oil executives, criminal justice lobbyists, and climate change deniers have all been able to algorithm based evidence for their agendas. Don't blame the algorithms, blame the people.
[deleted]
Your claim of "mostly helpful tools" was not backed up with facts or even an attempt at explanation.
I thought my comment was pretty clear. Most of the tools for trying to extract information about people by how they write is helpful. An example from the list is "Gender Detection from Names" under the "Discrimination" heading.
And then I gave several examples and an explanation. But to reiterate, tools that help us increase our understand of society are helpful, and can be used for good.
The list assumes that a tool that can help discern someone's gender will and can only be used to treat someone unfairly. But a researcher could just as well use exactly the same tools to analyze the gender composition of a set of people, for the purposes of advocating for them.
FOR EXAMPLE, in this repo:
https://github.com/waterdatacollaborative/Get-the-Lead-Out
What if a researcher were to use data mining tools to find out more information about the children most impacted by lead in the water, for the purposes of helping them. (And some have)
Is that an "awful" use of AI?
I would assert that actually yes some of these tools are quite helpful.
And then I gave several examples and an explanation. But to reiterate, tools that help us increase our understand of society are helpful, and can be used for good.
The list assumes that a tool that can help discern someone's gender will and can only be used to treat someone unfairly. But a researcher could just as well use exactly the same tools to analyze the gender composition of a set of people, for the purposes of advocating for them.
FOR EXAMPLE, in this repo:
https://github.com/waterdatacollaborative/Get-the-Lead-Out
What if a researcher were to use data mining tools to find out more information about the children most impacted by lead in the water, for the purposes of helping them. (And some have)
Is that an "awful" use of AI?
I would assert that actually yes some of these tools are quite helpful.
[deleted]
[deleted]
[deleted]
Is there a list of Nice-AI? Or generally unambiguously awesome tech?
If technology is a tool to make something easier, then I would say there's no such thing as "unambiguously awesome tech", as all tech can be used for good or evil...
82% 'unambiguously awesome tech'
12% 'bad stuff'
6% 'other'
A lot of these are less "woah, AI is real bad and we need to be better about using it appropriately" and more "people need to stop being fooled by snake oil products that promise impossible things."
You can blame users for expecting AI to solve everything, but that doesn't mean we give Engineers and Researchers free reign to package, publish, and distribute AI software that falls short of its promise (and introduces dangerous patterns along the way).
>that doesn't mean we give Engineers and Researchers free reign to package, publish, and distribute AI software that falls short of its promise
And who is going to monitor and enforce this? You?
And who is going to monitor and enforce this? You?
Ideally we would have a professional organization, like literally every other discipline that calls itself Engineering, both to educate, license, and enforce ethical and safety norms, and to support engineers asked to violate those boundaries by providing a public avenue for grievance against companies trying to behave unethically or unsafely.
Perhaps by "give free reign to [x]" they just mean "not consider morally liable for [doing x]"?
Trying to build in safeguards before deploying anything is an unreasonable standard, and would severely cripple the good work that many of us are trying to do.
I have worked on crime analysis software to help people. It is important to me to minimize the number of people getting hurt. And when the data shows that most people getting in fights on the subway are black, what then? What would you have us do?
We should dig deeper and find real solutions, not delude ourselves that these problems exist.
And the world isn't as rosy as we might imagine it is. There's are some really uncomfortable truths out there. But I think we should confront and understand these things.
I have worked on crime analysis software to help people. It is important to me to minimize the number of people getting hurt. And when the data shows that most people getting in fights on the subway are black, what then? What would you have us do?
We should dig deeper and find real solutions, not delude ourselves that these problems exist.
And the world isn't as rosy as we might imagine it is. There's are some really uncomfortable truths out there. But I think we should confront and understand these things.
Not relevant to my point, which is responding to "And who is going to monitor and enforce this? You?".
My comment is not arguing for or against the claim that the software should or shouldn't have whatever safeguards. It was extremely limited in its scope.
So, I'm not sure why you chose my comment to respond to with your "darkly hinting" comment mr. "apostacy".
My comment is not arguing for or against the claim that the software should or shouldn't have whatever safeguards. It was extremely limited in its scope.
So, I'm not sure why you chose my comment to respond to with your "darkly hinting" comment mr. "apostacy".
At a minimum, the engineers and researchers. Deploying an AI sentencing model or AI lie detector should be treated with the same seriousness as deploying a new train or bridge; engineers shouldn't be willing to sign off on such a thing without strong, well-founded reasons to believe it won't hurt anyone.
Let's stop calling them engineers then.
Certainly, but it's important to acknowledge the distinction. If people think that Tay turned racist because Microsoft engineers encoded their racial bias, or even just programmed it poorly, they're going to be tricked by a succession of new projects claiming they fixed the biases from the last one.
a lot of this is fear mongering
Doesn't seem like a well-curated list.
Some of these, e.g. Uber's God Mode, don't even fall under the normal usage of the term "AI".
Others, like Palantir or WeChat, seem to denigrate an entire company (rather than specific practices).
And the racist chatbot thing isn't even an actual product, nor was it intentionally designed to be racist.
> Nor was it intentionally designed to be racist
Can't think of a racist AI scandal where is was intentional. The point of those stories is that it's easy to do accidentally.
Can't think of a racist AI scandal where is was intentional. The point of those stories is that it's easy to do accidentally.
A huge portion of these seem to be "I don't like the fact that this heuristic is strongly predictive", not any sort of complaint about the implementation or use of the prediction engine.
Fairness and predictivity are different concerns. Many of my interests, hobbies, and behaviors are strongly predictive of poor public speaking skills, but it would still be very unfair if my company installed an AI saying I can't give conference speeches because I'll probably do poorly.
1) Why is unfair? If there is one speech, but 3 candidates. And the AI predicted based on past performances, why is it unfair? (It is a bad decision for you, or the AI could be making a poor decision based on data/algorithm limitations, but it is not unfair, the AI doesn't favor anyone outside of the data it's given). 2) AI doesn't make the decision that you can't give speeches. AI gives truthful predictions assuming the data is truthful. But it is your boss who uses the results to make that decision. The ethics part is on your boss using what data/algo to make such a prediction to determine who should go. Why would you blame the AI instead of the boss?
Yea, I mean there's a lot of strongly-predictive heuristics that are unethical to act on, though (e.g., "if applicant is pregnant, then she will take maternity leave, and therefore be less productive employee year 1, so don't hire" <- obviously immoral).
Why shouldn't automated decision-making processes be held to the same ethical standards as manual ones?
Why shouldn't automated decision-making processes be held to the same ethical standards as manual ones?
That's an important point, no matter if the algorithm is some trained neural net or a ruleset written by hand.
I'm often under the impression that moral standards are easier to ignore if "the computer says that X". E.g. we have to accept the fact that woman is much more likely to take maternety leave, and essentially an AI is okay to be aware of that, too. It even has to, else it finds another commom factor (eg womans chess club) - but it should be able to wilfully ignore this finding because using it against an applicant is immoral.
I'm often under the impression that moral standards are easier to ignore if "the computer says that X". E.g. we have to accept the fact that woman is much more likely to take maternety leave, and essentially an AI is okay to be aware of that, too. It even has to, else it finds another commom factor (eg womans chess club) - but it should be able to wilfully ignore this finding because using it against an applicant is immoral.
Exactly, but it's still a problem. Using existing trends that we should be changing to continue perpetuating them through prediction is what's at the heart of the issue.
I think there's 4 categories of issues:
- 1. AI that's just not good enough, e.g. it requires 100% confidence not to be problematic (Detecting whether it's a person or a gorilla. Autonomous driving)
- 2. AI that's correctly trained but discovers problematic trends and attempts to do prediction based on these trends (Train on poverty + race correlations)
- 3. AI that's incorrectly trained on inherently biased data and just reaffirms the problematic trend (Train only on male CVs for hiring prediction.)
- 4. Buggy code that spits out bad data.
- 1. AI that's just not good enough, e.g. it requires 100% confidence not to be problematic (Detecting whether it's a person or a gorilla. Autonomous driving)
- 2. AI that's correctly trained but discovers problematic trends and attempts to do prediction based on these trends (Train on poverty + race correlations)
- 3. AI that's incorrectly trained on inherently biased data and just reaffirms the problematic trend (Train only on male CVs for hiring prediction.)
- 4. Buggy code that spits out bad data.
Watch this:
https://youtu.be/stHLrBs-_iE
(Based off of the Stanford Prof’s example during a TED talk of using <$100 to assassinate people with AI, A CO2 canister, a nail, and a drone).
https://youtu.be/stHLrBs-_iE
(Based off of the Stanford Prof’s example during a TED talk of using <$100 to assassinate people with AI, A CO2 canister, a nail, and a drone).
Can we chill with the throbbing bass though?
Good quality version - https://youtu.be/9fa9lVwHHqg
On the flipside of things, here's a series I'm working on:
https://github.com/cbailes/awesome-ai
Currently:
https://github.com/cbailes/awesome-ai-cancer
https://github.com/cbailes/awesome-ai-cardiology
https://github.com/cbailes/awesome-deep-trading
https://github.com/cbailes/awesome-ai
Currently:
https://github.com/cbailes/awesome-ai-cancer
https://github.com/cbailes/awesome-ai-cardiology
https://github.com/cbailes/awesome-deep-trading
One of my favourites for your list: https://medium.com/@pguillaud/when-facebook-can-predict-your...
Using Machine learning to predict divorces on Facebook.
Using Machine learning to predict divorces on Facebook.