A Robot That Explains Its Actions(spectrum.ieee.org)
spectrum.ieee.org
A Robot That Explains Its Actions
https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/a-robot-that-explains-its-actions
46 comments
I understand your concerns, but there is a large body of XAI research that addresses them. For example, there are post-hoc explainability algorithms like LIME (https://github.com/marcotcr/lime) that "explain" the classification decisions of machine learning models. The obvious alternative is to use (deep) learning only to mine simpler, fully explainable models (sacrifice correlation power for intelligibility). What I don't think exist yet are approaches that go full scale with the "reasoning backwards" approach. Like a neural network making a BS classification decision and then trying to "reason backwards" in a symbolic way to defend the decision. But works like this one: https://openai.com/blog/debate/ go roughly into this direction (the work seemed to be, at the time of writing, in its infancy).
Humans are notoriously good at coming up with bullshit explanations for both random/statistical & deterministic phenomena. Why do we want AI to do the same? "Trust"?
Please.
I work in finance (investment strategy; ensemble trading models) and I trust my algorithms more than I'd trust 99% of the portfolio managers out there. And these PMs are incredibly capable at coming up with bullshit explanations for market moves on short notice.
Please.
I work in finance (investment strategy; ensemble trading models) and I trust my algorithms more than I'd trust 99% of the portfolio managers out there. And these PMs are incredibly capable at coming up with bullshit explanations for market moves on short notice.
You are right. In that sense, the AI that puts out the best bullshit will eventually "win", as in general with humanity.
Your perspective (and most readers here) is very different then those of the general population though. The creator of an AI has a deep understanding of how it came to live (even if it does not fully grasp the current learned decision making). But for the average Joe, it's impossible to correctly decide to trust an AI or not.
So even though you're completely right, it's the direction we'll go in anyway. Either humans or other AI's vouch for an algorithm, and 99% of the population will just go with it. It's crap stacked on other crap, and some entities in the right place hugely abuse their position, but it's the only way.
Your perspective (and most readers here) is very different then those of the general population though. The creator of an AI has a deep understanding of how it came to live (even if it does not fully grasp the current learned decision making). But for the average Joe, it's impossible to correctly decide to trust an AI or not.
So even though you're completely right, it's the direction we'll go in anyway. Either humans or other AI's vouch for an algorithm, and 99% of the population will just go with it. It's crap stacked on other crap, and some entities in the right place hugely abuse their position, but it's the only way.
> Humans are notoriously good at coming up with bullshit explanations
This is true, but bullshit stories have benefit not just to one side but both sides of the relationship more often than not, which is why evolution hasn't got ride of lying - its not a bug its a feature (for certain class of problems)- https://www.nationalgeographic.com/magazine/2017/06/lying-ho...
This is true, but bullshit stories have benefit not just to one side but both sides of the relationship more often than not, which is why evolution hasn't got ride of lying - its not a bug its a feature (for certain class of problems)- https://www.nationalgeographic.com/magazine/2017/06/lying-ho...
Uh, could you point out what part of that article you think supports your claim that "bullshit stories have benefit not just to one side but both sides of the relationship more often than not"? I just read it and didn't see anything that indicated that.
In the past few years I've worked pretty hard on building habits of honesty. Not from an ethical perspective--I'm an atheist and don't see any inherent reason to follow any prescriptive ideology--but from a self-benefit perspective. Cooperation is highly beneficial, moreso perhaps than at any time in history, and honesty allows a deeper level of cooperation with my peers. It's surprisingly difficult, as someone who viewed themselves initially as an already-honest person, but it yields major benefits. In the long term, I'm not even convinced that lying benefits the liar, let alone the person being lied to.
In the past few years I've worked pretty hard on building habits of honesty. Not from an ethical perspective--I'm an atheist and don't see any inherent reason to follow any prescriptive ideology--but from a self-benefit perspective. Cooperation is highly beneficial, moreso perhaps than at any time in history, and honesty allows a deeper level of cooperation with my peers. It's surprisingly difficult, as someone who viewed themselves initially as an already-honest person, but it yields major benefits. In the long term, I'm not even convinced that lying benefits the liar, let alone the person being lied to.
I just want to "second" this message. It's an attitude I've adopted and has been highly beneficial. It might seem like an uphill battle at first, with plenty of challenges, but ultimately has none of the challenges once the habit is steady. Rather, there are no challenges regarding honesty, because there is only one course of action available.
Some people may respond to it poorly, but I've learned to see this as a test of character. People worth interacting with and investing in will appreciate that honesty. People that don't appreciate it will likely be a source of conflict sooner or later anyway.
Perhaps the only valid "challenge" is learning how to be honest in a productive and beneficial way, but this challenge is distinct from challenges that _result_ from being honest.
Some people may respond to it poorly, but I've learned to see this as a test of character. People worth interacting with and investing in will appreciate that honesty. People that don't appreciate it will likely be a source of conflict sooner or later anyway.
Perhaps the only valid "challenge" is learning how to be honest in a productive and beneficial way, but this challenge is distinct from challenges that _result_ from being honest.
Doesn't the pie chart show to make others laugh, to protect others feelings, to maintain social norms, for economic/personal advantage (usually benefits family), to escape harm (again usually benefits your family when you don't get yourself killed while dealing with evil) etc
Both honesty and lying are tools. You choose how you use them. Whether to benefit others or to take advantage of others is a choice you make.
But there will be situations where the tool of honesty wont work, and if you walk into those situations having convinced yourself that lying isn't a tool available, then people who don't think that way will be better prepared than you to handle those situations.
Basically be aware where lies can be used. Don't just dismiss it totally as only useable for evil.
During Hitler's rise people like to point at the Honest folk who stood up against him like Carl Goerdeler, Dietrich Bonhoeffer. But there were people like Wilhelm Canaris who lied day in day out and did all kinds of damage and saved a whole lot of lives.
Both honesty and lying are tools. You choose how you use them. Whether to benefit others or to take advantage of others is a choice you make.
But there will be situations where the tool of honesty wont work, and if you walk into those situations having convinced yourself that lying isn't a tool available, then people who don't think that way will be better prepared than you to handle those situations.
Basically be aware where lies can be used. Don't just dismiss it totally as only useable for evil.
During Hitler's rise people like to point at the Honest folk who stood up against him like Carl Goerdeler, Dietrich Bonhoeffer. But there were people like Wilhelm Canaris who lied day in day out and did all kinds of damage and saved a whole lot of lives.
> Doesn't the pie chart show to make others laugh,
I'd question whether this actually is lying. Lying implies intent to deceive. For a scientific study, it makes sense to include this in the data, but I'm not sure it counts as lying. I'm not sure a priest, rabbi, and imam have ever walked into a bar together, but I'm pretty sure nobody is worried about whether it's true when I say they did.
> to protect others feelings,
As I've pointed out elsewhere, I don't think this actually helps the person being lied to in the long run.
> to maintain social norms,
Is there even an argument that this benefits the person being lied to?
> for economic/personal advantage (usually benefits family),
Doesn't benefit the person being lied to.
> to escape harm (again usually benefits your family when you don't get yourself killed while dealing with evil) etc
Again doesn't benefit the person being lied to. It's also a bit of a stretch to extrapolate "avoidance" from the chart to what you're saying, and I'd argue that the situations where you're lying to evil that might kill you are pretty unusual. I'd absolutely have no qualms lying to Nazis during the Third Reich, but that's not a fact which has any bearing on my life right now.
I'd question whether this actually is lying. Lying implies intent to deceive. For a scientific study, it makes sense to include this in the data, but I'm not sure it counts as lying. I'm not sure a priest, rabbi, and imam have ever walked into a bar together, but I'm pretty sure nobody is worried about whether it's true when I say they did.
> to protect others feelings,
As I've pointed out elsewhere, I don't think this actually helps the person being lied to in the long run.
> to maintain social norms,
Is there even an argument that this benefits the person being lied to?
> for economic/personal advantage (usually benefits family),
Doesn't benefit the person being lied to.
> to escape harm (again usually benefits your family when you don't get yourself killed while dealing with evil) etc
Again doesn't benefit the person being lied to. It's also a bit of a stretch to extrapolate "avoidance" from the chart to what you're saying, and I'd argue that the situations where you're lying to evil that might kill you are pretty unusual. I'd absolutely have no qualms lying to Nazis during the Third Reich, but that's not a fact which has any bearing on my life right now.
Warning: Do not try this advice when your wife asks if the pants make her butt look fat. The benefits of lying have been proven over and over again in this scenario.
That's simply not true in my experience. I suspect you haven't actually tried what you're saying over any significant period of time to make the claim you're making.
You're lying to avoid a conflict which is simply not that large in the grand scheme of a lifelong relationship. If I think pants make her butt look fat, that's not necessarily even a problem: it doesn't mean I'm not attracted to her in those pants, or that you even need to be attracted to her in any specific pants, or that she should even base her clothing choices on your opinions. And in a more general sense, why is she asking questions she doesn't want an answer to? If your relationship can't handle communicating honestly about very minor things like this, you're totally screwed when it comes to real issues, like the changing nature of attraction as you age, or asking for what you need to feel fulfilled in a relationship, or concern for the person's health at their weight. If you can't communicate in a really insignificant situation like this, how are you going to communicate when there's anything of actual significance?
Honesty with kindness is a skill, and it's certainly not trivial, but lying isn't the kinder option, even in this case.
Coming at it from the other side of things: when someone lies to me to spare my feelings, there's a lot of times where I know they're lying, and that means I can't trust them to give me honest feedback when I really don't know. If I can't trust someone to tell me something negative, then I can't trust them when they tell me something positive either.
You're lying to avoid a conflict which is simply not that large in the grand scheme of a lifelong relationship. If I think pants make her butt look fat, that's not necessarily even a problem: it doesn't mean I'm not attracted to her in those pants, or that you even need to be attracted to her in any specific pants, or that she should even base her clothing choices on your opinions. And in a more general sense, why is she asking questions she doesn't want an answer to? If your relationship can't handle communicating honestly about very minor things like this, you're totally screwed when it comes to real issues, like the changing nature of attraction as you age, or asking for what you need to feel fulfilled in a relationship, or concern for the person's health at their weight. If you can't communicate in a really insignificant situation like this, how are you going to communicate when there's anything of actual significance?
Honesty with kindness is a skill, and it's certainly not trivial, but lying isn't the kinder option, even in this case.
Coming at it from the other side of things: when someone lies to me to spare my feelings, there's a lot of times where I know they're lying, and that means I can't trust them to give me honest feedback when I really don't know. If I can't trust someone to tell me something negative, then I can't trust them when they tell me something positive either.
The correct answer to this is an appreciative "oh yes, yes they do."
It was probably lying that got you in this situation in the first place.
True.
I'm honest much earlier in relationships about much more significant things, so it's highly unlikely that I'll end up married to someone who couldn't handle honest in this situation.
I'm honest much earlier in relationships about much more significant things, so it's highly unlikely that I'll end up married to someone who couldn't handle honest in this situation.
But how do you handle people that take offense to honesty?
Well, that's gonna be complicated. The strategies are going to vary by situation. Honest communication is just a hard part of being a human. But it's worth the effort.
For example, with the "yes those pants make your butt look fat" conversation, that's an opportunity to set a boundary that improves your relationship: "Please don't ask me questions you don't want an answer to." Your relationship doesn't have to be a minefield, where she's quizzing you and any wrong answer could turn into a fight--that's not a pattern that's fun for either of you. Your wife probably just wants you to compliment her, so you can tell her that you'll make an effort to verbalize when you like how she looks or something she does. Those compliments will hold more weight, because she'll know that what you're saying is true. A compliment from someone who gives false compliments all the time is meaningless.
Some other general strategies:
1. Consider that I might be wrong, and if I was wrong, apologize. A side effect of being more honest with others is that I end up being more honest with myself as well, and I've often discovered that I believe some wrong things. Particularly with opinions, if you find that being honest about your opinions is offensive to people, consider that your opinions might be the problem, not the honesty. A lot of people who say "I'm just being honest!" when they offend people are actually just assholes. Lying wouldn't make them less assholes, it would just make them secret assholes.
2. Consider that my input wasn't wanted, and if I did that, apologize. A lot of the time the truth can just exist without being said. Does it need to be said? Does it need to be said now? Do I need to be the one to to say it? I'm particularly bad at this, personally.
3. Have a conversation with them about why I said what I said. A lot of times, like with the "yes those pants make your butt look fat" conversation, what the person is actually taking offense to is because they're assuming that what you said has other meaning. Just because I said the pants make her butt look fat doesn't mean that I don't love her, or that I'm not attracted to her, or that I think her butt looking fat is a bad thing. It may be that her negative reaction is because she's assumed one of those things, so clarifying would make her feel more secure.
4. This doesn't trivially apply to you wife, but in some cases, you can just stop wasting energy on the person. You can't please everyone, so don't try. If someone wants you to lie to them, that means they don't value what you have to say. Why would you want to talk to someone like that? But really, it rarely gets this far, because the reality is that most people don't take offense to the truth. Most people realize they have no choice but to accept reality when presented with it.
For example, with the "yes those pants make your butt look fat" conversation, that's an opportunity to set a boundary that improves your relationship: "Please don't ask me questions you don't want an answer to." Your relationship doesn't have to be a minefield, where she's quizzing you and any wrong answer could turn into a fight--that's not a pattern that's fun for either of you. Your wife probably just wants you to compliment her, so you can tell her that you'll make an effort to verbalize when you like how she looks or something she does. Those compliments will hold more weight, because she'll know that what you're saying is true. A compliment from someone who gives false compliments all the time is meaningless.
Some other general strategies:
1. Consider that I might be wrong, and if I was wrong, apologize. A side effect of being more honest with others is that I end up being more honest with myself as well, and I've often discovered that I believe some wrong things. Particularly with opinions, if you find that being honest about your opinions is offensive to people, consider that your opinions might be the problem, not the honesty. A lot of people who say "I'm just being honest!" when they offend people are actually just assholes. Lying wouldn't make them less assholes, it would just make them secret assholes.
2. Consider that my input wasn't wanted, and if I did that, apologize. A lot of the time the truth can just exist without being said. Does it need to be said? Does it need to be said now? Do I need to be the one to to say it? I'm particularly bad at this, personally.
3. Have a conversation with them about why I said what I said. A lot of times, like with the "yes those pants make your butt look fat" conversation, what the person is actually taking offense to is because they're assuming that what you said has other meaning. Just because I said the pants make her butt look fat doesn't mean that I don't love her, or that I'm not attracted to her, or that I think her butt looking fat is a bad thing. It may be that her negative reaction is because she's assumed one of those things, so clarifying would make her feel more secure.
4. This doesn't trivially apply to you wife, but in some cases, you can just stop wasting energy on the person. You can't please everyone, so don't try. If someone wants you to lie to them, that means they don't value what you have to say. Why would you want to talk to someone like that? But really, it rarely gets this far, because the reality is that most people don't take offense to the truth. Most people realize they have no choice but to accept reality when presented with it.
I'd like to point out that these "bullshit stories" are not necessarily lies. They could just be the post hoc rationalisation of hunches.
One of the issues is that humans don't reason what the current situation is prior to emotional response and action. Rather their emotions about the situation are inputs into the reasoning process.
e.g. The fear elicited upon perceiving a particular person causes the conclusion that he is dangerous, not vice versa.
The multifarious causes of emotions are notoriously difficult to trace, and can indeed be somewhat noisy. Thus most people are effectively just rationalising when asked why they behaved as they did.
e.g. The fear elicited upon perceiving a particular person causes the conclusion that he is dangerous, not vice versa.
The multifarious causes of emotions are notoriously difficult to trace, and can indeed be somewhat noisy. Thus most people are effectively just rationalising when asked why they behaved as they did.
This is a perspective I have not heard before. Iiuc, you are saying that there is a risk that the AI will come up with plausibly deniable things and just give us the rationalizations.
There are two ideas here, I think. First is that humans aren't actually very good at plausible explanations of decisions or causes. We're very good at providing acceptable justifications after the fact. So good at it, that when combined with the deep-seated human need for the world to have order, that people readily confuse the two.
Second, it may not be desirable to have AI that replicates this. Teaching a machine to give us post-hoc justifications the way we do to one another may not be as helpful as it sounds.
Second, it may not be desirable to have AI that replicates this. Teaching a machine to give us post-hoc justifications the way we do to one another may not be as helpful as it sounds.
We can only sometimes trust the minimally intelligent brains we already have to deal with.
At some point soon we're going to have to discuss whether we're going to go the way of the other extinct species. I think we will. I think that's fine, not being a raging egomaniac, I'd just like a slow pleasant transition, thanks.
Even if advanced machine intelligence could generate explanations, does that really establish trustworthiness? We can't even trust humans to explain their actions reasonably well. We're usually way better at rationalization than we are at introspection. How are we to know the same isn't true for the artificial intelligence?
Any (artificial) intelligent agent sophisticated enough to convincingly explain its reasons, is likely also sophisticated enough to lie about them.
Maybe, but imo a really good Chinese room that "learns" to tell us what we want to hear is way more likely in the foreseeable future than an AI with enough of a "mind", as we commonly understand it, to deliberately lie to us.
But what if it... lies.
Hire an AI-detective to know the truth.
what if the truth we believe is very well "inception-able"? Can we distinguish ideas planted by others(religion, government, pop cultures, movies etc)from our own, let alone "truth" from those AI who "study/spy" on our thoughts to better "serve/enslave" us?
the word that sticks out the most is "Trust". We haven't even solved trust with non-AI systems. Half the time I don't even trust myself and it's also not a binary state, and in constant flux. It's such a fuzzy, non-scientific concept that if we would replace every instance of the word "Trust" with "Faith" it would be clear that Trust is an unsolvable problem. "Trusting AI" is the ultimate oxymoron. Snake-oil vendors have also figured this out which is why we have the equally brain dead concept of "trustless"[1].
[1] the ultimate snake-oil is ofc combining the 2 words which gives you: https://trustless.ai
[1] the ultimate snake-oil is ofc combining the 2 words which gives you: https://trustless.ai
Доверя́й, но проверя́й (Trust, but verify)
Despite your well considered objections, we need to know that our computers are providing useful output at every level, especially at the highest levels which are once again being labeled AI.
The concept applies all up and down the stack of systems and organizations: When we don't trust our memory devices, we add error correction. When we don't trust our conventional software, we add QA testing. When we don't trust future governing officials, we add checks and balances. When we don't trust other countries we add verification protocols.
"Trusting AI" is not the ultimate oxymoron. It's just one more place we have to design systems to require as little verification as possible, and then provide the means to close the loop and allow verification of what remains. Trust in AI is big now because of the successes of non-symbolic processing. There's a real problem of understanding and explaining answers given by such systems. That the trust problem exists elsewhere is no reason to dismiss its importance here.
Despite your well considered objections, we need to know that our computers are providing useful output at every level, especially at the highest levels which are once again being labeled AI.
The concept applies all up and down the stack of systems and organizations: When we don't trust our memory devices, we add error correction. When we don't trust our conventional software, we add QA testing. When we don't trust future governing officials, we add checks and balances. When we don't trust other countries we add verification protocols.
"Trusting AI" is not the ultimate oxymoron. It's just one more place we have to design systems to require as little verification as possible, and then provide the means to close the loop and allow verification of what remains. Trust in AI is big now because of the successes of non-symbolic processing. There's a real problem of understanding and explaining answers given by such systems. That the trust problem exists elsewhere is no reason to dismiss its importance here.
Doesn't this assume the AI internally models the problem in a way that isn't foreign to the human? I believe the most effective AI will definitely not model problems like a human.
I'm not sure what you mean. Why would a foreign model inherently have the attribute of being inexplicable?
Isn't it more likely that we would look at a foreign model and think, "this makes sense, but i wouldnt have ever thought to do it this way".
Our senses might be fooled, but I think there would almost always exist a set of words to explain what the AI is doing assuming the AI is capable enough to find them
Isn't it more likely that we would look at a foreign model and think, "this makes sense, but i wouldnt have ever thought to do it this way".
Our senses might be fooled, but I think there would almost always exist a set of words to explain what the AI is doing assuming the AI is capable enough to find them
Well, in order to explain why you did a thing, you need to have some understanding of how the other party thinks in order to explain it in a way they will understand. And in order to explain it in that way, you would need to reflect upon the meaning of your own decisions in the context of how the other party models things.
Yes I think we would need a design goal to provide the AI with "some understanding" of how we think. It should be able to do its work in any model that works for it, which would then be translated back to our language.
Thankfully, language is flexible so it can express many models we do not understand in ways that can help us understand them. It would be interesting to see some sort of study on where the boundaries are on what human language is capable of modelling or not
Thankfully, language is flexible so it can express many models we do not understand in ways that can help us understand them. It would be interesting to see some sort of study on where the boundaries are on what human language is capable of modelling or not
If you want explainable AI you can fit a linear regression.
It works for a surprisingly large amounts of problems, including ones where people default to using DNNs, and at least with linear regression, when something goes wrong, you can look at it and tell what went wrong and why. Which is the whole point.
Fully autonomous robots are not yet completely brought into regular usage. So I think there is no need to worry much about it.
What happens when it's explanations stop making sense and it gives the equivalence of a "because" response?
So the AI that can lie most convincingly will earn humanity's trust?
[deleted]
[deleted]
let’s try this by substituting AI with human: A human that explains its actions is a first step towards a human we can maybe trust.
what a bunch of bs. what is trust? also, can we stop pretending we understand how the brain works and that the small incremental progress that is made in this area means that AI is just around the corner? it’s not
what a bunch of bs. what is trust? also, can we stop pretending we understand how the brain works and that the small incremental progress that is made in this area means that AI is just around the corner? it’s not
Yeah, let's try. Or maybe rephrase it to: a human whose actions you can understand is a human you can begin to trust.
We have it easy with other humans, because our brains are all alike. We all run on the same architecture, same firmware. In the occasional cases of unpredictable people, we tend to assume buggy or damaged hardware, and we either fix them or keep them away from anything where predictability matters (e.g. aviation, or gun ownership, or driving), up to and including locking the hardest cases up.
With algorithms, we're playing hard mode. There is no theory of mind for software. The "thoughts" of AIs are entirely unlike our own. So we need to do extra work to make them predictable.
We have it easy with other humans, because our brains are all alike. We all run on the same architecture, same firmware. In the occasional cases of unpredictable people, we tend to assume buggy or damaged hardware, and we either fix them or keep them away from anything where predictability matters (e.g. aviation, or gun ownership, or driving), up to and including locking the hardest cases up.
With algorithms, we're playing hard mode. There is no theory of mind for software. The "thoughts" of AIs are entirely unlike our own. So we need to do extra work to make them predictable.
Better yet, how would this strategy even be applicable to NN based learning models? We don't even have sufficient knowledge of how our own symbolic systems map to our brain activity. There is even evidence against a direct causal relationship between our symbolic narrative and our actions such as split brain experiments where the actions of the nonverbal hemisphere are "explained" or rationalized by the other post hoc. See also postdictive illusions and the varied arguments against free will.
My view is that the processes that constitute our minds are actually almost entirely non-verbal and the language oriented parts are merely a specialization of a more general rule. I'm not saying that language has no effect on our conscious processes, it wouldn't be a useful system if it played no part, but I do consider it to be more of a side channel or perhaps an abstraction of the processes that are driving what we call will or thought.
I believe this may be a fundamental flaw in how we approach understanding consciousness or any other aspect of intelligence.