Trapped priors as a basic problem of rationality(astralcodexten.substack.com)
astralcodexten.substack.com
Trapped priors as a basic problem of rationality
https://astralcodexten.substack.com/p/trapped-priors-as-a-basic-problem
79 comments
Rationality as a tool to serve yourself. A lot of SSC and AC10 is around this idea that rational intelligent thinking with self awareness of bias allows you to model the world accurately.
As for the rest, yes, I agree with you. Often, after having considered a great deal of evidence, I condense the finding into some strength of belief about something. Now, absent the original evidence I have a position. This is usually fine since I actually lack the ability to summon all evidence on demand.
It's okay, though, since I (and I suspect most people intuitively) have both a likelihood notion and a certainty in likelihood notion analogous to a Confidence Interval and P-value, if you will.
So I might hold the belief that GME is going to be between 120 and infinity on Jan 13 2022 and have a certainty in that belief that is like 10% but lose the evidence that made me think that.
This is also sort of a failure mode for fast Aumann Agreement even in co-incentivized individuals - the fact that memory is short.
So the problem is that my certainty holding is not well done.
Thanks for the thought prompt.
As for the rest, yes, I agree with you. Often, after having considered a great deal of evidence, I condense the finding into some strength of belief about something. Now, absent the original evidence I have a position. This is usually fine since I actually lack the ability to summon all evidence on demand.
It's okay, though, since I (and I suspect most people intuitively) have both a likelihood notion and a certainty in likelihood notion analogous to a Confidence Interval and P-value, if you will.
So I might hold the belief that GME is going to be between 120 and infinity on Jan 13 2022 and have a certainty in that belief that is like 10% but lose the evidence that made me think that.
This is also sort of a failure mode for fast Aumann Agreement even in co-incentivized individuals - the fact that memory is short.
So the problem is that my certainty holding is not well done.
Thanks for the thought prompt.
> Rationality as a tool to serve yourself. A lot of SSC and AC10 is around this idea that rational intelligent thinking with self awareness of bias allows you to model the world accurately.
That sounds like an extraordinary claim that would require extraordinary evidence. No doubt physics and engineering have made great advances, but that doesn't mean that every social science that dresses itself up in the language of rationalism will be able to make similar advances, or really any advance at all. In fact, there is a case it will lead to regression rather than advancement:
Societies are very complex things made of people who respond to changing incentives. I don't believe anyone understands them very well, nor has any field in the social sciences established a track record of making highly accurate out of sample predictions, especially for the efficacy of interventions 10, 100, 1000 generations on. Even the simple question of "what keeps a society together?" is still a puzzling one, especially now that post-modern Western societies are disintegrating before our eyes and are failing to even biologically reproduce at replacement levels.
And so we turn to values. At least with values, it is possible to gain experimental knowledge in the laboratory of history. Societies reproduce values and compete with each other as well as with a hostile environment, so long lived societies should have acquired value systems more conducive to social survival.
And I don't think we can make much stronger claims than that.
Certainly someone just imagining stuff or doing some small experiments is unlikely to come up with robust rules for organizing society that will stand the test of time better than what comes from the long lived traditions of successful societies.
That sounds like an extraordinary claim that would require extraordinary evidence. No doubt physics and engineering have made great advances, but that doesn't mean that every social science that dresses itself up in the language of rationalism will be able to make similar advances, or really any advance at all. In fact, there is a case it will lead to regression rather than advancement:
Societies are very complex things made of people who respond to changing incentives. I don't believe anyone understands them very well, nor has any field in the social sciences established a track record of making highly accurate out of sample predictions, especially for the efficacy of interventions 10, 100, 1000 generations on. Even the simple question of "what keeps a society together?" is still a puzzling one, especially now that post-modern Western societies are disintegrating before our eyes and are failing to even biologically reproduce at replacement levels.
And so we turn to values. At least with values, it is possible to gain experimental knowledge in the laboratory of history. Societies reproduce values and compete with each other as well as with a hostile environment, so long lived societies should have acquired value systems more conducive to social survival.
And I don't think we can make much stronger claims than that.
Certainly someone just imagining stuff or doing some small experiments is unlikely to come up with robust rules for organizing society that will stand the test of time better than what comes from the long lived traditions of successful societies.
Someone living 1000 or even 100 years ago did not have the information we have.(eta: to design a society) Technology also lets us afford things that past societies simply couldn't do.
We sometime learn from evolved biology to make machines, but sometimes we can also see that evolution made a bad design. By learning from what has worked it's not too difficult to design an alternate society that will work just fine for decades or centuries.
Some societies survive by preying on others. Some are stable but prey on their own. I don't want a life in service to a society that evolved to be strong. I want a society that exists for the benefit of the people in it, and improves as we learn better ways to do things. (The current system does change but different structures could be much more adaptable.)
We sometime learn from evolved biology to make machines, but sometimes we can also see that evolution made a bad design. By learning from what has worked it's not too difficult to design an alternate society that will work just fine for decades or centuries.
Some societies survive by preying on others. Some are stable but prey on their own. I don't want a life in service to a society that evolved to be strong. I want a society that exists for the benefit of the people in it, and improves as we learn better ways to do things. (The current system does change but different structures could be much more adaptable.)
I should clarify. I meant "more accurately than not engaging in that pattern of thought" not "accurately". The latter is an absolute statement, the former a relative one.
And, yes, one can use alternative schools of epistemology if one believes so. This group of people derive theirs significantly from Popper's line of "critical rationalism" which is different from "empiricism" (the names here are the ones that annoyed our cousin comment over there).
This school has certain affordances that permit them to reason as they do but if you disagree the differences between us are likely so large that it is unlikely that I can explain without expending far more effort than I am willing to.
And, yes, one can use alternative schools of epistemology if one believes so. This group of people derive theirs significantly from Popper's line of "critical rationalism" which is different from "empiricism" (the names here are the ones that annoyed our cousin comment over there).
This school has certain affordances that permit them to reason as they do but if you disagree the differences between us are likely so large that it is unlikely that I can explain without expending far more effort than I am willing to.
I've always found it odd that the people called "rationalists" are not rationalist. Talking about evidence and realistic priors all the time is good (they don't do enough evidence gathering though) but it is not rationalism, it is empiricism.
Surely a rationalist would be into logic and actively refuse to read studies, like the Austrian economists did.
Surely a rationalist would be into logic and actively refuse to read studies, like the Austrian economists did.
It's just a terminology thing. Sort of like how atheist actuaries still account for Acts of God.
The word 'rational' is one of the most abused in western intellectual history. Hegel considered his system 'rational' and I think it is fairly self-evident that that has nothing to do with these internet 'rationalists'.
Even if you shouldn't be convinced of a terminal value thing based on evidence, doesn't mean you shouldn't be convinced of an empirical thing that happens to be associated with or championed by those who are promoting different values.
So, for example, in the climate change example he gives, while it is of course appropriate to take into account possible biases of people promoting one position or another, and ways that those may result from the different values that the people with those biases, one should nonetheless endeavor to have a tendency to believe what is actually true about the factual matter, even if it were that the people with the wrong values were biased as an (indirect) result of those values, in a way that led them to be more likely to reach the correct conclusion about the empirical facts, than those with the good values.
(Not to say that I think this is the case in this example. Just, hypothetically. Or, what do you call something that is like a hypothetical but without expressing any position about whether the thing is actually true?)
In addition, while they may get tangled up and which is which may change for a given person over time (due to human thought being fuzzy and wobbly), I think it still makes sense to make a distinction between terminal values and instrumental values. Further, if one values something instrumentally, because one believes that it is instrumentally beneficial for one's terminal values, and it is demonstrably true that it actually does not benefit one's terminal values, then one should come to believe this truth, and so no longer instrumentally value the thing.
I also don't think cognitive error can be entirely reduced in all cases to revealed preferences, because, uh, -- well, ok, one can, of course, model all behavior as arising from a preference to take exactly the sequence of actions that one takes. But I think you will agree that this is not a reasonable model.
People will sometimes understand their past behavior as arising from a cognitive error, and endeavor to address this. I suppose you could regard this as simply conflicting preferences, or preferences about preferences, but I don't think this is the best way to understand things.
So, for example, in the climate change example he gives, while it is of course appropriate to take into account possible biases of people promoting one position or another, and ways that those may result from the different values that the people with those biases, one should nonetheless endeavor to have a tendency to believe what is actually true about the factual matter, even if it were that the people with the wrong values were biased as an (indirect) result of those values, in a way that led them to be more likely to reach the correct conclusion about the empirical facts, than those with the good values.
(Not to say that I think this is the case in this example. Just, hypothetically. Or, what do you call something that is like a hypothetical but without expressing any position about whether the thing is actually true?)
In addition, while they may get tangled up and which is which may change for a given person over time (due to human thought being fuzzy and wobbly), I think it still makes sense to make a distinction between terminal values and instrumental values. Further, if one values something instrumentally, because one believes that it is instrumentally beneficial for one's terminal values, and it is demonstrably true that it actually does not benefit one's terminal values, then one should come to believe this truth, and so no longer instrumentally value the thing.
I also don't think cognitive error can be entirely reduced in all cases to revealed preferences, because, uh, -- well, ok, one can, of course, model all behavior as arising from a preference to take exactly the sequence of actions that one takes. But I think you will agree that this is not a reasonable model.
People will sometimes understand their past behavior as arising from a cognitive error, and endeavor to address this. I suppose you could regard this as simply conflicting preferences, or preferences about preferences, but I don't think this is the best way to understand things.
Even if you shouldn't be convinced of a terminal value thing based on evidence, doesn't mean you shouldn't be convinced of an empirical thing that happens to be associated with or championed by those who are promoting different values.
That's true, and I have been convinced of some of those things. For example, I am empirically convinced that mass shootings by spree killers are shocking but not a major issue of public health or safety in the US. Even though I believe that Democratic politics gives disproportionate attention to mass shootings, weirdly mirroring how Republican politics gives disproportionate attention to Islamist terrorism, I'm not going to say so in the company of many Democrats. Because another thing I have been empirically convinced of is that people build effective political coalitions by emphasizing common ground and judiciously overlooking their allies' faults. I want action on climate issues, so I'm not going to alienate my natural allies by lecturing them about their irrational fear of semi-automatic rifles. Effective politics is almost the thematic opposite of a personal quest to aggressively probe all things in search of Truth.
I also don't think cognitive error can be entirely reduced in all cases to revealed preferences, because, uh, -- well, ok, one can, of course, model all behavior as arising from a preference to take exactly the sequence of actions that one takes. But I think you will agree that this is not a reasonable model.
It's a model that doesn't lend itself to mathematically grounded predictions. But is it actually not reasonable? Without access to another person's qualia it's really easy to misattribute whether their more surprising decisions are revealing a preference, demonstrating cognitive error, or servicing a fundamentally different value orientation. Like Hume's Problem of Induction I can't effectively grapple with it but I can't exactly dismiss it either.
That's true, and I have been convinced of some of those things. For example, I am empirically convinced that mass shootings by spree killers are shocking but not a major issue of public health or safety in the US. Even though I believe that Democratic politics gives disproportionate attention to mass shootings, weirdly mirroring how Republican politics gives disproportionate attention to Islamist terrorism, I'm not going to say so in the company of many Democrats. Because another thing I have been empirically convinced of is that people build effective political coalitions by emphasizing common ground and judiciously overlooking their allies' faults. I want action on climate issues, so I'm not going to alienate my natural allies by lecturing them about their irrational fear of semi-automatic rifles. Effective politics is almost the thematic opposite of a personal quest to aggressively probe all things in search of Truth.
I also don't think cognitive error can be entirely reduced in all cases to revealed preferences, because, uh, -- well, ok, one can, of course, model all behavior as arising from a preference to take exactly the sequence of actions that one takes. But I think you will agree that this is not a reasonable model.
It's a model that doesn't lend itself to mathematically grounded predictions. But is it actually not reasonable? Without access to another person's qualia it's really easy to misattribute whether their more surprising decisions are revealing a preference, demonstrating cognitive error, or servicing a fundamentally different value orientation. Like Hume's Problem of Induction I can't effectively grapple with it but I can't exactly dismiss it either.
[deleted]
> It is probably a good strategy to refuse to be talked into positions against your fundamental values by mere evidence. I'm joking but also serious.
I believe that too. There are some boxes so dangerous we need people not to open them even if they believe they should.
However there are a lot of things in the world that we can be wrong about. A strategy of "I'm going to use deontological reasoning" just means that we have to sort right from wrong by an evolutionary process over generations, getting to kids while they are young enough to be swayed, in a process that is a bit icky and disrespectful of people's beliefs. It would be better if people could be persuaded by evidence.
We don't get sewage, skyscrapers and subways by following our instincts or because some people somehow made them values. Everyone wants good sewage systems. The best parts of society come from following evidence and logic as best we are able - including, I point out, the legal system.
I believe that too. There are some boxes so dangerous we need people not to open them even if they believe they should.
However there are a lot of things in the world that we can be wrong about. A strategy of "I'm going to use deontological reasoning" just means that we have to sort right from wrong by an evolutionary process over generations, getting to kids while they are young enough to be swayed, in a process that is a bit icky and disrespectful of people's beliefs. It would be better if people could be persuaded by evidence.
We don't get sewage, skyscrapers and subways by following our instincts or because some people somehow made them values. Everyone wants good sewage systems. The best parts of society come from following evidence and logic as best we are able - including, I point out, the legal system.
However there are a lot of things in the world that we can be wrong about. A strategy of "I'm going to use deontological reasoning" just means that we have to sort right from wrong by an evolutionary process over generations, getting to kids while they are young enough to be swayed, in a process that is a bit icky and disrespectful of people's beliefs. It would be better if people could be persuaded by evidence.
I don't think that you necessarily need to sway kids while they're young. 10 years ago there were no states in the US that had legalized recreational cannabis use for adults. Today there are 17, and I expect at least another 17 over the next decade. What changed? Attitudes shifted among all age groups. It didn't seem like a process of grandparents dying out and their grandchildren entering the voting booth.
It didn't seem like an evidence based process either. I was well persuaded in the 1990s that the deleterious health effects and other negative externalities of cannabis use were no worse than those corresponding to alcohol. The historical example of Prohibition was more than enough evidence for me that fueling a black market was worse than letting adults impair themselves. I wonder why evidence didn't persuade people to change laws sooner, yet the shift happened far faster than generational replacement once it started.
If anyone reading has a convincing model for how the cannabis attitude shift happened: does it only rationalize history, or does it also predict other things likely to change in the near future? There are a lot of ways we can turn past series of events into coherent stories. There are fewer that successfully foretell what happens in the next chapter.
I don't think that you necessarily need to sway kids while they're young. 10 years ago there were no states in the US that had legalized recreational cannabis use for adults. Today there are 17, and I expect at least another 17 over the next decade. What changed? Attitudes shifted among all age groups. It didn't seem like a process of grandparents dying out and their grandchildren entering the voting booth.
It didn't seem like an evidence based process either. I was well persuaded in the 1990s that the deleterious health effects and other negative externalities of cannabis use were no worse than those corresponding to alcohol. The historical example of Prohibition was more than enough evidence for me that fueling a black market was worse than letting adults impair themselves. I wonder why evidence didn't persuade people to change laws sooner, yet the shift happened far faster than generational replacement once it started.
If anyone reading has a convincing model for how the cannabis attitude shift happened: does it only rationalize history, or does it also predict other things likely to change in the near future? There are a lot of ways we can turn past series of events into coherent stories. There are fewer that successfully foretell what happens in the next chapter.
The cannabis legalization happened because of two additional factors:
A) Overincarceration of non-violent offenders
B) the reveal of racist roots in the foundational war on drugs rhetoric (which led to more strict penalties for the use of crack [cheaper street drug], for instance, than straight cocaine [high-society, white-collar drug]).
At least that seems to be the lynchpin from my point of view. It was always going to fail from mmy point of view since prohibitions in the U.S. don't have stellar records.
A) Overincarceration of non-violent offenders
B) the reveal of racist roots in the foundational war on drugs rhetoric (which led to more strict penalties for the use of crack [cheaper street drug], for instance, than straight cocaine [high-society, white-collar drug]).
At least that seems to be the lynchpin from my point of view. It was always going to fail from mmy point of view since prohibitions in the U.S. don't have stellar records.
> There are some boxes so dangerous we need people not to open them even if they believe they should.
Religion is several of those boxes, which have already been opened for centuries.
Faith, (by which I mean deliberately attempting to believe something to a degree of certainty which exceeds what is warranted by the available evidence) is one of those boxes that should be sealed up and ejected into space, or at least made to be deemed immoral by the majority of people, since it obviously is immoral.
Religion is several of those boxes, which have already been opened for centuries.
Faith, (by which I mean deliberately attempting to believe something to a degree of certainty which exceeds what is warranted by the available evidence) is one of those boxes that should be sealed up and ejected into space, or at least made to be deemed immoral by the majority of people, since it obviously is immoral.
Few people carry that definition of faith though, but many, especially during the French Revolution, felt as you do, and acted on it, not by ejecting religious people into space, but by gathering them together in barns and burning them in mass executions.
https://en.m.wikipedia.org/wiki/Infernal_columns
https://en.m.wikipedia.org/wiki/Infernal_columns
The cognitive error being discussed here relates to a systematic failures to make accurate descriptive statements. The distinction between deontological and consequentialist thinking is entirely prescriptive. Being able to make accurate descriptive statements and understanding the accuracy of descriptive statements are valuable tools regardless of if you are a consequentialist, a deontologist or adhere to some other moral framework altogether.
I believe that torture is wrong and I also believe that torture doesn't work. I could theoretically be convinced that torture works, but I would still believe that it is wrong. If I were to insist that torture doesn't work in the face of strong evidence to the contrary because of my moral beliefs that would be a cognitive error.
Regarding your footnote: They cannot be the same thing because preferences are prescriptive and errors are (at least as they are being discussed here) are descriptive.
I believe that torture is wrong and I also believe that torture doesn't work. I could theoretically be convinced that torture works, but I would still believe that it is wrong. If I were to insist that torture doesn't work in the face of strong evidence to the contrary because of my moral beliefs that would be a cognitive error.
Regarding your footnote: They cannot be the same thing because preferences are prescriptive and errors are (at least as they are being discussed here) are descriptive.
> "cognitive error" and "revealed preference" may also be different names for the same thing. Are they experimentally distinguishable?
"I would still be against torture no matter what the research shows" is revealed preference, and "I'm against it, therefore it doesn't work, no matter what the research shows" is cognitive error.
This is clearer if you suppose that I published a good, thorough study showing that torture works very well. In the former case, you could accept it as accurate but retain your opinion; in the latter case you would be forced to either change your stance or accuse me of [ineptitude, fraud, bias, secret sadism, etc].
"I would still be against torture no matter what the research shows" is revealed preference, and "I'm against it, therefore it doesn't work, no matter what the research shows" is cognitive error.
This is clearer if you suppose that I published a good, thorough study showing that torture works very well. In the former case, you could accept it as accurate but retain your opinion; in the latter case you would be forced to either change your stance or accuse me of [ineptitude, fraud, bias, secret sadism, etc].
I think this is often confused because "works" is poorly defined leading both sides to interpret it as: works == achieves my aims == matches my values
[deleted]
I believe that the author makes the basic assumption that rationality is the end.
> But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it.
“We’re an empire now, and when we act, we create our own reality. And while you’re studying that reality — judiciously, as you will — we’ll act again, creating other new realities, which you can study too, and that’s how things will sort out. We’re history’s actors . . . and you, all of you, will be left to just study what we do.”
- Karl Rove
“We’re an empire now, and when we act, we create our own reality. And while you’re studying that reality — judiciously, as you will — we’ll act again, creating other new realities, which you can study too, and that’s how things will sort out. We’re history’s actors . . . and you, all of you, will be left to just study what we do.”
- Karl Rove
throwawaysea(1)
A lot of the time, this might not be a property of the update process so much as the prior itself.
A conspiracy theory is often something that can explain any lack of observational evidence for itself (because it's covered up!). If the prior on the conspiracy being true is too high, the update on contrary evidence will be very weak just because the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole, with probability mass mostly just being shifted within the context of the theory.
("They're covering up aliens!" --> no evidence of this after fifty years? --> "It's worse than I thought; they must be using mind control!")
The only cure for this might be preventative -- just maintaining a strong doubt on deep conspiracy theories. But that also means resigning yourself to never believing in one before it's revealed. If it turns out you live in a reality where the government is using mind control to conceal aliens, you'll not be the one who realizes it before everyone else.
A conspiracy theory is often something that can explain any lack of observational evidence for itself (because it's covered up!). If the prior on the conspiracy being true is too high, the update on contrary evidence will be very weak just because the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole, with probability mass mostly just being shifted within the context of the theory.
("They're covering up aliens!" --> no evidence of this after fifty years? --> "It's worse than I thought; they must be using mind control!")
The only cure for this might be preventative -- just maintaining a strong doubt on deep conspiracy theories. But that also means resigning yourself to never believing in one before it's revealed. If it turns out you live in a reality where the government is using mind control to conceal aliens, you'll not be the one who realizes it before everyone else.
> the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole
That may be the case for conspiracy theories, but I'd expect the theory that dogs are dangerous to make much stronger predictions. On the other hand, maybe people who're afraid of dogs don't expect them to behave differently, they just think that everything dogs do is scary (which is true, since they're scared by everything dogs do.)
That may be the case for conspiracy theories, but I'd expect the theory that dogs are dangerous to make much stronger predictions. On the other hand, maybe people who're afraid of dogs don't expect them to behave differently, they just think that everything dogs do is scary (which is true, since they're scared by everything dogs do.)
It’s about power; Dogs being violent isn’t super rare, and if a dog decides to get violent, I have no control over it. The probability of the danger is low, but it is sufficiently bad even in mild cases that I do not want to take any chances.
I think you're right; it could be either.
In this example, if their fear of dogs causes them to misperceive a dog's excitement as anger, then it's closer to the scenario in the article. But if their paranoia is such that they perceive the dog is acting friendly but interpret that as the dog luring them into a sense of safety to bite them later, then it's more like the conspiracy theory case.
In this example, if their fear of dogs causes them to misperceive a dog's excitement as anger, then it's closer to the scenario in the article. But if their paranoia is such that they perceive the dog is acting friendly but interpret that as the dog luring them into a sense of safety to bite them later, then it's more like the conspiracy theory case.
The problem that I always have with this author’s material and most material in the “rationalist movement” is that after making very valid points about biases, irrational behavior, etc., he seems to leave things at some sort of implicit epistemological nihilism. Since people on the left and the right in American politics both have biases, and both believe their beliefs just as strongly as the other side, apparently we can’t discern anything about their political views other than how their views make them feel? That’s the vibe I’m getting from stuff like this:
> I've had arguments with people who believe that no pro-life conservative really cares about fetuses, they just want to punish women for being sluts by denying them control over their bodies. And I've had arguments with people who believe that no pro-lockdown liberal really cares about COVID deaths, they just like the government being able to force people to wear masks as a sign of submission. Once you're at the point where all these things sound plausible, you are doomed.
Apparently you’re doomed if you believe one of those things describes reality more than the other? They’re both extreme generalizations, obviously, and are defined imprecisely (“no true Scotsman”), but couldn’t it be that one of those two groups of people have beliefs that correspond much more with reality? Or is the conclusion that, since this is politics, and political views is intensely personal and prone to bias, that both sides are equally correct?
Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not? Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group? Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct? Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
> I've had arguments with people who believe that no pro-life conservative really cares about fetuses, they just want to punish women for being sluts by denying them control over their bodies. And I've had arguments with people who believe that no pro-lockdown liberal really cares about COVID deaths, they just like the government being able to force people to wear masks as a sign of submission. Once you're at the point where all these things sound plausible, you are doomed.
Apparently you’re doomed if you believe one of those things describes reality more than the other? They’re both extreme generalizations, obviously, and are defined imprecisely (“no true Scotsman”), but couldn’t it be that one of those two groups of people have beliefs that correspond much more with reality? Or is the conclusion that, since this is politics, and political views is intensely personal and prone to bias, that both sides are equally correct?
Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not? Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group? Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct? Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
I think the author only means you're "doomed" in the sense that "your fate is sealed": you'll find it very hard to change your mind in the face of contrary evidence, because your observations will start to feel like they always reinforce your conclusion.
Right, but the implication is that your fate is sealed and that your belief doesn’t reflect reality or isn’t “rational.” I’m assuming the author wouldn’t use the same terminology to describe one’s unwillingness to change their views from something he considers “rational” to something he considers “irrational,” e.g. he probably wouldn’t say “if you believe the Earth is not flat, and you won’t change your beliefs unless presented with compelling evidence and explanations that the Earth is flat, then you are doomed.”
I think you're reading that implication in, when it wasn't there. Nobody can know with certainty what's really true or false, and the author isn't claiming to know either or judge people for disagreeing. We can only work on improving our processes for deciding what's true, and noticing that sometimes we can end up in a situation where our beliefs will become immune to contrary evidence is an important part of that. When that happens it doesn't mean those beliefs are certainly mistaken, but it does mean your fate is, more or less, sealed.
If your faith in a flat earth is strong enough that you'll come up with new theories of optics to justify continuing to believe in it even after you're launched into orbit to see it firsthand, your belief is unshakable. But that doesn't guarantee that you're wrong, maybe you're a genius who just revolutionized cosmology and optics. But you're probably wrong.
If your faith in a flat earth is strong enough that you'll come up with new theories of optics to justify continuing to believe in it even after you're launched into orbit to see it firsthand, your belief is unshakable. But that doesn't guarantee that you're wrong, maybe you're a genius who just revolutionized cosmology and optics. But you're probably wrong.
> Nobody can know with certainty what's really true or false, and the author isn't claiming to know either or judge people for disagreeing.
Yeah, that’s the kind of epistemological meat we really need to slice into. My view is that certainty is impossible but that knowledge is possible, and that we get ourselves into a lot of messes when we confuse knowledge with certainty. The quest for knowledge should not be a quest for certainty (either in the emotional sense or the Bayesian sense) or a quest for a perfect method of obtaining or justifying knowledge.
Yeah, that’s the kind of epistemological meat we really need to slice into. My view is that certainty is impossible but that knowledge is possible, and that we get ourselves into a lot of messes when we confuse knowledge with certainty. The quest for knowledge should not be a quest for certainty (either in the emotional sense or the Bayesian sense) or a quest for a perfect method of obtaining or justifying knowledge.
That's not perfectly parallel as a belief. The parallel statement would be "If you believe that flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth"...that would be equivalent. And I can't speak for Scott, but I think he'd say if you believed that, your ability to reason about the Earth being flat is basically dead. It happens to be that you have the right answer (the Earth is round), but only by chance...if you were wrong, you wouldn't be able to be convinced to change your mind.
> The parallel statement would be "If you believe that flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth"
But...isn’t that statement either true or false, just like the statement “the Earth is flat” is either true or false? What’s the difference between the two statements? Is knowledge about one statement possible, but not the other?
But...isn’t that statement either true or false, just like the statement “the Earth is flat” is either true or false? What’s the difference between the two statements? Is knowledge about one statement possible, but not the other?
It's the connection between the two statements.
If you believe "flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth" , this would pose difficulty for changing the belief "the earth isn't flat".
If someone believed "people who claim that 'flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth' don't actually care about whether flat-Earth advocates really [...], they just [idk, some absurd motivation for making the claim] ", that would pose difficulty for changing the belief "flat-Earth advocates really do believe that the Earth is flat".
If you believe "flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth" , this would pose difficulty for changing the belief "the earth isn't flat".
If someone believed "people who claim that 'flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth' don't actually care about whether flat-Earth advocates really [...], they just [idk, some absurd motivation for making the claim] ", that would pose difficulty for changing the belief "flat-Earth advocates really do believe that the Earth is flat".
> If you believe "flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth" , this would pose difficulty for changing the belief "the earth isn't flat".
I’m not sure why. The normal simple tests to distinguish between a flat Earth and a ball-shaped Earth ought to still work independent of what flat-Earth advocates believe.
I’m not sure why. The normal simple tests to distinguish between a flat Earth and a ball-shaped Earth ought to still work independent of what flat-Earth advocates believe.
The crux of the issue here is that it's not possible to know what someone else truly believes. You hear various forms of this argument all the time. Was Trump unable to string together a coherent sentence, or was his speech style a conscious choice to appeal to certain people? Are people who identify as LGBT doing so because that's what they truly feel about themselves, or because they want to be popular and fit in?
The statements being made aren't about the earth, but about people's perception of the earth. And ultimately you can't objectively determine someone else's qualia.
What GP is suggesting is that if the claimed flat-earth belief is simply a justification for a different goal (prevent space exploration), then providing contrary evidence doesn't matter, because the belief isn't core of the issue. But you can't really know that independently. And this is especially true where you get a sort of metacircularity in beliefs and belief systems. Identifying what the "real" axiom someone subscribes to is difficult.
To look at the conspiracy theory about flat earthers that GP proposed, imagine that tomorrow, suddenly, the Earth was flat. All of the historical record was the same. We have evidence and recordings that it was round, and that the Greeks agreed, but tomorrow we couldn't reproduce those experiments. The horizon suddenly looked different. A reasonable conclusion here might be that we're all suffering from a collective delusion (and that in fact the earth is still round[0]), because our prior that the earth could rearrange itself overnight is low. But, this person might believe that the collective delusion is the result of machinations of the flat earthers, trying to trick us all into stopping further space exploration by convincing us that the earth is flat. This despite there being no evidence that flat earthers were at fault, or even capable, of causing such a worldwide event.
[0]: This actually objectively doesn't make a whole lot of sense, if we're capable of being subject to a massive collective delusion at worldwide scale, it could be that we were doing so before, and that the change is the delusion wearing off, but I digress.
The statements being made aren't about the earth, but about people's perception of the earth. And ultimately you can't objectively determine someone else's qualia.
What GP is suggesting is that if the claimed flat-earth belief is simply a justification for a different goal (prevent space exploration), then providing contrary evidence doesn't matter, because the belief isn't core of the issue. But you can't really know that independently. And this is especially true where you get a sort of metacircularity in beliefs and belief systems. Identifying what the "real" axiom someone subscribes to is difficult.
To look at the conspiracy theory about flat earthers that GP proposed, imagine that tomorrow, suddenly, the Earth was flat. All of the historical record was the same. We have evidence and recordings that it was round, and that the Greeks agreed, but tomorrow we couldn't reproduce those experiments. The horizon suddenly looked different. A reasonable conclusion here might be that we're all suffering from a collective delusion (and that in fact the earth is still round[0]), because our prior that the earth could rearrange itself overnight is low. But, this person might believe that the collective delusion is the result of machinations of the flat earthers, trying to trick us all into stopping further space exploration by convincing us that the earth is flat. This despite there being no evidence that flat earthers were at fault, or even capable, of causing such a worldwide event.
[0]: This actually objectively doesn't make a whole lot of sense, if we're capable of being subject to a massive collective delusion at worldwide scale, it could be that we were doing so before, and that the change is the delusion wearing off, but I digress.
> The crux of the issue here is that it's not possible to know what someone else truly believes.
I don’t think there’s some process to follow to acquire knowledge with a guarantee that we haven’t made a mistake, but I don’t think that means it’s impossible to acquire knowledge. I think we can and do acquire knowledge about people’s beliefs and intentions and use that knowledge to solve problems. Obviously examples are convicting people of fraud and various crimes where the person’s intentions are relevant.
I don’t think there’s some process to follow to acquire knowledge with a guarantee that we haven’t made a mistake, but I don’t think that means it’s impossible to acquire knowledge. I think we can and do acquire knowledge about people’s beliefs and intentions and use that knowledge to solve problems. Obviously examples are convicting people of fraud and various crimes where the person’s intentions are relevant.
This gets more complex in the realm of, let's call it politics, where the quest for knowledge may be adversarial, and is played over long periods. With fraud and crimes of intent, you often have very clear evidence where the person was unguarded (they said a slur as they committed the crime, making it a hate crime, for example).
I'm not saying reliably discovering intentions is always impossible. I'm saying that it is not always possible.
I'm not saying reliably discovering intentions is always impossible. I'm saying that it is not always possible.
Reflecting reality is different from being rational. It's related to the old idea that true knowledge is believing the right thing for the right reason. The problem is that if your reasoning process is incorrect, what you believe is independent of reality, so whether you believe the truth is mostly random. Your opponents have equally strong (i.e. not) reasons to believe what they do. You should aim to do better.
That quote is NOT Scott saying that pro-life advocates are right about abortion or wrong about abortion. That quote is Scott saying that once you start to believe that "the enemy" is a cackling evil villain, who is lying about their true motivations and secretly is only advocating for their preferred policies because they want to do evil things...you are doomed, at least in terms of seeking truth. Because once you believe that, there is nothing that "the enemy" can ever say or do that will change your mind. It's always just a tactic to advance their evil plot.
Of course, Scott could be wrong about this thesis. It's possible that maybe believing that pro-life conservatives don't care at all about fetuses doesn't actually indicate that you have a trapped prior that will prevent you from ever hearing new arguments on the abortion issue. But that is the argument that Scott is advancing, not that you can't form opinions about the truth or falsehood of someone's political views at all.
Of course, Scott could be wrong about this thesis. It's possible that maybe believing that pro-life conservatives don't care at all about fetuses doesn't actually indicate that you have a trapped prior that will prevent you from ever hearing new arguments on the abortion issue. But that is the argument that Scott is advancing, not that you can't form opinions about the truth or falsehood of someone's political views at all.
I guess my question is whether the author believes that it’s possible to have knowledge about these sorts of things or not. Everything he says in this article sounds like he’s saying that obtaining such knowledge is impossible, or at least that neither side has obtained knowledge. He seems to only judge the validity of the opposing views by how strongly their proponents believe the views, and since the two views seem to be held equally strongly, he seems to conclude that there is no discernible difference in the validity of the views. But if it were possible to obtain such knowledge, then it would be possible to correctly say “both sides believe their views equally strongly, but side A is more correct and side B is less correct.”
There are outside groups with less of that type of bias that will often be able to see things in a more nuanced way that more closely reflects reality. For example, maybe someone from another country who is uninterested in politics.
That doesn't mean there are some people who have no bias at all. Just different ones that apply to other things.
In the examples you quoted, he was deliberately listing views extreme enough to be inaccurate. For example, both the liberal and conservative examples said the characterization applied to _everyone_ in those groups. The less-biased interpretations might be: _some_ conservatives really _do_ care about fetuses and lives, and _some_ liberals _do_ care about preventing COVID deaths.
That doesn't mean there are some people who have no bias at all. Just different ones that apply to other things.
In the examples you quoted, he was deliberately listing views extreme enough to be inaccurate. For example, both the liberal and conservative examples said the characterization applied to _everyone_ in those groups. The less-biased interpretations might be: _some_ conservatives really _do_ care about fetuses and lives, and _some_ liberals _do_ care about preventing COVID deaths.
Agnosticism about particular partisan issues is very appropriate when discussing general patterns of thinking and failure modes. And I think he would agree with your last paragraph to the extent of thinking it too obvious to say out loud. His own politics are clear enough from his other writing.
> Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not?
I think it's important to really carefully spell out what we mean by "not having good intentions". Because there are two different reasonable things that that can mean, and they're _really_ different, but the incentive to flip-flop between them can be strong.
The first meaning sounds like "having bad ideas". For example, let's say I'm in favor of some new law to...promote the arts. (Trying to pick a bland example.) You might think my law is actually going to hurt the arts, and argue against it for that reason. Or maybe you agree that it'll help the arts, but you think its other downsides outweigh its upsides. At maximum generality, maybe you don't object to my idea per se, but you'd just rather spend the same money on something else.
The second meaning sounds like "having bad values". For example, you could claim that I'm not actually interested in art, and just trying to divert money to my friends. Or maybe I could paint you as some sort of uncultured caveman, who can't understand why normal people like art.
When we put it that way, I think it's clear that we usually say "good intentions" to talk about the second meaning. Like, the whole point of saying it is usually to _contrast_ someone's good values with some bad outcome they're associated with, right?
On the other hand, it's much easier to take an argumentative position using the first meaning. The world is complicated, everything has downsides, etc. Plus object level arguments tend to be more sophisticated and interesting, with expertise and fancy math getting involved. So a tempting strategy can be to equivocate between "having bad ideas" and "having bad values", by making an argument about ideas or outcomes and yet drawing or implying a conclusion about values. This is the sort of fallacy that internet rationalists complain about constantly, calling it a "motte and bailey" or something like that. And I'm really sympathetic to that complaint: I think that it's very easy to make this sort of bad argument, that it's actually kind of hard _not_ to make it without some sort of shared community or commitment, and that the effect of this sort of argument on discourse is frustrating, unproductive, and ultimately dehumanizing.
So all that said, do I think that some groups objectively have bad values? I...oof...I feel kind of forced to admit that it's true sometimes. I can't sustain a really extreme position here like "everyone's values are special and lovely in their own way", at least not without diluting the concept of values so much that it's not very interesting anymore. But at the same time, I just feel...super skeptical of most "bad values" arguments in practice.
I think it's important to really carefully spell out what we mean by "not having good intentions". Because there are two different reasonable things that that can mean, and they're _really_ different, but the incentive to flip-flop between them can be strong.
The first meaning sounds like "having bad ideas". For example, let's say I'm in favor of some new law to...promote the arts. (Trying to pick a bland example.) You might think my law is actually going to hurt the arts, and argue against it for that reason. Or maybe you agree that it'll help the arts, but you think its other downsides outweigh its upsides. At maximum generality, maybe you don't object to my idea per se, but you'd just rather spend the same money on something else.
The second meaning sounds like "having bad values". For example, you could claim that I'm not actually interested in art, and just trying to divert money to my friends. Or maybe I could paint you as some sort of uncultured caveman, who can't understand why normal people like art.
When we put it that way, I think it's clear that we usually say "good intentions" to talk about the second meaning. Like, the whole point of saying it is usually to _contrast_ someone's good values with some bad outcome they're associated with, right?
On the other hand, it's much easier to take an argumentative position using the first meaning. The world is complicated, everything has downsides, etc. Plus object level arguments tend to be more sophisticated and interesting, with expertise and fancy math getting involved. So a tempting strategy can be to equivocate between "having bad ideas" and "having bad values", by making an argument about ideas or outcomes and yet drawing or implying a conclusion about values. This is the sort of fallacy that internet rationalists complain about constantly, calling it a "motte and bailey" or something like that. And I'm really sympathetic to that complaint: I think that it's very easy to make this sort of bad argument, that it's actually kind of hard _not_ to make it without some sort of shared community or commitment, and that the effect of this sort of argument on discourse is frustrating, unproductive, and ultimately dehumanizing.
So all that said, do I think that some groups objectively have bad values? I...oof...I feel kind of forced to admit that it's true sometimes. I can't sustain a really extreme position here like "everyone's values are special and lovely in their own way", at least not without diluting the concept of values so much that it's not very interesting anymore. But at the same time, I just feel...super skeptical of most "bad values" arguments in practice.
I think it's also worth remembering that many versions of "bad values" might not be bad faith feigning of good values, but it might be a disagreement over values (in the same way that you might agree on values, but disagree on ideas as you described).
Two people may have different weights on the proper tradeoffs of safety versus liberty, and each might think the other has "bad values" without disagreeing with the others ideas, or thinking they are being dishonest or underhanded in any way.
Two people may have different weights on the proper tradeoffs of safety versus liberty, and each might think the other has "bad values" without disagreeing with the others ideas, or thinking they are being dishonest or underhanded in any way.
> Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not?
There are people who have bad values, and want things that are bad, yes. (Though generally they don't want them because they think they are bad.)
> Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group?
yes.
> Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct?
I'm not sure what precisely you mean by intentions being "honest and good". If I believe that my intent is honest -- well, I suppose I could be fooling myself, thinking that some behavior of mine is "honest" by twisting the meaning of the word "honest" in my mind, while still intending to cause others to have a belief which I believe to be false. Also, by intentions being "good", do you mean that the things one intends are actually good things, or that the one intending them thinks them good for those they impact, and intends them for that reason?
> Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
Knowledge is more than a feeling of certainty, yes. And, either knowledge, or something very similar to knowledge, is possible.
For example, both of the beliefs quoted, are false. There may be related claims which are true, but neither of the claims, as stated, are. (I don't see the no true Scotsman aspect of either though?)
The point isn't that different sides are equally correct. The point is demonstrating a kind of error that can be made, and giving an example on multiple sides makes it easier to understand. If one only gave examples of a type of error being made by the side the reader agrees with, there would be a danger of this being interpreted as simply an attack on the side the reader agrees with, and so the reader would be less likely to appreciate the point. On the other hand, if one only gave examples of the type of error being made by the side the reader disagrees with, while the reader may take a more positive view of what is being written, they are less likely to internalize the point as a kind of error that they, or others on their side, are prone to.
For something to be a general type of error, the type of error should be one that can be made in multiple contexts/directions/ways.
Then, to demonstrate that something is a general type of error, why not demonstrate that, by showing that it can be made in multiple contexts/directions/ways ?
The point isn't nihilism or relativism. The point is vigilance.
There are people who have bad values, and want things that are bad, yes. (Though generally they don't want them because they think they are bad.)
> Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group?
yes.
> Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct?
I'm not sure what precisely you mean by intentions being "honest and good". If I believe that my intent is honest -- well, I suppose I could be fooling myself, thinking that some behavior of mine is "honest" by twisting the meaning of the word "honest" in my mind, while still intending to cause others to have a belief which I believe to be false. Also, by intentions being "good", do you mean that the things one intends are actually good things, or that the one intending them thinks them good for those they impact, and intends them for that reason?
> Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
Knowledge is more than a feeling of certainty, yes. And, either knowledge, or something very similar to knowledge, is possible.
For example, both of the beliefs quoted, are false. There may be related claims which are true, but neither of the claims, as stated, are. (I don't see the no true Scotsman aspect of either though?)
The point isn't that different sides are equally correct. The point is demonstrating a kind of error that can be made, and giving an example on multiple sides makes it easier to understand. If one only gave examples of a type of error being made by the side the reader agrees with, there would be a danger of this being interpreted as simply an attack on the side the reader agrees with, and so the reader would be less likely to appreciate the point. On the other hand, if one only gave examples of the type of error being made by the side the reader disagrees with, while the reader may take a more positive view of what is being written, they are less likely to internalize the point as a kind of error that they, or others on their side, are prone to.
For something to be a general type of error, the type of error should be one that can be made in multiple contexts/directions/ways.
Then, to demonstrate that something is a general type of error, why not demonstrate that, by showing that it can be made in multiple contexts/directions/ways ?
The point isn't nihilism or relativism. The point is vigilance.
> I'm not sure what precisely you mean by intentions being "honest and good".
It could mean something objectively moral, or if you’re not into that sort of thing, it could just mean intentions that both groups in question would agree are good.
It could mean something objectively moral, or if you’re not into that sort of thing, it could just mean intentions that both groups in question would agree are good.
Thanks for clarifying.
I do believe that at least many moral questions have an objective/real/true answer, and maybe all well-formed ones do. (And I’m confident that at least almost all of the ones that are relevant in practice have a true answer.)
Ok, so you meant that the intent is for something which is good.
Then yes, people can have an intent for something which they think is good, and have their intent for it be because they think it is good, while they are incorrect, and in truth the thing they intend is bad.
The reason I was unsure is because I think there’s an idiom or something(?) where “good intentions” vs “ill intentions” refers to whether they intend to do what they think is beneficial to the target, vs to do something they think is against the target.
Thanks again for clarifying .
I do believe that at least many moral questions have an objective/real/true answer, and maybe all well-formed ones do. (And I’m confident that at least almost all of the ones that are relevant in practice have a true answer.)
Ok, so you meant that the intent is for something which is good.
Then yes, people can have an intent for something which they think is good, and have their intent for it be because they think it is good, while they are incorrect, and in truth the thing they intend is bad.
The reason I was unsure is because I think there’s an idiom or something(?) where “good intentions” vs “ill intentions” refers to whether they intend to do what they think is beneficial to the target, vs to do something they think is against the target.
Thanks again for clarifying .
Has Bayesian verbiage become a form of virtue signaling?
Don't get me wrong. I took statistics in college, and we covered Bayesian methods. They can't be controversial because Bayes' Theorem is a theorem, meaning it's proven. But the attempt to apply its terminology to lengthy but "soft" arguments seems like pure clutter.
Don't get me wrong. I took statistics in college, and we covered Bayesian methods. They can't be controversial because Bayes' Theorem is a theorem, meaning it's proven. But the attempt to apply its terminology to lengthy but "soft" arguments seems like pure clutter.
I don't know if virtue signaling is the right term. The better term might be "shibboleth" for so-called rationalists. It's as if you took the kids who loved lists of logical fallacies and told them they could resolve all disputes if they just reduced them to first principles and used words like "prior."
Nevermind that the first principles lose all context of the original issue - we're in a world of pure logic, where we can use theorems! To the rationalist community, everything looks like a good opportunity to write a lengthy essay reinventing the wheel with Bayes theorem. It's a cargo cult of basic statistics dressing up otherwise uninteresting arguments.
It's very tedious for people like myself who actually studied statistics and apply it professionally. Bayes theorem is barely even a theorem - it's literally a basic rearrangement of the axioms of probability. It's not some supreme revelation that imbues arguments with more credibility. Most of the time I see it invoked in discussion, it's not even rigorously quantified. You can choose arbitrary priors, so at the end of the day you never arrive at an objective truth. It's profundity for its own sake, and nothing is really achieved.
Nevermind that the first principles lose all context of the original issue - we're in a world of pure logic, where we can use theorems! To the rationalist community, everything looks like a good opportunity to write a lengthy essay reinventing the wheel with Bayes theorem. It's a cargo cult of basic statistics dressing up otherwise uninteresting arguments.
It's very tedious for people like myself who actually studied statistics and apply it professionally. Bayes theorem is barely even a theorem - it's literally a basic rearrangement of the axioms of probability. It's not some supreme revelation that imbues arguments with more credibility. Most of the time I see it invoked in discussion, it's not even rigorously quantified. You can choose arbitrary priors, so at the end of the day you never arrive at an objective truth. It's profundity for its own sake, and nothing is really achieved.
No, I don't think it is. Separately from rationalist community, there is substantial amount of work showing that Bayesian updating and reasoning is in some sense the optimal way of operating under uncertainty.
It is also certainly true that garbage in and garbage out, i.e. if you feed garbage inputs, you'll get garbage out (Bayesian or not), but the advantage in my opinion of having a framework for making decisions in cases of uncertainties is extremely useful. It is often hard to really apply this for ordinary life situations, (and maybe often not worth it), but even then it's useful to think about priors/evidence, even if you don't particularly care about the final number.
It is also certainly true that garbage in and garbage out, i.e. if you feed garbage inputs, you'll get garbage out (Bayesian or not), but the advantage in my opinion of having a framework for making decisions in cases of uncertainties is extremely useful. It is often hard to really apply this for ordinary life situations, (and maybe often not worth it), but even then it's useful to think about priors/evidence, even if you don't particularly care about the final number.
> No, I don't think it is. Separately from rationalist community, there is substantial amount of work showing that Bayesian updating and reasoning is in some sense the optimal way of operating under uncertainty.
With the very important proviso that the events you're uncertain about are at least probabilistically quantifiable and the "priors" are credible. It's very easy to write a long winded essay arriving at any conclusion you want if it's a subject with significant uncertainty and you can make a halfway decent appeal to whatever priors you want.
With the very important proviso that the events you're uncertain about are at least probabilistically quantifiable and the "priors" are credible. It's very easy to write a long winded essay arriving at any conclusion you want if it's a subject with significant uncertainty and you can make a halfway decent appeal to whatever priors you want.
I see two main problems with the way the lesswrong-type people try to do Bayesianism as a life philosophy. The first is that claiming to be rational* and having "priors" and "updating" them instead of "beliefs" and "changing" them is humblebragging and doesn't admit that you have beliefs that might be wrong. Even if the original idea was to make you more humble, it's dangerous when it spreads to people like young male VCs and Elon Musk who already have the world's biggest egos.
The second is there's no way to set priors for silly ideas and they won't admit some things just ain't gonna happen, which means they've accidentally started a religion by refusing to not believe in various logic traps they call "existential risks" like Roko's basilisk, evil AI gods enslaving them, and so on.
* for some reason they call themselves rationalist which implies being "rational", aka "always right", even though they are explicitly not rationalist, but empiricist!
The second is there's no way to set priors for silly ideas and they won't admit some things just ain't gonna happen, which means they've accidentally started a religion by refusing to not believe in various logic traps they call "existential risks" like Roko's basilisk, evil AI gods enslaving them, and so on.
* for some reason they call themselves rationalist which implies being "rational", aka "always right", even though they are explicitly not rationalist, but empiricist!
That's fair. A lot of what I read about "Bayesian reasoning" boils down to: Check your assumptions, and update your beliefs when faced with new evidence. But those habits have been with us for millennia, and I'm not arguing against them. My parents were educated in an earlier rationalist movement, and I'm deeply influenced by them.
Being able to think about probabilities is useful, if they're quantifiable. And conditional probability plays a useful role in that process.
Being able to think about probabilities is useful, if they're quantifiable. And conditional probability plays a useful role in that process.
I think it's been common in the rationalist community for a while, in the 5-10 years that I've been aware of them.
I can’t speak to how others interpret it, but to me, I read it as a way to apply a pretense of mathematical rigor in otherwise unrigorous contexts (like psychology or behavioral economics).
I wish the scroll bar range conformed to the length of the article rather than the length of the article plus comments. SSC/ASC articles are already quite long. I was almost put off reading by the languid progression of the scroll bar marker.
> I wonder if it's the equivalent of making trauma victims describe the traumatic event in detail; an attempt to give higher weight to the raw experience pathway compared to the prior pathway.
> The other promising source of hope is psychedelics.
For me, what this part if the essay points to is that everyone needs to do their own work, in whatever context, and there's no getting around that.
Experts, authorities, and managers need to find the line between facilitating a process and doing the process themselves. You can't make anyone believe anything or do anything. You can only hope to facilitate the conditions where helpful beliefs and accomplishments arise.
I hope this is one significant cultural shift during my lifetime. The 20th century was about creating expertise and giving that expertise authority, which is just another expression of the dominator paradigm.
> The other promising source of hope is psychedelics.
For me, what this part if the essay points to is that everyone needs to do their own work, in whatever context, and there's no getting around that.
Experts, authorities, and managers need to find the line between facilitating a process and doing the process themselves. You can't make anyone believe anything or do anything. You can only hope to facilitate the conditions where helpful beliefs and accomplishments arise.
I hope this is one significant cultural shift during my lifetime. The 20th century was about creating expertise and giving that expertise authority, which is just another expression of the dominator paradigm.
So, can you stick the cynophobe in the room with the well-behaved Rottweiler and also give them some drug that reduces their anxiety just while they’re in the room?
There's been recent research on a drug that can help make exposure therapy faster and more effective in just this way, although I can't remember what it's called and wasn't able to find it with a quick search. :-(
Edit: I found it with a less-quick search; it's called D-cycloserine.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/
I think there is still other research on using other memory-affecting and/or emotion-affecting medications to facilitate exposure therapy.
Edit: I found it with a less-quick search; it's called D-cycloserine.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/
I think there is still other research on using other memory-affecting and/or emotion-affecting medications to facilitate exposure therapy.
Is the Rottweiler particularly anxious to be in the room?
The argument here is insufficiently clear on the mechanism by which priors get trapped. I'm thinking the argument goes something like this:
a Bayesian updater takes in new evidence, and the adjusts their priors based on that evidence. Crucially, the information that is fed back into the Bayesian machine should be independent of the prior.
People do not have direct access to evidence. They only have their perception of that evidence, and the content of those perceptions depend in various complicated ways on people's priors. Therefore, evidence that should move people's priors may move them less than they should.
Sounds plausible. I think reinforcement learning algorithms like temporal difference learning are probably a more accurate description of how people learn tha Bayesian update, but the issue described would apply regardless.
a Bayesian updater takes in new evidence, and the adjusts their priors based on that evidence. Crucially, the information that is fed back into the Bayesian machine should be independent of the prior.
People do not have direct access to evidence. They only have their perception of that evidence, and the content of those perceptions depend in various complicated ways on people's priors. Therefore, evidence that should move people's priors may move them less than they should.
Sounds plausible. I think reinforcement learning algorithms like temporal difference learning are probably a more accurate description of how people learn tha Bayesian update, but the issue described would apply regardless.
Its weird to me that people are not familiar with the concept of worldview or beliefs and that they have an often overwhelming influence over thinking.
And I think that everyone has a worldview. No one is impervious to this or truly non-partisan. Some people may be less partisan in some ways. But they still have worldviews that influence their thinking, possibly in somewhat different areas.
And I think that everyone has a worldview. No one is impervious to this or truly non-partisan. Some people may be less partisan in some ways. But they still have worldviews that influence their thinking, possibly in somewhat different areas.
Is it just me or does anyone else find the “just-so” style of explaining phenomenon in this article irritating?
Edit:
Less snarky response here. The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic.
Edit:
Less snarky response here. The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic.
What do you mean by "just-so" style? I'm not familiar with the term.
> In science and philosophy, a just-so story is an untestable narrative explanation for a cultural practice, a biological trait, or behavior of humans or other animals. The pejorative[1] nature of the expression is an implicit criticism that reminds the listener of the essentially fictional and unprovable nature of such an explanation. Such tales are common in folklore and mythology (where they are known as etiological myths—see etiology).
https://en.wikipedia.org/wiki/Just-so_story
https://en.wikipedia.org/wiki/Just-so_story
It's a neuromyth being applied to political domains so that the believers of political narratives don't have to feel accountable for anything.
> The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic.
This is endemic to the rationalist community. It's actually quite humorous once you begin to notice it.
This is endemic to the rationalist community. It's actually quite humorous once you begin to notice it.
But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it.
This could describe cognitive error [1] or it could describe someone who's using deontological reasoning more than consequentialist reasoning. (And perhaps lacking the vocabulary to say "consequentialist arguments won't sway my values.")
It is probably a good strategy to refuse to be talked into positions against your fundamental values by mere evidence. I'm joking but also serious. I have a vague sense that torturing suspects doesn't work to prevent terrorist attacks, but that's not the root of my opposition to torture. Maybe clever torturers could make it work more often than not. That wouldn't sway me, because torturing suspects is against my values. Documenting how well it can "work" won't make me reconsider.
[1] Further food for thought: "cognitive error" and "revealed preference" may also be different names for the same thing. Are they experimentally distinguishable?