Book Review: From Oversight To Overkill(astralcodexten.substack.com)
astralcodexten.substack.com
Book Review: From Oversight To Overkill
https://astralcodexten.substack.com/p/book-review-from-oversight-to-overkill
49 comments
> The IRB demanded that he give his patients consent forms warning that they could get smallpox. Dr. Knight tried to explain that smallpox had been extinct in the wild since the 1970s, the only remaining samples in US and Russian biosecurity labs...
> ...the IRB demanded patients sign consent forms in pen (not pencil) but the psychiatric ward would only allow patients to have pencils (not pen)...
> ...the IRB demanded patients sign consent forms in pen (not pencil) but the psychiatric ward would only allow patients to have pencils (not pen)...
Interesting. The thing that seems missing here is deeper comparison to other countries (there is only a brief mention of britian). Its presented as an american problem, so how does the american system compare to other systems? Do they have the same problems? Did everyone copy america?
Balancing safety against freedom and responsibility. Let’s hope the pendulum swings like free-range parenting (freedom, responsibility) is having a moment over helicopter parenting (safety):
https://www.freerangekids.com/
https://en.m.wikipedia.org/wiki/Free-range_parenting
https://www.freerangekids.com/
https://en.m.wikipedia.org/wiki/Free-range_parenting
In '84 I had a college class in experimental psychology, and the culmination of that was to perform our own experiment, which had to clear the IRB. Mine involved measuring the student-subject's galvanic skin response and showing them photos of art. To calibrate that device the protocol was for the experimenter to slap their hands together a few feet away from the subject. That was too much for the IRB even 40 years ago. They did not allow the slap. Too traumatic I guess. It didn't really make a difference because it was a toy study for a class. But that was a great lesson in IRBs.
If Scott's wondering whom specifically to blame: Accretion of safeguards. It's inevitable, it seems. Something goes wrong, the process is amended to avoid that failure case. Something else goes wrong, add another amendment.
All individual additions are likely net positive - positive in 99% of the cases, negative in the rest. Assuming events are mostly independent, you'll sooner or later are in a world were net positive outcomes are becoming less likely than net negative ones.
You can sometimes sidestep that by combining/rewriting rules to some extent, but in the limit, you can only avoid all additional risk if you prevent any changes.
That's not a fault of IRBs. That's a fault of trying to minimize risks. I'm not sure there is a good answer here outside of "here's the maximum amount of risk we're willing to take, and here are the few clear ethical boundaries we'll never overstep".
There's a reason any place that's dealing with a lot of constraints is concerned with risk budgets, not risk elimination. And I'm not sure there's a lot of thinking on probabilistic ethics. (Or maybe I just can't find it)
All individual additions are likely net positive - positive in 99% of the cases, negative in the rest. Assuming events are mostly independent, you'll sooner or later are in a world were net positive outcomes are becoming less likely than net negative ones.
You can sometimes sidestep that by combining/rewriting rules to some extent, but in the limit, you can only avoid all additional risk if you prevent any changes.
That's not a fault of IRBs. That's a fault of trying to minimize risks. I'm not sure there is a good answer here outside of "here's the maximum amount of risk we're willing to take, and here are the few clear ethical boundaries we'll never overstep".
There's a reason any place that's dealing with a lot of constraints is concerned with risk budgets, not risk elimination. And I'm not sure there's a lot of thinking on probabilistic ethics. (Or maybe I just can't find it)
> Ezra Klein calls this “vetocracy”, rule by safety-focused bureaucrats whose mandate is to stop anything that might cause harm, with no consideration of the harm of stopping too many things. It’s worst in medicine, but everywhere else is catching up.
This really scares me. It seems like every important facet of American life that can be touched by increased, predatory, and superfluous litigation and regulation has. If we take Scott's word for it, this 'vetocracy' costs 10 000 lives a year due to IRB over-diligence, makes Bay-area housing completely unaffordable, reduces the quality of education, and so much more.
It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach.
This really scares me. It seems like every important facet of American life that can be touched by increased, predatory, and superfluous litigation and regulation has. If we take Scott's word for it, this 'vetocracy' costs 10 000 lives a year due to IRB over-diligence, makes Bay-area housing completely unaffordable, reduces the quality of education, and so much more.
It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach.
> It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach.
The housing is also really, really important: the Silicon Valley, San Francisco and New York etc are some of the most productive parts of your country. If more people were allowed to live there [0], your economy could grow a lot. A bigger economy directly translates to lives saved.
To give a simple example: your highway engineering people work with a value of life of something like x million USD. Any improvement to safety that costs less than x million USD per life saved gets implemented. The value of x gets set in vague relation to how rich society is.
[0] No any one individual is directly barred from moving to SF, if they pony up enough money for rent; but the NIMBY climate effectively sets a limit on the total number of people allowed to dwell there.
The housing is also really, really important: the Silicon Valley, San Francisco and New York etc are some of the most productive parts of your country. If more people were allowed to live there [0], your economy could grow a lot. A bigger economy directly translates to lives saved.
To give a simple example: your highway engineering people work with a value of life of something like x million USD. Any improvement to safety that costs less than x million USD per life saved gets implemented. The value of x gets set in vague relation to how rich society is.
[0] No any one individual is directly barred from moving to SF, if they pony up enough money for rent; but the NIMBY climate effectively sets a limit on the total number of people allowed to dwell there.
Location has almost nothing to do with the productivity of workers in Silicon Valley.
Okay, but then why do folks move there? My impression is that it’s mostly for work.
Companies mandate going to the office and remote work only became acceptable en masse after the pandemic?
Why would profit maximizing companies have those mandates, if location is not important?
Also: Google (for example) has offices in lower cost locations, too. Why do they keep the offices in Silicon Valley open at all, if location is not important?
Also: Google (for example) has offices in lower cost locations, too. Why do they keep the offices in Silicon Valley open at all, if location is not important?
If that's true there is no reason for Silicon Valley to exist in the first place.
But it does exist.
But it does exist.
Im not much of english speaker so I wonder if my intuition is right here but its sounds like non sequitor. Why do You think there are no other possible scenarios which explains Silicon Valley existence? Like people herd mentality (even the high inteligent one are not resistant to it) or just simple power accumulation by few that make the decisions where to take up residency? With almost free capital and dollar as world reserve currency current Silicon Valley situation can be solvent by accident just long enough for people like You beliving it as not being irrational.
Google, Facebook, Apple etc already have offices in lower cost locations.
Why would they keep their high rent offices in Silicon Valley open?
Are you saying they decision makers are all irrational? They still have to compete in the market. Any company that could get equal productivity out of random locations could eat their lunch with a lower cost base.
Why would they keep their high rent offices in Silicon Valley open?
Are you saying they decision makers are all irrational? They still have to compete in the market. Any company that could get equal productivity out of random locations could eat their lunch with a lower cost base.
> your highway engineering people work with a value of life of something like x million USD.
This is not currently nor should it ever be how civil engineering works.
This is not currently nor should it ever be how civil engineering works.
You might like https://en.wikipedia.org/wiki/National_Institute_for_Health_... more. It's how the NHS decides what interventions to fund. (A similar idea was demonised in the US as https://en.wikipedia.org/wiki/Death_panel )
Why do you think decisions _should_not_ be made like this? How else do you want decisions to be made?
Why do you think decisions _should_not_ be made like this? How else do you want decisions to be made?
Good luck finding any engineering project which uses this.
Engineering projects select their safety features based on what funding is available for which features in a given community. I'm sure there's someone in the federal government somewhere who thinks anyone uses VSL guidance, but that doesn't make it so.
Engineering projects select their safety features based on what funding is available for which features in a given community. I'm sure there's someone in the federal government somewhere who thinks anyone uses VSL guidance, but that doesn't make it so.
There's probably nothing in modern history that hurts upward mobility and quality of life more than the disaster that is housing in western countries. Like most things it's a product of good intentions and previous winners using gov to kick the rug out for everyone else. The Bay Area is the classic example but this exists in urban (and increasingly suburban) areas in almost every wealthy western country: UK, Canada, Australia, etc.
The only major-changes that actually help are stuff like work arounds ala working from home or using AirBnB to offset mortgage costs.
I've read that the meme of "middle class is dead" is mostly bullshit, incomes have stayed consistent, the only major difference is cost of living, which housing ranks #1 by far.
The only major-changes that actually help are stuff like work arounds ala working from home or using AirBnB to offset mortgage costs.
I've read that the meme of "middle class is dead" is mostly bullshit, incomes have stayed consistent, the only major difference is cost of living, which housing ranks #1 by far.
Don't forget college. The [very good] public university I got my degree from in the 80s charged me about $180 per semester for tuition. The price now is $7500. That doesn't include books, housing, food, etc.
The US, the UK, Canada and Australia are all common law countries. Wealthy western countries that have administrative law, like Japan, Germany and France do not have this problem to the same extent.
That's an interesting point, I wasn't aware of that. I asked GPT to explain how this affects housing policy compared to the US and it does sound much better. Simpler processes based on clear sets of rules rather than complex caselaw, public participation is more limited and not imposed at every stage, and "there is typically a single permitting authority responsible for issuing permits and overseeing compliance, which can reduce bureaucratic obstacles and simplify the process for developers".
Some of these can be addressed without shifting this stuff to administrative law, but it's a good baseline.
Some of these can be addressed without shifting this stuff to administrative law, but it's a good baseline.
And like the IRB there's historical precedent for the laws the previous winners use to freeze development. Before the creation of laws like NEPA and CEQA in the US, civil engineering projects would frequently lead to environmental issues. Before local control over housing became the norm, minority neighborhoods and shops were razed to accommodate highways. Now those laws are being weaponized to freeze development.
It’s not bullshit, it’s just more nuanced than that, as things often are.
https://www.pewresearch.org/fact-tank/2022/04/20/how-the-ame...
https://www.pewresearch.org/fact-tank/2022/04/20/how-the-ame...
watch out for Chesterton’s Fence, though
> ... ethicist Jerry Menikoff [...] had decided maybe it was unethical to do RCTs on ventilator settings at all. He asked whether they might be able to give every patient the right setting while still doing the study. The study team tried to explain to him that they didn’t know which was the right setting, that was why they had to do the study. He wouldn’t budge.
> Finally, after a year, another panel of experts ruled in favor of the investigators and gave them permission to restart the study right away. They did, but the delay was responsible for thousands of deaths, and produced a chain effect on ventilator research that made us less prepared for the surge in ventilator demand around COVID.
People who do not understand or accept the role of risk-taking in the expansion of scientific knowledge and the betterment of the human condition should not be in positions of authority over the scientific process.
I have to ask: what went wrong here? Why would such a person be in this position? Surely a judgment like this is exceptionally bad...? Or has the rot spread much farther, to the point where incompetence this severe is normal?
> Finally, after a year, another panel of experts ruled in favor of the investigators and gave them permission to restart the study right away. They did, but the delay was responsible for thousands of deaths, and produced a chain effect on ventilator research that made us less prepared for the surge in ventilator demand around COVID.
People who do not understand or accept the role of risk-taking in the expansion of scientific knowledge and the betterment of the human condition should not be in positions of authority over the scientific process.
I have to ask: what went wrong here? Why would such a person be in this position? Surely a judgment like this is exceptionally bad...? Or has the rot spread much farther, to the point where incompetence this severe is normal?
That science requires risk taking means that the more institutionalized it becomes, the less scientific it becomes as institutions are naturally allergic to risk.
Institutions are allergic to short-term, acute risks that result from their actions. They are insensitive to long-term, chronic risks that results from their inaction.
Meta-risk, the risk that comes from not taking any risk, needs to be drilled into everyone's head: the public, journalists, politicians, IRBs, etc. I think the concept that not taking risks is risky should have its own label, but if someone doesn't like "meta-risk", maybe they will accept "risk tradeoffs".
Meta-risk, the risk that comes from not taking any risk, needs to be drilled into everyone's head: the public, journalists, politicians, IRBs, etc. I think the concept that not taking risks is risky should have its own label, but if someone doesn't like "meta-risk", maybe they will accept "risk tradeoffs".
> Institutions are allergic to short-term, acute risks that result from their actions. They are insensitive to long-term, chronic risks that results from their inaction.
That's not true in general. It all comes down to incentives on both the institution and individual actors.
To give a counter-example to your claim, there are plenty of companies who see the writing on the wall, and desperately try to do something to avoid the risk of inaction.
Facebook might be an interesting example: it's slowly sliding into less and less relevance, and the 'metaverse' is a big and bold bet to actively change that. (To make that example even better, as far as I can tell, it's not working out. So we can guess that there was considerable risk in taking that bet.)
That's not true in general. It all comes down to incentives on both the institution and individual actors.
To give a counter-example to your claim, there are plenty of companies who see the writing on the wall, and desperately try to do something to avoid the risk of inaction.
Facebook might be an interesting example: it's slowly sliding into less and less relevance, and the 'metaverse' is a big and bold bet to actively change that. (To make that example even better, as far as I can tell, it's not working out. So we can guess that there was considerable risk in taking that bet.)
In the context of this thread, I was referring to institutions that conduct basic science (i.e., primarily universities and government agencies) rather than business, which is not the first thing I associate with the label "institution".
I think Facebook and the metaverse is a great counter-example, but it also seems to highlight the general trend. I would be very surprised by any institution where power is diffuse understanding the value of avoiding meta-risk. Business can concentrate immense power in the CEO, particularly when they are also the largest shareholder, which enables them to act decisively.
I would love to see examples of where groups or co-operative without powerful individuals collectively chose to embrace risk to avoid meta-risk, since they could provide incredibly valuable lessons.
I think Facebook and the metaverse is a great counter-example, but it also seems to highlight the general trend. I would be very surprised by any institution where power is diffuse understanding the value of avoiding meta-risk. Business can concentrate immense power in the CEO, particularly when they are also the largest shareholder, which enables them to act decisively.
I would love to see examples of where groups or co-operative without powerful individuals collectively chose to embrace risk to avoid meta-risk, since they could provide incredibly valuable lessons.
I think to some extent, we're just not naturally wired to think of meta-risk. We have innate biases against negative outcomes, but since risks—or surefire outcomes, like in Scott's post—are more abstract, we just don't naturally think about them. I think you're right though, we have to start teaching people that not taking a small risk can have a much larger impact.
To have a labor market that demands people who can perceive abstract issues and interconnected system failures requires, in the first place, a prerequisite population that has the cognitive ability to conceive big picture models which include risk contemplation as a decisive factor in organizational success. These intelligent representatives will not be working for free without indoctrination projects like the lifelong instilling of patriotism. So, realistically, any executive officer who can meet the requirements for a demanding large scale management labor will be selective in what administration he serves. Leaving those unfortunate enterprises that can't satisfy natural requirements left to fend for themselves with whatever unsustainable capabilities they can acquire or start up with. All this is broad, globally occurring competition for talent, labor, and resources and it provides an explanation for why things are the way they are.
This comprehension on business management can lead to the conclusion that socialist visions will always have capitalist characteristics associated with their attempted executions. So it seems there's an extensive and influential capitalism that exists somewhere in the world, right now. The only way for socialist goals to prosper is to remove that wide-reaching capitalism.
So just educating people about risk might actually, ironically and funnily, be an unproductive decision and indicative of a much more interesting problem.
This comprehension on business management can lead to the conclusion that socialist visions will always have capitalist characteristics associated with their attempted executions. So it seems there's an extensive and influential capitalism that exists somewhere in the world, right now. The only way for socialist goals to prosper is to remove that wide-reaching capitalism.
So just educating people about risk might actually, ironically and funnily, be an unproductive decision and indicative of a much more interesting problem.
I've been thinking for a while that it's time to start fresh building institutions that can take over the tasks our current universities have largely drifted away from.
> Why would such a person be in this position?
Such a person will always be in this position, it is the type of personality that thrives in a bureaucratic institution. No amount of fiddling with the process will fix it, you need an alternative form of decision making
Such a person will always be in this position, it is the type of personality that thrives in a bureaucratic institution. No amount of fiddling with the process will fix it, you need an alternative form of decision making
I love how the word oversight has two opposite meanings.
You'll like https://en.wikipedia.org/wiki/Auto-antonym then.
Side note: the French example “apprendre” listed in the article seems like it might not really belong in this category. On it’s own it always means “to learn”, and it is only when you use it in specific grammatical constructions, for example following indicative personal pronouns, that it takes the opposite meaning: “lui apprendre” or “leur apprendre” means “to teach him/her/them”. The meaning is not merely inferred from context, it is determined by grammar.
I think there are two things at work. First, if you pay someone to find problems with a study, they're going to want to find problems with the study to justify their existence.
Associated with that is that many people don't consider the level of risk when they are considering whether something is or is not a problem.
And both things don't just apply to IRBs. A friend told me about when his workplace had a security assessment (for Reasons). the assessor's only finding: that a houseplant by the door had sharp leaves and could be used by a malicious intruder as a weapon. Yes, I suppose so. Job done by security assesso. But, just perhaps, a malicious intruder might not choose to rely on being able to slash someone with the houseplant, in which case worse things might possibly happen.
Associated with that is that many people don't consider the level of risk when they are considering whether something is or is not a problem.
And both things don't just apply to IRBs. A friend told me about when his workplace had a security assessment (for Reasons). the assessor's only finding: that a houseplant by the door had sharp leaves and could be used by a malicious intruder as a weapon. Yes, I suppose so. Job done by security assesso. But, just perhaps, a malicious intruder might not choose to rely on being able to slash someone with the houseplant, in which case worse things might possibly happen.
> if you pay someone to find problems with a study, they're going to want to find problems with the study to justify their existence.
This is the same reason a home inspection report during a real estate sale ALWAYS finds problems with the property for sale
This is the same reason a home inspection report during a real estate sale ALWAYS finds problems with the property for sale
> Lawyers sue institutions every time they harm someone (but not when they fail to benefit someone)
> pull the culture towards celebrating harm-avoidance as the greatest good, and cast suspicion on anyone who tries to add benefit-getting to the calculation
Minor nitpick, but it is more accurate to say that they prefer prevention of new harms over the prevention of old harms.
The current wording (plus the commentary over Gnosticism) kinda makes it sound like negative utilitarianism, or harm aversion (preferring the reduction of suffering over the addition of "positive pleasures") which would be a legitimate position. But the post is something more like "new risks aversion", i.e. preferring current harms over possible new harms regardless of the lower intensity of the "possible new harms", which sounds pretty irrational.
> pull the culture towards celebrating harm-avoidance as the greatest good, and cast suspicion on anyone who tries to add benefit-getting to the calculation
Minor nitpick, but it is more accurate to say that they prefer prevention of new harms over the prevention of old harms.
The current wording (plus the commentary over Gnosticism) kinda makes it sound like negative utilitarianism, or harm aversion (preferring the reduction of suffering over the addition of "positive pleasures") which would be a legitimate position. But the post is something more like "new risks aversion", i.e. preferring current harms over possible new harms regardless of the lower intensity of the "possible new harms", which sounds pretty irrational.
Let’s engage in some speculative thinking:
Somebody, let’s call them Dr. X, invents a computer algorithm that cures cancer. It works by running simulations of the patient’s cells. Its output is a list of diverse interventions, for example, pairs of RNA fragments and custom delivery mechanisms for each fragment. For a given patient, the algorithm outputs between 100 and 10000 interventions, all of them different and tailored to the patient. The simulation also takes into account possible adverse effects, it’s really a cure.
With the current regulatory framework, does Dr. X has any hope of his invention being used at hospitals in less than a decade, or all he can do is to use it on himself and/or make the algorithm available in the dark web?
Somebody, let’s call them Dr. X, invents a computer algorithm that cures cancer. It works by running simulations of the patient’s cells. Its output is a list of diverse interventions, for example, pairs of RNA fragments and custom delivery mechanisms for each fragment. For a given patient, the algorithm outputs between 100 and 10000 interventions, all of them different and tailored to the patient. The simulation also takes into account possible adverse effects, it’s really a cure.
With the current regulatory framework, does Dr. X has any hope of his invention being used at hospitals in less than a decade, or all he can do is to use it on himself and/or make the algorithm available in the dark web?
(Internal anger: 150/100)
Add another tally mark to "bureaucracies can only kill, never create".
Try saying "accept what we cannot avoid" to my skewed feet, OHRP. I WISH I had the finances to fix them, but because of people like you, any advancements on cheap treatments for this issue will be met with hundreds of miles of red tape just to form a simple test.
This quote sums up how I feel about the OHRP, and all bureaucracies similar to it:
> A survey of how researchers feel about IRBs did include one person who said “I hope all those at OHRP [the bureaucracy in charge of IRBs] and the ethicists die of diseases that we could have made significant progress on if we had [the research materials IRBs are banning us from using]”.
Add another tally mark to "bureaucracies can only kill, never create".
Try saying "accept what we cannot avoid" to my skewed feet, OHRP. I WISH I had the finances to fix them, but because of people like you, any advancements on cheap treatments for this issue will be met with hundreds of miles of red tape just to form a simple test.
This quote sums up how I feel about the OHRP, and all bureaucracies similar to it:
> A survey of how researchers feel about IRBs did include one person who said “I hope all those at OHRP [the bureaucracy in charge of IRBs] and the ethicists die of diseases that we could have made significant progress on if we had [the research materials IRBs are banning us from using]”.
"Doctors are told to weigh the benefits vs. costs of every treatment. So what are the benefits and costs of IRBs?
"Whitney can find five people who unexpectedly died from research in the past twenty-five years. ...
"What are the costs? ...the monetary costs are around the order of $1.6 billion.
"What about non-monetary costs? ... Low confidence estimate, but somewhere between 10,000 and 100,000 Americans probably die each year from IRB-related research delays.
"So the cost-benefit calculation looks like - save a tiny handful of people per year, while killing 10,000 to 100,000 more, for a price tag of $1.6 billion. If this were a medication, I would not prescribe it."
"Whitney can find five people who unexpectedly died from research in the past twenty-five years. ...
"What are the costs? ...the monetary costs are around the order of $1.6 billion.
"What about non-monetary costs? ... Low confidence estimate, but somewhere between 10,000 and 100,000 Americans probably die each year from IRB-related research delays.
"So the cost-benefit calculation looks like - save a tiny handful of people per year, while killing 10,000 to 100,000 more, for a price tag of $1.6 billion. If this were a medication, I would not prescribe it."
This is a big trolley problem, isn't it? Do studies that risk 2 deaths per decade, or do no studies and let tens of thousands die.
That Hans Jonas quote is insane. I hate insane americans, why do they have such outsize influence on culture there (and through cultural imperialism everywhere else)?
That Hans Jonas quote is insane. I hate insane americans, why do they have such outsize influence on culture there (and through cultural imperialism everywhere else)?
[deleted]
It seems a common human failing to not be able to cope with rules that affect them.
One you regularly see here is the complaint about Wikipedia: "I tried to edit the page about me/my work/my family and they wouldn't let me. But I'm the expert on me/my work/my family and it's currently wrong".
Which sounds outrageous!. But at the same time "people can freely edit their own pages" has obvious failure modes too.
Similarly, getting mental patients to sign in pen can be presented as pointless nonsense. Or it can be seen as part of a strategy to stop defenceless patients being taken advantage of by malicious doctors and stop vulnerable doctors being sued by malicious patients.
One you regularly see here is the complaint about Wikipedia: "I tried to edit the page about me/my work/my family and they wouldn't let me. But I'm the expert on me/my work/my family and it's currently wrong".
Which sounds outrageous!. But at the same time "people can freely edit their own pages" has obvious failure modes too.
Similarly, getting mental patients to sign in pen can be presented as pointless nonsense. Or it can be seen as part of a strategy to stop defenceless patients being taken advantage of by malicious doctors and stop vulnerable doctors being sued by malicious patients.
[deleted]
Meanwhile finding out the number of deaths caused by extra caution and delaying the experiment requires a calculation, and then a very lengthy blog post to be communicated.
Bureaucrats don't care about the second kind of deaths because they can hardly be linked to their actions. Now the first kind of deaths are directly their responsibility. And guess what, they can be avoided with a flick of a pen.