Analysis of the Imperial College Epidemiological Model(m.facebook.com)
m.facebook.com
Analysis of the Imperial College Epidemiological Model
https://m.facebook.com/scarlett.strong.1/posts/252437219500976
23 comments
According to the Atlantic
> Johnson instead offered a suite of soft advice—people with symptoms should stay home; no school trips abroad; people over 70 should avoid cruises.
So to be clear, even the most "do nothing" approach was not "do nothing", and at that time, people were alread voluntarily starting to socially distance.
> Johnson instead offered a suite of soft advice—people with symptoms should stay home; no school trips abroad; people over 70 should avoid cruises.
So to be clear, even the most "do nothing" approach was not "do nothing", and at that time, people were alread voluntarily starting to socially distance.
These options are so soft that they really are equivalent to doing nothing. Sick people have always been told to stay home; trips abroad and on ships might cause visible clusters early on, but they're a drop in the bucket later when the disease is already at home.
The real issue is that a strong response causes people to take things seriously, and a weak response causes them to not take it seriously. In other words, just like the model, we have a situation where predicted results inform the actions taken, which affect popular opinion, which in turn change the predicted results.
Yes, the optimal outcome is to somehow have the government do nothing, thus restricting nobody's freedoms, but simultaneously have society take it so seriously that they eradicate the disease themselves in two weeks. But that is not a genuine policy option.
The real issue is that a strong response causes people to take things seriously, and a weak response causes them to not take it seriously. In other words, just like the model, we have a situation where predicted results inform the actions taken, which affect popular opinion, which in turn change the predicted results.
Yes, the optimal outcome is to somehow have the government do nothing, thus restricting nobody's freedoms, but simultaneously have society take it so seriously that they eradicate the disease themselves in two weeks. But that is not a genuine policy option.
You say that, but your intuition about those measures isn't good enough, which is why we needed a reliable model to tell us whether those measures were going to be effective, that's the entire point of the research. If our starting point is "Oh well obviously we need to do lock down" then why are we wasting money on modelling it? And since we haven't modelled the system without public policy intervention, none of the results mean anything.
Some questions I ask myself when reading random posts with grand and important claims on any subject:
Where is this from? Who originally wrote it? Is this text’s origin really a random Facebook post, from a pseudonymous author with a cartoon profile picture and no claim of any serious credentials in the subject at hand? (Whether epidemiology or anything else)
Regardless of the merits of the text’s post (which I do not claim to be able to judge) all evidence has to be analysed for context as well as content. Simple “common sense” claims (with a couple of big words to impress non-epidemiologists like me) are made to debunk the models: where is the evidence rather than rhetoric, even some basic citations, and/or examples of or links to counter-modelling? The post doesn’t even link to the original model files from Imperial that they’re claiming to critique.
It’s perfectly _possible_ that the claims made in this Facebook post are correct, but it doesn’t mean anyone should take this post (and its conclusions) remotely seriously without asking some very robust questions of it.
Where is this from? Who originally wrote it? Is this text’s origin really a random Facebook post, from a pseudonymous author with a cartoon profile picture and no claim of any serious credentials in the subject at hand? (Whether epidemiology or anything else)
Regardless of the merits of the text’s post (which I do not claim to be able to judge) all evidence has to be analysed for context as well as content. Simple “common sense” claims (with a couple of big words to impress non-epidemiologists like me) are made to debunk the models: where is the evidence rather than rhetoric, even some basic citations, and/or examples of or links to counter-modelling? The post doesn’t even link to the original model files from Imperial that they’re claiming to critique.
It’s perfectly _possible_ that the claims made in this Facebook post are correct, but it doesn’t mean anyone should take this post (and its conclusions) remotely seriously without asking some very robust questions of it.
I have another view really. Which is reading a paper analyzing pandemic overshoot. The authors covered a half dozen models. From very simple ones built on a simple differential equations to complex ones that factored in network effects. My take away is actual models[1] in this space tend to be robust.
The Imperial College Model is a complex model designed to answer subtle questions about the spread and containment of an epidemic. But there is nothing subtle about COVID19. The model predicts catastrophe unless you turn all the knobs to contain it to 11 and do it now.
The model is validated by real experience. Italy, New York both blew up. And elsewhere half hearted measures merely slowed the virus down, not stop it.
The true the deniers can't escape is. If what's been thrown at the pandemic is unnecessary. Then if so, why hasn't the pandemic just collapsed?
[1] As opposed to models that fit a prior defined curve to data. Those are shit.
The Imperial College Model is a complex model designed to answer subtle questions about the spread and containment of an epidemic. But there is nothing subtle about COVID19. The model predicts catastrophe unless you turn all the knobs to contain it to 11 and do it now.
The model is validated by real experience. Italy, New York both blew up. And elsewhere half hearted measures merely slowed the virus down, not stop it.
The true the deniers can't escape is. If what's been thrown at the pandemic is unnecessary. Then if so, why hasn't the pandemic just collapsed?
[1] As opposed to models that fit a prior defined curve to data. Those are shit.
> The model is validated by real experience. Italy, New York both blew up.
Do you know where the first clusters in New York where?
Because in Northern Italy, and especially around Bergamo, hospitals and then nursing care homes turned into infection centers, and with the population there so skewed with the most vulnerable (along with imperfect knowledge on the pathology) it was easy for the virus to kill them.
In fact most of the initial clusters were in hospitals, and negligence turned Alzano Lombardo in a nice spreading place.
Would a model tuned on what we know now, taking into account different infection routes and places, work the same way? I don't know, but it is a question worth asking even if in the end the model proves to be absolutely correct.
Do you know where the first clusters in New York where?
Because in Northern Italy, and especially around Bergamo, hospitals and then nursing care homes turned into infection centers, and with the population there so skewed with the most vulnerable (along with imperfect knowledge on the pathology) it was easy for the virus to kill them.
In fact most of the initial clusters were in hospitals, and negligence turned Alzano Lombardo in a nice spreading place.
Would a model tuned on what we know now, taking into account different infection routes and places, work the same way? I don't know, but it is a question worth asking even if in the end the model proves to be absolutely correct.
"the model that shut down the world economy"
The model was released on the 16th of March. Large parts of the world had already "locked down" prior to this date.
edit: of course there is no doubt this model was influential in other countries' decision to lock down.
The model was released on the 16th of March. Large parts of the world had already "locked down" prior to this date.
edit: of course there is no doubt this model was influential in other countries' decision to lock down.
Australia's influenza modelling had plans for this style of lockdown prepared long before the 'rona hit. So you are right, this model did not shut down the global economy. If anything did, it was China's demonstration that such a lockdown could actually work.
To be fair, not all of China locked down as hard as the Hubei province.
John Carmack, who worked on the Imperial College model code, has a pretty positive attitude about it:
https://twitter.com/ID_AA_Carmack/status/1254872368763277313
https://twitter.com/ID_AA_Carmack/status/1254872368763277313
I think there are some really important points here, and one of the most shocking is that Imperial put out in their summary that optimal mitigation policies could reduce deaths by half when their model just clearly doesn't model the base case.
> The social distancing policies are being compared to a zombie alternative that cannot happen and will not happen.
This is absoutely correct, my company made us work from home signficantly before the lock down because they were quite rightly worried about business continuity. In the Imperial model, that sort of thing just isn't accounted for. We literally cannot say whether the natural societal reaction would've reduced that base case by 1% or 99%.
The second point I would make is that epidemics aren't different every time. Or rather, the underlying fundamentals of how diseases spread aren't that different every time. So how come, all of these institutions that are meant to be studying the spread of diseases are putting out papers based on hacked together research code from the 1980s? Like, what were you doing in the 17 years since SARS?
Where are the dozens of open source, high quality, back tested, parameterizable models sitting on the shelf ready to go?
In many ways putting out the results of some trashy model that you haven't even properly tested and trying to shape public policy with it is actually quite a damaging thing to do. Not only is it likely to lead to bad decision making, it's also going to destroy trust in institutions that desparately need to be trusted in these situations.
> The social distancing policies are being compared to a zombie alternative that cannot happen and will not happen.
This is absoutely correct, my company made us work from home signficantly before the lock down because they were quite rightly worried about business continuity. In the Imperial model, that sort of thing just isn't accounted for. We literally cannot say whether the natural societal reaction would've reduced that base case by 1% or 99%.
The second point I would make is that epidemics aren't different every time. Or rather, the underlying fundamentals of how diseases spread aren't that different every time. So how come, all of these institutions that are meant to be studying the spread of diseases are putting out papers based on hacked together research code from the 1980s? Like, what were you doing in the 17 years since SARS?
Where are the dozens of open source, high quality, back tested, parameterizable models sitting on the shelf ready to go?
In many ways putting out the results of some trashy model that you haven't even properly tested and trying to shape public policy with it is actually quite a damaging thing to do. Not only is it likely to lead to bad decision making, it's also going to destroy trust in institutions that desparately need to be trusted in these situations.
Everything described in the review is common practice in science.
Scientific publishing is broken. Is not only about the methodologies and how some reviews are conducted, but you can't get the files needed to reproduce them, not to mention datasets that are almost impossible to obtain.
Scientific publishing is broken. Is not only about the methodologies and how some reviews are conducted, but you can't get the files needed to reproduce them, not to mention datasets that are almost impossible to obtain.
I've seen a couple of criticisms of the model, let's say, some less charitable (condescending) than others. This seems an ok criticism, though it is missing one of (IMO) important points of the other criticism (which I won't link here but some might have seen in some "skeptical" places)
I've worked briefly with simulations, I think most of the items here are valid. The RNG especially is a very easy place to shoot yourself in the foot, especially if your main RNG is used to derive random numbers with different distributions, and your numbers might look off and you don't know why.
Example, your RNG is biased or does not have all the range/precision you think it has (similar to throwing a die when you are expecting a number between 1 and 10 for example - this sounds stupid, but think for example, your results come as 0-100 but biased towards smaller numbers, it will be a pain to find)
Now, some other criticisms were pointing to a potential multithreading problem in the model, which sounds like a worrying problem.
I've worked briefly with simulations, I think most of the items here are valid. The RNG especially is a very easy place to shoot yourself in the foot, especially if your main RNG is used to derive random numbers with different distributions, and your numbers might look off and you don't know why.
Example, your RNG is biased or does not have all the range/precision you think it has (similar to throwing a die when you are expecting a number between 1 and 10 for example - this sounds stupid, but think for example, your results come as 0-100 but biased towards smaller numbers, it will be a pain to find)
Now, some other criticisms were pointing to a potential multithreading problem in the model, which sounds like a worrying problem.
This is not really a great analysis. I would like to see an analysis from someone who:
-Is actually familiar with academic code so that the usual pearl clutching can be tempered with realism. That doesn't mean throwing up one's hands and just accepting that all academic code is like this, but it does mean coming from a place of knowing what normal academic code looks like (much worse than this).
-Is actually familiar with production forecasting in other contexts (outside of the small bubble of SV) the writer says they have "done some model review of financial models". I have done a lot of such reviews and many models are implemented in idiosyncratic Excel. Yes, models that drive policy decisions in government and large transactions in finance, models that are used in internal forecasting in many places. All Excel.
A recognition of the difference between should and is. There are claims here about what should be done in modelling which are correct but with little recognition that they almost never are.
Really I would like to see a review by someone who is familiar enough with epi modelling and with modern software engineering practices to give a balanced assessment.
The only material criticism of the model that isn't just "this should have been implemented better" is that it assumes that people will not voluntarily implement forms of social distancing when many people are dying and that this distorts upwards the death count. An agent based model would model people's time varying contact matrices based on what they can observe (so as death rates go up, voluntary social distancing goes up as well).
Presumably this leads to a higher peak than mandatory measures because deaths lag measures which are visible to policy makers but not to the public, therefore people will distance later.
This is indeed not addressed in the model. It could be addressed exogenously by making interactions dependent on the rolling average death count of the last several days as that is probably the signal that will the cause the most widespread panic.
However. There is some important context here which is that:
-This was not even the only model used by SAGE. The LSHTM model was run alongside it and gave similar results.
-The "no intervention" case was sense checked using the basic equations of epidemiology.
Models that drive decisions can be criticised on technical-procedural grounds (ideally this model should use fewer globals, and use psuedorandomness better so that it gives the exact same result on every run rather than more-or-less the same result on every run) and on model concept basis (the model assumes zombies) but the most important assessment of real models is how they drive decisions.
In that regard the model was successful and drove the same decision as the very different LSHTM model, the models used by almost every other country, and what could be estimated using Epi 101.
That is the necessity of mass social distancing. Even the Swedes, incidentally, accept this. They have chosen to make everything except for the closure of the highest risk establishments voluntary. I wonder though if isn't the case that these restrictions are in practice voluntary in many cases even in the UK. I could easily visit my family and friends - the police would never know. My employer could continue having us come into work - there is no absolute ban on office workers being required to come in, just guidance that people who can work from home should. The actual legislation can be considered not only as enforceable rules but also as high-cost signalling. There will be people who simply will not take it seriously otherwise on the grounds that "if it were really that bad, they would ban it"
So to come back to the one substantive criticism - what different advice would a model that implemented behaviour changes have led to? I don't think it would have changed anything, if you tell policymakers: "listen, we don't necessarily need to make these things mandatory, once the coffins pile up people will stay home by themselves" what do you think that is going to drive? They're hardly likely to decide that we can leave off the restrictions. We came up to 80% or so of ICU capacity as it was. That tells me that anything that led to less or later social distancing would have put us in the position that Bergamo or Madrid were - at least localised and temporary overwhelming of hospitals. Potentially much worse.
-Is actually familiar with academic code so that the usual pearl clutching can be tempered with realism. That doesn't mean throwing up one's hands and just accepting that all academic code is like this, but it does mean coming from a place of knowing what normal academic code looks like (much worse than this).
-Is actually familiar with production forecasting in other contexts (outside of the small bubble of SV) the writer says they have "done some model review of financial models". I have done a lot of such reviews and many models are implemented in idiosyncratic Excel. Yes, models that drive policy decisions in government and large transactions in finance, models that are used in internal forecasting in many places. All Excel.
A recognition of the difference between should and is. There are claims here about what should be done in modelling which are correct but with little recognition that they almost never are.
Really I would like to see a review by someone who is familiar enough with epi modelling and with modern software engineering practices to give a balanced assessment.
The only material criticism of the model that isn't just "this should have been implemented better" is that it assumes that people will not voluntarily implement forms of social distancing when many people are dying and that this distorts upwards the death count. An agent based model would model people's time varying contact matrices based on what they can observe (so as death rates go up, voluntary social distancing goes up as well).
Presumably this leads to a higher peak than mandatory measures because deaths lag measures which are visible to policy makers but not to the public, therefore people will distance later.
This is indeed not addressed in the model. It could be addressed exogenously by making interactions dependent on the rolling average death count of the last several days as that is probably the signal that will the cause the most widespread panic.
However. There is some important context here which is that:
-This was not even the only model used by SAGE. The LSHTM model was run alongside it and gave similar results.
-The "no intervention" case was sense checked using the basic equations of epidemiology.
Models that drive decisions can be criticised on technical-procedural grounds (ideally this model should use fewer globals, and use psuedorandomness better so that it gives the exact same result on every run rather than more-or-less the same result on every run) and on model concept basis (the model assumes zombies) but the most important assessment of real models is how they drive decisions.
In that regard the model was successful and drove the same decision as the very different LSHTM model, the models used by almost every other country, and what could be estimated using Epi 101.
That is the necessity of mass social distancing. Even the Swedes, incidentally, accept this. They have chosen to make everything except for the closure of the highest risk establishments voluntary. I wonder though if isn't the case that these restrictions are in practice voluntary in many cases even in the UK. I could easily visit my family and friends - the police would never know. My employer could continue having us come into work - there is no absolute ban on office workers being required to come in, just guidance that people who can work from home should. The actual legislation can be considered not only as enforceable rules but also as high-cost signalling. There will be people who simply will not take it seriously otherwise on the grounds that "if it were really that bad, they would ban it"
So to come back to the one substantive criticism - what different advice would a model that implemented behaviour changes have led to? I don't think it would have changed anything, if you tell policymakers: "listen, we don't necessarily need to make these things mandatory, once the coffins pile up people will stay home by themselves" what do you think that is going to drive? They're hardly likely to decide that we can leave off the restrictions. We came up to 80% or so of ICU capacity as it was. That tells me that anything that led to less or later social distancing would have put us in the position that Bergamo or Madrid were - at least localised and temporary overwhelming of hospitals. Potentially much worse.
>-Is actually familiar with academic code so that the usual pearl clutching can be tempered with realism.
I think this is a real and systemic problem. Academia has driven itself into this horrible cultural rut where they've defined for themselves this attitude towards code that it's not important and it's fine if it's bad, hacky, undocumented, not version controlled, copied and pasted over and over. Its just research code!
It's not pearl clutching to point out that this is the coding version of a teenager saving final_report.docx, final_report_final.docx, final_report_3/23/2020.docx, final_report_v3.docx. It's basically everyone outside of academia looking at academics and wondering why they don't just grow up, and it's a cultural issue.
The problem is that what you describe as pearl clutching is just the very reasonable observation that an entire profession has chosen to just refuse to do a core part of their job.
I think this is a real and systemic problem. Academia has driven itself into this horrible cultural rut where they've defined for themselves this attitude towards code that it's not important and it's fine if it's bad, hacky, undocumented, not version controlled, copied and pasted over and over. Its just research code!
It's not pearl clutching to point out that this is the coding version of a teenager saving final_report.docx, final_report_final.docx, final_report_3/23/2020.docx, final_report_v3.docx. It's basically everyone outside of academia looking at academics and wondering why they don't just grow up, and it's a cultural issue.
The problem is that what you describe as pearl clutching is just the very reasonable observation that an entire profession has chosen to just refuse to do a core part of their job.
The problem is that it is not everyone outside of academia. The vast majority of the world's modelling code for various values of modelling is written and "maintained" by subject matter experts who are not software engineers. Finance, insurance, you name it and there's a horrible Excel model driving it.
I think the core difference is that most places outside of academia there is a closed feedback loop. You make a crap prediction, you lose a bunch of money, you get fired. There's a self-correcting mechanism for insuring the models work correctly.
In academia you run your model, you write your paper, you publish your paper, you get your promotion. Because no decisions are made based on your model, there's no accountability. Even crappy excel models in business are more reliable than academic ones because people actually have to use the results- and that compounds over decades.
But the impact of lots of academics generating rubbish and publishing it is that we're literally polluting our knowledge pool. Which is why it should be of as important for your code to be right as it is for your experimental method to be right.
In academia you run your model, you write your paper, you publish your paper, you get your promotion. Because no decisions are made based on your model, there's no accountability. Even crappy excel models in business are more reliable than academic ones because people actually have to use the results- and that compounds over decades.
But the impact of lots of academics generating rubbish and publishing it is that we're literally polluting our knowledge pool. Which is why it should be of as important for your code to be right as it is for your experimental method to be right.
Sure, models that break along their common paths will lose people money. Models that don't produce the results they should won't lead to high impact publications.
In both cases, models can break in unfortunate ways along rarely used code paths or under unusual conditions. If a model just doesn't work, then of course it will be fixed. That's in any context. If it works most of the time until it breaks badly once, then it may well not be fixed before it has caused massive damage.
In both cases, models can break in unfortunate ways along rarely used code paths or under unusual conditions. If a model just doesn't work, then of course it will be fixed. That's in any context. If it works most of the time until it breaks badly once, then it may well not be fixed before it has caused massive damage.
>That doesn't mean throwing up one's hands and just accepting that all academic code is like this, but it does mean coming from a place of knowing what normal academic code looks like (much worse than this).
This paints an incredibly bleak picture of the state of modelling. Is there any chance that things are improving?
This paints an incredibly bleak picture of the state of modelling. Is there any chance that things are improving?
Sure, younger academics in modelling heavy subjects are much more aware of software engineering issues and newer code tends to be better. That being said, academia is a world where you are competing with the smartest, most driven people in the world for a small set of available jobs. Anything that does not directly impact how likely you are to get those jobs simply will not be done consistently.
This whole thing has made me grateful for the fact that only engineers at my company will know the horrors I have perpetrated in my career.
Must be awful for the poor folks whose work is being savaged by everyone.
Must be awful for the poor folks whose work is being savaged by everyone.
Who exactly is this person and what are her credentials?
This is a completely unfair criticism. When the relevant decisions were being made two months ago, a huge number of influential people in the UK were seriously proposing to do nothing to get herd immunity. This model, which intentionally depicts what would happen if people indeed did nothing, was a big part of why people changed their minds.
You can't say the model is wrong because people would take COVID seriously anyway, when the model is half the reason people take it seriously in the first place. We're somehow forgetting history that's only months old now.