Potential false-positive rate among the 'asymptomatic infected individuals'(ncbi.nlm.nih.gov)
ncbi.nlm.nih.gov
Potential false-positive rate among the 'asymptomatic infected individuals'
https://www.ncbi.nlm.nih.gov/pubmed/32133832
73 comments
One fallacy that seems universal with healthcare folks is they think the false positive rate is the chance that a given positive result is erroneous. If an illness is rare, a positive result in a test with a 1% error rate might have an overwhelming probability of being a false positive. This is why prior probabilities need to be taken into account in making decisions.
With all due respect, this comment shows a lack of understanding about how health professionals assess the quality of diagnostics.
My wife is a doctor (and I've learned a lot from her). In med school, they're specifically taught to evaluate diagnostics on their specificity and sensitivity - which essentially covers false positives and false negatives. If you hear a doctor talk about the "accuracy" of a test, it's likely because they're simplifying the concepts.
"Error rate" or "accuracy" is not used at the scientific level in medicine. Partly, for the reason you defined. It doesn't convey enough information about the outcome of the test.
A "99% accurate test" is pretty meaningless without understanding the specificity and/or sensitivity components. In fact, I've seen some headlines where they incorrectly refer to only one component as the "accuracy".
My wife is a doctor (and I've learned a lot from her). In med school, they're specifically taught to evaluate diagnostics on their specificity and sensitivity - which essentially covers false positives and false negatives. If you hear a doctor talk about the "accuracy" of a test, it's likely because they're simplifying the concepts.
"Error rate" or "accuracy" is not used at the scientific level in medicine. Partly, for the reason you defined. It doesn't convey enough information about the outcome of the test.
A "99% accurate test" is pretty meaningless without understanding the specificity and/or sensitivity components. In fact, I've seen some headlines where they incorrectly refer to only one component as the "accuracy".
The specificity (true positive) and sensitivity (true negative) do not solve the problem I am describing.
If something is rare, it has a low base rate. That even means a test with excellent specificity and sensitivity could still be wrong most of the time.
Decisions on test accuracy simply cannot be made coherently when ignoring the base rate. To make an intuitive example, suppose that one in a thousand people have a disease. A test for the disease has 90% specificity and 100% sensitivity. It will always correctly give a positive result if the person has the disease, and has a 99% chance that a given positive test is valid. Pretty good, much better than most tests.
Now suppose that 1/1000 of people have the disease. A person with a positive result has a 1% chance of not having the disease. If everyone is tested, then 1/1000 people will get true positive results. But, (999/1000 * 0.01) ~ 1% of people will get false positives.
Thus, a given person with a positive result has nearly a 10x chance of it being erroneous compared to it being accurate! As I said, the frequentist techniques that you describe and are taught in medical schools do not help with this.
Yet this is endemic in medicine. This sort of thing is why in a recent meta-study of 54 landmark cancer trials, only six could be replicated. That is frankly terrifying.
If something is rare, it has a low base rate. That even means a test with excellent specificity and sensitivity could still be wrong most of the time.
Decisions on test accuracy simply cannot be made coherently when ignoring the base rate. To make an intuitive example, suppose that one in a thousand people have a disease. A test for the disease has 90% specificity and 100% sensitivity. It will always correctly give a positive result if the person has the disease, and has a 99% chance that a given positive test is valid. Pretty good, much better than most tests.
Now suppose that 1/1000 of people have the disease. A person with a positive result has a 1% chance of not having the disease. If everyone is tested, then 1/1000 people will get true positive results. But, (999/1000 * 0.01) ~ 1% of people will get false positives.
Thus, a given person with a positive result has nearly a 10x chance of it being erroneous compared to it being accurate! As I said, the frequentist techniques that you describe and are taught in medical schools do not help with this.
Yet this is endemic in medicine. This sort of thing is why in a recent meta-study of 54 landmark cancer trials, only six could be replicated. That is frankly terrifying.
I get bored by more esoteric statistics terms in epidemiology, but accuracy has a simple enough mathematical formula: https://www.lexjansen.com/nesug/nesug10/hl/hl07.pdf
(True positives + True Negatives) / number of all tested
Similar concept comes up in measuring accuracy of computerized image segmentation, where you ignore the true negatives
true positive / (true positive + false positive + false negative)
where it is called intersection over Union (IOU).
I can’t ever remember the names, and just rebuild whatever metric I care about in terms of true vs false and positive vs negative.
Applying all this to the real world is tough because of the over fitting problem. Even if you got the test to be 100% accurate in your tested population, it doesn’t mean it won’t be wrong on the next person it tests. Generalization is hard. So doctors have to guess based on their understanding of the tested and untested population and the sensitivity and specificity of the test. You can go meta and give the doctor a sensitivity and specificity also.
(True positives + True Negatives) / number of all tested
Similar concept comes up in measuring accuracy of computerized image segmentation, where you ignore the true negatives
true positive / (true positive + false positive + false negative)
where it is called intersection over Union (IOU).
I can’t ever remember the names, and just rebuild whatever metric I care about in terms of true vs false and positive vs negative.
Applying all this to the real world is tough because of the over fitting problem. Even if you got the test to be 100% accurate in your tested population, it doesn’t mean it won’t be wrong on the next person it tests. Generalization is hard. So doctors have to guess based on their understanding of the tested and untested population and the sensitivity and specificity of the test. You can go meta and give the doctor a sensitivity and specificity also.
>If an illness is rare, a positive result in a test with a 1% error rate might have an overwhelming probability of being a false positive.
Can you elaborate on this a little more...?
Can you elaborate on this a little more...?
If a given test has a 1% chance of returning true, even when the actual result is false, then from a sample of say 1000 tests we would expect at least 10 trues, in addition to any actual true results. If the chance of having the disease in the general population is low (say 1 in a thousand for this example) then we would expect 11 true results in our thousand samples. Of which 91% are incorrect results - false positives.
So then you'd want to know if the cause of a false positive is random, or specific to the individual, right? If it's random, then how much would a retest change your certainty that an initial positive was a true positive?
ie, if someone was a false positive the first time would they still have a 1% chance of getting another false positive, or is it possible there's something about that individual that will always give them a positive result?
ie, if someone was a false positive the first time would they still have a 1% chance of getting another false positive, or is it possible there's something about that individual that will always give them a positive result?
Would I be correct with the following:
If the false positive rate is higher than the expected rate of disease in a given community, then the majority of positive tests will be false positives.
Does this relate to COVID in any way? Since the rates among affected communities seem to be growing rapidly. Would appreciate your thoughts.
If the false positive rate is higher than the expected rate of disease in a given community, then the majority of positive tests will be false positives.
Does this relate to COVID in any way? Since the rates among affected communities seem to be growing rapidly. Would appreciate your thoughts.
Looking at growth rate with false positives is a bit of a mindbender: if you limit your testing to the potential contacts of a positive (false or not), you could get a "false R0" virtual epidemic from testing alone, if and only if you test more contacts per positive than 1/false positive rate. Unfortunately, actual hospitalizations and and deaths rule out a virtual epidemic so this is not a hope to cling to.
> Unfortunately, actual hospitalizations and and deaths rule out a virtual epidemic so this is not a hope to cling to.
Not necessarily. In theory all the deaths could have some other cause, i.e. some fraction of people with a different underlying fatal condition had false positive tests for this coronavirus and then died of the other condition.
That's probably not what's happening, but it's theoretically possible. (It's also probable that some of the reported deaths are that, but who knows what percentage.)
Not necessarily. In theory all the deaths could have some other cause, i.e. some fraction of people with a different underlying fatal condition had false positive tests for this coronavirus and then died of the other condition.
That's probably not what's happening, but it's theoretically possible. (It's also probable that some of the reported deaths are that, but who knows what percentage.)
If the false positive rate is p and the false negative rate is q, and the infection rate is r, then you will have p·(1-r) false positives (as proportion of the tested population) and (1-q)·r true positives. Your hypothesis p>r is not enough to settle which of those two numbers is bigger.
(Edited to fix a silly mistake: The phone rang while I was posting, so I ended up being hasty.)
Edit the 2nd: Even in the simplified case q=0, you can't easily tell.
(Edited to fix a silly mistake: The phone rang while I was posting, so I ended up being hasty.)
Edit the 2nd: Even in the simplified case q=0, you can't easily tell.
Does 1% error rate mean it's positive 1% of the time or wrong 1% of the time?
How does it relate to universal Healthcare? I'm not an American so maybe I'm missing something.
A "fallacy that seems universal with healthcare folks" is not expected to be related to universal healthcare.
Yes, that was rather embarrasing. Sorry about that.
This is a classic result in Bayesian theory.
Here is someone else explaining it.
https://betterexplained.com/articles/an-intuitive-and-short-...
Here is someone else explaining it.
https://betterexplained.com/articles/an-intuitive-and-short-...
That is a really excellent explanation. Thanks for sharing it.
[deleted]
In the extreme case, imagine that nobody actually has the condition. Then every positive result is a false positive. The thing we call the "false positive" rate might still be only 1% though; that just means the test is correctly identifying 99% of people as negative and 1%, falsely, as positive.
As always, there's an XKCD for this: https://xkcd.com/1132/
Basically, imagine that it's a decade from now, and no one in the world has COVID-19 anymore. The test still has a 1% false-positive rate, though - so if you test a few thousand people, a few of them will test positive. Given that setup, every single one of the test positives will be false positives.
The same holds true if there's one infected person and you test 1,000,000.
Basically, imagine that it's a decade from now, and no one in the world has COVID-19 anymore. The test still has a 1% false-positive rate, though - so if you test a few thousand people, a few of them will test positive. Given that setup, every single one of the test positives will be false positives.
The same holds true if there's one infected person and you test 1,000,000.
I don’t follow but am interested in your point. Please elaborate.
Suppose a test has a 1/100 false positive rate, and the true incidence of the disease over the population that you're testing is 1/1000 (that is, you expect a true positive every 1/1000 times).
In this case, every positive test you observe is around 10x more likely to be a false positive than a true positive.
In this case, every positive test you observe is around 10x more likely to be a false positive than a true positive.
Got it. Thanks! I have a masters in public health and still need to get schooled by hacker news! Haha
This is just a form of the base rate fallacy, correct?
https://en.wikipedia.org/wiki/Base_rate_fallacy
https://en.wikipedia.org/wiki/Base_rate_fallacy
Yes, it’s a special case of Bayes’ Theorem.
I appreciate all the answers that point out Bayes's theorem. One thing to add is that in contrast to the usual classroom Bayes the "occurrence in the general population" is very much a variable rather than a constant. For example, Germany's low death rate can be a side effect of the false positive issue in the paper.
In fact, the stark difference between Italy and Germany would provide support to the paper's conclusion.
In fact, the stark difference between Italy and Germany would provide support to the paper's conclusion.
There can be other explanation for this difference http://www.savespain.eu/italy-vs-germany because there is huge disparity between Italy and Germany infected population demographic.
My hypothesis is that this comes from bigger inter-generation connections in Italy.
My hypothesis is that this comes from bigger inter-generation connections in Italy.
Right, so it's a variable, but if we can independently nail down a false negative/positive rate, then we could try to infer the true rate of infection, but problem is (assuming US numbers):
* assume false positive rate of 10%, assume true positive rate of 100% (it's not but let's be generous)
* maybe 1/10 of the population (30 million) gets tested
* lets say 100,000 people have COVID right now (50x the official number)
number of positive tests = 30 million * 0.1 + 100k * 1 = 3,100,000
fraction of postive tests that actually have the disease = 1 / 31 = 3.2%
problem is FPR (1) depends on the population tested (i.e. p(covid | positive test) != p(covid|positive test, some symptoms) != ...), (2) we need very accurate measurements of FPR because:
lets say we constrain FPR to 10% +/- 1% ==> 10% uncertainty in FPR -- that means our inference of the number of infected people is:
n_infected = (n_positive_tests - FPR * n_tested)
which is: -200,000 to 400,000
so...not very useful.
* assume false positive rate of 10%, assume true positive rate of 100% (it's not but let's be generous)
* maybe 1/10 of the population (30 million) gets tested
* lets say 100,000 people have COVID right now (50x the official number)
number of positive tests = 30 million * 0.1 + 100k * 1 = 3,100,000
fraction of postive tests that actually have the disease = 1 / 31 = 3.2%
problem is FPR (1) depends on the population tested (i.e. p(covid | positive test) != p(covid|positive test, some symptoms) != ...), (2) we need very accurate measurements of FPR because:
lets say we constrain FPR to 10% +/- 1% ==> 10% uncertainty in FPR -- that means our inference of the number of infected people is:
n_infected = (n_positive_tests - FPR * n_tested)
which is: -200,000 to 400,000
so...not very useful.
In Denmark we increased our testing 10-fold and found 300% more people infected. Our response to that increase has been to shut down the country completely for 2 weeks and expand our governments right to act: Forced entry into private property, forced isolation and treatment, forced testing. If this is all because of an error in the test kit I'm going to be super ticked off.
Or think about it this way, this means as a percentage of real people infected, the mortality rate of COVID-19 is higher than thought. This false-positive rate means that the number of real cases is lower than reported and therefore mortality rate is higher. In other words, COVID-19 is more dangerous than thought if you catch it. Then again, I'm sure the actual cases dwarfs those reported since not everyone has been tested.
Mortality is almost always much lower than the initially reported and observed numbers.
There could be many more people that never even get to the testing phase.
There could be many more people that never even get to the testing phase.
AFAIK, that wasn't the case with SARS.
The same would be true of things like the Flu then too. Which means the true mortality rate of Flu is much lower, so even if COVID-19 is lower than current estimates for the same reason, it is still an order of magnitude higher than something like the Flu.
It's possible to get more accurate numbers for the flu because it's so prevalent, which makes it possible to do random sampling of the general population. If you did random sampling of the general population for this coronavirus right now, ~100% of the positive results would be erroneous because probably less than 0.01% of the population actually has it.
That's an excellent point! Thank you!
Yes, and that's why absolute population normalized mortality rate is the only really comparable thing across populations.
The mortality rate in Denmark is still a flat 0.0
This is somewhat misleading information.
First, the country has NOT been completely shut down. I went shopping today and bought milk, yeast and flour. We didn't need toilet paper, but the store had plenty. All schools and most of the public sector closes down for two weeks on Monday. Some business (like restaurants, movie theaters and fitness gyms) are closing down on their own accord. But you can - if you will - still go shopping for clothes, gardening stuff, electronics and most importantly food.
Second, although the right to forced entry into private property was in the original draft it was removed before vote. Entry still follows the known rules of needing approval by a judge.
You are right, however, that forced testing, forced treatment, forced vaccination (if/when possible) and forced quarantine is mandated as per discretion of the public health authorities.
First, the country has NOT been completely shut down. I went shopping today and bought milk, yeast and flour. We didn't need toilet paper, but the store had plenty. All schools and most of the public sector closes down for two weeks on Monday. Some business (like restaurants, movie theaters and fitness gyms) are closing down on their own accord. But you can - if you will - still go shopping for clothes, gardening stuff, electronics and most importantly food.
Second, although the right to forced entry into private property was in the original draft it was removed before vote. Entry still follows the known rules of needing approval by a judge.
You are right, however, that forced testing, forced treatment, forced vaccination (if/when possible) and forced quarantine is mandated as per discretion of the public health authorities.
The situation is similar in Norway. We're a bit more strict, as parts of the private sector is also forcibly shut down: Gyms, pubs, hairdressers, movie theaters are all closed.
Our infection rate has grown dramatically in the past few days, and not as a result of increased testing AFAIK. Testing capacity has been limited, but is being drastically increased as of today. So maybe the already high growth rate will increase further as a result.
Our infection rate has grown dramatically in the past few days, and not as a result of increased testing AFAIK. Testing capacity has been limited, but is being drastically increased as of today. So maybe the already high growth rate will increase further as a result.
You are correct on all points, thanks for letting me know about the changed proposal.
Would this indicate that the mortality rate is actually much higher percentage-wise, since the denominator is actually artificially inflated?
I think so but wouldn't it also make the transmission rate estimates lower?
That was my thinking as well. I also wonder how many people who ARE symptomatic don't get tested and how much this balances out the false positives of asymptomatic people, if at all.
One big question is what percentage of positive tests are asymptomatic? If only a tiny percent are asymptomatic, then this false positive issue would not be elevating the total numbers much, right? I won’t claim to have the best resource here but one article stated:
“ Dr. Tedros noted that only 1 percent of cases in China are reported as “asymptomatic.” And of that 1 percent, 75 percent do go on to develop symptoms.”
https://arstechnica.com/science/2020/03/dont-panic-the-compr...
“ Dr. Tedros noted that only 1 percent of cases in China are reported as “asymptomatic.” And of that 1 percent, 75 percent do go on to develop symptoms.”
https://arstechnica.com/science/2020/03/dont-panic-the-compr...
How did they estimate this? If anybody can read the actual paper, I’d love to know.
If false positives dominate true positives then you’d expect total positives to depend primarily on number of tests given, right? Which sounds wrong to me, but I’d be interested in hearing other thoughts.
If false positives dominate true positives then you’d expect total positives to depend primarily on number of tests given, right? Which sounds wrong to me, but I’d be interested in hearing other thoughts.
Can someone clarify which type of test they analyzed? 80% seems way too high. I would expect something closer to 10% at most (which would still mean the probability of a true positive might be very low per Bayes' theorem)
Are you confused or am I misreading your comment? The result is that 80% of positives are false positives, not that 80% of all tests are false positives. (IMO it is still fishy.)
I wasn't implying that 80% of all tests were false positives. I'm talking about P(You don't have coronavirus | You test positive). 80% is far beyond what would be normal for these types of medical diagnostic tests. I would expect something like 1% or less for a really good test, and around 10% for a bad one, but I don't know if the test the Chinese were using is really bad in some way, which is why I'm asking for clarification there.
[deleted]
80%...in asymptomatic cases. So if most of the people who get tested DID show symptoms, the false positive rate generally could be far worse.
But, it would suggest downsides to more general testing.
But, it would suggest downsides to more general testing.
But if your diagnostic criteria is showing symptoms later, then you are ejecting the entire population of carriers who might be, say, teeming with the virus but showing zero symptoms, for perhaps a genetic or "dumb luck" reason.
Interesting, as this means that China would be quarantining more people than "necessary", which would help slow the pandemic anyway. I can't imagine an asymptomatic person would put stress on the hospital system? But maybe I'm wrong there.
I am curious if this also could indicate a false-positive problem with non asymptomatic people as well.
False-positives are also why the CDC tests had to be shipped back, although that was because it was showing false positives in other diseases it was testing for, not COVID-19.
I am curious if this also could indicate a false-positive problem with non asymptomatic people as well.
False-positives are also why the CDC tests had to be shipped back, although that was because it was showing false positives in other diseases it was testing for, not COVID-19.
> I can't imagine an asymptomatic person would put stress on the hospital system?
If s/he is in the hospital in quarantine you must give blankets and food, probably a nurse to check the temperature and symptoms two or three times per day, a medical doctor one a day just to be sure. Perhaps a blood analysis from time to time?
Luckily you don't have to handle visitors because they are in quarantine. (Or there are some visits? What if one patient tries to escape?) You must give an official reports for the family. Now you can assume the patient can send a WhatsApp message to the family saying s/he is fine, but you need probably still an official report. Paperwork, there is also paperwork.
How isolated are them from each other. If they are all together, you can transform the overcrowded false positives in real patients.
If s/he is in the hospital in quarantine you must give blankets and food, probably a nurse to check the temperature and symptoms two or three times per day, a medical doctor one a day just to be sure. Perhaps a blood analysis from time to time?
Luckily you don't have to handle visitors because they are in quarantine. (Or there are some visits? What if one patient tries to escape?) You must give an official reports for the family. Now you can assume the patient can send a WhatsApp message to the family saying s/he is fine, but you need probably still an official report. Paperwork, there is also paperwork.
How isolated are them from each other. If they are all together, you can transform the overcrowded false positives in real patients.
why would an asymptomatic person be in an hospital? Even people with minor symptoms are asked to self quarantine at home pretty much everywhere.
In China, asymptomatic with a positive test result still put you in their make-shift hospital facilities.
I'm starting to hear stories that the Chinese people demanded more quarantine than was needed. The problem got bad enough that the government decided to comply as it wouldn't hurt much more than the amount of quarantine was needed and might even help.
I really wonder how we're going to look back at this in a few years.
On the one hand, no matter what measures politics takes here (the Netherlands), every political party is clamouring for more. That suggests to me that we'll probably end up doing too much.
On the other hand, the scenes from Hubei and Italy are horrific, we're going to get them here as well, it could be far worse, we absolutely must act now.
On the third and fourth hands, the total number of deaths so far wouldn't even make a serious flu season in a single country, and fatality rates are all over the place depending on the country.
Very confusing. Still, better too much than too little.
On the one hand, no matter what measures politics takes here (the Netherlands), every political party is clamouring for more. That suggests to me that we'll probably end up doing too much.
On the other hand, the scenes from Hubei and Italy are horrific, we're going to get them here as well, it could be far worse, we absolutely must act now.
On the third and fourth hands, the total number of deaths so far wouldn't even make a serious flu season in a single country, and fatality rates are all over the place depending on the country.
Very confusing. Still, better too much than too little.
Can you look up demographic of the infected people in Netherlands?
It can be that you have somehow managed to avoid infecting older generation. I have longer discussion about this here http://www.savespain.eu/italy-vs-germany
It can be that you have somehow managed to avoid infecting older generation. I have longer discussion about this here http://www.savespain.eu/italy-vs-germany
The thing is, if all these measures stop the spread, even if they were all justified, the outcomes are pretty much indistinguishable from an over reaction: both accurate response and over reaction have similar outcomes.
Wait, so the actual mortality rate for COVID-19 is much higher than thought because of all the false positives on cases without symptoms?
But that also means that it doesn't spread as quickly/easily as originally thought. Right?
it all depends what "higher than thought" means.
If you mean the day-to-day numbers, those will always show variations, and paradoxically those will even show a substantial uptick once the spread slows down.
The most vulnerable are affected first and most severely.
If you mean the day-to-day numbers, those will always show variations, and paradoxically those will even show a substantial uptick once the spread slows down.
The most vulnerable are affected first and most severely.
Anyone save the full text? Every time I try to get to it I get a 404.
I believe the full article is in Chinese.
It's not in anything now. It's 404'd which is concerning.
If tests indeed have such a high false-positive rate, then all estimates of fatality rate calculated by dividing over the number of individuals identified as "infected" are too low, i.e., by implication the virus is actually deadlier than naively estimated.
EDIT: All else remaining the same. See AnthonyMouse's comment below for important clarifications and corrections.
EDIT: All else remaining the same. See AnthonyMouse's comment below for important clarifications and corrections.
That's assuming a large fraction of the people who have been tested are asymptomatic, otherwise a high false positive rate among asymptomatic people would have minimal effect on the numbers because they aren't being tested to begin with.
Meanwhile you also have the opposite happening for the same reason -- if even a small percentage of asymptomatic people are actually infected but not being tested, a small percentage of "asymptomatic people" (i.e. nearly the entire population) could represent a very large proportion of those infected and cause the fatality rate estimates to be much higher than the true number.
Meanwhile you also have the opposite happening for the same reason -- if even a small percentage of asymptomatic people are actually infected but not being tested, a small percentage of "asymptomatic people" (i.e. nearly the entire population) could represent a very large proportion of those infected and cause the fatality rate estimates to be much higher than the true number.
True. I updated my comment.