Medicine’s Uncomfortable Relationship with Math (2014)(archinte.jamanetwork.com)
archinte.jamanetwork.com
Medicine’s Uncomfortable Relationship with Math (2014)
http://archinte.jamanetwork.com/article.aspx?articleid=1861033
46 comments
The problem is more deep-rooted in most biology and medicine education. In many fields of biology and medicine people from faculty to undergraduate lack a systematic education in statistics and probability, and this problem keeping reflected in the research and publications. I collaborated with many people in these fields, and surprisingly the statistical analysis is just clicking a few buttons in a software and copy-pasting the alpha value. Another reason why this problem is so rampant in the fields, I think it is because the graduate and postgraduate training in these fields still like apprenticeship. So you learn mostly from what senior members do in the lab, what the classical papers in the field did, and what your paper's reviwers ask.
They are required courses for those degrees, so you might have a bad sample of people, or the field is simply bad at it entirely.
Since this is something about statistics and probability, I think enough samples for conclusion is required. ;)
You can read an example (http://science.sciencemag.org/content/351/6277/1037.2.long, http://www.be-md.ncbi.nlm.nih.gov/pubmed/26941311/#comments) I linked in another comment (https://news.ycombinator.com/item?id=11325254). There are quite a few blogpost on this also, and if you allow me to shorten the story in one sentence, that is a comment on Science arguing a previous study on Science did the statistical analysis wrong is under the critics that the comment itself did statistical analysis wrong.
And recently life sciences fields are in a "reproducibility" controversy which basically arises from education and application of the topic.
You can read an example (http://science.sciencemag.org/content/351/6277/1037.2.long, http://www.be-md.ncbi.nlm.nih.gov/pubmed/26941311/#comments) I linked in another comment (https://news.ycombinator.com/item?id=11325254). There are quite a few blogpost on this also, and if you allow me to shorten the story in one sentence, that is a comment on Science arguing a previous study on Science did the statistical analysis wrong is under the critics that the comment itself did statistical analysis wrong.
And recently life sciences fields are in a "reproducibility" controversy which basically arises from education and application of the topic.
I'm a physician, with an undergrad in physics. My research is very applied and very math heavy. When applying to med schools, I asked a couple very senior mathematicians at my college as to whether I should take a stats class. They assured me I would not need a stats class.
The first time I got a biostats lecture in medical school, I genuinely questioned their advice. By the third class, I was convinced I knew nothing about math. Luckily our biochem prof was a physics major and assured me I didn't need to worry about the stats.
Somewhere in residency, I realized I really had been getting all the stats questions right. But not because I did anything like the other residents. They have all these crazy stories and mnemonics to keep track of what kind of problem they're looking at. They seriously try to classify their way out of basic math. They classify. Everything.
I now agree with my profs. I didn't need a stats class. But the stats education in medical school seems designed to convince physicians that they need statisticians. Which is probably a coup for biostatisticians, but it's a damn shame for physicians trying to break into research outside major academic centers.
The first time I got a biostats lecture in medical school, I genuinely questioned their advice. By the third class, I was convinced I knew nothing about math. Luckily our biochem prof was a physics major and assured me I didn't need to worry about the stats.
Somewhere in residency, I realized I really had been getting all the stats questions right. But not because I did anything like the other residents. They have all these crazy stories and mnemonics to keep track of what kind of problem they're looking at. They seriously try to classify their way out of basic math. They classify. Everything.
I now agree with my profs. I didn't need a stats class. But the stats education in medical school seems designed to convince physicians that they need statisticians. Which is probably a coup for biostatisticians, but it's a damn shame for physicians trying to break into research outside major academic centers.
Irrelevant to your main point - I am surprised you didn't have at least an intro to probability and statistics, and later a statistical mechanics course as part of a physics undergrad curriculum.
Relevant to your main point - I have worked with many clinicians. Most of them have had a fairly tenuous grasp of the statistics they were working with, which matches your experience in training I think. A couple had much better understanding, but had driven that education themselves.
Relevant to your main point - I have worked with many clinicians. Most of them have had a fairly tenuous grasp of the statistics they were working with, which matches your experience in training I think. A couple had much better understanding, but had driven that education themselves.
Yes, we took a statistical mechanics course senior year, deriving Gibbs free energy from buckets of 1s and 0s, etc. Brutal class.
Ah, good. Had me worried there for a moment !
It is true that depending on how statistical mechanics is taught, it can be difficult to connect to other statistical reasoning (the distributions are "nice", you have a gazillion measurements, and your population sampling doesn't tend to have bias issues, etc.)
It is true that depending on how statistical mechanics is taught, it can be difficult to connect to other statistical reasoning (the distributions are "nice", you have a gazillion measurements, and your population sampling doesn't tend to have bias issues, etc.)
Nope, PhD in bioinformatics here and never took a single (formal) statistics course. There was Statistics I required for my undergrad, which I AP'ed out of by reading the book for a few hours. Nothing whatsoever in grad school. It's horrible and badly needs to be change.
I've had to do a lot of extracurricular statistics to get up to speed but a disturbingly large portion of people don't. People generally know t-tests and maybe ANOVA, and that's about it. Of course they don't know any of the theory behind it and often just apply t-tests to everything, even when inappropriate.
I've had to do a lot of extracurricular statistics to get up to speed but a disturbingly large portion of people don't. People generally know t-tests and maybe ANOVA, and that's about it. Of course they don't know any of the theory behind it and often just apply t-tests to everything, even when inappropriate.
> They are required courses for those degrees
How can you possibly say that? Are you familiar with the course prerequisites for every biology and medicine degree in the world off the top of your head?
How can you possibly say that? Are you familiar with the course prerequisites for every biology and medicine degree in the world off the top of your head?
Medical school hopefuls are a major source, probably the major source, of biology undergrads, and the pre-reqs for medical school, with few exceptions, are standardized under the AAMC. It's 2 semesters of calc, no stats required.
And quite frankly, the PPV trope is just that, a trope. The biostatisticians found a little trick to lord over their masters. If you only have 15 seconds to solve after reading the question, and you know you only do this once every 6 months, and you know most people get it wrong, answer C, next question.
Christ, the story that docs suck at stats is part of medical education. They don't teach biostatistics, they teach "you suck at biostats".
And quite frankly, the PPV trope is just that, a trope. The biostatisticians found a little trick to lord over their masters. If you only have 15 seconds to solve after reading the question, and you know you only do this once every 6 months, and you know most people get it wrong, answer C, next question.
Christ, the story that docs suck at stats is part of medical education. They don't teach biostatistics, they teach "you suck at biostats".
Are you aware that some people live outside the United States?
As with a lot of "simple" math, the trick is to actually write it down and calculate it, because our intuition (at least for some of us) is often not the best when it comes to this kind of calculation.
In this case we write down the contingency table. Assuming that the test perfectly detects what we are looking for we find
True positives: 1
False positive: 5% of 1000 = 50
True negative: 949
False negative: 0
Chance of disease given positive results = 1/51 = 1.96%
In this case we write down the contingency table. Assuming that the test perfectly detects what we are looking for we find
True positives: 1
False positive: 5% of 1000 = 50
True negative: 949
False negative: 0
Chance of disease given positive results = 1/51 = 1.96%
Interestingly, there is a discussion in comments of this paper claiming the authors got it wrong.
But as you said, under the assumption that the test perfectly detects, the results are correct.
But as you said, under the assumption that the test perfectly detects, the results are correct.
This reminds me a recent interesting publication. A comment on Science (http://science.sciencemag.org/content/351/6277/1037.2.long) arguing a previous study on Science did the statistical analysis wrong is under critics that the comment itself did statistical analysis wrong.
It really depends on how you define "False Positive Rate". There are at least 2 different ways of looking at this:
- Out of 1000 tests 51 will be positive, 50 of which are incorrect.
- Out of 1000 positive tests, 50 will be wrong, 950 will be correct.
The 2 interpretations give vastly different results.
- Out of 1000 tests 51 will be positive, 50 of which are incorrect.
- Out of 1000 positive tests, 50 will be wrong, 950 will be correct.
The 2 interpretations give vastly different results.
"False positive" has a precise meaning in statistical analysis. The second thing you're talking about is useful to know, but calling it the false positive rate is just wrong.
"Out of 1000 positive tests, 50 were false positives, 950 were true positives" - valid statement.
"The false positive rate was 50 out of 1000" - abuses a common technical term in a way that sounds valid on the face of it, but which is potentially VERY misleading.
We can't even calculate a valid false positive rate from the above data, since that requires taking the ratio vs. all tests and not just positive tests.
"Out of 1000 positive tests, 50 were false positives, 950 were true positives" - valid statement.
"The false positive rate was 50 out of 1000" - abuses a common technical term in a way that sounds valid on the face of it, but which is potentially VERY misleading.
We can't even calculate a valid false positive rate from the above data, since that requires taking the ratio vs. all tests and not just positive tests.
Here is a definition of False Positive from wikipedia: "In medical testing, and more generally in binary classification, a false positive is an error in data reporting in which a test result improperly indicates presence of a condition, such as a disease (the result is positive), when in reality it is not"
I don't see anything here that definitively clarifies which of the 2 scenarios above it can exclusively be applied to.
I don't see anything here that definitively clarifies which of the 2 scenarios above it can exclusively be applied to.
There are many ways in which you can incorrectly interpret statements on Wikipedia, which is why specialized textbooks and so forth still have use.
That's what a "false positive" is but Wikipedia also has a separate article on "false positive rate", which gives the formula
FP / (FP + TN)
Where FP is number of false positives, and TN is number of true negatives. So it's a third option:
- Out of 1000 actually negative samples, 50 were tested as positive.
So in the case of 1000 samples, 949 correctly testing as negative, 50 incorrectly testing as positive, and 1 correctly testing as positive, the false positive rate is 50 / 999.
FP / (FP + TN)
Where FP is number of false positives, and TN is number of true negatives. So it's a third option:
- Out of 1000 actually negative samples, 50 were tested as positive.
So in the case of 1000 samples, 949 correctly testing as negative, 50 incorrectly testing as positive, and 1 correctly testing as positive, the false positive rate is 50 / 999.
Right; this is the definition of the numerator (the number of false positives). The false positive RATE also has a denominator, which is defined as the total number of tests performed (the second case in the parent poster's question).
Dividing by the number of positive tests instead gives what's called the 'false discovery rate' which is pretty rarely used.
Dividing by the number of positive tests instead gives what's called the 'false discovery rate' which is pretty rarely used.
The traditional way (that I'm aware of) of defining the false positive rate is derived from the conditional probability of a positive prediction given that the true underlying state is negative:
False Pos. Rate = P(predict + | state -)
= P(predict + and state -) / P(state -)
= P(predict + and state -) / (P(predict + and state-) + P(predict - and state -))
~ #FP/samplesize / (#FP/samplesize + #TN/samplesize)
= #FP / (#FP + #TN)
The first way is wrong. To show you clearly way: If you apply 1000 tests to a sample in which everyone is ill, there is 0 percent of false positives because a false positive requires the person to be sane.
So to detect false positives in a mathematical way, you should apply the test only to sane people and now the proportion positive/total is an estimation of the false positive rate.
So to detect false positives in a mathematical way, you should apply the test only to sane people and now the proportion positive/total is an estimation of the false positive rate.
The latter ratio is called precision. There is a handy table at https://en.wikipedia.org/wiki/Precision_and_recall#Probabili...
[deleted]
Are there actually two accepted ways of defining type I error rates? (Genuinely curious, I am not a statistician)
Exhibit B: “A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves.” http://care.diabetesjournals.org/content/17/2/152.abstract
Cited by 272(!!): https://scholar.google.com/scholar?cites=1812909520721081729...
Cited by 272(!!): https://scholar.google.com/scholar?cites=1812909520721081729...
Context: In this 1994 paper, the researcher "invented" the trapezoidal rule for computing integrals[+]. This is a very common method for computing approximate integrals, and can probably be found in your local high school calculus class.
[+] https://en.wikipedia.org/wiki/Trapezoidal_rule
[+] https://en.wikipedia.org/wiki/Trapezoidal_rule
For what it's worth, these exact same questions do show up on board exams, so it's not like medical students don't ever learn this information.
What is a bit sad is that data analysis and quantitative skills continue to be de-emphasized in the pre-clinical curriculum for the sake of rote memorization of huge lists and tables, which would seem to be less and less important as instant access to references on smartphones becomes commonplace.
What is a bit sad is that data analysis and quantitative skills continue to be de-emphasized in the pre-clinical curriculum for the sake of rote memorization of huge lists and tables, which would seem to be less and less important as instant access to references on smartphones becomes commonplace.
I agree, people rarely enter medicine because they are particularly good at maths. All that memorization just happens to draw a different kind of personality. But it's hard to blame medicine for that: in the practical act of applying medical knowledge to heal people, the memorization is much more important than mathematical understanding.
Where it breaks down is medical science, which relies so much more on subtle mathematical interpretation than most other empirical sciences. But that is just what happens when there is so little opportunity for deliberate experimentation and such a large dataset of "naturally occurring" experiments to observe. (and it surely is helped by this millenia old tradition of nonempirical medicine, i guess all that bloodletting based on beautiful but hilariously wrong theories is to this day giving the theory-building part of the scientific method a bad reputation in medicine, making it work completely in "stupid mode empirism" where every part of thinking that is not strictly looking at numbers is heavily frowned upon)
Maybe there is a need for a stronger differentiation between academic paths leading to to applied medicine and those leading medical science. Biotech comes close, but it certainly isn't about teasing more reliable insights from clinical datasets. Other sciences take the easy road and simply train everybody for science, leaving practical application to learning on the job, but we sure would not want to get treated by that kind of MD.
Where it breaks down is medical science, which relies so much more on subtle mathematical interpretation than most other empirical sciences. But that is just what happens when there is so little opportunity for deliberate experimentation and such a large dataset of "naturally occurring" experiments to observe. (and it surely is helped by this millenia old tradition of nonempirical medicine, i guess all that bloodletting based on beautiful but hilariously wrong theories is to this day giving the theory-building part of the scientific method a bad reputation in medicine, making it work completely in "stupid mode empirism" where every part of thinking that is not strictly looking at numbers is heavily frowned upon)
Maybe there is a need for a stronger differentiation between academic paths leading to to applied medicine and those leading medical science. Biotech comes close, but it certainly isn't about teasing more reliable insights from clinical datasets. Other sciences take the easy road and simply train everybody for science, leaving practical application to learning on the job, but we sure would not want to get treated by that kind of MD.
I'm studying for these boards right now, and these equations do show up. However, it's not the curriculum (at least not at my school) pushing the rote memorization of drugs, side effects, contraindications, microbes, etc. It's the board exams. A huge component of the exams is raw memorization of random facts that have no relation to anything (did you know rifampin makes your tears turn orange?). Unfortunately, there seems to be no incentive for the board examiners to change their exam.
The issue of positive predictive value versus specificity comes up almost every day in my work as a pathologist. There is widespread misunderstanding, and in my own anecdotal experience, it very frequently results in unnecessary lab testing and misinterpretation of test results by clinicians.
When I was a pre-med student, for some reason the prerequisites required a year of calculus. That succeeded in weeding out people who can't make an A in freshman calculus, but I'm not sure what else it accomplished. Calculus has little to do with the daily practice of medicine, unless you're a radiation oncologist or doing some hardcore research.
A year of statistics would have served me and my patients much better. That goes double, given the current firehose of data that is part and parcel of the personalized medicine revolution.
When I was a pre-med student, for some reason the prerequisites required a year of calculus. That succeeded in weeding out people who can't make an A in freshman calculus, but I'm not sure what else it accomplished. Calculus has little to do with the daily practice of medicine, unless you're a radiation oncologist or doing some hardcore research.
A year of statistics would have served me and my patients much better. That goes double, given the current firehose of data that is part and parcel of the personalized medicine revolution.
heh, pathologist here as well. See my other comment in this thread (1). I did physics, so I don't know what the general pre-med curriculum is like, but I'm not so sure a stats class would necessarily help more than two semesters of calculus when you're up against big data. Understanding integration and continuity are more valuable in the big picture.
My experience has been that big data is more about linear algebra, which is usually several classes beyond entry level calc or stats. You have to be able to reason about arbitrarily large collections of partial differential equations (albeit reduced to difference equations).
For example, if you want to talk about genomics, Durbin's Biologic Sequence Analysis is probably the most foundational text available. It introduces Bayes's theorem on page 8, has stuff that looks suspiciously like calculus on page 40, and is into Markov chains by page 48. They hold off on a formal treatment of entropy until about half-way through the book.
And for phylogenetics, the equivalent books is Felsenstein's Inferring Phylogenies. He introduces linear algebra before integration.
My favorite quote from Felsenstein, particularly germaine for pathologists (surely the taxonomists of medicine):
"Knowing exactly how many tree topologies ... is not particularly important. The point is that there are very large numbers of them. ... one use for the numbers was "to frighten taxonomists."
1. https://news.ycombinator.com/item?id=11326178
My experience has been that big data is more about linear algebra, which is usually several classes beyond entry level calc or stats. You have to be able to reason about arbitrarily large collections of partial differential equations (albeit reduced to difference equations).
For example, if you want to talk about genomics, Durbin's Biologic Sequence Analysis is probably the most foundational text available. It introduces Bayes's theorem on page 8, has stuff that looks suspiciously like calculus on page 40, and is into Markov chains by page 48. They hold off on a formal treatment of entropy until about half-way through the book.
And for phylogenetics, the equivalent books is Felsenstein's Inferring Phylogenies. He introduces linear algebra before integration.
My favorite quote from Felsenstein, particularly germaine for pathologists (surely the taxonomists of medicine):
"Knowing exactly how many tree topologies ... is not particularly important. The point is that there are very large numbers of them. ... one use for the numbers was "to frighten taxonomists."
1. https://news.ycombinator.com/item?id=11326178
Reminds me of Gerd Gigerenzer's quizzes of over 1,000 gynaecologists on interpreting mammogram results - only 21% got it right. [1]
[1] http://www.bbc.com/news/magazine-28166019
[1] http://www.bbc.com/news/magazine-28166019
Did anyone else learn how to compute probabilities by drilling through the algebra? Like, if asked how to convert a plain English probabilistic query (e.g. "what is the chance of picking two red candies from a mixed bowl") into a formula, I would focus on the 'and', mentally recite something to the effect of 'conjunctions multiply probabilities', then write p(red)*p(red). There's no intuition or understanding, just rote memorization.
I've been trying to recognize and account for this deficiency by drawing mental decision trees enumerating probabilities instead ... anyone else doing anything similar?
I've been trying to recognize and account for this deficiency by drawing mental decision trees enumerating probabilities instead ... anyone else doing anything similar?
Not to be an asshole but, unless you replace the first candy after taking it out, your formula's wrong.
If you start w/ 50 red candies and 50 non-red candies, the chance of the first one red is 1/2 (from 50/100) but the chance the second one is red after first drawing a red candy is now 49/99. After removing one red candy, there are now 49 red candies and 99 total candies.
So, P(red) * P(red | removed 1 red) = 49/198
If you start w/ 50 red candies and 50 non-red candies, the chance of the first one red is 1/2 (from 50/100) but the chance the second one is red after first drawing a red candy is now 49/99. After removing one red candy, there are now 49 red candies and 99 total candies.
So, P(red) * P(red | removed 1 red) = 49/198
XD okay, fair enough. I was thinking of the ideal candy bowl, which is self-refilling. Do you grade undergraduate stats quizzes, by the way?
Drake's Fundamentals of Applied Probability Theory does emphasize the tree enumeration of possibilities in it's presentation of probability. It's an excellent introductory text. I remember remarking to a fellow student that the book was so good you didn't need to go to lecture. His reply was that Professor Drake's lectures were so good you didn't need to read the book.
However, in probability thing can quickly get complicated and, for me at least, intuition can't always be trusted. (I probably needed to go to the lectures and read the book.) For examples see either Schwarz's 40 Puzzles and Problems in Probability and Mathematical Statistics or Grimmett and Stirzaker's larger One Thousand Exercises in Probability.
However, in probability thing can quickly get complicated and, for me at least, intuition can't always be trusted. (I probably needed to go to the lectures and read the book.) For examples see either Schwarz's 40 Puzzles and Problems in Probability and Mathematical Statistics or Grimmett and Stirzaker's larger One Thousand Exercises in Probability.
Aah, that's excellent, thank you very much! I'm always on the lookout for interesting course ware and that's quite the strong recommendation.
And those look like excellent exercises as well; thank you doubly :)
And those look like excellent exercises as well; thank you doubly :)
I just wanted to point out that I think it is crucial that the authors included the statement "Assuming a perfectly sensitive test,..." Otherwise, if using only the information presented in the question, there would be no way to calculate the true positive rate, or P(test+ | disease), in Bayes' Theorem.
True, but any false negative rate below 100% would cause the answer to be lower than 2%. The problem they were focusing on was the subset of people who overestimated.
[deleted]
[deleted]