A popular algorithm to predict sepsis misses most cases and sends false alarms(statnews.com)
statnews.com
A popular algorithm to predict sepsis misses most cases and sends false alarms
https://www.statnews.com/2021/06/21/epic-sepsis-prediction-tool/
39 comments
I have this conversation with my attendings regularly regarding scaling back treatment. Everyone has their own comfort level with how aggressive they are doing that, and it would be interesting to see better heuristics that would allow for more appropriate de-escalation sooner (or longer, I guess)
Hot, breathing heard, with a fast heartbeat. For the laypeople out there. Saved you three lookups. Just call me the human memoizer.
I'm gonna call you the "being decent at communicating"izer, for using commonly understandable terms and not ego stroking about how much medical vocab you know
It seems like this should be a sepsis-risk calculation then, based on extrinsic factors (age, admittance reason, etc.); rather than intrinsic factors (hr, bp..).
The problem here then, i'd guess, is how many of those extrinsic factors are unmeasurable in practice; and how little data we'd have on them.
The problem here then, i'd guess, is how many of those extrinsic factors are unmeasurable in practice; and how little data we'd have on them.
Sure, and there's a lot of work in this area to predict sepsis in specific subgroups, but as with a lot of AI I think we're a long, long way off a 'general purpose' sepsis detector. For various reasons sepsis just presents differently in different patient groups so one person's risk factor is another's protective factor
Basically everyone admitted to the hospital is at some level or risk from sepsis which is why the proxy markers become important.
Sepsis is pretty well defined between SIRS and qSOFA. But the definition has a subjective component which has very poor inter-rater reliability. And you’re 100% right that the definition overlaps with other very common conditions, such as pain or having just had an operation.
check out these guys: https://www.luminaremed.com ; started because of the false alarms
Paywalled
Here's an alternative article on the same topic: https://labblog.uofmhealth.org/lab-report/popular-sepsis-pre...
Podcast on this was good: https://edhub.ama-assn.org/jn-learning/audio-player/18620183
The problem of looking at supposedly representative (but not actually) proxies for actual information is widespread in society, whether it be in hiring or medicine.
Here they just replicated all the same issues the current model of diagnosis faced.
Here they just replicated all the same issues the current model of diagnosis faced.
My wife recently had a burst apendice with perithonitis and a lot of puss in her abdomen and she had no fever.
After the surgery my parents(both doctors) told me to watch for fever as a sign of reinfection. I had to remind them there was no fever in the first place and that was likely a bad proxy for infection in her case.
After the surgery my parents(both doctors) told me to watch for fever as a sign of reinfection. I had to remind them there was no fever in the first place and that was likely a bad proxy for infection in her case.
As CRP testing is easy and cheap, why do we still cling to heuristics such as measuring fever instead of measuring the infection?
CRP has a three day delay - it gives you an idea of what was going on. Fevers are induced my macrophages encountering the antigen and are quite immediate.
I was replying to a case where there was no fever but CRP was high when measured.
I have been in a similar situation where after several days of illness my condition didn't appear bad enough and the doctor sent me home. This was some 15 years ago and they didn't have quick tests back then, but luckily they did prescribe a lab CRP test and called me back in for treatment immediately after they got the lab result later that day.
I have been in a similar situation where after several days of illness my condition didn't appear bad enough and the doctor sent me home. This was some 15 years ago and they didn't have quick tests back then, but luckily they did prescribe a lab CRP test and called me back in for treatment immediately after they got the lab result later that day.
Yes, all these tests have different characteristics. I think I was making the point that there isn't an "infection" test. Temperature's are useful because they are an immediate response by your body and they can be checked instantly in remote locations. CRP is a fairly standard test, but it is raised in infection/inflammation/surgery etc, and the delay can be unhelpful. Other tests include blood cultures (low sensitivity), white cell count (again quite none specific) and then more location-specific tests, like xrays, CTs, PET-CTs, urinalysis.
A lot depends on your index of suspicion, how well the person is, what the risks are of getting it wrong (in both directions), what safety nets you have available to you (i.e. are they home with someone sensible who can keep an eye on them).
In summary - there is no perfect test or perfect heuristic to apply to everyone.
A lot depends on your index of suspicion, how well the person is, what the risks are of getting it wrong (in both directions), what safety nets you have available to you (i.e. are they home with someone sensible who can keep an eye on them).
In summary - there is no perfect test or perfect heuristic to apply to everyone.
Would it maybe mean that CRP means nothing in isolation? Like a majority of what we test for?
As it's easy and cheap, you have no reason to use a measurement in isolation. For instance, you can track it every day to see if the infection is growing or shrinking. Here's a recent example regarding COVID-19: "We realized that whereas a single CRP lab value from hospital admission wasn't very practical as a predictor of who might get sicker, tracking the rate of change from Day 1 to Day 2 or 3 was a very powerful and very clinically predictive test" https://www.sciencedaily.com/releases/2020/11/201105112947.h...
Wow, incredible!
My wife did receive a CRP test that confirmed her infection in the hospital(it was over 200) -- that was among the first things the emergency department checked, then an ultrasound revealed the burst apendice, but I don't think there's any easy way to check it at home.
She already had a lot of burst veins from the hospital, so we decided to just do what her surgeon said, basically take amoxicillin for another 7 days and just about it.
She already had a lot of burst veins from the hospital, so we decided to just do what her surgeon said, basically take amoxicillin for another 7 days and just about it.
CRP can’t really confirm an infection. It’s a marker of inflammation. Inflammation often correlates with infection but not always. Can be lots of other things too, including a lot of the things that are false positives on these sepsis algos.
Ultimately doctors have to take the whole clinical picture into account instead of relying on a couple markers or vital signs.
You’re right that a big problem is the inability of our system to do testing at home, which might greatly expand the scope of the clinical picture. But it still wouldn’t be a cure-all
Ultimately doctors have to take the whole clinical picture into account instead of relying on a couple markers or vital signs.
You’re right that a big problem is the inability of our system to do testing at home, which might greatly expand the scope of the clinical picture. But it still wouldn’t be a cure-all
You have to compare to the current situation where we are overreliant on fever as a marker. If after an operation, your CRP levels shoot high unexpectedly, wouldn't you want to know and start investigating what's going on?
CRP normally does shoot up post operatively, and white cell count behaves similarly. Generally if there is a post op problem people know- maybe not a fever but people feel generally unwell. That is a good reason to investigate further. Sometimes for higher risk operations we do routine blood tests as an inpatient, but if people are will enough to go home, it’s reasonable to rely on people not feeling right.
I think what you are assuming is that early intervention in infection is good. Very likely to be true. And that we can reliably detect early infection. Much less likely to be true.
The biggest issue is the costs of false positives, which with any early intervention protocol are likely to be high and, questionably, on balance a net negative.
The biggest issue is the costs of false positives, which with any early intervention protocol are likely to be high and, questionably, on balance a net negative.
You should instead be doing a WBC count with a breakdown of neutrophils and other leukocytes, not a CRP which is highly non-specific
You should be doing both. WCC and neutrophils can also be misleading, and tend to fluctuate. Procalcitonin can be helpful in some situations but isn’t used as often.
A nurse can take a CRP quick test. Home test kits are available to order as well, but I haven't tried them.
[deleted]
Looks like they picked data that didn't require any work on their part.
I work for an AI company, so I can easily see that.
Original article https://jamanetwork.com/journals/jamainternalmedicine/fullar...
Not that "Epic" is the name of an Electronic Health Record system that embeds this proprietary algorithm for detecting sepsis.
Key Points
Question How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis?
Findings In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.
Meaning This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level.
Results We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.
Conclusions and Relevance This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
Not that "Epic" is the name of an Electronic Health Record system that embeds this proprietary algorithm for detecting sepsis.
Key Points
Question How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis?
Findings In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.
Meaning This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level.
Results We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.
Conclusions and Relevance This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
So that's euphemism for the algorithm doesn't work.
This case is worse than just not working: the algorithm generates alert fatigue and misleads doctors who would do a better job and come to better decisions without the algorithm. In the hospital context, a fair characterisation might be that the algorithm causes deaths.
Just like the days when every program would pop up warning after warning asking, "are you sure?". After a while users started to treat them as annoyances interfering with the task and automatically close them. They became worse than useless because among the noise there would sometimes be serious events that would lead to corruption or loss of data.
Prior to this algorithm most Epic customers had an alert for providers to check for Sepsis in a patient. In most cases the Epic tool is an improvement in over alerting.
That provides important context. However, based on the news, Epic has not been honest in their marketing of this feature. If you provide something that is less harmful but still harmful, you have to disclose that the real solution would be something else - fixing the insurance system, liabilities?
I can go for a run on a hot day- I'll be pyrexic, tachypnoeic, and tachycardic- enough to trigger a sepsis alert. Whereas an elderly patient who is a bit more confused than normal may have a very severe raging infection with few changes in their markers.