I work at a telemedicine company. We’ve benchmarked a few frontier LLMs on public medical imaging datasets. One test included high-quality and high-consensus otoscopic images. We didn’t anticipate the models to do well on something so niche, but what concerned us was how poorly calibrated the models were.
I know you can’t trust an LLM’s self-assessed “confidence” of a prediction, but I’ve found that confidence can at least be directionally correct for some tasks. For our benchmarks, however, confidence was poorly correlated. What’s worse is that binary classification models (“Do you see $diagnosis in this photo?”) highly influenced the LLM to confidently predict $diagnosis.
I’m concerned for those using LLMs for diagnostics, and getting confidently led to the wrong conclusion.
American healthcare is seldom affordable, accessible, or high-quality. We are fixing this for pediatrics. Blueberry is the most affordable option amongst our competitors. We practice the highest quality pediatric telemedicine, as evidenced by our exclusive hiring of board-certified pediatricians and the usage of at-home medical kits. And, we’re accessible 24 hours a day.
Our success is shown in the lives we save, the costs we save our insurers, and our exploding B2B and D2C business.
As you can imagine, pulling off affordable high-quality healthcare is a challenge. It requires a lot of engineering ingenuity, a C-suite aligned with positive patient outcomes above short-term profits, and a great product team.
We use Django, Hotwire Turbo (an HTMX-like framework), Pytorch, Sklearn, and Flutter. Experience in these technologies helps, but what’s more important is general full-stack knowledge, curiosity, and a strong work ethic.
American healthcare is seldom affordable, accessible, or high-quality. We are fixing this for pediatrics. Blueberry is the most affordable option amongst our competitors. We practice the highest quality pediatric telemedicine, as evidenced by our exclusive hiring of board-certified pediatricians and the usage of at-home medical kits. And, we’re accessible 24 hours a day.
Our success is shown in the lives we save, the costs we save our insurers, and our exploding B2B and D2C business.
As you can imagine, pulling off affordable high-quality healthcare is a challenge. It requires a lot of engineering ingenuity, a C-suite aligned with positive patient outcomes above short-term profits, and a great product team.
We use Django, Hotwire Turbo (an HTMX-like framework), Pytorch, Sklearn, and Flutter. Experience in these technologies helps, but what’s more important is general full-stack knowledge, curiosity, and a strong work ethic.
Agreed here. A key theme, which isn’t terribly explicit in this post, is that your codebase is your context.
I’ve found that when my agent flies off the rails, it’s due to an underlying weakness in the construction of my program. The organization of the codebase doesn’t implicitly encode the “map”. Writing a prompt library helps to overcome this weakness, but I’ve found that the most enduring guidance comes from updating the codebase itself to be more discoverable.
American healthcare is seldom affordable, accessible, or high-quality. We are fixing this for pediatrics. Blueberry is the most affordable option amongst our competitors. We practice the highest quality pediatric telemedicine, as evidenced by our exclusive hiring of board-certified pediatricians and the usage of at-home medical kits. And, we’re accessible 24 hours a day.
Our success is shown in the lives we save, the costs we save our insurers, and our exploding B2B and D2C business.
As you can imagine, pulling off affordable high-quality healthcare is a challenge. It requires a lot of engineering ingenuity, a C-suite aligned with positive patient outcomes above short-term profits, and a great product team.
We use Django, Hotwire Turbo (an HTMX-like framework), Pytorch, Sklearn, and Flutter. Experience in these technologies helps, but what’s more important is general full-stack knowledge, curiosity, and a strong work ethic.
I know you can’t trust an LLM’s self-assessed “confidence” of a prediction, but I’ve found that confidence can at least be directionally correct for some tasks. For our benchmarks, however, confidence was poorly correlated. What’s worse is that binary classification models (“Do you see $diagnosis in this photo?”) highly influenced the LLM to confidently predict $diagnosis.
I’m concerned for those using LLMs for diagnostics, and getting confidently led to the wrong conclusion.