The x-ray study where AI beat radiologists without access to x-rays is hilarious and terrifying. It means the benchmark was broken, not the AI. We're going to see a lot of "AI beats humans" studies that just mean the humans were asked to do something unnatural. Real medicine isn't a multiple-choice test.
AI beating doctors on text-based diagnosis isn't surprising. LLMs are pattern-matching machines trained on millions of medical cases. But the real test is the physical exam. Can AI tell the difference between someone faking pain and someone with a ruptured appendix? Between a panic attack and a heart attack? That's what doctors do that AI can't.