Here's an example that illustrates a shortcoming about how medicine has historically been practiced and how it might change with the advent of AI.
A person very familiar with me was having an interaction at a review board level at Stanford. They had this rare illness that they were treating and someone had the "bright idea" of suggesting of saying ... "hey, why don't we look at the ten other times we treated this exact illness and see what worked!" Everyone was delighted with this novel idea (discussion circa 2021). My person was a bit disgusted as this is simple feedback loop style improvement and WTF! they should be doing this all the time to get probabilistic style suggestions for many treatments. I know it happens within certain healthcare systems (e.g., Kaiser had full EMR back in early 2000s and saw right away that VIOXX was killing people. So, they stop prescribing it. citation: Kaiser panel member paraphrase at a healthcare conference in 2010). If you just observe the healthcare system, you can see that the healthcare systems and most EMRs don't typically capture the feedback loop (i.e, when's the last time a doctor followed up and said "did you feel better after the last treatment?" or measures the result.) AI itself can't solve this as it doesn't have access to the data feedback loop. However, maybe AI's within the EMR will help "suggest" evidence based treatments. I could go on and on, but as a math guy, I've often been shocked at the non-evidence based assertions some doctors make. My conclusion is that if you're not "in the fairway", they are typically just guessing.
My mom had cancer and she was on regular, suppressive chemotherapy. I put her info into an AI and it correctly noted that her chemotherapy had stopped being effective 2 months prior based on factual lab reports. She was unaware of this. I was able to be her health advocate much more effectively by respectfully asking her oncologist targeted questions. He was already on top of it and was addressing the issue. Our conversation was respectful and, due to my educating myself, went up another level. Ultimately, it was a positive interaction. I was satisfied that he was indeed expert at his craft, and he was satisfied that we were aware of the uncertainty of the new treatment with a risk-based understanding of the viability of success. This was a positive engagement with an expert. In parallel situations around non-health issues, I've found the ego of the expert seems to be the determinative factor in whether or not the interaction goes well.
Maybe a difference here is asking AI for conclusions. When I have it do a buyer's report for me, I ask it for "what questions should I be asking? What are typical things that go wrong with this type of vehicle?" I don't delegate conclusions to the AI but use it to educate myself. Then, I can gather further information to make MY decision .. to buy it or not.
A person very familiar with me was having an interaction at a review board level at Stanford. They had this rare illness that they were treating and someone had the "bright idea" of suggesting of saying ... "hey, why don't we look at the ten other times we treated this exact illness and see what worked!" Everyone was delighted with this novel idea (discussion circa 2021). My person was a bit disgusted as this is simple feedback loop style improvement and WTF! they should be doing this all the time to get probabilistic style suggestions for many treatments. I know it happens within certain healthcare systems (e.g., Kaiser had full EMR back in early 2000s and saw right away that VIOXX was killing people. So, they stop prescribing it. citation: Kaiser panel member paraphrase at a healthcare conference in 2010). If you just observe the healthcare system, you can see that the healthcare systems and most EMRs don't typically capture the feedback loop (i.e, when's the last time a doctor followed up and said "did you feel better after the last treatment?" or measures the result.) AI itself can't solve this as it doesn't have access to the data feedback loop. However, maybe AI's within the EMR will help "suggest" evidence based treatments. I could go on and on, but as a math guy, I've often been shocked at the non-evidence based assertions some doctors make. My conclusion is that if you're not "in the fairway", they are typically just guessing.