āļ„āļđāđˆāļĄāļ·āļ­āđāļ­āļ›āļžāļĨāļīāđ€āļ„āļŠāļąāļ™

āļāļēāļĢāđƒāļŠāđ‰ ChatGPT āļŠāļģāļŦāļĢāļąāļšāļāļēāļĢāļ§āļīāļ™āļīāļˆāļ‰āļąāļĒāđāļĒāļāđ‚āļĢāļ„

Large language models such as GPT-4 can produce plausible differential diagnosis lists from a case description, and in published studies they often include the correct diagnosis.

āļŸāļąāļ‡āļ„āļģāđāļ™āļ°āļ™āļģāļ™āļĩāđ‰āļšāļĢāļĢāļĒāļēāļĒāđ‚āļ”āļĒ AI · ~3 min

āļšāļ™āļŦāļ™āđ‰āļēāļ™āļĩāđ‰āļ­āđˆāļēāļ™ 4 āļ™āļēāļ—āļĩ
  1. āļ āļēāļžāļĢāļ§āļĄ
  2. āđ€āļˆāļēāļ°āļĨāļķāļ
  3. āļāļēāļĢāđƒāļŠāđ‰āļ‡āļēāļ™āļˆāļĢāļīāļ‡āđƒāļ™āđ‚āļĨāļāđāļŦāđˆāļ‡āļ„āļ§āļēāļĄāđ€āļ›āđ‡āļ™āļˆāļĢāļīāļ‡
  4. The Future of Using ChatGPT for Differential Diagnosis
  5. āļāļēāļĢāļ™āļģāđ„āļ›āļ›āļāļīāļšāļąāļ•āļī
  6. āļŠāļģāļĢāļ§āļˆāļ•āđˆāļ­āđ„āļ›
  7. āļ„āļģāļ–āļēāļĄāļ—āļĩāđˆāļžāļšāļšāđˆāļ­āļĒ

āļ āļēāļžāļĢāļ§āļĄ

They are not validated diagnostic devices. Their safest use is as a structured second-opinion brainstorm that a clinician checks against the patient, and doctors who understand both their strengths and their failure modes get the most out of them.

āđ€āļˆāļēāļ°āļĨāļķāļ

Much of the evidence comes from case vignettes. In a 2023 JAMA research letter, Kanjee and colleagues tested GPT-4 on 70 difficult cases from the New England Journal of Medicine clinicopathological conferences. The final diagnosis appeared in the model's differential in about 64 percent of cases and was its top diagnosis in about 39 percent. A randomized trial by Goh and colleagues, published in JAMA Network Open in 2024, gave 50 physicians either GPT-4 plus usual resources or usual resources alone for structured diagnostic cases. Access to GPT-4 did not significantly improve physicians' diagnostic reasoning scores. GPT-4 working alone, however, scored substantially higher than the physicians who used only usual resources. One interpretation is that physicians did not know how to use the tool well, or did not trust it when it disagreed with them. Research systems such as Google's AMIE have also been tested on diagnostic dialogue and differential generation, with strong results in controlled studies. These results need careful reading. Published case series may be in a model's training data. Vignettes arrive already cleaned up, with the relevant findings chosen and summarized, while real patients bring incomplete, contradictory and unstated information. Scoring "correct diagnosis somewhere in the list" rewards long lists, which can prompt extra testing. The common misconception is that benchmark accuracy equals bedside accuracy. It does not. Known failure modes include confident but wrong reasoning, invented references or lab thresholds, anchoring on whatever diagnosis the user hints at, over-weighting common textbook presentations, and missing time-critical conditions when the description leaves out vital signs. Privacy is a separate issue. Entering identifiable patient information into a consumer chatbot without a business associate agreement can breach HIPAA, so clinicians should use tools their organization has approved.

āļāļēāļĢāđƒāļŠāđ‰āļ‡āļēāļ™āļˆāļĢāļīāļ‡āđƒāļ™āđ‚āļĨāļāđāļŦāđˆāļ‡āļ„āļ§āļēāļĄāđ€āļ›āđ‡āļ™āļˆāļĢāļīāļ‡

A hospitalist types a de-identified summary of fever, rash, eosinophilia and a new anticonvulsant into her health system's approved AI tool and asks for can't-miss diagnoses. DRESS appears on the list, and she reviews the medication timeline.

A resident asks the model which findings would best tell apart his top three diagnoses for acute dyspnea, then uses the answer to plan a focused exam and tests, not to settle on a diagnosis.

A primary care physician who suspects a viral illness asks the model to argue against her leading diagnosis and list what she might be missing. This is a deliberate check on anchoring.

A clinician gets two different ranked lists after asking the same question twice with slightly different wording, and takes that as a sign the model's ordering is not a probability estimate.

The Future of Using ChatGPT for Differential Diagnosis

Diagnostic AI is moving from general chatbots toward tools built into EHRs and evidence platforms, where they can see structured data and cite sources. That may reduce some errors, but it adds the risk that clinicians defer to a confident suggestion. The Goh trial points to the open question: how clinicians and models should work together, not just how accurate the model is alone. Prospective studies with real patients and real outcomes are still scarce compared with vignette benchmarks. Until more exist, the defensible position is that these tools can widen a differential and prompt reconsideration. The clinician still owns the diagnosis.

āļāļēāļĢāļ™āļģāđ„āļ›āļ›āļāļīāļšāļąāļ•āļī

āļ—āļģāđ„āļĄāļĄāļąāļ™āļ–āļķāļ‡āļŠāļģāļ„āļąāļ

  • āļŠāļĢāđ‰āļēāļ‡āļ—āļēāļ‡āđ€āļĨāļ·āļ­āļ. āļāļēāļĢāļ­āļ­āļāđāļšāļšāļĢāļ°āļ”āļąāļšāđāļ­āļ›āļžāļĨāļīāđ€āļ„āļŠāļąāļ™āļˆāļ°āļāļģāļŦāļ™āļ”āļ§āđˆāļē AI āļˆāļ°āļ›āļĢāļąāļšāļ›āļĢāļļāļ‡āļœāļĨāļĨāļąāļžāļ˜āđŒāļ—āļĩāđˆāđāļ—āđ‰āļˆāļĢāļīāļ‡āļŦāļĢāļ·āļ­āđ„āļĄāđˆ
  • āļ—āļĩāļĄāļ‡āļēāļ™āđāļĨāļ°āļ‚āļąāđ‰āļ™āļ•āļ­āļ™āļāļēāļĢāļ—āļģāļ‡āļēāļ™. āļāļēāļĢāļšāļđāļĢāļ“āļēāļāļēāļĢāļ‚āļąāđ‰āļ™āļ•āļ­āļ™āļāļēāļĢāļ—āļģāļ‡āļēāļ™āļ—āļĩāđˆāļ”āļĩāļˆāļ°āļŠāđˆāļ§āļĒāđ€āļžāļīāđˆāļĄāļœāļĨāļœāļĨāļīāļ•āļ—āļĩāđˆāļœāļđāđ‰āđƒāļŠāđ‰āđ„āļ§āđ‰āļ§āļēāļ‡āđƒāļˆāđ„āļ”āđ‰
  • āļ„āļ§āļēāļĄāđ€āļŠāļĩāđˆāļĒāļ‡āđāļĨāļ°āļ„āļ§āļēāļĄāļ›āļĨāļ­āļ”āļ āļąāļĒ. āļāļĢāļ“āļĩāļāļēāļĢāđƒāļŠāđ‰āļ‡āļēāļ™āļ—āļĩāđˆāļĄāļĩāļ‚āļ­āļšāđ€āļ‚āļ•āļ”āļĩāļˆāļ°āļŠāđˆāļ§āļĒāļĨāļ”āļ„āļ§āļēāļĄāđ€āļŦāļ™āļ·āđˆāļ­āļĒāļĨāđ‰āļēāļ‚āļ­āļ‡āļāļēāļĢāđ€āļ›āļĨāļĩāđˆāļĒāļ™āđāļ›āļĨāļ‡āđāļĨāļ°āļ„āļ§āļēāļĄāđ€āļŠāļĩāđˆāļĒāļ‡āđƒāļ™āļāļēāļĢāļ”āļģāđ€āļ™āļīāļ™āļāļēāļĢ

āļĢāļ°āļ§āļąāļ‡

  • āļāļēāļĢāļ—āļģāđƒāļŦāđ‰āļāļĢāļ°āļšāļ§āļ™āļāļēāļĢāļ—āļĩāđˆāđ€āļŠāļĩāļĒāļŦāļēāļĒāđ€āļ›āđ‡āļ™āļ­āļąāļ•āđ‚āļ™āļĄāļąāļ•āļīāļŠāļēāļĄāļēāļĢāļ–āļ‚āļĒāļēāļĒāļ›āļąāļāļŦāļēāļ—āļĩāđˆāļĄāļĩāļ­āļĒāļđāđˆāđ„āļ”āđ‰
  • āļ—āļĩāļĄāļ­āļēāļˆāļ”āļģāđ€āļ™āļīāļ™āļāļēāļĢāļ­āļąāļ•āđ‚āļ™āļĄāļąāļ•āļīāļĄāļēāļāđ€āļāļīāļ™āđ„āļ›āđāļĨāļ°āļĨāļšāļ§āļīāļˆāļēāļĢāļ“āļāļēāļ“āļ‚āļ­āļ‡āļĄāļ™āļļāļĐāļĒāđŒāļ—āļĩāđˆāļˆāļģāđ€āļ›āđ‡āļ™āļ­āļ­āļ
  • āļ„āļļāļ“āļ āļēāļžāļ­āļēāļˆāļ„āļĨāļēāļ”āđ€āļ„āļĨāļ·āđˆāļ­āļ™āđ„āļ”āđ‰āļŦāļēāļāđ„āļĄāđˆāđ„āļ”āđ‰āļĢāļąāļšāļāļēāļĢāļ›āļĢāļ°āđ€āļĄāļīāļ™āļœāļĨāļĨāļąāļžāļ˜āđŒāļ­āļĒāđˆāļēāļ‡āļ•āđˆāļ­āđ€āļ™āļ·āđˆāļ­āļ‡

āļ‚āļąāđ‰āļ™āļ•āļ­āļ™āļ•āđˆāļ­āđ„āļ›

  1. āđāļĄāļ›āļ‚āļąāđ‰āļ™āļ•āļ­āļ™āļāļēāļĢāļ—āļģāļ‡āļēāļ™āļ›āļąāļˆāļˆāļļāļšāļąāļ™āđāļĨāļ°āļĢāļ°āļšāļļāļ‚āļąāđ‰āļ™āļ•āļ­āļ™āļ—āļĩāđˆāļĄāļĩāđāļĢāļ‡āđ€āļŠāļĩāļĒāļ”āļ—āļēāļ™āļŠāļđāļ‡āļŠāļļāļ”
  2. āļāļģāļŦāļ™āļ”āļˆāļļāļ”āļ•āļĢāļ§āļˆāļ‚āļ­āļ‡āļĄāļ™āļļāļĐāļĒāđŒāļāđˆāļ­āļ™āļĢāļ°āļšāļšāļ­āļąāļ•āđ‚āļ™āļĄāļąāļ•āļīāđ€āļ•āđ‡āļĄāļĢāļđāļ›āđāļšāļš
  3. āļāļķāļāļ­āļšāļĢāļĄāļœāļđāđ‰āđƒāļŠāđ‰āđ€āļāļĩāđˆāļĒāļ§āļāļąāļšāļāļēāļĢāđāļˆāđ‰āļ‡āđ€āļ•āļ·āļ­āļ™ āđ€āļŠāđ‰āļ™āļ—āļēāļ‡āļāļēāļĢāļĒāļāļĢāļ°āļ”āļąāļš āđāļĨāļ°āļĄāļēāļ•āļĢāļāļēāļ™āļ„āļļāļ“āļ āļēāļž
  4. āļ•āļīāļ”āļ•āļēāļĄāļœāļĨāļĨāļąāļžāļ˜āđŒāļĢāļ°āļ”āļąāļšāļ‡āļēāļ™āđ€āļžāļ·āđˆāļ­āļĒāļ·āļ™āļĒāļąāļ™āļ„āļļāļ“āļ„āđˆāļēāļ—āļĩāđˆāļĒāļąāđˆāļ‡āļĒāļ·āļ™

āļŠāļģāļĢāļ§āļˆāļ•āđˆāļ­āđ„āļ›

āļˆāļ”āļŦāļĄāļēāļĒāļ‚āđˆāļēāļ§āļŸāļĢāļĩ

āļĢāļąāļšāļŠāļĢāļļāļ› AI āļĢāļēāļĒāļ§āļąāļ™

āđ€āļĢāļ·āđˆāļ­āļ‡āļĢāļēāļ§ AI āļ—āļĩāđˆāļ•āļĢāļ§āļˆāļŠāļ­āļšāđāļĨāđ‰āļ§ 3 āđ€āļĢāļ·āđˆāļ­āļ‡āļ—āļļāļāļ§āļąāļ™āļ—āļģāļ‡āļēāļ™ āđ€āļŠāđ‰āļē, āđ€āļ‚āļĩāļĒāļ™āđ€āļ›āđ‡āļ™āļ āļēāļĐāļēāļ­āļąāļ‡āļāļĪāļĐāļ˜āļĢāļĢāļĄāļ”āļē āļŸāļĢāļĩāļ•āļĨāļ­āļ”āđ„āļ› āđ„āļĄāđˆāļĄāļĩāđ‚āļ†āļĐāļ“āļē

āļ­āļĩāđ€āļĄāļĨāļŦāļ™āļķāđˆāļ‡āļ‰āļšāļąāļšāļ—āļļāļāļ§āļąāļ™āļ—āļģāļ‡āļēāļ™ āļĒāļāđ€āļĨāļīāļāļāļēāļĢāļŠāļĄāļąāļ„āļĢāđƒāļ™āļ„āļĨāļīāļāđ€āļ”āļĩāļĒāļ§ āđ€āļĢāļēāđ„āļĄāđˆāđ€āļ„āļĒāļ‚āļēāļĒāļŦāļĢāļ·āļ­āđāļŠāļĢāđŒāļ—āļĩāđˆāļ­āļĒāļđāđˆāļ­āļĩāđ€āļĄāļĨāļ‚āļ­āļ‡āļ„āļļāļ“

āļ—āļ”āļŠāļ­āļšāļ•āļąāļ§āđ€āļ­āļ‡

Take the Using ChatGPT for Differential Diagnosis quiz

āļ‚āđ‰āļ­āđ€āļŠāļ™āļ­āđāļ™āļ°āļ—āļąāļ™āļ—āļĩāļŠāļģāļŦāļĢāļąāļšāļ—āļļāļāļ„āļģāļ•āļ­āļš āđāļĨāļ°āđƒāļšāļĢāļąāļšāļĢāļ­āļ‡āļ—āļĩāđˆāļŠāļēāļĄāļēāļĢāļ–āđāļŠāļĢāđŒāđ„āļ”āđ‰āļžāļĢāđ‰āļ­āļĄ ID āļ—āļĩāđˆāļ•āļĢāļ§āļˆāļŠāļ­āļšāđ„āļ”āđ‰āđ€āļĄāļ·āđˆāļ­āļ„āļļāļ“āļœāđˆāļēāļ™āļŦāļĨāļąāļāļŠāļđāļ•āļĢ

āđ€āļĢāļīāđˆāļĄāđāļšāļšāļ—āļ”āļŠāļ­āļš

āļŠāļ™āļąāļšāļŠāļ™āļļāļ™āļāļēāļĢāļĻāļķāļāļĐāļē AI āļŸāļĢāļĩ AI Understanding āđ€āļ›āđ‡āļ™āļ­āļ‡āļ„āđŒāļāļĢāđ„āļĄāđˆāđāļŠāļ§āļ‡āļŦāļēāļœāļĨāļāļģāđ„āļĢ 501(c)(3) — āđ„āļĄāđˆāļĄāļĩāđ‚āļ†āļĐāļ“āļē āđ„āļĄāđˆāļĄāļĩāļāļģāđāļžāļ‡āđ€āļ‡āļīāļ™, āļ•āļĨāļ­āļ”āđ„āļ› āļšāļĢāļīāļˆāļēāļ„

āļ„āļģāļ–āļēāļĄāļ—āļĩāđˆāļžāļšāļšāđˆāļ­āļĒ

What is Using ChatGPT for Differential Diagnosis?

Large language models such as GPT-4 can produce plausible differential diagnosis lists from a case description, and in published studies they often include the correct diagnosis. They are not validated diagnostic devices. Their safest use is as a structured second-opinion brainstorm that a clinician checks against the patient, and doctors who understand both their strengths and their failure modes get the most out of them.

In Kanjee and colleagues' 2023 test of GPT-4 on NEJM clinicopathological cases, about how often was the final diagnosis anywhere in the model's differential?

The final diagnosis was in the differential in about 64 percent of cases and was the top diagnosis in about 39 percent.

What was the main result of the 2024 Goh et al. randomized trial?

Physicians with GPT-4 did not significantly outperform physicians with usual resources, yet the model alone scored higher. That points to how people use the tool.

Why can vignette benchmarks overstate how well a model would do with real patients?

Pre-organized findings and possible training data contamination both make vignettes easier than messy real encounters.

A clinician asks, "Could this be lupus?" at the start of her prompt. Which failure mode does this invite?

Models tend to agree with the framing they are given, so hinting at a diagnosis pulls the differential toward it.

Why is scoring "correct diagnosis anywhere in the list" a flawed measure?

A longer list is more likely to contain the answer, but long differentials in practice can lead to extra and unnecessary workups.

āđ€āļĢāļĩāļĒāļ™āļĢāļđāđ‰āļ•āđˆāļ­āđ„āļ›

āļ„āļģāđāļ™āļ°āļ™āļģāļ—āļĩāđˆāđ€āļāļĩāđˆāļĒāļ§āļ‚āđ‰āļ­āļ‡

āļ„āļģāđāļ™āļ°āļ™āļģāđ€āļžāļīāđˆāļĄāđ€āļ•āļīāļĄāļ—āļĩāđˆāđ€āļĨāļ·āļ­āļāļŠāļģāļŦāļĢāļąāļšāļŦāļąāļ§āļ‚āđ‰āļ­āļ™āļĩāđ‰

āļĢāļąāļšāđƒāļšāļĢāļąāļšāļĢāļ­āļ‡ AI āļŸāļĢāļĩāļŦāļĨāļąāļāļŠāļđāļ•āļĢāđāļĨāļ°āļāļēāļĢāļŠāļ­āļšāļŠāļąāđ‰āļ™ āđ† āļŸāļĢāļĩ āđ„āļĄāđˆāļĄāļĩāļ„āđˆāļēāđƒāļŠāđ‰āļˆāđˆāļēāļĒ āđ„āļĄāđˆāļĄāļĩāļšāļąāļ•āļĢāđ€āļ„āļĢāļ”āļīāļ• āļ›āļĢāļ°āļĄāļēāļ“ 2.5 āļŠāļąāđˆāļ§āđ‚āļĄāļ‡āđ€āļĢāļīāđˆāļĄāļ•āđ‰āļ™āļŸāļĢāļĩ →