Using ChatGPT for Differential Diagnosis Quiz

Answers and explanations appear as you take the quiz.

1. In Kanjee and colleagues' 2023 test of GPT-4 on NEJM clinicopathological cases, about how often was the final diagnosis anywhere in the model's differential?
   • About 39 percent
   • About 64 percent
   • About 92 percent
   • About 15 percent
2. What was the main result of the 2024 Goh et al. randomized trial?
   • Physicians given GPT-4 scored far higher than those without it
   • GPT-4 alone performed worse than every physician
   • Giving physicians GPT-4 did not significantly improve their diagnostic reasoning scores, although GPT-4 alone scored higher
   • The trial was stopped for patient harm
3. Why can vignette benchmarks overstate how well a model would do with real patients?
   • Vignettes arrive with the relevant findings already chosen and cleaned up, and published cases may be in training data
   • Vignettes are always written in another language
   • Real patients have fewer symptoms than vignettes
   • Models cannot read case descriptions longer than a paragraph
4. A clinician asks, "Could this be lupus?" at the start of her prompt. Which failure mode does this invite?
   • Invented laboratory reference ranges
   • Refusing to answer medical questions
   • Losing the conversation history
   • Sycophantic anchoring toward the diagnosis the user suggested
5. Why is scoring "correct diagnosis anywhere in the list" a flawed measure?
   • It rewards long lists, which can prompt unnecessary testing
   • It ignores the patient's age
   • It only works for pediatric cases
   • It requires the model to cite references
6. Getting two differently ordered lists from nearly identical prompts shows what about the model's ranking?
   • The model is broken and should not be used
   • The second answer is always the correct one
   • The ranking is not a calibrated probability estimate and shifts with wording
   • The model is searching the live internet
7. What is the main privacy concern with pasting a patient's chart into a consumer chatbot?
   • The chatbot will refuse clinical questions
   • Entering identifiable information without a business associate agreement can breach HIPAA
   • Consumer chatbots cannot read lab values
   • It automatically bills the patient's insurer
8. Which prompt request best helps a clinician act on a model's differential?
   • "Give me the single correct diagnosis"
   • "Confirm my diagnosis is right"
   • "Write the discharge summary"
   • "List supporting and contradicting features for each item and the test that best separates the top candidates"