Evaluating Medical AI Tools as a Clinician Quiz

Answers and explanations appear as you take the quiz.

1. A vendor reports an AUC of 0.85 for its deterioration model. What does the guide say this number fails to tell a hospital?
   • How well the model ranks cases in the vendor's own dataset
   • Whether the model was built with machine learning
   • How the tool will behave with the hospital's own patients, workflow and alert threshold
   • Whether the model can be installed in an EHR
2. What did the 2021 external validation of a proprietary sepsis model, published in JAMA Internal Medicine, find?
   • Much weaker discrimination than the developer reported, with many missed cases and frequent alerts
   • Performance identical to the developer's figures
   • Better performance at the external site than at the development site
   • That the FDA had revoked the model's clearance
3. A model was validated where a condition affects 10 percent of patients. Why might it generate more false alarms on a unit where only 2 percent have the condition?
   • Calibration always improves at lower prevalence
   • AUC automatically falls when prevalence falls
   • Sensitivity rises sharply at lower prevalence
   • Positive predictive value depends on prevalence, so a larger share of alerts are false when the condition is rarer
4. According to the guide, which pathway do most AI medical devices use to reach the US market?
   • Premarket approval backed by randomized trials
   • The 510(k) pathway, based on substantial equivalence to an existing device
   • No FDA review of any kind
   • Emergency use authorization
5. What is a silent or shadow deployment of a clinical AI model?
   • Muting the sound on alert notifications
   • A vendor updating the model without telling the customer
   • Letting the model score live patients without showing results to clinicians, so local performance can be measured
   • Running the model only during overnight shifts
6. A dermatology group learns that darker skin tones were rare in a classifier's training images. Which evaluation area does this concern?
   • Post-deployment monitoring
   • Bias, meaning performance across patient subgroups
   • Regulatory status
   • Alert threshold selection
7. A model assigns 30 percent risk to a group of patients, but only 10 percent of them go on to have the outcome. What problem does this show?
   • A low AUC, necessarily
   • Substantial equivalence failure
   • High positive predictive value
   • Poor calibration
8. After go-live, a hospital switches to a new lab assay and the values feeding the model shift. What does the guide call this, and how should it be handled?
   • Input drift, which ongoing monitoring should detect
   • Algorithm aversion, which training should fix
   • Substantial equivalence, which the FDA handles
   • Prevalence shift, which needs no action