Automation Bias in Clinical AI

Automation bias was first studied closely in aviation and other safety-critical work. Human factors researchers found that operators working with reliable automation tend to monitor less and defer more. They described two error types. In an error of omission, a person misses a problem because the system did not flag it. In an error of commission, a person acts on a wrong suggestion. A 2012 systematic review by Goddard and colleagues found automation bias in clinical decision support, with effects shaped by task complexity, workload, experience and confidence. Medical imaging supplies the clearest evidence. Studies of early computer-aided detection in mammography reported that when the system failed to prompt a cancer, readers were sometimes less likely to find it than they would have been without the system. A 2023 study in Radiology gave radiologists deliberately incorrect suggestions presented as coming from AI when rating mammograms. Their accuracy dropped, and less experienced readers were affected most. A 2023 vignette study in JAMA found that clinicians shown a systematically biased model made less accurate diagnoses, and that adding model explanations did little to offset the harm. The opposite problem also exists. After seeing an algorithm make mistakes, people may stop using it even when it outperforms them. Researchers outside medicine have called this algorithm aversion. In hospitals it often looks like alert fatigue. Neither extreme is safe. The goal is appropriate reliance: trusting the tool where it is strong and checking it where it is weak. Two misconceptions matter: putting a "human in the loop" guarantees safety. If the human reliably defers, the loop adds little protection; and Explanations always help. Some research shows they can increase trust in wrong outputs as easily as in correct ones.