Automation Bias: Overtrusting AI

Automation bias describes a human tendency to use an automated decision aid as a heuristic replacement for vigilant information seeking and processing. It can create commission errors, when a person follows an incorrect recommendation, and omission errors, when a person fails to notice a condition because automation did not alert them. The term predates current generative AI and has been studied in aviation and decision-support settings. A polished interface or “human in the loop” label does not show that a reviewer meaningfully checked the evidence. The mechanism is not simply that people trust machines. Workload, time pressure, task complexity, expertise, interface design, the visibility of system confidence, and availability of independent evidence can all affect reliance. Mosier and Skitka describe automation bias as taking the path of least cognitive effort; their research found users can defer to decision aids even when they have other information. In medical decision support, studies reviewed in the literature have documented clinicians changing correct judgments after erroneous computerized advice. These findings do not mean every user or every AI system produces the same effect. Human oversight helps only when the reviewer has enough time, competence, authority, and access to source information to disagree. Teams can design workflows that require an initial independent assessment, make uncertainty and evidence visible, prompt verification of high-impact claims, and monitor both commission and omission errors. Training can help users recognize failure modes, though training alone is not a complete control. For generative AI, citation checks, source retrieval, and verification of actions in authoritative systems reduce reliance on confident but unsupported outputs. Measure behavior, not just the existence of a review step. Record overrides, missed alerts, unsupported accepted suggestions, and whether reviewers inspect source evidence. Test under realistic workload and UI conditions. If an organization cannot provide an effective review path, it should limit automation’s authority or avoid the use in that context.