Automation Bias

Automation bias is excessive reliance on automated output, including acceptance of wrong output despite relevant contrary evidence. Article 14 of the EU AI Act addresses awareness of this tendency in high-risk oversight, and NIST’s Generative AI Profile places it within human-AI configuration risks. Agreement with a system is not itself bias: the relevant question is whether reliance is appropriate to its evidence, limitations, and task.

A human check can fail quietly when the person lacks evidence, time, expertise, or authority. An approval log records decisions, but cannot by itself establish that review was effective. Nor does frequent agreement prove ineffectiveness: a properly examined sample may contain no errors.

When it grows

The tendency is not a character flaw and it is not constant. It can grow with volume, because a reviewer who has to clear two hundred items before lunch may lack time to examine each one adequately; with time pressure, for the same reason; and with a good track record, because the hundredth correct match makes the hundred-and-first easier to accept unread. Generative systems add a factor of their own: NIST’s profile notes that confabulated output is often stated confidently and may come with confabulated logic or citations, so a wrong answer arrives looking justified. Interface design adds the last factor. A screen that shows a conclusion and an accept button invites acceptance; a screen that shows the evidence and asks for a decision does not, or at least does so less.

A fictional case from payments shows the end state. An assistant matched supplier invoices to purchase orders and prepared the weekly payment run. The approval screen listed twelve matched invoices and a total. Over months the clerk had never found a mismatch, and approval took a few seconds. When one invoice carried a changed bank account, the change was in the run and not on the screen, and it was approved with the rest. Nothing about the clerk’s diligence had changed. The design had made diligence irrelevant.

Measuring it

Track overrides by reviewer, case type, and time, with the reviewed-case denominator, missing records, and routing rules. A fall toward zero is a reason to investigate, not proof of perfect performance or abandoned review. For example, a system with a 1% independent error probability has about a 36.6% chance of producing no errors in 100 cases. Independent audits and observation help distinguish explanations. In an isolated test, present realistic known errors and measure detection under stated workloads and sampling conditions. Prevent those test items from issuing payments, changing records, or affecting people. A missed error establishes a failed check on that case, not its psychological cause; missing evidence or expertise may explain it. The measured catch rate describes the test set, not automatically all production control coverage. human oversight develops the broader assessment.

Designing against it

Articles 14 and 26 connect effective oversight with understanding limitations, interpreting outputs, overriding or stopping where appropriate, and providing competence, training, authority, and support. The following design choices put those capabilities to work; each needs testing in the actual task.

• Show evidence, not a conclusion. The source document, the fields extracted from it, the rule applied, and any change with the old value beside the new one.
• Require entry for what matters. For a critical field, the reviewer types or selects the value rather than accepting a pre-filled one.
• Size the queue for a real check. Test whether the allowed time is sufficient. Sampling may support monitoring, but cannot replace required pre-action review of every high-impact item; reduce volume, add capacity, or stop the affected path when coverage is insufficient.
• Separate high-impact items. A change of bank details, a rejection, a deletion appears on its own, not inside a batch total.
• Show uncertainty where it exists. A low retrieval score or a disagreement between two checks is information the reviewer should see.
• Feed back. Tell reviewers which of their approvals turned out to be wrong, so that the track record they calibrate on is accurate.
• Give the person authority. A reviewer who cannot reject, delay, or stop the system, or who is measured only on throughput, has been asked to supervise without the means.

NIST also identifies algorithmic aversion: inappropriate reluctance to rely on a system can undermine a useful tool. Neither maximizing overrides nor maximizing agreement is the objective. Compare reviewer decisions with suitable evidence and outcomes, investigate errors, and assess appropriate reliance for the task. The Invoice Gave the Instructions connects this to deciding which actions need review.

References

• Regulation (EU) 2024/1689 (Artificial Intelligence Act), consolidated text of 27 July 2026, EUR-Lex, Articles 14 and 26.
• NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024.