New Benchmark Finds AI Agents Wrongly Block Approved Work 28% of the Time

Что случилось

Researchers Oguz Serdar and Cuneyt Mertayak posted a preprint describing SteerBench-Work, a benchmark for the pre-commit "proceed or hold" decision in workplace AI agents. They report that across 30 model conditions, models wrongly held authorized, evidence-cleared work on 28.1% of opportunities and wrongly allowed unsafe work on 1.0%.

Two researchers, Oguz Serdar and Cuneyt Mertayak, posted a preprint to arXiv on 12 August 2026 introducing SteerBench-Work, a benchmark built around a single decision inside a long-running AI agent's workflow: whether to commit an action or hold it for human or policy review. The authors call this the steering decision and place it at what they term the action boundary — the point just before an agent sends an email, merges a pull request, or wires a payment. The benchmark does not score whether an agent can complete a task. It scores whether the agent crosses or holds that boundary correctly.

The release described in the abstract is labelled v2026-05 and contains 106 scenarios spanning developer operations, customer service, finance, legal, medical, HR, and security. The scenarios are anchored in public incidents. Each is paired with what the authors call an evidence-reversed mirror — a version of the same situation in which the underlying evidence points the other way — alongside calibration controls. Labels are split nearly evenly between proceed and hold, which the authors say is deliberate: it gives the two error directions close to the same number of chances to occur, so a model cannot score well simply by defaulting to caution. In each item, a model is shown a proposed action and the available evidence, and returns a gate decision.

Across 30 model conditions, the authors report that failures run almost entirely in one direction. Models wrongly held authorized, evidence-cleared work on 28.1% of opportunities, while wrongly allowing unsafe work on 1.0%. The hardest category, according to the abstract, is what the authors call risk-resolved commits: cases where a genuine risk trigger fired but signed or structured evidence has already cleared it, and the correct answer is to proceed. Performance also diverged sharply between famous incidents and their mirrors — 98.5% on the incidents themselves versus 63.8% on the evidence-reversed versions.

The authors draw a further distinction between general capability and steering calibration. Higher-capability models, they write, often over-refuse at the commit boundary, and additional reasoning can repair a weak gate while leaving an already-calibrated one flat. The abstract points to a public leaderboard, but the source page renders the address as a placeholder rather than a working link.

Several things are not established by the material available here. The abstract does not name the models tested, does not define what counts as a distinct "model condition" among the 30, and gives no per-domain or per-model breakdown. It does not describe who wrote the ground-truth labels or how disagreements were resolved, and the version label v2026-05 is not explained against the August submission date. This is a version-one preprint with no indication of peer review, and none of the figures have been independently reproduced.

Подробности об источнике: arxiv.org ↗

Почему это важно

Most agent safety testing measures whether models block harmful actions. This work measures both error directions and finds the dominant failure is over-blocking, which carries real operating costs and can push humans into rubber-stamping. The numbers come from an unreviewed preprint and have not been independently verified.

AI agents are increasingly deployed to take actions rather than only produce text, and the point where a suggestion becomes an irreversible effect is where the risk actually lands. A great deal of published agent safety work asks one question — did the model refuse the harmful thing? — and a system that refuses everything scores perfectly on that question while being useless. By balancing proceed and hold labels and reporting both error rates, this benchmark measures the tradeoff instead of one side of it.

If the reported asymmetry generalizes, the practical problem in enterprise agent deployments may be closer to over-blocking than to runaway action. False holds are not free. Each one routes work back to a person, which erodes the efficiency case for automating the task at all, and adds latency to processes such as ticket resolution or code review where the delay itself has a cost. There is a subtler risk too: a review queue full of unnecessary escalations trains reviewers to approve quickly, which weakens the human oversight that the hold mechanism exists to provide. That downstream effect is a reasonable inference from the setup, not something the benchmark measures.

The gap between 98.5% on famous incidents and 63.8% on their evidence-reversed mirrors is the result most worth scrutiny. It is consistent with models recognizing the shape of a well-documented failure and responding to that recognition rather than reading the evidence in front of them. If that reading holds, it complicates how buyers should interpret agent safety scores generally: a model can look reliable on scenarios that resemble published cautionary tales while handling the same structural situation poorly when the facts are rearranged. The authors' "risk-resolved commits" category tests exactly this — evidence exists that clears a triggered risk, and the model must actually process it.

The findings also bear on how human-in-the-loop requirements are written. Policies that mandate review for categories of action assume the gate is the safe default. This work suggests the gate has its own error rate, in the direction of blocking cleared work, and that the rate should be measured rather than assumed to be near zero.

Weighing against all of this: the benchmark evaluates a single pre-commit judgment given a curated evidence packet, not an agent operating a real tool loop with incomplete information it must gather itself. Real deployments also sit behind permission systems and organizational context that the scenarios cannot fully represent. Whether 106 constructed scenarios, however carefully anchored, predict behavior in production is an open question, and the claims here are the authors' own.

Что посмотреть дальше

Whether the full paper, dataset, and leaderboard identify the models tested; whether independent groups reproduce the over-refusal gap; whether the evidence-reversed mirror results hold up as a sign of pattern-matching; and whether buyers and regulators start treating false holds as a measured failure rather than a safe default.

The most immediate thing to look for is disclosure. The full paper, any released dataset or code, and the leaderboard the abstract references would show which models were tested, how the 30 conditions were composed, and how the 28.1% figure distributes across models and domains. An aggregate number spanning 30 conditions can conceal wide variation, and the practical question for anyone selecting a model is which systems sit at which end of that range.

Independent replication matters more than the headline figure. Watch for other groups running the scenarios, for scrutiny of the ground-truth labels on the hardest "risk-resolved commit" cases, and for whether the incident-versus-mirror gap survives testing by people who did not build the benchmark. That gap is the paper's most consequential claim, and it is also the one most dependent on how the mirrors were constructed.

Also worth tracking is whether the single-step framing transfers. A model asked to gate one proposed action with evidence supplied is in a different position from an agent mid-task that must decide what evidence to gather before committing. Follow-up work placing the same scenarios inside full agent loops would test whether the calibration measured here predicts real behavior.

On the adoption side, the question is whether false-hold rates enter procurement and system cards alongside refusal rates, and whether model developers begin tuning for calibration at the action boundary rather than for caution alone. The authors' claim that added reasoning helps weak gates but not already-calibrated ones, if confirmed, would push against the assumption that more inference-time compute reliably improves safety-relevant judgment.

Finally, watch the versioning. A benchmark labelled v2026-05 and anchored in public incidents faces a contamination problem as its scenarios circulate and future models train on them. How the authors refresh the set — and whether mirror performance improves through genuine reasoning gains or through exposure — will determine how long the numbers mean what they mean today.

Сопутствующие руководства и викторины