Number 7 of 12 · November 12, 2026

DEPUTYBENCH: Evaluating Contextual Safety in Enterprise Agents

SI agents in the DEPUTYBENCH tests were mostly safe when the evidence was clear. They were more likely to act anyway when the evidence needed to confirm permission was missing.

What they found

The authors built DEPUTYBENCH, a runnable test with 79 base items and 375 instances. It checks whether an enterprise SI agent can tell if a normal request is allowed by company records and policies, while still doing legitimate work. Across 12 models, revised audits gave 88.0% pooled safety and 79.1% paired success. In a controlled study of 28 derived tasks and six models, restricted actions were run more often when evidence was missing than when permission was explicitly denied: 10.3% versus 3.5% on 27 matched tasks. This was a secondary analysis. A post-hoc look found 61 of 100 masked-condition executions in four privacy tasks. Those tasks needed complete cumulative counts or person-specific consent, and the gap was much smaller outside them.

Why it matters

Companies want SI agents to handle routine requests on their own. This paper suggests that good scores when the evidence is available do not show an agent will stop when permission cannot be checked. The authors also report that irrelevant rewrites of tasks did not change mean paired success beyond a five-point margin, and that strong-model safety stayed near ceiling. The authors say the concentration result describes task structure and audited outcomes, not an internal mechanism. Stage labels are still LLM-generated, though two security experts checked task ground truth by hand over two weeks.

How it changes the race

This is a testing tool, not a capability gain. It gives builders of enterprise SI agents a way to measure a specific gap: acting when permission cannot be verified. The result is limited to this benchmark and concentrated in a few privacy tasks.

Safety and control

Agents sometimes ran restricted actions when evidence of permission was missing, more often than when permission was explicitly denied. Safety checks for deployed agents should test missing evidence as well as clear refusals.

The abstract, by the authors

Enterprise agents must determine whether ordinary requests are permitted by organizational records and policies while continuing legitimate work. We introduce DEPUTYBENCH, an executable benchmark with 79 base items, 375 instances, paired permission contexts, and graded evidence requirements. Across 12 models, revised audits give 88.0% pooled safety and 79.1% paired success. A controlled study of 28 derived tasks and six models tests authorization-fact flips, evidence masking, and irrelevant rewrites. Outcomes track backend permissions, yet restricted execution is more frequent with missing evidence than explicit non-permission: 10.3% versus 3.5% on 27 matched tasks, a 6.8-percentage-point difference (95% interval: 1.9-13.0; secondary analysis). A post-hoc analysis locates 61 of 100 masked-condition executions in four privacy tasks requiring complete cumulative counts or person-specific consent. The difference persists when the most error-prone model or the largest contributing task is excluded, but is much smaller outside those four tasks. Thus high safety with available evidence does not establish reliable stopping when permission cannot be verified. Mean paired success is equivalent within a prespecified five-point margin under irrelevant rewrites, while strong-model safety remains near ceiling. Scorer validation improves from 178 to 266 correctly scored cases out of 284 held-out counterexamples. Two security experts manually verified task ground truth over a two-week period; stage labels remain LLM-generated. The concentration analysis describes task structure and audited outcomes, not an identified internal mechanism.

Quoted from arXiv:2611.13355. The research and its words belong to its authors.

The authors

Hao Zhao, Minghao Lin

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.