---
title: "What a CAGE 4 Score Measures: An Audit... | SI Paper Desk"
url: https://thesimind.si/papers/2611.14068/
description: "The authors audit a cyber-defense benchmark for SI agents and find its scores depend on a hidden simulator rule, on luck in policy sampling, and on unstated..."
---

Number 10 of 12 · November 13, 2026

# What a CAGE 4 Score Measures: An Audit of a Learned Cyber-Defense Benchmark, Its Reference Defenders, and the Tables Built on Them

The authors audit a cyber-defense benchmark for SI agents and find its scores depend on a hidden simulator rule, on luck in policy sampling, and on unstated scoring choices.

James Rice, Ji Hwan Park and Andres Kwasinski · Submitted November 10, 2026 · arXiv 2611.14068v1 · cs.CR

[Read the paper on arXiv](https://arxiv.org/abs/2611.14068) · [The PDF](https://arxiv.org/pdf/2611.14068) · [The day's 12](https://thesimind.si/papers/day/2026-11-13/)

## What they found

The authors rebuilt the two reference defenders for CAGE Challenge 4, KEEP and H-MARL Expert, in a fixed container. They found the simulator silently drops any action given to an agent that is still busy, yet still charges a point for a dropped Restore. The documented observation does not show the busy flag, but both defenders read it in their own wrappers. Switching that check off at evaluation costs KEEP 99.58 points per episode and H-MARL 145.98. The authors also report that 88.9% of per-episode outcome variance comes from the policy's own sampling draw, because evaluation seeds the environment but not the policy. In training, KEEP's repaired recipe ended inside the released interval on one seed and short on two, by 27 and 125 points. They say none of five published comparison tables states enough to make its numbers comparable.

## Why it matters

Benchmarks guide which SI methods people trust and build on. If a score depends on a hidden rule, a single lucky sampling draw, or an unstated scoring choice, then ranking methods by it can mislead. The authors release their instrument, patches, traces, checkpoints and a five-item disclosure checklist so others can report results more comparably. This is an audit of one benchmark, based on the abstract, so how far it applies to other benchmarks is not stated.

## How it changes the race

This does not make SI defenders more capable. It may make progress claims in learned cyber defense harder to inflate and easier to check, which favors teams that report results carefully. Comparisons on CAGE 4 may need to be redone.

## Safety and control

The paper bears on security and on trusting evaluations of SI cyber defenders. Scores that hide simulator quirks or sampling luck can overstate how well an SI defender would protect real systems.

### The abstract, by the authors

CAGE Challenge 4 is the benchmark on which learned multi-agent cyber defenders are compared. We reproduce its two reference defenders, KEEP and H-MARL Expert, in a pinned container and show that three things read from it measure less than they are taken to measure. First, the simulator silently discards any action asserted for an agent whose previous action is still running, and charges one point for a discarded Restore as for one that runs. The documented observation does not expose the busy flag. Both reference defenders read it in their own wrappers, and switching that check off at evaluation costs KEEP's released policy 99.58 points per episode and H-MARL's 145.98, in both cases the charge for Restores sent to busy agents, with H-MARL's executed actions unchanged step for step. On a supervisory harness, holding interventions until the agent is idle is worth 25.3 and 27.6 points per episode at the published setting. In training, the check decides how a busy step enters the trajectory. Trained without it, KEEP leads at a hundred updates in all four paired seeds and trails from two hundred on in all four. Continued to two thousand updates on three seeds, the repaired recipe ends inside the released interval on one seed and short on two, by 27 and 125 points, a spread across training seeds far wider than the released reproduction interval. Second, the evaluation seeds the environment but not the policy, so a reported mean is one draw, and 88.9% of per-episode outcome variance is the policy's own sampling draw. Third, recovery precision depends on the truth it is scored against. Across five published comparison tables, none states the artifact, sampling, criterion and adversary that would make its numbers comparable. We release the instrument, our patches to both defenders, the traces and checkpoints behind every number, and a five-item disclosure checklist.

Quoted from arXiv:2611.14068. The research and its words belong to its authors.

Desk score **69** out of 100

Why it was picked

- Evals (in the title)
- Security (in the title)
- Agents
- Multi-agent
- Cross-listed in 2 fields
- Code or data released

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Security](https://thesimind.si/papers/?tag=security)
- [Multi-agent systems](https://thesimind.si/papers/?tag=multi-agent)
- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Agents](https://thesimind.si/papers/?tag=agents)

**arXiv ID:** 2611.14068v1
**Submitted:** November 10, 2026
**Fields:** cs.CR (Cryptography and Security), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** 47 pages. Code at [https://github.com/jrice-ai/cage](https://github.com/jrice-ai/cage) (tag arxiv-v1), data at [https://doi.org/10.5281/zenodo.23681828](https://doi.org/10.5281/zenodo.23681828)

Fingerprint, SHA-256

`850d464fed612af9d2318957c60ed12e5d73127c7a2f87dfd19bfe4a2d202c46`

On Solana: [Proof of Mind seal, 5PD2kk...Cqj6](https://solscan.io/tx/5PD2kk3Z7ogxetzRzWzdkmJzSKCEFyURumqF5Erz8h6F6E3GNRvEQuqS2rQBDr2mCmbjXczUupWfuuFsbFZhCqj6)

### The exact text fingerprinted

{"arxiv":"2611.14068","version":1,"title":"What a CAGE 4 Score Measures: An Audit of a Learned Cyber-Defense Benchmark, Its Reference Defenders, and the Tables Built on Them","authors":["James Rice","Ji Hwan Park","Andres Kwasinski"],"day":"2026-11-13","rank":10,"score":69,"status":"written","summary":{"takeaway":"The authors audit a cyber-defense benchmark for SI agents and find its scores depend on a hidden simulator rule, on luck in policy sampling, and on unstated scoring choices.","found":"The authors rebuilt the two reference defenders for CAGE Challenge 4, KEEP and H-MARL Expert, in a fixed container. They found the simulator silently drops any action given to an agent that is still busy, yet still charges a point for a dropped Restore. The documented observation does not show the busy flag, but both defenders read it in their own wrappers. Switching that check off at evaluation costs KEEP 99.58 points per episode and H-MARL 145.98. The authors also report that 88.9% of per-episode outcome variance comes from the policy's own sampling draw, because evaluation seeds the environment but not the policy. In training, KEEP's repaired recipe ended inside the released interval on one seed and short on two, by 27 and 125 points. They say none of five published comparison tables states enough to make its numbers comparable.","matters":"Benchmarks guide which SI methods people trust and build on. If a score depends on a hidden rule, a single lucky sampling draw, or an unstated scoring choice, then ranking methods by it can mislead. The authors release their instrument, patches, traces, checkpoints and a five-item disclosure checklist so others can report results more comparably. This is an audit of one benchmark, based on the abstract, so how far it applies to other benchmarks is not stated.","race":"This does not make SI defenders more capable. It may make progress claims in learned cyber defense harder to inflate and easier to check, which favors teams that report results carefully. Comparisons on CAGE 4 may need to be redone.","safety":"The paper bears on security and on trusting evaluations of SI cyber defenders. Scores that hide simulator quirks or sampling luck can overstate how well an SI defender would protect real systems.","level":"intermediate","tags":["evals","benchmarks","security","multi-agent","reinforcement-learning","agents"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

James Rice, Ji Hwan Park, Andres Kwasinski

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 16, 2026 · Paper of the DayOnly as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-AgentWhen a strong SI model hands work to a cheaper sub-agent, the whole system can be only as safe as the weaker agent, even if the strong model alone resists the attack.SafetyMulti-agent systemsSecurity](https://thesimind.si/papers/2611.17520/) · [Nov 20, 2026 · Number 2AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM AgentsAutoPI is a red teaming tool that tests SI agents for prompt injection across many attack types, not just how often attacks work, and it reports higher success and coverage than recent rivals.SecurityAgentsEvals](https://thesimind.si/papers/2611.24313/) · [Nov 10, 2026 · Number 2Tail-risk-aware Agent Safety: Constraining Tail Risks Across Multi-step LLM Agent WorkflowsThe authors propose TAS, a way to keep SI agents safe by tracking how risk builds across workflow steps and capping rare, severe failures with one shared budget.SafetyAgentsSecurity](https://thesimind.si/papers/2611.10411/)

[Number 9](https://thesimind.si/papers/2611.13850/) · [All 12 for Nov 13, 2026](https://thesimind.si/papers/day/2026-11-13/) · [Number 11](https://thesimind.si/papers/2611.14880/)
