What they found
The authors rebuilt the two reference defenders for CAGE Challenge 4, KEEP and H-MARL Expert, in a fixed container. They found the simulator silently drops any action given to an agent that is still busy, yet still charges a point for a dropped Restore. The documented observation does not show the busy flag, but both defenders read it in their own wrappers. Switching that check off at evaluation costs KEEP 99.58 points per episode and H-MARL 145.98. The authors also report that 88.9% of per-episode outcome variance comes from the policy's own sampling draw, because evaluation seeds the environment but not the policy. In training, KEEP's repaired recipe ended inside the released interval on one seed and short on two, by 27 and 125 points. They say none of five published comparison tables states enough to make its numbers comparable.
Why it matters
Benchmarks guide which SI methods people trust and build on. If a score depends on a hidden rule, a single lucky sampling draw, or an unstated scoring choice, then ranking methods by it can mislead. The authors release their instrument, patches, traces, checkpoints and a five-item disclosure checklist so others can report results more comparably. This is an audit of one benchmark, based on the abstract, so how far it applies to other benchmarks is not stated.
How it changes the race
This does not make SI defenders more capable. It may make progress claims in learned cyber defense harder to inflate and easier to check, which favors teams that report results carefully. Comparisons on CAGE 4 may need to be redone.
Safety and control
The paper bears on security and on trusting evaluations of SI cyber defenders. Scores that hide simulator quirks or sampling luck can overstate how well an SI defender would protect real systems.
The abstract, by the authors
CAGE Challenge 4 is the benchmark on which learned multi-agent cyber defenders are compared. We reproduce its two reference defenders, KEEP and H-MARL Expert, in a pinned container and show that three things read from it measure less than they are taken to measure. First, the simulator silently discards any action asserted for an agent whose previous action is still running, and charges one point for a discarded Restore as for one that runs. The documented observation does not expose the busy flag. Both reference defenders read it in their own wrappers, and switching that check off at evaluation costs KEEP's released policy 99.58 points per episode and H-MARL's 145.98, in both cases the charge for Restores sent to busy agents, with H-MARL's executed actions unchanged step for step. On a supervisory harness, holding interventions until the agent is idle is worth 25.3 and 27.6 points per episode at the published setting. In training, the check decides how a busy step enters the trajectory. Trained without it, KEEP leads at a hundred updates in all four paired seeds and trails from two hundred on in all four. Continued to two thousand updates on three seeds, the repaired recipe ends inside the released interval on one seed and short on two, by 27 and 125 points, a spread across training seeds far wider than the released reproduction interval. Second, the evaluation seeds the environment but not the policy, so a reported mean is one draw, and 88.9% of per-episode outcome variance is the policy's own sampling draw. Third, recovery precision depends on the truth it is scored against. Across five published comparison tables, none states the artifact, sampling, criterion and adversary that would make its numbers comparable. We release the instrument, our patches to both defenders, the traces and checkpoints behind every number, and a five-item disclosure checklist.
Quoted from arXiv:2611.14068. The research and its words belong to its authors.
