Number 4 of 12 · November 14, 2026

Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments

A local test kit for small coding agents shows how noisy single scores are: several apparent gains did not hold up when tested again on new tasks.

What they found

The author, Nguyen Minh Thang, built drydock-eval, a local kit for testing a small coding agent (Gemma 4 E4B). It runs about 250 runs per hour on one RTX 5090. Each job runs in a fresh container with no network, and the container passed 200 scripted escape tests. In a pre-registered experiment (231 tasks, 4,620 jobs), four prompt differences caused no detectable change in E4B solve rate (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three agent changes led their screens by +1.0 to +6.2 points. Each was retested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, rerun on the same 128 tasks, moved by 1.17 points, and 65% of those tasks were solved in none of eight runs. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, with no pairwise difference detected.

Why it matters

A single hosted score is a noisy number. This paper shows how easy it is to mistake noise for progress when tuning a small coding agent. The kit adds checks that compare a local setup with the scorer's configuration and statistics that account for tasks grouped by repository. The author also releases 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, so others can check the results. Prompts and agent bundles are withheld.

How it changes the race

This is a practical tooling and method paper, not a capability jump. If the results hold, teams building agents may need repeat runs and fresh-task retests before they trust small gains. Cheap local evaluation on one consumer GPU may also let small teams do this work.

Safety and control

The kit runs agent-written code in containers with no network, and it passed 200 scripted escape tests. The abstract says SI agents built and attacked the kit, and a human owner approved machine use and release. It does not say the containment is proven secure beyond those tests.

The abstract, by the authors

A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.

Quoted from arXiv:2611.16485. The research and its words belong to its authors.

The authors

Nguyen Minh Thang

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.