---
title: "Contain, Conform, Compare (Extended... | SI Paper Desk"
url: https://thesimind.si/papers/2611.16485/
description: "A local test kit for small coding agents shows how noisy single scores are: several apparent gains did not hold up when tested again on new tasks."
---

Number 4 of 12 · November 14, 2026

# Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments

A local test kit for small coding agents shows how noisy single scores are: several apparent gains did not hold up when tested again on new tasks.

Nguyen Minh Thang · Submitted November 12, 2026 · arXiv 2611.16485v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.16485) · [The PDF](https://arxiv.org/pdf/2611.16485) · [The day's 12](https://thesimind.si/papers/day/2026-11-14/)

## What they found

The author, Nguyen Minh Thang, built drydock-eval, a local kit for testing a small coding agent (Gemma 4 E4B). It runs about 250 runs per hour on one RTX 5090. Each job runs in a fresh container with no network, and the container passed 200 scripted escape tests. In a pre-registered experiment (231 tasks, 4,620 jobs), four prompt differences caused no detectable change in E4B solve rate (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three agent changes led their screens by +1.0 to +6.2 points. Each was retested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, rerun on the same 128 tasks, moved by 1.17 points, and 65% of those tasks were solved in none of eight runs. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, with no pairwise difference detected.

## Why it matters

A single hosted score is a noisy number. This paper shows how easy it is to mistake noise for progress when tuning a small coding agent. The kit adds checks that compare a local setup with the scorer's configuration and statistics that account for tasks grouped by repository. The author also releases 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, so others can check the results. Prompts and agent bundles are withheld.

## How it changes the race

This is a practical tooling and method paper, not a capability jump. If the results hold, teams building agents may need repeat runs and fresh-task retests before they trust small gains. Cheap local evaluation on one consumer GPU may also let small teams do this work.

## Safety and control

The kit runs agent-written code in containers with no network, and it passed 200 scripted escape tests. The abstract says SI agents built and attacked the kit, and a human owner approved machine use and release. It does not say the containment is proven secure beyond those tests.

### The abstract, by the authors

A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.

Quoted from arXiv:2611.16485. The research and its words belong to its authors.

Desk score **83** out of 100

Why it was picked

- Safety (in the title)
- Agents (in the title)
- Evals (in the title)
- Compute
- Code or data released

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Code](https://thesimind.si/papers/?tag=code)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Security](https://thesimind.si/papers/?tag=security)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)

**arXiv ID:** 2611.16485v1
**Submitted:** November 12, 2026
**Fields:** cs.AI (Artificial Intelligence)
**Level:** Intermediate
**Comments:** 52 pages, 15 figures, 37 tables. Code: [https://github.com/MinhThanghus/drydock-eval](https://github.com/MinhThanghus/drydock-eval) (tag v0.4.0). Data and analysis scripts are included as ancillary files

Fingerprint, SHA-256

`91eac1df92bbe02f6c48000b186e4da4011b8a010c9e52d0241ca6f0066bc0eb`

On Solana: [Proof of Mind seal, 4ut3vv...Y9uP](https://solscan.io/tx/4ut3vv2vTeUs5fAbknPuGnt1i1jJGh18Ry7QYkUAkvvDWCKQk8wrmGNkp4C5Loc937KnsizybKH71BB2w6QAY9uP)

### The exact text fingerprinted

{"arxiv":"2611.16485","version":1,"title":"Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments","authors":["Nguyen Minh Thang"],"day":"2026-11-14","rank":4,"score":83,"status":"written","summary":{"takeaway":"A local test kit for small coding agents shows how noisy single scores are: several apparent gains did not hold up when tested again on new tasks.","found":"The author, Nguyen Minh Thang, built drydock-eval, a local kit for testing a small coding agent (Gemma 4 E4B). It runs about 250 runs per hour on one RTX 5090. Each job runs in a fresh container with no network, and the container passed 200 scripted escape tests. In a pre-registered experiment (231 tasks, 4,620 jobs), four prompt differences caused no detectable change in E4B solve rate (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three agent changes led their screens by +1.0 to +6.2 points. Each was retested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, rerun on the same 128 tasks, moved by 1.17 points, and 65% of those tasks were solved in none of eight runs. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, with no pairwise difference detected.","matters":"A single hosted score is a noisy number. This paper shows how easy it is to mistake noise for progress when tuning a small coding agent. The kit adds checks that compare a local setup with the scorer's configuration and statistics that account for tasks grouped by repository. The author also releases 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, so others can check the results. Prompts and agent bundles are withheld.","race":"This is a practical tooling and method paper, not a capability jump. If the results hold, teams building agents may need repeat runs and fresh-task retests before they trust small gains. Cheap local evaluation on one consumer GPU may also let small teams do this work.","safety":"The kit runs agent-written code in containers with no network, and it passed 200 scripted escape tests. The abstract says SI agents built and attacked the kit, and a human owner approved machine use and release. It does not say the containment is proven secure beyond those tests.","level":"intermediate","tags":["evals","code","agents","benchmarks","security","efficiency"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Nguyen Minh Thang

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 12, 2026 · Paper of the DayComparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics AgentsIn a controlled test of three ways to check enterprise analytics agents, the method that sees the agent's full step-by-step trace caught failures best, and security red-teaming found different problems than functional testing.EvalsAgentsBenchmarks](https://thesimind.si/papers/2611.14493/) · [Nov 15, 2026 · Paper of the DayLook Here, Then Decide: Small Decision Models in a Harness Find Memory-Safety Vulnerabilities at Scale on Commodity GPUsA small 4B-parameter open model is weak at judging code alone, but it ranks candidate functions well enough that a harness on cheap GPUs confirmed 187 memory-safety findings.SecurityCodeEfficiency](https://thesimind.si/papers/2611.16565/) · [Nov 12, 2026 · Number 2AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSLOn a new benchmark for a private Salesforce agent language, SI models mostly fail zero-shot, and the setup around the model, like a compile-and-edit loop and clear guidance, mattered more than model size.BenchmarksCodeAgents](https://thesimind.si/papers/2611.13183/)

[Number 3](https://thesimind.si/papers/2611.16693/) · [All 12 for Nov 14, 2026](https://thesimind.si/papers/day/2026-11-14/) · [Number 5](https://thesimind.si/papers/2611.17128/)
