What they found
The authors built GRACE, a pipeline where N SI agents each write and run Python code to answer a closed-form question about a table of data. Instead of voting on the final answers, a pairwise tournament trims the pool. Then a guardrail picks the best candidate or rejects all of them and gives a diagnosis, which triggers a retry with feedback for each agent. On InfiAgent-DABench (DAEval, M = 246) with Qwen3.6-35B-A3B, answer match rose from 39% for a single attempt to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. With the smaller Qwen3.5-9B it rose from 38% to 63% (+25 pp, 1.7x). A 3 x 3 study of the number of agents against the retry budget found the two largely interchangeable on the 35B model, so a cheap setting works. On DataBench, across 400 sampled questions, answer match rose from 70% to 85%.
Why it matters
Many SI agents that analyse data run once and accept whatever comes out. This paper shows that comparing several attempts, and being allowed to say "none of these is good enough," can help a lot on these benchmarks. The finding that more agents and more retries can substitute for each other gives practical guidance for deployment. The abstract reports results on two benchmarks and two model sizes, but does not say how the method fares on other kinds of tasks.
How it changes the race
This is mostly an engineering gain for systems built on top of existing models, not a new model capability. It suggests that careful selection and retry around a mid-sized model can close a good part of the gap on data analysis questions, at a cost the authors say can be kept low.
Safety and control
The ability to reject every candidate and abstain with a diagnosis, rather than always returning an answer, is a useful reliability feature. The abstract does not test it against misuse or safety risks.
The abstract, by the authors
LLM agents answer closed-form questions over a tabular dataframe by writing and executing Python, almost always as a single rollout: one interpretation, one script, one result, accepted without comparison. The few pipelines that produce several candidates reconcile them by answer-agreement voting, which needs a single comparable scalar and always returns one, even when every candidate is wrong. We present GRACE (Generative, Ranked, Adaptive Code Ensemble), a production-oriented pipeline that instead runs parallel best-of-N selection specialised to dataframe code. N agents run concurrently under a strict, machine-checkable output contract, diversified by persona hints, per-retry rotation, and a two-regime temperature schedule. Rather than vote, GRACE judges the executed artefacts: a pairwise tournament pre-filters the pool and a domain-specialised guardrail either selects the best candidate or, when none is defensible, rejects the whole pool and abstains with a diagnosis (an option voting structurally lacks), which in turn drives an adaptive retry with per-agent feedback. On InfiAgent-DABench (DAEval, M = 246), using Qwen3.6-35B-A3B as the backbone, GRACE raises answer match from 39% for a single-rollout baseline to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. On the same DAEval task, the smaller dense Qwen3.5-9B lifts answer match from 38% to 63% (+25 pp, 1.7x). A controlled 3 x 3 study that sweeps the number of parallel agents against the retry budget finds, on the 35B backbone, the two levers largely substitutable: once either one is generous, the other adds little, identifying a cheap operating point for deployment. The same directional pattern replicates on the independent DataBench benchmark, again on the Qwen3.6-35B-A3B backbone, where answer match rises by 15 percentage points (from 70% to 85%) across 400 sampled questions.
Quoted from arXiv:2611.20983. The research and its words belong to its authors.
