---
title: "GRACE: Parallel Best-of-N Multi-Agent... | SI Paper Desk"
url: https://thesimind.si/papers/2611.20983/
description: "GRACE runs several SI agents in parallel on spreadsheet-style questions, judges their executed work instead of voting, and can reject all answers and retry..."
---

Paper of the Day · November 18, 2026

# GRACE: Parallel Best-of-N Multi-Agent Analysis of Closed-Form Questions over Dataframes with Guardrail Selection and Adaptive Retry

GRACE runs several SI agents in parallel on spreadsheet-style questions, judges their executed work instead of voting, and can reject all answers and retry, lifting answer match from 39% to 71% on one benchmark.

Fabio Picariello, Salvatore Cipolla, Susanna Bonura and 1 more · Submitted November 16, 2026 · arXiv 2611.20983v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.20983) · [The PDF](https://arxiv.org/pdf/2611.20983) · [The day's 12](https://thesimind.si/papers/day/2026-11-18/)

## What they found

The authors built GRACE, a pipeline where N SI agents each write and run Python code to answer a closed-form question about a table of data. Instead of voting on the final answers, a pairwise tournament trims the pool. Then a guardrail picks the best candidate or rejects all of them and gives a diagnosis, which triggers a retry with feedback for each agent. On InfiAgent-DABench (DAEval, M = 246) with Qwen3.6-35B-A3B, answer match rose from 39% for a single attempt to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. With the smaller Qwen3.5-9B it rose from 38% to 63% (+25 pp, 1.7x). A 3 x 3 study of the number of agents against the retry budget found the two largely interchangeable on the 35B model, so a cheap setting works. On DataBench, across 400 sampled questions, answer match rose from 70% to 85%.

## Why it matters

Many SI agents that analyse data run once and accept whatever comes out. This paper shows that comparing several attempts, and being allowed to say "none of these is good enough," can help a lot on these benchmarks. The finding that more agents and more retries can substitute for each other gives practical guidance for deployment. The abstract reports results on two benchmarks and two model sizes, but does not say how the method fares on other kinds of tasks.

## How it changes the race

This is mostly an engineering gain for systems built on top of existing models, not a new model capability. It suggests that careful selection and retry around a mid-sized model can close a good part of the gap on data analysis questions, at a cost the authors say can be kept low.

## Safety and control

The ability to reject every candidate and abstain with a diagnosis, rather than always returning an answer, is a useful reliability feature. The abstract does not test it against misuse or safety risks.

### The abstract, by the authors

LLM agents answer closed-form questions over a tabular dataframe by writing and executing Python, almost always as a single rollout: one interpretation, one script, one result, accepted without comparison. The few pipelines that produce several candidates reconcile them by answer-agreement voting, which needs a single comparable scalar and always returns one, even when every candidate is wrong. We present GRACE (Generative, Ranked, Adaptive Code Ensemble), a production-oriented pipeline that instead runs parallel best-of-N selection specialised to dataframe code. N agents run concurrently under a strict, machine-checkable output contract, diversified by persona hints, per-retry rotation, and a two-regime temperature schedule. Rather than vote, GRACE judges the executed artefacts: a pairwise tournament pre-filters the pool and a domain-specialised guardrail either selects the best candidate or, when none is defensible, rejects the whole pool and abstains with a diagnosis (an option voting structurally lacks), which in turn drives an adaptive retry with per-agent feedback. On InfiAgent-DABench (DAEval, M = 246), using Qwen3.6-35B-A3B as the backbone, GRACE raises answer match from 39% for a single-rollout baseline to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. On the same DAEval task, the smaller dense Qwen3.5-9B lifts answer match from 38% to 63% (+25 pp, 1.7x). A controlled 3 x 3 study that sweeps the number of parallel agents against the retry budget finds, on the 35B backbone, the two levers largely substitutable: once either one is generous, the other adds little, identifying a cheap operating point for deployment. The same directional pattern replicates on the independent DataBench benchmark, again on the Qwen3.6-35B-A3B backbone, where answer match rises by 15 percentage points (from 70% to 85%) across 400 sampled questions.

Quoted from arXiv:2611.20983. The research and its words belong to its authors.

Desk score **86** out of 100

Why it was picked

- Safety (in the title)
- Agents (in the title)
- Evals
- Multi-agent (in the title)
- Studies frontier models (Qwen3.6)
- Accepted at Science
- Language models

Topics

- [Agents](https://thesimind.si/papers/?tag=agents)
- [Multi-agent systems](https://thesimind.si/papers/?tag=multi-agent)
- [Code](https://thesimind.si/papers/?tag=code)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)
- [Language models](https://thesimind.si/papers/?tag=language-models)

**arXiv ID:** 2611.20983v1
**Submitted:** November 16, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** Accepted at the 5th Italian Conference on Big Data and Data Science (ITADATA 2026). To appear in CEUR Workshop Proceedings

Fingerprint, SHA-256

`5f8dba7b54169041eff084e97c45ff52067c710d48d13561593809c8e02c73ff`

On Solana: [Proof of Mind seal, 5WzZUk...UPMg](https://solscan.io/tx/5WzZUksJeCL3fYgfzcQ6DSdVxipu97PAmY5Y9M59gCMXbJtnyBVsqaoPSDHWrP4AiZAh7k4iVso5ts7WvxdNUPMg)

### The exact text fingerprinted

{"arxiv":"2611.20983","version":1,"title":"GRACE: Parallel Best-of-N Multi-Agent Analysis of Closed-Form Questions over Dataframes with Guardrail Selection and Adaptive Retry","authors":["Fabio Picariello","Salvatore Cipolla","Susanna Bonura","Vito Morreale"],"day":"2026-11-18","rank":1,"score":86,"status":"written","summary":{"takeaway":"GRACE runs several SI agents in parallel on spreadsheet-style questions, judges their executed work instead of voting, and can reject all answers and retry, lifting answer match from 39% to 71% on one benchmark.","found":"The authors built GRACE, a pipeline where N SI agents each write and run Python code to answer a closed-form question about a table of data. Instead of voting on the final answers, a pairwise tournament trims the pool. Then a guardrail picks the best candidate or rejects all of them and gives a diagnosis, which triggers a retry with feedback for each agent. On InfiAgent-DABench (DAEval, M = 246) with Qwen3.6-35B-A3B, answer match rose from 39% for a single attempt to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. With the smaller Qwen3.5-9B it rose from 38% to 63% (+25 pp, 1.7x). A 3 x 3 study of the number of agents against the retry budget found the two largely interchangeable on the 35B model, so a cheap setting works. On DataBench, across 400 sampled questions, answer match rose from 70% to 85%.","matters":"Many SI agents that analyse data run once and accept whatever comes out. This paper shows that comparing several attempts, and being allowed to say \"none of these is good enough,\" can help a lot on these benchmarks. The finding that more agents and more retries can substitute for each other gives practical guidance for deployment. The abstract reports results on two benchmarks and two model sizes, but does not say how the method fares on other kinds of tasks.","race":"This is mostly an engineering gain for systems built on top of existing models, not a new model capability. It suggests that careful selection and retry around a mid-sized model can close a good part of the gap on data analysis questions, at a cost the authors say can be kept low.","safety":"The ability to reject every candidate and abstain with a diagnosis, rather than always returning an answer, is a useful reliability feature. The abstract does not test it against misuse or safety risks.","level":"intermediate","tags":["agents","multi-agent","code","evals","efficiency","language-models"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Fabio Picariello, Salvatore Cipolla, Susanna Bonura, Vito Morreale

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 4Correct First, Fast Second: What a Local 7B Code Model Can and Cannot Do for GPU Kernel OptimizationWith a small local 7B code model writing GPU kernels, getting code that works matters more than speed, and extra agent structure only helps when matched to what the model can do.CodeAgentsEfficiency](https://thesimind.si/papers/2611.12079/) · [Nov 15, 2026 · Paper of the DayLook Here, Then Decide: Small Decision Models in a Harness Find Memory-Safety Vulnerabilities at Scale on Commodity GPUsA small 4B-parameter open model is weak at judging code alone, but it ranks candidate functions well enough that a harness on cheap GPUs confirmed 187 memory-safety findings.SecurityCodeEfficiency](https://thesimind.si/papers/2611.16565/) · [Nov 12, 2026 · Number 2AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSLOn a new benchmark for a private Salesforce agent language, SI models mostly fail zero-shot, and the setup around the model, like a compile-and-edit loop and clear guidance, mattered more than model size.BenchmarksCodeAgents](https://thesimind.si/papers/2611.13183/)

[All 12 for Nov 18, 2026](https://thesimind.si/papers/day/2026-11-18/) · [Number 2](https://thesimind.si/papers/2611.20771/)
