Number 3 of 12 · November 16, 2026

More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs

When multi-agent SI systems are compared with fair baselines that use the same number of samples or the same compute, they usually do no better and often do slightly worse.

What they found

The author pulls together nine earlier empirical studies, covering 516 multi-agent setups. Many of those studies compared agents that talk to each other with a single greedy pass, which is one quick answer from one model. That mixes up three things: coordination, the number of samples, and the amount of compute. Against a single-pass baseline, multi-agent accuracy differences are negative overall. They are near zero if the largest study is left out. Against baselines matched on sample count or resource budget, multi-agent systems average -0.042 in accuracy. They equal or fall below the baseline in 74% of these comparable contrasts. The author argues that communication adds no new information when agents share the same evidence. Gains need split-up evidence, outside verification, or varied errors across different models.

Why it matters

Many claims that teams of SI agents beat one model may come from unfair comparisons. Part of the apparent gain may just come from drawing more samples or spending more compute. This paper gives builders and reviewers a way to check this. It offers a reviewer checklist and accounting tools to track resources. The abstract does not say how the nine studies were chosen, so the findings rest on that set.

How it changes the race

If the result holds, teams may get more from spending compute on sampling or longer thinking by a single model than from complex multi-agent designs. Multi-agent setups would then be worth the cost only when agents hold different evidence, use external checks, or come from different models.

Safety and control

The paper does not address safety directly. It does suggest that claims about agent teams need fairer testing before people trust them.

The abstract, by the authors

Evaluations of multi-agent language models often compare communicating architectures against a single greedy pass. This practice confounds coordination with sample multiplicity and inference compute. We synthesize nine primary empirical evaluations comprising 516 unique multi-agent configurations organized along a resource vector R with six dimensions and a baseline ladder with four categories. Against baselines with a single pass (Category A), multi-agent accuracy differences are negative overall and near zero when omitting the largest contributing study. When evaluated against baselines matched on sample count (Category B) or resource budget (Category C), multi-agent systems average -0.042 in accuracy, and multilevel random effects modeling on the log odds scale remains negative across sensitivity checks. Multi-agent accuracy equals or falls below the baseline in 74% of these comparable contrasts, and study means are nonpositive in almost all evaluated studies. Across studies reporting both baselines, moving from a greedy pass to a sampled baseline shifts accuracy differences by -0.038 on average. Because communication adds no information when agents share identical evidence, positive returns require partitioned evidence, external verification, or error diversity across models. We provide a reviewer checklist and accounting utilities.

Quoted from arXiv:2611.17610. The research and its words belong to its authors.

The authors

Seyed Mahmoud Sajjadi Mohammadabadi

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.