---
title: "Agent-as-a-Judge | SI Paper Desk"
url: https://thesimind.si/papers/2611.18067/
description: "Agent-as-a-Judge is a survey of SI evaluators that actively gather evidence before giving a verdict, and it finds that their gains over plain judges are uneven."
---

Paper of the Day · November 17, 2026

# Agent-as-a-Judge

Agent-as-a-Judge is a survey of SI evaluators that actively gather evidence before giving a verdict, and it finds that their gains over plain judges are uneven.

Runyang You, Hongru Cai, Caiqi Zhang and 5 more · Submitted November 13, 2026 · arXiv 2611.18067v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.18067) · [The PDF](https://arxiv.org/pdf/2611.18067) · [The day's 12](https://thesimind.si/papers/day/2026-11-17/)

## What they found

The authors review a growing line of work on judges that do more than read a fixed answer and score it. These "agentic" judges can check things before deciding, such as running tools, looking up facts, or inspecting what an SI agent did in its environment. The paper sorts this work by four design choices: multi-agent teamwork, tool use, memory, and planning. The authors also gather published scores and reproduce missing entries for key agentic judges. They say the comparison shows uneven gains from the agentic approach, and that this supports building dedicated tests for agentic evaluation. The abstract does not give specific numbers.

## Why it matters

SI systems are now compared, deployed, and trained using feedback from other SI models acting as judges. A judge that only sees a static log can miss facts that were never recorded or claims that need outside checking. This survey gives a map of how to build judges that collect that missing evidence. It also names open problems in cost, reliability, safety, and governance.

## How it changes the race

If the findings hold, better judges could give builders more trustworthy feedback for evaluating and post-training agents. The uneven gains suggest this is not a free upgrade, and teams will need to test when an agentic judge is actually worth its extra cost.

## Safety and control

The authors list safety and governance as open challenges for deploying agentic judges at scale. Judges that use tools and act in environments bring their own risks, and their verdicts need to be reliable before people lean on them.

### The abstract, by the authors

Artificial intelligence has progressed from task-specific systems toward general models spanning perception, generation, reasoning, and autonomous interaction, making reliable evaluation essential for comparison, deployment, and post-training feedback. At scale, modern evaluation pipelines increasingly rely on LLM-as-a-Judge to produce scores, preferences, and critiques; in its fixed-context setting, the judge relies on the supplied evaluand and evaluation materials. Yet a reliable verdict may require evidence beyond the supplied evaluand and context, either because critical environmental states remain unrecorded in the static execution log, or because controversial factual claims require external verification. Passive judges cannot obtain that missing evidence before deciding, especially when evaluating agent trajectories, executable artifacts, or domain-specific reports. Recent work therefore develops active evaluators that seek missing evidence and update their grounds for judgment before deciding. We bring these advances together through Agent-as-a-Judge, focusing on how evaluators construct and control checking processes within the broader LLM-based judging landscape. We define the scope through evaluation-directed checks and the use of their results, covering both structured workflows and observation-driven control. We organize the design space by multi-agent collaboration, tool integration, memory, and planning, and show how these mechanisms supply the checks and evidence on which a verdict rests. We consolidate published scores and reproduce missing entries for key agentic judges; the comparison reveals uneven agentic gains and motivates dedicated agentic evaluation. We conclude by characterizing application demands and open challenges in cost, reliability, safety, and governance that condition trusted deployment of agentic judges at scale.

Quoted from arXiv:2611.18067. The research and its words belong to its authors.

Desk score **82** out of 100

Why it was picked

- Agents (in the title)
- Safety
- Reasoning
- Evals
- Policy
- Multi-agent
- Language models

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Safety](https://thesimind.si/papers/?tag=safety)
- [Multi-agent systems](https://thesimind.si/papers/?tag=multi-agent)

**arXiv ID:** 2611.18067v1
**Submitted:** November 13, 2026
**Fields:** cs.CL (Computation and Language)
**Level:** Intermediate
**Comments:** Project page: [https://github.com/ModalityDance/Awesome-Agent-as-a-Judge](https://github.com/ModalityDance/Awesome-Agent-as-a-Judge)

Fingerprint, SHA-256

`7762eea2343bef972abfb8246ecd4adb4d18ff32a427a5941c5510f38ef2539a`

On Solana: [Proof of Mind seal, 2HNJWs...cNY9](https://solscan.io/tx/2HNJWsLa5Tp3F7wQaNr1CPR71N1aoGNptA7fgCafC5m3BwbmKwhNGTXyaSfyEa9jqug6kM55u5VmDvbSrRepcNY9)

### The exact text fingerprinted

{"arxiv":"2611.18067","version":1,"title":"Agent-as-a-Judge","authors":["Runyang You","Hongru Cai","Caiqi Zhang","Qiancheng Xu","Meng Liu","Tiezheng Yu","Yongqi Li","Wenjie Li"],"day":"2026-11-17","rank":1,"score":82,"status":"written","summary":{"takeaway":"Agent-as-a-Judge is a survey of SI evaluators that actively gather evidence before giving a verdict, and it finds that their gains over plain judges are uneven.","found":"The authors review a growing line of work on judges that do more than read a fixed answer and score it. These \"agentic\" judges can check things before deciding, such as running tools, looking up facts, or inspecting what an SI agent did in its environment. The paper sorts this work by four design choices: multi-agent teamwork, tool use, memory, and planning. The authors also gather published scores and reproduce missing entries for key agentic judges. They say the comparison shows uneven gains from the agentic approach, and that this supports building dedicated tests for agentic evaluation. The abstract does not give specific numbers.","matters":"SI systems are now compared, deployed, and trained using feedback from other SI models acting as judges. A judge that only sees a static log can miss facts that were never recorded or claims that need outside checking. This survey gives a map of how to build judges that collect that missing evidence. It also names open problems in cost, reliability, safety, and governance.","race":"If the findings hold, better judges could give builders more trustworthy feedback for evaluating and post-training agents. The uneven gains suggest this is not a free upgrade, and teams will need to test when an agentic judge is actually worth its extra cost.","safety":"The authors list safety and governance as open challenges for deploying agentic judges at scale. Judges that use tools and act in environments bring their own risks, and their verdicts need to be reliable before people lean on them.","level":"intermediate","tags":["evals","agents","language-models","benchmarks","safety","multi-agent"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Runyang You, Hongru Cai, Caiqi Zhang, Qiancheng Xu, Meng Liu, Tiezheng Yu, Yongqi Li, Wenjie Li

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 16, 2026 · Paper of the DayOnly as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-AgentWhen a strong SI model hands work to a cheaper sub-agent, the whole system can be only as safe as the weaker agent, even if the strong model alone resists the attack.SafetyMulti-agent systemsSecurity](https://thesimind.si/papers/2611.17520/) · [Nov 20, 2026 · Number 2AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM AgentsAutoPI is a red teaming tool that tests SI agents for prompt injection across many attack types, not just how often attacks work, and it reports higher success and coverage than recent rivals.SecurityAgentsEvals](https://thesimind.si/papers/2611.24313/) · [Nov 11, 2026 · Number 6Steering Emergent Behavior in Multi-Agent LLM Systems via Information NetworksIn groups of SI agents, showing agents what their peers did does not always help. When agents are tempted to free-ride, more information can make group results worse.Multi-agent systemsSafetyAgents](https://thesimind.si/papers/2611.11714/)

[All 12 for Nov 17, 2026](https://thesimind.si/papers/day/2026-11-17/) · [Number 2](https://thesimind.si/papers/2611.18397/)
