Paper of the Day · November 17, 2026

Agent-as-a-Judge

Agent-as-a-Judge is a survey of SI evaluators that actively gather evidence before giving a verdict, and it finds that their gains over plain judges are uneven.

What they found

The authors review a growing line of work on judges that do more than read a fixed answer and score it. These "agentic" judges can check things before deciding, such as running tools, looking up facts, or inspecting what an SI agent did in its environment. The paper sorts this work by four design choices: multi-agent teamwork, tool use, memory, and planning. The authors also gather published scores and reproduce missing entries for key agentic judges. They say the comparison shows uneven gains from the agentic approach, and that this supports building dedicated tests for agentic evaluation. The abstract does not give specific numbers.

Why it matters

SI systems are now compared, deployed, and trained using feedback from other SI models acting as judges. A judge that only sees a static log can miss facts that were never recorded or claims that need outside checking. This survey gives a map of how to build judges that collect that missing evidence. It also names open problems in cost, reliability, safety, and governance.

How it changes the race

If the findings hold, better judges could give builders more trustworthy feedback for evaluating and post-training agents. The uneven gains suggest this is not a free upgrade, and teams will need to test when an agentic judge is actually worth its extra cost.

Safety and control

The authors list safety and governance as open challenges for deploying agentic judges at scale. Judges that use tools and act in environments bring their own risks, and their verdicts need to be reliable before people lean on them.

The abstract, by the authors

Artificial intelligence has progressed from task-specific systems toward general models spanning perception, generation, reasoning, and autonomous interaction, making reliable evaluation essential for comparison, deployment, and post-training feedback. At scale, modern evaluation pipelines increasingly rely on LLM-as-a-Judge to produce scores, preferences, and critiques; in its fixed-context setting, the judge relies on the supplied evaluand and evaluation materials. Yet a reliable verdict may require evidence beyond the supplied evaluand and context, either because critical environmental states remain unrecorded in the static execution log, or because controversial factual claims require external verification. Passive judges cannot obtain that missing evidence before deciding, especially when evaluating agent trajectories, executable artifacts, or domain-specific reports. Recent work therefore develops active evaluators that seek missing evidence and update their grounds for judgment before deciding. We bring these advances together through Agent-as-a-Judge, focusing on how evaluators construct and control checking processes within the broader LLM-based judging landscape. We define the scope through evaluation-directed checks and the use of their results, covering both structured workflows and observation-driven control. We organize the design space by multi-agent collaboration, tool integration, memory, and planning, and show how these mechanisms supply the checks and evidence on which a verdict rests. We consolidate published scores and reproduce missing entries for key agentic judges; the comparison reveals uneven agentic gains and motivates dedicated agentic evaluation. We conclude by characterizing application demands and open challenges in cost, reliability, safety, and governance that condition trusted deployment of agentic judges at scale.

Quoted from arXiv:2611.18067. The research and its words belong to its authors.

The authors

Runyang You, Hongru Cai, Caiqi Zhang, Qiancheng Xu, Meng Liu, Tiezheng Yu, Yongqi Li, Wenjie Li

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.