---
title: "Auditing AI peer reviewers: a benchmark... | SI Paper Desk"
url: https://thesimind.si/papers/2611.17052/
description: "The authors built a test that plants known errors in real papers. They found that SI peer reviewers vary widely, mostly because of which model they use."
---

Number 8 of 12 · November 14, 2026

# Auditing AI peer reviewers: a benchmark on real scientific papers

The authors built a test that plants known errors in real papers. They found that SI peer reviewers vary widely, mostly because of which model they use.

Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro and 2 more · Submitted November 12, 2026 · arXiv 2611.17052v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.17052) · [The PDF](https://arxiv.org/pdf/2611.17052) · [The day's 12](https://thesimind.si/papers/day/2026-11-14/)

## What they found

The authors inject objective errors into real papers. The errors come in four types: general issues, mathematics, numerics and citations. Each has one of three severity levels. They measure how often each reviewer finds the errors, and how often it wrongly flags clean copies. They tested twelve reviewer systems on two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning, with 357 injected errors in total. Model choice mattered most: with the same prompt, three frontier LLMs differed by about 50 percentage points in total detection on both corpora. Tools helped when they supplied something the model could not. On astrophysics, Claude Code did better than a single Claude call on mathematics and numerics. The authors' own multi-agent reviewer, Skepthical, retrieves cited papers, checks mathematics with computer algebra and runs code for numerical claims. It had the highest citation error detection by a large margin on astrophysics. That lead did not appear on machine learning, where it was statistically indistinguishable from the best single LLM. False-positive rates were below 10% for most systems, and mathematics flags were the main source.

## Why it matters

LLM-based peer review is already used at production scale, but the authors say its evaluation is limited. This benchmark gives a concrete way to test what a reviewer catches and what it misses. It also suggests that extra tools pay off mainly where the model's own knowledge falls short. The abstract covers only two fields, so it does not show how far the results carry to others.

## How it changes the race

This is an evaluation advance, not a capability one. It gives builders of SI reviewers a shared test and a hint on where to spend effort: choose the model first, then add tools for the gaps.

## Safety and control

The benchmark helps audit SI systems that are used to judge scientific work. The large differences it reports between reviewers show why such systems need testing before people rely on them.

### The abstract, by the authors

LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.

Quoted from arXiv:2611.17052. The research and its words belong to its authors.

Desk score **74** out of 100

Why it was picked

- Evals (in the title)
- Agents
- Cross-listed in 4 fields
- Studies frontier models (Claude)
- Accepted at NeurIPS
- Multi-agent
- Language models

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Science](https://thesimind.si/papers/?tag=science)
- [Retrieval](https://thesimind.si/papers/?tag=retrieval)

**arXiv ID:** 2611.17052v1
**Submitted:** November 12, 2026
**Fields:** cs.CL (Computation and Language), astro-ph.IM, cs.AI (Artificial Intelligence), cs.DL
**Level:** Intermediate
**Comments:** Accepted at NeurIPS 2026

Fingerprint, SHA-256

`86a68f838b41a515cd2aaf493ebbcc9d2d8fc68bcea2c9b14363990e5fd2d257`

On Solana: [Proof of Mind seal, 4ut3vv...Y9uP](https://solscan.io/tx/4ut3vv2vTeUs5fAbknPuGnt1i1jJGh18Ry7QYkUAkvvDWCKQk8wrmGNkp4C5Loc937KnsizybKH71BB2w6QAY9uP)

### The exact text fingerprinted

{"arxiv":"2611.17052","version":1,"title":"Auditing AI peer reviewers: a benchmark on real scientific papers","authors":["Íñigo Zubeldia","Boris Bolliet","Francisco Villaescusa-Navarro","Pablo Villanueva-Domingo","Miles Cranmer"],"day":"2026-11-14","rank":8,"score":74,"status":"written","summary":{"takeaway":"The authors built a test that plants known errors in real papers. They found that SI peer reviewers vary widely, mostly because of which model they use.","found":"The authors inject objective errors into real papers. The errors come in four types: general issues, mathematics, numerics and citations. Each has one of three severity levels. They measure how often each reviewer finds the errors, and how often it wrongly flags clean copies. They tested twelve reviewer systems on two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning, with 357 injected errors in total. Model choice mattered most: with the same prompt, three frontier LLMs differed by about 50 percentage points in total detection on both corpora. Tools helped when they supplied something the model could not. On astrophysics, Claude Code did better than a single Claude call on mathematics and numerics. The authors' own multi-agent reviewer, Skepthical, retrieves cited papers, checks mathematics with computer algebra and runs code for numerical claims. It had the highest citation error detection by a large margin on astrophysics. That lead did not appear on machine learning, where it was statistically indistinguishable from the best single LLM. False-positive rates were below 10% for most systems, and mathematics flags were the main source.","matters":"LLM-based peer review is already used at production scale, but the authors say its evaluation is limited. This benchmark gives a concrete way to test what a reviewer catches and what it misses. It also suggests that extra tools pay off mainly where the model's own knowledge falls short. The abstract covers only two fields, so it does not show how far the results carry to others.","race":"This is an evaluation advance, not a capability one. It gives builders of SI reviewers a shared test and a hint on where to spend effort: choose the model first, then add tools for the gaps.","safety":"The benchmark helps audit SI systems that are used to judge scientific work. The large differences it reports between reviewers show why such systems need testing before people rely on them.","level":"intermediate","tags":["evals","benchmarks","agents","language-models","science","retrieval"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro, Pablo Villanueva-Domingo, Miles Cranmer

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/) · [Nov 21, 2026 · Number 3TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real LabsTwinLab-Bench tests whether SI agents can design their own lab experiments in simulated twins of real labs, and the best model scores only 45.0.BenchmarksAgentsScience](https://thesimind.si/papers/2611.26248/) · [Nov 17, 2026 · Number 7JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification ExamsJFinBench is a Japanese finance test built from the 2026 certification exams, made so it can be refreshed each year to limit leaked test questions.BenchmarksEvalsLanguage models](https://thesimind.si/papers/2611.18130/)

[Number 7](https://thesimind.si/papers/2611.16767/) · [All 12 for Nov 14, 2026](https://thesimind.si/papers/day/2026-11-14/) · [Number 9](https://thesimind.si/papers/2611.16879/)
