---
title: "SciResearch-Eval: Evaluating Scientific... | SI Paper Desk"
url: https://thesimind.si/papers/2611.21541/
description: "SciResearch-Eval tests whether SI agents can run lab-style experiments and report honest error ranges. When the simulator is missing physics, most models..."
---

Number 9 of 12 · November 18, 2026

# SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

SciResearch-Eval tests whether SI agents can run lab-style experiments and report honest error ranges. When the simulator is missing physics, most models state 95% confidence but are often wrong.

Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora · Submitted November 16, 2026 · arXiv 2611.21541v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.21541) · [The PDF](https://arxiv.org/pdf/2611.21541) · [The day's 12](https://thesimind.si/papers/day/2026-11-18/)

## What they found

The authors built an interactive test. An SI agent designs its own heating experiments, uses a simulator that may be flawed, and reports each material property with a 95% interval. The test covers 93 held-out tasks, 9 models and 2,511 episodes. Models handle procedural traps. When the simulator is missing physics, most report wrong values while claiming 95% confidence. Their ranges hold the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Asking for wider intervals does not help. Stating the strategy improves interval scores 1.6-2.5-fold, but coverage barely moves. At maximum effort, Opus improves more than tenfold, but it uses 13-31 times more reasoning. Older and open-weight models gain nothing.

## Why it matters

Autonomous SI agents are being pointed at real lab work. In that work, a confident wrong answer can be worse than an honest "I am not sure." This benchmark scores whether stated confidence matches the truth, using exact ground truth. It also shows that good results can come from smart experiment design, such as gentle heating, and not only from naming the cause of the problem. The abstract also shows a cost trade-off. Fable 5.2 is the most accurate at default effort, with an interval score 1.9 times better than Astra's. It writes 6 times as many tokens, and at Astra's token count it falls short.

## How it changes the race

The result suggests that judgment about when to trust a tool is a separate skill from raw ability. It may be latent in some models and only appear with much more reasoning. Labs building SI scientists may need to compare models at equal reasoning budgets, not only at default settings.

## Safety and control

The paper bears on trust in SI agents. Most models gave confident intervals that were wrong when the simulator was flawed, and asking for wider intervals did not fix this. Calibration checks like this one may be needed before agents are given autonomy in experiments.

### The abstract, by the authors

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.

Quoted from arXiv:2611.21541. The research and its words belong to its authors.

Desk score **77** out of 100

Why it was picked

- Agents (in the title)
- Evals (in the title)
- Reasoning
- Studies frontier models (Claude)
- Cross-listed in 3 fields
- Code or data released

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Science](https://thesimind.si/papers/?tag=science)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)
- [Safety](https://thesimind.si/papers/?tag=safety)

**arXiv ID:** 2611.21541v1
**Submitted:** November 16, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.LG (Machine Learning), physics.comp-ph
**Level:** Intermediate
**Comments:** 28 pages, 7 figures. Code and leaderboard: [https://github.com/Btwestyo/sciresearch-eval](https://github.com/Btwestyo/sciresearch-eval)

Fingerprint, SHA-256

`808c94123d749d15d1fa6f1765af2c03212d9d062a645319a34ba4c5270e03a2`

On Solana: [Proof of Mind seal, 5WzZUk...UPMg](https://solscan.io/tx/5WzZUksJeCL3fYgfzcQ6DSdVxipu97PAmY5Y9M59gCMXbJtnyBVsqaoPSDHWrP4AiZAh7k4iVso5ts7WvxdNUPMg)

### The exact text fingerprinted

{"arxiv":"2611.21541","version":1,"title":"SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents","authors":["Brian T. Weston","Alondra Torres Contreras","Aldair E. Gongora"],"day":"2026-11-18","rank":9,"score":77,"status":"written","summary":{"takeaway":"SciResearch-Eval tests whether SI agents can run lab-style experiments and report honest error ranges. When the simulator is missing physics, most models state 95% confidence but are often wrong.","found":"The authors built an interactive test. An SI agent designs its own heating experiments, uses a simulator that may be flawed, and reports each material property with a 95% interval. The test covers 93 held-out tasks, 9 models and 2,511 episodes. Models handle procedural traps. When the simulator is missing physics, most report wrong values while claiming 95% confidence. Their ranges hold the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Asking for wider intervals does not help. Stating the strategy improves interval scores 1.6-2.5-fold, but coverage barely moves. At maximum effort, Opus improves more than tenfold, but it uses 13-31 times more reasoning. Older and open-weight models gain nothing.","matters":"Autonomous SI agents are being pointed at real lab work. In that work, a confident wrong answer can be worse than an honest \"I am not sure.\" This benchmark scores whether stated confidence matches the truth, using exact ground truth. It also shows that good results can come from smart experiment design, such as gentle heating, and not only from naming the cause of the problem. The abstract also shows a cost trade-off. Fable 5.2 is the most accurate at default effort, with an interval score 1.9 times better than Astra's. It writes 6 times as many tokens, and at Astra's token count it falls short.","race":"The result suggests that judgment about when to trust a tool is a separate skill from raw ability. It may be latent in some models and only appear with much more reasoning. Labs building SI scientists may need to compare models at equal reasoning budgets, not only at default settings.","safety":"The paper bears on trust in SI agents. Most models gave confident intervals that were wrong when the simulator was flawed, and asking for wider intervals did not fix this. Calibration checks like this one may be needed before agents are given autonomy in experiments.","level":"intermediate","tags":["evals","benchmarks","agents","science","reasoning","safety"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/) · [Nov 11, 2026 · Number 8Late but Valuable: Why Deadlines Change the Measured Value of LLM ReasoningWhen SI answers must arrive before a deadline, judging answer quality and speed separately overstates the value of reasoning. Score the action that actually runs.EvalsAgentsReasoning](https://thesimind.si/papers/2611.11910/) · [Nov 15, 2026 · Number 5AnalogLayoutWorld: An Open Benchmark for Agentic Analog Layout DesignSI coding agents can often produce analog chip layout files, but only 23.0% of attempts pass the benchmark's acceptance criteria, so physical engineering remains hard.BenchmarksAgentsEvals](https://thesimind.si/papers/2611.16991/)

[Number 8](https://thesimind.si/papers/2611.20956/) · [All 12 for Nov 18, 2026](https://thesimind.si/papers/day/2026-11-18/) · [Number 10](https://thesimind.si/papers/2611.20779/)
