What they found
The authors built an interactive test. An SI agent designs its own heating experiments, uses a simulator that may be flawed, and reports each material property with a 95% interval. The test covers 93 held-out tasks, 9 models and 2,511 episodes. Models handle procedural traps. When the simulator is missing physics, most report wrong values while claiming 95% confidence. Their ranges hold the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Asking for wider intervals does not help. Stating the strategy improves interval scores 1.6-2.5-fold, but coverage barely moves. At maximum effort, Opus improves more than tenfold, but it uses 13-31 times more reasoning. Older and open-weight models gain nothing.
Why it matters
Autonomous SI agents are being pointed at real lab work. In that work, a confident wrong answer can be worse than an honest "I am not sure." This benchmark scores whether stated confidence matches the truth, using exact ground truth. It also shows that good results can come from smart experiment design, such as gentle heating, and not only from naming the cause of the problem. The abstract also shows a cost trade-off. Fable 5.2 is the most accurate at default effort, with an interval score 1.9 times better than Astra's. It writes 6 times as many tokens, and at Astra's token count it falls short.
How it changes the race
The result suggests that judgment about when to trust a tool is a separate skill from raw ability. It may be latent in some models and only appear with much more reasoning. Labs building SI scientists may need to compare models at equal reasoning budgets, not only at default settings.
Safety and control
The paper bears on trust in SI agents. Most models gave confident intervals that were wrong when the simulator was flawed, and asking for wider intervals did not fix this. Calibration checks like this one may be needed before agents are given autonomy in experiments.
The abstract, by the authors
We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.
Quoted from arXiv:2611.21541. The research and its words belong to its authors.
