---
title: "TwinLab-Bench: Evaluating AI Scientists... | SI Paper Desk"
url: https://thesimind.si/papers/2611.26248/
description: "TwinLab-Bench tests whether SI agents can design their own lab experiments in simulated twins of real labs, and the best model scores only 45.0."
---

Number 3 of 12 · November 21, 2026

# TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs

TwinLab-Bench tests whether SI agents can design their own lab experiments in simulated twins of real labs, and the best model scores only 45.0.

Yang Wan and Linchao Zhu · Submitted November 19, 2026 · arXiv 2611.26248v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.26248) · [The PDF](https://arxiv.org/pdf/2611.26248) · [The day's 12](https://thesimind.si/papers/day/2026-11-21/)

## What they found

The authors built TwinLab-Bench, with 52 tasks from labs in the physical sciences, the life sciences and engineering. The tested agent designs its own experiments. A separate "TwinLab agent" then runs them, using raw lab materials and calculators built from the lab's records. The tested agent's report is scored against a hidden answer taken from real laboratory records. The authors say the TwinLab agent almost eliminates readings that stray from the real records and inconsistent answers to a repeated request, compared with a language model that only sees the task description. Closed-book probes check for answers recalled from memory. The tasks stay hard: the strongest model, Claude Fable 5.2, averages 45.0, and the authors study where it fails.

## Why it matters

Many people expect SI agents to work as scientists. Most existing tests check only whether an agent can follow a given workflow or pick from a list of options. This benchmark tests the harder skill of designing an experiment from scratch. A low top score gives a clear measure of the gap between current models and that goal. The abstract does not give details on the failure study, so those findings are not covered here.

## How it changes the race

It gives frontier developers a harder, more realistic test of SI scientist skills than closed-loop benchmarks. If it holds up, it could steer effort toward open-ended experiment design. It is an evaluation, not a new capability.

### The abstract, by the authors

AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.

Quoted from arXiv:2611.26248. The research and its words belong to its authors.

Desk score **79** out of 100

Why it was picked

- Frontier lab: DeepSeek
- Agents (in the title)
- Evals
- Studies frontier models (Claude)
- Cross-listed in 3 fields
- Language models

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Science](https://thesimind.si/papers/?tag=science)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Language models](https://thesimind.si/papers/?tag=language-models)

**arXiv ID:** 2611.26248v1
**Submitted:** November 19, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.CL (Computation and Language), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** [https://twinlab-bench.github.io/](https://twinlab-bench.github.io/) contains tasks and code. 33 pages, 6 figures, 14 tables. LLM-agent benchmark for autonomous experimental design in digital twins of real labs (physics, chemistry, wet-lab biology, materials, engineering). 13 agents evaluated: Claude Fable 5.2 45.0, Opus 5.5 40.2...

Fingerprint, SHA-256

`5f69738d440d0ff1bc199c4f45c1c2453c8d4e8b47c7d2c121d5a4e4d56bd4da`

On Solana: [Proof of Mind seal, beudvj...We9f](https://solscan.io/tx/beudvjGVbBTGVZsMaL2MgMJjuMy9dK2a2K3ETkw5AFwjoARs9PQedxMxfrBaiJ5MetjT2AHxuhfTCyP34eQWe9f)

### The exact text fingerprinted

{"arxiv":"2611.26248","version":1,"title":"TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs","authors":["Yang Wan","Linchao Zhu"],"day":"2026-11-21","rank":3,"score":79,"status":"written","summary":{"takeaway":"TwinLab-Bench tests whether SI agents can design their own lab experiments in simulated twins of real labs, and the best model scores only 45.0.","found":"The authors built TwinLab-Bench, with 52 tasks from labs in the physical sciences, the life sciences and engineering. The tested agent designs its own experiments. A separate \"TwinLab agent\" then runs them, using raw lab materials and calculators built from the lab's records. The tested agent's report is scored against a hidden answer taken from real laboratory records. The authors say the TwinLab agent almost eliminates readings that stray from the real records and inconsistent answers to a repeated request, compared with a language model that only sees the task description. Closed-book probes check for answers recalled from memory. The tasks stay hard: the strongest model, Claude Fable 5.2, averages 45.0, and the authors study where it fails.","matters":"Many people expect SI agents to work as scientists. Most existing tests check only whether an agent can follow a given workflow or pick from a list of options. This benchmark tests the harder skill of designing an experiment from scratch. A low top score gives a clear measure of the gap between current models and that goal. The abstract does not give details on the failure study, so those findings are not covered here.","race":"It gives frontier developers a harder, more realistic test of SI scientist skills than closed-loop benchmarks. If it holds up, it could steer effort toward open-ended experiment design. It is an evaluation, not a new capability.","safety":"","level":"intermediate","tags":["benchmarks","agents","science","evals","language-models"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Yang Wan, Linchao Zhu

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 14, 2026 · Number 8Auditing AI peer reviewers: a benchmark on real scientific papersThe authors built a test that plants known errors in real papers. They found that SI peer reviewers vary widely, mostly because of which model they use.EvalsBenchmarksAgents](https://thesimind.si/papers/2611.17052/) · [Nov 19, 2026 · Number 12SciWorkBench: A Benchmark for Real Scientific WorkflowsSciWorkBench tests SI agents on real science tasks in four fields. The best agent scores 57.38, so current agents still struggle with realistic scientific workflows.BenchmarksAgentsScience](https://thesimind.si/papers/2611.22490/) · [Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/)

[Number 2](https://thesimind.si/papers/2611.26553/) · [All 12 for Nov 21, 2026](https://thesimind.si/papers/day/2026-11-21/) · [Number 4](https://thesimind.si/papers/2611.26545/)
