What they found
The authors built TwinLab-Bench, with 52 tasks from labs in the physical sciences, the life sciences and engineering. The tested agent designs its own experiments. A separate "TwinLab agent" then runs them, using raw lab materials and calculators built from the lab's records. The tested agent's report is scored against a hidden answer taken from real laboratory records. The authors say the TwinLab agent almost eliminates readings that stray from the real records and inconsistent answers to a repeated request, compared with a language model that only sees the task description. Closed-book probes check for answers recalled from memory. The tasks stay hard: the strongest model, Claude Fable 5.2, averages 45.0, and the authors study where it fails.
Why it matters
Many people expect SI agents to work as scientists. Most existing tests check only whether an agent can follow a given workflow or pick from a list of options. This benchmark tests the harder skill of designing an experiment from scratch. A low top score gives a clear measure of the gap between current models and that goal. The abstract does not give details on the failure study, so those findings are not covered here.
How it changes the race
It gives frontier developers a harder, more realistic test of SI scientist skills than closed-loop benchmarks. If it holds up, it could steer effort toward open-ended experiment design. It is an evaluation, not a new capability.
The abstract, by the authors
AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.
Quoted from arXiv:2611.26248. The research and its words belong to its authors.
