What they found
The authors built SciWorkBench, a benchmark (a standard test) of real science tasks in biology, chemistry, materials science and physics. Tasks come from peer-reviewed papers or research scenarios given by experts. The authors also propose a framework of scientific analysis stages, checked by domain experts, and each task covers one or more of those stages. They tested up-to-date models inside the Claude Code and Codex agent frameworks under one shared way of running and grading. The best agent got a mean score of 57.38. Their failure analysis points to three weaknesses: turning scientific knowledge into task-specific steps, combining evidence from different kinds of data, and correcting its own scientific mistakes.
Why it matters
Many people want to use SI agents to do research, not just answer questions. This benchmark checks whole workflows, which is closer to how scientists work than single questions. The abstract says current agents fall short, and it names where they fail. That gives builders and users a clearer picture of what to fix and what not to trust yet.
How it changes the race
It gives frontier builders a test aimed at end-to-end scientific work, with named weak spots to target. The abstract gives no comparison across models beyond the best score, so the effect is likely to be a new yardstick, not a shift in capability.
Safety and control
The abstract frames its goal as measuring progress toward reliable SI systems for scientific research. Weak self-correction is a reliability concern when agents work with little human checking.
The abstract, by the authors
Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
Quoted from arXiv:2611.22490. The research and its words belong to its authors.
