What they found
The authors studied SI models that make decisions in automated laboratories. If an answer arrives after the deadline, a safe fallback action runs instead, such as rejecting a sample. On synthetic decisions with a known best choice, they compared one model's fast mode with its thinking mode. In a prespecified replay of 1,200 new decisions, thinking answers that missed the deadline were worth about five times as much as on-time ones. Multiplying the on-time rate by the mean answer value predicted about twice the value thinking delivered. This overstated thinking's edge over fast mode by 0.0241 per decision, and by 0.0191 in a repeat on 300 new decisions. Both 95% confidence intervals excluded zero. In a single-GPU Qwen3-8B test with simultaneous request bursts, going from 8 to 56 requests cut the share of thinking answers meeting a 45 s deadline from 79.2% to 1.4%, though total output tripled.
Why it matters
Lab and factory SI agents often work on hard schedules, but common serving metrics report quality and speed apart. The authors show this can make slow reasoning look more useful than it is. The late answers were often the valuable ones, because they differed from the fallback. The results come from synthetic decisions and one model family, so real labs may differ.
How it changes the race
If the result holds, teams building SI agents for real-time work will need to judge reasoning by the value of the action that actually runs, not by quality and latency alone. Serving capacity under load becomes part of how capable a system really is.
Safety and control
The paper centers on a safe fallback action that runs when an answer is late. It shows that such a fallback can quietly discard the valuable reasoning, so safety design and value should be measured together.
The abstract, by the authors
Large language models (LLMs) increasingly plan and run experiments in automated laboratories. These experiments keep running while the model reasons and run on schedules with hard deadlines. If an LLM-generated answer arrives late, the system executes a safe fallback action instead, such as rejecting a sample. This creates a systems problem that standard serving metrics do not capture. An answer that matches the fallback adds no value, and answers that differ from the fallback may require more reasoning time. Reporting answer quality and latency separately can therefore overstate the value of reasoning. We test this on synthetic sample decisions, where the best choice is known exactly and a runtime enforces the deadline. We compare one LLM's fast mode, which answers directly, with its thinking mode, which reasons first. In a prespecified replay of 1,200 new decisions, thinking answers that missed a deadline were worth about five times as much as on-time ones. Multiplying the on-time rate by the mean answer value predicted about twice the value that thinking delivered. This prediction overstated thinking's advantage over fast mode by 0.0241 per decision, and by 0.0191 in a repeat on 300 new decisions. Both 95% confidence intervals excluded zero. In a single-GPU Qwen3-8B experiment using simultaneous request bursts, raising the number of requests from 8 to 56 reduced the share of thinking answers that met a 45 s deadline from 79.2% to 1.4%, although total output tripled. Evaluations of LLM agents for laboratories should therefore measure each answer's value and arrival time together and score the action that actually runs.
Quoted from arXiv:2611.11910. The research and its words belong to its authors.