What they found
The authors introduce PREWARN, a way to test monitors that watch a language model while it writes. It sets each monitor's alarm threshold on validation data to a chosen error rate. It then scores alarms against the token where the failure starts, for three jobs: classifying the response, detecting the failure, and warning before it. As a case study, they test repetitive degeneration (the model gets stuck repeating itself) in three open language models. They use linear probes on the model's internal activations and six untrained baselines. Response-level scores like AUROC saturate here, and a rule that reads only response length scores almost perfectly. Probes that AUROC ranks as equal warn very differently under PREWARN. Probes warn before the onset on most degenerate responses of two of the three models. Low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset. No untrained baseline warns much more often than random alarms.
Why it matters
A monitor is only useful for stopping or steering a model if it fires before the failure starts. Common scores ignore timing, so they can make weak monitors look strong. This paper shows a case where a trivial length rule looks almost perfect on those scores. A timing-aware test gives a more honest picture of what a monitor can do.
How it changes the race
This is a measurement advance, not a capability one. If it holds, teams building monitors for SI systems would have a stricter test for early warning. The abstract tests only one failure type in three open models.
Safety and control
The paper bears on runtime monitoring of SI models, since only a warning before a failure leaves time to stop or steer generation. The case study is repetitive degeneration, a mild failure. The abstract does not say whether the protocol carries over to more serious failures.
The abstract, by the authors
Many failures of language models begin at a specific token, the onset. A monitor that reads the model during generation can classify the finished response, detect the failure once it is under way, or warn before it begins, and only a warning leaves time to stop or steer the generation. Monitors are usually evaluated with response-level metrics such as AUROC, which ignore when the monitor fires. We introduce PREWARN, an evaluation protocol for any failure with an annotated onset. It calibrates each monitor's threshold on validation data to an error rate chosen for the application, runs the monitor, and scores its alarms relative to the onset on all three tasks. As a case study we apply it to repetitive degeneration in three open language models, with early warning as the main task, using linear probes on activations and six untrained baselines. Response-level metrics saturate on this failure, and a rule that reads only the response length scores almost perfectly, while probes that AUROC ranks as equal warn very differently under PREWARN. Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms. We release the probes with calibrated thresholds at https://huggingface.co/collections/marcodene/prewarn-degeneration-probes
Quoted from arXiv:2611.16803. The research and its words belong to its authors.
