Number 5 of 12 · November 22, 2026

Training LLMs to Follow Reasoning Instructions with Minimal Edits

The authors train a model on small edits at the first point where its own reasoning goes wrong, and report better reasoning instruction following with other skills largely kept.

What they found

The authors propose edit-based training. The model writes a reasoning answer. An editor then finds the first point where it fails a given rule and writes a minimal replacement for that span. The model is trained only on those edited spans, so the rest of its reasoning is left alone. They tested it on reasoning instruction following, using Qwen3-32B and the ReasonIF benchmark. They report that it substantially improves compliance with the reasoning instruction compared with full-trajectory imitation and trajectory-level reinforcement learning. They also report that it largely preserves skills on untargeted tasks. The abstract gives no specific numbers, and it does not say what kind of editor was used.

Why it matters

Shaping how a model reasons, not just what it answers, is hard with blunt training signals. Copying full demonstrations can overconstrain a model. Scoring whole answers with reinforcement learning does not show where the mistake happened. This work points to a more targeted option: fix the first error and leave the rest. The authors present it as a complement to existing methods, not a replacement.

How it changes the race

If the result holds, developers get a finer tool for steering reasoning behavior without retraining on whole demonstrations. This is a post-training refinement. The evidence in the abstract covers one model and one benchmark, so it is an incremental step for now.

Safety and control

The work is about making models follow instructions about how they reason, which relates to control. The abstract does not test safety directly, so any safety link is a possible use, not a result.

The abstract, by the authors

Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.

Quoted from arXiv:2611.26342. The research and its words belong to its authors.

The authors

Alexandra Yakovleva, Luat Gia Khoi Nguyen, Juho Kannala, Harri Valpola, Alexander Ilin

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.