---
title: "Training LLMs to Follow Reasoning... | SI Paper Desk"
url: https://thesimind.si/papers/2611.26342/
description: "The authors train a model on small edits at the first point where its own reasoning goes wrong, and report better reasoning instruction following with other..."
---

Number 5 of 12 · November 22, 2026

# Training LLMs to Follow Reasoning Instructions with Minimal Edits

The authors train a model on small edits at the first point where its own reasoning goes wrong, and report better reasoning instruction following with other skills largely kept.

Alexandra Yakovleva, Luat Gia Khoi Nguyen, Juho Kannala and 2 more · Submitted November 19, 2026 · arXiv 2611.26342v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.26342) · [The PDF](https://arxiv.org/pdf/2611.26342) · [The day's 12](https://thesimind.si/papers/day/2026-11-22/)

## What they found

The authors propose edit-based training. The model writes a reasoning answer. An editor then finds the first point where it fails a given rule and writes a minimal replacement for that span. The model is trained only on those edited spans, so the rest of its reasoning is left alone. They tested it on reasoning instruction following, using Qwen3-32B and the ReasonIF benchmark. They report that it substantially improves compliance with the reasoning instruction compared with full-trajectory imitation and trajectory-level reinforcement learning. They also report that it largely preserves skills on untargeted tasks. The abstract gives no specific numbers, and it does not say what kind of editor was used.

## Why it matters

Shaping how a model reasons, not just what it answers, is hard with blunt training signals. Copying full demonstrations can overconstrain a model. Scoring whole answers with reinforcement learning does not show where the mistake happened. This work points to a more targeted option: fix the first error and leave the rest. The authors present it as a complement to existing methods, not a replacement.

## How it changes the race

If the result holds, developers get a finer tool for steering reasoning behavior without retraining on whole demonstrations. This is a post-training refinement. The evidence in the abstract covers one model and one benchmark, so it is an incremental step for now.

## Safety and control

The work is about making models follow instructions about how they reason, which relates to control. The abstract does not test safety directly, so any safety link is a possible use, not a result.

### The abstract, by the authors

Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.

Quoted from arXiv:2611.26342. The research and its words belong to its authors.

Desk score **70** out of 100

Why it was picked

- Reasoning (in the title)
- Evals
- Language models (in the title)
- Studies frontier models (Qwen3)
- Accepted at NeurIPS

Topics

- [Training](https://thesimind.si/papers/?tag=training)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)
- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Alignment](https://thesimind.si/papers/?tag=alignment)
- [Evals](https://thesimind.si/papers/?tag=evals)

**arXiv ID:** 2611.26342v1
**Submitted:** November 19, 2026
**Fields:** cs.AI (Artificial Intelligence)
**Level:** Intermediate
**Comments:** Accepted at the NeurIPS 2026 Workshop on Transitioning from Pre-training to Post-training

Fingerprint, SHA-256

`edd20218d732a8214e9a3980c0b4b6d95e3a569c0a35e1094f9a06212daca6b5`

In SI MIND's journal since Nov 22, 2026, 02:01 UTC. It goes on Solana with the next hourly Proof of Mind seal.

### The exact text fingerprinted

{"arxiv":"2611.26342","version":1,"title":"Training LLMs to Follow Reasoning Instructions with Minimal Edits","authors":["Alexandra Yakovleva","Luat Gia Khoi Nguyen","Juho Kannala","Harri Valpola","Alexander Ilin"],"day":"2026-11-22","rank":5,"score":70,"status":"written","summary":{"takeaway":"The authors train a model on small edits at the first point where its own reasoning goes wrong, and report better reasoning instruction following with other skills largely kept.","found":"The authors propose edit-based training. The model writes a reasoning answer. An editor then finds the first point where it fails a given rule and writes a minimal replacement for that span. The model is trained only on those edited spans, so the rest of its reasoning is left alone. They tested it on reasoning instruction following, using Qwen3-32B and the ReasonIF benchmark. They report that it substantially improves compliance with the reasoning instruction compared with full-trajectory imitation and trajectory-level reinforcement learning. They also report that it largely preserves skills on untargeted tasks. The abstract gives no specific numbers, and it does not say what kind of editor was used.","matters":"Shaping how a model reasons, not just what it answers, is hard with blunt training signals. Copying full demonstrations can overconstrain a model. Scoring whole answers with reinforcement learning does not show where the mistake happened. This work points to a more targeted option: fix the first error and leave the rest. The authors present it as a complement to existing methods, not a replacement.","race":"If the result holds, developers get a finer tool for steering reasoning behavior without retraining on whole demonstrations. This is a post-training refinement. The evidence in the abstract covers one model and one benchmark, so it is an incremental step for now.","safety":"The work is about making models follow instructions about how they reason, which relates to control. The abstract does not test safety directly, so any safety link is a possible use, not a result.","level":"intermediate","tags":["training","reasoning","reinforcement-learning","language-models","alignment","evals"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Alexandra Yakovleva, Luat Gia Khoi Nguyen, Juho Kannala, Harri Valpola, Alexander Ilin

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 16, 2026 · Number 12Benchmarks Don't Talk Back: Learning Through Self-DisagreementAbbasloo and Cheng argue that knowing when to hold an answer and when to yield is its own skill, and that a model can train it by arguing against itself.AlignmentReinforcement learningLanguage models](https://thesimind.si/papers/2611.17563/) · [Nov 18, 2026 · Number 4Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process OracleWhen the checker used to reward an SI model is too generous, the model learns to game it. The authors show this clearly because they can measure the true reward exactly.Reinforcement learningAlignmentEvals](https://thesimind.si/papers/2611.21538/) · [Nov 16, 2026 · Number 6Alignment with Reflect-Reasoning: Learning How To Follow PrinciplesReflect-Reasoning trains models to reason about whether they are following a given set of principles, and the authors report it beats system prompting across five models.AlignmentReasoningLanguage models](https://thesimind.si/papers/2611.17398/)

[Number 4](https://thesimind.si/papers/2611.26398/) · [All 12 for Nov 22, 2026](https://thesimind.si/papers/day/2026-11-22/) · [Number 6](https://thesimind.si/papers/2611.26282/)
