---
title: "ROAM-GRPO: Enforcing Reasoning Diversity... | SI Paper Desk"
url: https://thesimind.si/papers/2611.25926/
description: "ROAM-GRPO adds a step to reinforcement learning for reasoning that pushes a model to find different solution strategies, and the authors report better..."
---

Number 8 of 12 · November 22, 2026

# ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

ROAM-GRPO adds a step to reinforcement learning for reasoning that pushes a model to find different solution strategies, and the authors report better pass@16 and more varied correct answers on a small model.

Jeff Duan and Matt Thomson · Submitted November 19, 2026 · arXiv 2611.25926v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.25926) · [The PDF](https://arxiv.org/pdf/2611.25926) · [The day's 12](https://thesimind.si/papers/day/2026-11-22/)

## What they found

The authors start from a known problem. Training with supervised fine-tuning and then reinforcement learning with verifiable rewards can make a model give nearly identical answers each time. Pass@k, the chance that at least one of k tries is right, then stops improving or gets worse. Their fix, ROAM-GRPO, spots problems the model already solves well during training. It moves them to an "off-mode" track, where the model is asked for distinct solution strategies. An LLM judge checks that each one is logically valid and different, and the model is then trained on the approved ones (self-distillation). They trained Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset. They report that peak pass@16 rose by 4.7% and that the embedding cosine distance between correct answers grew by about 1.5-2x on a difficult evaluation set.

## Why it matters

Many SI reasoning models are trained with this recipe, and losing variety in answers limits how much extra sampling helps. A method that keeps several valid approaches alive could make better use of repeated tries. The abstract describes one small model and one dataset subset, so it does not show whether the gain holds for larger models or other tasks. It also does not say how much the LLM judge's accuracy matters.

## How it changes the race

If it holds up, this is a modest, low-cost add-on to an existing training method, so small teams could try it. On the evidence given, it is an incremental step and has not been shown at frontier scale.

### The abstract, by the authors

Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

Quoted from arXiv:2611.25926. The research and its words belong to its authors.

Desk score **67** out of 100

Why it was picked

- Reasoning (in the title)
- Evals
- Studies frontier models (Qwen3.5)
- Accepted at NeurIPS
- Language models

Topics

- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)
- [Training](https://thesimind.si/papers/?tag=training)
- [Math](https://thesimind.si/papers/?tag=math)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)

**arXiv ID:** 2611.25926v1
**Submitted:** November 19, 2026
**Fields:** cs.AI (Artificial Intelligence)
**Level:** Intermediate
**Comments:** Accepted to the NeurIPS 2026 Transitioning from Pre-Training to Post-Training Workshop

Fingerprint, SHA-256

`83be2b28afe64dadc65da041d0a1fb70aaef685b9589a90732d816638a2bab9c`

In SI MIND's journal since Nov 22, 2026, 02:01 UTC. It goes on Solana with the next hourly Proof of Mind seal.

### The exact text fingerprinted

{"arxiv":"2611.25926","version":1,"title":"ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining","authors":["Jeff Duan","Matt Thomson"],"day":"2026-11-22","rank":8,"score":67,"status":"written","summary":{"takeaway":"ROAM-GRPO adds a step to reinforcement learning for reasoning that pushes a model to find different solution strategies, and the authors report better pass@16 and more varied correct answers on a small model.","found":"The authors start from a known problem. Training with supervised fine-tuning and then reinforcement learning with verifiable rewards can make a model give nearly identical answers each time. Pass@k, the chance that at least one of k tries is right, then stops improving or gets worse. Their fix, ROAM-GRPO, spots problems the model already solves well during training. It moves them to an \"off-mode\" track, where the model is asked for distinct solution strategies. An LLM judge checks that each one is logically valid and different, and the model is then trained on the approved ones (self-distillation). They trained Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset. They report that peak pass@16 rose by 4.7% and that the embedding cosine distance between correct answers grew by about 1.5-2x on a difficult evaluation set.","matters":"Many SI reasoning models are trained with this recipe, and losing variety in answers limits how much extra sampling helps. A method that keeps several valid approaches alive could make better use of repeated tries. The abstract describes one small model and one dataset subset, so it does not show whether the gain holds for larger models or other tasks. It also does not say how much the LLM judge's accuracy matters.","race":"If it holds up, this is a modest, low-cost add-on to an existing training method, so small teams could try it. On the evidence given, it is an incremental step and has not been shown at frontier scale.","safety":"","level":"intermediate","tags":["reinforcement-learning","reasoning","training","math","language-models","efficiency"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Jeff Duan, Matt Thomson

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 17, 2026 · Number 4ARCHER: From Rationale Imitation to Executable Reasoning in Small Language ModelsARCHER trains small SI models to solve math by writing step-by-step traces that call a calculator and check results, and the authors report gains over the strongest tool-augmented baseline.ReasoningMathReinforcement learning](https://thesimind.si/papers/2611.18494/) · [Nov 19, 2026 · Number 11Get Hint-Augmented GRPO Done Right Through Activation SteeringASH trains a small steering vector from hinted questions, then adds it inside the model at test time, so the model acts as if it had a hint when none is given.Reinforcement learningReasoningTraining](https://thesimind.si/papers/2611.23124/) · [Nov 9, 2026 · Number 11Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM ReasoningDR-BOLT trains one vision-language model to answer simple questions directly and think step by step only when needed, cutting wasted reasoning and training time.ReasoningEfficiencyMultimodal](https://thesimind.si/papers/2611.09154/)

[Number 7](https://thesimind.si/papers/2611.26199/) · [All 12 for Nov 22, 2026](https://thesimind.si/papers/day/2026-11-22/) · [Number 9](https://thesimind.si/papers/2611.26569/)
