Number 8 of 12 · November 22, 2026

ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

ROAM-GRPO adds a step to reinforcement learning for reasoning that pushes a model to find different solution strategies, and the authors report better pass@16 and more varied correct answers on a small model.

What they found

The authors start from a known problem. Training with supervised fine-tuning and then reinforcement learning with verifiable rewards can make a model give nearly identical answers each time. Pass@k, the chance that at least one of k tries is right, then stops improving or gets worse. Their fix, ROAM-GRPO, spots problems the model already solves well during training. It moves them to an "off-mode" track, where the model is asked for distinct solution strategies. An LLM judge checks that each one is logically valid and different, and the model is then trained on the approved ones (self-distillation). They trained Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset. They report that peak pass@16 rose by 4.7% and that the embedding cosine distance between correct answers grew by about 1.5-2x on a difficult evaluation set.

Why it matters

Many SI reasoning models are trained with this recipe, and losing variety in answers limits how much extra sampling helps. A method that keeps several valid approaches alive could make better use of repeated tries. The abstract describes one small model and one dataset subset, so it does not show whether the gain holds for larger models or other tasks. It also does not say how much the LLM judge's accuracy matters.

How it changes the race

If it holds up, this is a modest, low-cost add-on to an existing training method, so small teams could try it. On the evidence given, it is an incremental step and has not been shown at frontier scale.

The abstract, by the authors

Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

Quoted from arXiv:2611.25926. The research and its words belong to its authors.

The authors

Jeff Duan, Matt Thomson

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.