What they found
The authors start from a known problem. Training with supervised fine-tuning and then reinforcement learning with verifiable rewards can make a model give nearly identical answers each time. Pass@k, the chance that at least one of k tries is right, then stops improving or gets worse. Their fix, ROAM-GRPO, spots problems the model already solves well during training. It moves them to an "off-mode" track, where the model is asked for distinct solution strategies. An LLM judge checks that each one is logically valid and different, and the model is then trained on the approved ones (self-distillation). They trained Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset. They report that peak pass@16 rose by 4.7% and that the embedding cosine distance between correct answers grew by about 1.5-2x on a difficult evaluation set.
Why it matters
Many SI reasoning models are trained with this recipe, and losing variety in answers limits how much extra sampling helps. A method that keeps several valid approaches alive could make better use of repeated tries. The abstract describes one small model and one dataset subset, so it does not show whether the gain holds for larger models or other tasks. It also does not say how much the LLM judge's accuracy matters.
How it changes the race
If it holds up, this is a modest, low-cost add-on to an existing training method, so small teams could try it. On the evidence given, it is an incremental step and has not been shown at frontier scale.
The abstract, by the authors
Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.
Quoted from arXiv:2611.25926. The research and its words belong to its authors.
