What they found
The authors point out a bookkeeping problem in a common way of training multi-turn SI agents with reinforcement learning. Each episode is split into several rows and many tokens. This can count one episode's outcome several times, and it can give long episodes more total weight than short ones. Their fix, Outcome-Mass Alignment (OMA), has two parts: compute outcome statistics over unique episodes, and control the total weight each episode carries. On ALFWorld, at two model scales, OMA improved matched-horizon endpoint success by 7.0-9.4 percentage points. On ordinary WebShop the effects were mixed, but in a success-oriented setting called WebShop-SOL, success nAUC rose from 0.399 to 0.511.
Why it matters
Training agents on long tasks is costly, and small hidden choices in how results are counted can change what an agent learns. The paper says failed episodes often expand much more than successful ones on ALFWorld, so they get extra weight by default. The authors argue the effect depends on how episode length relates to outcomes. That means the right fix is not the same everywhere. The abstract does not say how the method fares on other tasks or at larger scale.
How it changes the race
This is a low-cost, incremental change to training that works with existing methods, with no redesign of the base optimizer. If it holds up, teams training SI agents could get better results from the same setup by fixing how episodes are counted.
Safety and control
")
The abstract, by the authors
To keep long interaction contexts tractable, stepwise group-relative training for multi-turn LLM agents materializes each source trajectory into multiple response rows and valid tokens. This expansion creates an outer-accounting problem distinct from within-trajectory credit assignment: row-based outcome statistics can count one episode outcome multiple times, while token-level reduction can assign unequal aggregate objective mass across trajectories. We term these effects outcome-statistics multiplicity and objective-mass imbalance. We introduce Outcome-Mass Alignment (OMA), a two-component method that makes both accounting choices explicit. Outcome Alignment computes outcome statistics over unique source trajectories, while Mass Alignment controls aggregate objective mass after within-trajectory aggregation of optimizer-native token objectives. Neither redesigns the base optimizer's credit assignment, importance ratios, or clipping. Across matched ALFWorld GRPO comparisons at two model scales, where failed trajectories typically expand much more than successful ones, OMA improves matched-horizon endpoint success by 7.0-9.4 percentage points. Ordinary WebShop exhibits weak or reversed expansion-outcome coupling and correspondingly mixed effects, whereas in a success-oriented WebShop setting, WebShop-SOL, OMA improves success nAUC from 0.399 to 0.511. Component ablations and GiGPO and GSPO comparisons further support component separability and cross-optimizer compatibility. These results show that trajectory expansion always defines an implicit weighting rule, but its practical effect depends on how expansion couples to outcomes and optimizer-native updates.
Quoted from arXiv:2611.16816. The research and its words belong to its authors.
