Number 6 of 12 · November 15, 2026

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

A SI model that is bad at acting in a long task can still be a good teacher, because judging actions after the episode ends shows their real consequences.

What they found

The authors study "decision models," which give a typed answer such as a choice among options with a probability for each. One such model can pick actions, judge past actions, or score every action to train a policy with reinforcement learning. On ALFWorld, they tested three decision models and three general LLMs as actors and judges. As actors, all six were pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credited the steps toward that subgoal. Asking whether each action helped, instead of whether the task will succeed, separated useful from useless actions. As a teacher, JEV raised a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop.

Why it matters

Earlier work mostly judged decision models by how well they act. This paper suggests that is the wrong test if the goal is to train other models. The authors argue that judging an action after the episode ends reveals consequences that an actor must guess in advance. If that holds, models that look weak as agents may still be useful sources of training signal for long tasks.

How it changes the race

If the result holds, builders of agent SI could get stronger training signals from models they would have written off as poor actors. That could help small policies improve on long tasks without needing a top-performing actor as the guide. The abstract reports only two benchmarks and one small policy, so the reach is not yet clear.

Safety and control

The paper does not address safety directly. It does bear on how SI agents are trained, since judge models that score each action shape what a policy learns.

The abstract, by the authors

Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.

Quoted from arXiv:2611.17167. The research and its words belong to its authors.

The authors

Haochen Liu, Qianhao Zhu, Zixuan Wang, Dingming Li, Tongbo Chen, Peng Qiu, Yingqi Qu, Weiming Lu, Jun Xiao, Yongliang Shen

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.