---
title: "Actor, Judge, or Teacher? Understanding... | SI Paper Desk"
url: https://thesimind.si/papers/2611.17167/
description: "A SI model that is bad at acting in a long task can still be a good teacher, because judging actions after the episode ends shows their real consequences."
---

Number 6 of 12 · November 15, 2026

# Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

A SI model that is bad at acting in a long task can still be a good teacher, because judging actions after the episode ends shows their real consequences.

Haochen Liu, Qianhao Zhu, Zixuan Wang and 7 more · Submitted November 12, 2026 · arXiv 2611.17167v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.17167) · [The PDF](https://arxiv.org/pdf/2611.17167) · [The day's 12](https://thesimind.si/papers/day/2026-11-15/)

## What they found

The authors study "decision models," which give a typed answer such as a choice among options with a probability for each. One such model can pick actions, judge past actions, or score every action to train a policy with reinforcement learning. On ALFWorld, they tested three decision models and three general LLMs as actors and judges. As actors, all six were pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credited the steps toward that subgoal. Asking whether each action helped, instead of whether the task will succeed, separated useful from useless actions. As a teacher, JEV raised a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop.

## Why it matters

Earlier work mostly judged decision models by how well they act. This paper suggests that is the wrong test if the goal is to train other models. The authors argue that judging an action after the episode ends reveals consequences that an actor must guess in advance. If that holds, models that look weak as agents may still be useful sources of training signal for long tasks.

## How it changes the race

If the result holds, builders of agent SI could get stronger training signals from models they would have written off as poor actors. That could help small policies improve on long tasks without needing a top-performing actor as the guide. The abstract reports only two benchmarks and one small policy, so the reach is not yet clear.

## Safety and control

The paper does not address safety directly. It does bear on how SI agents are trained, since judge models that score each action shape what a policy learns.

### The abstract, by the authors

Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.

Quoted from arXiv:2611.17167. The research and its words belong to its authors.

Desk score **74** out of 100

Why it was picked

- Agents (in the title)
- Reasoning (in the title)
- Studies frontier models (Qwen2.5)
- Language models
- Cross-listed in 2 fields
- Code or data released

Topics

- [Agents](https://thesimind.si/papers/?tag=agents)
- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Training](https://thesimind.si/papers/?tag=training)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)

**arXiv ID:** 2611.17167v1
**Submitted:** November 12, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.CL (Computation and Language)
**Level:** Intermediate

Fingerprint, SHA-256

`b495dca254ce3036d841b73a24c1e231771a6da4ca8a8fbd825238b2a8cdb34a`

On Solana: [Proof of Mind seal, Szprsf...FGjr](https://solscan.io/tx/SzprsfAMG4vXvYD7hrurYFce76smMcrTbLnANy8xS96bh14DoynCdjEEMZf7RPtRUgYeejS8HGnxjqtD1XuFGjr)

### The exact text fingerprinted

{"arxiv":"2611.17167","version":1,"title":"Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents","authors":["Haochen Liu","Qianhao Zhu","Zixuan Wang","Dingming Li","Tongbo Chen","Peng Qiu","Yingqi Qu","Weiming Lu","Jun Xiao","Yongliang Shen"],"day":"2026-11-15","rank":6,"score":74,"status":"written","summary":{"takeaway":"A SI model that is bad at acting in a long task can still be a good teacher, because judging actions after the episode ends shows their real consequences.","found":"The authors study \"decision models,\" which give a typed answer such as a choice among options with a probability for each. One such model can pick actions, judge past actions, or score every action to train a policy with reinforcement learning. On ALFWorld, they tested three decision models and three general LLMs as actors and judges. As actors, all six were pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credited the steps toward that subgoal. Asking whether each action helped, instead of whether the task will succeed, separated useful from useless actions. As a teacher, JEV raised a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop.","matters":"Earlier work mostly judged decision models by how well they act. This paper suggests that is the wrong test if the goal is to train other models. The authors argue that judging an action after the episode ends reveals consequences that an actor must guess in advance. If that holds, models that look weak as agents may still be useful sources of training signal for long tasks.","race":"If the result holds, builders of agent SI could get stronger training signals from models they would have written off as poor actors. That could help small policies improve on long tasks without needing a top-performing actor as the guide. The abstract reports only two benchmarks and one small policy, so the reach is not yet clear.","safety":"The paper does not address safety directly. It does bear on how SI agents are trained, since judge models that score each action shape what a policy learns.","level":"intermediate","tags":["agents","reinforcement-learning","training","evals","language-models","reasoning"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Haochen Liu, Qianhao Zhu, Zixuan Wang, Dingming Li, Tongbo Chen, Peng Qiu, Yingqi Qu, Weiming Lu, Jun Xiao, Yongliang Shen

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 18, 2026 · Number 7GPRM: Global Perspective Process Reward ModelGPRM is a step-by-step reasoning judge that sees earlier steps, their scores, and later steps, and the authors report it beats strong baselines on PRMBench and ProcessBench with less data and smaller models.ReasoningEvalsReinforcement learning](https://thesimind.si/papers/2611.20755/) · [Nov 19, 2026 · Number 8WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning LoopsWATERFALL is a workflow that lets SI models write and improve reward rules for reinforcement learning by watching video of agent behavior, with no task description or hidden information.Reinforcement learningAgentsMultimodal](https://thesimind.si/papers/2611.22854/) · [Nov 16, 2026 · Number 12Benchmarks Don't Talk Back: Learning Through Self-DisagreementAbbasloo and Cheng argue that knowing when to hold an answer and when to yield is its own skill, and that a model can train it by arguing against itself.AlignmentReinforcement learningLanguage models](https://thesimind.si/papers/2611.17563/)

[Number 5](https://thesimind.si/papers/2611.16991/) · [All 12 for Nov 15, 2026](https://thesimind.si/papers/day/2026-11-15/) · [Number 7](https://thesimind.si/papers/2611.16123/)
