---
title: "WATERFALL: Workflow for Adaptive... | SI Paper Desk"
url: https://thesimind.si/papers/2611.22854/
description: "WATERFALL is a workflow that lets SI models write and improve reward rules for reinforcement learning by watching video of agent behavior, with no task..."
---

Number 8 of 12 · November 19, 2026

# WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning Loops

WATERFALL is a workflow that lets SI models write and improve reward rules for reinforcement learning by watching video of agent behavior, with no task description or hidden information.

Eleftherios Triantafyllidis, Filippos Christianos, Zhibin Li and 1 more · Submitted November 17, 2026 · arXiv 2611.22854v1 · cs.LG

[Read the paper on arXiv](https://arxiv.org/abs/2611.22854) · [The PDF](https://arxiv.org/pdf/2611.22854) · [The day's 12](https://thesimind.si/papers/day/2026-11-19/)

## What they found

The authors build WATERFALL, a loop that creates many candidate reward programs, using different "personas" to get variety. Vision-language models (SI models that read images and text) watch video of what an agent does and judge it in a Swiss-system tournament, a pairing format where similar performers meet, to pick the best candidates. The history of past judgments then guides which candidates to change, push further, or drop. The authors test it on fully observable and partially observable environments, from discrete to continuous. They report that it beats sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines, even though those baselines keep their usual information advantages, with the clearest gains in the partially observable case. Ablations show that visual evidence, tournament selection and iterative refinement all matter together.

## Why it matters

Designing rewards, the signals that tell an RL agent what counts as success, is a major bottleneck, especially when rewards are rare and tasks are long. Many recent methods quietly rely on task descriptions, demonstrations or ground-truth metrics. This paper tries to remove those aids and keep the setting strict. If it holds, agents could discover useful goals through interaction instead of depending on human-written rewards. The abstract gives no specific numbers, so the size of the gains is not clear from it.

## How it changes the race

If the result holds, it could cut the manual reward work needed to train agents in hard, partly hidden environments. That would lower the expertise needed to build such systems. The abstract reports only benchmark environments, so real-world reach is unknown.

## Safety and control

The system writes its own reward programs and judges them with vision-language models, so the rewards it learns may not match what humans want. The abstract does not discuss this, so oversight of automatically discovered rewards is an open question.

### The abstract, by the authors

Reward design remains a central bottleneck in reinforcement learning (RL), particularly in sparse-reward, long-horizon and partially observable settings requiring dependency chaining, memory, and deceptive affordance disambiguation. While recent foundation-model approaches reduce manual reward engineering, most assume goal-conditioning, task descriptions, expert demonstrations, or ground-truth metrics. These assumptions are problematic in partially observable environments, where task-relevant objectives, affordances, and dynamics must be discovered through interaction rather than disclosed a priori. We formalise this problem setting by introducing WATERFALL, an iterative population-based workflow for automated reward discovery under the strict observability constraints inherent in POMDP environments. WATERFALL achieves this by (i) synthesising diverse programmatic reward candidates via persona-conditioned generation, (ii) evaluating candidate behaviour from raw visual rollouts utilising a Swiss-system evolutionary tournament judged by vision-language models to identify elites, and (iii) leveraging longitudinal assessment histories to iteratively mutate, escalate, or prune candidates. We evaluate WATERFALL across fully observable and reveal-gated partially observable environments ranging from discrete to continuous, comparing against sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines. Our results indicate that WATERFALL discovers and refines reward programs without access to privileged information, outperforming baselines that retain their native information advantages, most clearly in the POMDP regime. Extensive ablations substantiate that visual evidence, tournament-based selection, and iterative refinement are jointly important. Enforcing this contract is what separates autonomous reward discovery from benchmark-privileged reward shaping.

Quoted from arXiv:2611.22854. The research and its words belong to its authors.

Desk score **74** out of 100

Why it was picked

- Safety
- Agents
- Reasoning
- Evals
- Accepted at NeurIPS
- Language models
- Cross-listed in 2 fields

Topics

- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Multimodal](https://thesimind.si/papers/?tag=multimodal)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Training](https://thesimind.si/papers/?tag=training)
- [Language models](https://thesimind.si/papers/?tag=language-models)

**arXiv ID:** 2611.22854v1
**Submitted:** November 17, 2026
**Fields:** cs.LG (Machine Learning), cs.AI (Artificial Intelligence)
**Level:** Advanced
**Comments:** Accepted at NeurIPS 2026 (poster). 78 pages, 13 figures, 19 tables. Code: [https://github.com/etriantafyllidis/waterfall](https://github.com/etriantafyllidis/waterfall) OpenReview: [https://openreview.net/forum?id=TI1lG64bTY](https://openreview.net/forum?id=TI1lG64bTY)

Fingerprint, SHA-256

`3c83bd5d1c00667e964e08806465b4cc0eef65ffdf3c6dc5cf6d1d958c27f545`

On Solana: [Proof of Mind seal, 3rBYfi...QvEH](https://solscan.io/tx/3rBYfiFP3PxcUbtJzCwpcVq5MngquU5iAtJFeiP7PFVgnsuoTjwwJPjNDYgKxQzUnVq5keQT2KEgX46YTdHqQvEH)

### The exact text fingerprinted

{"arxiv":"2611.22854","version":1,"title":"WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning Loops","authors":["Eleftherios Triantafyllidis","Filippos Christianos","Zhibin Li","Bernd Bickel"],"day":"2026-11-19","rank":8,"score":74,"status":"written","summary":{"takeaway":"WATERFALL is a workflow that lets SI models write and improve reward rules for reinforcement learning by watching video of agent behavior, with no task description or hidden information.","found":"The authors build WATERFALL, a loop that creates many candidate reward programs, using different \"personas\" to get variety. Vision-language models (SI models that read images and text) watch video of what an agent does and judge it in a Swiss-system tournament, a pairing format where similar performers meet, to pick the best candidates. The history of past judgments then guides which candidates to change, push further, or drop. The authors test it on fully observable and partially observable environments, from discrete to continuous. They report that it beats sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines, even though those baselines keep their usual information advantages, with the clearest gains in the partially observable case. Ablations show that visual evidence, tournament selection and iterative refinement all matter together.","matters":"Designing rewards, the signals that tell an RL agent what counts as success, is a major bottleneck, especially when rewards are rare and tasks are long. Many recent methods quietly rely on task descriptions, demonstrations or ground-truth metrics. This paper tries to remove those aids and keep the setting strict. If it holds, agents could discover useful goals through interaction instead of depending on human-written rewards. The abstract gives no specific numbers, so the size of the gains is not clear from it.","race":"If the result holds, it could cut the manual reward work needed to train agents in hard, partly hidden environments. That would lower the expertise needed to build such systems. The abstract reports only benchmark environments, so real-world reach is unknown.","safety":"The system writes its own reward programs and judges them with vision-language models, so the rewards it learns may not match what humans want. The abstract does not discuss this, so oversight of automatically discovered rewards is an open question.","level":"advanced","tags":["reinforcement-learning","agents","multimodal","evals","training","language-models"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Eleftherios Triantafyllidis, Filippos Christianos, Zhibin Li, Bernd Bickel

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 15, 2026 · Number 6Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon AgentsA SI model that is bad at acting in a long task can still be a good teacher, because judging actions after the episode ends shows their real consequences.AgentsReinforcement learningTraining](https://thesimind.si/papers/2611.17167/) · [Nov 19, 2026 · Number 5Evolution Under Fire: Evolving Agent Safety Harnesses with Adaptive Injection StressorsThe authors let an automated search build and improve the safety wrapper around an SI agent, testing it against evolving attacks, and cut attack success sharply on the benchmarks they report.SecurityAgentsSafety](https://thesimind.si/papers/2611.22583/) · [Nov 18, 2026 · Number 7GPRM: Global Perspective Process Reward ModelGPRM is a step-by-step reasoning judge that sees earlier steps, their scores, and later steps, and the authors report it beats strong baselines on PRMBench and ProcessBench with less data and smaller models.ReasoningEvalsReinforcement learning](https://thesimind.si/papers/2611.20755/)

[Number 7](https://thesimind.si/papers/2611.23010/) · [All 12 for Nov 19, 2026](https://thesimind.si/papers/day/2026-11-19/) · [Number 9](https://thesimind.si/papers/2611.22483/)
