---
title: "FOE-Dreamer: Deployment-Efficient... | SI Paper Desk"
url: https://thesimind.si/papers/2611.14880/
description: "FOE-Dreamer trains a cyber defense SI directly in a live test network, not a simulator, and reports about 2× better episode loss than Rainbow and IQN..."
---

Number 11 of 12 · November 13, 2026

# FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks

FOE-Dreamer trains a cyber defense SI directly in a live test network, not a simulator, and reports about 2× better episode loss than Rainbow and IQN baselines within 3 days on one GPU.

Yinuo Du and Christopher Kiekintveld · Submitted November 11, 2026 · arXiv 2611.14880v1 · cs.LG

[Read the paper on arXiv](https://arxiv.org/abs/2611.14880) · [The PDF](https://arxiv.org/pdf/2611.14880) · [The day's 12](https://thesimind.si/papers/day/2026-11-13/)

## What they found

The authors built FOE-Dreamer, which learns a defender by first learning a model of the network (a "world model") and then practicing inside that model. It splits what the model tracks into three parts: effects of the defender's own actions, normal user activity, and a guess about the attacker made only from signals the defender can see. This split lowers the variance (random noise) in the policy gradients, the signals used to improve the defender. They tested it on an 8-host OpenStack testbed in the cloud, with two scripted attacker profiles and user activity emulated by the GHOSTS framework. It trained within a 3-day budget on a single GPU and reached about 2× better episode loss than Rainbow and IQN baselines under matched compute. The authors say both the factoring and the opponent model add to the gain, and the learned latents match the intended separation.

## Why it matters

Most cyber defense SI is trained in simulators, which miss things like login delays, partly failed actions, service dependencies and real user traffic. Policies can then perform worse on real networks. Training in the real network avoids that gap, but each step is slow and costly, so learning must need few interactions. This work suggests a model-based method can make that practical, at least on a small testbed. The abstract does not report results on larger networks or on attackers that are not scripted.

## How it changes the race

If it holds up beyond this small testbed, it could lower the cost of building defenders that learn in real environments, instead of relying on simulator quality. It is an applied step in sample efficiency for security agents, not a change in general frontier capability.

## Safety and control

The paper bears on autonomous cyber defense. Letting a learning agent act online in a live network raises control questions, and the abstract does not say how mistakes during learning are limited. The tests used scripted attackers in a testbed.

### The abstract, by the authors

Autonomous cyber defense is widely studied with deep reinforcement learning, but most work trains policies inside simulators because learning requires many trial-and-error interactions. Simulator-based training carries a fidelity gap: trained policies face authentication delays, partial action failures, service dependencies, and user traffic patterns that the simulator did not represent, and degrade on transfer. Training directly in the network being defended removes the gap but makes every interaction slow and costly, so sample efficiency becomes the binding constraint. Model-based reinforcement learning has reduced sample requirements on games and physical robots by orders of magnitude by planning inside a learned world model, but it has not been applied to defenders trained in operational networks. We present FOE-Dreamer (Factored Opponent-Enhanced Dreamer), a model-based reinforcement learning framework that trains a defender online and directly in a network on cloud infrastructure, without a simulator stage. FOE-Dreamer separates three sources of variation in the latent state: a controllable component driven by defender actions, an exogenous component driven by benign user activity, and an opponent embedding inferred from defender-observable signals alone. Separating the components reduces the variance in policy gradients during planning in imagination. Against two scripted attacker profiles with user activity emulated by the GHOSTS framework on an 8-host OpenStack testbed, FOE-Dreamer trains within a 3-day budget on a single GPU and achieves about 2× better episode loss than Rainbow and IQN baselines under matched compute. Both the factoring and the opponent model contribute to the gain, and the learned latents are consistent with the intended separation.

Quoted from arXiv:2611.14880. The research and its words belong to its authors.

Desk score **65** out of 100

Why it was picked

- Security (in the title)
- Agents
- Reasoning
- Compute
- Cross-listed in 2 fields

Topics

- [Security](https://thesimind.si/papers/?tag=security)
- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)
- [Training](https://thesimind.si/papers/?tag=training)

**arXiv ID:** 2611.14880v1
**Submitted:** November 11, 2026
**Fields:** cs.LG (Machine Learning), cs.CR (Cryptography and Security)
**Level:** Intermediate
**Comments:** 14 pages, 5 figures. To appear in the proceedings of the Annual Computer Security Applications Conference (ACSAC)

Fingerprint, SHA-256

`6f09e9005ec0abaf84614e701c165b56246646735c4e12545e2f10f7c7e6e018`

On Solana: [Proof of Mind seal, 5PD2kk...Cqj6](https://solscan.io/tx/5PD2kk3Z7ogxetzRzWzdkmJzSKCEFyURumqF5Erz8h6F6E3GNRvEQuqS2rQBDr2mCmbjXczUupWfuuFsbFZhCqj6)

### The exact text fingerprinted

{"arxiv":"2611.14880","version":1,"title":"FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks","authors":["Yinuo Du","Christopher Kiekintveld"],"day":"2026-11-13","rank":11,"score":65,"status":"written","summary":{"takeaway":"FOE-Dreamer trains a cyber defense SI directly in a live test network, not a simulator, and reports about 2× better episode loss than Rainbow and IQN baselines within 3 days on one GPU.","found":"The authors built FOE-Dreamer, which learns a defender by first learning a model of the network (a \"world model\") and then practicing inside that model. It splits what the model tracks into three parts: effects of the defender's own actions, normal user activity, and a guess about the attacker made only from signals the defender can see. This split lowers the variance (random noise) in the policy gradients, the signals used to improve the defender. They tested it on an 8-host OpenStack testbed in the cloud, with two scripted attacker profiles and user activity emulated by the GHOSTS framework. It trained within a 3-day budget on a single GPU and reached about 2× better episode loss than Rainbow and IQN baselines under matched compute. The authors say both the factoring and the opponent model add to the gain, and the learned latents match the intended separation.","matters":"Most cyber defense SI is trained in simulators, which miss things like login delays, partly failed actions, service dependencies and real user traffic. Policies can then perform worse on real networks. Training in the real network avoids that gap, but each step is slow and costly, so learning must need few interactions. This work suggests a model-based method can make that practical, at least on a small testbed. The abstract does not report results on larger networks or on attackers that are not scripted.","race":"If it holds up beyond this small testbed, it could lower the cost of building defenders that learn in real environments, instead of relying on simulator quality. It is an applied step in sample efficiency for security agents, not a change in general frontier capability.","safety":"The paper bears on autonomous cyber defense. Letting a learning agent act online in a live network raises control questions, and the abstract does not say how mistakes during learning are limited. The tests used scripted attackers in a testbed.","level":"intermediate","tags":["security","reinforcement-learning","agents","efficiency","training"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Yinuo Du, Christopher Kiekintveld

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 20, 2026 · Number 7PIPETTE: Prompt Injection Purple-Teaming for Robust LLM AgentsHigh attack success in red-teaming does not mean better defense. PIPETTE rewards varied attacks, and the authors report agents that resist prompt injection while still doing their normal tasks.SecurityAgentsSafety](https://thesimind.si/papers/2611.24362/) · [Nov 17, 2026 · Number 2Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game AgentGamer is an SI agent trained on tens of thousands of hours of keyboard and mouse play. The authors say it plays 3D open-world games with efficiency comparable to skilled players.AgentsReinforcement learningMultimodal](https://thesimind.si/papers/2611.18397/) · [Nov 14, 2026 · Number 2Bait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward FunctionThe authors show that a cheap harmful agentic RL attack on an RL-as-a-service API can strip a model's safety training, and they propose Bait, a defense that probes the reward function.SecuritySafetyReinforcement learning](https://thesimind.si/papers/2611.16369/)

[Number 10](https://thesimind.si/papers/2611.14068/) · [All 12 for Nov 13, 2026](https://thesimind.si/papers/day/2026-11-13/) · [Number 12](https://thesimind.si/papers/2611.14244/)
