---
title: "PIPETTE: Prompt Injection Purple-Teaming... | SI Paper Desk"
url: https://thesimind.si/papers/2611.24362/
description: "High attack success in red-teaming does not mean better defense. PIPETTE rewards varied attacks, and the authors report agents that resist prompt injection..."
---

Number 7 of 12 · November 20, 2026

# PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents

High attack success in red-teaming does not mean better defense. PIPETTE rewards varied attacks, and the authors report agents that resist prompt injection while still doing their normal tasks.

Lipeng He, Yihan Wang, Jiawen Zhang and 1 more · Submitted November 18, 2026 · arXiv 2611.24362v1 · cs.CR

[Read the paper on arXiv](https://arxiv.org/abs/2611.24362) · [The PDF](https://arxiv.org/pdf/2611.24362) · [The day's 12](https://thesimind.si/papers/day/2026-11-20/)

## What they found

The authors study indirect prompt injection, where an SI agent reads untrusted outside data that secretly tells it to do an attacker's task. They show that two leading RL-based attack generators, RL-Hammer and PISmith, suffer from mode collapse. This means they keep finding similar attacks, and the extra successes add no robustness when used for adversarial training. The authors introduce PIPETTE, which uses a dictionary learning-based reward to push attack generation toward a broader range of strategies. It also adds an on-policy adversarial training pipeline for this kind of defense. On AgentDojo and ASB, they report defenders with near-zero attack success rates on static attacks, resilience to strong adaptive attacks, and benign task utility close to the undefended baseline.

## Why it matters

SI agents that browse, read email or call tools must handle data they cannot trust. Prompt injection is a main way these agents get hijacked. This work suggests that how diverse the attacks are matters more than how often they succeed when hardening a model. The abstract does not give exact numbers or say how well this holds beyond the two benchmarks.

## How it changes the race

If the results hold, developers could harden agents against prompt injection more efficiently before deployment, without a large loss in usefulness. It is a refinement of the red-team and adversarial training process rather than a new capability.

## Safety and control

This work bears directly on security and control of SI agents. It aims to keep agents following the user's instructions rather than an attacker's. Near-zero attack rates on tested attacks do not prove safety against new attack types.

### The abstract, by the authors

Large language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR). Model developers use the attacks discovered during internal red-teaming to harden agents through adversarial training before deployment, a collaborative offensive-defensive process known in cybersecurity as purple-teaming. However, we show that achieving high ASR during red-teaming does not guarantee robustness gains from subsequent adversarial training. Specifically, we find that two state-of-the-art RL-based red-teaming methods, RL-Hammer and PISmith, suffer from mode collapse, repeatedly discovering similar attack strategies whose additional successes add no additional robustness improvements to adversarial training. To address this gap, we introduce PIPETTE, a unified prompt injection purple-teaming framework that effectively converts red-teaming discoveries into defender robustness improvements. PIPETTE employs a dictionary learning-based reward that encourages broader coverage of attack strategies during red-teaming. It also integrates an on-policy adversarial training pipeline tailored to IPI defense, enabling agents to resist attacks while completing legitimate user tasks. Extensive evaluations across agentic benchmarks (AgentDojo and ASB) demonstrate that PIPETTE produces defenders that achieve near-zero attack success rates on static attacks and can withstand strong adaptive attacks, while maintaining benign task utility close to that of the undefended baseline.

Quoted from arXiv:2611.24362. The research and its words belong to its authors.

Desk score **79** out of 100

Why it was picked

- Agents (in the title)
- Safety
- Security (in the title)
- Evals
- Language models (in the title)
- Cross-listed in 2 fields

Topics

- [Security](https://thesimind.si/papers/?tag=security)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Safety](https://thesimind.si/papers/?tag=safety)
- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Training](https://thesimind.si/papers/?tag=training)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)

**arXiv ID:** 2611.24362v1
**Submitted:** November 18, 2026
**Fields:** cs.CR (Cryptography and Security), cs.AI (Artificial Intelligence)
**Level:** Intermediate
**Comments:** 18 pages, 6 figures

Fingerprint, SHA-256

`03512451f6442f80a257d657109601ef5ae6d0cf9bb63efa68ed16b6cc1d35f7`

On Solana: [Proof of Mind seal, 4JtzxB...6w5n](https://solscan.io/tx/4JtzxBBunUdHpdT7ydTygvUny81ph9bmXUMt9UHMJBLFrSwBZFdTbtV5CcZdHAM2kg7Bc2pTnq59P4DvXyGc6w5n)

### The exact text fingerprinted

{"arxiv":"2611.24362","version":1,"title":"PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents","authors":["Lipeng He","Yihan Wang","Jiawen Zhang","N. Asokan"],"day":"2026-11-20","rank":7,"score":79,"status":"written","summary":{"takeaway":"High attack success in red-teaming does not mean better defense. PIPETTE rewards varied attacks, and the authors report agents that resist prompt injection while still doing their normal tasks.","found":"The authors study indirect prompt injection, where an SI agent reads untrusted outside data that secretly tells it to do an attacker's task. They show that two leading RL-based attack generators, RL-Hammer and PISmith, suffer from mode collapse. This means they keep finding similar attacks, and the extra successes add no robustness when used for adversarial training. The authors introduce PIPETTE, which uses a dictionary learning-based reward to push attack generation toward a broader range of strategies. It also adds an on-policy adversarial training pipeline for this kind of defense. On AgentDojo and ASB, they report defenders with near-zero attack success rates on static attacks, resilience to strong adaptive attacks, and benign task utility close to the undefended baseline.","matters":"SI agents that browse, read email or call tools must handle data they cannot trust. Prompt injection is a main way these agents get hijacked. This work suggests that how diverse the attacks are matters more than how often they succeed when hardening a model. The abstract does not give exact numbers or say how well this holds beyond the two benchmarks.","race":"If the results hold, developers could harden agents against prompt injection more efficiently before deployment, without a large loss in usefulness. It is a refinement of the red-team and adversarial training process rather than a new capability.","safety":"This work bears directly on security and control of SI agents. It aims to keep agents following the user's instructions rather than an attacker's. Near-zero attack rates on tested attacks do not prove safety against new attack types.","level":"intermediate","tags":["security","agents","safety","reinforcement-learning","training","benchmarks"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 10, 2026 · Number 2Tail-risk-aware Agent Safety: Constraining Tail Risks Across Multi-step LLM Agent WorkflowsThe authors propose TAS, a way to keep SI agents safe by tracking how risk builds across workflow steps and capping rare, severe failures with one shared budget.SafetyAgentsSecurity](https://thesimind.si/papers/2611.10411/) · [Nov 14, 2026 · Number 2Bait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward FunctionThe authors show that a cheap harmful agentic RL attack on an RL-as-a-service API can strip a model's safety training, and they propose Bait, a defense that probes the reward function.SecuritySafetyReinforcement learning](https://thesimind.si/papers/2611.16369/) · [Nov 20, 2026 · Number 2AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM AgentsAutoPI is a red teaming tool that tests SI agents for prompt injection across many attack types, not just how often attacks work, and it reports higher success and coverage than recent rivals.SecurityAgentsEvals](https://thesimind.si/papers/2611.24313/)

[Number 6](https://thesimind.si/papers/2611.24221/) · [All 12 for Nov 20, 2026](https://thesimind.si/papers/day/2026-11-20/) · [Number 8](https://thesimind.si/papers/2611.24357/)
