---
title: "Can Mechanistic Interpretability Help... | SI Paper Desk"
url: https://thesimind.si/papers/2611.18514/
description: "Tools that look inside language models can spot and partly fix a web agent's made-up actions, but they explain the failure only in part and break under..."
---

Number 5 of 12 · November 17, 2026

# Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

Tools that look inside language models can spot and partly fix a web agent's made-up actions, but they explain the failure only in part and break under small prompt changes.

Mohamed Aghzal, Mor Geva, Gregory J. Stein and 1 more · Submitted November 13, 2026 · arXiv 2611.18514v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.18514) · [The PDF](https://arxiv.org/pdf/2611.18514) · [The day's 12](https://thesimind.si/papers/day/2026-11-17/)

## What they found

The authors studied one failure: web agents that try actions a website does not offer. They ran several interpretability analyses on Qwen2.5-7B-Instruct while it used live websites. A simple linear probe (a small classifier) on the last prompt token predicted hallucinations before generation began, with 0.80 AUC. Steering the model along the probe direction corrected 85% of hallucinations but disrupted 7% of valid actions. The authors say this is only a partial explanation. The probe direction was sensitive to prompt formatting and lost much of its steering effect after a minor change to how actions are shown. Circuit finding was hard because the computation is spread across many components and positions, and the information the agent uses differs from example to example.

## Why it matters

Interpretability aims to explain why SI systems do what they do. This paper tests that aim on a realistic agent failure instead of a clean lab task. It shows where standard methods for finding circuits and features run into trouble. The authors outline future directions for making these methods better suited to agent failures. The abstract reports one model and one failure type, so how far the findings carry over is not stated.

## How it changes the race

This is a case study and does not change capability or cost for frontier builders. It suggests that interpretability tools may need rework before they can explain the failures of SI agents in real settings.

## Safety and control

The paper bears on human understanding and control of SI agents. A probe that flags likely errors before they happen is a useful signal. But the fragile steering and the partial explanation suggest these tools should not yet be trusted as reliable safeguards.

### The abstract, by the authors

Mechanistic Interpretability (MI) has made significant progress in explaining the internal computations of Large Language Models (LLMs); however, these advances have yet to demonstrate comparable success in explaining the decision-making processes of LLM-based agents. This paper examines the challenges MI tools face when applied to realistic agentic failures. We focus on one such failure: web agents hallucinating actions that are not available on a website. We apply a series of mechanistic analyzes to Qwen2.5-7B-Instruct as it interacts with live websites. We find that a linear probe trained on the last prompt token can predict hallucinations before generation begins (0.80 AUC) and steering along a probe direction corrects 85% of hallucinations, but disrupts 7% of valid actions. However, these results only provide a partial explanation of why the agent hallucinates. The probe direction is sensitive to prompt formatting, losing much of its steering effect after a minor change to how actions are represented. Additionally, circuit identification can be difficult as the computation is distributed across multiple components and positions, and the information the agent relies on varies across examples. We discuss how these observations challenge the assumptions underlying standard MI methods for circuit discovery and feature identification, and outline future directions for making these methods better suited to explain failures in LLM-based agents.

Quoted from arXiv:2611.18514. The research and its words belong to its authors.

Desk score **76** out of 100

Why it was picked

- Interpretability (in the title)
- Agents (in the title)
- Language models (in the title)
- Studies frontier models (Qwen2.5)
- Cross-listed in 2 fields

Topics

- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Safety](https://thesimind.si/papers/?tag=safety)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Evals](https://thesimind.si/papers/?tag=evals)

**arXiv ID:** 2611.18514v1
**Submitted:** November 13, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.CL (Computation and Language)
**Level:** Intermediate

Fingerprint, SHA-256

`e87316be6a8dad723c5a0d1fa5462c55047d6ba7b84c3a1a2ae820939139fc11`

On Solana: [Proof of Mind seal, 2HNJWs...cNY9](https://solscan.io/tx/2HNJWsLa5Tp3F7wQaNr1CPR71N1aoGNptA7fgCafC5m3BwbmKwhNGTXyaSfyEa9jqug6kM55u5VmDvbSrRepcNY9)

### The exact text fingerprinted

{"arxiv":"2611.18514","version":1,"title":"Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination","authors":["Mohamed Aghzal","Mor Geva","Gregory J. Stein","Ziyu Yao"],"day":"2026-11-17","rank":5,"score":76,"status":"written","summary":{"takeaway":"Tools that look inside language models can spot and partly fix a web agent's made-up actions, but they explain the failure only in part and break under small prompt changes.","found":"The authors studied one failure: web agents that try actions a website does not offer. They ran several interpretability analyses on Qwen2.5-7B-Instruct while it used live websites. A simple linear probe (a small classifier) on the last prompt token predicted hallucinations before generation began, with 0.80 AUC. Steering the model along the probe direction corrected 85% of hallucinations but disrupted 7% of valid actions. The authors say this is only a partial explanation. The probe direction was sensitive to prompt formatting and lost much of its steering effect after a minor change to how actions are shown. Circuit finding was hard because the computation is spread across many components and positions, and the information the agent uses differs from example to example.","matters":"Interpretability aims to explain why SI systems do what they do. This paper tests that aim on a realistic agent failure instead of a clean lab task. It shows where standard methods for finding circuits and features run into trouble. The authors outline future directions for making these methods better suited to agent failures. The abstract reports one model and one failure type, so how far the findings carry over is not stated.","race":"This is a case study and does not change capability or cost for frontier builders. It suggests that interpretability tools may need rework before they can explain the failures of SI agents in real settings.","safety":"The paper bears on human understanding and control of SI agents. A probe that flags likely errors before they happen is a useful signal. But the fragile steering and the partial explanation suggest these tools should not yet be trusted as reliable safeguards.","level":"intermediate","tags":["interpretability","agents","safety","language-models","evals"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Mohamed Aghzal, Mor Geva, Gregory J. Stein, Ziyu Yao

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 16, 2026 · Paper of the DayOnly as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-AgentWhen a strong SI model hands work to a cheaper sub-agent, the whole system can be only as safe as the weaker agent, even if the strong model alone resists the attack.SafetyMulti-agent systemsSecurity](https://thesimind.si/papers/2611.17520/) · [Nov 20, 2026 · Number 2AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM AgentsAutoPI is a red teaming tool that tests SI agents for prompt injection across many attack types, not just how often attacks work, and it reports higher success and coverage than recent rivals.SecurityAgentsEvals](https://thesimind.si/papers/2611.24313/) · [Nov 11, 2026 · Paper of the DayMitigating Agentic Deception via Evaluation-Side Partially Observable GamesThe authors treat SI agent deception as a game between an agent and a monitor that checks the whole interaction, and report fewer deceptive behaviors and better task success in tests on small models.SafetyAlignmentAgents](https://thesimind.si/papers/2611.12835/)

[Number 4](https://thesimind.si/papers/2611.18494/) · [All 12 for Nov 17, 2026](https://thesimind.si/papers/day/2026-11-17/) · [Number 6](https://thesimind.si/papers/2611.18346/)
