---
title: "Tracking Representations in LLM... | SI Paper Desk"
url: https://thesimind.si/papers/2611.23403/
description: "The authors find that an SI model builds and updates an internal picture of a game while it reasons, and that once it locks onto a wrong belief, it rarely..."
---

Number 11 of 12 · November 20, 2026

# Tracking Representations in LLM Reasoning Traces for Iterated Games

The authors find that an SI model builds and updates an internal picture of a game while it reasons, and that once it locks onto a wrong belief, it rarely recovers.

Chaipat Tirapongprasert, Rajarshi Das and Parijat Dube · Submitted November 17, 2026 · arXiv 2611.23403v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.23403) · [The PDF](https://arxiv.org/pdf/2611.23403) · [The day's 12](https://thesimind.si/papers/day/2026-11-20/)

## What they found

The authors studied how SI agents make choices in repeated games where players lack full information. They used mechanistic interpretability, which means tools that look at a model's inner activations, not only its written output. They report that the model forms an internal representation of the game and revises it step by step during its chain-of-thought (the written reasoning before an answer). They also report that once the model commits too early to an incorrect belief, it rarely recovers. This failure shows up both in the written trace and in the layer activations. The abstract gives no numbers, model names or comparisons.

## Why it matters

People often read an SI model's written reasoning as a window into how it decides. The abstract says little is understood about how that trace relates to the model's real computation. This work links the two in one setting, iterated games. It suggests that errors in strategic reasoning can be traced to specific points where a wrong belief took hold. That could help diagnose why multi-agent SI systems fail.

## How it changes the race

If it holds beyond these games, builders could use internal tracking to find where reasoning goes wrong, not only check final answers. This is a diagnostic step, not a new capability, and the abstract does not say how far the results generalize.

## Safety and control

The paper bears on oversight. It looks at how far a visible reasoning trace matches what happens inside the model, and it offers a way to diagnose strategic errors in multi-agent SI. The abstract does not claim it can detect deception or control behavior.

### The abstract, by the authors

Large language model (LLM) agents achieve state-of-the-art performance across domains through chain-of-thought (CoT) reasoning. Despite their ubiquitous use, little is understood about the relationship between the externalized trace and the model's actual computation. We address this by investigating how LLM agents make decisions in iterative games with incomplete information. Using a series of mechanistic interpretability techniques, we find that the model forms and iteratively revises an internal representation of the game throughout the CoT. Once the model prematurely commits to an incorrect belief, it rarely recovers. This failure is manifested in both the literal trace and the layer activations. Ultimately, as multi-agent systems are more prevalent, our work opens the door to a more focused diagnosis of their strategic reasoning errors. For illustrative purposes, we offer an interactive visualizer of our methodology on selected sample traces at https://interp-self.github.io/gametrace/

Quoted from arXiv:2611.23403. The research and its words belong to its authors.

Desk score **76** out of 100

Why it was picked

- Reasoning (in the title)
- Interpretability
- Agents
- Language models (in the title)
- Cross-listed in 3 fields
- Multi-agent

Topics

- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)
- [Multi-agent systems](https://thesimind.si/papers/?tag=multi-agent)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Safety](https://thesimind.si/papers/?tag=safety)

**arXiv ID:** 2611.23403v1
**Submitted:** November 17, 2026
**Fields:** cs.CL (Computation and Language), cs.AI (Artificial Intelligence), cs.GT (Computer Science and Game Theory)
**Level:** Intermediate
**Comments:** Accepted to SoCalNLP 2026. 15 pages, 11 figures

Fingerprint, SHA-256

`a6aa499350bf71be6c02891066bd7eb129f50250cd8703d377781b4453fd6b44`

On Solana: [Proof of Mind seal, 4JtzxB...6w5n](https://solscan.io/tx/4JtzxBBunUdHpdT7ydTygvUny81ph9bmXUMt9UHMJBLFrSwBZFdTbtV5CcZdHAM2kg7Bc2pTnq59P4DvXyGc6w5n)

### The exact text fingerprinted

{"arxiv":"2611.23403","version":1,"title":"Tracking Representations in LLM Reasoning Traces for Iterated Games","authors":["Chaipat Tirapongprasert","Rajarshi Das","Parijat Dube"],"day":"2026-11-20","rank":11,"score":76,"status":"written","summary":{"takeaway":"The authors find that an SI model builds and updates an internal picture of a game while it reasons, and that once it locks onto a wrong belief, it rarely recovers.","found":"The authors studied how SI agents make choices in repeated games where players lack full information. They used mechanistic interpretability, which means tools that look at a model's inner activations, not only its written output. They report that the model forms an internal representation of the game and revises it step by step during its chain-of-thought (the written reasoning before an answer). They also report that once the model commits too early to an incorrect belief, it rarely recovers. This failure shows up both in the written trace and in the layer activations. The abstract gives no numbers, model names or comparisons.","matters":"People often read an SI model's written reasoning as a window into how it decides. The abstract says little is understood about how that trace relates to the model's real computation. This work links the two in one setting, iterated games. It suggests that errors in strategic reasoning can be traced to specific points where a wrong belief took hold. That could help diagnose why multi-agent SI systems fail.","race":"If it holds beyond these games, builders could use internal tracking to find where reasoning goes wrong, not only check final answers. This is a diagnostic step, not a new capability, and the abstract does not say how far the results generalize.","safety":"The paper bears on oversight. It looks at how far a visible reasoning trace matches what happens inside the model, and it offers a way to diagnose strategic errors in multi-agent SI. The abstract does not claim it can detect deception or control behavior.","level":"intermediate","tags":["interpretability","reasoning","multi-agent","agents","language-models","safety"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Chaipat Tirapongprasert, Rajarshi Das, Parijat Dube

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 11Explaining and interrupting collective misalignment in language model systemsThe author presents Collective Dependence Analysis, a method that finds which messages between SI agents enabled a policy violation, so removing a few of them can stop it.SafetyMulti-agent systemsAlignment](https://thesimind.si/papers/2611.11724/) · [Nov 16, 2026 · Paper of the DayOnly as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-AgentWhen a strong SI model hands work to a cheaper sub-agent, the whole system can be only as safe as the weaker agent, even if the strong model alone resists the attack.SafetyMulti-agent systemsSecurity](https://thesimind.si/papers/2611.17520/) · [Nov 16, 2026 · Number 2Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM AgentsSoteria checks an SI agent's plan against written rules before any tool runs, then enforces those rules during execution. The authors report more completed tasks and no policy violations.AgentsSafetyReasoning](https://thesimind.si/papers/2611.17880/)

[Number 10](https://thesimind.si/papers/2611.24609/) · [All 12 for Nov 20, 2026](https://thesimind.si/papers/day/2026-11-20/) · [Number 12](https://thesimind.si/papers/2611.24765/)
