Number 11 of 12 · November 20, 2026

Tracking Representations in LLM Reasoning Traces for Iterated Games

The authors find that an SI model builds and updates an internal picture of a game while it reasons, and that once it locks onto a wrong belief, it rarely recovers.

What they found

The authors studied how SI agents make choices in repeated games where players lack full information. They used mechanistic interpretability, which means tools that look at a model's inner activations, not only its written output. They report that the model forms an internal representation of the game and revises it step by step during its chain-of-thought (the written reasoning before an answer). They also report that once the model commits too early to an incorrect belief, it rarely recovers. This failure shows up both in the written trace and in the layer activations. The abstract gives no numbers, model names or comparisons.

Why it matters

People often read an SI model's written reasoning as a window into how it decides. The abstract says little is understood about how that trace relates to the model's real computation. This work links the two in one setting, iterated games. It suggests that errors in strategic reasoning can be traced to specific points where a wrong belief took hold. That could help diagnose why multi-agent SI systems fail.

How it changes the race

If it holds beyond these games, builders could use internal tracking to find where reasoning goes wrong, not only check final answers. This is a diagnostic step, not a new capability, and the abstract does not say how far the results generalize.

Safety and control

The paper bears on oversight. It looks at how far a visible reasoning trace matches what happens inside the model, and it offers a way to diagnose strategic errors in multi-agent SI. The abstract does not claim it can detect deception or control behavior.

The abstract, by the authors

Large language model (LLM) agents achieve state-of-the-art performance across domains through chain-of-thought (CoT) reasoning. Despite their ubiquitous use, little is understood about the relationship between the externalized trace and the model's actual computation. We address this by investigating how LLM agents make decisions in iterative games with incomplete information. Using a series of mechanistic interpretability techniques, we find that the model forms and iteratively revises an internal representation of the game throughout the CoT. Once the model prematurely commits to an incorrect belief, it rarely recovers. This failure is manifested in both the literal trace and the layer activations. Ultimately, as multi-agent systems are more prevalent, our work opens the door to a more focused diagnosis of their strategic reasoning errors. For illustrative purposes, we offer an interactive visualizer of our methodology on selected sample traces at https://interp-self.github.io/gametrace/

Quoted from arXiv:2611.23403. The research and its words belong to its authors.

The authors

Chaipat Tirapongprasert, Rajarshi Das, Parijat Dube

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.