Number 7 of 12 · November 16, 2026

DECODE: Diagnosing Interpretability Failures in Automated Interpretation Process

DECODE is a visual tool that puts humans back into automated SI interpretation, helping find where explanations of a model's inner features go wrong and why.

What they found

The authors build DECODE, a visual analytics system (an interactive tool with charts and views) for checking automated interpretation of LLMs. In these pipelines, an LLM both writes and scores explanations of what a model's internal parts do, so there is little human oversight and no ground truth. DECODE splits diagnosis into three stages, so failures can be traced from structural defects to explanation-level misalignments. It treats agreement among independently generated explanations and their evaluation metrics as a stand-in for reliability, an idea borrowed from inter-coder consensus in qualitative research. A three-phase active seeding protocol recommends which items humans should review first. In a case study on 16,384 Sparse Autoencoder features, the authors say it can efficiently find and attribute interpretability failures at scale. The abstract gives no numbers on accuracy or on how it compares with other methods.

Why it matters

Interpretability tools are meant to help people understand what is happening inside SI models. If the explanations come from another LLM and are scored by an LLM, errors can hide inside the loop. This work tackles a practical question: when an explanation looks wrong, is the fault in the tool's extracted features or in the LLM that explained them? Better ways to answer that make interpretability results easier to trust.

How it changes the race

This is a supporting tool, not a capability advance. If it holds up, it could make large-scale interpretability work more reliable and cut the human effort needed to audit it.

Safety and control

The paper bears on human oversight of SI. It aims to make automated interpretation checkable by people, which matters if interpretability is used to audit models. The abstract only reports one case study, so how well it works more broadly is not shown.

The abstract, by the authors

Tools for interpreting the internal representations of Large Language Models (LLMs) increasingly rely on LLMs themselves to both generate and evaluate explanations, creating a self-referential automated interpretation process with minimal human oversight. This automation raises significant reliability concerns, which are exacerbated by the absence of ground truth. In practice, it is difficult to validate explanations or to attribute interpretability failures to their source - whether they originate from the tool's extracted representations or from the LLM used in the process. To address these challenges, we present DECODE, a visual analytics system that reintroduces human oversight into the automated interpretation process. Our approach decomposes the diagnostic workflow into three progressive stages, isolating failure sources ranging from structural defects to explanation-level misalignments. Drawing inspiration from inter-coder consensus in qualitative analysis, we utilize consensus among independently generated explanations and their evaluation metrics as a proxy for reliability. To further scale human inspection, we employ a three-phase active seeding protocol that strategically recommends high-priority instances for manual review. Finally, we demonstrate the effectiveness of our approach through a case study on 16,384 Sparse Autoencoder features, showing that it can efficiently identify and attribute interpretability failures at scale.

Quoted from arXiv:2611.17856. The research and its words belong to its authors.

The authors

Dohyun Kim, Jaeung Lee, Donghee Hong, Jaemin Jo

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.