What they found
The authors study two common interpretability measurements in toy models and synthetic dictionaries where the true answer is known: feature dimensionality and the feature recovery rate (FRR) with a cosine threshold. They show exactly where feature dimensionality is discontinuous, meaning it can jump with tiny changes. They replace it with a ridge (regularized) version that has sharp stability bounds. The ridge leverage score changes by at most 3sqrt3/(8sqrtλ) times the operator-norm change of the weights, and this constant cannot be improved. They also prove certificates that say when a claim "feature i is recovered" must survive a given change of the dictionary. The certificates held in every test and are informative for small perturbations. They certify no feature non-trivially across independently trained dictionaries. At the standard threshold, whether a feature counted as recovered changed with the training seed for 55 percent of features.
Why it matters
Interpretability results are used to support safety claims, such as "this dictionary found most of the model's features." This paper shows that such claims can depend on arbitrary thresholds, training seeds and small weight changes. It gives a way to say when a claim is stable and when it is not. That helps readers judge how much weight such claims deserve.
How it changes the race
This is a methods and rigor paper, not a capability result. If it holds beyond toy models, teams building frontier SI would need to report stability checks with their interpretability claims, and some current claims may look weaker.
Safety and control
The paper bears directly on safety evidence. It warns that interpretability-based safety claims, like feature recovery rates, can reverse under a different seed or threshold. The experiments use toy models only, so the abstract does not show how this applies to large SI systems. The authors also state that SI agents drafted the proofs, code and text, with an independent blind SI reviewer and numerical tests as checks.
The abstract, by the authors
Interpretability tools increasingly support safety claims, such as that a sparse dictionary has recovered most of a model's features. Such claims rest on measurements with arbitrary thresholds, and small changes to the model weights, a different training seed or a different threshold can reverse them. We study two widely used measurements in toy models and synthetic dictionaries with known ground truth: the feature dimensionality of Elhage et al. and the feature recovery rate (FRR) with a cosine threshold. We characterize exactly where feature dimensionality is discontinuous and replace it with a regularized (ridge) version with sharp stability bounds: the ridge leverage score changes by at most 3sqrt3/(8sqrtλ) times the operator-norm change of the weights, and this constant cannot be improved; for the regularized dimensionality the stability constant grows like ρ/4 in the scale ρ=R/sqrtλ, with explicit upper and lower bounds. We also prove certificates stating when a claim "feature i is recovered" must survive a given change of the dictionary. The certificates held in every test and are informative for small perturbations, but certify no feature non-trivially across independently trained dictionaries; at the standard threshold, whether a feature counts as recovered changed with the training seed for 55 percent of features. Experiments use toy models only. The proofs, code and text were drafted by AI agents directed by the authors and checked by an independent blind AI reviewer and by numerical tests; the authors take full responsibility for the content.
Quoted from arXiv:2611.16190. The research and its words belong to its authors.
