What they found
The authors study Cross-Layer Transcoders (CLTs), tools that break a language model's internal activations into human-readable features and trace how they link across layers. They propose Cross-Layer TopK, which sets a single L0 budget (the count of active features) for the whole model. It then moves that budget between layers during training, since some layers, especially intermediate ones, are harder to reconstruct. On Qwen3-8B-Base, they report lower FVU (a measure of reconstruction error) while using 20% fewer active latents than JumpReLU. They also report more complete circuits, with higher replacement and completeness scores. Pairing the method with sparse decoding gives a 2.9x speedup per training step, and the abstract says it follows scaling laws in latent size and sparsity.
Why it matters
Interpretability tools only help if they are accurate and affordable to train. A method that reconstructs a model's inner workings better with fewer active features could make circuit tracing in large models more practical. The authors have also released the weights of their CLT, so others can use and check it. The abstract does not give details on how the circuits were judged beyond the stated scores.
How it changes the race
This is an efficiency and tooling gain for interpretability work. If it holds, it lowers the cost of building CLTs for large models, which may let more groups study frontier-scale SI systems.
Safety and control
Better circuit tracing can help people understand why SI models behave as they do, which supports oversight. The abstract does not test any safety use directly.
The abstract, by the authors
Cross-Layer Transcoders (CLTs) decompose the activations of large language models into human-interpretable features and trace how these features interact across layers to form circuits, making them a key tool for mechanistic interpretability. However, training CLTs effectively and efficiently is hard. Beyond the usual trade-off between sparsity and reconstruction quality, a CLT has to model all layers at once, which adds a further challenge. The common JumpReLU recipe controls sparsity only indirectly through a penalty, while the direct extension of TopK selects a fixed number of latents in every layer. Yet reconstruction difficulty differs widely across layers, and intermediate layers are much harder than others, so giving every layer the same budget is suboptimal. We propose Cross-Layer TopK, which sets a single L0 budget for the whole model and dynamically allocates it across layers during training. In a comparison with JumpReLU and other TopK variants on Qwen3-8B-Base, this gives lower FVU while using 20% fewer active latents than JumpReLU. The better reconstruction carries over to circuit tracing, yielding more complete circuits with higher replacement and completeness scores. The fixed per-token budget of Cross-Layer TopK also makes it easy to pair with sparse decoding, which cuts both computation and communication and gives a 2.9x speedup per training step. We further show that Cross-Layer TopK follows scaling laws in both latent size and sparsity. We have open-sourced the weights of our CLT at https://huggingface.co/ByteDance-Seed/Cross-Layer-TopK-CLT.
Quoted from arXiv:2611.09233. The research and its words belong to its authors.