---
title: "Cross-Layer TopK: Dynamically Allocating... | SI Paper Desk"
url: https://thesimind.si/papers/2611.09233/
description: "Cross-Layer TopK sets one sparsity budget for the whole model and shifts it between layers during training, giving better reconstruction with fewer active..."
---

Number 10 of 12 · November 9, 2026

# Cross-Layer TopK: Dynamically Allocating Sparsity Across Layers in Transcoders

Cross-Layer TopK sets one sparsity budget for the whole model and shifts it between layers during training, giving better reconstruction with fewer active latents on Qwen3-8B-Base.

Anyuan Zhuo, Shaohua Zhang, Tianle Li and 4 more · Submitted November 6, 2026 · arXiv 2611.09233v1 · cs.LG

[Read the paper on arXiv](https://arxiv.org/abs/2611.09233) · [The PDF](https://arxiv.org/pdf/2611.09233) · [The day's 12](https://thesimind.si/papers/day/2026-11-09/)

## What they found

The authors study Cross-Layer Transcoders (CLTs), tools that break a language model's internal activations into human-readable features and trace how they link across layers. They propose Cross-Layer TopK, which sets a single L0 budget (the count of active features) for the whole model. It then moves that budget between layers during training, since some layers, especially intermediate ones, are harder to reconstruct. On Qwen3-8B-Base, they report lower FVU (a measure of reconstruction error) while using 20% fewer active latents than JumpReLU. They also report more complete circuits, with higher replacement and completeness scores. Pairing the method with sparse decoding gives a 2.9x speedup per training step, and the abstract says it follows scaling laws in latent size and sparsity.

## Why it matters

Interpretability tools only help if they are accurate and affordable to train. A method that reconstructs a model's inner workings better with fewer active features could make circuit tracing in large models more practical. The authors have also released the weights of their CLT, so others can use and check it. The abstract does not give details on how the circuits were judged beyond the stated scores.

## How it changes the race

This is an efficiency and tooling gain for interpretability work. If it holds, it lowers the cost of building CLTs for large models, which may let more groups study frontier-scale SI systems.

## Safety and control

Better circuit tracing can help people understand why SI models behave as they do, which supports oversight. The abstract does not test any safety use directly.

### The abstract, by the authors

Cross-Layer Transcoders (CLTs) decompose the activations of large language models into human-interpretable features and trace how these features interact across layers to form circuits, making them a key tool for mechanistic interpretability. However, training CLTs effectively and efficiently is hard. Beyond the usual trade-off between sparsity and reconstruction quality, a CLT has to model all layers at once, which adds a further challenge. The common JumpReLU recipe controls sparsity only indirectly through a penalty, while the direct extension of TopK selects a fixed number of latents in every layer. Yet reconstruction difficulty differs widely across layers, and intermediate layers are much harder than others, so giving every layer the same budget is suboptimal. We propose Cross-Layer TopK, which sets a single L0 budget for the whole model and dynamically allocates it across layers during training. In a comparison with JumpReLU and other TopK variants on Qwen3-8B-Base, this gives lower FVU while using 20% fewer active latents than JumpReLU. The better reconstruction carries over to circuit tracing, yielding more complete circuits with higher replacement and completeness scores. The fixed per-token budget of Cross-Layer TopK also makes it easy to pair with sparse decoding, which cuts both computation and communication and gives a 2.9x speedup per training step. We further show that Cross-Layer TopK follows scaling laws in both latent size and sparsity. We have open-sourced the weights of our CLT at https://huggingface.co/ByteDance-Seed/Cross-Layer-TopK-CLT.

Quoted from arXiv:2611.09233. The research and its words belong to its authors.

Desk score **71** out of 100

Why it was picked

- Frontier lab: ByteDance
- Interpretability
- Compute
- Studies frontier models (Qwen3)
- Language models
- Code or data released

Topics

- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Scaling](https://thesimind.si/papers/?tag=scaling)
- [Training](https://thesimind.si/papers/?tag=training)

**arXiv ID:** 2611.09233v1
**Submitted:** November 6, 2026
**Fields:** cs.LG (Machine Learning)
**Level:** Advanced

Fingerprint, SHA-256

`6afb76114845961bb28724b105e68c56e5eaf59050c5d5b15c68849ddf153149`

On Solana: [Proof of Mind seal, 437gF7...BR92](https://solscan.io/tx/437gF725DAG8gRKPBLFDmKiCEc8recBwc9f2nJbDSHirzNkcCnPsgSDCZzxXS15qHXCpomLuKrJuXUTtZbKhBR92)

### The exact text fingerprinted

{"arxiv":"2611.09233","version":1,"title":"Cross-Layer TopK: Dynamically Allocating Sparsity Across Layers in Transcoders","authors":["Anyuan Zhuo","Shaohua Zhang","Tianle Li","Yuan Lin","Diandian Gu","Jun Wang","Pinyan Lu"],"day":"2026-11-09","rank":10,"score":71,"status":"written","summary":{"takeaway":"Cross-Layer TopK sets one sparsity budget for the whole model and shifts it between layers during training, giving better reconstruction with fewer active latents on Qwen3-8B-Base.","found":"The authors study Cross-Layer Transcoders (CLTs), tools that break a language model's internal activations into human-readable features and trace how they link across layers. They propose Cross-Layer TopK, which sets a single L0 budget (the count of active features) for the whole model. It then moves that budget between layers during training, since some layers, especially intermediate ones, are harder to reconstruct. On Qwen3-8B-Base, they report lower FVU (a measure of reconstruction error) while using 20% fewer active latents than JumpReLU. They also report more complete circuits, with higher replacement and completeness scores. Pairing the method with sparse decoding gives a 2.9x speedup per training step, and the abstract says it follows scaling laws in latent size and sparsity.","matters":"Interpretability tools only help if they are accurate and affordable to train. A method that reconstructs a model's inner workings better with fewer active features could make circuit tracing in large models more practical. The authors have also released the weights of their CLT, so others can use and check it. The abstract does not give details on how the circuits were judged beyond the stated scores.","race":"This is an efficiency and tooling gain for interpretability work. If it holds, it lowers the cost of building CLTs for large models, which may let more groups study frontier-scale SI systems.","safety":"Better circuit tracing can help people understand why SI models behave as they do, which supports oversight. The abstract does not test any safety use directly.","level":"advanced","tags":["interpretability","efficiency","language-models","scaling","training"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Anyuan Zhuo, Shaohua Zhang, Tianle Li, Yuan Lin, Diandian Gu, Jun Wang, Pinyan Lu

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 9, 2026 · Number 9BrainCoT: A Multi-Task Zero-Shot Brain Signal Foundation Model with Neurometric-Anchored Chain-of-Thought ReasoningBrainCoT is a single brain signal model that handles several tasks with no extra training, and it gives step by step reasoning tied to measurable brain evidence.Language modelsReasoningHealth](https://thesimind.si/papers/2611.09082/) · [Nov 9, 2026 · Number 11Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM ReasoningDR-BOLT trains one vision-language model to answer simple questions directly and think step by step only when needed, cutting wasted reasoning and training time.ReasoningEfficiencyMultimodal](https://thesimind.si/papers/2611.09154/) · [Nov 9, 2026 · Number 2Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDPIn multiple-choice safety tests, forcing a model to pick a letter seems to crowd out its refusal, so these scores may measure knowledge more than safety.SafetyInterpretabilityEvals](https://thesimind.si/papers/2611.08639/)

[Number 9](https://thesimind.si/papers/2611.09082/) · [All 12 for Nov 9, 2026](https://thesimind.si/papers/day/2026-11-09/) · [Number 11](https://thesimind.si/papers/2611.09154/)