---
title: "When and How SignSGD Outperforms SGD: A... | SI Paper Desk"
url: https://thesimind.si/papers/2611.16560/
description: "The authors show, in a simplified math setting, that SignSGD can beat plain SGD on hard tasks and at large batch sizes, because gradient noise changes how..."
---

Number 12 of 12 · November 14, 2026

# When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective

The authors show, in a simplified math setting, that SignSGD can beat plain SGD on hard tasks and at large batch sizes, because gradient noise changes how the sign step behaves.

Zilin Wang, Binghui Li, Jia-Nan Wang and 3 more · Submitted November 12, 2026 · arXiv 2611.16560v1 · stat.ML

[Read the paper on arXiv](https://arxiv.org/abs/2611.16560) · [The PDF](https://arxiv.org/pdf/2611.16560) · [The day's 12](https://thesimind.si/papers/day/2026-11-14/)

## What they found

The authors study SignSGD, an optimizer that updates by the sign (plus or minus) of each gradient value. They work in power-law kernel regression with diagonal features, a simplified math setting. There they derive a functional scaling law (FSL), a formula for how training loss changes over time under general learning rate schedules. The FSL has two parts: signal learning and noise forgetting. Gradient noise smooths the sign step, so SignSGD acts like SGD with a preconditioner, which changes the effective Hessian (the curvature of the loss) from H to H^1/2. This makes signal learning faster but noise forgetting slower than SGD. As a result, SignSGD uses data more efficiently on hard tasks. On easy tasks both match on data efficiency, but SignSGD keeps that efficiency at larger batch sizes and needs fewer steps for a fixed data budget. Kernel regression experiments support this, and LLM pre-training experiments suggest the SignSGD FSL fits batch-size effects for sign-based optimizers better than the original SGD FSL.

## Why it matters

Sign-based optimizers are used in training large models, but it is not well understood when they help. This paper offers a clear mechanism and a rule for when each optimizer should win. The theory covers a simplified setting, and the abstract only calls the LLM results suggestive.

## How it changes the race

If the result holds, teams training large SI models could better choose optimizers and batch sizes. Larger batches with fewer steps could shorten training time. This is an incremental, theory-led guide, not a new capability.

### The abstract, by the authors

We study how the sign operator interacts with gradient noise to shape the learning dynamics and scaling behavior of sign-based optimizers. In power-law kernel regression with diagonal features, we establish a functional scaling law (FSL) for SignSGD under general learning rate schedules. The FSL describes the learning dynamics through two processes: signal learning and noise forgetting. The key mechanism is noise-induced preconditioning: gradient noise smooths the sign operator, inducing an effective preconditioned SGD. Noise-curvature alignment enables this preconditioning to transform the effective Hessian from H to H^1/2. The resulting dynamics exhibit faster signal learning but slower noise forgetting than SGD. This tradeoff yields a regime-dependent comparison with SGD: (i) SignSGD achieves improved data efficiency in the hard-task regime; (ii) both optimizers achieve the same data efficiency in the easy-task regime, but SignSGD maintains this efficiency at larger batch sizes, requiring fewer optimization steps for a fixed data budget. Power-law kernel regression experiments support the proposed mechanism and predicted scaling behavior. LLM pre-training experiments further suggest that the SignSGD FSL better captures the batch-size dependence of loss dynamics for sign-based optimizers than the original SGD FSL.

Quoted from arXiv:2611.16560. The research and its words belong to its authors.

Desk score **64** out of 100

Why it was picked

- Compute (in the title)
- Alignment
- Accepted at NeurIPS
- Language models
- Cross-listed in 2 fields

Topics

- [Theory](https://thesimind.si/papers/?tag=theory)
- [Training](https://thesimind.si/papers/?tag=training)
- [Scaling](https://thesimind.si/papers/?tag=scaling)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)
- [Language models](https://thesimind.si/papers/?tag=language-models)

**arXiv ID:** 2611.16560v1
**Submitted:** November 12, 2026
**Fields:** stat.ML (Statistics: Machine Learning), cs.LG (Machine Learning)
**Level:** Advanced
**Comments:** Accepted at NeurIPS 2026 as a spotlight paper

Fingerprint, SHA-256

`1b8496d5fc577afa00d3ace41e01e7c48e2f96021bbb4c66f93cf8bbe0250263`

On Solana: [Proof of Mind seal, 4ut3vv...Y9uP](https://solscan.io/tx/4ut3vv2vTeUs5fAbknPuGnt1i1jJGh18Ry7QYkUAkvvDWCKQk8wrmGNkp4C5Loc937KnsizybKH71BB2w6QAY9uP)

### The exact text fingerprinted

{"arxiv":"2611.16560","version":1,"title":"When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective","authors":["Zilin Wang","Binghui Li","Jia-Nan Wang","Lean Wang","Jinbo Wang","Lei Wu"],"day":"2026-11-14","rank":12,"score":64,"status":"written","summary":{"takeaway":"The authors show, in a simplified math setting, that SignSGD can beat plain SGD on hard tasks and at large batch sizes, because gradient noise changes how the sign step behaves.","found":"The authors study SignSGD, an optimizer that updates by the sign (plus or minus) of each gradient value. They work in power-law kernel regression with diagonal features, a simplified math setting. There they derive a functional scaling law (FSL), a formula for how training loss changes over time under general learning rate schedules. The FSL has two parts: signal learning and noise forgetting. Gradient noise smooths the sign step, so SignSGD acts like SGD with a preconditioner, which changes the effective Hessian (the curvature of the loss) from H to H^1/2. This makes signal learning faster but noise forgetting slower than SGD. As a result, SignSGD uses data more efficiently on hard tasks. On easy tasks both match on data efficiency, but SignSGD keeps that efficiency at larger batch sizes and needs fewer steps for a fixed data budget. Kernel regression experiments support this, and LLM pre-training experiments suggest the SignSGD FSL fits batch-size effects for sign-based optimizers better than the original SGD FSL.","matters":"Sign-based optimizers are used in training large models, but it is not well understood when they help. This paper offers a clear mechanism and a rule for when each optimizer should win. The theory covers a simplified setting, and the abstract only calls the LLM results suggestive.","race":"If the result holds, teams training large SI models could better choose optimizers and batch sizes. Larger batches with fewer steps could shorten training time. This is an incremental, theory-led guide, not a new capability.","safety":"","level":"advanced","tags":["theory","training","scaling","efficiency","language-models"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Zilin Wang, Binghui Li, Jia-Nan Wang, Lean Wang, Jinbo Wang, Lei Wu

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 9, 2026 · Number 10Cross-Layer TopK: Dynamically Allocating Sparsity Across Layers in TranscodersCross-Layer TopK sets one sparsity budget for the whole model and shifts it between layers during training, giving better reconstruction with fewer active latents on Qwen3-8B-Base.InterpretabilityEfficiencyLanguage models](https://thesimind.si/papers/2611.09233/) · [Nov 15, 2026 · Number 11ArchSpace: Recording, Reviewing, and Referencing LLM Architecture ResearchArchSpace is a public platform where SI model architecture ideas are recorded with their experiments, reviewed by the community, and reused, using scaling laws to judge ideas before costly large training runs.ScalingLanguage modelsEvals](https://thesimind.si/papers/2611.17332/) · [Nov 18, 2026 · Number 2Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational AgentsNVIDIA's authors post-trained a small open model from Gemma-4-E4B-it for fast local assistants. They report the top average accuracy on 21 benchmarks and faster decoding with a drafter model.EfficiencyLanguage modelsAgents](https://thesimind.si/papers/2611.20771/)

[Number 11](https://thesimind.si/papers/2611.16816/) · [All 12 for Nov 14, 2026](https://thesimind.si/papers/day/2026-11-14/)
