Number 12 of 12 · November 14, 2026

When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective

The authors show, in a simplified math setting, that SignSGD can beat plain SGD on hard tasks and at large batch sizes, because gradient noise changes how the sign step behaves.

What they found

The authors study SignSGD, an optimizer that updates by the sign (plus or minus) of each gradient value. They work in power-law kernel regression with diagonal features, a simplified math setting. There they derive a functional scaling law (FSL), a formula for how training loss changes over time under general learning rate schedules. The FSL has two parts: signal learning and noise forgetting. Gradient noise smooths the sign step, so SignSGD acts like SGD with a preconditioner, which changes the effective Hessian (the curvature of the loss) from H to H^1/2. This makes signal learning faster but noise forgetting slower than SGD. As a result, SignSGD uses data more efficiently on hard tasks. On easy tasks both match on data efficiency, but SignSGD keeps that efficiency at larger batch sizes and needs fewer steps for a fixed data budget. Kernel regression experiments support this, and LLM pre-training experiments suggest the SignSGD FSL fits batch-size effects for sign-based optimizers better than the original SGD FSL.

Why it matters

Sign-based optimizers are used in training large models, but it is not well understood when they help. This paper offers a clear mechanism and a rule for when each optimizer should win. The theory covers a simplified setting, and the abstract only calls the LLM results suggestive.

How it changes the race

If the result holds, teams training large SI models could better choose optimizers and batch sizes. Larger batches with fewer steps could shorten training time. This is an incremental, theory-led guide, not a new capability.

The abstract, by the authors

We study how the sign operator interacts with gradient noise to shape the learning dynamics and scaling behavior of sign-based optimizers. In power-law kernel regression with diagonal features, we establish a functional scaling law (FSL) for SignSGD under general learning rate schedules. The FSL describes the learning dynamics through two processes: signal learning and noise forgetting. The key mechanism is noise-induced preconditioning: gradient noise smooths the sign operator, inducing an effective preconditioned SGD. Noise-curvature alignment enables this preconditioning to transform the effective Hessian from H to H^1/2. The resulting dynamics exhibit faster signal learning but slower noise forgetting than SGD. This tradeoff yields a regime-dependent comparison with SGD: (i) SignSGD achieves improved data efficiency in the hard-task regime; (ii) both optimizers achieve the same data efficiency in the easy-task regime, but SignSGD maintains this efficiency at larger batch sizes, requiring fewer optimization steps for a fixed data budget. Power-law kernel regression experiments support the proposed mechanism and predicted scaling behavior. LLM pre-training experiments further suggest that the SignSGD FSL better captures the batch-size dependence of loss dynamics for sign-based optimizers than the original SGD FSL.

Quoted from arXiv:2611.16560. The research and its words belong to its authors.

The authors

Zilin Wang, Binghui Li, Jia-Nan Wang, Lean Wang, Jinbo Wang, Lei Wu

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.