What this work contributes
Studies how unequal per-example clipping factors can distort the singular-vector geometry used by Muon. The proposed inner-momentum method averages each sampled example's gradients across the current and recent models before clipping. The preprint gives a distortion analysis and reports improved BLEU and ROUGE-L over DP-Muon in seed-matched private GPT-2 fine-tuning on E2E and DART at ε ∈ {1, 2, 4, 8}.
When to read this work
Read this when studying differentially private optimizers, gradient-clipping geometry, or the interaction between Muon's orthogonalization and private language-model fine-tuning.
Topics: Differential privacy · Private optimization · Muon · Gradient clipping · Language-model fine-tuning
Abstract
Differentially private training clips each per-example gradient before adding noise. This clipping is radial for each example, yet unequal clipping factors can distort the relative singular-vector geometry of their average. Muon is particularly exposed to this effect, since its update is an approximate polar factor UV^T that depends only on the singular vectors that clipping can shift. To curb this degradation, we propose averaging each sampled example’s Muon gradient over the current model and a short history of recent models before clipping. The clipped batch matrix then separates into a common rescaling and a covariance residual R between sampled gradients and clipping values, bounding the clipping-induced distortion directly. We further show that a finite Newton-Schulz iteration preserves the polar factor of its input under these spectral conditions, confirming that our correction survives orthogonalization. In private GPT-2 fine-tuning on E2E and DART at epsilon in 1, 2, 4, 8, DP-Muon-IM improves BLEU and ROUGE-L over DP-Muon in every seed-matched comparison, and non-private diagnostics show 2-4% lower pre-noise polar error.
Tutorials and reading guides
- Why gradient clipping changes Muon's geometry: a two-dimensional example — A runnable Python example shows how unequal per-example clipping rotates a matrix's polar factor, motivating the DP-Muon-IM preprint.
Cite this work
Download BibTeX · Download RIS · This record describes a preprint.
@article{bhusal2026dpmuon,
title = {Inner Momentum for Differentially Private Muon},
author = {Bhusal, Bishnu and Vu, Minh and Southworth, Ben and Zollicoffer, Geigh and Chadha, Rohit and Bhattarai, Manish},
journal = {arXiv preprint arXiv:2610.02738},
year = {2026},
month = oct,
eprint = {2610.02738},
archiveprefix = {arXiv},
primaryclass = {cs.LG},
url = {https://arxiv.org/abs/2610.02738}
}