fix: attribute muon-25bt larger std to Muon characteristic, not training duration alone

The muon-25bt (Muon, 25B) vs kami/norm (AdamW, 15B) comparison confounds
optimizer choice with training duration. Reframe as "Muon produces larger
post-convergence weight variance" rather than "training duration drives
variance growth."
This commit is contained in:
2026-07-20 11:59:09 +08:00
parent a83555f326
commit 4b37f289c0
+10 -12
View File
@@ -520,21 +520,18 @@ are visible in the per-component weight std
(Table~\ref{tab:weight_std}, Appendix~\ref{app:weight_std}):
\begin{itemize}[nosep]
\item \textbf{Training duration drives variance growth}: the
\texttt{muon-25bt} checkpoint exhibits the largest weight std
($\sim$0.024 for attention projections), exceeding both 15B
checkpoints ($\sim$0.015--0.021), reflecting continued weight
drift from $\sigma_0 = 0.02$ as training progresses.
\item \textbf{Muon produces larger post-convergence weight
variance}: the \texttt{muon-25bt} checkpoint (Muon, 25B tokens)
exhibits the largest weight std ($\sim$0.024), exceeding both
15B AdamW checkpoints ($\sim$0.015--0.021), consistent with
Muon allowing wider parameter distributions after convergence.
\item \textbf{Residual scaling constrains early-stage drift}: at
equal training budget (15B tokens) and with the same AdamW
optimizer, the GPT-2-scaled \texttt{kami-15bt} shows
\emph{smaller} std ($\sim$0.015) than the Normal-init
\texttt{norm-15bt} ($\sim$0.021), indicating that residual
scaling itself limits weight drift beyond its initialization
effect. The Muon-trained \texttt{muon-25bt} (25B tokens)
exhibits the largest std because cumulative training steps
eventually overtake both initialization and per-step optimizer
constraints.
effect.
\end{itemize}
Critically, the residual-scaled projections maintain consistently
@@ -847,9 +844,10 @@ distribution analysis in Section~\ref{sec:num-stability}.
\begin{table}[H]
\centering
\caption{Weight std by component across checkpoints. Non-scaled weights broaden with training duration; residual scaling
constrains drift at equal token count (15B) but is overtaken by
longer training (25B).
\caption{Weight std by component across checkpoints. Non-scaled weights
broaden with training; residual scaling constrains drift at equal
token count (15B). Muon produces larger post-convergence weight
variance than AdamW.
Residual-scaled projections ($\mathbf{W}_o$, $\mathbf{W}_{\text{down}}$)
remain bounded.}
\label{tab:weight_std}