fix: attribute muon-25bt larger std to Muon characteristic, not training duration alone
The muon-25bt (Muon, 25B) vs kami/norm (AdamW, 15B) comparison confounds optimizer choice with training duration. Reframe as "Muon produces larger post-convergence weight variance" rather than "training duration drives variance growth."
This commit is contained in:
@@ -520,21 +520,18 @@ are visible in the per-component weight std
|
||||
(Table~\ref{tab:weight_std}, Appendix~\ref{app:weight_std}):
|
||||
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{Training duration drives variance growth}: the
|
||||
\texttt{muon-25bt} checkpoint exhibits the largest weight std
|
||||
($\sim$0.024 for attention projections), exceeding both 15B
|
||||
checkpoints ($\sim$0.015--0.021), reflecting continued weight
|
||||
drift from $\sigma_0 = 0.02$ as training progresses.
|
||||
\item \textbf{Muon produces larger post-convergence weight
|
||||
variance}: the \texttt{muon-25bt} checkpoint (Muon, 25B tokens)
|
||||
exhibits the largest weight std ($\sim$0.024), exceeding both
|
||||
15B AdamW checkpoints ($\sim$0.015--0.021), consistent with
|
||||
Muon allowing wider parameter distributions after convergence.
|
||||
\item \textbf{Residual scaling constrains early-stage drift}: at
|
||||
equal training budget (15B tokens) and with the same AdamW
|
||||
optimizer, the GPT-2-scaled \texttt{kami-15bt} shows
|
||||
\emph{smaller} std ($\sim$0.015) than the Normal-init
|
||||
\texttt{norm-15bt} ($\sim$0.021), indicating that residual
|
||||
scaling itself limits weight drift beyond its initialization
|
||||
effect. The Muon-trained \texttt{muon-25bt} (25B tokens)
|
||||
exhibits the largest std because cumulative training steps
|
||||
eventually overtake both initialization and per-step optimizer
|
||||
constraints.
|
||||
effect.
|
||||
\end{itemize}
|
||||
|
||||
Critically, the residual-scaled projections maintain consistently
|
||||
@@ -847,9 +844,10 @@ distribution analysis in Section~\ref{sec:num-stability}.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\caption{Weight std by component across checkpoints. Non-scaled weights broaden with training duration; residual scaling
|
||||
constrains drift at equal token count (15B) but is overtaken by
|
||||
longer training (25B).
|
||||
\caption{Weight std by component across checkpoints. Non-scaled weights
|
||||
broaden with training; residual scaling constrains drift at equal
|
||||
token count (15B). Muon produces larger post-convergence weight
|
||||
variance than AdamW.
|
||||
Residual-scaled projections ($\mathbf{W}_o$, $\mathbf{W}_{\text{down}}$)
|
||||
remain bounded.}
|
||||
\label{tab:weight_std}
|
||||
|
||||
Reference in New Issue
Block a user