ViperEkura/AstrAI-paper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: attribute muon-25bt larger std to Muon characteristic, not training duration alone

Browse Source 
The muon-25bt (Muon, 25B) vs kami/norm (AdamW, 15B) comparison confounds
optimizer choice with training duration. Reframe as "Muon produces larger
post-convergence weight variance" rather than "training duration drives
variance growth."
This commit is contained in:
ViperEkura
2026-07-20 11:59:09 +08:00
parent a83555f326
commit 4b37f289c0
1 changed files with 10 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
main.tex
+10 -12
View File
@@ -520,21 +520,18 @@ are visible in the per-component weight std
(Table~\ref{tab:weight_std}, Appendix~\ref{app:weight_std}): (Table~\ref{tab:weight_std}, Appendix~\ref{app:weight_std}):
\begin{itemize}[nosep] \begin{itemize}[nosep]
\item \textbf{Training duration drives variance growth}: the \item \textbf{Muon produces larger post-convergence weight
\texttt{muon-25bt} checkpoint exhibits the largest weight std variance}: the \texttt{muon-25bt} checkpoint (Muon, 25B tokens)
($\sim$0.024 for attention projections), exceeding both 15B exhibits the largest weight std ($\sim$0.024), exceeding both
checkpoints ($\sim$0.015--0.021), reflecting continued weight 15B AdamW checkpoints ($\sim$0.015--0.021), consistent with
drift from $\sigma_0 = 0.02$ as training progresses. Muon allowing wider parameter distributions after convergence.
\item \textbf{Residual scaling constrains early-stage drift}: at \item \textbf{Residual scaling constrains early-stage drift}: at
equal training budget (15B tokens) and with the same AdamW equal training budget (15B tokens) and with the same AdamW
optimizer, the GPT-2-scaled \texttt{kami-15bt} shows optimizer, the GPT-2-scaled \texttt{kami-15bt} shows
\emph{smaller} std ($\sim$0.015) than the Normal-init \emph{smaller} std ($\sim$0.015) than the Normal-init
\texttt{norm-15bt} ($\sim$0.021), indicating that residual \texttt{norm-15bt} ($\sim$0.021), indicating that residual
scaling itself limits weight drift beyond its initialization scaling itself limits weight drift beyond its initialization
effect. The Muon-trained \texttt{muon-25bt} (25B tokens) effect.
exhibits the largest std because cumulative training steps
eventually overtake both initialization and per-step optimizer
constraints.
\end{itemize} \end{itemize}
Critically, the residual-scaled projections maintain consistently Critically, the residual-scaled projections maintain consistently
@@ -847,9 +844,10 @@ distribution analysis in Section~\ref{sec:num-stability}.
\begin{table}[H] \begin{table}[H]
\centering \centering
\caption{Weight std by component across checkpoints. Non-scaled weights broaden with training duration; residual scaling \caption{Weight std by component across checkpoints. Non-scaled weights
constrains drift at equal token count (15B) but is overtaken by broaden with training; residual scaling constrains drift at equal
longer training (25B). token count (15B). Muon produces larger post-convergence weight
variance than AdamW.
Residual-scaled projections ($\mathbf{W}_o$, $\mathbf{W}_{\text{down}}$) Residual-scaled projections ($\mathbf{W}_o$, $\mathbf{W}_{\text{down}}$)
remain bounded.} remain bounded.}
\label{tab:weight_std} \label{tab:weight_std}