diff --git a/main.tex b/main.tex index 7d64a4c..38cf0dc 100644 --- a/main.tex +++ b/main.tex @@ -520,21 +520,18 @@ are visible in the per-component weight std (Table~\ref{tab:weight_std}, Appendix~\ref{app:weight_std}): \begin{itemize}[nosep] - \item \textbf{Training duration drives variance growth}: the - \texttt{muon-25bt} checkpoint exhibits the largest weight std - ($\sim$0.024 for attention projections), exceeding both 15B - checkpoints ($\sim$0.015--0.021), reflecting continued weight - drift from $\sigma_0 = 0.02$ as training progresses. + \item \textbf{Muon produces larger post-convergence weight + variance}: the \texttt{muon-25bt} checkpoint (Muon, 25B tokens) + exhibits the largest weight std ($\sim$0.024), exceeding both + 15B AdamW checkpoints ($\sim$0.015--0.021), consistent with + Muon allowing wider parameter distributions after convergence. \item \textbf{Residual scaling constrains early-stage drift}: at equal training budget (15B tokens) and with the same AdamW optimizer, the GPT-2-scaled \texttt{kami-15bt} shows \emph{smaller} std ($\sim$0.015) than the Normal-init \texttt{norm-15bt} ($\sim$0.021), indicating that residual scaling itself limits weight drift beyond its initialization - effect. The Muon-trained \texttt{muon-25bt} (25B tokens) - exhibits the largest std because cumulative training steps - eventually overtake both initialization and per-step optimizer - constraints. + effect. \end{itemize} Critically, the residual-scaled projections maintain consistently @@ -847,9 +844,10 @@ distribution analysis in Section~\ref{sec:num-stability}. \begin{table}[H] \centering -\caption{Weight std by component across checkpoints. Non-scaled weights broaden with training duration; residual scaling -constrains drift at equal token count (15B) but is overtaken by -longer training (25B). +\caption{Weight std by component across checkpoints. Non-scaled weights +broaden with training; residual scaling constrains drift at equal +token count (15B). Muon produces larger post-convergence weight +variance than AdamW. Residual-scaled projections ($\mathbf{W}_o$, $\mathbf{W}_{\text{down}}$) remain bounded.} \label{tab:weight_std}