Compare commits

...
2 Commits
Author SHA1 Message Date
ViperEkura 0c2bc916f2 fix: loss_compare caption token count (20B -> 5B)
The loss_compare.png x-axis only spans 0-5B tokens. The caption
incorrectly claimed ~20B tokens.
2026-07-24 06:21:52 +08:00
ViperEkura 4e70e827ff remove: drop ckpt_weight_density_per_run.png figure
The per-run weight density figure contained misleading legacy
iteration labels (500k/1M iter) that contradicted the paper's
stated token budgets (15B). Removing it avoids confusion; the
per-category density plot (ckpt_weight_density.png) and Table 7
remain as the primary evidence.
2026-07-24 06:20:08 +08:00
+1 -12
View File
@@ -302,7 +302,7 @@ Sequence length & 2,048 tokens \\
\centering
\includegraphics[width=0.50\linewidth]{data/loss_compare.png}
\caption{Training loss curves: GPT-2 residual scaling vs.~Kaiming
initialization over $\sim$20B tokens.}
initialization over $\sim$5B tokens.}
\label{fig:loss}
\end{figure}
@@ -547,17 +547,6 @@ spread.}
\label{fig:ckpt_weight_density}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.95\linewidth]{data/ckpt_weight_density_per_run.png}
\caption{Per-checkpoint weight density breakdowns. The three checkpoints
are \texttt{norm-15bt} (AdamW, Normal init, 15B tokens), \texttt{kami-15bt}
(AdamW, GPT-2 residual scaling, 15B tokens), and \texttt{muon-25bt}
(Muon, 25B tokens). Note: the iteration labels inside the figure reflect
legacy script metadata and should be ignored in favour of the token
budgets stated here.}
\label{fig:ckpt_weight_density_per_run}
\end{figure}
% ======================================================================
\section{Conclusion}