Refine abstract, switch to Times font, remove date, fix overfull hboxes with microtype

This commit is contained in:
ViperEkura 2026-07-04 22:31:45 +08:00
parent dd412b3c9c
commit b40fcc86c7
1 changed files with 13 additions and 11 deletions

View File

@ -3,8 +3,9 @@
% ===== Packages ===== % ===== Packages =====
\usepackage[utf8]{inputenc} \usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc} \usepackage[T1]{fontenc}
\usepackage{newtxtext,newtxmath}
\usepackage[margin=1in]{geometry} \usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb} \usepackage{amsmath}
\usepackage{booktabs} \usepackage{booktabs}
\usepackage{graphicx} \usepackage{graphicx}
\usepackage{hyperref} \usepackage{hyperref}
@ -12,6 +13,7 @@
\usepackage{caption} \usepackage{caption}
\usepackage{enumitem} \usepackage{enumitem}
\usepackage{url} \usepackage{url}
\usepackage{microtype}
\DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\Var}{Var}
@ -19,7 +21,7 @@
\large Data Pipeline, Distributed Training, and BF16 Numerical Stability via Residual Scaling} \large Data Pipeline, Distributed Training, and BF16 Numerical Stability via Residual Scaling}
\author{AstrAI Contributors} \author{AstrAI Contributors}
\date{June 2026} \date{}
\begin{document} \begin{document}
@ -36,14 +38,14 @@ a 24-layer decoder-only architecture with Grouped Query Attention and
SwiGLU, and distributed training via DDP/FSDP with cosine scheduling. SwiGLU, and distributed training via DDP/FSDP with cosine scheduling.
A central focus is the numerical stability of BF16-precision training A central focus is the numerical stability of BF16-precision training
in deep Transformers. Through variance propagation analysis, we show in deep Transformers. Through variance propagation analysis, we show
that GPT-2 residual scaling ($\sigma_o = 0.02 / \sqrt{2L}$) on output that GPT-2 residual scaling on output projections reduces per-block
projections reduces per-block residual variance by a factor of $2L=48$, residual variance by a factor of 48, containing post-24-layer variance
containing post-24-layer variance at $1.34$ compared to $17.5$ without at 1.34 compared to 17.5 without scaling. Empirical evaluations over
scaling. Empirical evaluations over 15B training tokens demonstrate that 15B training tokens demonstrate that residual scaling consistently
residual scaling consistently outperforms Kaiming initialization, with outperforms Kaiming initialization, with the gap widening to 0.79 in
the gap widening to $0.79$ in the mid-training regime before narrowing the mid-training regime before narrowing to 0.38 at convergence. These
to $0.38$ at convergence. These results establish residual scaling as a results establish residual scaling as a practical necessity for BF16
practical necessity for BF16 Transformer training at scale. Transformer training at scale.
\end{abstract} \end{abstract}
% ====================================================================== % ======================================================================
@ -74,7 +76,7 @@ via a JSON specification that defines:
(e.g.,~mask user input, compute loss on assistant response). (e.g.,~mask user input, compute loss on assistant response).
\item \textbf{Packing}: Documents concatenated via \texttt{simple} \item \textbf{Packing}: Documents concatenated via \texttt{simple}
(sequential), \texttt{bfd} (best-fit decreasing), or (sequential), \texttt{bfd} (best-fit decreasing), or
\texttt{bfd\_split} strategies. \texttt{bfd\_\allowbreak{}split} strategies.
\item \textbf{Position IDs}: \texttt{none}, \texttt{doc\_reset} (per-document \item \textbf{Position IDs}: \texttt{none}, \texttt{doc\_reset} (per-document
boundary), or \texttt{continuous}. boundary), or \texttt{continuous}.
\item \textbf{Output}: Tokenized sequences written to \texttt{.h5} or \item \textbf{Output}: Tokenized sequences written to \texttt{.h5} or