Refine abstract, switch to Times font, remove date, fix overfull hboxes with microtype

This commit is contained in:
ViperEkura 2026-07-04 22:31:45 +08:00
parent dd412b3c9c
commit b40fcc86c7
1 changed files with 13 additions and 11 deletions

View File

@ -3,8 +3,9 @@
% ===== Packages =====
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{newtxtext,newtxmath}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{hyperref}
@ -12,6 +13,7 @@
\usepackage{caption}
\usepackage{enumitem}
\usepackage{url}
\usepackage{microtype}
\DeclareMathOperator{\Var}{Var}
@ -19,7 +21,7 @@
\large Data Pipeline, Distributed Training, and BF16 Numerical Stability via Residual Scaling}
\author{AstrAI Contributors}
\date{June 2026}
\date{}
\begin{document}
@ -36,14 +38,14 @@ a 24-layer decoder-only architecture with Grouped Query Attention and
SwiGLU, and distributed training via DDP/FSDP with cosine scheduling.
A central focus is the numerical stability of BF16-precision training
in deep Transformers. Through variance propagation analysis, we show
that GPT-2 residual scaling ($\sigma_o = 0.02 / \sqrt{2L}$) on output
projections reduces per-block residual variance by a factor of $2L=48$,
containing post-24-layer variance at $1.34$ compared to $17.5$ without
scaling. Empirical evaluations over 15B training tokens demonstrate that
residual scaling consistently outperforms Kaiming initialization, with
the gap widening to $0.79$ in the mid-training regime before narrowing
to $0.38$ at convergence. These results establish residual scaling as a
practical necessity for BF16 Transformer training at scale.
that GPT-2 residual scaling on output projections reduces per-block
residual variance by a factor of 48, containing post-24-layer variance
at 1.34 compared to 17.5 without scaling. Empirical evaluations over
15B training tokens demonstrate that residual scaling consistently
outperforms Kaiming initialization, with the gap widening to 0.79 in
the mid-training regime before narrowing to 0.38 at convergence. These
results establish residual scaling as a practical necessity for BF16
Transformer training at scale.
\end{abstract}
% ======================================================================
@ -74,7 +76,7 @@ via a JSON specification that defines:
(e.g.,~mask user input, compute loss on assistant response).
\item \textbf{Packing}: Documents concatenated via \texttt{simple}
(sequential), \texttt{bfd} (best-fit decreasing), or
\texttt{bfd\_split} strategies.
\texttt{bfd\_\allowbreak{}split} strategies.
\item \textbf{Position IDs}: \texttt{none}, \texttt{doc\_reset} (per-document
boundary), or \texttt{continuous}.
\item \textbf{Output}: Tokenized sequences written to \texttt{.h5} or