Refine abstract, switch to Times font, remove date, fix overfull hboxes with microtype
This commit is contained in:
parent
dd412b3c9c
commit
b40fcc86c7
24
main.tex
24
main.tex
|
|
@ -3,8 +3,9 @@
|
|||
% ===== Packages =====
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{newtxtext,newtxmath}
|
||||
\usepackage[margin=1in]{geometry}
|
||||
\usepackage{amsmath,amssymb}
|
||||
\usepackage{amsmath}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{hyperref}
|
||||
|
|
@ -12,6 +13,7 @@
|
|||
\usepackage{caption}
|
||||
\usepackage{enumitem}
|
||||
\usepackage{url}
|
||||
\usepackage{microtype}
|
||||
|
||||
\DeclareMathOperator{\Var}{Var}
|
||||
|
||||
|
|
@ -19,7 +21,7 @@
|
|||
\large Data Pipeline, Distributed Training, and BF16 Numerical Stability via Residual Scaling}
|
||||
|
||||
\author{AstrAI Contributors}
|
||||
\date{June 2026}
|
||||
\date{}
|
||||
|
||||
\begin{document}
|
||||
|
||||
|
|
@ -36,14 +38,14 @@ a 24-layer decoder-only architecture with Grouped Query Attention and
|
|||
SwiGLU, and distributed training via DDP/FSDP with cosine scheduling.
|
||||
A central focus is the numerical stability of BF16-precision training
|
||||
in deep Transformers. Through variance propagation analysis, we show
|
||||
that GPT-2 residual scaling ($\sigma_o = 0.02 / \sqrt{2L}$) on output
|
||||
projections reduces per-block residual variance by a factor of $2L=48$,
|
||||
containing post-24-layer variance at $1.34$ compared to $17.5$ without
|
||||
scaling. Empirical evaluations over 15B training tokens demonstrate that
|
||||
residual scaling consistently outperforms Kaiming initialization, with
|
||||
the gap widening to $0.79$ in the mid-training regime before narrowing
|
||||
to $0.38$ at convergence. These results establish residual scaling as a
|
||||
practical necessity for BF16 Transformer training at scale.
|
||||
that GPT-2 residual scaling on output projections reduces per-block
|
||||
residual variance by a factor of 48, containing post-24-layer variance
|
||||
at 1.34 compared to 17.5 without scaling. Empirical evaluations over
|
||||
15B training tokens demonstrate that residual scaling consistently
|
||||
outperforms Kaiming initialization, with the gap widening to 0.79 in
|
||||
the mid-training regime before narrowing to 0.38 at convergence. These
|
||||
results establish residual scaling as a practical necessity for BF16
|
||||
Transformer training at scale.
|
||||
\end{abstract}
|
||||
|
||||
% ======================================================================
|
||||
|
|
@ -74,7 +76,7 @@ via a JSON specification that defines:
|
|||
(e.g.,~mask user input, compute loss on assistant response).
|
||||
\item \textbf{Packing}: Documents concatenated via \texttt{simple}
|
||||
(sequential), \texttt{bfd} (best-fit decreasing), or
|
||||
\texttt{bfd\_split} strategies.
|
||||
\texttt{bfd\_\allowbreak{}split} strategies.
|
||||
\item \textbf{Position IDs}: \texttt{none}, \texttt{doc\_reset} (per-document
|
||||
boundary), or \texttt{continuous}.
|
||||
\item \textbf{Output}: Tokenized sequences written to \texttt{.h5} or
|
||||
|
|
|
|||
Loading…
Reference in New Issue