Refine abstract, switch to Times font, remove date, fix overfull hboxes with microtype
This commit is contained in:
parent
dd412b3c9c
commit
b40fcc86c7
24
main.tex
24
main.tex
|
|
@ -3,8 +3,9 @@
|
||||||
% ===== Packages =====
|
% ===== Packages =====
|
||||||
\usepackage[utf8]{inputenc}
|
\usepackage[utf8]{inputenc}
|
||||||
\usepackage[T1]{fontenc}
|
\usepackage[T1]{fontenc}
|
||||||
|
\usepackage{newtxtext,newtxmath}
|
||||||
\usepackage[margin=1in]{geometry}
|
\usepackage[margin=1in]{geometry}
|
||||||
\usepackage{amsmath,amssymb}
|
\usepackage{amsmath}
|
||||||
\usepackage{booktabs}
|
\usepackage{booktabs}
|
||||||
\usepackage{graphicx}
|
\usepackage{graphicx}
|
||||||
\usepackage{hyperref}
|
\usepackage{hyperref}
|
||||||
|
|
@ -12,6 +13,7 @@
|
||||||
\usepackage{caption}
|
\usepackage{caption}
|
||||||
\usepackage{enumitem}
|
\usepackage{enumitem}
|
||||||
\usepackage{url}
|
\usepackage{url}
|
||||||
|
\usepackage{microtype}
|
||||||
|
|
||||||
\DeclareMathOperator{\Var}{Var}
|
\DeclareMathOperator{\Var}{Var}
|
||||||
|
|
||||||
|
|
@ -19,7 +21,7 @@
|
||||||
\large Data Pipeline, Distributed Training, and BF16 Numerical Stability via Residual Scaling}
|
\large Data Pipeline, Distributed Training, and BF16 Numerical Stability via Residual Scaling}
|
||||||
|
|
||||||
\author{AstrAI Contributors}
|
\author{AstrAI Contributors}
|
||||||
\date{June 2026}
|
\date{}
|
||||||
|
|
||||||
\begin{document}
|
\begin{document}
|
||||||
|
|
||||||
|
|
@ -36,14 +38,14 @@ a 24-layer decoder-only architecture with Grouped Query Attention and
|
||||||
SwiGLU, and distributed training via DDP/FSDP with cosine scheduling.
|
SwiGLU, and distributed training via DDP/FSDP with cosine scheduling.
|
||||||
A central focus is the numerical stability of BF16-precision training
|
A central focus is the numerical stability of BF16-precision training
|
||||||
in deep Transformers. Through variance propagation analysis, we show
|
in deep Transformers. Through variance propagation analysis, we show
|
||||||
that GPT-2 residual scaling ($\sigma_o = 0.02 / \sqrt{2L}$) on output
|
that GPT-2 residual scaling on output projections reduces per-block
|
||||||
projections reduces per-block residual variance by a factor of $2L=48$,
|
residual variance by a factor of 48, containing post-24-layer variance
|
||||||
containing post-24-layer variance at $1.34$ compared to $17.5$ without
|
at 1.34 compared to 17.5 without scaling. Empirical evaluations over
|
||||||
scaling. Empirical evaluations over 15B training tokens demonstrate that
|
15B training tokens demonstrate that residual scaling consistently
|
||||||
residual scaling consistently outperforms Kaiming initialization, with
|
outperforms Kaiming initialization, with the gap widening to 0.79 in
|
||||||
the gap widening to $0.79$ in the mid-training regime before narrowing
|
the mid-training regime before narrowing to 0.38 at convergence. These
|
||||||
to $0.38$ at convergence. These results establish residual scaling as a
|
results establish residual scaling as a practical necessity for BF16
|
||||||
practical necessity for BF16 Transformer training at scale.
|
Transformer training at scale.
|
||||||
\end{abstract}
|
\end{abstract}
|
||||||
|
|
||||||
% ======================================================================
|
% ======================================================================
|
||||||
|
|
@ -74,7 +76,7 @@ via a JSON specification that defines:
|
||||||
(e.g.,~mask user input, compute loss on assistant response).
|
(e.g.,~mask user input, compute loss on assistant response).
|
||||||
\item \textbf{Packing}: Documents concatenated via \texttt{simple}
|
\item \textbf{Packing}: Documents concatenated via \texttt{simple}
|
||||||
(sequential), \texttt{bfd} (best-fit decreasing), or
|
(sequential), \texttt{bfd} (best-fit decreasing), or
|
||||||
\texttt{bfd\_split} strategies.
|
\texttt{bfd\_\allowbreak{}split} strategies.
|
||||||
\item \textbf{Position IDs}: \texttt{none}, \texttt{doc\_reset} (per-document
|
\item \textbf{Position IDs}: \texttt{none}, \texttt{doc\_reset} (per-document
|
||||||
boundary), or \texttt{continuous}.
|
boundary), or \texttt{continuous}.
|
||||||
\item \textbf{Output}: Tokenized sequences written to \texttt{.h5} or
|
\item \textbf{Output}: Tokenized sequences written to \texttt{.h5} or
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue