diff --git a/main.tex b/main.tex index dce904c..2063933 100644 --- a/main.tex +++ b/main.tex @@ -144,9 +144,8 @@ The model is trained on next-token cross-entropy loss: \end{equation} Training uses AdamW~\cite{loshchilov2019adamw} with cosine learning rate -scheduling (5\% warmup), global L2 gradient clipping, and periodic -validation. The framework supports DDP and FSDP for multi-GPU distribution, -with gradient accumulation and activation checkpointing to manage memory. +scheduling (5\% warmup) and global L2 gradient clipping. The framework supports DDP and FSDP for multi-GPU distribution, +with gradient accumulation to manage memory. Table~\ref{tab:train_params} lists the key hyperparameters. \begin{table}[H] @@ -162,7 +161,7 @@ Optimizer & AdamW, $\eta=7.5\times10^{-6}$ \\ Betas & $(0.9, 0.95)$, weight decay $0.01$ \\ Gradient clip & Global L2, max norm $1.0$ \\ Scheduler & Cosine, warmup ratio $0.05$ \\ -Batch size & 4 per device $\times$ 4 GPUs $\times$ 8 accumulation \\ +Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\ Sequence length & 2,048 tokens \\ Total steps & 950,000 \\ \bottomrule