fix training config: grad_accum 32, remove unused validation & activation ckpt

This commit is contained in:
ViperEkura 2026-06-28 15:35:15 +08:00
parent d3fbe0ecc6
commit 13089b002a
1 changed files with 3 additions and 4 deletions

View File

@ -144,9 +144,8 @@ The model is trained on next-token cross-entropy loss:
\end{equation}
Training uses AdamW~\cite{loshchilov2019adamw} with cosine learning rate
scheduling (5\% warmup), global L2 gradient clipping, and periodic
validation. The framework supports DDP and FSDP for multi-GPU distribution,
with gradient accumulation and activation checkpointing to manage memory.
scheduling (5\% warmup) and global L2 gradient clipping. The framework supports DDP and FSDP for multi-GPU distribution,
with gradient accumulation to manage memory.
Table~\ref{tab:train_params} lists the key hyperparameters.
\begin{table}[H]
@ -162,7 +161,7 @@ Optimizer & AdamW, $\eta=7.5\times10^{-6}$ \\
Betas & $(0.9, 0.95)$, weight decay $0.01$ \\
Gradient clip & Global L2, max norm $1.0$ \\
Scheduler & Cosine, warmup ratio $0.05$ \\
Batch size & 4 per device $\times$ 4 GPUs $\times$ 8 accumulation \\
Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\
Sequence length & 2,048 tokens \\
Total steps & 950,000 \\
\bottomrule