fix training config: grad_accum 32, remove unused validation & activation ckpt
This commit is contained in:
parent
d3fbe0ecc6
commit
13089b002a
7
main.tex
7
main.tex
|
|
@ -144,9 +144,8 @@ The model is trained on next-token cross-entropy loss:
|
|||
\end{equation}
|
||||
|
||||
Training uses AdamW~\cite{loshchilov2019adamw} with cosine learning rate
|
||||
scheduling (5\% warmup), global L2 gradient clipping, and periodic
|
||||
validation. The framework supports DDP and FSDP for multi-GPU distribution,
|
||||
with gradient accumulation and activation checkpointing to manage memory.
|
||||
scheduling (5\% warmup) and global L2 gradient clipping. The framework supports DDP and FSDP for multi-GPU distribution,
|
||||
with gradient accumulation to manage memory.
|
||||
Table~\ref{tab:train_params} lists the key hyperparameters.
|
||||
|
||||
\begin{table}[H]
|
||||
|
|
@ -162,7 +161,7 @@ Optimizer & AdamW, $\eta=7.5\times10^{-6}$ \\
|
|||
Betas & $(0.9, 0.95)$, weight decay $0.01$ \\
|
||||
Gradient clip & Global L2, max norm $1.0$ \\
|
||||
Scheduler & Cosine, warmup ratio $0.05$ \\
|
||||
Batch size & 4 per device $\times$ 4 GPUs $\times$ 8 accumulation \\
|
||||
Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\
|
||||
Sequence length & 2,048 tokens \\
|
||||
Total steps & 950,000 \\
|
||||
\bottomrule
|
||||
|
|
|
|||
Loading…
Reference in New Issue