ViperEkura
/
AstrAI-paper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix training config: grad_accum 32, remove unused validation & activation ckpt

This commit is contained in:
ViperEkura 2026-06-28 15:35:15 +08:00
parent d3fbe0ecc6
commit 13089b002a
1 changed files with 3 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
main.tex
View File

@ -144,9 +144,8 @@ The model is trained on next-token cross-entropy loss:
\end{equation} \end{equation}
Training uses AdamW~\cite{loshchilov2019adamw} with cosine learning rate Training uses AdamW~\cite{loshchilov2019adamw} with cosine learning rate
scheduling (5\% warmup), global L2 gradient clipping, and periodic scheduling (5\% warmup) and global L2 gradient clipping. The framework supports DDP and FSDP for multi-GPU distribution,
validation. The framework supports DDP and FSDP for multi-GPU distribution, with gradient accumulation to manage memory.
with gradient accumulation and activation checkpointing to manage memory.
Table~\ref{tab:train_params} lists the key hyperparameters. Table~\ref{tab:train_params} lists the key hyperparameters.
\begin{table}[H] \begin{table}[H]
@ -162,7 +161,7 @@ Optimizer & AdamW, $\eta=7.5\times10^{-6}$ \\
Betas & $(0.9, 0.95)$, weight decay $0.01$ \\ Betas & $(0.9, 0.95)$, weight decay $0.01$ \\
Gradient clip & Global L2, max norm $1.0$ \\ Gradient clip & Global L2, max norm $1.0$ \\
Scheduler & Cosine, warmup ratio $0.05$ \\ Scheduler & Cosine, warmup ratio $0.05$ \\
Batch size & 4 per device $\times$ 4 GPUs $\times$ 8 accumulation \\ Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\
Sequence length & 2,048 tokens \\ Sequence length & 2,048 tokens \\
Total steps & 950,000 \\ Total steps & 950,000 \\
\bottomrule \bottomrule