ViperEkura
/
AstrAI-paper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix training hyperparams to match pretraining script: lr 1.5e-4, wd 0.1, warmup 0.02

This commit is contained in:
ViperEkura 2026-06-28 16:57:30 +08:00
parent adc284920a
commit 73cb02f0aa
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
main.tex
View File

@ -195,10 +195,10 @@ Table~\ref{tab:train_params} lists the key hyperparameters.
\textbf{Hyperparameter} & \textbf{Value} \\ \textbf{Hyperparameter} & \textbf{Value} \\
\midrule \midrule
Precision & BF16 (weights + AdamW states) \\ Precision & BF16 (weights + AdamW states) \\
Optimizer & AdamW, $\eta=7.5\times10^{-6}$ \\ Optimizer & AdamW, $\eta=1.5\times10^{-4}$ \\
Betas & $(0.9, 0.95)$, weight decay $0.01$ \\ Betas & $(0.9, 0.95)$, weight decay $0.1$ \\
Gradient clip & Global L2, max norm $1.0$ \\ Gradient clip & Global L2, max norm $1.0$ \\
Scheduler & Cosine, warmup ratio $0.05$ \\ Scheduler & Cosine, warmup ratio $0.02$ \\
Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\ Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\
Sequence length & 2,048 tokens \\ Sequence length & 2,048 tokens \\
Total steps & 950,000 \\ Total steps & 950,000 \\