diff --git a/main.tex b/main.tex index 69c1d3d..0e10a82 100644 --- a/main.tex +++ b/main.tex @@ -195,10 +195,10 @@ Table~\ref{tab:train_params} lists the key hyperparameters. \textbf{Hyperparameter} & \textbf{Value} \\ \midrule Precision & BF16 (weights + AdamW states) \\ -Optimizer & AdamW, $\eta=7.5\times10^{-6}$ \\ -Betas & $(0.9, 0.95)$, weight decay $0.01$ \\ +Optimizer & AdamW, $\eta=1.5\times10^{-4}$ \\ +Betas & $(0.9, 0.95)$, weight decay $0.1$ \\ Gradient clip & Global L2, max norm $1.0$ \\ -Scheduler & Cosine, warmup ratio $0.05$ \\ +Scheduler & Cosine, warmup ratio $0.02$ \\ Batch size & 4 per device $\times$ 4 GPUs $\times$ 32 accumulation \\ Sequence length & 2,048 tokens \\ Total steps & 950,000 \\