Update DPO iteration count: 1,500 -> 3,000

This commit is contained in:
2026-07-20 03:59:55 +08:00
parent c775a2b3e0
commit da0f536526
+2 -2
View File
@@ -364,7 +364,7 @@ We use $\beta=0.1$, a batch of 64 preference pairs per step, and the
same hybrid Muon/AdamW optimizer. Unlike the pretraining and SFT same hybrid Muon/AdamW optimizer. Unlike the pretraining and SFT
phases, which both employ WSD (warmup--stable--decay) scheduling to phases, which both employ WSD (warmup--stable--decay) scheduling to
maintain a long high-learning-rate plateau, DPO alignment uses a maintain a long high-learning-rate plateau, DPO alignment uses a
cosine schedule with short linear warmup. The shorter ~1{,}500-step cosine schedule with short linear warmup. The shorter ~3{,}000-step
alignment run does not benefit from an extended stable phase; instead, alignment run does not benefit from an extended stable phase; instead,
cosine decay lowers the learning rate steadily, which discourages cosine decay lowers the learning rate steadily, which discourages
over-optimisation away from the reference distribution and matches the over-optimisation away from the reference distribution and matches the
@@ -373,7 +373,7 @@ observed convergence pattern in Figure~\ref{fig:dpo_metrics}.
\begin{figure}[H] \begin{figure}[H]
\centering \centering
\includegraphics[width=0.95\linewidth]{data/dpo_metrics.png} \includegraphics[width=0.95\linewidth]{data/dpo_metrics.png}
\caption{DPO training metrics over $\sim$1{,}500 alignment steps: \caption{DPO training metrics over $\sim$3{,}000 alignment steps:
preference loss (raw and 100-step moving average), learning rate preference loss (raw and 100-step moving average), learning rate
schedule, and gradient norm.} schedule, and gradient norm.}
\label{fig:dpo_metrics} \label{fig:dpo_metrics}