Update DPO iteration count: 1,500 -> 3,000
This commit is contained in:
@@ -364,7 +364,7 @@ We use $\beta=0.1$, a batch of 64 preference pairs per step, and the
|
|||||||
same hybrid Muon/AdamW optimizer. Unlike the pretraining and SFT
|
same hybrid Muon/AdamW optimizer. Unlike the pretraining and SFT
|
||||||
phases, which both employ WSD (warmup--stable--decay) scheduling to
|
phases, which both employ WSD (warmup--stable--decay) scheduling to
|
||||||
maintain a long high-learning-rate plateau, DPO alignment uses a
|
maintain a long high-learning-rate plateau, DPO alignment uses a
|
||||||
cosine schedule with short linear warmup. The shorter ~1{,}500-step
|
cosine schedule with short linear warmup. The shorter ~3{,}000-step
|
||||||
alignment run does not benefit from an extended stable phase; instead,
|
alignment run does not benefit from an extended stable phase; instead,
|
||||||
cosine decay lowers the learning rate steadily, which discourages
|
cosine decay lowers the learning rate steadily, which discourages
|
||||||
over-optimisation away from the reference distribution and matches the
|
over-optimisation away from the reference distribution and matches the
|
||||||
@@ -373,7 +373,7 @@ observed convergence pattern in Figure~\ref{fig:dpo_metrics}.
|
|||||||
\begin{figure}[H]
|
\begin{figure}[H]
|
||||||
\centering
|
\centering
|
||||||
\includegraphics[width=0.95\linewidth]{data/dpo_metrics.png}
|
\includegraphics[width=0.95\linewidth]{data/dpo_metrics.png}
|
||||||
\caption{DPO training metrics over $\sim$1{,}500 alignment steps:
|
\caption{DPO training metrics over $\sim$3{,}000 alignment steps:
|
||||||
preference loss (raw and 100-step moving average), learning rate
|
preference loss (raw and 100-step moving average), learning rate
|
||||||
schedule, and gradient norm.}
|
schedule, and gradient norm.}
|
||||||
\label{fig:dpo_metrics}
|
\label{fig:dpo_metrics}
|
||||||
|
|||||||
Reference in New Issue
Block a user