fix: keep async rollouts version-consistent

- serialize shared-model optimizer updates with generation - reject future or over-lagged rollout results after asynchronous scoring - close cache publication races - persist policy versions in online checkpoints
2026-09-03 12:01:24 +08:00
parent ce2f9d13b3
commit 587b0ee046
16 changed files with 531 additions and 49 deletions
@@ -355,5 +355,6 @@ class TrainContextBuilder:
                generator=generator,
                reward_model=cfg.reward_model_fn(),
                rollout_interval=cfg.rollout_interval,
+                max_policy_lag=cfg.rollout_max_policy_lag,
            )
        )