feat: version rollout weight updates
Track a monotonic policy version across optimizer steps, scheduler updates, and rollout results. Serialize synchronous generation with weight acknowledgements and invalidate reusable prefix KV entries so cached samples remain attributable to the behavior policy that generated them.
This commit is contained in:
@@ -139,6 +139,12 @@ attention backends share the same rotary dispatch — it is backend-agnostic.
|
||||
4. Decode → Run single-token forward for each same-position group
|
||||
```
|
||||
|
||||
For in-process training rollout, `InferenceScheduler.update_weights(version)`
|
||||
acknowledges that the shared model was updated in place. Versions are monotonic;
|
||||
the scheduler rejects updates while requests are queued and invalidates reusable
|
||||
prefix KV pages before exposing the new version. Synchronous `run_batch()` and
|
||||
weight updates are serialized so a generation cannot straddle two versions.
|
||||
|
||||
## Sampling (Strategy Pattern)
|
||||
|
||||
```
|
||||
|
||||
@@ -166,6 +166,13 @@ them with a `BaseRewardModel`. It refreshes cached rollouts every
|
||||
`rollout_interval` optimizer steps. `online_grpo` synchronizes `old_model` when
|
||||
a fresh rollout is produced.
|
||||
|
||||
Every successful optimizer step advances a monotonic `policy_version` and
|
||||
acknowledges the shared-model weight update to the rollout scheduler. The
|
||||
scheduler invalidates reusable KV prefixes before accepting the new version.
|
||||
`RawRollout` and `RolloutResult` retain the version that actually generated
|
||||
their behavior log-probabilities, so cached rollout samples remain attributable
|
||||
even while later optimizer steps advance the live policy.
|
||||
|
||||
Online strategies require `TrainConfig.reward_model_fn`. `train.py` exposes the
|
||||
rollout sampling parameters but does not yet offer a CLI argument for the reward
|
||||
model factory.
|
||||
|
||||
Reference in New Issue
Block a user