2 Commits

Author	SHA1	Message	Date
ViperEkura	2c7a71a9c0	refactor: separate old policy and ref model in GRPO strategy ... - Split single ref_model into old_model (importance sampling ratio) and ref_model (frozen KL regularizer) - Move ref_model/old_model creation from strategy __init__ to TrainContextBuilder, pass as explicit parameters - Remove periodic sync_ref_model + sync_interval; add sync_old_model for external rollout loop to call - DPOStrategy also receives ref_model from builder - Fix std to use unbiased=False (population std per GRPO paper) - Remove redundant tests (test_grpo_kl_zero_at_init, test_grpo_no_sync_interval_param) - Remove --grpo_sync_interval CLI arg	2026-07-14 20:03:45 +08:00
ViperEkura	9bcd696580	fix: token-level ratio and prompt masking in GRPO strategy ... - Mask prompt tokens to 0 so their logprobs excluded from ratio/KL - Switch to token-level ratio + PPO clipping via reduction='none' - Slice response token logprobs from full sequence output - Replace k3 KL estimator with non-negative k1 estimator - Fix epsilon from finifo.eps (~1e-38) to 1e-8 - Remove unused 'reduction' param from GRPOStrategy.__init__ - Clarify offline batch semantics in docstring - Add 11 unit tests for masking, advantage, KL, sync, clipping - Sync training.md and architecture.md docs	2026-07-12 21:24:09 +08:00