fix: token-level ratio and prompt masking in GRPO strategy
- Mask prompt tokens to 0 so their logprobs excluded from ratio/KL - Switch to token-level ratio + PPO clipping via reduction='none' - Slice response token logprobs from full sequence output - Replace k3 KL estimator with non-negative k1 estimator - Fix epsilon from finifo.eps (~1e-38) to 1e-8 - Remove unused 'reduction' param from GRPOStrategy.__init__ - Clarify offline batch semantics in docstring - Add 11 unit tests for masking, advantage, KL, sync, clipping - Sync training.md and architecture.md docs
This commit is contained in:
@@ -499,7 +499,6 @@ classDiagram
|
||||
+float clip_eps
|
||||
+float kl_coef
|
||||
+int group_size
|
||||
+str reduction
|
||||
+int sync_interval
|
||||
+compute_loss(batch) Tensor
|
||||
+sync_ref_model()
|
||||
|
||||
Reference in New Issue
Block a user