fix: token-level ratio and prompt masking in GRPO strategy

- Mask prompt tokens to 0 so their logprobs excluded from ratio/KL
- Switch to token-level ratio + PPO clipping via reduction='none'
- Slice response token logprobs from full sequence output
- Replace k3 KL estimator with non-negative k1 estimator
- Fix epsilon from finifo.eps (~1e-38) to 1e-8
- Remove unused 'reduction' param from GRPOStrategy.__init__
- Clarify offline batch semantics in docstring
- Add 11 unit tests for masking, advantage, KL, sync, clipping
- Sync training.md and architecture.md docs
This commit is contained in:
2026-07-12 21:24:09 +08:00
parent 8f89c82d55
commit 9bcd696580
4 changed files with 266 additions and 23 deletions
-1
View File
@@ -499,7 +499,6 @@ classDiagram
+float clip_eps
+float kl_coef
+int group_size
+str reduction
+int sync_interval
+compute_loss(batch) Tensor
+sync_ref_model()