fix : correct prefill sampling and record alignment

- sample the first token from prefill logits without duplicating the prompt tail
- reject incomplete multi-output records before preprocessing alignment
- cover cached generation and partial DPO records with regression tests
This commit is contained in:
2026-08-05 22:20:29 +08:00
parent a317a4756b
commit 654e6eb0d1
7 changed files with 143 additions and 80 deletions
+2
View File
@@ -22,6 +22,8 @@ def test_task_next_pos():
task.input_tokens = 5
assert task.next_pos == 5
task.output_ids.append(4)
assert task.next_pos == 5
task.output_ids.append(5)
assert task.next_pos == 6