fix : correct prefill sampling and record alignment
- sample the first token from prefill logits without duplicating the prompt tail - reject incomplete multi-output records before preprocessing alignment - cover cached generation and partial DPO records with regression tests
This commit is contained in:
@@ -22,6 +22,8 @@ def test_task_next_pos():
|
||||
task.input_tokens = 5
|
||||
assert task.next_pos == 5
|
||||
task.output_ids.append(4)
|
||||
assert task.next_pos == 5
|
||||
task.output_ids.append(5)
|
||||
assert task.next_pos == 6
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user