fix: resolve audited training and inference bugs
- reject prompts that encode to zero tokens in add_task instead of admitting a task whose prefill can never run, and surface empty-id run_batch calls as prompt_empty errors - deliver the STOP stream callback when cancelling a live task so clients observe termination instead of hanging until socket timeout - strip the torch.compile _orig_mod. prefix at every unwrap_model site and when loading checkpoints so FSDP state dicts and saved weights no longer leak the wrapper name into downstream keys - reject online_* train strategies with nprocs > 1 at config validation time, explaining the NCCL all-gather deadlock they would otherwise hit mid-run - apply the frequency penalty before temperature scaling (OpenAI semantics) so the penalty survives temperature=0 instead of being annihilated by the 1e8 logit blowup, and exclude penalty pipelines from the greedy fast path - return logprobs from the raw pre-strategy distribution so they match training-side policy logprobs for PPO/GRPO importance ratios
This commit is contained in:
@@ -444,6 +444,10 @@ class InferenceScheduler:
|
||||
tasks: List[Optional[Task]] = []
|
||||
error_reasons: List[Optional[str]] = []
|
||||
for ids in prompt_ids_list:
|
||||
if not ids:
|
||||
tasks.append(None)
|
||||
error_reasons.append("prompt_empty")
|
||||
continue
|
||||
if len(ids) >= seq_cap:
|
||||
tasks.append(None)
|
||||
error_reasons.append("prompt_too_long")
|
||||
|
||||
Reference in New Issue
Block a user