perf: batch ragged prefill requests

- Pack prompts with a shared prefix start and attention backend into one forward. - Select per-request final logits from cumulative query lengths. - Cover ragged tokens, logprobs, scheduling, and documentation.
2026-09-02 15:00:16 +08:00
parent 800981d85a
commit 01bcd0d105
5 changed files with 117 additions and 20 deletions
@@ -822,7 +822,7 @@ classDiagram
            +InferenceWorkspace _workspace
            +Optional[str] device
            +Optional[torch.dtype] dtype
-            +execute_prefill(tasks, prompt_len, start_pos=0)
+            +execute_prefill(tasks, start_pos=0)
            +execute_decode(tasks, return_logprobs=False) Union[List[int], List[Tuple[int, float]]]
        }