refactor: centralize batch state and split scheduler duties

- make decode steady state self-validating: gate the token fill and sampling reuse on the decode cache's own task signature, drop TaskCacheManager.last_task_signature_matches whose req-index signature recycles with slot reuse
- extract PolicyVersionGuard (version protocol plus generation/weight mutex) and Stepper (shared one-token advancement) out of the scheduler, keeping its public API unchanged
- pin the input staging buffer only on CUDA devices so CPU-only workspaces allocate
This commit is contained in:
2026-09-04 14:44:22 +08:00
parent ae7fc3059a
commit 8e39d9d8c9
7 changed files with 304 additions and 185 deletions
+4 -1
View File
@@ -80,7 +80,10 @@ class InferenceWorkspace:
(max_batch_size,), dtype=torch.long, device=device
)
self._pin = torch.empty(
(max_batch_size,), dtype=torch.long, pin_memory=True
(max_batch_size,),
dtype=torch.long,
pin_memory=torch.cuda.is_available()
and torch.device(device).type == "cuda",
)
# KV-cache bind metadata (fixed shape, written by