refactor: centralize batch state and split scheduler duties
- make decode steady state self-validating: gate the token fill and sampling reuse on the decode cache's own task signature, drop TaskCacheManager.last_task_signature_matches whose req-index signature recycles with slot reuse - extract PolicyVersionGuard (version protocol plus generation/weight mutex) and Stepper (shared one-token advancement) out of the scheduler, keeping its public API unchanged - pin the input staging buffer only on CUDA devices so CPU-only workspaces allocate
This commit is contained in:
@@ -80,7 +80,10 @@ class InferenceWorkspace:
|
||||
(max_batch_size,), dtype=torch.long, device=device
|
||||
)
|
||||
self._pin = torch.empty(
|
||||
(max_batch_size,), dtype=torch.long, pin_memory=True
|
||||
(max_batch_size,),
|
||||
dtype=torch.long,
|
||||
pin_memory=torch.cuda.is_available()
|
||||
and torch.device(device).type == "cuda",
|
||||
)
|
||||
|
||||
# KV-cache bind metadata (fixed shape, written by
|
||||
|
||||
Reference in New Issue
Block a user