feat: add CUDA-graph capture for decode forward
- New CudaGraphContext class: warmup -> capture -> replay lifecycle - One graph per batch_size key, all inputs at fixed workspace addresses - Added position_ids buffer to InferenceWorkspace (required for graph capture) - Graph only activates when CUDA backend is the current backend - Default off (opt-in) due to slight numerical divergence in graph replay - Sampling stays outside the graph (torch.multinomial uses mutable RNG) - Resolved circular import: KVCache -> TYPE_CHECKING in attention_backend.py
This commit is contained in:
@@ -88,6 +88,11 @@ class InferenceWorkspace:
|
||||
(max_batch_size, 1), dtype=torch.long, device=device
|
||||
)
|
||||
|
||||
# Per-step position IDs (must be at a fixed address for CUDA-graph capture).
|
||||
self.position_ids = torch.empty(
|
||||
(max_batch_size,), dtype=torch.long, device=device
|
||||
)
|
||||
|
||||
# Split-KV partial-result buffers for decode (persistent, one global
|
||||
# alloc per process — mirrors FlashInfer's workspace pattern).
|
||||
# Shape: [max_batch_size, max_q_heads, _MAX_SPLITS, head_dim] (o_part)
|
||||
|
||||
Reference in New Issue
Block a user