feat: add CUDA-graph capture for decode forward

- New CudaGraphContext class: warmup -> capture -> replay lifecycle
- One graph per batch_size key, all inputs at fixed workspace addresses
- Added position_ids buffer to InferenceWorkspace (required for graph capture)
- Graph only activates when CUDA backend is the current backend
- Default off (opt-in) due to slight numerical divergence in graph replay
- Sampling stays outside the graph (torch.multinomial uses mutable RNG)
- Resolved circular import: KVCache -> TYPE_CHECKING in attention_backend.py
This commit is contained in:
2026-08-06 19:57:12 +08:00
parent 6f67ba8942
commit 6054b8dbd4
4 changed files with 165 additions and 32 deletions
+5
View File
@@ -88,6 +88,11 @@ class InferenceWorkspace:
(max_batch_size, 1), dtype=torch.long, device=device
)
# Per-step position IDs (must be at a fixed address for CUDA-graph capture).
self.position_ids = torch.empty(
(max_batch_size,), dtype=torch.long, device=device
)
# Split-KV partial-result buffers for decode (persistent, one global
# alloc per process — mirrors FlashInfer's workspace pattern).
# Shape: [max_batch_size, max_q_heads, _MAX_SPLITS, head_dim] (o_part)