perf: move decode split partials to InferenceWorkspace
- Replace per-.cu-file static cached tensors with workspace-managed pre-allocated buffers - InferenceWorkspace now owns decode_o_part / decode_ml_part (mirrors FlashInfer's workspace pattern) - KVCache carries the buffers through the backend -> C++ kernel chain - C++ kernels accept optional pre-allocated buffers; fallback to alloc_split_partials for backward compat - Pre-allocates once at Executor init, zero allocation in the decode hot loop - Prerequisite for CUDA-graph capture (all kernel addresses are stable)
This commit is contained in:
@@ -441,6 +441,8 @@ class CudaBackend(AttentionBackend):
|
||||
kv_indptr,
|
||||
kv_cache.max_len,
|
||||
is_causal=True,
|
||||
o_part_buf=kv_cache.decode_o_part,
|
||||
ml_part_buf=kv_cache.decode_ml_part,
|
||||
)
|
||||
return out.unsqueeze(1).flatten(2)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user