perf: preload V in decode split-kv shared mem and cache partial tensors

- Preload V into shared memory alongside K to eliminate per-element KV address lookups in the inner softmax/accum loop (doubles smem)
- Cache split-KV partial tensors (o_part, ml_part) with static tensors instead of per-call allocation in both decode and paged-decode paths
- Force is_causal=True in CUDA decode backend (decode is always causal)
This commit is contained in:
2026-08-06 18:27:00 +08:00
parent 4f2e03880b
commit d0c5debbab
5 changed files with 31 additions and 14 deletions
+1 -2
View File
@@ -440,8 +440,7 @@ class CudaBackend(AttentionBackend):
kv_cache.req_pool_indices,
kv_indptr,
kv_cache.max_len,
mask=attn_mask,
is_causal=is_causal,
is_causal=True,
)
return out.unsqueeze(1).flatten(2)