perf: preload V in decode split-kv shared mem and cache partial tensors
- Preload V into shared memory alongside K to eliminate per-element KV address lookups in the inner softmax/accum loop (doubles smem) - Cache split-KV partial tensors (o_part, ml_part) with static tensors instead of per-call allocation in both decode and paged-decode paths - Force is_causal=True in CUDA decode backend (decode is always causal)
This commit is contained in:
@@ -440,8 +440,7 @@ class CudaBackend(AttentionBackend):
|
||||
kv_cache.req_pool_indices,
|
||||
kv_indptr,
|
||||
kv_cache.max_len,
|
||||
mask=attn_mask,
|
||||
is_causal=is_causal,
|
||||
is_causal=True,
|
||||
)
|
||||
return out.unsqueeze(1).flatten(2)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user