feat: add CudaBackend and rename to fwd_decode/fwd_prefill

- CudaBackend: paged decode via attn_paged_decode, prefill via attn_prefill
- Decode uses req_to_token as page_table with page_size=1
- Falls back to TorchNativeBackend when kernel unavailable
- Rename forward_decode/forward_extend to fwd_decode/fwd_prefill
- Register ATTN_BACKEND.CUDA in _BACKEND_REGISTRY
This commit is contained in:
2026-07-30 18:45:33 +08:00
parent 21bf37dd83
commit 32fd03a025
2 changed files with 117 additions and 12 deletions
+5 -2
View File
@@ -6,17 +6,19 @@ Public API:
- ``attn_paged_decode`` — paged decode attention (direct page-table access)
- ``AttentionBackend`` — ABC for attention computation strategies
- ``TorchNativeBackend`` — default SDPA backend with KV cache I/O
- ``CudaBackend`` — CUDA kernel backend with paged decode + prefill
Layout convention: all q/k/v are ``[batch, seq_len, n_heads, head_dim]``
(blhd). Scale is always ``1/sqrt(head_dim)``.
Each wrapper dispatches to its compiled CUDA kernel (``astrai.extension.attn_*``)
when available, otherwise falls back to ``torch.nn.functional.scaled_dot_product_attention``.
Each wrapper calls its compiled CUDA kernel directly. Fallback to torch
SDPA is handled by the attention backend, not the wrapper functions.
"""
from astrai.extension.attention_backend import (
ATTN_BACKEND,
AttentionBackend,
CudaBackend,
TorchNativeBackend,
attn_backend,
get_backend,
@@ -31,6 +33,7 @@ from astrai.extension.loader import KERNEL_NAMES, is_available
__all__ = [
"ATTN_BACKEND",
"AttentionBackend",
"CudaBackend",
"TorchNativeBackend",
"attn_backend",
"get_backend",