ViperEkura
3e67b4f88d
- Single-kernel rotary embedding (cos/sin lookup + rotation) replaces PyTorch complex-multiply path (3 kernel launches + f32 upcast per call) - RotaryEmbedding now stores cos_table/sin_table and returns (cos, sin) f32 tuple instead of a complex tensor - apply_rotary_emb in rotary_backend.py auto-dispatches: CUDA kernel if available, else torch complex-multiply fallback; backend-agnostic (both attention backends benefit) - Kernel: 256-thread blocks, grid-stride loop, vectorized __nv_bfloat162 load/store, f32 compute, bf16 out - Standalone kernel 6-9x faster than torch across decode/prefill shapes, max diff 0 (decode) to 3e-2 (large prefill, bf16) - Benchmark (L20, bf16, CUDA backend): B=1 9.48->7.25ms (+31%), B=4 10.73->7.67ms (+40%), B=8 10.77->7.81ms (+38%), B=16 10.79->7.83ms (+38%)
50 lines
1.4 KiB
Python
50 lines
1.4 KiB
Python
"""CUDA attention kernel wrappers with torch fallback.
|
|
|
|
Public API:
|
|
- ``attn_decode`` — single-query decode attention
|
|
- ``attn_prefill`` — multi-query prefill attention
|
|
- ``attn_paged_decode`` — paged decode attention (direct page-table access)
|
|
- ``AttentionBackend`` — ABC for attention computation strategies
|
|
- ``TorchNativeBackend`` — default SDPA backend with KV cache I/O
|
|
- ``CudaBackend`` — CUDA kernel backend with paged decode + prefill
|
|
|
|
Layout convention: all q/k/v are ``[batch, seq_len, n_heads, head_dim]``
|
|
(blhd). Scale is always ``1/sqrt(head_dim)``.
|
|
|
|
Each wrapper calls its compiled CUDA kernel directly. Fallback to torch
|
|
SDPA is handled by the attention backend, not the wrapper functions.
|
|
"""
|
|
|
|
from astrai.extension.attention_backend import (
|
|
ATTN_BACKEND,
|
|
AttentionBackend,
|
|
CudaBackend,
|
|
TorchNativeBackend,
|
|
attention,
|
|
attn_backend,
|
|
get_backend,
|
|
)
|
|
from astrai.extension.attention_ops import (
|
|
attn_decode,
|
|
attn_paged_decode,
|
|
attn_prefill,
|
|
)
|
|
from astrai.extension.loader import KERNEL_NAMES, is_available
|
|
from astrai.extension.rotary_backend import apply_rotary_emb
|
|
|
|
__all__ = [
|
|
"ATTN_BACKEND",
|
|
"AttentionBackend",
|
|
"CudaBackend",
|
|
"TorchNativeBackend",
|
|
"attention",
|
|
"attn_backend",
|
|
"get_backend",
|
|
"attn_decode",
|
|
"attn_paged_decode",
|
|
"attn_prefill",
|
|
"is_available",
|
|
"KERNEL_NAMES",
|
|
"apply_rotary_emb",
|
|
]
|