ViperEkura
1798474316
- split-K removed entirely: tiled kernel walks K in one pass, no partials/semas workspace, no memset, single launch per call
- skinny GEMM (M<=8) dispatch table replaces the hand-written switch
- shape-driven four-family table replaces plan_gemm: wide-N (n>=4096) default {16,64,64,3,128} with BM=32 at M>16; narrow-N deep-K rings {16,32,256,2,64} while the grid fits one wave, {16,32,128,2,64} past it
- narrow-N is K-serial: widening the grid measurably does nothing (BN 64->32 ties, doubled m_tiles tie, kv at 4 blocks ties q/o at 24); deeper K chunks win until 72KB smem forces one CTA per SM and past one wave the 2-wave quantization loses to BK=128
- launch-check macros in common/launch.cuh; smem opt-in for the 72KB/60KB rings
- rename kernels/bf16_*.cu to gemm.cu/swiglu.cu; module names unchanged
- Python gate: lm_head (N>32768) falls back to cuBLAS, band narrows to M<=32
- drop the stale per-op benchmark narratives; fold the live numbers into cuda_kernels.md
Benchmark: NVIDIA L20 (sm_89, 92 SMs), CUDA 12.8, bf16, L2-thrash weight rotation, per-call medians at M=16: q/o 9.5us, kv 8.6us, gate/up 33.3us, down 33.7us (down -29% vs prior default). End-to-end 1B decode (gen 128, 3 trials, tokens/s vs cuBLAS): B=1 260 vs 252, B=8 1660 vs 1446, B=16 2464 vs 2437, B=32 3620 vs 3690. Prior split-K dispatch measured B=16 2243 / B=32 3393.
97 lines
2.3 KiB
Python
97 lines
2.3 KiB
Python
"""CUDA kernel wrappers, operator dispatch, and backend selection.
|
|
|
|
Public API:
|
|
- ``attention``, ``linear``, ``swiglu``, ``apply_rotary_emb`` — op
|
|
families with safe torch fallbacks (see ``astrai.extension.backend``)
|
|
- ``attn_decode`` / ``attn_prefill`` / ``attn_paged_decode`` /
|
|
``attn_paged_prefill`` — direct attention kernel wrappers
|
|
- ``bf16_gemm`` / ``bf16_swiglu`` — directly callable linear/MLP kernels
|
|
- ``AttentionBackend`` / ``TorchNativeBackend`` / ``CudaBackend`` /
|
|
``FlashAttnBackend`` — attention backend strategies
|
|
- ``resolve`` / ``explain`` / ``op_backend`` / ``env_mode`` — the shared
|
|
operator dispatcher (see ``astrai.extension.dispatch``)
|
|
|
|
Layout convention: all q/k/v are ``[batch, seq_len, n_heads, head_dim]``
|
|
(blhd). Scale is always ``1/sqrt(head_dim)``. Wrapper functions call their
|
|
compiled CUDA kernels directly; fallback is the backend's responsibility.
|
|
"""
|
|
|
|
from astrai.extension.backend import (
|
|
ATTN_BACKEND,
|
|
AttentionBackend,
|
|
AttentionBackendFactory,
|
|
CudaBackend,
|
|
FlashAttnBackend,
|
|
TorchNativeBackend,
|
|
apply_rotary_emb,
|
|
attention,
|
|
attn_backend,
|
|
get_backend,
|
|
linear,
|
|
swiglu,
|
|
)
|
|
from astrai.extension.dispatch import (
|
|
Axes,
|
|
ExplicitSelectionError,
|
|
ImplRecord,
|
|
Resolution,
|
|
Spec,
|
|
axis,
|
|
env_mode,
|
|
explain,
|
|
explain_plan,
|
|
op_backend,
|
|
register_env_alias,
|
|
register_family,
|
|
resolve,
|
|
resolve_plan,
|
|
tensor_axes,
|
|
)
|
|
from astrai.extension.loader import KERNEL_NAMES, is_available
|
|
from astrai.extension.ops import (
|
|
TensorLayout,
|
|
attn_decode,
|
|
attn_paged_decode,
|
|
attn_prefill,
|
|
bf16_gemm,
|
|
bf16_swiglu,
|
|
)
|
|
|
|
__all__ = [
|
|
"ATTN_BACKEND",
|
|
"AttentionBackend",
|
|
"AttentionBackendFactory",
|
|
"CudaBackend",
|
|
"TorchNativeBackend",
|
|
"FlashAttnBackend",
|
|
"TensorLayout",
|
|
"attention",
|
|
"attn_backend",
|
|
"get_backend",
|
|
"linear",
|
|
"swiglu",
|
|
"attn_decode",
|
|
"attn_paged_decode",
|
|
"attn_prefill",
|
|
"bf16_gemm",
|
|
"bf16_swiglu",
|
|
"is_available",
|
|
"KERNEL_NAMES",
|
|
"apply_rotary_emb",
|
|
"Axes",
|
|
"ExplicitSelectionError",
|
|
"ImplRecord",
|
|
"Resolution",
|
|
"Spec",
|
|
"axis",
|
|
"env_mode",
|
|
"explain",
|
|
"explain_plan",
|
|
"op_backend",
|
|
"register_env_alias",
|
|
"register_family",
|
|
"resolve",
|
|
"resolve_plan",
|
|
"tensor_axes",
|
|
]
|