perf: tune bf16 gemv and add opt-in fused swiglu
- deepen common-shape BF16 GEMV tuning with warp-row tiling for LLaMA/Qwen2/GPT-NeoX/OPT decode projections - add fused BF16 up/gate SwiGLU CUDA primitive with ASTRAI_SWIGLU=0/1/auto dispatch - keep the unfused linear backend as the default path; auto enables no shape until per-architecture checkpoint gates pass - fall back to the linear/torch chain when kernels are absent, on CPU, in training, or outside supported M/K/dtype shapes - add gemv/swiglu benchmark scripts, dispatch and parity tests, and kernel documentation Benchmark: NVIDIA L20 (sm_89), CUDA 12.8, PyTorch 2.11.0+cu128, idle GPU. AstrAI 1B config (24 layers, hidden 1536, vocab 100000), BF16, prompt 128, 32 greedy decode tokens, CUDA graphs enabled, A/B in separate interleaved processes (3 rounds, 8 trials each, medians). Default vs ASTRAI_SWIGLU=1 per generate call: batch 1 134.8->129.1 ms (+4.44%), batch 2 136.2->130.9 ms (+4.06%), batch 4 145.5->140.3 ms (+3.66%). Greedy output identical at batch 1, differs at batch 2/4, so auto stays unfused by default; kernelless fallback verified bit-identical greedy.
This commit is contained in:
@@ -27,6 +27,7 @@ from astrai.extension.backend import (
|
||||
attn_backend,
|
||||
get_backend,
|
||||
linear,
|
||||
swiglu,
|
||||
)
|
||||
from astrai.extension.dispatch import (
|
||||
Axes,
|
||||
@@ -51,6 +52,7 @@ from astrai.extension.ops import (
|
||||
attn_paged_decode,
|
||||
attn_prefill,
|
||||
bf16_gemv,
|
||||
bf16_swiglu,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -65,10 +67,12 @@ __all__ = [
|
||||
"attn_backend",
|
||||
"get_backend",
|
||||
"linear",
|
||||
"swiglu",
|
||||
"attn_decode",
|
||||
"attn_paged_decode",
|
||||
"attn_prefill",
|
||||
"bf16_gemv",
|
||||
"bf16_swiglu",
|
||||
"is_available",
|
||||
"KERNEL_NAMES",
|
||||
"apply_rotary_emb",
|
||||
|
||||
Reference in New Issue
Block a user