perf: tune bf16 gemv and add opt-in fused swiglu
- deepen common-shape BF16 GEMV tuning with warp-row tiling for LLaMA/Qwen2/GPT-NeoX/OPT decode projections - add fused BF16 up/gate SwiGLU CUDA primitive with ASTRAI_SWIGLU=0/1/auto dispatch - keep the unfused linear backend as the default path; auto enables no shape until per-architecture checkpoint gates pass - fall back to the linear/torch chain when kernels are absent, on CPU, in training, or outside supported M/K/dtype shapes - add gemv/swiglu benchmark scripts, dispatch and parity tests, and kernel documentation Benchmark: NVIDIA L20 (sm_89), CUDA 12.8, PyTorch 2.11.0+cu128, idle GPU. AstrAI 1B config (24 layers, hidden 1536, vocab 100000), BF16, prompt 128, 32 greedy decode tokens, CUDA graphs enabled, A/B in separate interleaved processes (3 rounds, 8 trials each, medians). Default vs ASTRAI_SWIGLU=1 per generate call: batch 1 134.8->129.1 ms (+4.44%), batch 2 136.2->130.9 ms (+4.06%), batch 4 145.5->140.3 ms (+3.66%). Greedy output identical at batch 1, differs at batch 2/4, so auto stays unfused by default; kernelless fallback verified bit-identical greedy.
This commit is contained in:
@@ -1,9 +1,9 @@
|
||||
"""Inference-only dispatch for AstrAI linear layers.
|
||||
|
||||
The CUDA GEMV path is deliberately narrow: automatic selection is enabled
|
||||
only for single-row BF16 shapes measured to beat ``F.linear`` on a supported
|
||||
architecture. Every training, prefill, unsupported-layout, and unmeasured
|
||||
call falls back to PyTorch.
|
||||
only for small decode batches and BF16 shapes measured to beat ``F.linear``
|
||||
on a supported architecture. Every training, prefill, unsupported-layout,
|
||||
and unmeasured call falls back to PyTorch.
|
||||
"""
|
||||
|
||||
import logging
|
||||
@@ -30,20 +30,72 @@ from astrai.extension.ops.gemv import bf16_gemv
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Shape keys are (N, K) for Y[M, N] = X[M, K] @ W[N, K].T. A band is
|
||||
# automatic only after both the per-shape >=5% and end-to-end decode >=3%
|
||||
# gates pass and checkpoint greedy output remains stable. M=1 and M=8 remain
|
||||
# empty on SM89; the safe M=2/4 bands improve real-engine throughput by
|
||||
# 11.8-14.0%.
|
||||
# automatic only after both the per-shape >=5% and projection-chain/engine
|
||||
# >=3% gates pass and output argmax remains stable. M=1 is limited to OPT 1.3B;
|
||||
# M=8 remains empty because at least one projection in each measured family
|
||||
# misses the per-shape gate even when its aggregate chain result is positive.
|
||||
_COMMON_TRANSFORMER_SM89_SHAPES = frozenset(
|
||||
{
|
||||
(1024, 4096), # LLaMA 3 8B K/V
|
||||
(4096, 4096), # LLaMA 2/3 7B/8B Q/O
|
||||
(11008, 4096), # LLaMA 2 7B gate/up
|
||||
(4096, 11008), # LLaMA 2 7B down
|
||||
(14336, 4096), # LLaMA 3 8B gate/up
|
||||
(4096, 14336), # LLaMA 3 8B down
|
||||
(5120, 5120), # LLaMA 2 13B Q/K/V/O
|
||||
(13824, 5120), # LLaMA 2 13B gate/up
|
||||
(5120, 13824), # LLaMA 2 13B down
|
||||
(16384, 4096), # GPT-NeoX MLP up
|
||||
(4096, 16384), # GPT-NeoX MLP down
|
||||
}
|
||||
)
|
||||
_COMMON_TRANSFORMER_SM89_M4_SHAPES = _COMMON_TRANSFORMER_SM89_SHAPES - {
|
||||
(4096, 4096),
|
||||
(11008, 4096),
|
||||
(4096, 11008),
|
||||
}
|
||||
_QWEN2_7B_SM89_SHAPES = frozenset(
|
||||
{
|
||||
(512, 3584), # K/V
|
||||
(3584, 3584), # Q/O
|
||||
(18944, 3584), # gate/up
|
||||
(3584, 18944), # down
|
||||
}
|
||||
)
|
||||
_LLAMA3_70B_SM89_SHAPES = frozenset(
|
||||
{
|
||||
(1024, 8192), # K/V
|
||||
(8192, 8192), # Q/O
|
||||
(28672, 8192), # gate/up
|
||||
(8192, 28672), # down
|
||||
}
|
||||
)
|
||||
_OPT_1_3B_SM89_SHAPES = frozenset(
|
||||
{
|
||||
(2048, 2048), # Q/K/V/O
|
||||
(8192, 2048), # MLP up
|
||||
(2048, 8192), # MLP down
|
||||
}
|
||||
)
|
||||
|
||||
_AUTO_GEMV_SHAPES: dict[tuple[int, int], dict[int, frozenset[tuple[int, int]]]] = {
|
||||
(8, 9): {
|
||||
2: frozenset(
|
||||
1: _OPT_1_3B_SM89_SHAPES,
|
||||
2: _COMMON_TRANSFORMER_SM89_SHAPES
|
||||
| _QWEN2_7B_SM89_SHAPES
|
||||
| _LLAMA3_70B_SM89_SHAPES
|
||||
| _OPT_1_3B_SM89_SHAPES
|
||||
| frozenset(
|
||||
{
|
||||
(256, 1536),
|
||||
(1536, 1536),
|
||||
(100000, 1536),
|
||||
}
|
||||
),
|
||||
4: frozenset({(256, 1536), (1536, 1536)}),
|
||||
4: _COMMON_TRANSFORMER_SM89_M4_SHAPES
|
||||
| _QWEN2_7B_SM89_SHAPES
|
||||
| _LLAMA3_70B_SM89_SHAPES
|
||||
| frozenset({(256, 1536), (1536, 1536)}),
|
||||
}
|
||||
}
|
||||
_AUTO_GEMV_M = frozenset(
|
||||
|
||||
Reference in New Issue
Block a user