perf: vectorize bf16 gemv and extend M support to 1-8
- Replace per-element loads with 128-bit uint4 vectorized loads (8 halves per access), improving every measured shape: q/k/v at M=2 from 6.0us to 5.4us, q_proj speedup 2.28-2.45x, mlp_down at M=4 2.76x, lm_head at M=1 +6-8%
- Extend kernel M support from {1,2,4,8} to all M in 1-8 via new BLOCK_M cases 3,5,6,7, since cuBLAS wmma templates pad small M to 8/16 rows and waste compute
- Keep the auto-dispatch allowlist unchanged: a 64-step greedy-walk probe on the real decode path showed mlp_down (K=6912) divergence at step 1 and argmax flips for every candidate odd-M band, the same noise class already present in the merged M=2/4 entries, so no entry has the stability evidence the gate requires
- Rejected alternatives with measurements: split-K accumulation (k/v shapes regress 6.0us to 9.2us, code removed) and MMA tiles (small M is DRAM-bound at ~1 FLOP/byte vs the ~138 needed)
- Update test_gemv M-rejection case to M=9 and test_linear_dispatch multirow fallback to M=9 for the widened range
Benchmark: 8x L20 (sm_89, CUDA 12.8), single-GPU microbench, 200 iters after 20 warmup, weights L2-resident; q(1536x1536) M=3 8.9->5.3us, kv(256x1536) M=3 8.7->3.0us, down(1536x6912) M=3 53.5->10.3us; full gate 691 passed, test_bf16_gemv_uses_current_stream passes in isolation after GPU contention rerun
This commit is contained in:
@@ -73,7 +73,7 @@ def _axes(
|
||||
x_shape = tuple(x.shape)
|
||||
weight_shape = tuple(weight.shape)
|
||||
m = 1 if x.ndim == 1 else (x.shape[0] if x.ndim == 2 else None)
|
||||
supported_m = m in (1, 2, 4, 8)
|
||||
supported_m = m is not None and 1 <= m <= 8
|
||||
shape_matches = (
|
||||
weight.ndim == 2
|
||||
and x.ndim in (1, 2)
|
||||
@@ -164,7 +164,7 @@ def _gemv_capable(x: Tensor, weight: Tensor, bias: Optional[Tensor]) -> bool:
|
||||
or weight.dtype != torch.bfloat16
|
||||
or weight.ndim != 2
|
||||
or x.ndim not in (1, 2)
|
||||
or (x.ndim == 2 and x.shape[0] not in (1, 2, 4, 8))
|
||||
or (x.ndim == 2 and not 1 <= x.shape[0] <= 8)
|
||||
or x.shape[-1] != weight.shape[1]
|
||||
or weight.shape[1] % 2 != 0
|
||||
or x.device != weight.device
|
||||
@@ -239,7 +239,7 @@ def linear(x: Tensor, weight: Tensor, bias: Optional[Tensor] = None) -> Tensor:
|
||||
"""Apply a linear projection with safe inference-only GEMV dispatch.
|
||||
|
||||
``ASTRAI_GEMV=0`` always uses PyTorch, ``1`` forces GEMV whenever the
|
||||
primitive can safely handle an M in ``{1, 2, 4, 8}``, and ``auto`` (the
|
||||
primitive can safely handle any M in ``{1, ..., 8}``, and ``auto`` (the
|
||||
default) uses only architecture/shape bands backed by benchmark evidence.
|
||||
"""
|
||||
# Preserve the shared dispatcher for explicit/context selection and
|
||||
|
||||
Reference in New Issue
Block a user