perf: drop swiglu warp-rows variant for an M=8 block-size rule
- delete the warp-per-row kernel and the (6912,1536) M=2/4/8 dispatch table; under rotated cold weights the warp path is 2-6% slower than CTA reuse at M=2/4, and the table had been tuned against L2-resident timing - a single CTA-reuse kernel now serves all M in [1, 8]; block size is 256 threads for M in [1, 7] and 128 for M=8, where the shorter shared-memory reduction tree wins - document in docs/developer/swiglu_benchmark.md that the earlier operator numbers were L2-resident: the fused kernel sits at the dual-stream cold-read floor (702 vs 699 GB/s at (6912,1536); 369 vs 370 GB/s at (11008,4096)) and wide matrices cap at ~370-400 GB/s even for pure reads, so the reported M=8 -23% regression does not survive the cold regime - update docs/developer/cuda_kernels.md accordingly Benchmark: L20 (sm_89), PyTorch 2.11.0+cu128, rotated weight copies >= 240 MB to defeat the 96 MB L2; end-to-end through the built module at (6912,1536) reaches 738-752 GB/s for M in [1, 4] and 702 GB/s at M=8, about +8% at M=2/4 and +6% at M=8 over the removed warp path
This commit is contained in:
@@ -125,12 +125,14 @@ BF16 inputs with `M` in `[1, 8]` and K divisible by 8. It preserves the BF16
|
||||
rounding boundaries of the two projection outputs, SiLU output, and final
|
||||
product while accumulating dot products in FP32.
|
||||
|
||||
The kernel contains two output-row tilings. A CTA-reuse path reads each up/gate
|
||||
weight chunk once and applies it to all M rows. The native AstrAI 1B shape
|
||||
`(N,K)=(6912,1536)` uses one warp per decode row for M=2/4/8; on L20 this
|
||||
removes the shared reductions and barrier and reduces M=4 CUDA-Graph latency
|
||||
from 0.0324 ms to 0.0181 ms. Wider LLaMA/GPT-NeoX matrices keep CTA reuse,
|
||||
because duplicating their weight reads across row warps regressed 1.3-4.2%.
|
||||
The kernel is a single CTA-reuse tiling: one CTA per output column reads each
|
||||
up/gate weight chunk once and applies it to all M rows. Block size is 256
|
||||
threads for M in [1, 7] and 128 for M=8, where the shorter shared-memory
|
||||
reduction tree wins under cold-HBM decode traffic. An earlier per-shape
|
||||
`(6912,1536)` warp-per-row variant and its dispatch table were removed: HBM
|
||||
measurements with rotated weights showed the table was tuned against L2-cache
|
||||
regime timing and was up to 6% slower than CTA reuse at M=2/4; the kernel is
|
||||
bandwidth-bound, so finer variant selection is noise.
|
||||
|
||||
Dense `MLP` modules route through the SwiGLU backend. `ASTRAI_SWIGLU=0` keeps
|
||||
the unfused linear backend, and `1` explicitly forces the fused primitive.
|
||||
|
||||
Reference in New Issue
Block a user