ViperEkura
27abb7c5e7
- delete the warp-per-row kernel and the (6912,1536) M=2/4/8 dispatch table; under rotated cold weights the warp path is 2-6% slower than CTA reuse at M=2/4, and the table had been tuned against L2-resident timing - a single CTA-reuse kernel now serves all M in [1, 8]; block size is 256 threads for M in [1, 7] and 128 for M=8, where the shorter shared-memory reduction tree wins - document in docs/developer/swiglu_benchmark.md that the earlier operator numbers were L2-resident: the fused kernel sits at the dual-stream cold-read floor (702 vs 699 GB/s at (6912,1536); 369 vs 370 GB/s at (11008,4096)) and wide matrices cap at ~370-400 GB/s even for pure reads, so the reported M=8 -23% regression does not survive the cold regime - update docs/developer/cuda_kernels.md accordingly Benchmark: L20 (sm_89), PyTorch 2.11.0+cu128, rotated weight copies >= 240 MB to defeat the 96 MB L2; end-to-end through the built module at (6912,1536) reaches 738-752 GB/s for M in [1, 4] and 702 GB/s at M=8, about +8% at M=2/4 and +6% at M=8 over the removed warp path
84 lines
3.8 KiB
Markdown
84 lines
3.8 KiB
Markdown
# Fused SwiGLU benchmark
|
|
|
|
`scripts/tools/benchmark_swiglu.py` compares the directly callable fused BF16
|
|
SwiGLU primitive with both `F.linear` and the existing two-GEMV chain. It covers
|
|
the native AstrAI 1B MLP plus LLaMA 2 7B/13B, LLaMA 3 8B, and GPT-NeoX 20B
|
|
up/gate shapes at M=1/2/4/8 in eager and CUDA Graph modes.
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 python scripts/tools/benchmark_swiglu.py \
|
|
--output results/swiglu.json \
|
|
--markdown-output results/swiglu.md \
|
|
--m-values 1,2,4,8 --mode both \
|
|
--warmup 20 --iterations 100 --trials 10
|
|
```
|
|
|
|
Each trial uses A-B-C-C-B-A ordering to balance clock, cache, and temperature
|
|
drift. The generated JSON records every timing sample, p50/p90/p99, CUDA launch
|
|
count, maximum/mean absolute error, and cosine similarity.
|
|
|
|
## L20 findings
|
|
|
|
Hardware was one NVIDIA L20 (sm_89), PyTorch 2.11.0+cu128, CUDA 12.8. The
|
|
existing GPU5 inference service remained resident (15.4 GiB) but idle at the
|
|
sampling boundaries; no process or container was stopped.
|
|
|
|
For AstrAI 1B `(N,K)=(6912,1536)`, CUDA Graph medians were:
|
|
|
|
| M | torch (ms) | GEMV chain (ms) | fused (ms) | vs best unfused |
|
|
|---:|---:|---:|---:|---:|
|
|
| 1 | 0.02564 | 0.02298 | 0.01375 | +67.13% |
|
|
| 2 | 0.02484 | 0.02628 | 0.01416 | +75.40% |
|
|
| 4 | 0.02507 | 0.03839 | 0.01806 | +38.82% |
|
|
| 8 | 0.02563 | 0.07007 | 0.03339 | -23.24% |
|
|
|
|
The wide traditional shapes are weight-bandwidth dominated. CTA reuse keeps
|
|
the fused primitive within roughly -1.2% to +0.9% of the best unfused chain,
|
|
so none is eligible for automatic selection. This negative crossover is kept
|
|
in the raw evidence rather than hidden by a favorable subset.
|
|
|
|
The real 24-layer AstrAI checkpoint was then run through `InferenceEngine`,
|
|
including scheduler, sampling, and CUDA Graph. A-B-B-A medians were:
|
|
|
|
| Batch | unfused (ms/step) | forced fused (ms/step) | throughput gain |
|
|
|---:|---:|---:|---:|
|
|
| 1 | 4.125 | 3.925 | +5.10% |
|
|
| 2 | 4.245 | 4.055 | +4.69% |
|
|
| 4 | 4.475 | 4.305 | +3.95% |
|
|
|
|
## Dispatch decision
|
|
|
|
Direct correctness stayed close (`max_abs <= 2.4e-4`, cosine approximately
|
|
1.0), but deterministic greedy generations changed at M=1, M=2, and M=4.
|
|
|
|
For that reason no SM89 shape is enabled in `auto`. The default path stays on
|
|
the existing unfused linear backend, including any independently qualified
|
|
GEMV dispatch. `ASTRAI_SWIGLU=1` remains an explicit benchmark/experimentation
|
|
switch for callers that accept normal BF16 reduction-order variation. A future
|
|
automatic band must repeat both the performance and checkpoint-output gates.
|
|
|
|
## HBM re-measurement and kernel simplification
|
|
|
|
The operator numbers above are L2-resident: the AstrAI pair is 40.5 MB,
|
|
smaller than the 96 MB L2, so a tight timing loop re-reads warm weights
|
|
(13.75 us implies ~3.1 TB/s, far above the 864 GB/s spec). Real decode rotates
|
|
~1 GB of per-layer weights through L2 every step, so every call is cold.
|
|
|
|
Re-measuring with rotated weight copies (>= 240 MB working set) on the same
|
|
L20 showed:
|
|
|
|
- The fused CTA-reuse kernel sits at the dual-stream cold-read floor
|
|
(702 vs 699 GB/s at (6912,1536); 369 vs 370 GB/s at (11008,4096)). Wide
|
|
LLaMA matrices cap at ~370-400 GB/s regardless of kernel, even for a
|
|
pure-read loop, so the old per-variant gaps there were noise.
|
|
- The `(6912,1536)` warp-per-row variant (formerly M=2/4/8) is 2-6% slower
|
|
than CTA reuse at M=2/4 under cold weights and no longer wins at M=8 once
|
|
the CTA drops to 128 threads. It and its dispatch table were deleted.
|
|
- New rule: 256 threads for M in [1, 7], 128 threads for M=8. End-to-end
|
|
through the built module at (6912,1536): 738-752 GB/s for M in [1, 4] and
|
|
702 GB/s at M=8 (+6% over the removed warp path).
|
|
|
|
The M=8 CUDA-Graph regression reported above (`-23.24%`) does not survive the
|
|
cold-weight regime: cuBLAS reaches L2 bandwidth in the warm loop while both
|
|
fused paths converge to the same HBM floor.
|