ViperEkura
1798474316
- split-K removed entirely: tiled kernel walks K in one pass, no partials/semas workspace, no memset, single launch per call
- skinny GEMM (M<=8) dispatch table replaces the hand-written switch
- shape-driven four-family table replaces plan_gemm: wide-N (n>=4096) default {16,64,64,3,128} with BM=32 at M>16; narrow-N deep-K rings {16,32,256,2,64} while the grid fits one wave, {16,32,128,2,64} past it
- narrow-N is K-serial: widening the grid measurably does nothing (BN 64->32 ties, doubled m_tiles tie, kv at 4 blocks ties q/o at 24); deeper K chunks win until 72KB smem forces one CTA per SM and past one wave the 2-wave quantization loses to BK=128
- launch-check macros in common/launch.cuh; smem opt-in for the 72KB/60KB rings
- rename kernels/bf16_*.cu to gemm.cu/swiglu.cu; module names unchanged
- Python gate: lm_head (N>32768) falls back to cuBLAS, band narrows to M<=32
- drop the stale per-op benchmark narratives; fold the live numbers into cuda_kernels.md
Benchmark: NVIDIA L20 (sm_89, 92 SMs), CUDA 12.8, bf16, L2-thrash weight rotation, per-call medians at M=16: q/o 9.5us, kv 8.6us, gate/up 33.3us, down 33.7us (down -29% vs prior default). End-to-end 1B decode (gen 128, 3 trials, tokens/s vs cuBLAS): B=1 260 vs 252, B=8 1660 vs 1446, B=16 2464 vs 2437, B=32 3620 vs 3690. Prior split-K dispatch measured B=16 2243 / B=32 3393.
37 lines
1.8 KiB
Plaintext
37 lines
1.8 KiB
Plaintext
// Launch-and-check macros — pure CUDA, no torch, so out-of-tree harnesses
|
|
// (tile sweeps, csrc/tests) share the exact production launch discipline.
|
|
//
|
|
// Include order matters for overrides: define ASTRAI_LAUNCH_FAIL before
|
|
// including this header (directly or via another kernel header) to swap
|
|
// print+abort for a throwing check, as the torch entry units do with
|
|
// C10_CUDA_CHECK.
|
|
|
|
#pragma once
|
|
|
|
#include <cstdio>
|
|
#include <cstdlib>
|
|
#include <cuda_runtime.h>
|
|
|
|
|
|
#define ASTRAI_LAUNCH_FAIL(err, what) \
|
|
do { \
|
|
std::fprintf( \
|
|
stderr, "ASTRAI: %s failed: %s (%s:%d)\n", what, \
|
|
cudaGetErrorString(err), __FILE__, __LINE__ \
|
|
); \
|
|
std::abort(); \
|
|
} while (0)
|
|
|
|
#define ASTRAI_CUDA_CHECK(expr) \
|
|
do { \
|
|
cudaError_t astrai_err_ = (expr); \
|
|
if (astrai_err_ != cudaSuccess) { \
|
|
ASTRAI_LAUNCH_FAIL(astrai_err_, #expr); \
|
|
} \
|
|
} while (0)
|
|
|
|
// Check a kernel launch. Wrap the raw <<<>>> with this on the next line;
|
|
// a rejected configuration must fail loudly instead of silently measuring
|
|
// as a constant ~3us no-op (the tile-sweep lesson).
|
|
#define ASTRAI_LAUNCH_CHECK() ASTRAI_CUDA_CHECK(cudaGetLastError())
|