perf: pack gqa q-heads per prefill block to reuse kv tiles

- pack HB = min(G, WARPS) q heads per block; K/V tiles stream once per block instead of once per q head
- G=1 keeps the old grid; paged path splits 64-row host Q tiles into HB blocks along grid.x (host maps unchanged)

Benchmark: NVIDIA RTX 6000D, short-q/long-kv prefill 1.4-3.4x (G=8 B=16 q=16 kv=16k 4.22 -> 1.26 ms); full prefill/MHA/paged unchanged (compute-bound); verified vs SDPA G in {1,2,3,4,8,32}, 99 tests pass
This commit is contained in:
2026-08-31 13:39:47 +08:00
parent 7dd184a4e5
commit a1a1a6bf0f
5 changed files with 91 additions and 20 deletions
+11 -2
View File
@@ -88,8 +88,17 @@ struct PrefillLauncherMMA {
static void launch(AttentionParams<bf16>& p, cudaStream_t stream) {
using Config = PrefillConfigMap<HEAD_DIM, IsCausal>;
using Traits = KernelTraits<HEAD_DIM, Config::BC, Config::WARPS, Config::STAGES>;
constexpr int ROWS = Traits::BR * Config::WARPS;
dim3 grid(QSchedule::host_q_blocks(p, ROWS), p.q_head,
// GQA head packing: HB = min(G, WARPS) q-heads of one kv-head group
// share each block's K/V stream (~HB× less global K/V traffic).
// Each head gets WPH = WARPS/HB 16-row chunks per block, so per-head
// rows drop from 64 to BR*WPH while total mma work per K/V byte is
// unchanged. G=1 (MHA) reproduces the historical grid exactly.
const int G = p.q_head / p.kv_head;
const int HB = std::min(G, Config::WARPS);
const int WPH = Config::WARPS / HB;
constexpr int BR = Traits::BR;
dim3 grid(QSchedule::packed_grid_x(p, BR * WPH),
p.kv_head * ((G + HB - 1) / HB),
QSchedule::host_grid_batch(p));
dim3 block(Traits::NUM_THREADS);
attn_prefill_split_q_mma_kernel<Traits, QSchedule, KV, IsCausal, HasMask>