- replace the per-request python loop in the torch-native 3-d inference path with a single F.scaled_dot_product_attention call over [B, max_q, max_kv] padded tensors
- fold the per-request causal offset (seq_len - q_len) and kv padding into one bool mask [B, 1, max_q, max_kv]; padded q rows gather row 0 and are dropped by the [q_valid] unpack, which restores the flat qo_indptr order
- decode drops from B sdpa launches plus ~3B host syncs (int() on qo_indptr/seq_lens per request) to one call with two syncs
- masked lanes contribute exact-zero weights, so prefill logits stay bitwise identical and test_prefill_with_kv_cache_matches_torch still passes its diff == 0.0 requirement
Benchmark: NVIDIA L20, CUDA 12.8, torch 2.11.0+cu128, bf16 decode microbench (GQA 32/8 heads, head_dim 128, batch 32, seq_lens 200-2000), 20 trials
- per-layer decode attention: 117.5 -> 9.5 ms (~12x)
- max abs diff vs the per-request loop: 0.002 (bf16 noise; the decode invariant tolerates 0.05)
- tests: extension + inference suites 350 passed
ViperEkura
c9ceca1eb9