- replace the per-request python loop in the torch-native 3-d inference path with a single F.scaled_dot_product_attention call over [B, max_q, max_kv] padded tensors
- fold the per-request causal offset (seq_len - q_len) and kv padding into one bool mask [B, 1, max_q, max_kv]; padded q rows gather row 0 and are dropped by the [q_valid] unpack, which restores the flat qo_indptr order
- decode drops from B sdpa launches plus ~3B host syncs (int() on qo_indptr/seq_lens per request) to one call with two syncs
- masked lanes contribute exact-zero weights, so prefill logits stay bitwise identical and test_prefill_with_kv_cache_matches_torch still passes its diff == 0.0 requirement
Benchmark: NVIDIA L20, CUDA 12.8, torch 2.11.0+cu128, bf16 decode microbench (GQA 32/8 heads, head_dim 128, batch 32, seq_lens 200-2000), 20 trials
- per-layer decode attention: 117.5 -> 9.5 ms (~12x)
- max abs diff vs the per-request loop: 0.002 (bf16 noise; the decode invariant tolerates 0.05)
- tests: extension + inference suites 350 passed
- split KVCache into phase-specific PrefillKVCache/DecodeKVCache types selected by start_pos
- unify steady-state detection in TaskCacheManager
- guard decode steady-state reuse with the cached task signature so recycled req slots cannot replay a prior generation's tokens and positions
- collapse attention backend fwd_decode/fwd_prefill into a single subclass-owned forward with a shared _check_fwd guard
- fix thread-safety gap in weight update and validate prefill inputs before KV allocation
- centralize magic constants in InferenceConfig and align docs with behavior
- add astrai/extension/dispatch.py: per-family decision tables over composable Specs with explicit-strict / implicit-loose resolution, ASTR_OPS env overrides, profile presets, and explain traces
- make the axis schema family-owned: register_family takes an axes extractor that snapshots whatever decision axes that family needs from the call, and the core only supplies the axis() predicate vocabulary plus a tensor_axes helper
- drop the central CallContext dataclass; resolve and explain take the raw call arguments, so unregistered handles are probed through supports_call on the same args
- migrate attention and rotary onto family-owned axes with behavior-preserving specs and spec-vs-supports_call mirror tests
- replace the non-ASCII member-of glyph in spec descriptions with plain ASCII " in "
- Move task_alloc/task_free/task_extend/task_cached/task_record_hashes and bind from the PagePool card to a new TaskCacheManager card matching pool.py
- Drop the nonexistent Executor tokenizer attribute and association, add task_cache instead
- Add AllocationStrategy/ContiguousStrategy/PagedStrategy cards and point Allocator/RadixCache composition at PagedStrategy
- Add TaskCacheManager and the allocation strategies to the module overview, add _task_cache to InferenceScheduler
- Fix the design-pattern count in the table of contents (15 -> 16)
- Rewrite the FlashAttnBackend class docstring: packed decode gathers flat K/V via req_to_token and calls flash_attn_varlen_func; dense prefill uses flash_attn_func (no flash_attn_with_kvcache exists)
- Apply the same correction to the backend bullets in internals.md and cuda_kernels.md
- Rename the stale fp8_mma_test.cu reference to fp8_test.cu in cuda_kernels.md
ViperEkura