docs: fix stale developer documentation claims
- Move task_alloc/task_free/task_extend/task_cached/task_record_hashes and bind from the PagePool card to a new TaskCacheManager card matching pool.py - Drop the nonexistent Executor tokenizer attribute and association, add task_cache instead - Add AllocationStrategy/ContiguousStrategy/PagedStrategy cards and point Allocator/RadixCache composition at PagedStrategy - Add TaskCacheManager and the allocation strategies to the module overview, add _task_cache to InferenceScheduler - Fix the design-pattern count in the table of contents (15 -> 16) - Rewrite the FlashAttnBackend class docstring: packed decode gathers flat K/V via req_to_token and calls flash_attn_varlen_func; dense prefill uses flash_attn_func (no flash_attn_with_kvcache exists) - Apply the same correction to the backend bullets in internals.md and cuda_kernels.md - Rename the stale fp8_mma_test.cu reference to fp8_test.cu in cuda_kernels.md
This commit is contained in:
@@ -694,12 +694,13 @@ class CudaBackend(AttentionBackend):
|
||||
class FlashAttnBackend(AttentionBackend):
|
||||
"""FlashAttention backend via the optional ``flash-attn`` package.
|
||||
|
||||
Decode (q_len=1, contiguous cache): uses ``flash_attn_with_kvcache``,
|
||||
which reads K/V directly from the flat pool via cache_batch_idx +
|
||||
cache_seqlens — no materialized KV gather.
|
||||
Decode (q_len=1, contiguous cache): writes K/V to the pool, gathers
|
||||
flat K/V via the ``req_to_token`` page table, and calls
|
||||
``flash_attn_varlen_func`` over the ragged batch
|
||||
(``qo_indptr``/``kv_indptr``).
|
||||
|
||||
Prefill / non-contiguous decode: falls back to KV gather +
|
||||
``flash_attn_func``.
|
||||
Prefill: packed 3-D calls share the ``flash_attn_varlen_func`` path;
|
||||
dense 4-D calls go through ``flash_attn_func`` (mask-free only).
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
|
||||
Reference in New Issue
Block a user