4 Commits

Include renames

Author	SHA1	Message	Date
ViperEkura	deb2d7e127	refactor: rebuild KV cache with three-layer separation architecture ... - Replace CacheView/ContiguousCache/PageCache with SGLang-inspired design: KVStorage (flat token-level NHD buffers [n_layers, size, H, D]), ReqToTokenPool (index table [req_idx, pos] -> token_slot), Allocator + PrefixCache (slot allocation with LRU and prefix sharing) - Add KVCache as pure dataclass passed to model: k_buffer, v_buffer, req_to_token, req_pool_indices, seq_lens, out_cache_loc - PagePool orchestrates all three layers, supports contiguous mode (pre-allocated per-request blocks, default) and paged mode (page_size=1 or >1 with dynamic allocation and prefix caching) - Attention layers now do raw buffer indexing instead of opaque write/gather method calls on CacheView objects - Update executor.bind_tasks signature: seq_lens list + start_pos - Rename paged_cache -> kv_cache throughout model/ and inference/	2026-07-30 17:19:06 +08:00
ViperEkura	5416c2e8fb	perf: replace paged KV cache with contiguous ContiguousCache, decode all groups ... - Add KVCache/CacheView abstract base classes in cache.py - Add ContiguousCache (contiguous per-slot buffer, default) alongside PageCache (paged, renamed from old KVCache) - Merge make_table_tensor + bind into bind_tasks on KVCache interface - Remove task_cached/task_record_hashes from base class (PageCache-only) - Scheduler: decode all position groups instead of just the largest (eliminates 63% group skip rate) - Scheduler: accept optional cache param for swapping implementations - Model layer type hints use CacheView base class - Batch 1-32: 1-7% speedup from eliminating Storage.gather overhead - All 183 inference tests pass	2026-07-05 11:34:36 +08:00
ViperEkura	205b40bd28	refactor: 重构 cache 和 inference 参数体系，分离存储与分配 ... - 合并 GenerationRequest/GenerationParams，统一 max_tokens 参数名 - PagePool/PrefixCache 分离为 Allocator + PrefixCache + PagePool - 拆分 KV 存储为独立 Storage 类，PagedCache → KVCache，CacheView → KvcacheView - Allocator.inc_ref 移除 LRU 防止竞争，Storage.write 增加负页防御 - Allocator/PrefixCache/TaskTable 加 threading.Lock 保证线程安全 - server.py uvicorn.run 改为传 app 对象修复导入错误 - benchmark.py 适配 KVCache 新 API	2026-05-14 20:05:08 +08:00
ViperEkura	2196c34c52	refactor: 重构 inference 模块架构，引入设计模式并分组文件 ... - 新增 protocol.py 协议层，Template Method 模式消除流/非流分支 45% 重复 - SSEBuilder 统一 SSE 构造，StopChecker 独立 stop_sequence 检测 - AnthropicHandler 追踪已产出文本，修复 stop 时重复 delta - server.py 路由从约 100 行缩减至 3 行 - 拆分为 core/（cache/executor/scheduler/task）和 api/（protocol/server） - 外部保持二级导入路径（from astrai.inference import Name） - 删除所有分隔线注释，代码按语义自然分组	2026-05-14 17:42:37 +08:00