ViperEkura/AstrAI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
658 Commits 1 Branch 14 Tags
d0e5d910dea58c9d20d49abc93cd1944e8e4430b
Commit Graph
10 Commits
Author SHA1 Message Date
ViperEkura a03504a280 perf: preallocate inference decode buffers 
- add InferenceWorkspace with fixed-shape per-step buffers (input_ids, decode mask, KV bind metadata) for CUDA-graph capture
- bind_tasks derives seq_lens from the pool's own _task_len tracking, dropping the seq_lens parameter
- update decode metadata in-place (position_ids, seq_lens, kv_indptr) instead of re-allocating per step
- task_extend advances _task_len in contiguous mode so the pool tracks current length
- skip log_softmax when logprobs are not requested
 2026-08-03 00:55:26 +08:00
ViperEkura df979b4469 refactor: use single-index access and update docs for cache architecture 
- Replace all buffer[layer_id][loc] double indexing with buffer[layer_id, loc] single advanced indexing in cache.py and attention.py
- Revert KVStorage buffers back to 4D [n_layers, size, n_kv_heads, head_dim], remove leftover 3D reshape/view in MLA path
- Update docs/guides/inference.md, docs/developer/internals.md, docs/developer/architecture.md to reflect new PagePool/KVStorage/ReqToTokenPool/KVCache classes
 2026-07-30 17:47:04 +08:00
ViperEkura deb2d7e127 refactor: rebuild KV cache with three-layer separation architecture 
- Replace CacheView/ContiguousCache/PageCache with SGLang-inspired design: KVStorage (flat token-level NHD buffers [n_layers, size, H, D]), ReqToTokenPool (index table [req_idx, pos] -> token_slot), Allocator + PrefixCache (slot allocation with LRU and prefix sharing)
- Add KVCache as pure dataclass passed to model: k_buffer, v_buffer, req_to_token, req_pool_indices, seq_lens, out_cache_loc
- PagePool orchestrates all three layers, supports contiguous mode (pre-allocated per-request blocks, default) and paged mode (page_size=1 or >1 with dynamic allocation and prefix caching)
- Attention layers now do raw buffer indexing instead of opaque write/gather method calls on CacheView objects
- Update executor.bind_tasks signature: seq_lens list + start_pos
- Rename paged_cache -> kv_cache throughout model/ and inference/
 2026-07-30 17:19:06 +08:00
ViperEkura 5416c2e8fb perf: replace paged KV cache with contiguous ContiguousCache, decode all groups 
- Add KVCache/CacheView abstract base classes in cache.py
- Add ContiguousCache (contiguous per-slot buffer, default) alongside PageCache (paged, renamed from old KVCache)
- Merge make_table_tensor + bind into bind_tasks on KVCache interface
- Remove task_cached/task_record_hashes from base class (PageCache-only)
- Scheduler: decode all position groups instead of just the largest (eliminates 63% group skip rate)
- Scheduler: accept optional cache param for swapping implementations
- Model layer type hints use CacheView base class
- Batch 1-32: 1-7% speedup from eliminating Storage.gather overhead
- All 183 inference tests pass
 2026-07-05 11:34:36 +08:00
ViperEkura 205b40bd28 refactor: 重构 cache 和 inference 参数体系，分离存储与分配 
- 合并 GenerationRequest/GenerationParams，统一 max_tokens 参数名
- PagePool/PrefixCache 分离为 Allocator + PrefixCache + PagePool
- 拆分 KV 存储为独立 Storage 类，PagedCache → KVCache，CacheView → KvcacheView
- Allocator.inc_ref 移除 LRU 防止竞争，Storage.write 增加负页防御
- Allocator/PrefixCache/TaskTable 加 threading.Lock 保证线程安全
- server.py uvicorn.run 改为传 app 对象修复导入错误
- benchmark.py 适配 KVCache 新 API
 2026-05-14 20:05:08 +08:00
ViperEkura 2196c34c52 refactor: 重构 inference 模块架构，引入设计模式并分组文件 
- 新增 protocol.py 协议层，Template Method 模式消除流/非流分支 45% 重复
- SSEBuilder 统一 SSE 构造，StopChecker 独立 stop_sequence 检测
- AnthropicHandler 追踪已产出文本，修复 stop 时重复 delta
- server.py 路由从约 100 行缩减至 3 行
- 拆分为 core/（cache/executor/scheduler/task）和 api/（protocol/server）
- 外部保持二级导入路径（from astrai.inference import Name）
- 删除所有分隔线注释，代码按语义自然分组
 2026-05-14 17:42:37 +08:00
ViperEkura 6d6ef99e66 perf: 消除 PagedCache.write 中的 position_ids GPU 同步，解码提速 15% 
- CacheView.write 用 total_len - k.size(1) 推导 start_pos，替代 position_ids[0,0].item()

- 移除 GQA/MLA/DecoderBlock 中不再使用的 position_ids 参数

- PagedCache.write 参数 position_ids:Tensor → start_pos:int
 2026-05-14 15:37:48 +08:00
ViperEkura c0effc9f5b refactor: 位置编码改用 position_ids [B,S]，简化 attention mask 构建 
- RotaryEmbedding/CacheView 接受 position_ids 替代 start_pos

- process_attention_mask 用 position_ids >= arange 做逐位置 causal

- 训练/无 KV cache 时 position_ids=None 内部自动处理

- 移除 executor/benchmark 中冗余的 input_mask 构造
 2026-05-14 13:26:31 +08:00
ViperEkura df0845e916 chore: 解耦 Executor/Scheduler/TaskManager，修复 stop 页泄漏，移除 ServerState 全局单例 2026-05-12 13:47:55 +08:00
ViperEkura 7d4029c2a4 test: inference 模块补全单元测试，cache/sample/engine/task 
- test_cache: page_hash, PagePool, PrefixCache, TaskTable, PagedCache write/gather
- test_sample: TemperatureStrategy, TopKStrategy, TopPStrategy, SamplingPipeline, sample()
- test_engine: _Result 线程安全, generate stream/non-stream batch/single
- test_task: Task 生命周期, TaskManager 队列操作
- 4 新文件, +771 行, 116 total tests
 2026-05-12 12:17:57 +08:00