AstrAI/tests/inference
ViperEkura 5416c2e8fb perf: replace paged KV cache with contiguous ContiguousCache, decode all groups
- Add KVCache/CacheView abstract base classes in cache.py
- Add ContiguousCache (contiguous per-slot buffer, default) alongside PageCache (paged, renamed from old KVCache)
- Merge make_table_tensor + bind into bind_tasks on KVCache interface
- Remove task_cached/task_record_hashes from base class (PageCache-only)
- Scheduler: decode all position groups instead of just the largest (eliminates 63% group skip rate)
- Scheduler: accept optional cache param for swapping implementations
- Model layer type hints use CacheView base class
- Batch 1-32: 1-7% speedup from eliminating Storage.gather overhead
- All 183 inference tests pass
2026-07-05 11:34:36 +08:00
..
conftest.py refactor : FastAPI 懒加载单例,消除模块级副作用 2026-06-04 15:52:27 +08:00
test_cache.py perf: replace paged KV cache with contiguous ContiguousCache, decode all groups 2026-07-05 11:34:36 +08:00
test_engine.py refactor: 重构 inference 模块架构,引入设计模式并分组文件 2026-05-14 17:42:37 +08:00
test_protocol.py feat : BaseToolParser.feed 增加可选 token_ids 参数 2026-06-06 11:19:30 +08:00
test_sample.py fix: 修复推理引擎 batch decode 中多项正确性与并发问题 2026-05-14 21:31:39 +08:00
test_scheduler.py test: 补充推理协议层单测覆盖 2026-05-26 00:21:52 +08:00
test_server.py refactor : FastAPI 懒加载单例,消除模块级副作用 2026-06-04 15:52:27 +08:00
test_task.py refactor: 重构 inference 模块架构,引入设计模式并分组文件 2026-05-14 17:42:37 +08:00
test_tool_parser.py refactor : 压缩测试代码,消除重复 2026-06-19 14:54:39 +08:00