ViperEkura
d08a92c7bd
feat: add frequency penalty to inference sampling pipeline
...
- Add FrequencyPenaltyStrategy (logit -= penalty * count)
- Per-task rep_window for penalty history lookup
- Wire through engine, task, executor, API layer
- Add --frequency_penalty and --rep_window to stream_chat.py
- 9 unit tests for frequency penalty strategy
2026-07-17 21:28:31 +08:00
ViperEkura
849e1e00a3
refactor: clean up inference design patterns
...
1. KVCache base: add default task_cached/task_record_hashes, remove getattr from scheduler
2. Remove page_size param from scheduler constructor (ContiguousCache-only)
3. InferenceEngine expose cache param for KVCache injection
4. Rename page_cache -> kv_cache in Executor
5. Move stream_callback from Task to TaskManager._callbacks dict
6. TaskManager.clear_queues clears callbacks
2026-07-05 11:41:54 +08:00
ViperEkura
5416c2e8fb
perf: replace paged KV cache with contiguous ContiguousCache, decode all groups
...
- Add KVCache/CacheView abstract base classes in cache.py
- Add ContiguousCache (contiguous per-slot buffer, default) alongside PageCache (paged, renamed from old KVCache)
- Merge make_table_tensor + bind into bind_tasks on KVCache interface
- Remove task_cached/task_record_hashes from base class (PageCache-only)
- Scheduler: decode all position groups instead of just the largest (eliminates 63% group skip rate)
- Scheduler: accept optional cache param for swapping implementations
- Model layer type hints use CacheView base class
- Batch 1-32: 1-7% speedup from eliminating Storage.gather overhead
- All 183 inference tests pass
2026-07-05 11:34:36 +08:00
ViperEkura
cb8dcb97ea
refactor : 移除 -> None 返回值标注，拆分 FSDP 参数，新增 mmap 数据集存储
...
- 删除所有 def 函数 -> None 返回值类型标注
- FSDPExecutor 参数从 **kwargs 拆为显式声明，None 值自动过滤
- 新增 MmapStorage (bin) 存储后端，基于 numpy.memmap 零拷贝加载
- 新增 save_bin/load_bin/json_to_bin 工具函数
- detect_format 支持 bin 格式自动检测
2026-05-28 13:57:06 +08:00
ViperEkura
e3382f6bb5
fix: 修复推理引擎 batch decode 中多项正确性与并发问题
...
- scheduler: decode 分组由幂次分桶改为精确 next_pos，消除 KV cache 位置错乱
- task: activate() 加锁操作 active_tasks，消除数据竞争
- engine: wait_completion 加超时，防止分配失败时永久死锁
- sample: TopKStrategy 向量化为 per-sample threshold，尊重各 task 的 top_k
- cache: Storage.write/gather 中 -1 页改用 mask 处理，防数据污染
- executor: prefill 逐 task 循环改为单次 tensor 调用
2026-05-14 21:31:39 +08:00
ViperEkura
205b40bd28
refactor: 重构 cache 和 inference 参数体系，分离存储与分配
...
- 合并 GenerationRequest/GenerationParams，统一 max_tokens 参数名
- PagePool/PrefixCache 分离为 Allocator + PrefixCache + PagePool
- 拆分 KV 存储为独立 Storage 类，PagedCache → KVCache，CacheView → KvcacheView
- Allocator.inc_ref 移除 LRU 防止竞争，Storage.write 增加负页防御
- Allocator/PrefixCache/TaskTable 加 threading.Lock 保证线程安全
- server.py uvicorn.run 改为传 app 对象修复导入错误
- benchmark.py 适配 KVCache 新 API
2026-05-14 20:05:08 +08:00
ViperEkura
18fe6e9339
refactor: 消除多处重复模式，统一工厂和参数传递
...
- AutoModel 继承 BaseFactory，消除自建 Registry（-30 行）
- executor.execute_prefill 删除重复 forward 代码块（bug）
- train_callback 移除 Protocol 上矛盾的 issubclass 检查
- engine.py 内部方法统一传 GenerationParams，校验内聚
- protocol.py SSEBuilder 类→函数，handle() 用 GenerationParams
- StreamContext 动态属性改为显式 dataclass 字段
- BaseFactory 新增 get_component_class 方法
2026-05-14 18:00:50 +08:00
ViperEkura
2196c34c52
refactor: 重构 inference 模块架构，引入设计模式并分组文件
...
- 新增 protocol.py 协议层，Template Method 模式消除流/非流分支 45% 重复
- SSEBuilder 统一 SSE 构造，StopChecker 独立 stop_sequence 检测
- AnthropicHandler 追踪已产出文本，修复 stop 时重复 delta
- server.py 路由从约 100 行缩减至 3 行
- 拆分为 core/（cache/executor/scheduler/task）和 api/（protocol/server）
- 外部保持二级导入路径（from astrai.inference import Name）
- 删除所有分隔线注释，代码按语义自然分组
2026-05-14 17:42:37 +08:00
ViperEkura