refactor: 重构推理引擎控制逻辑,修复连续批处理核心缺陷
- 修复 decode 阶段新任务覆盖已有任务的严重缺陷 - 修复线程安全问题(热路径无锁竞争) - 修复前缀缓存引用计数管理不当导致缓存被驱逐 - 修复 pad_id 缺失导致全量 prefill 崩溃 - 修复 RoPE 位置错乱(不同位置任务共用 start_pos) - 新增 slot 版本追踪实现前缀缓存零拷贝复用 - 新增异步流式生成接口避免阻塞事件循环 - 添加完整英文文档字符串
This commit is contained in:
@@ -32,8 +32,15 @@ def mock_model_param():
|
||||
@pytest.fixture
|
||||
def mock_engine():
|
||||
"""Create a mock InferenceEngine."""
|
||||
|
||||
async def _async_gen():
|
||||
yield "chunk1"
|
||||
yield "chunk2"
|
||||
yield "[DONE]"
|
||||
|
||||
mock = MagicMock()
|
||||
mock.generate.return_value = "mock response"
|
||||
mock.generate_async.return_value = _async_gen()
|
||||
mock.get_stats.return_value = {
|
||||
"total_tasks": 0,
|
||||
"total_tokens": 0,
|
||||
|
||||
Reference in New Issue
Block a user