refactor: 推理协议层重构为策略/建造者模式
- ProtocolHandler 改为具体类,格式化委托给 ResponseBuilder - 新增 api/protocols/ 目录,含 OpenAIResponseBuilder、AnthropicResponseBuilder - GenContext、StopInfo 参数对象替代 StreamContext - 消除 Builder 的实例可变状态(accumulated、_yielded) - SSE 工具和停止检测收归 ProtocolHandler 统一管理 - prepare() 方法合并原来的 build_prompt、create_response_id - 参数校验去重:仅 GenerationRequest.init 负责校验 - Prefill 阶段提前短路完全命中的缓存任务
This commit is contained in:
@@ -108,7 +108,10 @@ class InferenceScheduler:
|
||||
continue
|
||||
|
||||
to_prefill = [
|
||||
t for t in self._task_mgr.get_active_tasks() if t.output_tokens == 0
|
||||
t
|
||||
for t in self._task_mgr.get_active_tasks()
|
||||
if t.output_tokens == 0
|
||||
and self._page_cache.task_cached(t.task_id) < len(t.prompt_ids)
|
||||
]
|
||||
if to_prefill:
|
||||
for t in to_prefill:
|
||||
|
||||
Reference in New Issue
Block a user