fix: 修复 remove_task 未释放 KV cache slot 导致第二轮对话死锁
- remove_task() 现在释放 KV cache slot 和 prefix cache 引用 - _refill_active_batch 中 alloc 失败时将剩余 task 推回 waiting_queue - 主循环增加 try/except 异常兜底,发送 _STOP 给所有 task - 重构:server.py 全局变量改为 ServerState 类;automodel.py 使用 Registry 替代裸 dict;合并 TrainContextBuilder 的 with_* 方法到 build()
This commit is contained in:
@@ -2,7 +2,6 @@
|
||||
|
||||
import asyncio
|
||||
import gc
|
||||
import logging
|
||||
import threading
|
||||
from typing import Any, AsyncGenerator, Dict, Generator, List, Optional, Union
|
||||
|
||||
@@ -12,8 +11,6 @@ import torch.nn as nn
|
||||
from astrai.inference.scheduler import _STOP, InferenceScheduler
|
||||
from astrai.tokenize import AutoTokenizer
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GenerationRequest:
|
||||
"""Request parameters for text generation.
|
||||
|
||||
Reference in New Issue
Block a user