refactor: decouple task cache from PagePool and unify steady-state detection
- TaskCacheRegistry -> TaskCacheManager (independent, held by scheduler) - TaskCacheState co-locates 5 parallel dicts into one dataclass - AllocationStrategy base class + PagedStrategy subclass (page_size is a parameter) - _rollback() helper for unified cleanup (no duplicate free paths) - Task._kv_len + prefill_done property (explicit, no output_tokens proxy) - Steady-state detection single-sourced in TaskCacheManager.bind() - PagePool is now pure physical layer (no task knowledge) - Removed dead _page_to_hash dict in RadixCache
This commit is contained in:
@@ -7,6 +7,7 @@ from astrai.inference.core.cache import (
|
||||
PagePool,
|
||||
RadixCache,
|
||||
ReqToTokenPool,
|
||||
TaskCacheManager,
|
||||
page_hash,
|
||||
)
|
||||
from astrai.inference.core.executor import Executor
|
||||
@@ -21,6 +22,7 @@ __all__ = [
|
||||
"PagePool",
|
||||
"RadixCache",
|
||||
"ReqToTokenPool",
|
||||
"TaskCacheManager",
|
||||
"page_hash",
|
||||
"Executor",
|
||||
"InferenceScheduler",
|
||||
|
||||
Reference in New Issue
Block a user