refactor: clean up inference design patterns
1. KVCache base: add default task_cached/task_record_hashes, remove getattr from scheduler 2. Remove page_size param from scheduler constructor (ContiguousCache-only) 3. InferenceEngine expose cache param for KVCache injection 4. Rename page_cache -> kv_cache in Executor 5. Move stream_callback from Task to TaskManager._callbacks dict 6. TaskManager.clear_queues clears callbacks
This commit is contained in:
@@ -303,6 +303,13 @@ class KVCache(ABC):
|
||||
self, task_ids: List[str], total_len: int, device: torch.device
|
||||
) -> CacheView: ...
|
||||
|
||||
def task_cached(self, task_id: str) -> int:
|
||||
return 0
|
||||
|
||||
def task_record_hashes(
|
||||
self, task_id: str, prompt_ids: List[int], start_logical_page: int = 0
|
||||
): ...
|
||||
|
||||
|
||||
class PageCacheView(CacheView):
|
||||
"""Bundles Storage + page_table + total_len for attention layers."""
|
||||
|
||||
Reference in New Issue
Block a user