ViperEkura d033b2ef0f perf: cache per-step decode tensor construction ...

- SamplingBatchInfo: sample params built once per task set (top_k int32, pinned async H2D)
- position_ids advances by +1 on steady-state decode instead of re-building
- DecodeBindCache: bind_tasks increments seq_lens/kv_indptr, reuses req_pool_indices
- saves ~240us of python/launch overhead per decode step

2026-08-02 20:32:53 +08:00

..

api

refactor: migrate config system to Pydantic dataclasses

2026-07-30 08:25:32 +08:00

core

perf: cache per-step decode tensor construction

2026-08-02 20:32:53 +08:00

__init__.py

refactor: rebuild KV cache with three-layer separation architecture

2026-07-30 17:19:06 +08:00

engine.py

refactor: rebuild KV cache with three-layer separation architecture

2026-07-30 17:19:06 +08:00

sample.py

perf: reduce decode overhead in scheduler and executor

2026-07-31 14:50:16 +08:00