ViperEkura 4b25664c79 perf: precompute kv_indptr once per decode step ...

- bind_tasks builds kv_indptr (prefix sum of seq_lens) a single time
- fwd_decode/fwd_prefill reuse it instead of rebuilding per layer
- Removes 24 cumsum launches per decode step (was ~1ms/step at B=4)
- Decode B=4: 9.60 -> 7.82 ms/step (-18.5%), +22.8% tok/s

2026-08-01 16:09:26 +08:00

..

__init__.py

refactor: rebuild KV cache with three-layer separation architecture

2026-07-30 17:19:06 +08:00

cache.py

perf: precompute kv_indptr once per decode step

2026-08-01 16:09:26 +08:00

executor.py

perf: reduce decode overhead in scheduler and executor

2026-07-31 14:50:16 +08:00

scheduler.py

perf: reduce decode overhead in scheduler and executor

2026-07-31 14:50:16 +08:00

task.py

perf: use Rust-native DecodeStream for O(n) streaming decode

2026-07-28 14:32:10 +08:00