ViperEkura af25833fab fix: add out_buf to attn_paged_decode for CUDA graph capture compatibility ...

- Pre-allocate decode_out in InferenceWorkspace so attn_paged_decode does not call torch::empty inside graph capture
- Wire decode_out through KVCache, PagePool.bind_tasks, and CudaBackend.fwd_decode
- Run live forward before graph capture to get valid output (graph pool memory is zeroed after capture block exits)
- Greedy generation with graph replay is bit-exact across all batch sizes
- Decode speedups vs no-graph: B=1 2.09x, B=4 1.80x, B=8 1.94x, B=16 1.76x

2026-08-07 19:45:59 +08:00

..

api

refactor: migrate config system to Pydantic dataclasses

2026-07-30 08:25:32 +08:00

core

fix: add out_buf to attn_paged_decode for CUDA graph capture compatibility

2026-08-07 19:45:59 +08:00

__init__.py

feat : add radix prefix cache

2026-08-06 11:45:52 +08:00

engine.py

fix : handle zero-token batch generation

2026-08-06 12:31:09 +08:00

sample.py

refactor: unify greedy check with _is_greedy helper

2026-08-07 14:14:15 +08:00