ViperEkura 3d3ea47d37 refactor: standardize packed 3d inference ...

- keep training attention on dense 4d tensors
- use packed 3d tensors with KV cache for inference
- extend CUDA rotary embedding to packed 3d inputs
- adapt torch, CUDA and FlashAttention backend dispatch

Benchmark: NVIDIA L20, BF16, 1B model, paged KV cache, CUDA Graph, prompt 512, generation 128 (median of 3 alternating runs)
- batch 1: 234.5 -> 242.6 tok/s (1.034x, +3.4%)
- batch 8: 1243.1 -> 1286.6 tok/s (1.035x, +3.5%)

2026-08-19 00:36:53 +08:00

..

__init__.py

refactor: separate extension ops and backends

2026-08-16 21:15:52 +08:00

attention.py

refactor: standardize packed 3d inference

2026-08-19 00:36:53 +08:00

fp8.py

perf: accelerate FP8 backward with fused fast kernel

2026-08-18 23:46:43 +08:00

rotary.py

refactor: separate extension ops and backends

2026-08-16 21:15:52 +08:00