5 Commits

Include renames

Author	SHA1	Message	Date
ViperEkura	3406157431	refactor: standardize packed 3d inference ... - keep training attention on dense 4d tensors - use packed 3d tensors with KV cache for inference - extend CUDA rotary embedding to packed 3d inputs - adapt torch, CUDA and FlashAttention backend dispatch	2026-08-16 13:24:02 +08:00
ViperEkura	3439e3104e	perf: launch CUDA kernels on torch's current stream ... - Thread a cudaStream_t through attn dispatchers onto torch's current stream - Scope the device guard to the entry function so kernels run on tensor device - DISPATCH_HEAD_DIM now forwards varargs so stream reaches each dispatch - Parallelize CPU reference kernels with OpenMP (paged test 31s -> 7s) - Merge decode/prefill standalone tests into attn_test.cu with correctness tables - Drop bench error column (CPU ref too slow at large sizes) - Update cuda_kernels.md for the merged test layout	2026-08-02 13:20:14 +08:00
ViperEkura	7feeb0b93e	refactor: replace magic layout ints with TensorLayout enum ... - Add TensorLayout enum (C++ + Python) to replace magic layout ints - Add C10_CUDA_CHECK post-launch error checking to all kernel entries - Add CUDAGuard + freqs_cis shape validation to rotary_emb.cu - Cache SM count to eliminate per-call cudaDeviceGetAttribute - Add DISPATCH_CAUSAL_MASK macro to deduplicate dispatcher if/else - Convert mask type hints from X|None to Optional[X]	2026-08-01 11:05:52 +08:00
ViperEkura	7aa5ed09d9	refactor: unify rotary embedding interface and update docs ... - Merge cos/sin into single freqs_cis tensor [batch, seq, dim/2, 2] throughout the pipeline: RotaryEmbedding buffer, forward return type, apply_rotary_emb signature, CUDA kernel interface - CUDA kernel now takes freqs_cis directly and reads cos/sin via stride offset internally, eliminating Python-side slice/copy overhead - Kernel interface: rotary_emb(x, freqs_cis) replaces rotary_emb(x, cos, sin) - All call sites pass rotary_emb as Tensor (was tuple), type annotations consistent - Update build threads from 8 to 16 - Fix all docs: get-started, inference, training, cuda_kernels, architecture, internals — reflect new rotary interface, KVCache fields, rotary backend dispatch, .so path, kernel registry count, file layout	2026-07-31 16:52:25 +08:00
ViperEkura	3e67b4f88d	perf: add fused CUDA rotary embedding kernel ... - Single-kernel rotary embedding (cos/sin lookup + rotation) replaces PyTorch complex-multiply path (3 kernel launches + f32 upcast per call) - RotaryEmbedding now stores cos_table/sin_table and returns (cos, sin) f32 tuple instead of a complex tensor - apply_rotary_emb in rotary_backend.py auto-dispatches: CUDA kernel if available, else torch complex-multiply fallback; backend-agnostic (both attention backends benefit) - Kernel: 256-thread blocks, grid-stride loop, vectorized __nv_bfloat162 load/store, f32 compute, bf16 out - Standalone kernel 6-9x faster than torch across decode/prefill shapes, max diff 0 (decode) to 3e-2 (large prefill, bf16) - Benchmark (L20, bf16, CUDA backend): B=1 9.48->7.25ms (+31%), B=4 10.73->7.67ms (+40%), B=8 10.77->7.81ms (+38%), B=16 10.79->7.83ms (+38%)	2026-07-31 15:27:31 +08:00