refactor: unify rotary embedding interface and update docs

- Merge cos/sin into single freqs_cis tensor [batch, seq, dim/2, 2] throughout the pipeline: RotaryEmbedding buffer, forward return type, apply_rotary_emb signature, CUDA kernel interface
- CUDA kernel now takes freqs_cis directly and reads cos/sin via stride offset internally, eliminating Python-side slice/copy overhead
- Kernel interface: rotary_emb(x, freqs_cis) replaces rotary_emb(x, cos, sin)
- All call sites pass rotary_emb as Tensor (was tuple), type annotations consistent
- Update build threads from 8 to 16
- Fix all docs: get-started, inference, training, cuda_kernels, architecture, internals — reflect new rotary interface, KVCache fields, rotary backend dispatch, .so path, kernel registry count, file layout
This commit is contained in:
2026-07-31 16:52:25 +08:00
parent 75411ce0cc
commit 7aa5ed09d9
11 changed files with 120 additions and 83 deletions
+3 -3
View File
@@ -17,14 +17,14 @@ cd AstrAI
# Basic install (pure PyTorch, no custom CUDA kernels)
pip install -e .
# With CUDA kernels (optional, for fused attention)
# With CUDA kernels (optional, for fused attention and rotary embedding)
# CSRC_KERNELS=true pip install -e . --no-build-isolation
# With dev dependencies (pytest, ruff)
# pip install -e ".[dev]"
```
> **CUDA kernels** are opt-in. They are not built by default. When built, they can be activated via `with attn_backend(ATTN_BACKEND.CUDA):` for accelerated decode/prefill. You can skip them for normal usage.
> **CUDA kernels** are opt-in. They are not built by default. When built, they can be activated via `with attn_backend(ATTN_BACKEND.CUDA):` for accelerated decode/prefill, and the fused rotary embedding kernel is auto-dispatched when available. You can skip them for normal usage.
## 2. Download Model Weights
@@ -232,4 +232,4 @@ docker compose up -d
| System architecture | [Architecture](developer/architecture.md) |
| Data pipeline internals | [Data Flow](developer/dataflow.md) |
> Document Update Time: 2026-07-30
> Document Update Time: 2026-07-31