docs: fix stale docs and align with code

- update cuda_kernels layout, arch flags, and add FP8 section
- fix install docs: kernels auto-build when nvcc + CUDA detected
- mark ignored OpenAI request params and complete KVCache fields
- add docker docs to indexes and astrai.optim to module overview
- refresh document update timestamps
This commit is contained in:
2026-08-23 14:23:28 +08:00
parent 75304d084d
commit 10fec8dca1
10 changed files with 134 additions and 58 deletions
+16 -7
View File
@@ -54,7 +54,12 @@ KVCache
├── out_cache_loc [batch, seq_len] — write indices for this forward
├── max_len int — max(seq_lens), avoids GPU sync in decode
├── kv_indptr [batch + 1] int32 — prefix sum of seq_lens, precomputed once per step
── qo_indptr [batch + 1] int32 — prefix sum of per-request q_lens (prefill), precomputed once per step
── qo_indptr [batch + 1] int32 — prefix sum of per-request q_lens (prefill), precomputed once per step
├── q_tile_to_batch [num_q_tiles] int32 — prefill: Q tile → request (precomputed once per step)
├── q_tile_to_index [num_q_tiles] int32 — prefill: Q tile → request-local tile index
├── decode_o_part [batch, n_heads, head_dim] — decode split-K partial output buffer
├── decode_ml_part [batch, n_heads] — decode split-K partial max/logsum buffer
└── decode_out [batch, n_heads, head_dim] — decode output accumulator
```
Attention layers do raw buffer indexing: `k_buffer[layer_id, out_cache_loc] = k` to write, `k_buffer[layer_id, indices]` to gather.
@@ -94,7 +99,7 @@ with attn_backend(ATTN_BACKEND.CUDA):
Environment and context selections are strict: if the selected backend cannot
handle the call, inference raises an error rather than silently switching.
`CudaBackend` decode path: writes K/V to cache, then calls `attn_paged_decode` with `page_size=1` — the `req_to_token` table serves directly as the page table, each token slot is a single-token "page". No explicit K/V gather needed.
`CudaBackend` decode path: writes K/V via `new_k`/`new_v` while calling `attn_paged_decode` — the `req_to_token` table serves directly as the page table (conceptually a single-token "page" per slot, i.e. `page_size=1`; the op itself takes no `page_size` argument). No explicit K/V gather needed.
`CudaBackend` prefill path: writes K/V, then calls `attn_paged_prefill` — a ragged-batch (paged) prefill kernel that reads K/V directly from the flat pool via `req_to_token`, addressing each request's `q_len`/`kv_len` through `qo_indptr` and `kv_indptr`. No explicit K/V gather needed.
@@ -253,14 +258,18 @@ The HTTP protocols and direct engine API have distinct request models and defaul
| `max_tokens` | Optional[int] | 2048 | Max generation length |
| `stream` | Optional[bool] | False | Stream output |
| `stop` | Optional[Union[str, List[str]]] | None | Stop sequences |
| `n` | Optional[int] | 1 | Number of choices requested |
| `presence_penalty` | Optional[float] | 0.0 | Presence penalty (-2.0 to 2.0) |
| `n` | Optional[int] | 1 | Accepted for API compatibility, **ignored** (always returns a single choice) |
| `presence_penalty` | Optional[float] | 0.0 | Accepted for API compatibility, **ignored** |
| `frequency_penalty` | Optional[float] | 0.0 | Frequency penalty (-2.0 to 2.0) |
| `logit_bias` | Optional[Dict[int, float]] | None | Per-token logit bias |
| `user` | Optional[str] | None | End-user identifier |
| `logit_bias` | Optional[Dict[int, float]] | None | Accepted for API compatibility, **ignored** |
| `user` | Optional[str] | None | Accepted for API compatibility, **ignored** |
| `tools` | Optional[List[ToolDef]] | None | Tool definitions for function calling |
| `tool_choice` | Optional[Union[str, Dict[str, Any]]] | `"auto"` | Tool selection mode or explicit tool choice |
> `n`, `presence_penalty`, `logit_bias`, and `user` are validated by the request
> model but ignored by the server (a warning is logged when a non-default value
> is supplied).
**Anthropic** (`MessagesRequest`):
| Param | Type | Default | Description |
@@ -371,4 +380,4 @@ async for token in engine.generate_async("Hello", ...): # -> AsyncGenerator[s
print(token)
```
> Document Update Time: 2026-08-16
> Document Update Time: 2026-08-22
+2 -2
View File
@@ -28,7 +28,7 @@
|-----------|-------------|---------|
| `--warmup_ratio` | Fraction of total steps used for LR warmup | 0.05 |
| `--max_lr` | Maximum learning rate (cosine decay after warmup) | 3e-4 |
| `--max_grad_norm` | Maximum gradient norm for clipping; the current CLI requires a positive number | 1.0 |
| `--max_grad_norm` | Maximum gradient norm for clipping; `TrainConfig` validates it as positive (or `None`) | 1.0 |
### Optimizer
@@ -281,4 +281,4 @@ See [Preprocessing Guide](preprocessing.md) for config file format and examples.
---
> Document Update Time: 2026-07-20
> Document Update Time: 2026-08-22