docs: 重构 README 结构,全文档添加目录导航

- README 新增 Getting Started 端到端流程,整合快速开始与演示,去重精简
- 中文 README 同步英文版结构,预处理配置改用 seq 策略
- inference.md 补充 SSE 流式格式、错误响应、/stats 端点文档
- params.md 扩展为 CLI 参考,覆盖 server/generate/preprocess 参数表
- dataflow.md 拆分 tokenization/format detection/backend 子节,新增流程图
- architecture/training/inference/preprocessing 均添加目录导航
- 移除 README CI badge
This commit is contained in:
2026-06-19 13:53:22 +08:00
parent d096b6e29e
commit 8ab7564d02
8 changed files with 392 additions and 154 deletions
+98 -1
View File
@@ -1,5 +1,16 @@
# Inference
## Contents
- [KV Cache](#kv-cache)
- [KVCache System](#kvcache-system)
- [Continuous Batching](#continuous-batching)
- [Sampling](#sampling-strategy-pattern)
- [Protocol Handlers](#protocol-handlers-strategy-pattern)
- [Engine & GenerateResult](#engine--generateresult)
- [HTTP API](#http-api) — endpoints, SSE, errors, stats
- [Engine API](#engine-api)
## KV Cache
At decode time, only the last query token matters. All previous K/V are cached to avoid recomputation:
@@ -133,6 +144,92 @@ Supports `stop_sequences` and streaming via `event: content_block_delta`.
| `max_tokens` | Optional[int] | None | Max generation length |
| `stream` | bool | False | Stream output |
### SSE Streaming Format
**OpenAI** (`/v1/chat/completions`, `stream=true`):
```
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"astrai",
"choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk",...,
"choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk",...,
"choices":[{"index":0,"delta":{},"finish_reason":"stop"}],
"usage":{"prompt_tokens":5,"completion_tokens":1,"total_tokens":6}}
data: [DONE]
```
**Anthropic** (`/v1/messages`, `stream=true`):
```
event: message_start
data: {"type":"message_start","message":{"id":"msg_...","model":"astrai","role":"assistant",
"content":[],"stop_reason":null,...}}
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
event: content_block_stop
data: {"type":"content_block_stop","index":0}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{...}}
event: message_stop
data: {"type":"message_stop"}
```
### Error Responses
All endpoints use standard HTTP status codes:
| Status | Meaning |
|--------|---------|
| 200 | Success |
| 400 | Invalid request (bad JSON, missing fields, validation error) |
| 405 | Method not allowed |
| 422 | Unprocessable entity (Pydantic validation) |
| 500 | Internal server error (model crash, OOM, scheduler failure) |
| 503 | Service unavailable (model not loaded, engine not ready) |
Error response body:
```json
{
"error": {
"message": "Invalid request: max_tokens must be > 0",
"type": "invalid_request_error",
"code": 400
}
}
```
### Stats Endpoint
```
GET /stats
```
Response:
```json
{
"active_requests": 3,
"waiting_requests": 2,
"total_requests": 128,
"cache_usage": 0.45,
"tokens_generated": 10240
}
```
`cache_usage` is the fraction of KV cache pages currently in use (0.01.0).
## Engine API
```python
@@ -149,4 +246,4 @@ async for token in engine.generate_async("Hello", ...): # -> AsyncGenerator[s
print(token)
```
> Document Update Time: 2026-05-30
> Document Update Time: 2026-06-19