docs: 更新设计文档

This commit is contained in:
2026-04-05 00:17:35 +08:00
parent 2b26f03bd3
commit ff43a2fab8
3 changed files with 89 additions and 73 deletions
+12 -12
View File
@@ -83,15 +83,15 @@
### Usage Example
```python
from astrai.config.param_config import ModelParameter
from astrai.inference.generator import StreamGenerator, GenerationRequest
from astrai.config import ModelParameter
from astrai.inference import InferenceEngine, GenerationRequest
# Load model
param = ModelParameter.load("your_model_dir")
param.to(device="cuda", dtype=torch.bfloat16)
# Create generator
generator = StreamGenerator(param)
# Create engine
engine = InferenceEngine(param)
# Build request
request = GenerationRequest(
@@ -102,14 +102,14 @@ request = GenerationRequest(
top_k=50,
)
# Generate
response = generator.generate(request)
# Generate (streaming)
for token in engine.generate_with_request(request):
print(token, end="", flush=True)
```
### Three Types of Generators
### Generation Modes
| Generator | Usage |
|-----------|-------|
| `StreamGenerator` | Streaming output, returns word by word |
| `LoopGenerator` | Non-streaming output, returns at once |
| `BatchGenerator` | Batch generation, processes multiple queries simultaneously |
| Mode | Description |
|------|-------------|
| `stream=True` | Streaming output, yields token by token |
| `stream=False` | Non-streaming output, returns complete result |