feat: unify Docker serving configuration in YAML

- Add server.py --config serve.yaml; explicit CLI flags override YAML
- Add scripts/serve.sh and serve_runtime.py for the Compose lifecycle
- Template server/cpu ports and param mounts in docker-compose.yml
- Document schema in docs/developer/docker-serving.md and params guide
- Add tests for runtime parsing and server CLI merge logic
This commit is contained in:
2026-08-21 23:16:45 +08:00
parent dcc96de12a
commit cb21af38ba
10 changed files with 817 additions and 6 deletions
+25
View File
@@ -171,6 +171,31 @@ InferenceEngine
`GenerateResult` uses `Condition` for non-streaming (`wait_completion()`) and `Event` for streaming (`wait()`). Stream callback is `cb(token)`.
## Launching the Server
`scripts/tools/server.py` accepts every option as a CLI flag or from a YAML
config file (`--config serve.yaml`); explicit CLI flags override YAML values.
The YAML `server:` section mirrors the flags:
```yaml
server:
host: 0.0.0.0
port: 8000
device: cuda
dtype: bfloat16
max_batch_size: 16
max_seq_len: null
```
```bash
python scripts/tools/server.py --config serve.yaml
python scripts/tools/server.py --config serve.yaml --port 9000 # CLI wins
```
In Docker, `scripts/serve.sh` drives the same YAML (a `runtime:` section
controls ports/GPU/mounts); see
[Docker Serving](../developer/docker-serving.md).
## HTTP API
```