feat: unify Docker serving configuration in YAML
- Add server.py --config serve.yaml; explicit CLI flags override YAML - Add scripts/serve.sh and serve_runtime.py for the Compose lifecycle - Template server/cpu ports and param mounts in docker-compose.yml - Document schema in docs/developer/docker-serving.md and params guide - Add tests for runtime parsing and server CLI merge logic
This commit is contained in:
@@ -171,6 +171,31 @@ InferenceEngine
|
||||
|
||||
`GenerateResult` uses `Condition` for non-streaming (`wait_completion()`) and `Event` for streaming (`wait()`). Stream callback is `cb(token)`.
|
||||
|
||||
## Launching the Server
|
||||
|
||||
`scripts/tools/server.py` accepts every option as a CLI flag or from a YAML
|
||||
config file (`--config serve.yaml`); explicit CLI flags override YAML values.
|
||||
The YAML `server:` section mirrors the flags:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 8000
|
||||
device: cuda
|
||||
dtype: bfloat16
|
||||
max_batch_size: 16
|
||||
max_seq_len: null
|
||||
```
|
||||
|
||||
```bash
|
||||
python scripts/tools/server.py --config serve.yaml
|
||||
python scripts/tools/server.py --config serve.yaml --port 9000 # CLI wins
|
||||
```
|
||||
|
||||
In Docker, `scripts/serve.sh` drives the same YAML (a `runtime:` section
|
||||
controls ports/GPU/mounts); see
|
||||
[Docker Serving](../developer/docker-serving.md).
|
||||
|
||||
## HTTP API
|
||||
|
||||
```
|
||||
|
||||
@@ -203,6 +203,7 @@ nohup python scripts/tools/train.py \
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `--config`, `-c` | path | `None` | Serving YAML config. CLI flags override YAML values |
|
||||
| `--host` | str | `0.0.0.0` | Host address |
|
||||
| `--port` | int | `8000` | Port number |
|
||||
| `--param_path` | path | `project_root/params` | Path to model parameters |
|
||||
@@ -217,6 +218,22 @@ Usage:
|
||||
python scripts/tools/server.py --param_path ./params --device cuda --dtype bfloat16
|
||||
```
|
||||
|
||||
YAML config (a `server:` section; explicit CLI flags override YAML values):
|
||||
```bash
|
||||
python scripts/tools/server.py --config serve.yaml
|
||||
```
|
||||
```yaml
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 8000
|
||||
device: cuda
|
||||
dtype: bfloat16
|
||||
max_batch_size: 16
|
||||
max_seq_len: null
|
||||
```
|
||||
`serve.yaml` also carries a `runtime:` section for the Docker wrapper; see
|
||||
[Docker Serving](../developer/docker-serving.md).
|
||||
|
||||
See [Inference Guide](inference.md) for HTTP API documentation.
|
||||
|
||||
## Generate (`generate.py`)
|
||||
|
||||
Reference in New Issue
Block a user