feat: unify Docker serving configuration in YAML
- Add server.py --config serve.yaml; explicit CLI flags override YAML - Add scripts/serve.sh and serve_runtime.py for the Compose lifecycle - Template server/cpu ports and param mounts in docker-compose.yml - Document schema in docs/developer/docker-serving.md and params guide - Add tests for runtime parsing and server CLI merge logic
This commit is contained in:
@@ -0,0 +1,108 @@
|
||||
# Containerized Serving Deployment
|
||||
|
||||
AstrAI uses one serving YAML as the declaration for both host-side container
|
||||
runtime settings and in-container server settings. `scripts/serve.sh` wraps the
|
||||
Compose commands so preflight validation and container lifecycle stay
|
||||
consistent with the trainer.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
serve.yaml
|
||||
├── runtime parsed on the host before Docker starts
|
||||
└── server parsed by server.py inside the container
|
||||
│
|
||||
scripts/serve.sh preflight, Compose wrapper, lifecycle
|
||||
└── docker-compose.yml GPU passthrough, mounts, image, port mapping
|
||||
└── server.py --config /run/astrai/serve.yaml
|
||||
```
|
||||
|
||||
`scripts/tools/serve_runtime.py` reads `runtime:` plus the two container-side
|
||||
values Compose needs (`server.port` for the port mapping, `server.device` for
|
||||
the preflight GPU check). `scripts/tools/server.py --config` reads `server:`.
|
||||
Explicit CLI arguments to `server.py` override `server:` YAML values.
|
||||
|
||||
## Runtime Schema
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
job_name: serve
|
||||
port: 8000
|
||||
paths:
|
||||
param: ./params
|
||||
gpu:
|
||||
enabled: true # false → cpu profile (server-cpu service)
|
||||
devices: all # all | [0]
|
||||
container:
|
||||
cuda_tag: cu128
|
||||
# environment:
|
||||
# TOKENIZERS_PARALLELISM: "false"
|
||||
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 8000
|
||||
device: cuda # cuda | cpu
|
||||
dtype: bfloat16 # bfloat16 | float16 | float32
|
||||
max_batch_size: 16
|
||||
max_seq_len: null # falls back to model config
|
||||
```
|
||||
|
||||
- Relative paths resolve from the YAML file's directory, not the current shell.
|
||||
- `runtime.port` is the host publish port; `server.port` is the port the
|
||||
container listens on. The Compose mapping is
|
||||
`${SERVE_PORT}:${SERVE_CONTAINER_PORT}`.
|
||||
- `runtime.gpu.enabled: true` (default) selects the `server` service with an
|
||||
NVIDIA device reservation; `false` selects `server-cpu` (no GPU passthrough).
|
||||
When disabled, `server.device` must be `cpu`.
|
||||
- `runtime.gpu.devices` is either `all` or a single-device list such as `[0]`;
|
||||
the list becomes `CUDA_VISIBLE_DEVICES`. Compose passes `count: 1`.
|
||||
- `environment` values are explicitly passed to the serving container. Keep
|
||||
host-specific settings here; they are not universal defaults.
|
||||
- `server.device` must agree with `runtime.gpu.enabled`; `preflight` enforces it.
|
||||
|
||||
## Fixed Container Paths
|
||||
|
||||
| Runtime path | Container path | Access |
|
||||
|---|---|---|
|
||||
| `runtime.paths.param` | `/app/params` | read-only |
|
||||
| the selected YAML | `/run/astrai/serve.yaml` | read-only |
|
||||
|
||||
`server.param_path` is optional: the server default is
|
||||
`project_root/params`, which is exactly `/app/params` inside the container
|
||||
(the working directory is `/app`). Set it explicitly only when serving from a
|
||||
different location; in Docker it must be a container path.
|
||||
|
||||
## Operations
|
||||
|
||||
The config argument defaults to `./serve.yaml`:
|
||||
|
||||
```bash
|
||||
bash scripts/serve.sh init [CONFIG]
|
||||
bash scripts/serve.sh preflight [CONFIG]
|
||||
bash scripts/serve.sh up [CONFIG]
|
||||
bash scripts/serve.sh run [CONFIG]
|
||||
bash scripts/serve.sh down [CONFIG]
|
||||
bash scripts/serve.sh restart [CONFIG]
|
||||
bash scripts/serve.sh logs [CONFIG]
|
||||
bash scripts/serve.sh status [CONFIG]
|
||||
```
|
||||
|
||||
`preflight` validates Docker, the model directory
|
||||
(`config.json` + `model.safetensors`), GPU/device consistency, and the
|
||||
rendered Compose configuration. `up` starts the container detached and
|
||||
rebuilds the image when the code changed (`--build`); `run` keeps it in the
|
||||
foreground. The wrapper manages a fixed container name
|
||||
(`astrai-server` or `astrai-server-<job_name>`); the plain
|
||||
`docker compose up -d` / `docker compose --profile cpu up -d` path keeps
|
||||
working with defaults (port 8000, `./params`).
|
||||
|
||||
## Hard Rules
|
||||
|
||||
1. Keep Docker settings in `runtime` and server settings in `server`.
|
||||
2. Filter GPUs once: the `server` service reserves one device; a `devices`
|
||||
list becomes `CUDA_VISIBLE_DEVICES`.
|
||||
3. `runtime.gpu.enabled: false` requires `server.device: cpu`.
|
||||
4. In Docker, `server.port` must match the published container port (default
|
||||
`8000`); change `runtime.port` to publish on a different host port.
|
||||
5. The image user is built with the host UID/GID so the mounted model
|
||||
directory stays readable.
|
||||
@@ -171,6 +171,31 @@ InferenceEngine
|
||||
|
||||
`GenerateResult` uses `Condition` for non-streaming (`wait_completion()`) and `Event` for streaming (`wait()`). Stream callback is `cb(token)`.
|
||||
|
||||
## Launching the Server
|
||||
|
||||
`scripts/tools/server.py` accepts every option as a CLI flag or from a YAML
|
||||
config file (`--config serve.yaml`); explicit CLI flags override YAML values.
|
||||
The YAML `server:` section mirrors the flags:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 8000
|
||||
device: cuda
|
||||
dtype: bfloat16
|
||||
max_batch_size: 16
|
||||
max_seq_len: null
|
||||
```
|
||||
|
||||
```bash
|
||||
python scripts/tools/server.py --config serve.yaml
|
||||
python scripts/tools/server.py --config serve.yaml --port 9000 # CLI wins
|
||||
```
|
||||
|
||||
In Docker, `scripts/serve.sh` drives the same YAML (a `runtime:` section
|
||||
controls ports/GPU/mounts); see
|
||||
[Docker Serving](../developer/docker-serving.md).
|
||||
|
||||
## HTTP API
|
||||
|
||||
```
|
||||
|
||||
@@ -203,6 +203,7 @@ nohup python scripts/tools/train.py \
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `--config`, `-c` | path | `None` | Serving YAML config. CLI flags override YAML values |
|
||||
| `--host` | str | `0.0.0.0` | Host address |
|
||||
| `--port` | int | `8000` | Port number |
|
||||
| `--param_path` | path | `project_root/params` | Path to model parameters |
|
||||
@@ -217,6 +218,22 @@ Usage:
|
||||
python scripts/tools/server.py --param_path ./params --device cuda --dtype bfloat16
|
||||
```
|
||||
|
||||
YAML config (a `server:` section; explicit CLI flags override YAML values):
|
||||
```bash
|
||||
python scripts/tools/server.py --config serve.yaml
|
||||
```
|
||||
```yaml
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 8000
|
||||
device: cuda
|
||||
dtype: bfloat16
|
||||
max_batch_size: 16
|
||||
max_seq_len: null
|
||||
```
|
||||
`serve.yaml` also carries a `runtime:` section for the Docker wrapper; see
|
||||
[Docker Serving](../developer/docker-serving.md).
|
||||
|
||||
See [Inference Guide](inference.md) for HTTP API documentation.
|
||||
|
||||
## Generate (`generate.py`)
|
||||
|
||||
Reference in New Issue
Block a user