From a7d4cb25c52f10bf058a2013b9d73c4e5ffd39c9 Mon Sep 17 00:00:00 2001 From: ViperEkura <3081035982@qq.com> Date: Sat, 29 Aug 2026 14:01:40 +0800 Subject: [PATCH] docs: scope trainer environment variables per job - Add a Per-Job Environment section explaining that runtime.environment reaches only the GPUs declared in the same job YAML, with one-YAML-per-GPU-group examples for local, cross-PCIe workaround, and NVSwitch NVLink tuning setups - Replace the NCCL workaround pair in the runtime schema example with ASTR_LOG_LEVEL and ASTR_BACKEND and document value semantics (str() rendering, null exports empty, no host-shell passthrough) - Comment out the blanket NCCL exports in the get-started multi-GPU example so they are opt-in per docs/guides/distributed.md - Add a hard rule against copying NCCL workarounds into every training config --- docs/developer/docker-training.md | 89 ++++++++++++++++++++++++++++--- docs/get-started.md | 5 +- 2 files changed, 86 insertions(+), 8 deletions(-) diff --git a/docs/developer/docker-training.md b/docs/developer/docker-training.md index 57b8ee5..0049ae8 100644 --- a/docs/developer/docker-training.md +++ b/docs/developer/docker-training.md @@ -42,10 +42,11 @@ runtime: stop_timeout_seconds: 600 checkpoint_keep_last: 5 # max_duration_hours: 12 - # Add host-specific workarounds only when required: + # Optional; entries are passed verbatim into the trainer container + # (see "Per-Job Environment"): # environment: - # NCCL_P2P_DISABLE: "1" - # NCCL_NET_GDR_LEVEL: "0" + # ASTR_LOG_LEVEL: DEBUG + # ASTR_BACKEND: torch_native ``` - Relative paths resolve from the YAML file's directory, not the current shell. @@ -57,11 +58,85 @@ runtime: Use `fsdp` explicitly when model sharding is required. - To select specific physical GPUs, replace `all` with a list such as `devices: [0, 1]`. -- `environment` values are explicitly passed to the training container. Keep - host-specific NCCL workarounds here; they are not universal defaults. +- `environment` entries apply only to the job defined by this YAML file, not to + the host or to other jobs. Keep the section omitted unless this job's GPU + selection needs it; see [Per-Job Environment](#per-job-environment). - `max_duration_hours` starts a detached host timer that calls the same graceful `stop` command. A manual stop cancels the timer. +## Per-Job Environment + +`runtime.environment` is scoped to one job. `start` passes only the entries of +the config file it was given, so a variable reaches exactly the GPUs declared +in that file's `runtime.gpu.devices` and nothing else. Two jobs on the same +machine can therefore differ: a job whose GPUs have working peer-to-peer keeps +the section omitted, a job whose GPUs cross broken PCIe/NVLink paths declares +the NCCL workarounds, and a job on an NVSwitch fabric can pin the NVLink fast +path on. + +Because of that scoping, the effective pattern is one YAML per GPU group +rather than one shared YAML that gets edited whenever the device list changes: + +```yaml +# train-local.yaml: GPUs with working peer-to-peer; nothing to declare +runtime: + gpu: + devices: [0, 1] + +# train-cross-pcie.yaml: this GPU set crosses broken paths, so only this job +# declares the workarounds (confirm first; see docs/guides/distributed.md) +runtime: + gpu: + devices: [4, 5, 6, 7] + environment: + NCCL_P2P_DISABLE: "1" + NCCL_NET_GDR_LEVEL: "0" +``` + +The same mechanism carries positive tuning, not just workarounds. On an +NVSwitch node (Hopper-class GPUs with fabric manager running), NVLink SHARP +multicast (NVLS) is the fast allreduce path and NCCL enables it automatically +where supported. A job may pin it on explicitly and raise channel parallelism +when benchmarks show the NVLink bandwidth is underused: + +```yaml +# train-nvlink.yaml: NVSwitch node; keep the disables OUT and pin the fast +# path on instead (verify support with NCCL_DEBUG=INFO first) +runtime: + gpu: + devices: [0, 1, 2, 3] + environment: + NCCL_NVLS_ENABLE: "1" + NCCL_MIN_NCHANNELS: "8" + # NCCL_ALGO: NVLS # force one algorithm; unsupported values fail loudly +``` + +NVLS requires NVSwitch multicast support; on plain NVLink bridges or PCIe-only +sets, keep the section omitted and let NCCL pick Ring/Tree with P2P. Newer +drivers list the actual interconnect and NVLS support directly in +`nvidia-smi topo -m`, so check that before assuming. + +Confirm a variable is needed before adding it, and only in the YAML of the job +that hits the problem: + +```bash +nvidia-smi topo -m # check P2P support between exactly the selected GPUs +NCCL_DEBUG=INFO # confirm NCCL transport errors before disabling them +``` + +See `docs/guides/distributed.md` for what each troubleshooting variable +disables. The two directions are mutually exclusive: `NCCL_P2P_DISABLE` and +`NCCL_NET_GDR_LEVEL` remove bandwidth and must never appear in the same +environment as the NVLink entries above. + +Semantics: + +- Values must be scalars and are rendered with `str()`, so quote them + explicitly (`"1"`, `"0"`) instead of relying on YAML booleans or numbers. +- A `null` value exports the name with an empty value. +- This section is the only path for extra host variables into the trainer + container; variables exported in the host shell do not pass through Compose. + ## Fixed Container Paths | Runtime path | Container path | Access | @@ -123,5 +198,7 @@ the Docker timeout expires. 3. Do not force DDP for a model that requires FSDP; declare the mode explicitly. 4. Do not use `kill -9` for routine shutdown; use `scripts/train.sh stop CONFIG`. 5. The image user is built with the host UID/GID so mounted checkpoints retain usable ownership. +6. Scope `runtime.environment` to the job YAML that needs it; do not copy NCCL + workarounds into every config. -> Document Update Time: 2026-08-22 +> Document Update Time: 2026-08-29 diff --git a/docs/get-started.md b/docs/get-started.md index fc73f11..890f6bc 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -190,8 +190,9 @@ python scripts/tools/train.py \ ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 -export NCCL_P2P_DISABLE=1 -export NCCL_NET_GDR_LEVEL=0 +# Only if this host's NCCL transport is broken; see docs/guides/distributed.md: +# export NCCL_P2P_DISABLE=1 +# export NCCL_NET_GDR_LEVEL=0 python scripts/tools/train.py \ --train_type=seq \