fix: align docs with actual code (40+ inconsistencies)

- Remove nonexistent Muon class from architecture diagram
- Fix Checkpoint/TrainConfig/TrainContext field names (iteration -> consumed_samples, start_batch -> start_samples)
- Add missing fields: neftune_alpha, val_split, grad_norm, optimizer_step, tool_calls/tools
- Fix CLI param defaults: --log_interval 1, --metrics [loss,lr,grad_norm], --start_samples
- Add missing scheduler CLI params; remove nonexistent --num_workers from preprocess docs
- Fix inference SSE format, stats response keys, error codes to match actual server output
- Fix preprocessing docs: BOS once, shard_0000 layout, from_json->from_file, GRPO prompts_mask
- Fix dataflow detect_format/_normalize descriptions; correct callback order in training.md
This commit is contained in:
2026-06-30 20:47:23 +08:00
parent aabb0d83e9
commit dc9faca3b1
6 changed files with 86 additions and 74 deletions
+4 -4
View File
@@ -46,10 +46,10 @@ The output `meta.json` records the storage format, key names, dtype, total token
### Format Detection
`detect_format(load_path)` inspects the directory:
`detect_format(load_path)` inspects the path:
- If `*.h5` files exist → `"h5"` (HDF5 backend)
- If `*.bin` + `meta.json` files exist → `"bin"` (memory-mapped backend)
- If `load_path` is a file: checks suffix — `.h5`/`.hdf5``"h5"`, unknown suffix raises `ValueError`
- If `load_path` is a directory: recursively globs for `*.h5`/`*.hdf5` files → `"h5"`, or `*.bin` + `**/meta.json` `"bin"`
### Store Backends
@@ -83,7 +83,7 @@ DatasetFactory.load(train_type, load_path, window_size, stride=None, storage_typ
→ detect_format(load_path)
→ StoreFactory.create(storage_type)
→ Store.load(load_path)
H5Store._normalize() / MmapStore._normalize()
→ _normalize(raw) # base Store, shared by both backends
→ Store._data[Dict[str, List[Tensor]]] + _cum[Dict[str, List[int]]]
→ BaseDataset.__getitem__(idx)
→ get_index(idx) → [begin, end)