fix: align docs with actual code (40+ inconsistencies)
- Remove nonexistent Muon class from architecture diagram - Fix Checkpoint/TrainConfig/TrainContext field names (iteration -> consumed_samples, start_batch -> start_samples) - Add missing fields: neftune_alpha, val_split, grad_norm, optimizer_step, tool_calls/tools - Fix CLI param defaults: --log_interval 1, --metrics [loss,lr,grad_norm], --start_samples - Add missing scheduler CLI params; remove nonexistent --num_workers from preprocess docs - Fix inference SSE format, stats response keys, error codes to match actual server output - Fix preprocessing docs: BOS once, shard_0000 layout, from_json->from_file, GRPO prompts_mask - Fix dataflow detect_format/_normalize descriptions; correct callback order in training.md
This commit is contained in:
@@ -46,10 +46,10 @@ The output `meta.json` records the storage format, key names, dtype, total token
|
||||
|
||||
### Format Detection
|
||||
|
||||
`detect_format(load_path)` inspects the directory:
|
||||
`detect_format(load_path)` inspects the path:
|
||||
|
||||
- If `*.h5` files exist → `"h5"` (HDF5 backend)
|
||||
- If `*.bin` + `meta.json` files exist → `"bin"` (memory-mapped backend)
|
||||
- If `load_path` is a file: checks suffix — `.h5`/`.hdf5` → `"h5"`, unknown suffix raises `ValueError`
|
||||
- If `load_path` is a directory: recursively globs for `*.h5`/`*.hdf5` files → `"h5"`, or `*.bin` + `**/meta.json` → `"bin"`
|
||||
|
||||
### Store Backends
|
||||
|
||||
@@ -83,7 +83,7 @@ DatasetFactory.load(train_type, load_path, window_size, stride=None, storage_typ
|
||||
→ detect_format(load_path)
|
||||
→ StoreFactory.create(storage_type)
|
||||
→ Store.load(load_path)
|
||||
→ H5Store._normalize() / MmapStore._normalize()
|
||||
→ _normalize(raw) # base Store, shared by both backends
|
||||
→ Store._data[Dict[str, List[Tensor]]] + _cum[Dict[str, List[int]]]
|
||||
→ BaseDataset.__getitem__(idx)
|
||||
→ get_index(idx) → [begin, end)
|
||||
|
||||
Reference in New Issue
Block a user