refactor: split Store into StreamStore and RecordStore
- StreamStore: fetch(begin, end, key) for stream access (SEQ/SFT) - RecordStore: mixin with fetch_record(i, key) for record access - H5Store/MmapStore/JsonlStore now dual-inherit both (C3 MRO) - JsonlStore supports lazy mode via processor= (no TokenizeTransform) - RecordDataset base class holds processor, DPO/GRPO simplified - dpo_tokenize pure function for on-the-fly JSONL tokenisation - DatasetFactory builds processor for jsonl+record datasets - train.py passes tokenizer_path=param_path uniformly - progress: len(dataset) returns sample count (stream=windows, record=records) - json no longer auto-detected as jsonl format
This commit is contained in:
@@ -460,6 +460,7 @@ def train(
|
||||
load_path=data_root_path,
|
||||
window_size=window_size,
|
||||
stride=stride,
|
||||
tokenizer_path=param_path,
|
||||
)
|
||||
|
||||
optimizer_fn = partial(
|
||||
|
||||
Reference in New Issue
Block a user