refactor: split Store into StreamStore and RecordStore

- StreamStore: fetch(begin, end, key) for stream access (SEQ/SFT)
- RecordStore: mixin with fetch_record(i, key) for record access
- H5Store/MmapStore/JsonlStore now dual-inherit both (C3 MRO)
- JsonlStore supports lazy mode via processor= (no TokenizeTransform)
- RecordDataset base class holds processor, DPO/GRPO simplified
- dpo_tokenize pure function for on-the-fly JSONL tokenisation
- DatasetFactory builds processor for jsonl+record datasets
- train.py passes tokenizer_path=param_path uniformly
- progress: len(dataset) returns sample count (stream=windows, record=records)
- json no longer auto-detected as jsonl format
This commit is contained in:
2026-07-18 23:04:31 +08:00
parent b33250dc28
commit b133fc9c07
5 changed files with 461 additions and 277 deletions
+4
View File
@@ -9,8 +9,10 @@ from astrai.dataset.storage import (
H5Store,
JsonlStore,
MmapStore,
RecordStore,
Store,
StoreFactory,
StreamStore,
detect_format,
)
from astrai.serialization import (
@@ -26,6 +28,8 @@ __all__ = [
"dpo_collate_fn",
"grpo_collate_fn",
"Store",
"StreamStore",
"RecordStore",
"StoreFactory",
"H5Store",
"MmapStore",