ViperEkura e220413035 feat: support raw JSON files in dataset pipeline and JsonlStore ... - detect_format now recognizes .json directories as jsonl store - JsonlStore loads .json arrays and dicts alongside .jsonl - tokenizer_path defaults to dataset dir when omitted - Pipeline._iter_items handles .json files (arrays/single dict) - Tests: detect_format, seq load, self-contained dataset dir		2026-07-17 12:20:03 +08:00
..
__init__.py	feat: add JSONL dataset store with on-the-fly tokenization	2026-07-04 15:42:33 +08:00
dataset.py	feat: add JSONL dataset store with on-the-fly tokenization	2026-07-04 15:42:33 +08:00
sampler.py	fix: reset sampler iter at epoch end so progress bar shows total after first epoch	2026-07-04 06:35:55 +08:00
storage.py	feat: support raw JSON files in dataset pipeline and JsonlStore	2026-07-17 12:20:03 +08:00