ViperEkura d7cd69fef5 feat: add streaming IterableDataset for pretraining ...

- StreamingSeqDataset yields windows sequentially through each shard
- Shard-level shuffle, distributed and multi-worker shard partitioning
- __len__ returns total window count for scheduler total_steps
- Better OS page-cache locality than random-access map-style datasets

2026-08-08 16:15:10 +08:00

..

__init__.py

feat: add streaming IterableDataset for pretraining

2026-08-08 16:15:10 +08:00

dataset.py

refactor: break JsonlStore→preprocessing circular dependency

2026-08-07 23:22:32 +08:00

sampler.py

feat: add record-mode to Store for DPO/GRPO

2026-07-18 21:02:29 +08:00

storage.py

refactor: break JsonlStore→preprocessing circular dependency

2026-08-07 23:22:32 +08:00

streaming.py

feat: add streaming IterableDataset for pretraining

2026-08-08 16:15:10 +08:00