feat: add streaming IterableDataset for pretraining
- StreamingSeqDataset yields windows sequentially through each shard - Shard-level shuffle, distributed and multi-worker shard partitioning - __len__ returns total window count for scheduler total_steps - Better OS page-cache locality than random-access map-style datasets
This commit is contained in:
@@ -14,6 +14,7 @@ from astrai.dataset.storage import (
|
||||
Streamable,
|
||||
detect_format,
|
||||
)
|
||||
from astrai.dataset.streaming import StreamingSeqDataset
|
||||
from astrai.serialization import (
|
||||
load_bin,
|
||||
save_bin,
|
||||
@@ -34,4 +35,5 @@ __all__ = [
|
||||
"save_bin",
|
||||
"load_bin",
|
||||
"RDSampler",
|
||||
"StreamingSeqDataset",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user