ViperEkura
8999ca89b8
- Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py - Reuse PipelineConfig schema for JSONL dataset configuration - Update detect_format to recognize JSONL directories and files - Move save_h5/load_h5/save_bin/load_bin to astrai/serialization - Split astrai/serialization.py into checkpoint/dataset submodules - Add tests for JSONL detection, seq/SFT stores, and config roundtrip |
||
|---|---|---|
| .. | ||
| conftest.py | ||
| test_checkpoint.py | ||
| test_dataset.py | ||
| test_preprocess_builder.py | ||
| test_preprocess_config.py | ||
| test_preprocess_pipeline.py | ||
| test_sampler.py | ||