AstrAI

Files

T

ViperEkura 8999ca89b8 feat: add JSONL dataset store with on-the-fly tokenization

- Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py
- Reuse PipelineConfig schema for JSONL dataset configuration
- Update detect_format to recognize JSONL directories and files
- Move save_h5/load_h5/save_bin/load_bin to astrai/serialization
- Split astrai/serialization.py into checkpoint/dataset submodules
- Add tests for JSONL detection, seq/SFT stores, and config roundtrip

2026-07-04 15:42:33 +08:00

conftest.py

refactor : 压缩测试代码，消除重复

2026-06-19 14:54:39 +08:00

test_checkpoint.py

refactor : replace iteration with consumed_samples

2026-06-30 18:42:42 +08:00

test_dataset.py

feat: add JSONL dataset store with on-the-fly tokenization

2026-07-04 15:42:33 +08:00

test_preprocess_builder.py

refactor : 压缩测试代码，消除重复