AstrAI

Commit Graph

Author	SHA1	Message	Date
ViperEkura	b092316385	feat : add distributed checkpoint via executor checkpoint_context - Add checkpoint_context context manager to BaseExecutor with entry/exit barrier - Add _gather_state_dict hook overridden per executor (template method) - DDPExecutor skips unwrap on non-rank-0 to avoid redundant state_dict gather - FSDPExecutor uses rank0_only=True to reduce memory on non-writers - Remove redundant rank-0 guard from Checkpoint.save and manual barrier from Callback	2026-07-13 12:27:09 +08:00
ViperEkura	8999ca89b8	feat: add JSONL dataset store with on-the-fly tokenization - Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py - Reuse PipelineConfig schema for JSONL dataset configuration - Update detect_format to recognize JSONL directories and files - Move save_h5/load_h5/save_bin/load_bin to astrai/serialization - Split astrai/serialization.py into checkpoint/dataset submodules - Add tests for JSONL detection, seq/SFT stores, and config roundtrip	2026-07-04 15:42:33 +08:00

Author

SHA1

Message

Date

ViperEkura

b092316385

feat : add distributed checkpoint via executor checkpoint_context

- Add checkpoint_context context manager to BaseExecutor with entry/exit barrier
- Add _gather_state_dict hook overridden per executor (template method)
- DDPExecutor skips unwrap on non-rank-0 to avoid redundant state_dict gather
- FSDPExecutor uses rank0_only=True to reduce memory on non-writers
- Remove redundant rank-0 guard from Checkpoint.save and manual barrier from Callback

2026-07-13 12:27:09 +08:00

ViperEkura

8999ca89b8

feat: add JSONL dataset store with on-the-fly tokenization

- Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py
- Reuse PipelineConfig schema for JSONL dataset configuration
- Update detect_format to recognize JSONL directories and files
- Move save_h5/load_h5/save_bin/load_bin to astrai/serialization
- Split astrai/serialization.py into checkpoint/dataset submodules
- Add tests for JSONL detection, seq/SFT stores, and config roundtrip

2026-07-04 15:42:33 +08:00

2 Commits