3 Commits

Include renames

Author	SHA1	Message	Date
ViperEkura	115192c67c	refactor: remove H5 storage backend in favor of mmap bin ... - Remove H5Store, H5Writer, save_h5/load_h5 and h5py dependency - MmapStore (bin) is the sole pre-tokenized storage backend - Move setup_logging after imports to fix E402 in __init__.py - Clean up unused imports across test files - Move inline test imports to file top	2026-07-29 12:50:27 +08:00
ViperEkura	a74e5b91a3	feat: add record-mode to Store for DPO/GRPO ... - Store gains fetch_record/num_records alongside stream fetch/__len__ - save_bin/load_bin support per-record offsets via record_keys param - H5Store/MmapStore/JsonlStore all support dual stream+record access - DPODataset/GRPODataset use fetch_record, no cross-record concat - dpo_collate_fn + collate_fn wired through TrainConfig - fixes attention context leakage in DPO from windowed concatenation	2026-07-18 21:02:29 +08:00
ViperEkura	8999ca89b8	feat: add JSONL dataset store with on-the-fly tokenization ... - Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py - Reuse PipelineConfig schema for JSONL dataset configuration - Update detect_format to recognize JSONL directories and files - Move save_h5/load_h5/save_bin/load_bin to astrai/serialization - Split astrai/serialization.py into checkpoint/dataset submodules - Add tests for JSONL detection, seq/SFT stores, and config roundtrip	2026-07-04 15:42:33 +08:00