ViperEkura 8999ca89b8 feat: add JSONL dataset store with on-the-fly tokenization ... - Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py - Reuse PipelineConfig schema for JSONL dataset configuration - Update detect_format to recognize JSONL directories and files - Move save_h5/load_h5/save_bin/load_bin to astrai/serialization - Split astrai/serialization.py into checkpoint/dataset submodules - Add tests for JSONL detection, seq/SFT stores, and config roundtrip		2026-07-04 15:42:33 +08:00
..
__init__.py	refactor : pipeline 策略化拆分，消除 _flush if/else	2026-06-06 00:45:33 +08:00
builder.py	refactor : BaseFactory 基类类型自动推导 + 移除冗余代码	2026-06-06 21:23:41 +08:00
packing.py	fix: 修复预处理流水线 4 个致命问题	2026-06-18 17:38:01 +08:00
pipeline.py	fix: 修复预处理流水线 4 个致命问题	2026-06-18 17:38:01 +08:00
position_id.py	fix : 修复策略相关文件的类型注解与抽象方法体	2026-06-14 16:20:10 +08:00
writer.py	feat: add JSONL dataset store with on-the-fly tokenization	2026-07-04 15:42:33 +08:00