ViperEkura
115192c67c
refactor: remove H5 storage backend in favor of mmap bin
...
- Remove H5Store, H5Writer, save_h5/load_h5 and h5py dependency
- MmapStore (bin) is the sole pre-tokenized storage backend
- Move setup_logging after imports to fix E402 in __init__.py
- Clean up unused imports across test files
- Move inline test imports to file top
2026-07-29 12:50:27 +08:00
ViperEkura
8999ca89b8
feat: add JSONL dataset store with on-the-fly tokenization
...
- Add JsonlStore registered under "jsonl" in astrai/dataset/storage.py
- Reuse PipelineConfig schema for JSONL dataset configuration
- Update detect_format to recognize JSONL directories and files
- Move save_h5/load_h5/save_bin/load_bin to astrai/serialization
- Split astrai/serialization.py into checkpoint/dataset submodules
- Add tests for JSONL detection, seq/SFT stores, and config roundtrip
2026-07-04 15:42:33 +08:00
ViperEkura
d88a41f8f1
fix: 修复预处理流水线 4 个致命问题
...
- pipeline: 单条数据异常不再崩溃整条流水线, 改 log warning 后跳过
- pipeline: _align_bucket 统一用 len(ids) 填充, 修复多输出模式下长度错配
- writer: BinWriter/H5Writer 写入失败自动清理残留文件并记录详细错误
- packing: BFDPacking 真正将序列打包进 bin 而非仅重排, 减少碎片
2026-06-18 17:38:01 +08:00
ViperEkura
e7b18b7c03
refactor : BaseFactory 基类类型自动推导 + 移除冗余代码
...
- _validate_component 从 BaseFactory[T] 泛型参数自动解析基类类型，9 个子类覆写移除
- Registry 类内联到 BaseFactory._entries，移除未用的 list_by_category/list_by_priority
- _component_base 在 __init_subclass__ 时立即解析
- 数据集 4 个子类冗余 __init__ 移除
2026-06-06 21:23:41 +08:00
ViperEkura
31bc7f5c2a
refactor : pipeline 策略化拆分，消除 _flush if/else
...
- PackingStrategy / PositionIdStrategy / StoreWriter 独立文件 + Factory
- Pipeline._flush 零 if/else，纯编排
- SectionRenderer 从 SectionedMaskBuilder 分离
- OutputConfig.position_ids_mode 默认改为 ""none""
2026-06-06 00:45:33 +08:00
ViperEkura