5 Commits

Include renames

Author	SHA1	Message	Date
ViperEkura	33c8720d69	feat: 并行 batch tokenization + cache_jsonl 批处理支持 ... - pipeline/tokenize/tokenizer.py: encode() 全部走 encode_batch(支持单条/批量) - pipeline/processors/base.py: BaseProcessor 新增 process_batch() - pipeline/processors/pretrain.py: PreTrainProcessor 覆盖 process_batch() 批量编码 - pipeline/io/export.py: cache_jsonl 新增 batch_size 参数默认 1000, 批量处理 - scripts/cache_h5.py: 新增 --batch-size 参数, 默认 tokenizer 路径改为 ../AstrAI/params	2026-07-25 12:19:45 +08:00
ViperEkura	cb6bfcb976	refactor: 重构流水线架构，添加Pipeline抽象并拆分IOHandler	2026-04-23 19:45:57 +08:00
ViperEkura	a38334f4ce	refactor: 修改模型架构	2026-04-16 18:57:21 +08:00
ViperEkura	f44ad6912e	fix: 修复特殊token 问题	2026-04-02 16:16:02 +08:00
ViperEkura	35963bcb08	reafactor: 重构项目	2026-03-30 20:58:51 +08:00