 ViperEkura
|
33c8720d69
|
feat: 并行 batch tokenization + cache_jsonl 批处理支持
- pipeline/tokenize/tokenizer.py: encode() 全部走 encode_batch(支持单条/批量)
- pipeline/processors/base.py: BaseProcessor 新增 process_batch()
- pipeline/processors/pretrain.py: PreTrainProcessor 覆盖 process_batch() 批量编码
- pipeline/io/export.py: cache_jsonl 新增 batch_size 参数默认 1000, 批量处理
- scripts/cache_h5.py: 新增 --batch-size 参数, 默认 tokenizer 路径改为 ../AstrAI/params
|
2026-07-25 12:19:45 +08:00 |
|
|
ViperEkura
|
e6787a2036
|
feat: add BELLE SFT export script, remove Firefly-1.1M-Rephrased
|
2026-07-17 23:15:18 +08:00 |
|
|
ViperEkura
|
fec37545e2
|
Add max_chunks=18 to dolma-30b export, remove max_chunks=5 from english-wiki, rename project to datapipline
|
2026-07-06 08:53:18 +08:00 |
|
|
ViperEkura
|
900cd91798
|
feat: MinHash+LSH 去重 + Strategy/Factory 存储后端
|
2026-07-04 14:47:38 +08:00 |
|
|
ViperEkura
|
816c02dab0
|
refactor: SFT 统一 messages 格式 + ChatML 纯 jinja 渲染
|
2026-07-04 14:32:35 +08:00 |
|
|
ViperEkura
|
598e1ce4ae
|
refactor: 重构打包模块，新增 BFD/FFD/Greedy 三种 bin-packing 算法，默认 BFD
- 将 pipeline/packing.py 拆分为 packing/ 子包 (base/stream/binpack)
- 新增 BfdPacker(默认)/FfDPacker/GreedyPacker，移除 StreamingPacker
- 超长序列直接截断至 pack_size
- group_size 语义改为"每 N 个 chunk 合并为一块"，默认 1000
- 新增 AutoTokenizer.token_to_id()，修复 ChatML 中 hacky 的 nl_id 获取
- pad_value 默认改为 2(pad_token_id)，position_ids pad=0, loss_mask pad=False
- 新增 position_ids 打包后归零一致性测试
- scripts/cache_h5.py 新增 --pack-algo 参数
|
2026-07-03 16:17:27 +08:00 |
|
|
ViperEkura
|
2f919e9243
|
feat: add MetaMathQA SFT export script and fix ChatML newline encoding
|
2026-07-02 17:40:03 +08:00 |
|
|
ViperEkura
|
fd99b9d852
|
feat: add OpenHermes-2.5 SFT export script
|
2026-06-04 14:27:04 +08:00 |
|
 jiachun
|
efba7a009f
|
feat: add sft scripts for Magicoder-Evol-Instruct-110K and alpaca-gpt4-data
- sft_magicoder.py: export ise-uiuc/Magicoder-Evol-Instruct-110K
- sft_alpaca_gpt4.py: export llm-wizard/alpaca-gpt4-data with instruction+input
- remove deprecated SFT scripts
|
2026-06-01 15:50:13 +08:00 |
|
|
ViperEkura
|
625695fd71
|
fix: padding 默认值改为 0，避免 loss_mask 误计 padding 位置
|
2026-05-15 16:59:42 +08:00 |
|
|
ViperEkura
|
cb6bfcb976
|
refactor: 重构流水线架构，添加Pipeline抽象并拆分IOHandler
|
2026-04-23 19:45:57 +08:00 |
|
|
ViperEkura
|
a38334f4ce
|
refactor: 修改模型架构
|
2026-04-16 18:57:21 +08:00 |
|
|
ViperEkura
|
f44ad6912e
|
fix: 修复特殊token 问题
|
2026-04-02 16:16:02 +08:00 |
|
|
ViperEkura
|
35963bcb08
|
reafactor: 重构项目
|
2026-03-30 20:58:51 +08:00 |
|
|
ViperEkura
|
07ef471aa6
|
chore: 增加 h5_cache 脚本
|
2026-03-21 21:33:38 +08:00 |
|
|
ViperEkura
|
909f1b400c
|
refactor: 修改项目结构
|
2026-03-21 21:25:59 +08:00 |
|