AstrAI

Commit Graph

Author	SHA1	Message	Date
ViperEkura	cd14d53707	feat: implement bfd_split packing strategy - BFDSplitPacking splits over-length sequences into chunks before BFD - All keys (loss_mask, position_ids, ...) split in lockstep for alignment - No tokens lost vs bfd which truncates over-length sequences - Tests: token preservation, chunk alignment, short unchanged, vs bfd	2026-07-17 12:38:56 +08:00
ViperEkura	25d4ea3f91	refactor : 压缩测试代码，消除重复 - fixture 替代重复实例化和 tokenizer 落盘 - parametrize 合并同构测试 - helper 消除 save_h5 + DatasetFactory.load 样板 - 净减 272 行	2026-06-19 14:54:39 +08:00
ViperEkura	02a7cb9fa0	feat : preprocessing 支持 DPO/GRPO 多输出格式 - InputConfig 新增 sources 字段驱动多输出映射 - SectionedMaskBuilder 提取 _process_sections/_build_multi 模板方法 - Pipeline 泛化 accumulate 逻辑处理多 key 结果 - 测试拆分为 config/builder/pipeline 三文件，纯函数风格	2026-06-03 10:32:10 +08:00

Author

SHA1

Message

Date

ViperEkura

cd14d53707

feat: implement bfd_split packing strategy

- BFDSplitPacking splits over-length sequences into chunks before BFD
- All keys (loss_mask, position_ids, ...) split in lockstep for alignment
- No tokens lost vs bfd which truncates over-length sequences
- Tests: token preservation, chunk alignment, short unchanged, vs bfd

2026-07-17 12:38:56 +08:00

ViperEkura

25d4ea3f91

refactor : 压缩测试代码，消除重复

- fixture 替代重复实例化和 tokenizer 落盘
- parametrize 合并同构测试
- helper 消除 save_h5 + DatasetFactory.load 样板
- 净减 272 行

2026-06-19 14:54:39 +08:00

ViperEkura

02a7cb9fa0

feat : preprocessing 支持 DPO/GRPO 多输出格式

- InputConfig 新增 sources 字段驱动多输出映射
- SectionedMaskBuilder 提取 _process_sections/_build_multi 模板方法
- Pipeline 泛化 accumulate 逻辑处理多 key 结果
- 测试拆分为 config/builder/pipeline 三文件，纯函数风格

2026-06-03 10:32:10 +08:00

3 Commits