ViperEkura cd14d53707 feat: implement bfd_split packing strategy ... - BFDSplitPacking splits over-length sequences into chunks before BFD - All keys (loss_mask, position_ids, ...) split in lockstep for alignment - No tokens lost vs bfd which truncates over-length sequences - Tests: token preservation, chunk alignment, short unchanged, vs bfd		2026-07-17 12:38:56 +08:00
..
__init__.py	refactor: split mask builder by single/multi output	2026-07-08 21:18:34 +08:00
builder.py	refactor: split mask builder by single/multi output	2026-07-08 21:18:34 +08:00
packing.py	feat: implement bfd_split packing strategy	2026-07-17 12:38:56 +08:00
pipeline.py	feat: support raw JSON files in dataset pipeline and JsonlStore	2026-07-17 12:20:03 +08:00
position_id.py	fix : 修复策略相关文件的类型注解与抽象方法体	2026-06-14 16:20:10 +08:00
writer.py	feat: add JSONL dataset store with on-the-fly tokenization	2026-07-04 15:42:33 +08:00