ViperEkura cd14d53707 feat: implement bfd_split packing strategy ...

- BFDSplitPacking splits over-length sequences into chunks before BFD
- All keys (loss_mask, position_ids, ...) split in lockstep for alignment
- No tokens lost vs bfd which truncates over-length sequences
- Tests: token preservation, chunk alignment, short unchanged, vs bfd

2026-07-17 12:38:56 +08:00

..

conftest.py

refactor: split mask builder by single/multi output

2026-07-08 21:18:34 +08:00

test_checkpoint.py

refactor : replace iteration with consumed_samples

2026-06-30 18:42:42 +08:00

test_dataset.py

feat: support raw JSON files in dataset pipeline and JsonlStore

2026-07-17 12:20:03 +08:00

test_preprocess_builder.py

refactor: split mask builder by single/multi output

2026-07-08 21:18:34 +08:00

test_preprocess_config.py

refactor : 清理工厂和配置系统中的死代码与冗余抽象

2026-06-07 11:39:50 +08:00

test_preprocess_pipeline.py

feat: implement bfd_split packing strategy

2026-07-17 12:38:56 +08:00

test_sampler.py

chore: 将data 模块命名为dataset

2026-04-04 16:16:27 +08:00