AstrAI

Files

T

ViperEkura cd14d53707 feat: implement bfd_split packing strategy

- BFDSplitPacking splits over-length sequences into chunks before BFD
- All keys (loss_mask, position_ids, ...) split in lockstep for alignment
- No tokens lost vs bfd which truncates over-length sequences
- Tests: token preservation, chunk alignment, short unchanged, vs bfd

2026-07-17 12:38:56 +08:00

__init__.py

refactor: split mask builder by single/multi output

2026-07-08 21:18:34 +08:00

builder.py

refactor: split mask builder by single/multi output

2026-07-08 21:18:34 +08:00

packing.py

feat: implement bfd_split packing strategy

2026-07-17 12:38:56 +08:00

pipeline.py

feat: support raw JSON files in dataset pipeline and JsonlStore