- BFDSplitPacking splits over-length sequences into chunks before BFD - All keys (loss_mask, position_ids, ...) split in lockstep for alignment - No tokens lost vs bfd which truncates over-length sequences - Tests: token preservation, chunk alignment, short unchanged, vs bfd |
||
|---|---|---|
| .. | ||
| config | ||
| dataset | ||
| extension | ||
| inference | ||
| model | ||
| parallel | ||
| preprocessing | ||
| serialization | ||
| tokenize | ||
| trainer | ||
| __init__.py | ||
| factory.py | ||
| protocols.py | ||