refactor: deduplicate preprocessing kernel and BFD packing

- Extract shared core (mask building, primary-id extraction, tensorisation, position-id generation) to astrai/preprocessing/core.py; Pipeline and TokenizeTransform both consume it, eliminating ~60% duplicated logic
- Promote BFD _plan to module-level plan_bfd(lengths, max_len) returning pure index bins; BFDPacking.apply and evaluate_ifd._pack_bins both call it, removing the second BFD implementation
- Split Pipeline._flush (49 lines) into _inject_doc_reset_position_ids + _inject_continuous_position_ids + _to_tensors; split Pipeline.run by delegating record iteration to core.iter_raw_records
- Remove dead no-op pop/塞回 in Pipeline.run (L110-111)
This commit is contained in:
2026-07-19 12:27:56 +08:00
parent 17127f8b3c
commit 31c22dc043
6 changed files with 297 additions and 146 deletions
+2
View File
@@ -8,6 +8,7 @@ from astrai.preprocessing.builder import (
from astrai.preprocessing.packing import (
PackingStrategy,
PackingStrategyFactory,
plan_bfd,
)
from astrai.preprocessing.pipeline import Pipeline, filter_by_length
from astrai.preprocessing.position_id import (
@@ -35,4 +36,5 @@ __all__ = [
"StoreWriterFactory",
"TokenizeTransform",
"filter_by_length",
"plan_bfd",
]