refactor: decouple tokenizer from Store into Transform layer

- Extract tokenization/mask/position logic from JsonlStore into TokenizeTransform
- JsonlStore now pure reader: reads JSON records, delegates to transform
- Store no longer imports tokenizer or preprocessing components
- Replace per_record param with segments_are_records class attribute
- Store subclasses declare segment semantics as format-level property
This commit is contained in:
2026-07-18 21:37:31 +08:00
parent a74e5b91a3
commit b33250dc28
3 changed files with 153 additions and 91 deletions
+2
View File
@@ -14,6 +14,7 @@ from astrai.preprocessing.position_id import (
PositionIdStrategy,
PositionIdStrategyFactory,
)
from astrai.preprocessing.transform import TokenizeTransform
from astrai.preprocessing.writer import (
StoreWriter,
StoreWriterFactory,
@@ -32,5 +33,6 @@ __all__ = [
"SingleOutputMaskBuilder",
"StoreWriter",
"StoreWriterFactory",
"TokenizeTransform",
"filter_by_length",
]