refactor: decouple tokenizer from Store into Transform layer
- Extract tokenization/mask/position logic from JsonlStore into TokenizeTransform - JsonlStore now pure reader: reads JSON records, delegates to transform - Store no longer imports tokenizer or preprocessing components - Replace per_record param with segments_are_records class attribute - Store subclasses declare segment semantics as format-level property
This commit is contained in:
@@ -14,6 +14,7 @@ from astrai.preprocessing.position_id import (
|
||||
PositionIdStrategy,
|
||||
PositionIdStrategyFactory,
|
||||
)
|
||||
from astrai.preprocessing.transform import TokenizeTransform
|
||||
from astrai.preprocessing.writer import (
|
||||
StoreWriter,
|
||||
StoreWriterFactory,
|
||||
@@ -32,5 +33,6 @@ __all__ = [
|
||||
"SingleOutputMaskBuilder",
|
||||
"StoreWriter",
|
||||
"StoreWriterFactory",
|
||||
"TokenizeTransform",
|
||||
"filter_by_length",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user