refactor : 基于声明式 JSON 配置的预处理管线重构
- 用工厂注册的 MaskBuilder(chat/instruction/text)替换硬编码的 _transform_* 方法 - mask 规则以 role-to-action 映射声明在配置中,与 chat_template 完全解耦 - 单次编码 + role-span 追踪替代两次编码 + 长度差计算 mask 的方式 - 支持多轮对话训练:所有 assistant 轮次参与训练,而非仅最后一轮 - 新建 astrai.preprocessing 包(builder.py + pipeline.py),删除 astrai/preprocess.py - CLI 精简为 --config 参数,所有参数通过 PipelineConfig JSON 配置 - 新增 PipelineConfig、InputConfig、ProcessingConfig、OutputConfig dataclass - 文档:assets/docs/preprocessing.md - 27 个测试覆盖 mask builder、pipeline、配置序列化、工厂注册
This commit is contained in:
@@ -0,0 +1,19 @@
|
||||
from astrai.preprocessing.builder import (
|
||||
BaseMaskBuilder,
|
||||
ChatMaskBuilder,
|
||||
InstructionMaskBuilder,
|
||||
MaskBuilderFactory,
|
||||
TextMaskBuilder,
|
||||
)
|
||||
from astrai.preprocessing.pipeline import Pipeline, dedup_signature, filter_by_length
|
||||
|
||||
__all__ = [
|
||||
"BaseMaskBuilder",
|
||||
"ChatMaskBuilder",
|
||||
"InstructionMaskBuilder",
|
||||
"MaskBuilderFactory",
|
||||
"TextMaskBuilder",
|
||||
"Pipeline",
|
||||
"dedup_signature",
|
||||
"filter_by_length",
|
||||
]
|
||||
Reference in New Issue
Block a user