ViperEkura
a1ea26d367
fix: rewrite GRPO data pipeline for offline record-level access
...
- process_list_field returns List[List[int]] preserving per-response boundaries
- GRPODataset rewritten to record-level __getitem__ (no windowing/stride)
- grpo_collate_fn pads variable-length responses into [B, G, R] tensors
- JsonlStore detects nested List[List[int]] and stores List[Tensor] per record
- Store._normalize skips nested-list keys from cumsum bookkeeping
- Pipeline._flush handles nested lists without cross-record flattening
- Export grpo_collate_fn from astrai.dataset
- 6 new GRPO tests + 2 updated builder tests, 114 total pass
2026-07-17 14:34:41 +08:00
ViperEkura
841a582b28
refactor: split mask builder by single/multi output
...
- Extract SingleOutputMaskBuilder for SFT and pretrain configs
- Extract MultiOutputMaskBuilder for DPO and GRPO configs
- Keep SectionedMaskBuilder as backward-compatible facade
- Register "single" and "multi" names in MaskBuilderFactory
- Add parity and rejection tests for concrete builders
2026-07-08 21:18:34 +08:00
ViperEkura
e7b18b7c03
refactor : BaseFactory 基类类型自动推导 + 移除冗余代码
...
- _validate_component 从 BaseFactory[T] 泛型参数自动解析基类类型，9 个子类覆写移除
- Registry 类内联到 BaseFactory._entries，移除未用的 list_by_category/list_by_priority
- _component_base 在 __init_subclass__ 时立即解析
- 数据集 4 个子类冗余 __init__ 移除
2026-06-06 21:23:41 +08:00
ViperEkura
31bc7f5c2a
refactor : pipeline 策略化拆分，消除 _flush if/else
...
- PackingStrategy / PositionIdStrategy / StoreWriter 独立文件 + Factory
- Pipeline._flush 零 if/else，纯编排
- SectionRenderer 从 SectionedMaskBuilder 分离
- OutputConfig.position_ids_mode 默认改为 ""none""
2026-06-06 00:45:33 +08:00
ViperEkura
02a7cb9fa0
feat : preprocessing 支持 DPO/GRPO 多输出格式
...
- InputConfig 新增 sources 字段驱动多输出映射
- SectionedMaskBuilder 提取 _process_sections/_build_multi 模板方法
- Pipeline 泛化 accumulate 逻辑处理多 key 结果
- 测试拆分为 config/builder/pipeline 三文件，纯函数风格
2026-06-03 10:32:10 +08:00
ViperEkura
01ce1fb9e3
refactor : Pipeline 去除去重，ids 重命名为 sequence，泛型透传
...
- 移除 Pipeline 内置去重逻辑及 dedup_signature 工具函数
- 删除 ProcessingConfig.deduplicate 字段
- builder 返回 'sequence' 替代 'ids'，与 dataset 层统一
- pipeline 纯透传，泛型处理任意 key 补齐默认值
2026-05-31 15:14:27 +08:00
ViperEkura
dbe5891201
refactor : 统一 SectionedMaskBuilder，支持可配置 dtype
...
- 三合一 MaskBuilder，移除 chat/instruction/text，统一为 sections 配置
- OutputConfig 增加 dtype 字段 (per-key，默认 int32)
- 移除 from __future__ import annotations
- 测试适配新配置格式
2026-05-31 14:24:10 +08:00
ViperEkura
69207e2c57
refactor : 基于声明式 JSON 配置的预处理管线重构
...
- 用工厂注册的 MaskBuilder（chat/instruction/text）替换硬编码的 _transform_* 方法
- mask 规则以 role-to-action 映射声明在配置中，与 chat_template 完全解耦
- 单次编码 + role-span 追踪替代两次编码 + 长度差计算 mask 的方式
- 支持多轮对话训练：所有 assistant 轮次参与训练，而非仅最后一轮
- 新建 astrai.preprocessing 包（builder.py + pipeline.py），删除 astrai/preprocess.py
- CLI 精简为 --config 参数，所有参数通过 PipelineConfig JSON 配置
- 新增 PipelineConfig、InputConfig、ProcessingConfig、OutputConfig dataclass
- 文档：assets/docs/preprocessing.md
- 27 个测试覆盖 mask builder、pipeline、配置序列化、工厂注册
2026-05-30 20:45:09 +08:00
ViperEkura