refactor : Pipeline 去除去重，ids 重命名为 sequence，泛型透传

- 移除 Pipeline 内置去重逻辑及 dedup_signature 工具函数 - 删除 ProcessingConfig.deduplicate 字段 - builder 返回 'sequence' 替代 'ids'，与 dataset 层统一 - pipeline 纯透传，泛型处理任意 key 补齐默认值
2026-05-31 15:14:27 +08:00
parent 14f83cbdac
commit 01ce1fb9e3
6 changed files with 32 additions and 52 deletions
@@ -3,13 +3,12 @@ from astrai.preprocessing.builder import (
    MaskBuilderFactory,
    SectionedMaskBuilder,
 )
-from astrai.preprocessing.pipeline import Pipeline, dedup_signature, filter_by_length
+from astrai.preprocessing.pipeline import Pipeline, filter_by_length

 __all__ = [
    "BaseMaskBuilder",
    "MaskBuilderFactory",
    "SectionedMaskBuilder",
    "Pipeline",
-    "dedup_signature",
    "filter_by_length",
 ]