refactor : Pipeline 去除去重，ids 重命名为 sequence，泛型透传

- 移除 Pipeline 内置去重逻辑及 dedup_signature 工具函数 - 删除 ProcessingConfig.deduplicate 字段 - builder 返回 'sequence' 替代 'ids'，与 dataset 层统一 - pipeline 纯透传，泛型处理任意 key 补齐默认值
2026-05-31 15:14:27 +08:00
parent 14f83cbdac
commit 01ce1fb9e3
6 changed files with 32 additions and 52 deletions
@@ -16,7 +16,6 @@ class ProcessingConfig(BaseConfig):
    max_seq_len: int = 2048
    min_chars: int = 50
    max_chars: int = 2_000_000
-    deduplicate: bool = True
    max_items: Optional[int] = None