fix : 修复 created 时间戳、bin 多 shard 覆盖与文档遗漏

- openai.py/anthropic.py: created 从 0 改为 int(time.time())
- openai.py: ChatCompletionRequest 不支持参数非默认值时 warning
- pipeline.py: bin 多 shard 使用子目录避免静默覆盖
- storage.py: MmapStore/detect_format 支持多 shard 聚合加载
- architecture.md: mermaid 类图新增 Pipeline 类
- preprocessing.md: 新增多 shard 输出布局与 Python API 示例
- protocol.py: docstring "6 methods" 改为 "5 methods"
This commit is contained in:
2026-05-30 23:03:42 +08:00
parent 1c2ff05a6d
commit 2a65c3314c
8 changed files with 122 additions and 11 deletions
+12
View File
@@ -363,6 +363,16 @@ classDiagram
class TextMaskBuilder {
+build(item, config, tokenizer) Optional[dict]
}
class Pipeline {
+PipelineConfig config
+List[str] paths
+str output_dir
+str tokenizer_path
+BaseMaskBuilder mask_builder
+transform(item) Optional[dict]
+run()
}
}
namespace tokenize {
@@ -1092,6 +1102,8 @@ classDiagram
KvcacheView o-- Storage
SamplingPipeline o-- BaseSamplingStrategy
BaseDataset o-- Store
Pipeline o-- PipelineConfig
Pipeline o-- BaseMaskBuilder
%% --- Dependency (uses temporarily) ---
TrainConfig ..> BaseStrategy : selects