docs : 更新架构文档与 storage 注释,同步 Store 重构

- architecture.md: 类图/关系线全部更新 (BaseStorage→Store, StorageFactory→StoreFactory, 新增 MmapStore)
- architecture.md: 移除 BaseSegmentFetcher/MultiSegmentFetcher 类图与关系
- dataflow.md: 管线加入 .bin 格式, Store._data + _cum 架构
- storage.py: module docstring 改用缩进式注释风格
This commit is contained in:
2026-05-28 14:36:18 +08:00
parent 6e150ea6d0
commit 0a708fff24
3 changed files with 50 additions and 73 deletions
+8 -7
View File
@@ -5,21 +5,22 @@ This document describes the data pipeline: from raw text to model input tensors.
## Overview
```
Raw Text → AutoTokenizer → Token IDs → .h5/.json → Dataset → Sampler → DataLoader → Training/Inference
Raw Text → AutoTokenizer → Token IDs → .h5/.json/.bin → Dataset → Sampler → DataLoader → Training/Inference
```
## Data Preparation
Raw text is tokenized via `AutoTokenizer.encode()` and saved as HDF5 (`.h5`) or JSON (`.json`/`.jsonl`) files with keyed tensor groups.
Raw text is tokenized via `AutoTokenizer.encode()` and saved as HDF5 (`.h5`), JSON (`.json`/`.jsonl`), or binary (`.bin` + `meta.json`) files with keyed tensor groups.
Storage format is auto-detected by `detect_format()`; backends are dispatched via registry:
```
StorageFactory.create("h5") → H5Storage
StorageFactory.create("json") → JSONStorage
StoreFactory.create("h5") → H5Store
StoreFactory.create("json") → JSONStore
StoreFactory.create("bin") → MmapStore
```
Both support shared memory via `.share_memory_()`.
H5 and JSON backends support shared memory via `.share_memory_()`. Bin (mmap) uses OS page-cache sharing natively.
## Data Keys by Training Type
@@ -34,8 +35,8 @@ Both support shared memory via `.share_memory_()`.
```
DatasetFactory.load(train_type, path, window_size, stride, storage_type, tokenizer)
→ StorageFactory.create(detect_format(path))
MultiSegmentFetcher(BaseSegmentFetcher per key)
→ StoreFactory.create(detect_format(path))
Store._data[Dict[str, List[Tensor]]] + _cum[Dict[str, List[int]]]
→ BaseDataset.__getitem__(idx)
→ sliding window [begin, end) via get_index(idx)
```