docs : 更新架构文档与 storage 注释,同步 Store 重构

- architecture.md: 类图/关系线全部更新 (BaseStorage→Store, StorageFactory→StoreFactory, 新增 MmapStore)
- architecture.md: 移除 BaseSegmentFetcher/MultiSegmentFetcher 类图与关系
- dataflow.md: 管线加入 .bin 格式, Store._data + _cum 架构
- storage.py: module docstring 改用缩进式注释风格
This commit is contained in:
2026-05-28 14:36:18 +08:00
parent 6e150ea6d0
commit 0a708fff24
3 changed files with 50 additions and 73 deletions
+13 -29
View File
@@ -1,36 +1,20 @@
"""Storage backends for different data formats.
Design
------
Three-layer architecture:
1. **I/O layer** — ``save_*`` / ``load_*`` functions that read/write raw files
(HDF5, JSON, binary) and return ``Dict[str, List[Tensor]]`` (multi-segment).
These are format-specific, low-level helpers — no abstraction, no state.
2. **Store (ABC)** — the central abstraction. Each concrete ``Store`` calls the
I/O layer during ``load()``, then **normalizes** multi-segment data into a
single contiguous tensor per key via ``_normalize()``. After that, ``fetch()``
is just a vanilla slice — no ``bisect``, no segment bookkeeping.
Data format inside a ``Store``::
self._data = {"sequence": Tensor, "loss_mask": Tensor, ...}
self._length = N # min first-dim size across keys, O(1)
3. **Dataset layer** — ``BaseDataset`` owns a ``Store`` and only calls
``store.fetch(begin, end, key)``. It never knows whether the data came
from HDF5, JSON, or mmap.
Layers:
- I/O layer: save_* / load_* functions, read/write raw files (HDF5/JSON/bin)
return Dict[str, List[Tensor]] — format-specific, no state
- Store (ABC): central abstraction, normalizes multi-segment into
Dict[str, List[Tensor]] per key via _normalize(),
fetch() uses bisect across segments — no forced concat
- Dataset layer: BaseDataset owns a Store, only calls store.fetch(begin, end, key)
Key properties:
- **Explicit length**: ``_length`` is set during ``load()`` and exposed via
``__len__`` (O(1)). No hidden computation inside a fetcher.
- **Zero-copy mmap**: ``MmapStore`` wraps ``np.memmap(mode="r")`` tensors.
Multiple DataLoader workers share the same OS page-cache pages.
- **Lazy concat**: ``H5Store`` / ``JSONStore`` concatenate segments at load
time, so fetch-time logic is trivial.
- Multi-segment: segments kept as-is, no forced concatenation — safe for
datasets larger than RAM
- Explicit length: _length = min(total elements across keys), set at load,
__len__ returns O(1)
- Zero-copy mmap: MmapStore wraps np.memmap(mode="r"), all DataLoader
workers share OS page-cache pages
"""
import bisect