Files

T

- 移除 JSONStore 引用（该类不存在）
- 修正 Store.load() 和 DatasetFactory.load() 签名（无 tokenizer 参数）
- 修正 TrainContextBuilder.with_resume_dir() 命名
- 修正 Checkpoint config 字段和 meta.json 描述
- 修正 ProtocolHandler.handle() 异步签名
- 修正采样继承图（平行子类，非线性）
- 修正训练循环：回调移入 accumulate 块内
- 更新文档日期至 2026-05-28

2026-05-28 21:01:47 +08:00

1.8 KiB

Raw Permalink Blame History

Data Flow

This document describes the data pipeline: from raw text to model input tensors.

Overview

Raw Text → AutoTokenizer → Token IDs → .h5/.bin → Dataset → Sampler → DataLoader → Training/Inference

Data Preparation

Raw text is tokenized via AutoTokenizer.encode() and saved as HDF5 (.h5) or binary (.bin + meta.json) files with keyed tensor groups.

Storage format is auto-detected by detect_format(); backends are dispatched via registry:

StoreFactory.create("h5")  → H5Store
StoreFactory.create("bin") → MmapStore

H5 backend supports shared memory via .share_memory_(). Bin (mmap) uses OS page-cache sharing natively.

Data Keys by Training Type

Type	Storage Keys
`seq`	`sequence` (→ input_ids, target_ids via offset-by-1)
`sft`	`sequence`, `loss_mask`
`dpo`	`chosen`, `rejected`, `chosen_mask`, `rejected_mask`
`grpo`	`prompts`, `responses`, `masks`, `rewards`

Dataset Architecture

DatasetFactory.load(train_type, load_path, window_size, stride, storage_type)
  → StoreFactory.create(detect_format(path))
    → Store._data[Dict[str, List[Tensor]]] + _cum[Dict[str, List[int]]]
      → BaseDataset.__getitem__(idx)
        → sliding window [begin, end) via get_index(idx)

window_size = max input length, stride = step between consecutive samples (defaults to window_size).

Sampler

ResumableDistributedSampler supports checkpoint-aware distributed sampling:

Tracks start_epoch / start_iter for resume
Shuffle via torch.Generator(seed + epoch)
Per-replica index slicing for DDP

DataLoader

Standard PyTorch DataLoader with configurable batch_size, num_workers, pin_memory, prefetch_factor. Sampler produces indices; dataloader fetches tensor batches via __getitem__.

Document Update Time: 2026-05-28

1.8 KiB Raw Permalink Blame History