docs : 按代码反向修正所有文档错误

- 更新预处理模块目录结构和类名(SectionedMaskBuilder)
- 修正 ResponseBuilder.prepare 签名(tokenizer → engine)
- 补全缺失的 CLI 参数、配置字段和数据键名
- 修正 README 中 download.py 的描述
This commit is contained in:
2026-06-06 01:06:30 +08:00
parent 31bc7f5c2a
commit cf9c60841b
6 changed files with 44 additions and 20 deletions
+4
View File
@@ -243,6 +243,9 @@ When `sources` is set, `sections` is ignored.
| `min_chars` | int | `50` | Skip text-mode items shorter than this |
| `max_chars` | int | `2000000` | Skip text-mode items longer than this |
| `max_items` | int or null | `null` | Stop after N documents |
| `packing_strategy` | str | `"simple"` | Packing strategy: `"simple"`, `"bfd"`, `"bfd_split"` |
| `max_packed_len` | int | `8192` | Maximum length of a packed bin |
| `truncation_mode` | str | `"keep_start"` | How to truncate sequences: `"keep_start"` or `"keep_end"` |
### `output`
@@ -252,6 +255,7 @@ When `sources` is set, `sections` is ignored.
| `storage_format` | str | `"bin"` | `"bin"` (mmap) or `"h5"` |
| `max_tokens_per_shard` | int | `100000000` | Flush threshold in cumulative tokens |
| `dtype` | dict[str, str] | `{}` | Per-key tensor dtype override (e.g. `{"loss_mask": "bool"}`) |
| `position_ids_mode` | str | `"none"` | How to compute position_ids: `"none"`, `"doc_reset"`, `"continuous"` |
---