ViperEkura/AstrAI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix : SFT pipeline position_ids default & doc boundary preservation

Browse Source 
- change position_ids_mode default from "none" to "doc_reset" so SFT preprocessing always generates position_ids (was causing dataset load KeyError)
- generate per-doc position_ids before packing (doc_reset mode), preserving document boundaries for BFD packing (cross-doc attention leak fix)
- change _align_bucket padding from [1] to [0] to avoid accidentally training on loss_mask padding
This commit is contained in:
ViperEkura
2026-07-04 15:59:11 +08:00
parent 8999ca89b8
commit 4e508afa2d
3 changed files with 13 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/docs/preprocessing.md
+1 -1
View File
@@ -268,7 +268,7 @@ When `sources` is set, `sections` is ignored.
| `storage_format` | str | `"bin"` | `"bin"` (mmap) or `"h5"` |
| `max_tokens_per_shard` | int | `100000000` | Flush threshold in cumulative tokens |
| `dtype` | dict[str, str] | `{}` | Per-key tensor dtype override (e.g. `{"loss_mask": "bool"}`) |
| `position_ids_mode` | str | `"none"` | How to compute position_ids: `"none"`, `"doc_reset"`, `"continuous"` |
| `position_ids_mode` | str | `"doc_reset"` | How to compute position_ids: `"none"`, `"doc_reset"`, `"continuous"` |
---