feat: SFT 增加 position_ids 边界处理

- SFTProcessor 输出 per-sample position_ids(torch.arange),每个样本从 0 开始
- position_ids 与 sequence/loss_mask 一同打包,边界处自然重置
- PT 路径不生成 position_ids
- 新增 TestPositionIds 测试及 SFTProcessor 相关测试
This commit is contained in:
2026-06-04 14:19:32 +08:00
parent efba7a009f
commit aa2ea4f3a6
4 changed files with 193 additions and 20 deletions
-1
View File
@@ -5,7 +5,6 @@ import logging
import os
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional, Union
from datasets import Dataset
from tqdm import tqdm