feat: SFT 增加 position_ids 边界处理
- SFTProcessor 输出 per-sample position_ids(torch.arange),每个样本从 0 开始 - position_ids 与 sequence/loss_mask 一同打包,边界处自然重置 - PT 路径不生成 position_ids - 新增 TestPositionIds 测试及 SFTProcessor 相关测试
This commit is contained in:
@@ -5,7 +5,6 @@ import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional, Union
|
||||
|
||||
from datasets import Dataset
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
Reference in New Issue
Block a user