fix : 修复存储层 bug,JSON 切换为 JSONL,补齐测试覆盖
- save_bin/load_bin: save_json/load_json 替换为直接 json.dump/json.load,修复致命 bug - _normalize: 空 cum 列表 guard,防止 IndexError - load_json: 改为仅支持 JSONL 逐行解析 (json.loads),移除 .json 支持 - detect_format: 只匹配 *.jsonl,不再匹配 *.json - save_json: 输出扩展名改为 .jsonl - GRPODataset.__getitem__: 补齐 .to(dtype=torch.long/bool) 与其他数据集一致 - load_bin: np.memmap mode='r+' 消除 PyTorch 不可写 tensor 警告 - 新增 16 个测试: bin roundtrip, mmap load, 空 key, JSONL 多行/文本, GRPO dtype/load, detect_format bin/jsonl, fetch multi-key/越界, json_to_bin 转换, DPO from JSONL, 显式 storage_type
This commit is contained in:
@@ -306,9 +306,11 @@ class GRPODataset(BaseDataset):
|
||||
def __getitem__(self, index: int) -> Dict[str, Tensor]:
|
||||
begin_idx, end_idx = self.get_index(index)
|
||||
|
||||
prompts = self._fetch_data(begin_idx, end_idx, "prompts")
|
||||
responses = self._fetch_data(begin_idx, end_idx, "responses")
|
||||
masks = self._fetch_data(begin_idx, end_idx, "masks")
|
||||
prompts = self._fetch_data(begin_idx, end_idx, "prompts").to(dtype=torch.long)
|
||||
responses = self._fetch_data(begin_idx, end_idx, "responses").to(
|
||||
dtype=torch.long
|
||||
)
|
||||
masks = self._fetch_data(begin_idx, end_idx, "masks").to(dtype=torch.bool)
|
||||
rewards = self._fetch_data(begin_idx, end_idx, "rewards")
|
||||
|
||||
return {
|
||||
|
||||
Reference in New Issue
Block a user