fix: rewrite GRPO data pipeline for offline record-level access

- process_list_field returns List[List[int]] preserving per-response boundaries
- GRPODataset rewritten to record-level __getitem__ (no windowing/stride)
- grpo_collate_fn pads variable-length responses into [B, G, R] tensors
- JsonlStore detects nested List[List[int]] and stores List[Tensor] per record
- Store._normalize skips nested-list keys from cumsum bookkeeping
- Pipeline._flush handles nested lists without cross-record flattening
- Export grpo_collate_fn from astrai.dataset
- 6 new GRPO tests + 2 updated builder tests, 114 total pass
This commit is contained in:
2026-07-17 14:34:41 +08:00
parent c17aa0dc54
commit a1ea26d367
7 changed files with 502 additions and 75 deletions
+18 -3
View File
@@ -180,9 +180,24 @@ class Pipeline:
dt = _STR_TO_DTYPE.get(
self.config.output.dtype.get(key, "int32"), torch.int32
)
tensors[key] = [
torch.tensor(list(chain.from_iterable(ids_list)), dtype=dt)
]
# GRPO multi-response keys store List[List[int]] per record
# (responses/masks). Rewards store List[float] per record.
# Both produce List[Tensor] (one tensor per record), but
# responses need inner flattening while rewards do not.
if ids_list and isinstance(ids_list[0], list):
tensors[key] = [
torch.tensor(
list(chain.from_iterable(ids))
if ids and isinstance(ids[0], list)
else ids,
dtype=dt,
)
for ids in ids_list
]
else:
tensors[key] = [
torch.tensor(list(chain.from_iterable(ids_list)), dtype=dt)
]
if mode == "continuous" and original_sequences:
pos_ids = self._position_id.generate(keys.get("sequence", []))