AstrAI/astrai/preprocessing
ViperEkura a1ea26d367 fix: rewrite GRPO data pipeline for offline record-level access
- process_list_field returns List[List[int]] preserving per-response boundaries
- GRPODataset rewritten to record-level __getitem__ (no windowing/stride)
- grpo_collate_fn pads variable-length responses into [B, G, R] tensors
- JsonlStore detects nested List[List[int]] and stores List[Tensor] per record
- Store._normalize skips nested-list keys from cumsum bookkeeping
- Pipeline._flush handles nested lists without cross-record flattening
- Export grpo_collate_fn from astrai.dataset
- 6 new GRPO tests + 2 updated builder tests, 114 total pass
2026-07-17 14:34:41 +08:00
..
__init__.py refactor: split mask builder by single/multi output 2026-07-08 21:18:34 +08:00
builder.py fix: rewrite GRPO data pipeline for offline record-level access 2026-07-17 14:34:41 +08:00
packing.py feat: implement bfd_split packing strategy 2026-07-17 12:38:56 +08:00
pipeline.py fix: rewrite GRPO data pipeline for offline record-level access 2026-07-17 14:34:41 +08:00
position_id.py fix : 修复策略相关文件的类型注解与抽象方法体 2026-06-14 16:20:10 +08:00
writer.py feat: add JSONL dataset store with on-the-fly tokenization 2026-07-04 15:42:33 +08:00