ViperEkura b33250dc28 refactor: decouple tokenizer from Store into Transform layer ...

- Extract tokenization/mask/position logic from JsonlStore into TokenizeTransform
- JsonlStore now pure reader: reads JSON records, delegates to transform
- Store no longer imports tokenizer or preprocessing components
- Replace per_record param with segments_are_records class attribute
- Store subclasses declare segment semantics as format-level property

2026-07-18 21:37:31 +08:00

..

config

feat: add record-mode to Store for DPO/GRPO

2026-07-18 21:02:29 +08:00

dataset

refactor: decouple tokenizer from Store into Transform layer

2026-07-18 21:37:31 +08:00

extension

refactor: stride-based attn interface with layout and causal mask

2026-07-14 21:34:42 +08:00

inference

fix: incremental decode to avoid U+FFFD in streaming

2026-07-18 13:05:36 +08:00

model

perf: replace paged KV cache with contiguous ContiguousCache, decode all groups

2026-07-05 11:34:36 +08:00

parallel

feat : add distributed checkpoint via executor checkpoint_context

2026-07-13 12:27:09 +08:00

preprocessing

refactor: decouple tokenizer from Store into Transform layer

2026-07-18 21:37:31 +08:00

serialization

feat: add record-mode to Store for DPO/GRPO

2026-07-18 21:02:29 +08:00

tokenize

refactor : Pipeline 去除去重，ids 重命名为 sequence，泛型透传

2026-05-31 15:14:27 +08:00

trainer

feat: add record-mode to Store for DPO/GRPO

2026-07-18 21:02:29 +08:00

__init__.py

feat: add record-mode to Store for DPO/GRPO

2026-07-18 21:02:29 +08:00

factory.py

fix: ForwardRef._evaluate Python 3.12 compatibility

2026-07-03 18:41:19 +08:00

protocols.py

refactor: 重构训练后端为 Executor 模式

2026-05-24 20:35:44 +08:00