ViperEkura and 0z5a
1fad50d847
fix: publish checkpoints atomically
...
- Write checkpoint payloads to a hidden sibling staging directory, add a versioned checksum manifest, fsync the completed payload, and publish it with an atomic rename
- Republishing an existing step retires the old payload under a hidden sibling name before the atomic rename, so re-runs into the same output directory replace the previous checkpoint instead of raising FileExistsError
- Keep legacy checkpoints loadable, add optional checksum verification, and align metric flushing with checkpoint publication
Co-authored-by: 0z5a <dezhen.lu@student.uni-tuebingen.de >
2026-09-02 15:29:22 +08:00
ViperEkura
aabb0d83e9
refactor : replace iteration with consumed_samples
...
- Replace context.iteration with consumed_samples (global sample count)
- Add optimizer_step property derived from consumed_samples
- Checkpoint meta.json stores consumed_samples, drops iteration
- CLI --start_batch renamed to --start_samples (per-rank samples)
- Checkpoint dir naming: epoch_X_step_Y instead of epoch_X_iter_Y
- Metric log entries use step and consumed_samples fields
- Backward compat removed (old iteration checkpoints unsupported)
2026-06-30 18:42:42 +08:00
ViperEkura
65ab69543b
refactor: 统一序列化层，消除分散的 I/O 路径
...
- Checkpoint 改为 @dataclass，内聚 save/load 方法
- 提取 save_safetensors/load_safetensors/save_json/load_json 共享工具
- 新增 save_model/load_model_config/load_model_weights 模块函数
- automodel 和 lora 统一委托到 serialization 模块
2026-05-26 16:44:40 +08:00
ViperEkura
7dea929788
refactor: checkpoint 按 HF 方式存独立 .pt 文件，callback 接管恢复
...
- Checkpoint.save/load: extra 逐 key 写为 {key}.pt 而非单个 extra.pt
- meta.json 新增 timestamp
- CheckpointCallback: save_extra/load_extra 静态方法 + extra_keys 类属性
- on_train_begin 接管 optimizer/scheduler 恢复，TrainContextBuilder 不再传 load_extra_fn
2026-05-16 18:29:04 +08:00
ViperEkura
39766aa1dc
chore: 修改类名，优化导入顺序
2026-04-05 22:27:57 +08:00
ViperEkura
bd9741dc5f
refactor: 从data 模块分离tokenizer
2026-04-04 16:12:58 +08:00
ViperEkura
b531232a9b
style: 修改为显式导入
2026-04-04 16:02:49 +08:00
ViperEkura
0852b852f8
refactor: 优化参数传递，清理导入样式
2026-04-03 22:06:32 +08:00
ViperEkura
2e009cf59a
chore: 更新项目名称
2026-03-31 09:34:11 +08:00
ViperEkura
426af2d75f
style: 使用ruff 工具优化代码风格
2026-03-30 23:32:28 +08:00
ViperEkura
3e8f2eba81
fix: 修复路径问题
2026-03-20 21:14:02 +08:00
ViperEkura
5713b55500
refactor: 修改 StepMonitorCallback, 分离职责
2026-03-04 19:45:39 +08:00
ViperEkura
8a8d6369bc
fix: 修复 dataset 和 checkpoint 的 bug
2026-03-02 11:12:21 +08:00
ViperEkura
9dab96c31f
test(checkpoint): 添加多进程检查点测试功能
2026-01-08 22:04:39 +08:00
ViperEkura
4da70785b5
refactor(tests): 重构测试文件目录结构
2026-01-08 21:34:52 +08:00
ViperEkuraand0z5a