ViperEkura/AstrAI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: publish checkpoints atomically

Browse Source 
- Write checkpoint payloads to a hidden sibling staging directory, add a versioned checksum manifest, fsync the completed payload, and publish it with an atomic rename
- Republishing an existing step retires the old payload under a hidden sibling name before the atomic rename, so re-runs into the same output directory replace the previous checkpoint instead of raising FileExistsError
- Keep legacy checkpoints loadable, add optional checksum verification, and align metric flushing with checkpoint publication

Co-authored-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
This commit is contained in:
ViperEkura
2026-09-02 15:29:22 +08:00
committed by 0z5a
co-authored by 0z5a
parent 01bcd0d105
commit 1fad50d847
8 changed files with 292 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/developer/docker-training.md
+6
View File
@@ -183,8 +183,14 @@ config.json
model.safetensors
optimizer.pt
scheduler.pt
manifest.json
```
New checkpoints write `manifest.json` after every payload file, sync the complete
staging directory, and then atomically rename that directory into place. Legacy
checkpoints without a manifest remain resumable when the original required files
are complete.
`start` resumes the latest complete checkpoint and ignores partial writes. If no
complete checkpoint exists, `/models/base/config.json` and
`/models/base/model.safetensors` are required. `stop` sends `SIGTERM`; the