fix: publish checkpoints atomically

- Write checkpoint payloads to a hidden sibling staging directory, add a versioned checksum manifest, fsync the completed payload, and publish it with an atomic rename
- Republishing an existing step retires the old payload under a hidden sibling name before the atomic rename, so re-runs into the same output directory replace the previous checkpoint instead of raising FileExistsError
- Keep legacy checkpoints loadable, add optional checksum verification, and align metric flushing with checkpoint publication

Co-authored-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>

tests/trainer/test_callbacks.py

@@ -170,3 +170,7 @@ def test_checkpoint_captures_completed_optimizer_step(
     )
     assert checkpoint.extra["optimizer"]["state"]
     assert checkpoint.extra["scheduler"]["last_epoch"] == 1
+    assert checkpoint.meta["optimizer_step"] == 1
+    assert (
+        Path(base_test_env["test_dir"]) / "epoch_0_step_1" / "metric.jsonl"
+    ).is_file()