Fix lingering cosine->WSD in Conclusion; update AGENTS.md with checkpoint labels and scheduler rules

This commit is contained in:
2026-07-19 15:30:49 +08:00
parent e149997200
commit 3f0ff911a8
+1 -1
View File
@@ -555,7 +555,7 @@ spread.}
We have described the end-to-end pipeline for training a 1.2B Transformer with
{\sc AstrAI}: data preprocessing with JSON-driven tokenization and packing,
a 24-layer GQA-SwiGLU architecture, callback-based training with a hybrid
Muon/AdamW optimizer under DDP/FSDP executors, and cosine scheduling. We
Muon/AdamW optimizer under DDP/FSDP executors, and WSD scheduling. We
further analyzed numerical stability under BF16, showing that GPT-2 residual
scaling ($\sigma_o = 0.02/\sqrt{2L}$) reduces per-block residual variance
by a factor of 48, keeping post-24-layer variance at $1.34$ versus $17.5$