Fix lingering cosine->WSD in Conclusion; update AGENTS.md with checkpoint labels and scheduler rules
This commit is contained in:
@@ -555,7 +555,7 @@ spread.}
|
||||
We have described the end-to-end pipeline for training a 1.2B Transformer with
|
||||
{\sc AstrAI}: data preprocessing with JSON-driven tokenization and packing,
|
||||
a 24-layer GQA-SwiGLU architecture, callback-based training with a hybrid
|
||||
Muon/AdamW optimizer under DDP/FSDP executors, and cosine scheduling. We
|
||||
Muon/AdamW optimizer under DDP/FSDP executors, and WSD scheduling. We
|
||||
further analyzed numerical stability under BF16, showing that GPT-2 residual
|
||||
scaling ($\sigma_o = 0.02/\sqrt{2L}$) reduces per-block residual variance
|
||||
by a factor of 48, keeping post-24-layer variance at $1.34$ versus $17.5$
|
||||
|
||||
Reference in New Issue
Block a user