ViperEkura/AstrAI-paper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix lingering cosine->WSD in Conclusion; update AGENTS.md with checkpoint labels and scheduler rules

Browse Source
This commit is contained in:
ViperEkura
2026-07-19 15:30:49 +08:00
parent e149997200
commit 3f0ff911a8
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
main.tex
+1 -1
View File
@@ -555,7 +555,7 @@ spread.}
We have described the end-to-end pipeline for training a 1.2B Transformer with We have described the end-to-end pipeline for training a 1.2B Transformer with
{\sc AstrAI}: data preprocessing with JSON-driven tokenization and packing, {\sc AstrAI}: data preprocessing with JSON-driven tokenization and packing,
a 24-layer GQA-SwiGLU architecture, callback-based training with a hybrid a 24-layer GQA-SwiGLU architecture, callback-based training with a hybrid
Muon/AdamW optimizer under DDP/FSDP executors, and cosine scheduling. We Muon/AdamW optimizer under DDP/FSDP executors, and WSD scheduling. We
further analyzed numerical stability under BF16, showing that GPT-2 residual further analyzed numerical stability under BF16, showing that GPT-2 residual
scaling ($\sigma_o = 0.02/\sqrt{2L}$) reduces per-block residual variance scaling ($\sigma_o = 0.02/\sqrt{2L}$) reduces per-block residual variance
by a factor of 48, keeping post-24-layer variance at $1.34$ versus $17.5$ by a factor of 48, keeping post-24-layer variance at $1.34$ versus $17.5$