diff --git a/main.tex b/main.tex index b66a419..8488662 100644 --- a/main.tex +++ b/main.tex @@ -555,7 +555,7 @@ spread.} We have described the end-to-end pipeline for training a 1.2B Transformer with {\sc AstrAI}: data preprocessing with JSON-driven tokenization and packing, a 24-layer GQA-SwiGLU architecture, callback-based training with a hybrid -Muon/AdamW optimizer under DDP/FSDP executors, and cosine scheduling. We +Muon/AdamW optimizer under DDP/FSDP executors, and WSD scheduling. We further analyzed numerical stability under BF16, showing that GPT-2 residual scaling ($\sigma_o = 0.02/\sqrt{2L}$) reduces per-block residual variance by a factor of 48, keeping post-24-layer variance at $1.34$ versus $17.5$