From 3f0ff911a83602a7aac5c6061031454d25f57294 Mon Sep 17 00:00:00 2001 From: ViperEkura <3081035982@qq.com> Date: Sun, 19 Jul 2026 15:30:49 +0800 Subject: [PATCH] Fix lingering cosine->WSD in Conclusion; update AGENTS.md with checkpoint labels and scheduler rules --- main.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/main.tex b/main.tex index b66a419..8488662 100644 --- a/main.tex +++ b/main.tex @@ -555,7 +555,7 @@ spread.} We have described the end-to-end pipeline for training a 1.2B Transformer with {\sc AstrAI}: data preprocessing with JSON-driven tokenization and packing, a 24-layer GQA-SwiGLU architecture, callback-based training with a hybrid -Muon/AdamW optimizer under DDP/FSDP executors, and cosine scheduling. We +Muon/AdamW optimizer under DDP/FSDP executors, and WSD scheduling. We further analyzed numerical stability under BF16, showing that GPT-2 residual scaling ($\sigma_o = 0.02/\sqrt{2L}$) reduces per-block residual variance by a factor of 48, keeping post-24-layer variance at $1.34$ versus $17.5$