Make Nora+NAdamW the default optimizer

2026-07-31 23:16:39 +08:00
parent 7aa5ed09d9
commit 04899a2b15
11 changed files with 1010 additions and 105 deletions
@@ -107,7 +107,9 @@ nohup python scripts/tools/train.py \
    --batch_per_device=4 \
    --grad_accum_steps=8 \
    --warmup_ratio=0.05 \
+    --optimizer=nora_nadamw \
    --max_lr=1e-4 \
+    --nora_lr=5e-3 \
    --max_grad_norm=1.0 \
    --weight_decay=0.1 \
    --window_size=2048 \
@@ -262,4 +264,4 @@ SSE 流式格式、错误码和统计端点详见[推理文档](guides/inference

 <div align="center">
  <em>专为高性能与易用性设计的轻量级 Transformer 框架。</em>
-</div>
+</div>