feat : NEFTune 噪声注入 + label_smoothing 默认值修正

- Embedding.forward 训练时注入 randn 噪声，缩放系数 neftune_noise_alpha / sqrt(seq_len) - TrainConfig.neftune_alpha 通过 config 传递（默认 0=关闭） - TrainContextBuilder 将 config.neftune_alpha 写入 embed_tokens - --neftune_alpha CLI 参数（典型值 5.0） - label_smoothing 默认值 0.05 -> 0.0
2026-06-11 15:32:43 +08:00
parent 6ae1828449
commit 445378667f
4 changed files with 22 additions and 2 deletions
@@ -63,6 +63,7 @@ class TrainContextBuilder:

        model = cfg.model_fn()
        model = model.to(device=device)
+        model.embed_tokens.neftune_noise_alpha = cfg.neftune_alpha

        model_config = {}
        if self._resume_dir: