feat : NEFTune 噪声注入 + label_smoothing 默认值修正

- Embedding.forward 训练时注入 randn 噪声,缩放系数 neftune_noise_alpha / sqrt(seq_len)
- TrainConfig.neftune_alpha 通过 config 传递(默认 0=关闭)
- TrainContextBuilder 将 config.neftune_alpha 写入 embed_tokens
- --neftune_alpha CLI 参数(典型值 5.0)
- label_smoothing 默认值 0.05 -> 0.0
This commit is contained in:
2026-06-11 15:32:43 +08:00
parent 6ae1828449
commit 445378667f
4 changed files with 22 additions and 2 deletions
+4
View File
@@ -128,6 +128,10 @@ class TrainConfig(BaseConfig):
default=1000,
metadata={"help": "Number of optimizer steps between validation runs."},
)
neftune_alpha: float = field(
default=0.0,
metadata={"help": "NEFTune noise alpha (0=disabled, typical: 5.0)."},
)
executor_kwargs: dict = field(
default_factory=dict,