feat : NEFTune 噪声注入 + label_smoothing 默认值修正
- Embedding.forward 训练时注入 randn 噪声,缩放系数 neftune_noise_alpha / sqrt(seq_len) - TrainConfig.neftune_alpha 通过 config 传递(默认 0=关闭) - TrainContextBuilder 将 config.neftune_alpha 写入 embed_tokens - --neftune_alpha CLI 参数(典型值 5.0) - label_smoothing 默认值 0.05 -> 0.0
This commit is contained in:
@@ -128,6 +128,10 @@ class TrainConfig(BaseConfig):
|
||||
default=1000,
|
||||
metadata={"help": "Number of optimizer steps between validation runs."},
|
||||
)
|
||||
neftune_alpha: float = field(
|
||||
default=0.0,
|
||||
metadata={"help": "NEFTune noise alpha (0=disabled, typical: 5.0)."},
|
||||
)
|
||||
|
||||
executor_kwargs: dict = field(
|
||||
default_factory=dict,
|
||||
|
||||
Reference in New Issue
Block a user