refactor : neftune_alpha 在 Embedding 构造时传入,由模型配置链路负责

- BaseModelConfig 添加 neftune_alpha 字段 (默认 0.0)
- Embedding.__init__ 接受 neftune_alpha 参数,不再外部 set
- AutoRegressiveLM / EmbeddingEncoder 从 config 传入 neftune_alpha
- train.py 将 CLI 参数注入 config 后再创建模型
- TrainContextBuilder 移除 neftune 设置(不再是其职责)
This commit is contained in:
2026-06-19 14:23:27 +08:00
parent b1adc40cfb
commit 39985840c7
6 changed files with 13 additions and 5 deletions
+5 -2
View File
@@ -7,10 +7,13 @@ from torch import Tensor
class Embedding(nn.Module):
def __init__(self, vocab_size: int, embedding_dim: int):
def __init__(self, vocab_size: int, embedding_dim: int, neftune_alpha: float = 0.0):
super().__init__()
self.weight = nn.Parameter(torch.empty((vocab_size, embedding_dim)))
self.neftune_noise_alpha = 0.0
self.neftune_noise_alpha = neftune_alpha
def set_neftune_alpha(self, alpha: float):
self.neftune_noise_alpha = alpha
def reset_parameters(self):
nn.init.normal_(self.weight, mean=0.0, std=0.02)