refactor: 改用递归子模块 init 替代统一 normal_(0.006)

- Embedding.reset_parameters: normal_(std=0.02)
- Linear.reset_parameters: kaiming_uniform_ + uniform_ bias
- Transformer._init_weights 通过 apply 递归调用子模块 reset_parameters
- 移除全局 normal_(0.006) 覆盖,各模块使用更合适的分布
This commit is contained in:
2026-05-17 10:44:18 +08:00
parent ad9f4d9cf6
commit 1d54491809
3 changed files with 14 additions and 5 deletions
+4 -5
View File
@@ -93,12 +93,11 @@ class Transformer(AutoModel):
if self.config.tie_weight is True:
self.lm_head.weight = self.embed_tokens.weight
self._init_weights()
self.apply(self._init_weights)
def _init_weights(self):
for param in self.parameters():
if param.dim() > 1:
nn.init.normal_(param, mean=0.0, std=0.006)
def _init_weights(self, module):
if hasattr(module, "reset_parameters"):
module.reset_parameters()
def load_state_dict(self, state_dict: Mapping[str, Any], strict=True, assign=False):
lm_head_key = "lm_head.weight"