ViperEkura 88ec63121d feat : GPT-2 residual scaling weight init ...

- Linear: normal(0, init_std) replaces kaiming_uniform_(a=sqrt(5))
- o_proj / mlp.down: init_std = 0.02 / sqrt(2 * n_layers)
- MoE: expert down scaled by 1/sqrt(1/n_shared + 1/K)
- Embedding: normal(0, 0.02), unchanged

2026-06-25 15:08:31 +08:00

..

__init__.py

refactor: 拆分 module.py 为 components 子包

2026-05-15 20:08:36 +08:00

attention.py

feat : GPT-2 residual scaling weight init

2026-06-25 15:08:31 +08:00

decoder_block.py

feat : GPT-2 residual scaling weight init

2026-06-25 15:08:31 +08:00

embedding.py

refactor : neftune_alpha 在 Embedding 构造时传入，由模型配置链路负责

2026-06-19 14:23:27 +08:00

linear.py

feat : GPT-2 residual scaling weight init

2026-06-25 15:08:31 +08:00

lora.py

refactor: 统一序列化层，消除分散的 I/O 路径

2026-05-26 16:44:40 +08:00

mlp.py

feat : GPT-2 residual scaling weight init

2026-06-25 15:08:31 +08:00

norm.py

refactor: 拆分 module.py 为 components 子包

2026-05-15 20:08:36 +08:00

rope.py

feat: 新增NTK-Aware RoPE缩放支持

2026-05-25 21:22:07 +08:00