ViperEkura
398e8a3ea3
refactor: deduplicate low-risk code paths
2026-08-19 16:17:40 +08:00
ViperEkura
3406157431
refactor: standardize packed 3d inference
...
- keep training attention on dense 4d tensors
- use packed 3d tensors with KV cache for inference
- extend CUDA rotary embedding to packed 3d inputs
- adapt torch, CUDA and FlashAttention backend dispatch
2026-08-16 13:24:02 +08:00
ViperEkura
a317a4756b
refactor: stateless MoE routing with grouped dispatch
...
- replace per-expert mask scan with sort+bincount grouped dispatch
- carry router stats in forward output instead of module state
- keep MoE diagnostics working under DDP/FSDP wrappers
- remove unused _load_balancing_loss helper
2026-08-05 18:42:12 +08:00
ViperEkura
9b7e6c205f
feat: add moe auxloss and metrics
2026-08-05 18:12:28 +08:00
ViperEkura
1c7369f293
feat: add MoE auxiliary loss metrics
...
- Propagates MoE load-balancing loss through model outputs
- Logs task, auxiliary, and weighted losses across strategies
- Computes only explicitly requested callback metrics
- Preserves tensor compute_loss API and adds regression tests
2026-08-02 06:30:43 +08:00
ViperEkura
d7db37a70f
fix: preserve MoE routing defaults
2026-08-02 05:30:26 +08:00
Gaolingx
6d98bb4f9f
20260801-moe model impl
...
need to add aux loss for load balancing
2026-08-01 22:48:58 +08:00
ViperEkura
88ec63121d
feat : GPT-2 residual scaling weight init
...
- Linear: normal(0, init_std) replaces kaiming_uniform_(a=sqrt(5))
- o_proj / mlp.down: init_std = 0.02 / sqrt(2 * n_layers)
- MoE: expert down scaled by 1/sqrt(1/n_shared + 1/K)
- Embedding: normal(0, 0.02), unchanged
2026-06-25 15:08:31 +08:00
ViperEkura
6ae1828449
refactor : 清理工厂和配置系统中的死代码与冗余抽象
...
- 删除 Registry 中未使用的 category/priority 字段，_entries 简化为直接存储类引用
- 修正 __init_subclass__ 避免叶子类（AutoRegressiveLM 等）创建空注册表
- 删除 5 个工厂的薄 create() 覆写，统一使用 BaseFactory.create(name, *args, **kwargs)
- 删除 3 处零调用的 available_types/available_strategies 别名死代码
- 删除零调用的 BaseModelConfig.to_file 死代码
- 将 BaseConfig.from_json/to_json 重命名为 from_file/to_file，消除与子类重复
- 移除两个 inference builder 中总是被覆写的 prompt_tokens=0
2026-06-07 11:39:50 +08:00
ViperEkura
48a53121ba
refactor: 工厂 kwargs 过滤及组件参数清理
...
- BaseFactory.create() 按 __init__ 签名过滤多余 kwargs
- 移除 GQA/MLA/MLP/DeepSeekMoE 中多余的 **kwargs
- MLP/DeepSeekMoE 参数名统一为 dim_ffn
- scheduler max_seq_len 增加 None 显式判断
- 默认 max_prompt_len 提升至 2048
2026-05-16 16:47:41 +08:00
ViperEkura
e12f1a7ee5
feat: BaseModelConfig + DeepSeekMoE + 工厂模式替代 if/else
...
- BaseModelConfig: fields() 精确字段匹配 + 类型矫正 + 未知key警告
- DeepSeekMoE: 共享专家 + 路由专家 + top-K 门控
- AttnFactory/FFNFactory: 装饰器注册，DecoderBlock 零分支
- config 用 attn_type/ffn_type 驱动组件选择
2026-05-15 20:34:52 +08:00
ViperEkura
ef25efffa2
refactor: 拆分 module.py 为 components 子包
...
- rope/linear/norm/embedding/mlp/attention/decoder_block 各自独立文件
- 依赖单向无循环
- 公开接口不变，外部无需修改
2026-05-15 20:08:36 +08:00
ViperEkura
Gaolingx