ViperEkura
9b7e6c205f
feat: add moe auxloss and metrics
2026-08-05 18:12:28 +08:00
ViperEkura
1c7369f293
feat: add MoE auxiliary loss metrics
...
- Propagates MoE load-balancing loss through model outputs
- Logs task, auxiliary, and weighted losses across strategies
- Computes only explicitly requested callback metrics
- Preserves tensor compute_loss API and adds regression tests
2026-08-02 06:30:43 +08:00
ViperEkura
d7db37a70f
fix: preserve MoE routing defaults
2026-08-02 05:30:26 +08:00
Gaolingx
6d98bb4f9f
20260801-moe model impl
...
need to add aux loss for load balancing
2026-08-01 22:48:58 +08:00
ViperEkura
5ba21f4eb3
refactor: eliminate test duplication via shared helpers
...
- Add tests/helpers.py with shared config, dataset, tokenizer, executor, and assertion helpers
- Replace 15 copies of device one-liner with session-scoped fixture
- Collapse 5 near-identical Dataset subclasses into RandomTokenDataset
- Remove duplicate _make_config/_make_model/_make_frozen and FakeTokenizer/FakeExecutor definitions
- Make test_callbacks and test_early_stopping use existing train_config_factory
- Replace 6 duplicate meta.json read blocks with load_shard_meta
- Fix mkdtemp leaks in test_lora.py with TemporaryDirectory
2026-07-27 22:34:53 +08:00
ViperEkura
0c86c89af4
refactor : align config field names with Hugging Face
...
- dim -> hidden_size, n_layers -> num_hidden_layers
- dim_ffn -> intermediate_size, n_heads -> num_attention_heads
- n_kv_heads -> num_key_value_heads, max_len -> max_position_embeddings
- norm_eps -> rms_norm_eps, tie_weight -> tie_word_embeddings
- update model, inference, training, scripts, tests, docs
2026-07-20 22:05:31 +08:00
ViperEkura
97c7ac0f4f
refactor: Transformer更名为AutoRegressiveLM并新增EmbeddingEncoder
...
- AutoRegressiveLM 注册名改为 autoregressive_lm
- 新增 EmbeddingEncoder 支持 mean/cls/last pooling
- ModelConfig 增加 pooling_type / normalize_embeddings 字段
- 导入、注释、测试全部同步更新
2026-05-17 15:29:20 +08:00
ViperEkura
a44fd22a99
fix: 修复训练与模型参数传递问题
...
- state_dict_fn 传入 CheckpointCallback，修复多卡 DDP 下 key 前缀丢失
- MLA 增加 use_qk_norm 支持，消除参数静默丢失
- moe_topk_method 统一命名为 topk_method
- checkpoint 回调移至最前
2026-05-17 11:20:13 +08:00
ViperEkura
0ba8c70ce1
fix: 修复 MLA 多个 bug 并缩小测试模型参数
...
- MLA kv_b_proj 输出维度和 q_rope 切分偏移修复
- 打通 MLA 配置从 ModelConfig 到 DecoderBlock 的传递路径
- rope_theta 配置不再被忽略，MLA 使用 qk_rope_head_dim
- tie_weight 使用 is True 避免 None 隐式生效
- norm_eps/rope base 类型标注修正
- 测试模型参数缩小 (dim=8, head_dim=4)
- 新增 6 种架构配置 × 2 场景的前向传播测试
2026-05-16 14:57:43 +08:00
ViperEkura
Gaolingx