feat: add MoE auxiliary loss metrics

- Propagates MoE load-balancing loss through model outputs
- Logs task, auxiliary, and weighted losses across strategies
- Computes only explicitly requested callback metrics
- Preserves tensor compute_loss API and adds regression tests
This commit is contained in:
2026-08-02 06:30:43 +08:00
parent 0fc1b1bd46
commit 1c7369f293
14 changed files with 370 additions and 68 deletions
+1 -1
View File
@@ -70,7 +70,7 @@ class EmbeddingEncoder(AutoModel):
attn_mask = process_attention_mask(input_mask)
for layer in self.layers:
x = layer(x, rotary_emb, attn_mask)
x = layer(x, rotary_emb, attn_mask)["hidden_states"]
hidden_states = self.norm(x)