feat: add MoE auxiliary loss metrics
- Propagates MoE load-balancing loss through model outputs - Logs task, auxiliary, and weighted losses across strategies - Computes only explicitly requested callback metrics - Preserves tensor compute_loss API and adds regression tests
This commit is contained in:
@@ -70,7 +70,7 @@ class EmbeddingEncoder(AutoModel):
|
||||
attn_mask = process_attention_mask(input_mask)
|
||||
|
||||
for layer in self.layers:
|
||||
x = layer(x, rotary_emb, attn_mask)
|
||||
x = layer(x, rotary_emb, attn_mask)["hidden_states"]
|
||||
|
||||
hidden_states = self.norm(x)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user