Commit Graph

4 Commits

Author SHA1 Message Date
ViperEkura c17aa0dc54 fix: eval script bugs and add missing features
- evaluate_mmlu: fix double few-shot injection (build_prompt no longer
  adds few-shot, apply_chat handles it once)
- evaluate_humaneval: fix pass@k k-filtering to be per-problem instead
  of using first problem's n globally; reuse ProcessPoolExecutor across
  problems; fix closure UnboundLocalError in test_one; handle None in
  report when k > n
- evaluate_ifd: remove dead code (score_plain/score_messages); add
  multi-file/directory input support with --input_path/--output_dir;
  add summary.json aggregation and --max_samples; add --dtype flag
- evaluate_ppl: add --device and --dtype flags (was hardcoded to cuda)
- evaluate_ifeval: fix docstring path (scripts/tools -> scripts/eval)
- analyze_weights: add --output JSON export; fix dead code filter
  ("_norm" not in r was always True)
2026-07-17 14:02:58 +08:00
ViperEkura b4587c5d08 refactor : metric_logger 改用事件类型 (type=step/validation/epoch)
- 每种事件独立 schema,不再混入 null 字段
- 回调顺序 validation 移到 metric_logger 之前,确保 on_optimizer_step 先跑
- 用内部 _last_val_loss 代替 TrainContext.last_val_iter 判断新验证
- 修复 factory.py 未使用导入、evaluate_ifeval.py 多余 f 前缀
2026-06-25 17:18:20 +08:00
ViperEkura 376e9eba80 feat: IFEval 使用 chat template 格式化 prompt,添加 model.eval()
- generate_one 用 tokenizer.apply_chat_template 包 user 消息
- 新增 model.eval() 关闭 dropout,确保确定性输出
2026-06-18 16:45:16 +08:00
ViperEkura 4e8d1ee24e feat: 新增 IFEval 指令遵循评测
- 实现 25 种正则约束 verifier
- 将评测脚本从 scripts/tools/ 移至 scripts/eval/
2026-06-16 21:57:34 +08:00