fix: eval script bugs and add missing features
- evaluate_mmlu: fix double few-shot injection (build_prompt no longer
adds few-shot, apply_chat handles it once)
- evaluate_humaneval: fix pass@k k-filtering to be per-problem instead
of using first problem's n globally; reuse ProcessPoolExecutor across
problems; fix closure UnboundLocalError in test_one; handle None in
report when k > n
- evaluate_ifd: remove dead code (score_plain/score_messages); add
multi-file/directory input support with --input_path/--output_dir;
add summary.json aggregation and --max_samples; add --dtype flag
- evaluate_ppl: add --device and --dtype flags (was hardcoded to cuda)
- evaluate_ifeval: fix docstring path (scripts/tools -> scripts/eval)
- analyze_weights: add --output JSON export; fix dead code filter
("_norm" not in r was always True)
This commit is contained in:
@@ -5,7 +5,7 @@ Supports all IFEval constraint types except language detection.
|
||||
|
||||
Usage::
|
||||
|
||||
python scripts/tools/evaluate_ifeval.py --param_path ./params \
|
||||
python scripts/eval/evaluate_ifeval.py --param_path ./params \
|
||||
--data_path ifeval.jsonl --output results.json \
|
||||
--temperature 0.1 --max_tokens 512
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user