AstrAI

Commit Graph

Author	SHA1	Message	Date
ViperEkura	599a51f4f7	fix: reliable test timeout, separate generate/test phases, dynamic pass@k - Replace SIGALRM+exec() with subprocess.run(timeout=) for test execution - Add --test_only flag to skip generation and test existing completions - Add --generate_only flag for generation-only runs - Derive pass@k values from num_samples (filter k > n) - Support loading completions from array JSON (not just JSONL)	2026-07-05 08:47:30 +08:00
ViperEkura	17d6eaa2f2	refactor: rewrite humaneval evaluation with functional pipeline design - fix KeyError race condition in inference cache touch() - EvalConfig dataclass for centralized configuration - load->generate->extract->test->score->report pipeline - two-phase generation+testing for max GPU utilization - signal-based SIGALRM timeout protection for code exec - suppress subprocess stdout/stderr pollution	2026-07-05 07:58:28 +08:00
ViperEkura	4e8d1ee24e	feat: 新增 IFEval 指令遵循评测 - 实现 25 种正则约束 verifier - 将评测脚本从 scripts/tools/ 移至 scripts/eval/	2026-06-16 21:57:34 +08:00

Author

SHA1

Message

Date

ViperEkura

599a51f4f7

fix: reliable test timeout, separate generate/test phases, dynamic pass@k

- Replace SIGALRM+exec() with subprocess.run(timeout=) for test execution
- Add --test_only flag to skip generation and test existing completions
- Add --generate_only flag for generation-only runs
- Derive pass@k values from num_samples (filter k > n)
- Support loading completions from array JSON (not just JSONL)

2026-07-05 08:47:30 +08:00

ViperEkura

17d6eaa2f2

refactor: rewrite humaneval evaluation with functional pipeline design

- fix KeyError race condition in inference cache touch()
- EvalConfig dataclass for centralized configuration
- load->generate->extract->test->score->report pipeline
- two-phase generation+testing for max GPU utilization
- signal-based SIGALRM timeout protection for code exec
- suppress subprocess stdout/stderr pollution

2026-07-05 07:58:28 +08:00

ViperEkura

4e8d1ee24e

feat: 新增 IFEval 指令遵循评测

- 实现 25 种正则约束 verifier
- 将评测脚本从 scripts/tools/ 移至 scripts/eval/

2026-06-16 21:57:34 +08:00

3 Commits