ViperEkura
/
AstrAI
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
464 Commits 1 Branch 9 Tags 4.1 MiB
Commit Graph

15 Commits

Author SHA1 Message Date
ViperEkura db9b39b084 fix: resolve IFD token-set asymmetry and support single-token answers 
- Sentinel-anchored unconditional pass: both branches now predict the same N response tokens
- Single-token responses (rl=1) fully supported
- ctx_len tracked per sample; skip_reason replaces silent None
- --per_token flag for per-token IFD breakdown
 2026-07-05 17:48:26 +08:00
ViperEkura 599a51f4f7 fix: reliable test timeout, separate generate/test phases, dynamic pass@k 
- Replace SIGALRM+exec() with subprocess.run(timeout=) for test execution
- Add --test_only flag to skip generation and test existing completions
- Add --generate_only flag for generation-only runs
- Derive pass@k values from num_samples (filter k > n)
- Support loading completions from array JSON (not just JSONL)
 2026-07-05 08:47:30 +08:00
ViperEkura 17d6eaa2f2 refactor: rewrite humaneval evaluation with functional pipeline design 
- fix KeyError race condition in inference cache touch()
- EvalConfig dataclass for centralized configuration
- load->generate->extract->test->score->report pipeline
- two-phase generation+testing for max GPU utilization
- signal-based SIGALRM timeout protection for code exec
- suppress subprocess stdout/stderr pollution
 2026-07-05 07:58:28 +08:00
ViperEkura 2d908639e9 feat : add ROUGE evaluation script (manual impl, no deps) 
- ROUGE-1/2 via n-gram overlap (Counter)
- ROUGE-L via LCS (DP)
- CLI: python scripts/eval/evaluate_rouge.py --data_path ... --output ...
- Library: compute_rouge(ref, cand) -> dict of precision/recall/f1
 2026-07-05 01:15:01 +08:00
ViperEkura c7158418dd perf: add BFD bin-packing and custom attention mask to IFD batch scoring 2026-07-04 18:58:13 +08:00
ViperEkura 4d3c9341c1 refactor: rewrite IFD evaluation with clean three-layer architecture 2026-07-04 18:33:51 +08:00
ViperEkura 1adca39cd8 fix: handle long sequences and optimize IFD computation 2026-07-04 08:35:45 +08:00
ViperEkura 204873fa2f fix: handle long sequences and optimize IFD computation 2026-07-04 08:23:32 +08:00
ViperEkura 84d4769163 feat: SVD 有效秩/权重统计分析脚本 2026-06-29 21:39:22 +08:00
ViperEkura b4587c5d08 refactor : metric_logger 改用事件类型 (type=step/validation/epoch) 
- 每种事件独立 schema，不再混入 null 字段
- 回调顺序 validation 移到 metric_logger 之前，确保 on_optimizer_step 先跑
- 用内部 _last_val_loss 代替 TrainContext.last_val_iter 判断新验证
- 修复 factory.py 未使用导入、evaluate_ifeval.py 多余 f 前缀
 2026-06-25 17:18:20 +08:00
ViperEkura 7348bac6ab fix: 规范 generate.py 命令行接口 
- generate.py 清理描述文字，help 统一标注默认值
- max_tokens 默认改为 None，回退 model config max_len
- evaluate_ppl.py 同步清理描述文字
- params.md 同步 max_tokens 默认值
 2026-06-19 14:03:02 +08:00
ViperEkura 376e9eba80 feat: IFEval 使用 chat template 格式化 prompt，添加 model.eval() 
- generate_one 用 tokenizer.apply_chat_template 包 user 消息
- 新增 model.eval() 关闭 dropout，确保确定性输出
 2026-06-18 16:45:16 +08:00
ViperEkura a62c2e11a2 feat: IFD 默认使用 chat template，支持裸文本模式 
- 新增 _compute_ifd_with_template，用 tokenizer chat template 格式化后计算 IFD
- 默认开启 chat template，可通过 --no_chat_template 切换回裸拼接
- chat template 缺失时给出 RuntimeError 提示
 2026-06-18 16:35:05 +08:00
ViperEkura 1818d06576 feat: 新增 IFD 数据质量评分工具, 移动 ppl 至 eval 
- 计算指令遵循难度分数用于数据筛选
- IFD = 条件交叉熵 / 无条件交叉熵
- perplexity 移至 scripts/eval/
 2026-06-16 22:03:45 +08:00
ViperEkura 4e8d1ee24e feat: 新增 IFEval 指令遵循评测 
- 实现 25 种正则约束 verifier
- 将评测脚本从 scripts/tools/ 移至 scripts/eval/
 2026-06-16 21:57:34 +08:00