AstrAI

Commit Graph

Author	SHA1	Message	Date
ViperEkura	b12b24eadc	feat: rewrite evaluate_ppl with token-level loss and multi-file support - Support multiple input files, glob patterns, and directory input - Add --token_level flag: per-record token_ids + log_probs JSONL output - Add --max_samples for random subsampling per file - LossAccumulator: streaming mode (histogram-based percentiles, low memory) vs exact mode (full token list) - Token type analysis (ascii/cjk/non_ascii/special) when token_level=True - Fix token_ids/log_probs alignment (shift offset) - Cache frozenset(stop_ids) outside loop for performance - Aggregate stats: mean/median/ppl/p50/p90/p95/p99 - Summary JSON with all datasets in one file	2026-07-17 13:14:15 +08:00
ViperEkura	7348bac6ab	fix: 规范 generate.py 命令行接口 - generate.py 清理描述文字，help 统一标注默认值 - max_tokens 默认改为 None，回退 model config max_len - evaluate_ppl.py 同步清理描述文字 - params.md 同步 max_tokens 默认值	2026-06-19 14:03:02 +08:00
ViperEkura	1818d06576	feat: 新增 IFD 数据质量评分工具, 移动 ppl 至 eval - 计算指令遵循难度分数用于数据筛选 - IFD = 条件交叉熵 / 无条件交叉熵 - perplexity 移至 scripts/eval/	2026-06-16 22:03:45 +08:00

Author

SHA1

Message

Date

ViperEkura

b12b24eadc

feat: rewrite evaluate_ppl with token-level loss and multi-file support

- Support multiple input files, glob patterns, and directory input
- Add --token_level flag: per-record token_ids + log_probs JSONL output
- Add --max_samples for random subsampling per file
- LossAccumulator: streaming mode (histogram-based percentiles, low memory) vs exact mode (full token list)
- Token type analysis (ascii/cjk/non_ascii/special) when token_level=True
- Fix token_ids/log_probs alignment (shift offset)
- Cache frozenset(stop_ids) outside loop for performance
- Aggregate stats: mean/median/ppl/p50/p90/p95/p99
- Summary JSON with all datasets in one file

2026-07-17 13:14:15 +08:00

ViperEkura

7348bac6ab

fix: 规范 generate.py 命令行接口

- generate.py 清理描述文字，help 统一标注默认值
- max_tokens 默认改为 None，回退 model config max_len
- evaluate_ppl.py 同步清理描述文字
- params.md 同步 max_tokens 默认值

2026-06-19 14:03:02 +08:00

ViperEkura

1818d06576

feat: 新增 IFD 数据质量评分工具, 移动 ppl 至 eval

- 计算指令遵循难度分数用于数据筛选
- IFD = 条件交叉熵 / 无条件交叉熵
- perplexity 移至 scripts/eval/

2026-06-16 22:03:45 +08:00

3 Commits