ViperEkura b12b24eadc feat: rewrite evaluate_ppl with token-level loss and multi-file support ... - Support multiple input files, glob patterns, and directory input - Add --token_level flag: per-record token_ids + log_probs JSONL output - Add --max_samples for random subsampling per file - LossAccumulator: streaming mode (histogram-based percentiles, low memory) vs exact mode (full token list) - Token type analysis (ascii/cjk/non_ascii/special) when token_level=True - Fix token_ids/log_probs alignment (shift offset) - Cache frozenset(stop_ids) outside loop for performance - Aggregate stats: mean/median/ppl/p50/p90/p95/p99 - Summary JSON with all datasets in one file		2026-07-17 13:14:15 +08:00
..
demo	feat: add model_path temperature top_p top_k max_tokens system_prompt args to stream_chat	2026-07-04 07:33:32 +08:00
eval	feat: rewrite evaluate_ppl with token-level loss and multi-file support	2026-07-17 13:14:15 +08:00
tools	feat: add --resume flag to decouple weight loading from training resumption	2026-07-16 14:23:23 +08:00
docker.sh	fix: docker-compose UID/GID 添加默认值，修复 docker.sh logs 命令	2026-05-18 14:24:00 +08:00
pre_commit.sh	ci: 优化 GitHub Actions 工作流	2026-04-05 22:40:16 +08:00