ViperEkura/AstrAI-paper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
34 Commits 1 Branch 0 Tags
e6be33aa5319457568dca762f33482e66ff0ca96
Commit Graph
34 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
ViperEkura e6be33aa53 doc: add SFT length-filter rationale — high PPL + high variance from A.3 IFD figure 
- Add 15-token length floor to SFT samples, with explicit reference
  to Appendix A.3 / Figure 5 (ifd_length_grid)
- Short replies (<10 tokens) show both high per-token perplexity
  (L_uncond ~6-8, PPL ~400-3000 vs long replies ~2-3, PPL ~7-20)
  and wide variance in L_cond (span 0-17.5), which distorts
  downstream IFD-based difficulty estimates.
 2026-07-24 06:37:35 +08:00
ViperEkura 0a1d0573ae doc: add SFT length filter (15-token floor) with IFD bias rationale 
Per-field length filtering in SFT drops instruction--response pairs
with responses shorter than 15 tokens. Short replies exhibit high
per-token variance in both conditional and unconditional loss
(Appendix A.3 / Figure 5), which would distort downstream IFD-based
difficulty estimates.
 2026-07-24 06:27:10 +08:00
ViperEkura 0c2bc916f2 fix: loss_compare caption token count (20B -> 5B) 
The loss_compare.png x-axis only spans 0-5B tokens. The caption
incorrectly claimed ~20B tokens.
 2026-07-24 06:21:52 +08:00
ViperEkura 4e70e827ff remove: drop ckpt_weight_density_per_run.png figure 
The per-run weight density figure contained misleading legacy
iteration labels (500k/1M iter) that contradicted the paper's
stated token budgets (15B). Removing it avoids confusion; the
per-category density plot (ckpt_weight_density.png) and Table 7
remain as the primary evidence.
 2026-07-24 06:20:08 +08:00
ViperEkura a7bbc7b29f fix: align SFT/DPO figures and text with actual training data 
- SFT: 1,000 steps/WSD → ~3,800 steps/cosine; loss ~2.5→1.6 → ~2.1→1.5
- DPO: fix preference-loss narrative to match training-loss curve;
  correct initial grad-norm (~50 → ~200)
- Table 2: peak LR 1.5e-4 → 2.0e-4 (matches pt_metric.png)
- Clarify scheduling: pretraining=WSD, SFT+DPO=cosine
- Add disclaimer to ckpt_weight_density_per_run caption for legacy iter labels
 2026-07-24 06:15:42 +08:00
ViperEkura 4b37f289c0 fix: attribute muon-25bt larger std to Muon characteristic, not training duration alone 
The muon-25bt (Muon, 25B) vs kami/norm (AdamW, 15B) comparison confounds
optimizer choice with training duration. Reframe as "Muon produces larger
post-convergence weight variance" rather than "training duration drives
variance growth."
 2026-07-20 11:59:09 +08:00
ViperEkura a83555f326 fix: align conclusion, abstract, introduction with body data and full paper scope 
- Fix sigma=0.003 (init value) to consistently narrower than non-scaled
  (body 4.4, conclusion, abstract)
- Add SFT/DPO to conclusion, abstract, and introduction
- Fix checkpoint label: kami-15bt (Muon) -> (AdamW, GPT-2 residual scaling)
  in effective rank appendix
- Fix table weight_std caption: Muon -> residual scaling constrains drift
- Fix 4.2 math: 0.689/L -> 0.689/(2L)
- Fix length filter: per-sample -> per-text-field skip
 2026-07-20 11:51:48 +08:00
ViperEkura d93ff48320 docs: fix storage backends (2->3) and per-field length filter 
- Storage Backends: add JsonlStore (lazy on-the-fly tokenization)
- Describe memory-efficiency tradeoff across three backends
- Fix length filter: per-text-field skip, not per-sample global
- Abstract: HDF5/mmap -> tiered storage backends
- Rename metric figures (singular) and update references
 2026-07-20 11:27:39 +08:00
ViperEkura da0f536526 Update DPO iteration count: 1,500 -> 3,000 2026-07-20 03:59:55 +08:00
ViperEkura c775a2b3e0 Update DPO metrics figure 2026-07-20 03:59:04 +08:00
ViperEkura 3f0ff911a8 Fix lingering cosine->WSD in Conclusion; update AGENTS.md with checkpoint labels and scheduler rules 2026-07-19 15:30:49 +08:00
ViperEkura e149997200 Add DPO data generation & training; fix checkpoint labels; switch to WSD scheduler; update optimizer ablations in abstract/conclusion 2026-07-19 15:25:42 +08:00
ViperEkura 6c2e04a86f Add DPO training subsection with dpo_metrics figure and Rafailov et al. citation 2026-07-19 15:01:23 +08:00
ViperEkura 411354eeb1 Update sft_metrics.png 2026-07-18 00:52:52 +08:00
ViperEkura a0c39601a0 Add effective rank analysis, weight distribution evolution, Muon/SFT training figures 
- Add SVD effective rank (ER@99%) and condition number analysis across
  three checkpoints (kami-15bt, norm-15bt, muon-25bt) in appendix
- Add post-training weight distribution analysis as §5.4, verifying
  residual scaling persists throughout training
- Add per-component weight std table in appendix showing Muon constrains
  early-stage drift vs Normal init
- Add Muon optimizer training dynamics figure (muon_pt.png)
- Add SFT training metrics figure (sft_metrics.png) on mixed CN-EN data
- Restructure: move effective rank to appendix, weight distribution to
  §5.4, delete standalone §6, tighten abstract to prose-only
- Update conclusion to reference new analyses
 2026-07-17 23:05:21 +08:00
ViperEkura bfc8ff6098 Fix training budget (19k steps / ~20B tokens), add Muon optimizer, correct variance scaling math, move figures to §Training Config, tighten abstract 2026-07-08 22:49:43 +08:00
ViperEkura 4a143b056d Tighten abstract, add Muon optimizer mention, update ckpt_comparison figure 2026-07-08 22:27:33 +08:00
ViperEkura 5726bf6c84 Add optimizer/initialization comparison figure (ckpt_comparison) 2026-07-08 21:59:56 +08:00
ViperEkura ff13a45d95 Restructure data pipeline: replace IFD analysis with length bias appendix 
- Remove Section 2.4 (IFD analysis) from main text
- Delete obsolete IFD figures (ifd_compare_clean, ifd_density_dist,
  ifd_both_vs_lossratio, ifd_loss_ratio_density)
- Update ifd_length_grid.png with new data
- Rewrite Appendix A: IFD definition + quantitative summary +
  representative samples table + length bias analysis
- Update abstract to remove IFD findings
- Fix .gitignore to only track .png in data/
- Clean up stale analysis scripts
 2026-07-05 18:59:29 +08:00
ViperEkura 2d52a00149 Update ifd_loss_ratio_density.png 2026-07-05 12:38:43 +08:00
ViperEkura f2c2b159ba Add IFD loss ratio density analysis, weight distribution appendix, and rewrite abstract 2026-07-05 12:31:40 +08:00
ViperEkura 459523f0e4 Fix IFD citation: correct author list and venue (NAACL 2024, not NeurIPS) 2026-07-05 00:24:43 +08:00
ViperEkura 4a7bf7ac3f Add GQA/SwiGLU/RoPE formula blocks, Alpaca-GPT4 citation, and improve IFD-Loss Ratio interpretation 2026-07-05 00:17:09 +08:00
ViperEkura 5c915514b3 Add correlation analysis interpretation to IFD bias section 2026-07-04 23:46:25 +08:00
ViperEkura bc8d4066c1 Add IFD analysis: density/length-bias figures and appendix 2026-07-04 23:33:12 +08:00
ViperEkura 9534316bac Fix duplicate paragraph and misleading table caption 2026-07-04 22:39:16 +08:00
ViperEkura b40fcc86c7 Refine abstract, switch to Times font, remove date, fix overfull hboxes with microtype 2026-07-04 22:31:45 +08:00
ViperEkura dd412b3c9c Add IFD analysis, appendix, and improve abstract 2026-07-04 22:25:18 +08:00
ViperEkura 73cb02f0aa fix training hyperparams to match pretraining script: lr 1.5e-4, wd 0.1, warmup 0.02 2026-06-28 16:57:30 +08:00
ViperEkura adc284920a add Alembic SFT data cleaning section with MinHash algorithm and fix bibliography order 2026-06-28 16:52:11 +08:00
ViperEkura 13089b002a fix training config: grad_accum 32, remove unused validation & activation ckpt 2026-06-28 15:35:15 +08:00
ViperEkura d3fbe0ecc6 refactor: numerical stability analysis with residual scaling comparison 2026-06-28 14:36:12 +08:00
ViperEkura 3db1096c78 fix: add caption package, fix overfull hbox, correct RoFormer author names 2026-06-25 16:53:26 +08:00
ViperEkura 6f9278b35b Initial commit: BF16 weight lock-in paper for 1.2B Transformer training with AstrAI 2026-06-25 15:51:05 +08:00