ViperEkura
a7bbc7b29f
fix: align SFT/DPO figures and text with actual training data
...
- SFT: 1,000 steps/WSD → ~3,800 steps/cosine; loss ~2.5→1.6 → ~2.1→1.5
- DPO: fix preference-loss narrative to match training-loss curve;
correct initial grad-norm (~50 → ~200)
- Table 2: peak LR 1.5e-4 → 2.0e-4 (matches pt_metric.png)
- Clarify scheduling: pretraining=WSD, SFT+DPO=cosine
- Add disclaimer to ckpt_weight_density_per_run caption for legacy iter labels
2026-07-24 06:15:42 +08:00
ViperEkura
d93ff48320
docs: fix storage backends (2->3) and per-field length filter
...
- Storage Backends: add JsonlStore (lazy on-the-fly tokenization)
- Describe memory-efficiency tradeoff across three backends
- Fix length filter: per-text-field skip, not per-sample global
- Abstract: HDF5/mmap -> tiered storage backends
- Rename metric figures (singular) and update references
2026-07-20 11:27:39 +08:00
ViperEkura
c775a2b3e0
Update DPO metrics figure
2026-07-20 03:59:04 +08:00
ViperEkura
e149997200
Add DPO data generation & training; fix checkpoint labels; switch to WSD scheduler; update optimizer ablations in abstract/conclusion
2026-07-19 15:25:42 +08:00
ViperEkura
6c2e04a86f
Add DPO training subsection with dpo_metrics figure and Rafailov et al. citation
2026-07-19 15:01:23 +08:00
ViperEkura
411354eeb1
Update sft_metrics.png
2026-07-18 00:52:52 +08:00
ViperEkura
a0c39601a0
Add effective rank analysis, weight distribution evolution, Muon/SFT training figures
...
- Add SVD effective rank (ER@99%) and condition number analysis across
three checkpoints (kami-15bt, norm-15bt, muon-25bt) in appendix
- Add post-training weight distribution analysis as §5.4, verifying
residual scaling persists throughout training
- Add per-component weight std table in appendix showing Muon constrains
early-stage drift vs Normal init
- Add Muon optimizer training dynamics figure (muon_pt.png)
- Add SFT training metrics figure (sft_metrics.png) on mixed CN-EN data
- Restructure: move effective rank to appendix, weight distribution to
§5.4, delete standalone §6, tighten abstract to prose-only
- Update conclusion to reference new analyses
2026-07-17 23:05:21 +08:00
ViperEkura
bfc8ff6098
Fix training budget (19k steps / ~20B tokens), add Muon optimizer, correct variance scaling math, move figures to §Training Config, tighten abstract
2026-07-08 22:49:43 +08:00
ViperEkura
4a143b056d
Tighten abstract, add Muon optimizer mention, update ckpt_comparison figure
2026-07-08 22:27:33 +08:00
ViperEkura
5726bf6c84
Add optimizer/initialization comparison figure (ckpt_comparison)
2026-07-08 21:59:56 +08:00
ViperEkura
ff13a45d95
Restructure data pipeline: replace IFD analysis with length bias appendix
...
- Remove Section 2.4 (IFD analysis) from main text
- Delete obsolete IFD figures (ifd_compare_clean, ifd_density_dist,
ifd_both_vs_lossratio, ifd_loss_ratio_density)
- Update ifd_length_grid.png with new data
- Rewrite Appendix A: IFD definition + quantitative summary +
representative samples table + length bias analysis
- Update abstract to remove IFD findings
- Fix .gitignore to only track .png in data/
- Clean up stale analysis scripts
2026-07-05 18:59:29 +08:00
ViperEkura
2d52a00149
Update ifd_loss_ratio_density.png
2026-07-05 12:38:43 +08:00
ViperEkura
f2c2b159ba
Add IFD loss ratio density analysis, weight distribution appendix, and rewrite abstract
2026-07-05 12:31:40 +08:00
ViperEkura
4a7bf7ac3f
Add GQA/SwiGLU/RoPE formula blocks, Alpaca-GPT4 citation, and improve IFD-Loss Ratio interpretation
2026-07-05 00:17:09 +08:00
ViperEkura
5c915514b3
Add correlation analysis interpretation to IFD bias section
2026-07-04 23:46:25 +08:00
ViperEkura
bc8d4066c1
Add IFD analysis: density/length-bias figures and appendix
2026-07-04 23:33:12 +08:00
ViperEkura
dd412b3c9c
Add IFD analysis, appendix, and improve abstract
2026-07-04 22:25:18 +08:00
ViperEkura
d3fbe0ecc6
refactor: numerical stability analysis with residual scaling comparison
2026-06-28 14:36:12 +08:00
ViperEkura