docs: fix storage backends (2->3) and per-field length filter

Browse Source

- Storage Backends: add JsonlStore (lazy on-the-fly tokenization)
- Describe memory-efficiency tradeoff across three backends
- Fix length filter: per-text-field skip, not per-sample global
- Abstract: HDF5/mmap -> tiered storage backends
- Rename metric figures (singular) and update references

...

This commit is contained in:

ViperEkura

2026-07-20 11:27:39 +08:00

parent da0f536526

commit d93ff48320

7 changed files with 31 additions and 18 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/dpo_metric.png

BIN

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 92 KiB

data/dpo_metrics.png

BIN

Binary file not shown.

Side by Side Before Width:  |  Height:  |  Size: 166 KiB

data/muon_pt.png

BIN

Binary file not shown.

Side by Side Before Width:  |  Height:  |  Size: 128 KiB

data/pt_metric.png

BIN

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 68 KiB

data/sft_metric.png

BIN

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 85 KiB

data/sft_metrics.png

BIN

Binary file not shown.

Side by Side Before Width:  |  Height:  |  Size: 172 KiB