ViperEkura
e220413035
- detect_format now recognizes .json directories as jsonl store - JsonlStore loads .json arrays and dicts alongside .jsonl - tokenizer_path defaults to dataset dir when omitted - Pipeline._iter_items handles .json files (arrays/single dict) - Tests: detect_format, seq load, self-contained dataset dir |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| builder.py | ||
| packing.py | ||
| pipeline.py | ||
| position_id.py | ||
| writer.py | ||