diff --git a/assets/docs/architecture.md b/assets/docs/architecture.md index 0be1bfc..07b5921 100644 --- a/assets/docs/architecture.md +++ b/assets/docs/architecture.md @@ -924,7 +924,6 @@ classDiagram +AutoTokenizer tokenizer +int max_batch_size +int max_seq_len - +int max_prompt_len +Deque waiting_queue +List active_tasks +add_task(prompt, max_tokens, temperature, top_p, top_k, stream_callback) str diff --git a/assets/docs/dataflow.md b/assets/docs/dataflow.md index 0c676e6..80e35a0 100644 --- a/assets/docs/dataflow.md +++ b/assets/docs/dataflow.md @@ -83,9 +83,11 @@ All backends normalise tensors into `Store._data[Dict[str, List[Tensor]]]` + `St ## Dataset Architecture ``` -DatasetFactory.load(train_type, load_path, window_size, stride=None, - storage_type=None, tokenizer_path=None, - max_position_embeddings=2048, store=None) +DatasetFactory.load( + train_type, load_path=None, window_size=0, stride=None, + storage_type=None, tokenizer_path=None, + max_len=2048, store=None +) → BaseDataset.load(load_path, storage_type=None) → detect_format(load_path) → StoreFactory.create(storage_type) diff --git a/assets/docs/params.md b/assets/docs/params.md index cfb6ed9..3d359e8 100644 --- a/assets/docs/params.md +++ b/assets/docs/params.md @@ -84,7 +84,7 @@ Combined optimizer: matrix parameters via **Muon**, non-matrix via **AdamW** (`f | Parameter | Description | Default | |-----------|-------------|---------| | `--nprocs` | Number of GPUs / processes | 1 | -| `--parallel_mode` | Parallel strategy (`none`, `ddp`, or `fsdp`) | none | +| `--parallel_mode` | Parallel strategy (`none`, `ddp`, `fsdp`, or `fsdp2`) | none | | `--device_type` | Device type | cuda | | `--start_method` | Multiprocessing start method (`spawn`, `fork`, `forkserver`) | spawn | | `--backend` | Distributed training backend | nccl | @@ -164,6 +164,7 @@ nohup python scripts/tools/train.py \ | `--device` | str | `cuda` | Device to load model on | | `--dtype` | str | `bfloat16` | Model weights dtype (`bfloat16`, `float16`, `float32`) | | `--max_batch_size` | int | `16` | Maximum batch size for continuous batching | +| `--max_seq_len` | int | model config `max_position_embeddings` | Maximum sequence length (KV cache size + prompt truncation) | | `--reload` | flag | `False` | Enable auto-reload for development | Usage: @@ -186,7 +187,11 @@ See [Inference Guide](inference.md) for HTTP API documentation. | `--top_k` | int | `30` | Top-k filtering | | `--top_p` | float | `0.95` | Nucleus sampling threshold | | `--batch_size` | int | `1` | Batch size for generation | +| `--num_samples` | int | `1` | Responses per prompt | | `--max_tokens` | int | model config `max_position_embeddings` | Maximum tokens to generate | +| `--cache_len` | int | `2048` | KV cache length | +| `--frequency_penalty` | float | `0.0` | Frequency penalty | +| `--rep_window` | int | `64` | Window size for frequency penalty | Usage: ```bash diff --git a/assets/docs/preprocessing.md b/assets/docs/preprocessing.md index 342d8f0..4fc13fe 100644 --- a/assets/docs/preprocessing.md +++ b/assets/docs/preprocessing.md @@ -256,6 +256,7 @@ When `sources` is set, `sections` is ignored. | `min_chars` | int | `50` | Skip text-mode items shorter than this | | `max_chars` | int | `2000000` | Skip text-mode items longer than this | | `max_items` | int or null | `null` | Stop after N documents | +| `batch_size` | int | `256` | Records per tokenization batch | | `packing_strategy` | str | `"simple"` | Packing strategy: `"simple"`, `"bfd"`, `"bfd_split"` | | `max_packed_len` | int | `8192` | Maximum length of a packed bin | | `truncation_mode` | str | `"keep_start"` | How to truncate sequences: `"keep_start"` or `"keep_end"` |