refactor: 修改分词器部分结构，更新特殊token等

2026-04-03 14:52:35 +08:00
parent 94c6a015c8
commit c5560740b6
3 changed files with 158 additions and 78 deletions
@@ -61,7 +61,7 @@ flowchart LR

 #### 1.1 Tokenizer (`tokenizer.py`)
 - Implemented based on Byte-Level BPE (BBPE)
- Supports special tokens: `<bos>`, `<eos>`, `<pad>`, `<|im_start|>`, `<|im_end|>`
+- Supports special tokens: `<｜begin▁of▁sentence｜>`, `<｜end▁of▁sentence｜>`, `<｜▁pad▁｜>`, `<｜im▁start｜>`, `<｜im▁end｜>`
 - Provides `encode`/`decode` methods for mutual conversion between text and token IDs
 - Learns vocabulary from corpus during training, saved as `.json` files