ViperEkura 958df58f9d refactor: unify tokenizer encode and apply_chat_template for batch support ...

- encode(str) single-thread, encode(List[str]) Rust parallel encode_batch
- apply_chat_template accepts single Messages or List[Messages] for batch
- add Message/Messages type aliases at module level

2026-07-20 17:49:26 +08:00

..

__init__.py

refactor: unify tokenizer encode and apply_chat_template for batch support

2026-07-20 17:49:26 +08:00

chat_template.py

fix: make tokenizer picklable for spawn multiprocessing

2026-07-19 11:59:55 +08:00

tokenizer.py

refactor: unify tokenizer encode and apply_chat_template for batch support

2026-07-20 17:49:26 +08:00