refactor: unify tokenizer encode and apply_chat_template for batch support

- encode(str) single-thread, encode(List[str]) Rust parallel encode_batch - apply_chat_template accepts single Messages or List[Messages] for batch - add Message/Messages type aliases at module level
2026-07-20 17:49:26 +08:00
parent e0f102c4d9
commit 958df58f9d
2 changed files with 61 additions and 28 deletions
@@ -1,8 +1,10 @@
 from astrai.tokenize.chat_template import ChatTemplate, MessageType
-from astrai.tokenize.tokenizer import AutoTokenizer
+from astrai.tokenize.tokenizer import AutoTokenizer, Message, Messages

 __all__ = [
    "AutoTokenizer",
    "ChatTemplate",
    "MessageType",
+    "Message",
+    "Messages",
 ]