fix: make tokenizer picklable for spawn multiprocessing

- ChatTemplate: defer Jinja2 compilation to cached_property, exclude compiled template from __getstate__ (its dynamic root function has __module__=None and falls back to __main__, breaking pickle) - AutoTokenizer: bypass __getattr__ for underscore-prefixed attrs to prevent infinite recursion during unpickle when __dict__ is empty
2026-07-19 11:59:55 +08:00
parent d7695b40e3
commit 17127f8b3c
2 changed files with 21 additions and 1 deletions
@@ -164,7 +164,14 @@ class AutoTokenizer:
          - tokenizer.bos_token   → returns string
          - tokenizer.bos_token_id → returns corresponding integer ID
          - tokenizer.stop_ids → returns list of corresponding integer IDs for all special tokens
+
+        Internal/private attrs are not intercepted: during unpickle
+        ``__dict__`` is empty, so probing ``self._special_token_map``
+        would recurse infinitely.
        """
+        if key.startswith("_"):
+            raise AttributeError(key)
+
        # Handle stop_ids - return IDs for all special tokens
        if key == "stop_ids":
            stop_ids = []