Tokenizer Issue (when running on Sagemaker)

#2
by RoiTabach - opened

Hi, I'm running on Sagemaker (using the huggingface reccomended script) and getting this stacktrace on the Enpoint Deployment -

 File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 196, in serve_inner
    model = get_model(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 377, in get_model
    return FlashMistral(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_mistral.py", line 526, in __init__
    super(FlashMistral, self).__init__(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_mistral.py", line 310, in __init__
    tokenizer = LlamaTokenizerFast.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained
    return cls._from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 124, in __init__
    super().__init__(
  File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)

Exception: data did not match any variant of untagged enum ModelWrapper at line 205277 column 3

Both StackOverflow and HF discussions seems to think the solution to this will come from training with a PreTokenizer:
Huggingface comment with suggested solution
@yam-peleg if you have any ideas

RoiTabach changed discussion title from Issue running on Sagemaker to Tokenizer Issue (when running on Sagemaker)
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment