Fast tokenizer, general VLLM support
Is this being working on ? I did not see any PRs for transformers
yes, transformers support is in progress, hope we can submit the the PR within next week
vLLM support is in this PR:https://github.com/vllm-project/vllm/pull/20114/files
about the faster tokenizer, could give some example on this items (some link inside of transformers lib ?)
yes, transformers support is in progress, hope we can submit the the PR within next week
vLLM support is in this PR:https://github.com/vllm-project/vllm/pull/20114/files
about the faster tokenizer, could give some example on this items (some link inside of transformers lib ?)
Tokenizer mode:
"auto" will use the fast tokenizer if available.
"slow" will always use the slow tokenizer.
"mistral" will always use the tokenizer from mistral_common.
"custom" will use --tokenizer to select the preregistered tokenizer.
This has to do with transformers support, so when the PR is in in transformers and or vllm support. This will work. I think the tokenizer is being run with a python script inside the vllm docker image.