Hugging Face's logo

Spaces:
Xenova
/
the-tokenizer-playground
Running

App Files Community
11

Could I get a simple workflow to add custom tokenizer?

#3
by sh0416 - opened
Discussion
sh0416

I want to see the tokenized result of phi tokenizer.
How to modify the code to add that tokenizer?
Thank you.

Xenova
 Owner

The source code for this demo can be found here, and you can modify it to add support for new tokenizers.

Xenova changed discussion status to closed
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment