README.md · xu3kev/the-tokenizer-playground at main

metadata

title: The Tokenizer Playground
emoji: 📝
colorFrom: indigo
colorTo: purple
sdk: static
pinned: false
short_description: Experiment with and compare different tokenizers
models:
  - Xenova/gpt-4
  - Xenova/text-davinci-003
  - Xenova/gpt-3
  - Xenova/grok-1-tokenizer
  - Xenova/claude-tokenizer
  - Xenova/mistral-tokenizer
  - Xenova/gemma-tokenizer
  - Xenova/llama-tokenizer
  - Xenova/c4ai-command-r-v01-tokenizer
  - Xenova/t5-small
  - Xenova/bert-base-cased
  - xai-org/grok-1

Link to source code: https://github.com/xenova/transformers.js/tree/main/examples/tokenizer-playground.