Jhayase commited on
Commit
08a038f
·
1 Parent(s): b7055d5

Link models

Browse files
Files changed (1) hide show
  1. README.md +9 -14
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- title: The Tokenizer Playground
3
  emoji: 📝
4
  colorFrom: indigo
5
  colorTo: purple
@@ -7,18 +7,13 @@ sdk: static
7
  pinned: false
8
  short_description: Experiment with and compare different tokenizers
9
  models:
10
- - Xenova/gpt-4
11
- - Xenova/text-davinci-003
12
- - Xenova/gpt-3
13
- - Xenova/grok-1-tokenizer
14
- - Xenova/claude-tokenizer
15
- - Xenova/mistral-tokenizer
16
- - Xenova/gemma-tokenizer
17
- - Xenova/llama-tokenizer
18
- - Xenova/c4ai-command-r-v01-tokenizer
19
- - Xenova/t5-small
20
- - Xenova/bert-base-cased
21
- - xai-org/grok-1
22
  ---
23
 
24
- Link to source code: https://github.com/xenova/transformers.js/tree/main/examples/tokenizer-playground.
 
1
  ---
2
+ title: SuperBPE Playground
3
  emoji: 📝
4
  colorFrom: indigo
5
  colorTo: purple
 
7
  pinned: false
8
  short_description: Experiment with and compare different tokenizers
9
  models:
10
+ - Xenova/gpt-4o
11
+ - Xenova/llama-3-tokenizer
12
+ - UW/OLMo2-8B-SuperBPE-t180k
13
+ - UW/OLMo2-11B-SuperBPE-t180k
14
+ - UW/OLMo2-8B-SuperBPE-t160k
15
+ - UW/OLMo2-8B-SuperBPE-t80k
16
+ - UW/OLMo2-8B-BPE
 
 
 
 
 
17
  ---
18
 
19
+ Link to source code: https://github.com/xenova/transformers.js/tree/main/examples/tokenizer-playground.