Hugging Face's logo

meelu
/
DA-MIXED-CEREBRAS-TOKEN

tokenizers
Danish
Mixed Tokenization
CerebrasGPT
Model card Files Community
MikkelWK commited on
Commit
6e42c6f
·
verified ·
1 Parent(s): 2afd54f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -1,8 +1,8 @@
1
  ---
2
  library_name: tokenizers
3
- tags: [Danish, Mixed Tokenization, LLaMA]
4
  ---
5
 
6
- ### DA-MIXED-LLAMA3.2-TOKEN
7
 
8
- A mixed tokenization approach combining morphology and BPE strategies, developed for the LLaMA architecture to enhance Danish text processing.
 
1
  ---
2
  library_name: tokenizers
3
+ tags: [Danish, Mixed Tokenization, CerebrasGPT]
4
  ---
5
 
6
+ ### DA-MIXED-CEREBRAS-TOKEN
7
 
8
+ This tokenizer combines morphological and BPE tokenization strategies for use with the CerebrasGPT architecture, aiming to balance linguistic insights and subword efficiency.