llama-lang-adapt
/

yo-extend-tokenizer

Model card Files Files and versions Community

yo-extend-tokenizer / README.md

nvassilyev's picture

Create README.md

dd98d75 over 1 year ago

|

history blame contribute delete

80 Bytes

Llama tokenizer with extended vocab:

32000 Llama tokens
~15000 Yoruba tokens