llama-lang-adapt
/

yo-extend-tokenizer

Model card Files Files and versions Community

yo-extend-tokenizer / README.md

nvassilyev's picture

Create README.md

dd98d75 over 1 year ago

|

history blame contribute delete

80 Bytes

	Llama tokenizer with extended vocab:
	- 32000 Llama tokens
	- ~15000 Yoruba tokens