llama-lang-adapt
/

yo-extend-tokenizer

Model card Files Files and versions Community

nvassilyev commited on Nov 8, 2023

Commit

dd98d75

·

1 Parent(s): 538c872

Create README.md

Files changed (1) hide show

README.md +3 -0

README.md ADDED Viewed

	@@ -0,0 +1,3 @@

+Llama tokenizer with extended vocab:
+- 32000 Llama tokens
+- ~15000 Yoruba tokens