manangarg
/

ind-llm-tokenizer

Model card Files Files and versions Community

ind-llm-tokenizer / README.md

manangarg's picture

Update README.md

5d0daa4 verified 10 months ago

|

history blame contribute delete

142 Bytes

metadata

license: apache-2.0

• Indic Language LLM Tokenizer

This is an indic language NLP tokenizer which is merged with LLaMA 2 tokenizer.