Update README.md

README.md

@@ -10,4 +10,24 @@ datasets:
 # sail-slimpajama-6B-32768-BPE-tokenizer
 
 
-This is simply an 'explicit' loadable repo with the 32768 vocab size tokenizer from the [sail vocab scaling laws study](https://huggingface.co/sail/scaling-with-vocab-trained-tokenizers/tree/main/hf_slimpajama-6B-32768-BPE)
+This is simply an 'explicit' loadable repo with the 32768 vocab size tokenizer from the [sail vocab scaling laws study](https://huggingface.co/sail/scaling-with-vocab-trained-tokenizers/tree/main/hf_slimpajama-6B-32768-BPE)
+
+
+## Usage
+
+
+```py
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+tk = AutoTokenizer.from_pretrained('pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer')
+```
+
+details:
+
+```
+LlamaTokenizerFast(name_or_path='pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer', vocab_size=32768, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
+	0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
+	1: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
+	2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
+	3: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
+}
+```