File size: 1,321 Bytes
92303d7 0ec3f76 92303d7 0ec3f76 92303d7 af8e22f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
---
library_name: transformers
tags:
- tokenizer
license: mit
datasets:
- DKYoon/SlimPajama-6B
---
# sail-slimpajama-6B-32768-BPE-tokenizer
This is simply an 'explicit' loadable repo with the 32768 vocab size tokenizer from the [sail vocab scaling laws study](https://huggingface.co/sail/scaling-with-vocab-trained-tokenizers/tree/main/hf_slimpajama-6B-32768-BPE)
## Usage
```py
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tk = AutoTokenizer.from_pretrained('pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer')
```
details:
```
LlamaTokenizerFast(name_or_path='pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer', vocab_size=32768, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
1: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
3: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
```
|