README.md

metadata

library_name: transformers
tags:
  - tokenizer
license: mit
datasets:
  - DKYoon/SlimPajama-6B

sail-slimpajama-6B-32768-BPE-tokenizer

This is simply an 'explicit' loadable repo with the 32768 vocab size tokenizer from the sail vocab scaling laws study

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tk = AutoTokenizer.from_pretrained('pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer')

details:

LlamaTokenizerFast(name_or_path='pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer', vocab_size=32768, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
    0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
    1: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
    2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
    3: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}