kl3m-004-char-8k-cased / tokenizer_config.json
alea-institute's picture
Upload folder using huggingface_hub
893bf77 verified
raw
history blame contribute delete
269 Bytes
{"unk_token": "<|unk|>", "bos_token": "<|start|>", "eos_token": "<|end|>", "pad_token": "<|pad|>", "sep_token": "<|sep|>", "cls_token": "<|cls|>", "mask_token": "<|mask|>", "add_prefix_space": false, "do_lower_case": false, "tokenizer_class": "PreTrainedTokenizerFast"}