tokent1 / added_tokens.json
caiom's picture
Upload tokenizer
bb7c5c2 verified
raw
history blame
297 Bytes
{
"<|assistant|>": 32005,
"<|continue|>": 32006,
"<|endoftext|>": 32000,
"<|end|>": 32001,
"<|function_call|>": 32008,
"<|function_list|>": 32007,
"<|function_output|>": 32009,
"<|raw|>": 32011,
"<|step|>": 32010,
"<|system|>": 32002,
"<|tag|>": 32003,
"<|user|>": 32004
}