Llama3-ChatQA-1.5-70B / tokenizer.json
zjyhf's picture
The tokenizer adds a special token '<|im_end|>' to solve the problem of non-stop generation when encountering <|im_end|>.
26140d9 verified
raw
history
9.09 MB
File too large to display, you can check the raw version instead.