add missing special tokens in tokenizer.json

#10

by mmoskal - opened 8 days ago

base: refs/heads/main

←

from: refs/pr/10

Discussion Files changed

+32

-0

add missing special tokens in tokenizer.json88426c96

mmoskal

8 days ago

tokenizer_config.json has <think>, </think> and <tool_response> tokens, which are missing from tokenizer.json. If I understand correctly, the files should match.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Cannot merge

This branch has merge conflicts in the following files:

tokenizer.json

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment