add missing special tokens in tokenizer.json

#10
by mmoskal - opened

tokenizer_config.json has <think>, </think> and <tool_response> tokens, which are missing from tokenizer.json. If I understand correctly, the files should match.

Cannot merge
This branch has merge conflicts in the following files:
  • tokenizer.json
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment