PAD tokens is present in `tokenizer.json` but not in `special_tokens_map.json`.

by sadra-barikbin - opened Jun 6, 2024

Jun 6, 2024

Hi there!
PAD token is present in tokenizer's added tokens in tokenizer.json as a special token with content "<|padding|>", while it's not in special_tokens_map.json and also in tokenizer_config.json which results in tokenizer.pad_token being None.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment