Update README.md

README.md

@@ -10,6 +10,21 @@ Original tokenizer pad vocabulary to correct size with `<extra_N>` tokens but en
 
 modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer 
 
+```
+Changes:
+1. Remove non-base model tokens
+2. Keep/Add `<|pad|>` special token to make sure padding can be differentiated from eos/bos.
+
+ "100277": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+```
+
 # DBRX Instruct Tokenizer
 
 A 🤗-compatible version of the **DBRX Instruct** (adapted from [databricks/dbrx-instruct](https://huggingface.co/databricks/dbrx-instruct)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).