Update README.md
Browse files
README.md
CHANGED
|
@@ -5,6 +5,11 @@ tags:
|
|
| 5 |
- tokenizers
|
| 6 |
---
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
# DBRX Instruct Tokenizer
|
| 9 |
|
| 10 |
A 🤗-compatible version of the **DBRX Instruct** (adapted from [databricks/dbrx-instruct](https://huggingface.co/databricks/dbrx-instruct)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
|
|
|
|
| 5 |
- tokenizers
|
| 6 |
---
|
| 7 |
|
| 8 |
+
## Why should you use this and not the titotken included in the orignal model?
|
| 9 |
+
Original tokenizer pad vocabulary to correct size with `<extra_N>` tokens but encoder never uses them causing inconsistency and deterimental to training code that may want to use the unused `<extra_N>` tokens.
|
| 10 |
+
|
| 11 |
+
modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer
|
| 12 |
+
|
| 13 |
# DBRX Instruct Tokenizer
|
| 14 |
|
| 15 |
A 🤗-compatible version of the **DBRX Instruct** (adapted from [databricks/dbrx-instruct](https://huggingface.co/databricks/dbrx-instruct)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
|