Update README.md
Browse files
README.md
CHANGED
|
@@ -5,7 +5,10 @@ tags:
|
|
| 5 |
- tokenizers
|
| 6 |
---
|
| 7 |
|
| 8 |
-
##
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
# DBRX Instruct Tokenizer
|
| 11 |
|
|
|
|
| 5 |
- tokenizers
|
| 6 |
---
|
| 7 |
|
| 8 |
+
## Why should you use this and not the titotken included in the orignal model?
|
| 9 |
+
Original tokenizer pad vocabulary to correct size with `<extra_N>` tokens but encoder never uses them causing inconsistency and deterimental to training code that may want to use the unused `<extra_N>` tokens.
|
| 10 |
+
|
| 11 |
+
modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer
|
| 12 |
|
| 13 |
# DBRX Instruct Tokenizer
|
| 14 |
|