vibhorag101
commited on
Update README.md
Browse filesupdated tokenizer and doc
README.md
CHANGED
@@ -47,6 +47,13 @@ If a question does not make any sense, or is not factually coherent, explain why
|
|
47 |
|
48 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64eb1e4a55e4f0ecb9c4f406/PsbTFlswJexLuwrJYtvly.png)
|
49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
50 |
|
51 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
52 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vibhorag101__llama-2-7b-chat-hf-phr_mental_health-2048)
|
|
|
47 |
|
48 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64eb1e4a55e4f0ecb9c4f406/PsbTFlswJexLuwrJYtvly.png)
|
49 |
|
50 |
+
## Tokenizer Details
|
51 |
+
The tokenizer should be loaded as follows for correct inference:
|
52 |
+
```
|
53 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True,use_fast=False,add_eos_token=True)
|
54 |
+
tokenizer.pad_token_id = 18610
|
55 |
+
```
|
56 |
+
|
57 |
|
58 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
59 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vibhorag101__llama-2-7b-chat-hf-phr_mental_health-2048)
|