rename
Browse files
README.md
CHANGED
@@ -8,7 +8,7 @@ language:
|
|
8 |
- en
|
9 |
---
|
10 |
|
11 |
-
## Model Details: Neural-Chat-v3-3-int4-inc
|
12 |
|
13 |
This model is an int4 model with group_size 128 of [Intel/neural-chat-7b-v3-3](https://huggingface.co/Intel/neural-chat-7b-v3-3) generated by [intel/auto-round](https://github.com/intel/auto-round).
|
14 |
|
@@ -50,7 +50,7 @@ Install [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first
|
|
50 |
|
51 |
```python
|
52 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
53 |
-
quantized_model_dir = "Intel/neural-chat-v3-3-int4-inc"
|
54 |
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
|
55 |
device_map="auto",
|
56 |
trust_remote_code=False,
|
@@ -66,7 +66,7 @@ print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adv
|
|
66 |
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
|
67 |
|
68 |
```bash
|
69 |
-
lm_eval --model hf --model_args pretrained="Intel/neural-chat-v3-3-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge --batch_size 128
|
70 |
```
|
71 |
|
72 |
| Metric | FP16 | INT4 |
|
|
|
8 |
- en
|
9 |
---
|
10 |
|
11 |
+
## Model Details: Neural-Chat-7b-v3-3-int4-inc
|
12 |
|
13 |
This model is an int4 model with group_size 128 of [Intel/neural-chat-7b-v3-3](https://huggingface.co/Intel/neural-chat-7b-v3-3) generated by [intel/auto-round](https://github.com/intel/auto-round).
|
14 |
|
|
|
50 |
|
51 |
```python
|
52 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
53 |
+
quantized_model_dir = "Intel/neural-chat-7b-v3-3-int4-inc"
|
54 |
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
|
55 |
device_map="auto",
|
56 |
trust_remote_code=False,
|
|
|
66 |
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
|
67 |
|
68 |
```bash
|
69 |
+
lm_eval --model hf --model_args pretrained="Intel/neural-chat-7b-v3-3-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge --batch_size 128
|
70 |
```
|
71 |
|
72 |
| Metric | FP16 | INT4 |
|