Update README.md
Browse files
README.md
CHANGED
@@ -101,11 +101,6 @@ print(f"{save_to} model:", benchmark_fn(quantized_model.generate, **inputs, max_
|
|
101 |
# Model Quality
|
102 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
103 |
|
104 |
-
## Installing the nightly version to get most recent updates
|
105 |
-
```
|
106 |
-
pip install git+https://github.com/EleutherAI/lm-evaluation-harness
|
107 |
-
```
|
108 |
-
|
109 |
## baseline
|
110 |
```
|
111 |
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 8
|
@@ -116,9 +111,6 @@ lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks
|
|
116 |
lm_eval --model hf --model_args pretrained=pytorch/Phi-4-mini-instruct-float8dq --tasks hellaswag --device cuda:0 --batch_size 8
|
117 |
```
|
118 |
|
119 |
-
`TODO: more complete eval results`
|
120 |
-
|
121 |
-
|
122 |
| Benchmark | | |
|
123 |
|----------------------------------|----------------|---------------------|
|
124 |
| | Phi-4 mini-Ins | phi4-mini-int4wo |
|
@@ -155,12 +147,6 @@ lm_eval --model hf --model_args pretrained=pytorch/Phi-4-mini-instruct-float8dq
|
|
155 |
|
156 |
Note the result of latency (benchmark_latency) is in seconds, and serving (benchmark_serving) is in number of requests per second.
|
157 |
|
158 |
-
## Download vllm source code and install vllm
|
159 |
-
```
|
160 |
-
git clone [email protected]:vllm-project/vllm.git
|
161 |
-
VLLM_USE_PRECOMPILED=1 pip install .
|
162 |
-
```
|
163 |
-
|
164 |
## Download dataset
|
165 |
Download sharegpt dataset: `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json`
|
166 |
|
|
|
101 |
# Model Quality
|
102 |
We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate the quality of the quantized model.
|
103 |
|
|
|
|
|
|
|
|
|
|
|
104 |
## baseline
|
105 |
```
|
106 |
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 8
|
|
|
111 |
lm_eval --model hf --model_args pretrained=pytorch/Phi-4-mini-instruct-float8dq --tasks hellaswag --device cuda:0 --batch_size 8
|
112 |
```
|
113 |
|
|
|
|
|
|
|
114 |
| Benchmark | | |
|
115 |
|----------------------------------|----------------|---------------------|
|
116 |
| | Phi-4 mini-Ins | phi4-mini-int4wo |
|
|
|
147 |
|
148 |
Note the result of latency (benchmark_latency) is in seconds, and serving (benchmark_serving) is in number of requests per second.
|
149 |
|
|
|
|
|
|
|
|
|
|
|
|
|
150 |
## Download dataset
|
151 |
Download sharegpt dataset: `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json`
|
152 |
|