metascroy commited on
Commit
0b831a8
·
verified ·
1 Parent(s): 0851691

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -2
README.md CHANGED
@@ -156,7 +156,17 @@ lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks
156
 
157
  ## int8 dynamic activation and int4 weight quantization (8da4w)
158
  ```
159
- lm_eval --model hf --model_args pretrained=pytorch/Phi-4-mini-instruct-8da4w --tasks hellaswag --device cuda:0 --batch_size 64
 
 
 
 
 
 
 
 
 
 
160
  ```
161
 
162
  | Benchmark | | |
@@ -187,7 +197,7 @@ We can run the quantized model on a mobile phone using [ExecuTorch](https://gith
187
  Once ExecuTorch is [set-up](https://pytorch.org/executorch/main/getting-started.html), exporting and running the model on device is a breeze.
188
 
189
  We first convert the quantized checkpoint to one ExecuTorch's LLM export script expects by renaming some of the checkpoint keys.
190
- The following script does this for you.
191
  ```
192
  python -m executorch.examples.models.phi_4_mini.convert_weights pytorch_model.bin phi4-mini-8da4w-converted.bin
193
  ```
 
156
 
157
  ## int8 dynamic activation and int4 weight quantization (8da4w)
158
  ```
159
+ import lm_eval
160
+ from lm_eval import evaluator
161
+ from lm_eval.utils import (
162
+ make_table,
163
+ )
164
+
165
+ lm_eval_model = lm_eval.models.huggingface.HFLM(pretrained=quantized_model, batch_size=64)
166
+ results = evaluator.simple_evaluate(
167
+ lm_eval_model, tasks=["hellaswag"], device="cuda:0", batch_size="auto"
168
+ )
169
+ print(make_table(results))
170
  ```
171
 
172
  | Benchmark | | |
 
197
  Once ExecuTorch is [set-up](https://pytorch.org/executorch/main/getting-started.html), exporting and running the model on device is a breeze.
198
 
199
  We first convert the quantized checkpoint to one ExecuTorch's LLM export script expects by renaming some of the checkpoint keys.
200
+ The following script does this for you. We have uploaded phi4-mini-8da4w-converted.bin here for convenience.
201
  ```
202
  python -m executorch.examples.models.phi_4_mini.convert_weights pytorch_model.bin phi4-mini-8da4w-converted.bin
203
  ```