Update README.md
Browse files
README.md
CHANGED
@@ -17,11 +17,11 @@ base_model:
|
|
17 |
pipeline_tag: text-generation
|
18 |
---
|
19 |
|
20 |
-
[Phi4-mini](https://huggingface.co/microsoft/Phi-4-mini-instruct)
|
21 |
|
22 |
# Quantization Recipe
|
23 |
|
24 |
-
|
25 |
```
|
26 |
pip install git+https://github.com/huggingface/transformers@main
|
27 |
pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu126
|
@@ -29,7 +29,7 @@ pip install torch
|
|
29 |
pip install accelerate
|
30 |
```
|
31 |
|
32 |
-
|
33 |
```
|
34 |
import torch
|
35 |
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig
|
@@ -159,7 +159,9 @@ We can use the following code to get a sense of peak memory usage during inferen
|
|
159 |
| Peak Memory (GB) | 8.91 | 2.98 (67% reduction) |
|
160 |
|
161 |
|
162 |
-
##
|
|
|
|
|
163 |
|
164 |
```
|
165 |
import torch
|
@@ -203,9 +205,7 @@ print(f"Peak Memory Usage: {mem:.02f} GB")
|
|
203 |
|
204 |
# Model Performance
|
205 |
|
206 |
-
Our int4wo is only optimized for batch size 1, so
|
207 |
-
and decode tokens per second will be more important than time to first token.
|
208 |
-
|
209 |
|
210 |
## Results (A100 machine)
|
211 |
| Benchmark (Latency) | | |
|
|
|
17 |
pipeline_tag: text-generation
|
18 |
---
|
19 |
|
20 |
+
[Phi4-mini](https://huggingface.co/microsoft/Phi-4-mini-instruct) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
|
21 |
|
22 |
# Quantization Recipe
|
23 |
|
24 |
+
Install the required packages:
|
25 |
```
|
26 |
pip install git+https://github.com/huggingface/transformers@main
|
27 |
pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu126
|
|
|
29 |
pip install accelerate
|
30 |
```
|
31 |
|
32 |
+
Use the following code to get the quantized model:
|
33 |
```
|
34 |
import torch
|
35 |
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig
|
|
|
159 |
| Peak Memory (GB) | 8.91 | 2.98 (67% reduction) |
|
160 |
|
161 |
|
162 |
+
## Peak Memory
|
163 |
+
|
164 |
+
We can use the following code to get a sense of peak memory usage during inference:
|
165 |
|
166 |
```
|
167 |
import torch
|
|
|
205 |
|
206 |
# Model Performance
|
207 |
|
208 |
+
Our int4wo is only optimized for batch size 1, so expect some slowdown with larger batch sizes, we expect this to be used in local server deployment for single or a few users where the decode tokens per second will matters more than the time to first token.
|
|
|
|
|
209 |
|
210 |
## Results (A100 machine)
|
211 |
| Benchmark (Latency) | | |
|