File size: 2,130 Bytes
5bd90d1 b2d16a0 5bd90d1 35b0e46 5bd90d1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
---
library_name: transformers
tags:
- phi-3
- phi-3-medium
- phi-3-medium-4k-instruct
- conversational
- text-generation-inference
pipeline_tag: text-generation
language:
- en
---
Official quantization of [microsoft/Phi-3-medium-4k-instruct](Phi-3-medium-4k-instruct](https://huggingface.co/microsoft/Phi-3-medium-4k-instruct) using [PV-Tuning](https://arxiv.org/abs/2405.14852) on top of [AQLM](https://arxiv.org/abs/2401.06118).
For this quantization, we used 1 codebook of 16 bits for groups of 8 weights.
Results (0-shot `acc`):
Results:
| Model | Quantization | WikiText-2 | C4 | Model size, Gb |
|------|------|-------|------|------|
| [microsoft/Phi-3-medium-4k-instruct](https://huggingface.co/microsoft/Phi-3-medium-4k-instruct) | None | | | 27.9 |
| | [1x16g8 (2-bit, this model)](https://huggingface.co/ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-2Bit-1x16-hf) | 5.18 | 8.56 | 4.2Gb |
| | [1x16g16 (1-bit, model link)](https://huggingface.co/ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-1Bit-1x16-hf) | 7.42 | 10.40 | 2.7Gb |
Phi-3-**medium** is not included in the original [PV-Tuining paper](https://arxiv.org/abs/2405.14852). As of yet, we did not have the bandwidth to evaluate it properly. We hope to eventually run the zero-shot evaluation suite, or you can help us by running it yourself and opening a pull-request to the readme!
In general, we always recommend the 2-bit models for best accuracy-size trade-offs. If tempted to use the 1-bit model, try a smaller model ,
e.g. Phi-3-**mini** quantized with AQLM+PV [(quantized model link)](https://huggingface.co/ISTA-DASLab/Phi-3-mini-4k-instruct-AQLM-PV-2Bit-1x16-hf) and compare the results, or check our [AQLM+PV collection](https://huggingface.co/collections/ISTA-DASLab/aqlmpv-66564dff5d84f00a893ba93f) for a more appropriate size.
To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the [official GitHub repo](https://github.com/Vahe1994/AQLM).
The original code for PV-Tuning can be found in the [AQLM@pv-tuning](https://github.com/Vahe1994/AQLM/tree/pv-tuning) branch. |