File size: 1,856 Bytes
ef19c52 ed41f8e e9d6045 ef19c52 ed41f8e 03e1550 ed41f8e dcd2ef9 ed41f8e 3ca9d76 ed41f8e a69aed2 ed41f8e a69aed2 ed41f8e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
---
license: mit
datasets:
- databricks/databricks-dolly-15k
language:
- en
pipeline_tag: text-generation
tags:
- dolly
- dolly-v2
- instruct
- sharded
- 8bit
- quantized
inference: false
---
# dolly-v2-12b: sharded **8bit** checkpoint
<a href="https://colab.research.google.com/gist/pszemraj/1bc9cea67e6c8dc450b868e0cfc18163/dolly-v2-12b-8bit-inference.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
This is a sharded checkpoint (with ~4GB shards) of the `databricks/dolly-v2-12b` model **in `8bit` precision** using `bitsandbytes`.
Refer to the [original model](https://huggingface.co/databricks/dolly-v2-12b) for all details w.r.t. to the model. For more info on loading 8bit models, refer to the [example repo](https://huggingface.co/ybelkada/bloom-1b7-8bit) and/or the `4.28.0` [release info](https://github.com/huggingface/transformers/releases/tag/v4.28.0).
- total model size is only ~12.5 GB!
- this enables low-RAM loading, i.e. Colab :)
- **update**: generation speed can be greatly improved by setting `use_cache=True` and generating via contrastive search. [example notenook here](https://colab.research.google.com/gist/pszemraj/12c832952c88d77f6924c0718a2d257d/dolly-v2-12b-8bit-use_cache-bettertransformer.ipynb)
## Basic Usage
install/upgrade `transformers`, `accelerate`, and `bitsandbytes`. For this to work **you must have** `transformers>=4.28.0` and `bitsandbytes>0.37.2`.
```bash
pip install -U -q transformers bitsandbytes accelerate
```
Load the model. As it is serialized in 8bit you don't need to do anything special:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "ethzanalytics/dolly-v2-12b-sharded-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
``` |