File size: 1,245 Bytes
07c718f f5e8289 07c718f f5e8289 b70666d f5e8289 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
---
license: mit
datasets:
- databricks/databricks-dolly-15k
language:
- en
pipeline_tag: text-generation
tags:
- dolly
- dolly-v2
- instruct
- sharded
inference: False
---
# dolly-v2-12b: sharded checkpoint
<a href="https://colab.research.google.com/gist/pszemraj/6eb7ccce28ea6aa07b8ec86388ac010e/sharded-instruction-model-playground.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
This is a sharded checkpoint (with ~4GB shards) of the `databricks/dolly-v2-12b` model. Refer to the [original model](https://huggingface.co/databricks/dolly-v2-12b) for all details.
- this enables low-RAM loading, i.e. Colab :)
## Basic Usage
install `transformers`, `accelerate`, and `bitsandbytes`.
```bash
pip install -U -q transformers bitsandbytes accelerate
```
Load the model in 8bit, then [run inference](https://huggingface.co/docs/transformers/generation_strategies#contrastive-search):
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "ethzanalytics/dolly-v2-12b-sharded"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name, load_in_8bit=True, device_map="auto",
)
``` |