File size: 9,362 Bytes
9dc5119
 
6d3dd74
9dc5119
365f858
5b7191a
 
 
 
 
365f858
5b7191a
 
365f858
5b7191a
 
365f858
6d3dd74
 
 
81e2785
6d3dd74
81e2785
6d3dd74
 
 
81e2785
 
6d3dd74
 
 
 
 
 
 
 
 
 
 
 
 
 
05c51c6
 
 
39878e0
05c51c6
39878e0
05c51c6
39878e0
 
 
5b7191a
05c51c6
6b3d216
39878e0
05c51c6
39878e0
05c51c6
 
 
 
 
5b7191a
05c51c6
 
d29e3fb
05c51c6
39878e0
 
 
 
 
 
 
d29e3fb
 
5b7191a
74ae984
6d3dd74
1dd6eed
6d3dd74
 
1dd6eed
6d3dd74
 
1dd6eed
dd08d1c
1dd6eed
39878e0
1dd6eed
 
 
5b7191a
6d3dd74
 
05c51c6
6d3dd74
 
 
05c51c6
6d3dd74
 
 
 
 
 
 
 
 
 
 
 
 
 
 
05c51c6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6d3dd74
 
 
 
05c51c6
6d3dd74
365f858
 
5b7191a
365f858
5b7191a
365f858
5b7191a
365f858
5b7191a
365f858
 
 
 
 
 
 
 
 
5b7191a
365f858
 
 
 
 
6d3dd74
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5b7191a
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
---
license: other
inference: false
---
<!-- header start -->
<div style="width: 100%;">
    <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
</div>
<div style="display: flex; justify-content: space-between; width: 100%;">
    <div style="display: flex; flex-direction: column; align-items: flex-start;">
        <p><a href="https://discord.gg/Jq4vkcDakD">Chat & support: my new Discord server</a></p>
    </div>
    <div style="display: flex; flex-direction: column; align-items: flex-end;">
        <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
    </div>
</div>
<!-- header end -->

# OpenAssistant LLaMA 30B SFT 7 GPTQ

This in a repo of GPTQ format 4bit quantised models for [OpenAssistant's LLaMA 30B SFT 7](https://huggingface.co/OpenAssistant/oasst-sft-7-llama-30b-xor).

It is the result of merging the XORs from the above repo with the original Llama 30B weights, and then quantising to 4bit GPU inference using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).

This is epoch 7 of OpenAssistant's training of their Llama 30B model.

**Please note that these models will need 24GB VRAM or greater to use effectively**

## Repositories available

* [4bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/OpenAssistant-SFT-7-Llama-30B-GPTQ).
* [4bit and 5bit GGML models for CPU inference](https://huggingface.co/TheBloke/OpenAssistant-SFT-7-Llama-30B-GGML).
* [Unquantised 16bit model in HF format](https://huggingface.co/TheBloke/OpenAssistant-SFT-7-Llama-30B-HF).

## PROMPT TEMPLATE

This model requires the following prompt template:

```
<|prompter|> prompt goes here
<|assistant|>:
```

## CHOICE OF MODELS

Three sets of models are provided:

* Groupsize = None
  * Should work reliably in 24GB VRAM
  * Uses --act-order for the best possible inference quality given its lack of group_size.
* Groupsize = 1024
  * Theoretically higher inference accuracy
  * May OOM on long context lengths in 24GB VRAM
* Groupsize = 128
  * Optimal setting for highest inference quality
  * Will definitely need more than 24GB VRAM on longer context lengths (1000-1500+ tokens returned)

For the 128g and 1024g models, two versions are available:
* `compat.no-act-order.safetensor`
  * Works with all versions of GPTQ-for-LLaMa, including the version in text-generation-webui one-click-installers
* `latest.act-order.safetensors`
  * uses `--act-order` for higher inference quality
  * requires more recent GPTQ-for-LLaMa code, therefore will not currently work with one-click-installers

## HOW TO CHOOSE YOUR MODEL

I have used branches to separate the models. This means you can clone the branch you want and not got model files you don't need.

If you have 24GB VRAM you are strongly recommended to use the file in `main`, with group_size = None.  This is fully compatible, and won't OOM.

* Branch: **main** = groupsize None, `OpenAssistant-SFT-7-Llama-30B-GPTQ-4bit.safetensors` file
* Branch: **1024-compat** = groupsize 1024, `compat.no-act-order.safetensors` file
* Branch: **1024-latest** = groupsize 1024, `latest.act-order.safetensors` file
* Branch: **128-compat** = groupsize 128, `compat.no-act-order.safetensors` file
* Branch: **128-latest** = groupsize 128, `latest.act-order.safetensors` file

![branches](https://i.imgur.com/PdiHnLxm.png)

## How to easily download and run the 1024g compat model in text-generation-webui

Open the text-generation-webui UI as normal.

1. Click the **Model tab**.
2. Under **Download custom model or LoRA**, enter `TheBloke/OpenAssistant-SFT-7-Llama-30B-GPTQ`.
3. Click **Download**.
4. Wait until it says it's finished downloading.
5. Click the **Refresh** icon next to **Model** in the top left.
6. In the **Model drop-down**: choose the model you just downloaded, `OpenAssistant-SFT-7-Llama-30B-GPTQ`.
7. If you see an error in the bottom right, ignore it - it's temporary.
8. Fill out the `GPTQ parameters` on the right: `Bits = 4`, `Groupsize = None`, `model_type = Llama`
9. Click **Save settings for this model** in the top right.
10. Click **Reload the Model** in the top right.
11. Once it says it's loaded, click the **Text Generation tab** and enter a prompt!

## Manual instructions for `text-generation-webui`

The `compat.no-act-order.safetensors` files can be loaded the same as any other GPTQ file, without requiring any updates to [oobaboogas text-generation-webui](https://github.com/oobabooga/text-generation-webui).

[Instructions on using GPTQ 4bit files in text-generation-webui are here](https://github.com/oobabooga/text-generation-webui/wiki/GPTQ-models-\(4-bit-mode\)).

The `latest.act-order.safetensors` files were created using `--act-order` to give the maximum possible quantisation quality, but this means it requires that the latest GPTQ-for-LLaMa is used inside the UI.

If you want to use the act-order `safetensors` files and need to update the Triton branch of GPTQ-for-LLaMa, here are the commands I used to clone the Triton branch of GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:
```
# Clone text-generation-webui, if you don't already have it
git clone https://github.com/oobabooga/text-generation-webui
# Make a repositories directory
mkdir text-generation-webui/repositories
cd text-generation-webui/repositories
# Clone the latest GPTQ-for-LLaMa code inside text-generation-webui
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
```

Then install this model into `text-generation-webui/models` and launch the UI as follows:
```
cd text-generation-webui
python server.py --model OpenAssistant-SFT-7-Llama-30B-GPTQ --wbits 4 --groupsize 128 --model_type Llama # add any other command line args you want
```

To update the CUDA branch of GPTQ-for-LLaMa, you can do the following. **This requires a C/C++ compiler and the CUDA toolkit installed!**
```
# Clone text-generation-webui, if you don't already have it
git clone https://github.com/oobabooga/text-generation-webui
# Make a repositories directory
mkdir text-generation-webui/repositories
cd text-generation-webui/repositories
# Clone the latest GPTQ-for-LLaMa code inside text-generation-webui
git clone -b cuda https://github.com/qwopqwop200/GPTQ-for-LLaMa
cd GPTQ-for-LLaMa
pip uninstall quant-cuda # uninstall existing CUDA version
python setup_cuda.py install # install latest version
```

The above commands assume you have installed all dependencies for GPTQ-for-LLaMa and text-generation-webui. Please see their respective repositories for further information.

If you can't update GPTQ-for-LLaMa or don't want to, please use a `compat.no-act-order.safetensor` file.

<!-- footer start -->
## Discord

For further support, and discussions on these models and AI in general, join us at:

[TheBloke AI's Discord server](https://discord.gg/Jq4vkcDakD)

## Thanks, and how to contribute.

Thanks to the [chirper.ai](https://chirper.ai) team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

* Patreon: https://patreon.com/TheBlokeAI
* Ko-Fi: https://ko-fi.com/TheBlokeAI

**Patreon special mentions**: Aemon Algiz, Dmitriy Samsonov, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, Jonathan Leane, Talal Aujan, V. Lukas, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Sebastain Graf, Johann-Peter Hartman.

Thank you to all my generous patrons and donaters!
<!-- footer end -->
# Original model card

```
llama-30b-sft-7:
  dtype: fp16
  log_dir: "llama_log_30b"
  learning_rate: 1e-5
  model_name: /home/ubuntu/Open-Assistant/model/model_training/.saved/llama-30b-super-pretrain/checkpoint-3500
  #model_name: OpenAssistant/llama-30b-super-pretrain
  output_dir: llama_model_30b
  deepspeed_config: configs/zero3_config_sft.json
  weight_decay: 0.0
  residual_dropout: 0.0
  max_length: 2048
  use_flash_attention: true
  warmup_steps: 20
  gradient_checkpointing: true
  gradient_accumulation_steps: 12
  per_device_train_batch_size: 2
  per_device_eval_batch_size: 3
  eval_steps: 101
  save_steps: 485
  num_train_epochs: 4
  save_total_limit: 3
  use_custom_sampler: true
  sort_by_length: false
  #save_strategy: steps
  save_strategy: epoch
  datasets:
    - oasst_export:
        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
        input_file_path: 2023-04-12_oasst_release_ready_synth.jsonl.gz
        val_split: 0.05
    - vicuna:
        val_split: 0.05
        max_val_set: 800
        fraction: 1.0
    - dolly15k:
        val_split: 0.05
        max_val_set: 300
    - grade_school_math_instructions:
        val_split: 0.05
    - code_alpaca:
        val_split: 0.05
        max_val_set: 250
```

- **OASST dataset paper:** https://arxiv.org/abs/2304.07327