File size: 4,420 Bytes
4d9810e 833888d ea9a4cd 2a74db6 833888d ea9a4cd 4d9810e 833888d ea9a4cd 833888d 2a74db6 833888d f0db376 833888d ea9a4cd 833888d cbe7f49 833888d ea9a4cd 833888d 9877e68 4d9810e 833888d 4d9810e ea9a4cd 833888d 9877e68 4d9810e 2a74db6 833888d 4d9810e ea9a4cd 833888d ea9a4cd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
---
base_model: meta-llama/Llama-2-7b-chat-hf
tags:
- generated_from_trainer
- trl
metrics:
- accuracy
model-index:
- name: llama-2-7b-reward-oasst1
results: []
datasets:
- tasksource/oasst1_pairwise_rlhf_reward
library_name: peft
---
# llama-2-7b-reward-oasst1
This model is a fine-tuned version of [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) on the first 10000 rows of the [tasksource/oasst1_pairwise_rlhf_reward](https://huggingface.co/datasets/tasksource/oasst1_pairwise_rlhf_reward) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5713
- Accuracy: 0.7435
See also [vincentmin/llama-2-13b-reward-oasst1](https://huggingface.co/vincentmin/llama-2-13b-reward-oasst1) for a 13b version of this model.
## Model description
This is a reward model trained with QLoRA in 4bit precision. The base model is [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) for which you need to have accepted the license in order to be able use it. Once you've been given permission, you can load the reward model as follows:
```
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer
peft_model_id = "vincentmin/llama-2-7b-reward-oasst1"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForSequenceClassification.from_pretrained(
config.base_model_name_or_path,
num_labels=1,
load_in_4bit=True,
torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path, use_auth_token=True)
model.eval()
with torch.no_grad():
reward = model(**tokenizer("prompter: hello world. assistant: foo bar", return_tensors='pt')).logits
reward
```
For best results, one should use the prompt format used during training:
```
prompt = "prompter: <prompt_1> assistant: <response_1> prompter: <prompt_2> ..."
```
Please use a version of peft where [#755](https://github.com/huggingface/peft/pull/755) has been merged to make sure the model is loaded correctly. You can install `peft` with `pip install git+https://github.com/huggingface/peft.git` to make sure this is the case.
## Intended uses & limitations
Since the model was trained on oasst1 data, the reward will reflect any biases present in the oasst1 data.
## Training and evaluation data
The model was trained using QLoRA and the `trl` library's `RewardTrainer` on the [tasksource/oasst1_pairwise_rlhf_reward](https://huggingface.co/datasets/tasksource/oasst1_pairwise_rlhf_reward) dataset. Examples with more than 1024 tokens were filtered out and the training data was restricted to the first 10000 rows of the filtered dataset.
### Training hyperparameters
The following `bitsandbytes` quantization config was used during training:
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float16
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- max_seq_length: 1024
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.8409 | 0.1 | 250 | 0.8243 | 0.6220 |
| 0.6288 | 0.2 | 500 | 0.7539 | 0.6715 |
| 0.5882 | 0.3 | 750 | 0.6792 | 0.7075 |
| 0.7671 | 0.4 | 1000 | 0.6130 | 0.7334 |
| 0.5782 | 0.5 | 1250 | 0.6115 | 0.7255 |
| 0.5691 | 0.6 | 1500 | 0.5795 | 0.7413 |
| 0.6579 | 0.7 | 1750 | 0.5774 | 0.7469 |
| 0.6107 | 0.8 | 2000 | 0.5691 | 0.7402 |
| 0.6255 | 0.9 | 2250 | 0.5710 | 0.7435 |
| 0.7034 | 1.0 | 2500 | 0.5713 | 0.7435 |
### Framework versions
- PEFT 0.5.0.dev0 (with https://github.com/huggingface/peft/pull/755)
- Transformers 4.32.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.14.0
- Tokenizers 0.13.3 |