---
datasets:
  - PKU-Alignment/PKU-SafeRLHF
language:
  - en
tags:
  - reinforcement-learning-from-human-feedback
  - reinforcement-learning
  - beaver
  - safety
  - llama
  - ai-safety
  - deepspeed
  - rlhf
  - alpaca
library_name: safe-rlhf
---

# 🦫 Beaver's Cost Model

## Model Details

The Beaver cost model is a preference model trained using the [PKU-SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) dataset.
It can play a role in the safe RLHF algorithm, helping the Beaver model become more safe and harmless.

- **Developed by:** the [PKU-Alignment](https://github.com/PKU-Alignment) Team.
- **Model Type:** An auto-regressive language model based on the transformer architecture.
- **License:** Non-commercial license.
- **Fine-tuned from model:** [LLaMA](https://arxiv.org/abs/2302.13971), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca).

## Model Sources

- **Repository:** <https://github.com/PKU-Alignment/safe-rlhf>
- **Beaver:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0>
- **Dataset:** <https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF>
- **Reward Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-reward>
- **Cost Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-cost>
- **Dataset Paper:** <https://arxiv.org/abs/2307.04657>
- **Paper:** <https://arxiv.org/abs/2310.12773>

## How to Use the Cost Model

```python
import torch
from transformers import AutoTokenizer
from safe_rlhf.models import AutoModelForScore

model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-v3.0-cost', torch_dtype=torch.bfloat16, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-v3.0-cost')

input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?'

input_ids = tokenizer(input, return_tensors='pt')
output = model(**input_ids)
print(output)

# ScoreModelOutput(
#     scores=tensor([[[ 3.4844],
#          [ 0.9414],
#          [ 1.9766],
#          [ 0.9688],
#          [ 1.4219],
#          [ 0.5781],
#          [ 0.7500],
#          [ 0.3516],
#          [-0.2305],
#          [-0.6055],
#          [-1.0625],
#          [-1.1875],
#          [-0.5820],
#          [ 0.0182],
#          [-1.0000],
#          [ 0.1279],
#          [-0.5820],
#          [-0.3691],
#          [ 0.5430],
#          [-0.2266],
#          [ 0.6797],
#          [ 1.0938],
#          [ 0.7188],
#          [ 0.6797],
#          [ 0.3613],
#          [ 0.1416],
#          [ 0.4238],
#          [ 0.4023]]], grad_fn=<ToCopyBackward0>),
#     end_scores=tensor([[0.4023]], grad_fn=<ToCopyBackward0>),
#     last_hidden_state=tensor([[[-0.2832, -0.0139, -0.1904,  ...,  0.4141, -0.5859, -1.2734],
#          [ 0.2168, -1.1953, -0.4707,  ..., -0.0806,  0.2500,  0.6016],
#          [ 0.5078,  0.2334,  0.1348,  ..., -0.1416, -0.1699, -0.3008],
#          ...,
#          [ 0.6328, -0.0108, -0.7188,  ..., -0.8828,  0.2812,  0.5352],
#          [ 0.4434,  0.3281, -0.1245,  ..., -0.7812,  0.7734,  0.8164],
#          [ 0.5078,  0.2637,  0.5508,  ...,  0.3477,  1.5625,  0.5547]]],
#        dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
#     end_last_hidden_state=tensor([[0.5078, 0.2637, 0.5508,  ..., 0.3477, 1.5625, 0.5547]],
#        dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
#     end_index=tensor([27])
# )
```