Hugging Face's logo

nvidia
/
Llama3-70B-SteerLM-RM

NeMo
English
nvidia
steerlm
llama3
reward model
Model card Files Community
4
Llama3-70B-SteerLM-RM / README.md
zhilinw's picture
zhilinw
Update README.md
2992f66 verified
preview code
|
raw
history blame
7.91 kB
metadata
license: llama3
library_name: nemo
language:
  - en
inference: false
fine-tuning: false
tags:
  - nvidia
  - steerlm
  - llama3
  - reward model
datasets:
  - nvidia/HelpSteer2

Llama3-70B-SteerLM-RM

License

The use of this model is governed by the Llama 3 Community License Agreement

Description:

Llama3-70B-SteerLM-RM is a 70 billion parameter language model (with context of up to 8,192 tokens) used as an Attribute Prediction Model, a multi-aspect Reward Model that rates model responses on various aspects that makes a response desirable instead of a singular score in a conventional Reward Model.

Given a conversation with multiple turns between user and assistant, it rates the following attributes (between 0 and 4) for every assistant turn.

  1. Helpfulness: Overall helpfulness of the response to the prompt.
  2. Correctness: Inclusion of all pertinent facts without errors.
  3. Coherence: Consistency and clarity of expression.
  4. Complexity: Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).
  5. Verbosity: Amount of detail included in the response, relative to what is asked for in the prompt.

Nonetheless, if you are only interested in using it as a conventional reward model that outputs a singular scalar, we recommend using the weights [0, 0, 0, 0, 0.65, 0.8, 0.45, 0.55, -0.4] to do elementwise multiplication with the predicted attributes (which outputs 9 float values in line with Llama2-13B-SteerLM-RM but the first four are not trained or used)

Llama3-70B-SteerLM-RM is trained from Llama 3 70B Base with the HelpSteer2 dataset

HelpSteer Paper : HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM

SteerLM Paper: SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

Llama3-70B-SteerLM-RM is trained with NVIDIA NeMo-Aligner, a scalable toolkit for performant and efficient model alignment. NeMo-Aligner is built using the NeMo Framework which allows for scaling training with data and model parallelism for all components of alignment. All of our checkpoints are compatible with the NeMo ecosystem, allowing for inference deployment and further customization.

RewardBench Primary Dataset LeaderBoard

Model Type of Model Overall Chat Chat-Hard Safety Reasoning
Nemotron-4-340B-RM Trained with Permissive Licensed Data 92.0 95.8 87.1 91.5 93.7
ArmoRM-Llama3-8B-v0.1 Trained with GPT4 Generated Data 90.8 96.9 76.8 92.2 97.3
Cohere May 2024 Proprietary LLM 89.5 96.4 71.3 92.7 97.7
Llama3-70B-SteerLM-RM Trained with Permissive Licensed Data 88.8 91.3 80.3 92.8 90.7
Google Gemini Pro 1.5 Proprietary LLM 88.1 92.3 80.6 87.5 92.0
RLHFlow-Llama3-8B Trained with GPT4 Generated Data 87.1 98.3 65.8 89.7 94.7
Cohere March 2024 Proprietary LLM 87.1 94.7 65.1 90.3 98.7
GPT-4-0125-Preview Proprietary LLM 85.9 95.3 74.3 87.2 86.9
Claude 3 Opus 0229 Proprietary LLM 80.7 94.7 60.3 89.1 78.7
Llama3 70B Instruct Trained with Permissive Licensed Data 76.0 97.6 58.9 69.2 78.5

Last updated: 12 Jun 2024

Note that we only consider the first four categories in RewardBench, because the optional fifth category (Prior Sets) is

  1. Heavily towards models trained on Anthropic HHH, Anthropic Helpful, OpenAI Summarize and Stanford Human Preferences (constituent datasets for the Prior Sets category) and therefore can be easily gamed (see About page on RewardBench)
  2. Extremely noisy with many constituent datasets (e.g. Anthropic Helpful, OpenAI summarize) not being able to reach val accuracy beyond ~0.70 even if training on the training set alone, suggesting unchecked errors in annotation (see https://arxiv.org/abs/2401.06080 for details)
  3. Not reported by several models such as Google Gemini Pro 1.5 and Claude 3 Opus 0229, making comparisons unfair since Prior Sets typically has lower scores than other categories

Usage:

You can use the model with NeMo Aligner following SteerLM training user guide.

  1. Spin up an inference server within the NeMo Aligner container
python /opt/NeMo-Aligner/examples/nlp/gpt/serve_reward_model.py \
      rm_model_file=Llama3-70B-SteerLM-RM \
      trainer.num_nodes=1 \
      trainer.devices=8 \
      ++model.tensor_model_parallel_size=8 \
      ++model.pipeline_model_parallel_size=1 \
      inference.micro_batch_size=2 \
      inference.port=1424
  1. Annotate data files using the served reward model. As an example, this can be the Open Assistant train/val files. Then follow the next step to train a SteerLM model based on SteerLM training user guide .
python /opt/NeMo-Aligner/examples/nlp/data/steerlm/preprocess_openassistant_data.py --output_directory=data/oasst

python /opt/NeMo-Aligner/examples/nlp/data/steerlm/attribute_annotate.py \
      --input-file=data/oasst/train.jsonl \
      --output-file=data/oasst/train_labeled.jsonl \
      --port=1424
  1. Alternatively, this can be any conversational data file (in .jsonl) in the following format, where each line looks like
{
    "conversations": [
              {"value": <user_turn_1>, "from": "User", "label": None},
              {"value": <assistant_turn_1>, "from": "Assistant", "label": <formatted_label_1>},
              {"value": <user_turn_2>, "from": "User", "label": None},
              {"value": <assistant_turn_2>, "from": "Assistant", "label": <formatted_label_2>},
          ],
    "mask": "User"
}

Ideally, each <formatted_label_n> refers to the ground truth label for the assistant turn but if they are not available, we can also use helpfulness:4,correctness:4,coherence:4,complexity:2,verbosity:2 (i.e. defaulting to moderate complexity and verbosity, adjust if needed. or simply helpfulness:-1. It must not be None or an empty string.

Contact

E-Mail: Zhilin Wang

Citation

If you find this dataset useful, please cite the following works

@misc{wang2023helpsteer,
      title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM}, 
      author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev},
      year={2023},
      eprint={2311.09528},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{dong2023steerlm,
      title={SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF}, 
      author={Yi Dong and Zhilin Wang and Makesh Narsimhan Sreedhar and Xianchao Wu and Oleksii Kuchaiev},
      year={2023},
      eprint={2310.05344},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}