Update README.md

README.md

@@ -21,8 +21,8 @@ tags:
 
 <!-- Provide a quick summary of what the model is/does. -->
 
-SteamSHP-Large is a preference model trained to predict human preferences, given some context and two possible responses.
-It can be used for NLG evaluation or to train a smaller reward model for RLHF.
+SteamSHP-Large is a preference model trained to predict -- given some context and two possible responses -- which response humans will find more helpful. 
+It can be used for NLG evaluation, question-answering evalation, or to train a smaller reward model for RLHF.
 
 It is a FLAN-T5-large model (780M parameters) finetuned on:
 1. The [Stanford Human Preferences Dataset (SHP)](https://huggingface.co/datasets/stanfordnlp/SHP), which contains collective human preferences sourced from 18 different communities on Reddit (e.g., `askculinary`, `legaladvice`, etc.).
@@ -106,13 +106,18 @@ SteamSHP-Large gets an average 72.0% accuracy across all domains:
 
 
 
-### Biases and Limitations
+## Biases and Limitations
 
-Biases in the datasets used to train SteamSHP-Large may be propagated downstream to the model predictions.
+SteamSHP is trained to predict which of two responses humans will find *more helpful*, not which response is *less harmful*.
+It should not be used to detect toxicity, make ethical judgments, or for a similar purpose.
+
+Biases and misinformation in the datasets used to train SteamSHP may also be propagated downstream to the model predictions.
 Although SHP filtered out posts with NSFW (over 18) content, chose subreddits that were well-moderated and had policies against harassment and bigotry, some of the data may contain discriminatory or harmful language.
-Reddit users on the subreddits covered by SHP are also not representative of the broader population. They are disproportionately from developed, Western, and English-speaking countries. 
+The responses that humans collectively found more helpful are also not guaranteed to be more factual.
 
-It is also worth noting that the more preferred response in SHP or HH-RLHF is not necessarily the more correct one -- the data just reflects the collective preference of Reddit users (in SHP's case) and individuals' preferences (in HH-RLHF's case).
+The people whose preferences are captured in SHP and HH-RLHF are not representative of the broader population. 
+Although specific demographic information is not available, overall, the Reddit users whose preferences are captured in SHP are disproportionately male and from developed, Western, and English-speaking countries (Pew Research).
+ 
 [Past work](https://www.anthropic.com/model-written-evals.pdf) by Anthropic has found that models optimized for human preference can be obsequious, at the expense of the truth.