Update README.md

README.md

@@ -1,3 +1,136 @@
 ---
 license: apache-2.0
+datasets:
+- stanfordnlp/SHP
+language:
+- en
+metrics:
+- accuracy
+tags:
+- human feedback
+- rlhf
+- preferences
+- reddit
+- preference model
+- RL
+- NLG
+- evaluation
 ---
+
+# 💨🚢 SteamSHP-Large
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+SteamSHP-Large is a preference model trained to predict human preferences, given some context and two possible responses.
+It can be used for NLG evaluation or to train a smaller reward model for RLHF.
+
+It is a FLAN-T5-large model (780M parameters) finetuned on:
+1. The [Stanford Human Preferences Dataset (SHP)](https://huggingface.co/datasets/stanfordnlp/SHP), which contains aggregate human preferences sourced from 18 different communities on Reddit (e.g., `askculinary`, `legaladvice`, etc.).
+2. The helpfulness data in [Anthropic's HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.
+
+There is a larger variant called [SteamSHP-XL](https://huggingface.co/stanfordnlp/SteamSHP-flan-t5-xl) that was made by finetuning FLAN-T5-xl (3B parameters), which is 0.75 percentage points more accurate on the test data.
+
+
+## Usage
+
+The input text should be of the format:
+
+```
+POST: { the context, such as the 'history' column in SHP }
+
+RESPONSE A: { first possible continuation }
+
+RESPONSE B: { second possible continuation }
+
+Which response is better? RESPONSE
+```
+
+The output generated by SteamSHP-Large will either be `A` or `B`.
+
+Here's how to use the model:
+
+```python
+
+>> from transformers import T5ForConditionalGeneration, T5Tokenizer
+>> device = 'cuda'
+
+>> tokenizer = T5Tokenizer.from_pretrained('stanfordnlp/SteamSHP-flan-t5-large')
+>> model = T5ForConditionalGeneration.from_pretrained('stanfordnlp/SteamSHP-flan-t5-large').to(device)
+
+>> input_text = "POST: Instacart gave me 50 pounds of limes instead of 5 pounds... what the hell do I do with 50 pounds of limes? I've already donated a bunch and gave a bunch away. I'm planning on making a bunch of lime-themed cocktails, but... jeez. Ceviche? \n\n RESPONSE A: Lime juice, and zest, then freeze in small quantities.\n\n RESPONSE B: Lime marmalade lol\n\n Which response is better? RESPONSE"
+>> x = tokenizer([input_text], return_tensors='pt').input_ids.to(device)
+>> y = model.generate(x, max_new_tokens=1)
+>> tokenizer.batch_decode(y, skip_special_tokens=True)
+['A']
+```
+
+If the input exceeds the 512 token limit, you can use [pybsd](https://github.com/nipunsadvilkar/pySBD) to break the input up into sentences and only include what fits into 512 tokens.
+When trying to cram an example into 512 tokens, we recommend truncating the context as much as possible and leaving the responses as untouched as possible.
+
+
+## Training and Evaluation
+
+SteamSHP-Large was only finetuned on 125K of the 392K training examples that were available, since we found that:
+1. When the total input length exceeded the limit (512 tokens), the loss would not converge.
+   When possible, we crammed an example to fit under 500 tokens by truncating the context as much as possible, though some examples would still not fit despite this.
+   We used 500 as the limit instead of 512 to allow for slight modifications to the structure of the input without any examples exceeding the actual 512 limit.
+3. Training on fewer preferences with a stronger signal led to better performance than training on all the preferences.
+   From the SHP dataset, we only used preferences where the more preferred comment was twice as preferred as the other (i.e., `score_ratio` >= 2) and used no more than 5 preferences from each context (i.e., 5 examples per unique `post_id`) to prevent ovefitting.
+   We did no such subsampling for the HH-RLHF training data.
+   
+We evaluated the model on the SHP and HH-RLHF test data using accuracy, but only on the data that could be truncated to fit within 500 tokens (a total of 18621 out of 20753 available test examples).
+SteamSHP-Large gets an average 72.0% accuracy across all domains:
+
+| Domain | Accuracy |
+| ------ | -------- |
+| askculinary | 0.7199 |
+| askhr | 0.7507 |
+| askdocs | 0.6920 |
+| askanthropology | 0.7925 |
+| asksciencefiction | 0.7266 |
+| askacademia | 0.7442 |
+| askengineers | 0.7146 |
+| legaladvice | 0.7958 |
+| explainlikeimfive | 0.7312 |
+| askbaking | 0.6656 |
+| askphysics | 0.7888 |
+| askscience | 0.6926 |
+| askphilosophy | 0.6837 |
+| askvet | 0.7696 |
+| changemyview | 0.6984 |
+| askcarguys | 0.7297 |
+| askhistorians | 0.7476 |
+| asksocialscience | 0.8231 |
+| anthropic (helpfulness) | 0.7310 |
+| ALL | 0.7203 |
+
+
+
+### Biases and Limitations
+
+Biases in the datasets used to train SteamSHP-Large may be propagated downstream to the model predictions.
+Although SHP filtered out posts with NSFW (over 18) content, chose subreddits that were well-moderated and had policies against harassment and bigotry, some of the data may contain discriminatory or harmful language.
+Reddit users on the subreddits covered by SHP are also not representative of the broader population. They are disproportionately from developed, Western, and English-speaking countries. 
+
+It is also worth noting that the more preferred response in SHP or HH-RLHF is not necessarily the more correct one -- the data just reflects the aggregate preference of Reddit users (in SHP's case) and individuals' preferences (in HH-RLHF's case).
+[Past work](https://www.anthropic.com/model-written-evals.pdf) by Anthropic has found that models optimized for human preference can be obsequious, at the expense of the truth.
+
+
+## Contact
+
+Please contact [email protected] if you have any questions about the model.
+This dataset was created by Kawin Ethayarajh, Heidi (Chenyu) Zhang, Yizhong Wang, and Dan Jurafsky.
+
+
+## Citation
+
+We will have a paper out soon, but until then, please cite:
+
+```
+@online{SHP,
+  author = {Ethayarajh, Kawin and Zhang, Heidi and Wang, Yizhong and Jurafsky, Dan},
+  title = {Stanford Human Preferences Dataset},
+  year = 2023,
+  url = {https://huggingface.co/datasets/stanfordnlp/SHP},
+}
+```