Maartje M A De Graaf
Bertram F Malle
Anca Dragan
Tom Ziemke

Explainable Robotic Systems

As artificial agents, and especially socially interactive robots, enter human society, the demands for these agents to be transparent and explainable grow rapidly. When systems are able, for example, to explain how they made classifications or arrived at a decision, users are better able to judge the systems' accuracy and have more calibrated trust in them (e.g.,  [8]    [10] ). More and more AI systems process vast amounts of information and make classifications or recommendations that humans use for financial, employment, medical, military, and political decisions. More precariously yet, autonomous social robots, by definition, make decisions reaching beyond direct commands and perform actions with social and moral significance for humans. The demand for these social robots to become transparent and explainable is particularly urgent. However, to make robots explainable, we need to understand how people interpret the behavior of such systems and what expectations they have of them. In this workshop, we will address the topics of transparency and explainability, for robots in particular, from both the cognitive science perspective and the computer science and robotics perspective. Cognitive science elucidates how people interpret robot behavior; computer science and robotics elucidate how the computational mechanisms underlying robot behavior can be structured and communicated so as to be human-interpretable. The implementation and use of explainable robotic systems may prevent the potentially frightening confusion over why a robot is behaving the way it is. Moreover, explainable robot systems may allow people to better calibrate their expectations of the robot's capabilities and be less prone to treating robots as almost-humans.
 BACKGROUND When people interact with a robotic system, they construct mental models to understand and predict its actions. However, people's mental models of robots stem from their interactions with living beings. Thus, people easily run the risk of establishing incorrect or inadequate models of robotic systems, which may result in self-deception or even harm  [12] . Moreover, a long-term study  [9]  showed that initially established (incorrect) mental models of an intelligent information system remained robust over time, even when details of the system's implementation were given and initial beliefs were challenged with contradictory evidence. Incorrect mental models of AIS can have significant consequences for trust in such systems and, as a result, for acceptance of and collaboration with these systems  [10] . Several studies indicate that people distrust a robotic system when they are unable to understand its actions. When a robot fails to communicate its intentions, people not only perceive the robot as creepy or unsettling  [11] , they also perceive such robots as erratic and untrustworthy even when they follow a clear decision-making process  [5] . Indeed, when a robot is not transparent about its intentions (i.e., not providing any explanations for its behavior), people may even question its correct task performance and blame the robotic agent for its alleged errors  [4] . In addition to such cases of distrust, incorrect mental models of AIS can also lead to the opposite situation. People sometimes over-trust artificial agents, such as when they comply with a faulty robot's unusual requests  [7]  or follow the lead of a potentially inept robot  [6] . Thus, there should be little doubt about the value of making robotic systems perform easily interpretable actions and explain their own decisions and behaviors. This goal for explainable robots demands innovative work in computer science, robotics, and cognitive science, and our proposed workshop provides strong representation from all these fields. Additionally, we invite research that examines if and how such structures lead to better acceptance of robotic systems, to more accurate mental models of such systems, and to calibrated trust in these systems.
 WORKSHOP ACTIVITIES The aim of this full-day workshop is to provide a forum to share and learn about recent research on requirements for artificial agents' explanations as well as the implementation of transparent, predictable and explainable robotic systems. Extended time for discussions will highlight and document promising approaches and encourage further work. We welcome prospective participants to submit extended abstracts (max. 2 pages) that contribute to the discussion of people's interpretation of robot actions as well as the implementation of transparent, predictable, and explainable behaviors in robotic systems. After the conference, and with permission of the authors, we will provide online access to the workshop proceedings on a dedicated workshop website: explainableroboticsystems.wordpress.com. We encourage researchers to attend the workshop even without a paper submission, as a major portion of the workshop involves community engagement and discussion to foster uptake of concepts regarding explainable robotic systems within the field of HRI. The morning session will cover invited talks from Rachid Alami, Joanna Bryson, Bradley Hayes, and Alessandra Sciutti, as well as short presentations by authors of accepted papers, followed by general discussion. The afternoon will be devoted to break-out sessions to discuss the next steps in research and development of explainable and transparent robotic systems. Groups will be composed of representatives of different disciplines in order to work on integrating the multiple necessary perspectives in this endeavor. To boost the discussions, we will ask presenters to prepare questions or raise pressing issues that provide starting points for the discussion groups.  is a postdoctoral research associate at the Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, USA. She has a Bachelor of Business Administration in Communication Management (2005), a Master of Science in Media Communication (2011), and a PhD in Communication Science and Human-Robot Interaction (2015). Her research research interest focuses on peopleâĂŹs social, emotional and cognitive responses to robots including the societal and ethical consequences of those responses. Bertram Malle is Professor of Psychology in the Department of Cognitive, Linguistic, and Psychological Sciences at Brown University and Co-Director of the Humanity-Centered Robotics Initiative at Brown. He was trained in psychology, philosophy, and linguistics at the University of Graz, Austria, and received his Ph.D. in Psychology from Stanford University in 1995. His research focuses on social cognition, moral psychology, and human-robot interaction. Anca Dragan is an Assistant Professor in Electrical Engineering and Computer Sciences at UC Berkeley. She received her PhD from the Robotics Institute at Carnegie Mellon University in 2015 and now runs a lab on algorithmic human-robot interaction. She has expertise in transparent and explainable robotics and AI, with applications in personal robots and autonomous cars. Tom Ziemke is Professor of Cognitive Science at the University of Skövde and Professor of Cognitive Systems at Linköping University, Sweden. He received his Ph.D. in Computer Science from Sheffield University, UK, in 2000. His main research interests are embodied cognition and social interaction, and in recent years in particular people's interaction with different types of autonomous systems, ranging from social robots to automated vehicles.

Abstract: The increasing complexity of robotic systems are pressing the need for them to be transparent and trustworthy. When people interact with a robotic system, they will inevitably construct mental models to understand and predict its actions. However, people's mental models of robotic systems stem from their interactions with living beings, which induces the risk of establishing incorrect or inadequate mental models of robotic systems and may lead people to either under-and over-trust these systems. We need to understand the inferences that people make about robots from their behavior, and leverage this understanding to formulate and implement behaviors into robotic systems that support the formation of correct mental models of and fosters trust calibration. This way, people will be better able to predict the intentions of these systems, and thus more accurately estimate their capabilities, better understand their actions, and potentially correct their errors. The aim of this full-day workshop is to provide a forum for researchers and practitioners to share and learn about recent research on people's inferences of robot actions, as well as the implementation of transparent, predictable, and explainable behaviors into robotic systems.


Nisan Stiennon
Long Ouyang
Jeff Wu
Daniel M Ziegler
Ryan Lowe
Chelsea Voss
Alec Radford
Dario Amodei
Paul Christiano
Openai

Learning to summarize from human feedback

Introduction Large-scale language model pretraining has become increasingly prevalent for achieving high performance on a variety of natural language processing (NLP) tasks. When applying these models to a specific task, they are usually fine-tuned using supervised learning, often to maximize the log probability of a set of human demonstrations. While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective-maximizing the likelihood of human-written text-and what we care about-generating high-quality outputs as determined by humans. This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g. making up facts  [41] ) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models
 Supervised learning
 Human feedback
 Pretrain only
 Reference summaries Figure  1 : Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.  4  Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in Appendix F; length differences explain about a third of the gap between feedback and supervised learning at 6.7B. are incentivized to place probability mass on all human demonstrations, including those that are low-quality; and distributional shift during sampling can degrade performance  [56, 52] . Quality can often be improved significantly by non-uniform sampling strategies such as beam search  [51] , but these can lead to repetition and other undesirable artifacts  [69, 23] . Optimizing for quality may be a principled approach to overcoming these problems. Our goal in this paper is to advance methods for training language models on objectives that more closely capture the behavior we care about. To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community  [16, 8, 54, 59, 50] , and is a subjective task where we believe it is difficult to quantify summary quality without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE  [39] , have received criticism for poor correlation with human judgments  [55, 45, 6, 33] . We follow the works of  [3, 73] , who fine-tune language models from human feedback using reward learning  [35] . We first collect a dataset of human preferences between pairs of summaries, then train a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each 'time step', and is updated using the PPO algorithm  [58]  based on the RM 'reward' given to the entire generated summary. We can then gather more human data using samples from the resulting policy, and repeat the process. We follow the works of  [48, 4]  and use large pretrained GPT-3 models with as many as 6.7 billion parameters. Our main contributions are four-fold. (1) We show that training with human feedback significantly outperforms very strong baselines on English summarization. When applying our methods on a version of the Reddit TL;DR dataset  [63] , we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure  1 ). (2) We show human feedback models generalize much better to new domains than supervised models. Our Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any news-specific fine-tuning, almost matching the quality of the dataset's reference summaries. We perform several checks to ensure that these human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section C.2), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices F and G.7). (3) We conduct extensive empirical analyses of our policy and reward model. We examine the impact of model and data size (Figure  6 ), study performance as we continue to optimize a given reward model (Section 4.3), and analyze reward model performance using synthetic and humanwritten perturbations of summaries (Section 4.3). We confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences, and that optimizing our reward model directly results in better summaries than optimizing ROUGE according to humans (Section 4.4). (4) We publicly release our human feedback dataset for further research. The dataset contains 64,832 summary comparisons on the TL;DR dataset, as well as our evaluation data on both TL;DR (comparisons and Likert scores) and CNN/DM (Likert scores). The methods we present in this paper are motivated in part by longer-term concerns about the misalignment of AI systems with what humans want them to do. When misaligned summarization models make up facts, their mistakes are fairly low-risk and easy to spot. However, as AI systems become more powerful and are given increasingly important tasks, the mistakes they make will likely become more subtle and safety-critical, making this an important area for further research.
 Related work Most directly related to our work is previous work using human feedback to train summarization models with RL  [3, 73] . Bohm et al.  [3]  learn a reward function from a dataset of human ratings of 2.5k CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing ROUGE. Our work is most similar to  [73] , who also train Transformer models  [62]  to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets. Unlike us, they train in an online manner and find the model highly extractive. They note that their labelers prefer extractive summaries and have low agreement rates with researchers. Compared to  [73] , we use significantly larger models, move to the batch setting for collecting human feedback, ensure high labeler-researcher agreement, and make some algorithmic modifications, such as separating the policy and value networks. Human feedback has also been used as a reward to train models in other domains such as dialogue  [25, 68, 21] , translation  [32, 1] , semantic parsing  [34] , story generation  [72] , review generation  [7] , and evidence extraction  [46] . Our reward modeling approach was developed in prior work on learning to rank  [40] , which has been applied to ranking search results using either explicit feedback  [2, 18]  or implicit feedback in the form of click-through data  [29, 30] . In a related line of research, human feedback has been used to train agents in simulated environments  [10, 24] . There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization  [50, 65, 45, 15, 19] , BLEU for translation  [50, 66, 1, 43] , and other domains  [61, 27, 26] . Finally, there has been extensive research on modifying architectures  [22, 59]  and pre-training procedures  [70, 36, 49, 60, 53, 14]  for improving summarization performance. 3 Method and experiment details
 High-level methodology Our approach is similar to the one outlined in  [73] , adapted to the batch setting. We start with an initial policy that is fine-tuned via supervised learning on the desired dataset (in our case, the Reddit TL;DR summarization dataset). The process (illustrated in Figure  2 ) then consists of three steps that can be repeated iteratively. Step 1: Collect samples from existing policies and send comparisons to humans. For each Reddit post, we sample summaries from several sources including the current policy, initial policy, original reference summaries and various baselines. We send a batch of pairs of summaries to our human evaluators, who are tasked with selecting the best summary of a given Reddit post. Step 2: Learn a reward model from human comparisons. Given a post and a candidate summary, we train a reward model to predict the log odds that this summary is the better one, as judged by our labelers. Step 3: Optimize a policy against the reward model. We treat the logit output of the reward model as a reward that we optimize using reinforcement learning, specifically with the PPO algorithm  [58] .
 Collect human feedback "j is better than k" "j is better than k" A Reddit post is sampled from the Reddit TL;DR dataset. Various policies are used to sample a set of summaries. Two summaries are selected for evaluation. A human judges which is a better summary of the post.
 Train reward model One post with two summaries judged by a human are fed to the reward model. The reward model calculates a reward r for each summary. The loss is calculated based on the rewards and human label, and is used to update the reward model.
 Train policy with PPO A new post is sampled from the dataset. The reward model calculates a reward for the summary. The reward is used to update the policy via PPO. We provide a more thorough description of our procedure, including details of the reward model and policy training and our quality control process, in the following sections. In practice, rather than precisely iterating this sequence of three steps, we updated our data collection and training procedures over the course of the project while accumulating labels (see Appendix C.6 for details).
 Datasets and task Datasets. We use the TL;DR summarization dataset  [63] , which contains ~3 million posts from reddit.com across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs). We additionally filter this dataset (see Appendix A) to ensure quality, including using a whitelist of subreddits that are understandable to the general population. Crucially, we also filter to include only posts where the human-written summaries contain between 24 and 48 tokens, to minimize the potential effect of summary length on quality (see Section 4.1 and Appendix F). Our final filtered dataset contains 123,169 posts, and we hold out ~5% as a validation set. For the remainder of this paper, we refer to this dataset simply as TL;DR. We chose the TL;DR dataset over the more commonly used CNN/DM dataset primarily because very strong performance can be attained on CNN/DM with simple extractive baselines. We find in Section 4.2 that our labelers prefer lead-3 over the CNN/DM reference summaries,  5  and that the supervised T5 model  [49]  with low-temperature sampling already surpasses the reference summary quality, while copying extensively from the article. On the other hand, simple extractive baselines perform poorly on TL;DR in our human evaluations (see Appendix G.2). Instead of training on CNN/DM, we study the transfer performance of our human feedback models to CNN/DM after being trained to summarize Reddit posts. Task. We define our ground-truth task as producing a model that generates summaries fewer than 48 tokens long that are as good as possible, according to our judgments. We judge summary quality by how faithfully the summary conveys the original post to a reader who can only read the summary and not the post (see Appendix C.5 for further discussion of criteria). Since we have limited capacity to do comparisons, we hire labelers to do the comparisons for us. We rely on detailed procedures to ensure high agreement between labelers and us on the task, which we describe in the next section. [r/dating_advice] First date ever, going to the beach. Would like some tips Hey Reddit! I (20M) would like some tips, because I have my first ever date tomorrow (although I've had a gf for 3 years, but no actual dating happened), and we're going to the beach. I met this girl, we have mutual friends, at a festival a few days ago. We didn't kiss, but we talked, held hands, danced a bit. I asked her to go on a date with me, which was super hard as it is the first time I've asked this to anybody. What I mean to say is, it's not like a standard *first* date because we already spent some time together. I'm really nervous and excited. I'm going to pick her up tomorrow, we're cycling to the beach which will take  30     1 : Example of post and samples on the TL;DR dataset, chosen to be particularly short. For random samples (along with posts), see Appendix H and our website.
 Collecting human feedback Previous work on fine-tuning language models from human feedback  [73]  reported "a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated", leading to model-generated summaries that were high-quality according to the labelers, but fairly low-quality according to the researchers. Compared to  [73] , we implement two changes to improve human data quality. First, we transition entirely to the offline setting, where we alternate between sending large batches of comparison data  6  to our human labelers and re-training our models on the cumulative collected data. Second, we maintain a hands-on relationship with labelers:  7  we on-board them with detailed instructions, answer their questions in a shared chat room, and provide regular feedback on their performance. We train all labelers to ensure high agreement with our judgments, and continuously monitor labeler-researcher agreement over the course of the project. See Appendix C.1 and C.5 for details. As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of comparison tasks, labelers agree with researchers 77% ± 2% of the time, while researchers agree with each other 73% ± 4% of the time. We provide more analysis of our human data quality in Appendix C.2.
 Models All of our models are Transformer decoders  [62]  in the style of GPT-3  [47, 4] . We conduct our human feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters. Pretrained models. Similarly to  [12, 47] , we start with models pretrained to autoregressively predict the next token in a large text corpus. As in  [48, 4] , we use these models as 'zero-shot' baselines by padding the context with examples of high-quality summaries from the dataset. We provide details on pretraining in Appendix B, and on our zero-shot procedure in Appendix B.2. Supervised baselines. We next fine-tune these models via supervised learning to predict summaries from our filtered TL;DR dataset (see Appendix B for details). We use these supervised models to sample initial summaries for collecting comparisons, to initialize our policy and reward models, and as baselines for evaluation. In our final human evaluations, we use T=0 to sample from all models, as we found it performed better than higher temperatures or nucleus sampling (see Appendix B.1). To validate that our supervised models are indeed strong baselines for comparison, we run our supervised fine-tuning procedure with our 6.7B model on the CNN/DM dataset, and find that we achieve slightly better ROUGE scores than SOTA models  [71]  from mid-2019 (see Appendix G.4). Reward models. To train our reward models, we start from a supervised baseline, as described above, then add a randomly initialized linear head that outputs a scalar value. We train this model to predict which summary y ∈ {y 0 , y 1 } is better as judged by a human, given a post x. If the summary preferred by the human is y i , we can write the RM loss as: loss(r θ ) = E (x,y0,y1,i)∼D [log(σ(r θ (x, y i ) − r θ (x, y 1−i )))] where r θ (x, y) is the scalar output of the reward model for post x and summary y with parameters θ, and D is the dataset of human judgments. At the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0. Human feedback policies. We want to use the reward model trained above to train a policy that generates higher-quality outputs as judged by humans. We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm  [58] , where each time step is a BPE token.  8  We initialize our policy to be the model fine-tuned on Reddit TL;DR. Importantly, we include a term in the reward that penalizes the KL divergence between the learned RL policy π RL φ with parameters φ and this original supervised model π SFT , as previously done in  [25] . The full reward R can be written as: R(x, y) = r θ (x, y) − β log[π RL φ (y|x)/π SFT (y|x)] This KL term serves two purposes. First, it acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode. Second, it ensures the policy doesn't learn to produce outputs that are too different from those that the reward model has seen during training. For the PPO value function, we use a Transformer with completely separate parameters from the policy. This prevents updates to the value function from partially destroying the pretrained policy early in training (see ablation in Appendix G.1). We initialize the value function to the parameters of the reward model. In our experiments, the reward model, policy, and value function are the same size.
 Results
 Summarizing Reddit posts from human feedback Policies trained with human feedback are preferred to much larger supervised policies. Our main results evaluating our human feedback policies on TL;DR are shown in Figure  1 . We measure policy quality as the percentage of summaries generated by that policy that humans prefer over the reference summaries in the dataset. Our policies trained with human feedback significantly outperform our supervised baselines on this metric, with our 1.3B human feedback model significantly outperforming a supervised model 10× its size (61% versus 43% raw preference score against reference summaries). Our 6.7B model in turn significantly outperforms our 1.3B model, suggesting that training with human feedback also benefits from scale. Additionally, both of our human feedback models are judged by humans to be superior to the human demonstrations used in the dataset. Controlling for summary length. When judging summary quality, summary length is a confounding factor. The target length of a summary is implicitly part of the summarization task; depending on the desired trade-off between conciseness and coverage, a shorter or longer summary might be better. Since our models learned to generate longer summaries, length could account for much of our quality improvements. We find that after controlling for length (Appendix F), the preference of our human feedback models vs. reference summaries drops by ~5%; even so, our 6.7B model summaries are still preferred to the reference summaries ~65% of the time. How do our policies improve over the baselines? To better understand the quality of our models' summaries compared to the reference summaries and those of our supervised baselines, we conduct an additional analysis where human labelers assess summary quality across four dimensions (or "axes") using a 7-point Likert scale  [38] . Labelers rated summaries for coverage (how much important information from the original post is covered), accuracy (to what degree the statements in the summary are stated in the post), coherence (how easy the summary is to read on its own), and overall quality.  The results (Figure  3 ) indicate that our human feedback models outperform the supervised baselines across every dimension of quality, but particularly coverage. Although our human labelers had a high bar for giving perfect overall scores, summaries from our 6.7B PPO model achieve a 7/7 overall score 45% of the time (compared to 20% and 23% for the 6.7B supervised baseline and reference summaries, respectively).
 Transfer to summarizing news articles Our human feedback models can also generate excellent summaries of CNN/DM news articles without any further training (Figure  4 ). Our human feedback models significantly outperform models trained via supervised learning on TL;DR and models trained only on pretraining corpora. In fact, our 6.7B human feedback model performs almost as well as a 6.7B model that was fine-tuned on the CNN/DM reference summaries, despite generating much shorter summaries. Since our human feedback models transferred to CNN/DM have little overlap in summary length distribution with models trained on CNN/DM, with about half as many tokens on average, they are difficult to compare directly. Thus our evaluations in Figure  4  use a 7-point Likert scale on four quality dimensions, as in Section 4.1 (see Appendix C.5 for labeler instructions). In Figure  4b  we show the average overall score at different summary lengths, which suggests our human feedback models would perform even better if they generated longer summaries. Qualitatively, CNN/DM summaries from our human feedback models are consistently fluent and reasonable representations of the article; we show examples on our website and in Appendix H.
 Understanding the reward model What happens as we optimize the reward model? Optimizing against our reward model is supposed to make our policy align with human preferences. But the reward model isn't a perfect representation of our labeler preferences, as it has limited capacity and only sees a small amount of comparison data from a relatively narrow distribution of summaries. While we can hope our reward model generalizes to summaries unseen during training, it's unclear how much one can optimize against the reward model until it starts giving useless evaluations. To answer this question, we created a range of policies optimized against an earlier version of our reward model, with varying degrees of optimization strength, and asked labelers to compare samples from them to the reference summaries. Figure  5  shows the results for PPO at a range of KL penalty coefficients (β). Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences. Though this is clearly undesirable, we note that this over-optimization also happens with ROUGE (see  [45]  and Appendix G.3). Similar behavior has been observed in learned reward functions in the robotics domain  [5] . How does reward modeling scale with increasing model and data size? We conduct an ablation to determine how data quantity and model size affect reward modeling performance. We train 7 reward models ranging from 160M to 13B parameters, on 8k to 64k human comparisons from our dataset. We find that doubling the training data amount leads to a ~1.1% increase in the reward model validation set accuracy, whereas doubling the model size leads to a ~1.8% increase (Figure  6 ). What has the reward model learned? We probe our reward model by evaluating it on several validation sets. We show the full results in Appendix G.6, and highlight them here. We find that our reward models generalize to evaluating CNN/DM summaries (Appendix G.7), agreeing with labeler preferences 62.4% and 66.5% of the time (for our 1.3B and 6.7B models, respectively). Our 6.7B reward model nearly matches the inter-labeler agreement value of 66.9%. We also find that our reward models are sensitive to small but semantically important details in the summary. We construct an additional validation set by having labelers make minimal edits to summaries to improve them. Our RMs prefer the edited summaries almost as often (79.4% for 1.3B and 82.8% for 6.7B) as a separate set of human evaluators (84.1%). Further, when comparing the reference summaries to perturbed summaries where the participants' roles are reversed, our models reliably select the original summary (92.9% of the time for 1.3B, 97.2% for 6.7B). However, our RMs are biased towards longer summaries: our 6.7B RM prefers improving edits that make the summary shorter only 62.6% of the time (vs. 76.4% for humans).
 Analyzing automatic metrics for summarization Evaluation. We study how well various automatic metrics act as predictors for human preferences, and compare them to our RMs. Specifically, we examine ROUGE, summary length, amount of copying from the post,  9  and log probability under our baseline supervised models. We present a full matrix of agreement rates between these metrics in Appendix G.7. We find that our learned reward models consistently outperform other metrics, even on the CNN/DM dataset on which it was never trained. We also find that ROUGE fails to track sample quality as our Figure  7 : Summary quality as a function of metric optimized and amount of optimization, using best-of-N rejection sampling. We evaluate ROUGE, our main reward models, and an earlier iteration of the 1.3B model trained on approximately 75% as much data (see Table  11  for details). ROUGE appears to peak both sooner and at a substantially lower preference rate than all reward models. Details in Appendix G.3. models improve. While ROUGE has ~57% agreement with labelers when comparing samples from our supervised baseline models, this drops to ~50% for samples from our human feedback model. Similarly, log probability agreement with humans drops to ≤50% on comparisons between samples from our human feedback models, while our RMs still perform above chance (62%). Scaling up the size of the supervised model does not reliably improve log probability's agreement with labelers. Optimization. In Figure  7 , we show that optimizing ROUGE using a simple optimization scheme doesn't consistently increase quality, as has been noted in  [45] . Optimization against ROUGE peaks both sooner and at a substantially lower quality rate than optimization against our reward models.
 Discussion Limitations. One limitation of our work is the time and cost required to produce our final models. Notably, fine-tuning our 6.7B model with RL required approximately 320 GPU-days. Our data collection procedure is also expensive compared to prior work -the training set took thousands of labeler hours and required significant researcher time to ensure quality. For this reason, we were unable to collect baselines such as an equivalent amount of high-quality human demonstrations for supervised baselines. See D for more discussion. We leave this ablation to future work. Nevertheless, we believe reward modeling is more likely to scale to tasks where it is extremely skill-intensive or time-consuming to provide good demonstrations. Future directions. The methods in this paper could be applied to any task where humans can compare samples, including dialogue, machine translation, question answering, speech synthesis, and music generation. We expect this method to be particularly important for generating long samples, where the distributional shift and degeneracy of maximum likelihood samples can be problematic. It may be possible to improve sample efficiency by training to predict feedback across many tasks  [42] . We are particularly interested in scaling human feedback to tasks where humans can't easily evaluate the quality of model outputs. In this setting, it is particularly challenging to identify whether an ML system is aligned with the human designer's intentions. One approach is to train ML systems to help humans perform the evaluation task quickly and accurately  [9] . There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models  [28, 17, 44, 64] . For example, we could solicit high-quality demonstrations from labelers, have labelers edit model outputs to make them better, or have labelers provide explanations for why they preferred one model output over another. All of this feedback could be leveraged as a signal to train more capable reward models and policies. Broader impacts. The techniques we explore in this paper are generic techniques that could be used in a wide variety of machine learning applications, for any task where it is feasible for humans to evaluate the quality of model outputs. Thus, the potential implications are quite broad. Our research is primarily motivated by the potential positive effects of aligning machine learning algorithms with the designer's preferences. Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait  [11] . In the short term, improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human well-being. In the long term, as machine learning systems become more capable it will likely become increasingly difficult to ensure that they are behaving safely: the mistakes they make might be more difficult to spot, and the consequences will be more severe. For instance, writing an inaccurate summary of a news article is both easy to notice (one simply has to read the original article) and has fairly low consequences. On the other hand, imitating human driving may be substantially less safe than driving to optimize human preferences. We believe that the techniques we explore in this paper are promising steps towards mitigating the risks from such capable systems, and better aligning them with what humans care about. Unfortunately, our techniques also enable malicious actors to more easily train models that cause societal harm. For instance, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs, or to induce dependence of humans on the technology, or to generate large amounts of toxic or hurtful content intended to harm specific individuals. Avoiding these outcomes is a significant challenge for which there are few obvious solutions. Large-scale models trained with human feedback could have significant impacts on many groups. Thus, it is important to be careful about how we define the 'good' model behavior that human labelers will reinforce. Deciding what makes a good summary is fairly straightforward, but doing this for tasks with more complex objectives, where different humans might disagree on the correct model behavior, will require significant care. In these cases, it is likely not appropriate to use researcher labels as the 'gold standard'; rather, individuals from groups impacted by the technology should be included in the process to define 'good' behavior, and hired as labelers to reinforce this behavior in the model. We chose to train on the Reddit TL;DR dataset because the summarization task is significantly more challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with minimal moderation, they often contain content that is offensive or reflects harmful social biases. This means our models can generate biased or offensive summaries, as they have been trained to summarize such content. For this reason, we recommend that the potential harms of our models be thoroughly studied before deploying them in user-facing applications. Finally, by improving the ability of machine learning algorithms to perform tasks that were previously only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially leading to significant job loss. Without suitable policies targeted at mitigating the effects of large-scale unemployment, this could also lead to significant societal harm.  Figure 2 : 2 Figure 2: Diagram of our human feedback, reward model training, and policy training procedure.
 Figure 4 : 4 Figure 4: Transfer results on CNN/DM. (a) Overall summary quality on CNN/DM as a function of model size. Full results across axes shown in Appendix G.2. (b) Overall scores vs. length for the 6.7B TL;DR supervised baseline, the 6.7B TL;DR human feedback model, and T5 fine-tuned on CNN/DM summaries. At similar summary lengths, our 6.7B TL;DR human feedback model nearly matches T5 despite never being trained to summarize news articles.
 Figure 3 : 3 Figure 3: Evaluations of four axes of summary quality on the TL;DR dataset.
 Table minutes, and then what? I'm a bit scared. Should I bring something (the weather, although no rain and sunny, is not super so no swimming), should we do something. I'd like all the tips I can get. Thanks! Human written reference TL;DR 6.7B supervised model 6.7B human feedback model First date after 3 years in a relation- Going on a date with a girl I met Going on my first ever date tomor- ship, going to the beach, terrified. a few days ago, going to the beach. row, cycling to the beach. Would What to bring with me, what to do? What should I bring, what should like some tips on what to do and we do? bring. I'm a bit nervous and excited. Thanks!
 Uifalean, Morris Stuttard, Russell Bernandez, Tasmai Dave, Rachel Wallace, Jenny Fletcher, Jian Ouyang, Justin Dill, Maria Orzek, Megan Niffenegger, William Sells, Emily Mariner, Andrew Seely, Lychelle Ignacio, Jelena Ostojic, Nhan Tran, Purev Batdelgar, Valentina Kezic, Michelle Wilkerson, Kelly Guerrero, Heather Scott, Sarah Mulligan, Gabriel Ricafrente, Kara Bell, Gabriel Perez, and Alfred Lee.
			 Throughout the paper, error bars represent 1 standard error.
			 We manually check this result in Appendix E and find we generally agree with labeler ratings.
			 Our decision to collect comparisons rather than Likert scores is supported by recent work, e.g. [37] . 7  We recruited labelers from a freelancing platform, Upwork, and two labeling services, Scale and Lionbridge.
			 Note that the reward model only gives rewards for entire summaries, and not at intermediate time steps. In RL terminology, each episode terminates when the policy outputs the EOS token, and the discount factor γ = 1.
			 We measure copying by computing the longest common subsequence of bigrams with the original Reddit post or news article, and dividing by the number of bigrams in the summary.

Abstract: As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about-summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts  [63]  and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles  [22] , producing summaries nearly as good as the human reference without any news-specific fine-tuning.  2  We conduct extensive analyses to understand our human feedback dataset and fine-tuned models.  3  We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want. * This was a joint project of the OpenAI Reflection team. Author order was randomized amongst {LO, JW, DZ, NS}; CV and RL were full-time contributors for most of the duration. PC is the team lead. 2 Samples from all of our models can be viewed on our website.  3  We provide inference code for our 1.3B models and baselines, as well as a model card and our human feedback dataset with over 64k summary comparisons, here.


Multitasking: Efficient Optimal Planning for Bandit Superprocesses

INTRODUCTION Multitasking is no doubt an activity familiar to the reader: one faces several decision problems but can act on only one (or perhaps a bounded number) at a time. Such prob-lems are ubiquitous for individuals, corporations, armies, and governments. Multitasking problems are expressed by the class of bandit superprocesses  [Nash, 1973]  or BSPs. A k-armed BSP M consists of k independent Markov decision processes M 1 , . . . , M k ; the MDPs are coupled by a common discount factor and by the constraint that at each time step the agent can act in only one MDP. Since the MDPs are independent of each other, one might imagine that the optimal policy for M is obtained by solving each MDP-turning the BSP into a multi-armed bandit-and then using Gittins indices to choose a sequence of arms. In fact, the optimal policy for a BSP may include actions that are suboptimal from the point of view of the constituent MDP in which they are taken. The reason for this is that the availability of other MDPs in which to act changes the balance between short-term and long-term rewards in a component MDP; in fact, it tends to lead to greedier behavior in each MDP because aiming for longterm reward in one MDP would delay rewards in all the other MDPs. The globally and locally optimal policies necessarily coincide only when the discount factor is 1. Hence, in addition to their general importance in the real world, a second reason to study multitasking problems is that they undermine an assumption implicit in practical applications of sequential decision analysis: the assumption that an optimal solution for the user's decision problem is optimal for a user who faces multiple decision problems. Despite these considerations, there has been remarkably little research on bandit superprocesses and almost none in AI. Section 2 summarizes what is known. Obviously, there are connections to multi-armed bandits (MABs), which are a special case of BSPs in which each arm only allows one choice of action rather than several. Unfortunately, the index theorems that simplify the computation of optimal MAB policies are not valid for BSPs. There are also strong connections-hitherto unexplored-between BSPs and sums of games as studied in combinatorial game theory  [Conway, 2000] . The principal question we address in this paper is whether an algorithm exists that is substantially more efficient than applying a standard MDP solver to the "cross-product" MDP obtained by combining the states spaces of the constituent MDPs. The sole known case where planning for a BSP is provably more efficient is the case where a dominating policy exists: if a single policy is optimal across the family of retirement processes associated with each arm then a BSP can be reduced to an equivalent MAB  [Whittle, 1980] . (Retirement processes provide a measure of how the optimal policy depends on context and are defined in Section 3.) However, this condition is seldom satisfied in practice. This paper provides three contributions. First, we give a concise survey of the bandit superprocess literature tailored to the AI community. Second, we provide a novel upper bound on the global value function of a BSP. For a BSP, M , with arms {M 1 , . . . , M k } we relax each arm to obtain an MAB M = {M 1 , . . . , M k } by adding actions to ensure that dominating policies exists. We show that this upper bound is equivalent to the Whittle integral, an existing upper bound for BSPs  [Brown and Smith, 2013] . Because our bound is defined in terms of an explicit relaxation, it provides insight into the nature of the bound and opens an avenue to extend this work to more general MDPs. Finally, we describe a practical computational approach for solving BSPs: we derive a simple method for computing the Whittle integral upper bound that can use an arbitrary MDP solver as a black box; then we combine this upper bound with a lower bound to derive an efficient algorithm for -optimal decision making in a BSP. We present empirical results to show that it substantially improves over more general optimal factored MDP algorithms; we use it to compute provably optimal actions for problems with upwards of 10 30 states in the full state space.
 RELATED WORK Robbins  [1952]  provided the first formulation of multiarmed bandits (MABs) in their modern form. The famous Gittins index theorem  [Gittins, 1979]  showed that optimal MAB policies are obtained by ranking the arms according to an index function defined on each seperately. An immediate corollary is that optimal decision making in an MAB is linear in the number of arms. Problems with this property are said to be indexable. Bandit superprocesses (BSPs) were introduced by  Nash [1973]  as a generalization of multi-armed bandits to study allocation of resources among research projects.  Whittle [1980]  provided an alternate proof of the Gittins index theorem that extends to bandit superprocesses with dominating policies. His proof utilized a construction called the Whittle integral, which allows one to compute the value of a composite state in an MAB.  Glazebrook [1982]  provides a proof that bandit superprocesses are not indexable in general.  Glazebrook [1993]  considers bandit superprocesses where the arms can exert limited influence on each other and shows a result analogous to  Whittle's. Brown and Smith [2013]  were the first to identify the signifigance of the Whittle integral for sequential decision making. They developed a version of policy iteration to compute it, and recognized that it upper bounds the value function of a BSP, 1 but did not derive an algorithm for solving BSPs. Our work provides two further contributions regarding the Whittle integral: a short proof that it is an upper bound and a simpler algorithm to compute it. Within the AI community, there has been limited study of loosely coupled Markov decision processes.  Singh and Cohn [1998]  consider optimal solutions to simultaneous MDPs, where an agent can take actions in a number of MDPs. Their formulation is more general than ours, as it is possible to act in multiple MDPs at once. They derive bounds for this problem and give an algorithm that combines a form of real-time dynamic programming with pruning steps to remove provably suboptimal actions. However, their bounds are substantially looser than ours, and our experiments show that the corresponding algorithm can be impractical for simple BSPs with many arms.  Meuleau et al. [1998]  examine MDPs that are coupled by constraints on the use of shared resources. BSPs fit within this class if we view the restriction to a single MDP at a time as a constraint on the agent's attention. Interestingly, their heuristics are also defined in terms of a parameterized value function for the component MDPs-it would be interesting to attempt to generalize the approaches considered here to their problem domain. In this work, we leverage the particular structure of our resource to compute optimal solutions; Meuleau et al. construct a heuristic policy.
 TECHNICAL BACKGROUND MARKOV DECISION PROCESSES Definition 1. (Markov Decision Process  [Puterman, 2009] ) A (finite-state, discounted) MDP, M , is a tuple M = S, A, T, R, γ . S is a set of states. A is a set of actions. T : S × A × S → [0, 1] is a function that assigns probability to state transitions for each state-action pair. R is a (bounded) reward function that maps state-action pairs to (positive) rewards R : S × A → R + . γ ∈ [0, 1) is a discount factor. A solution to M is a policy, π, that maps states to actions. The value of a state, s, under π is the sum of expected discounted rewards received by starting in s and selecting ac-tions according to π: V π (s) = E ∞ t=0 γ t R(s t )|s 0 = s, π . The optimal policy, π * , maximizes this value. In the above definition, and the ones that follow, we use superscripts to indicate dependence on the agent's policy. To simplify notation, we will omit these superscripts when the policy referred to is the optimal policy (e.g., V (s) = V π * (s)). The Q-function for the state-action pair, (s, a), is the value of taking a in s and selecting future actions according to π * .
 RETIREMENT PROCESSES Given an MDP, Whittle defines a family of optimal stopping problems: Definition 2. (Retirement Process  [Whittle, 1980] ) Let M be an MDP. For ρ ≥ 0, the retirement process for M with retirement reward, ρ, is an MDP, M ρ , with a single additional state, s R , and action, a R . a R transitions deterministically to s R and receives reward ρ. s R is a sink state that accrues zero reward. We refer to a decision to select a R as a decision to retire. We denote the retirement process value function as a function of a state and retirement reward, V (s, ρ). We let the optimal policy for retirement reward ρ be π * ρ . We write the set of states where the policy, π, retires as τ π ρ . We use ρ − (π) and ρ + (π) to denote the interval of retirement rewards so that π is optimal: ρ ∈ [ρ + (π), ρ + (π)] ⇒ V π (s, ρ) = V (s, ρ). We adopt the convention from the MAB/BSP literature and abuse notation to denote the (random) number of steps prior to retirement as τ π ρ (s). For s ∈ τ π ρ we let P retire (s |s, ρ, π) be the probability that s is the first state in τ π ρ the agent will reach given that it is in state s and executes policy π. We denote the expected discounted reward accrued prior to retirement, starting in s, as R π ρ (s). In regions of retirement reward where the optimal policy and stopping rule do not change, ∂V ∂ρ (s, ρ) is defined and is equal to the expected value of the discount parameter at retirement. This allows us to write the following expression for the retirement process value function: V (s, ρ) = R ρ (s) + E[γ τρ(s) ]ρ. (1) V (s, ρ) is piecewise linear in ρ. Figure 1 shows V (s, ρ) for the example BSP in Ex. 1. A policy that is optimal for every setting of ρ is called a dominating policy. Definition 3. (Dominating Policy) Let π be a policy for an MDP, M . π is a dominating policy iff ∀ρ ≥ 0 ∀s / ∈ τ ρ π(s) = π * ρ (s).
 MULTI-ARMED BANDITS AND BANDIT SUPERPROCESSES Multi-armed bandits (MAB) are a restricted class of MDPs that have received extensive study. Of particular interest are the form of the optimal policy and its utility in modelling "exploration vs exploitation" trade-offs. An MAB consists of a set of Markov reward processes (MRPs), where an MRP is an MDP with a single action. Each MRP is referred to as an arm of the problem. At each time-step, the agent selects an arm, that arm transitions according to its transition distribution, and the agent receives the corresponding reward. We adopt a summation notation to indicate the combination of several MRPs into an MAB. For example, if X and Y are MRPs and Z is the MAB with arms X and Y , we will write Z = X + Y . A famous result is that the optimal policy for any MAB is an index policy  [Gittins, 1979] . Each state, s i , in the individual arms is assigned an index, I si . In joint state s = {s 1 , . . . , s k }, the optimal action is to select argmax i I si . For MAB arm M i in state s i , the index is defined as the value of retirement reward such that the agent is indifferent between immediate retirement and following the optimal stopping policy  [Whittle, 1980] : I si = min ρ {ρ ; V i (s i , ρ) = ρ}. This means that, in a sense, context for a multi-armed bandit can always be summarized by a single number. To model a multi-tasking problem, we consider a generalization of multi-armed bandits: bandit superprocesses (BSPs). Bandit superprocesses allow arms that are arbitrary MDPs (and so are a generalization of MABs); at each time step, the agent selects both an arm and an action to take within that arm. Definition 4. (Bandit Superprocess  [Nash, 1973] ) Given k MDPs, {M i = S i , A i , T i , R i , γ }, we define M = i M i = S, A, T, R, γ to be the bandit superprocess with arms {M i }. S = × i S i and A = ∪ i A i . The transition distribution is stationary for arms that are not selected and follows the identical reward and transition distributions for the selected arm. Naturally, this makes planning more difficult. To see how, consider the following proposal: Conjecture 1. Let X = S X , A X , T X , R X , γ be an MDP. Define Y similarly and let Z = X + Y be the bandit superprocess that is their sum. Let a ∈ A X , s X ∈ S X be a state-action pair from X. Suppose that this transition is suboptimal in every retirement process: ∀ ρ ≥ 0, V X (s X , ρ) > Q X ((s X , a), ρ). Then a is suboptimal for any state in Z with s X as one of the components: ∀s Y ∈ S Y , V Z ({s X , s Y }) > Q Z ({s X , s Y }, a). This conjecture essentially proposes that state-action pairs can be safely ignored if they are suboptimal for all settings of a constant alternative. Unfortunately, as the following example illustrates, this is not the case. Example 1. Define X to be a deterministic reward chain and Y to be an initial choice between three reward chains (y 0 , y 1 , y 2 ) defined as follows: V (y 0 , ρ) = max 100 X = 28, 2 t=0 γ t + γ 3 ρ, ρ V (y 1 , ρ) = max 99 2 t=0 γ t + γ 3 max 1.4 1 − γ , ρ , ρ V (y 2 , ρ) = max 28 1 − γ , ρ . If we let γ = .9, then we have ∀280 > ρ ≥ 0 max{V (y 0 , ρ), V (y 2 , ρ)} > V (y 1 , ρ). Thus, any policy for a retirement process retirement process derived from Y that initially selects y 1 is suboptimal. We can apply the same strategy to compute the value of each combination of reward streams: V (y 0 + X) = 100 2 t=0 γ t + γ 3 28 5 t =0 γ t V (y 1 + X) = 99 2 t=0 γ t + γ 3 28 5 t =0 γ t + γ 6 1.4 1 − γ V (y 2 + X) = 28 1 − γ From this we can see that the optimal policy for the BSP that combines these MDPs, Z = X + Y , initially collects reward from y 1 . This contradicts Conjecture 1. In this section we show a bound on the value function of a bandit superprocess. We derive this bound by adding actions to a BSP so that it is equivalent to an MAB. The value function of our relaxed BSP is equivalent to the Whittle integral bound derived in Brown and  Smith [2013]  and so it yields insight into that computation. We begin by defining the Whittle integral and show some basic results in order to motivate our relaxation. Definition 5.  (Whittle Integral [Brown and Smith, 2013] ) Let M be a BSP. Let i index the arms of M. For any state, s = {s i }, and ρ ≥ 0, the Whittle integral of s is defined as V (s, ρ) = I − I x=ρ dx i ∂V i ∂ρ (s i , x). ( 2 ) Where I ≥ max i I si . When the arms of a BSP admit a dominating policy the Whittle integral is equal to the value function: Theorem 1. (Whittle Condition  [Whittle, 1980] ) Let M be a k-armed BSP with components {M i } and state space S. If each M i has a dominating policy, then ∀s ∈ S, ∀ρ ≥ 0, V (s, ρ) = V (s, ρ). MRPs have a single action per state, so the Whittle condition is trivially satisfied for all multi-armed bandits. V (s, 0) = V (s), so V (s, 0) provides an efficient method to compute the value of an MAB. For BSPs an arm that satisfies the Whittle condition can be replaced with an MRP that selects actions according to π * . The formula in Eq. 2 lends itself to a straightforward implementation, but is very challenging to interpret. We show below that this is equivalent to evaluating the retirement process value function for a single arm with a set of rewards determined by the other arms 3 . We refer to this set of rewards as the critical points of those arms. Definition 6. (Critical Points of an MDP) Let M be a Markov decision process. The critical points of M , C(M ) = {ρ i }, are the values of retirement reward such that the optimal stopping rule or policy changes. These are points where there is a discontinuity in ∂V ∂ρ . We let ∆ M (s, ρ) = lim δ→0 ∂V M ∂ρ (s, ρ + δ) − ∂V M ∂ρ (s, ρ − δ) be the size of the corresponding discontinuities. This is equivalent to the expected increase in E [γ τ ] under the new stopping rule. Theorem 2 shows that ∆ and C characterize the interaction between arms of an MAB. Theorem 2. Let X, Y be Markov reward processes. Let Z = X + Y be the 2-armed bandit. ∀s = {s X , s Y } ∈ S Z , V Z (s) = ρ∈C(Y ) V X (s X , ρ)∆ Y (s Y , ρ). (3) Proof. (sketch) As a first step, we follow the steps in Whittle  [1980]  and integrate Equation 2 by parts. This expresses V as the following integral: VZ (s) = Is X ρ =ρ V X (s X , ρ ) ∂ 2 V Y (s Y , ρ ) ∂ρ 2 dρ . V Y is piecewise linear with respect to ρ so ∂ 2 V Y ∂ρ 2 is a weighted sum of delta functions centered at V Y 's kinks. C(Y ) and ∆ Y respectively characterize these kinks and weights. Thus, VZ (s) is equal to the rhs of Eq. 3. X and Y are MRPs, so an appeal to Theorem 1 shows the result. A similar property holds for arbitrary k-armed bandits. This shows that we can summarize the context for an arm as a collection of weighted retirement rewards. Turning to BSPs, this lends insight into the approximation the Whittle integral introduces. To see how, we reconsider Ex. 1. Recall that this example consists of the BSP Z = X + Y and that Y is a choice between three reward chains, {y 0 , y 1 , y 2 }. Theorem 2 allows us to write the gap between the Whittle integral upper bound and the value of selecting y 1 as ρ∈C(X) max i V yi (s i , ρ) − V y1 (s 1 , ρ) ∆ X (s X , ρ) Figure  1  shows the retirement process value functions for this example. While the retirement process value of y 1 is always less than that of either y 0 or y 2 , it is close enough to their maximum that choosing y 1 essentially achieves the upper bound. The Whittle integral for Z is a weighted combination of distinct retirement process value functions (V y0 , V y2 ) but, in reality, the agent will be forced to pick a single policy of the two.
 DOMINATED RELAXATION OF AN MDP In this section, we introduce our primary theoretical result: a relaxation for the arms of a BSP so that a dominating policy exists. This reduces the BSP to an MAB whose value upper bounds the value of states in the BSP. We can show that the Whittle integral computes the value of states in this MAB and arrive at a straightforward proof that the Whittle integral is an upper bound. We call the result the dominated relaxation of an MDP. Before providing the details, we illustrate the main ideas with an example. The relaxed MDP is actually a Semi-MDP (SMDP): a generalization of an MDP where each action, a, has a duration, δ(a) ∈ R + . Example 2. Let MDP M be an initial choice between MRPs: M = 15, 0, 0, 0 . . . 10, 10, 10, 3, 3, 3, . . . . Let T (B) be the top (bottom) MRP. Let π T (π B ) be the policy that selects T (B) in s 0 . We can relax M with the addition of a single durative action, a , that transitions from the sink state in T to the sink state B. We set δ(a ) = 2. We take ρ such that V π T (s 0 , ρ) = V π B (s 0 , ρ) and set the rewards associated with a to be R π B ρ (s 0 ) − R π T ρ (s 0 ) = 10 2 0 γ t − 15 = 12.1 . With this change, at low settings of retirement reward, the agent is indifferent between a policy that opts for the top chain then selects a and the bottom reward chain. For high settings of retirement reward, the optimal policy retires immediately or retires after collecting the reward of 15. This SMDP satisfies the Whittle condition and can be replaced by an MRP in any bandit superprocess. Fig.  1  (c) shows an illustration of the state space and the introduced action. In this example, we connected the state where π T retires to the state where π B retires. This lets the agent collect shortterm and long-term rewards with the same policy. To do this in general, we introduce multiple copies of the state space, one for each policy that is optimal for some ρ. Definition 7. (Dominated Relaxation of an MDP) Let M be an MDP with discount factor γ and state space S. Let s be a state in M . The dominated relaxation of M for s, M D (s), is a semi-Markov decision process that fixes s as an initial state. Let {π i } be the policies that are optimal for some ρ: {π * ρ |ρ ∈ C(M )}. This sequence is ordered so that ρ − (π i ) is increasing in i 4 . For each i, we introduce a copy of the state space, S i , where the agent is restricted to following π i . Let s i be the analogue of s in S i . For s i ∈ τ ρ−(πi) , we introduce a single durative action, a i , that takes the agent from S i to S i−1 and characterize it as follows: • R(s i ) = R ρ+(πi−1) (s) − R ρ−(πi) (s) • T (s i , a i , s i−1 ) = P retire (s |π i−1 , ρ + (π i−1 ), s) • δ(a i ) = log γ E γ τ π i−1 ρ + (s) − log γ E γ τ π i ρ − (s) For each i, we introduce an action that transitions from s to s i with δ = 0. Let V D (s) represent the value of s in M D (s). Theorem 3. Let M be an MDP with state space S. The following statements are true for s ∈ S and ρ ≥ 0: 1. M D (s) satisfies the Whittle condition. 2. V D (s, ρ) = V (s, ρ).
 V (s, ρ) ≥ V (s, ρ) Proof. See supplementary materials
 AN -OPTIMAL ALGORITHM FOR BANDIT SUPERPROCESSES In this section, we present two algorithms related to BSPs. The first is a novel algorithm to compute V (s, ρ) that can use any method to solve the underlying MDP. The second is an efficient algorithm to compute optimal actions for a BSP. Our approach, Branch-and-Bound Value Iteration (BBVI), uses Whittle integrals to compute upper and lower bounds on value. Then, we apply a modified Branch-and-Bound search to find provably optimal actions.
 COMPUTING A RETIREMENT PROCESS VALUE FUNCTION Before we present BBVI, we give a simple algorithm to compute a retirement process value function that uses an (arbitrary) MDP solver as a black box. Brown and  Smith [2013]  describe an algorithm to compute retirement process value functions, but their approach requires custom implementation of a modified simplex algorithm. Our approach is simple and based on an algorithm that initially appeared in the solutions manual for  Russell and Norvig [2010] . Our goal is to identify each component of V (•, ρ), so our output will be a list of critical points and the associated slopes. First, we will need the following result. Lemma 1. Let M be an MDP with state space S. Let ρ 0 < ρ 1 . Consider a retirement reward in the interior of this interval, ρ ∈ (ρ 0 , ρ 1 ). If, ∀s ∈ S V (s, ρ) = V (s, ρ 0 ) + V (s, ρ 1 ) − V (s, ρ 0 ) ρ 1 − ρ 0 (ρ − ρ 0 ), ( 4 ) then there is at least one policy and stopping rule that is optimal for every ρ ∈ [ρ 0 , ρ 1 ]. The expected value of the discount factor at retirement is the slope between those points: E γ τ * (s) = V (s, ρ 1 ) − V (s, ρ 0 ) ρ 1 − ρ 0 . ( 5 ) Proof. This follows from the form of Equation 1 and the fact that V (s, ρ) is piecewise linear, increasing, and convex. This test allows us to implement a binary search over retirement rewards. Our approach relies on a given MDP solution method, SOLVE, that returns the vector of values for a given MDP. We initialize an interval that contains all critical points and make a call to SOLVE to compute the retirement process value at each endpoint. Then we call SOLVE to compute the retirement process value for the midpoint of our interval and apply the test in Eq. 4. If it succeeds, we return the lower endpoint and the corresponding slope. If this does not, we sub-divide our interval and recurse. We can concatenate the results to get the breakpoints and slopes for the retirement process value function over this interval. Algorithm 1 shows pseudocode for this algorithm. Algorithm 1 Computing an MDP's Critical Points Define: CRITICALPOINTS(M [, ρ − , ρ + , V − , V + ]) Input: MDP, M ; Interval of retirement values, ρ − , ρ + ; Values at interval endpoints, V − , V + if ρ − , ρ + are not set then ρ − ← 0 ρ + ← maxReward(M ) 1−γ V − ← SOLVE(M ρ − ) V + ← ρ + ## Retirement is initially optimal end if ρ ← ρ + −ρ − 2 V ← SOLVE(M ρ ) if |V − V − + (ρ − ρ − ) V + −V − ρ + −ρ − | < then return [ρ − ], V + −V − ρ + −ρ − else pts − , slope − ←CRITICALPOINTS(M, ρ − , ρ, V − , V ) pts + , slope + ←CRITICALPOINTS(M, ρ, ρ + , V, V + ) ## Merge adjacent intervals with the same slope if slope − [−1] = slope + [0] then del pts + [0],del slope + [0] end if return pts − ∪ pts + , slopes − ∪ slopes + end if
 BRANCH-AND-BOUND VALUE ITERATION Now, we present a practical algorithm to compute optimal actions in a bandit superprocess. While a BSP is not indexable, we would like to be able to plan efficiently when the arms are 'close' to indexable-when there are only a few states where the optimal policy changes in response to the opportunity cost of delayed rewards in other arms. Our approach is based on envelope dynamic programming methods to solve MDPs: we compute value estimates for a given initial state by solving a dynamic program defined over a reachable subset of the state space  [Gardiol, Natalia H and Kaelbling, Leslie P, 2003, Hansen and Zilberstein, 2001] . The primary difference between our approach and standard envelope methods is that we use dynamic pro-gramming on our envelope to update an upper bound and a lower bound on the value. This is useful in a BSP because the Whittle integral allows efficient calculation of both bounds. Our goal is to compute the optimal action for a given state s in a BSP, M = M 1 , . . . , M K . We can obtain an upper bound on V (s) with a Whittle integral. We use a Whittle integral for the MAB that solves each arm independently to compute a lower bound. Our algorithm maintains upper and lower bounds on the Q-function for each action that could be executed from s. We write these bounds as Q + (s, a) and Q − (s, a) respectively. If there is a pair of actions a, a such that Q + (s, a ) < Q − (s, a) + , then we can conclude that a is at least -suboptimal. If this test removes all but a single action, then we have found a -optimal action and return it. In the event that more than one action remains, we do a partial expansion of the state space around s. We keep track of a set of expanded states, E, and a set of boundary states, B. States in the boundary set, B, are states that some expanded state can transition to but have not yet been added to E. We can use these states to update the bounds on Q(s, •) by solving a related MDP. This is formalized in the following theorem. Theorem 4. Let M be an MDP with state space S and action space A. Let S ⊆ S where S = B ∪ E and ∀ s ∈ E, T (s, a, s ) = 0 ⇒ s ∈ S . Let M + be an MDP with state space S ∪ {α}. M + 's transition distribution and rewards are identical for states in E (expanded states). Each state s ∈ B transitions deterministically to α and receives a reward that is an upper bound on V M (s). α is a sink state that accrues 0 reward. Then the value of a state in M + is an upper bound on its value in M : V M + (s) ≥ V M (s). Proof. See supplementary materials. It is straightforward to show that an updated lower bound can be computed by fixing boundary states to have value equal to a lower bound. Our approach alternates between pruning actions based on dominance relations, adding states to E, and computing the value of an MDP defined over E. We interleave value iteration with a branch-andbound search, so we call it Branch-and-Bound Value Iteration (BBVI). Algorithm 2 shows pseudocode to implement this method. At termination, the action we return is guaranteed to be at most -suboptimal. Theorem 5. Let M be a BSP and let s be a state in M . Let a be the action returned by BBV I(M, s, ). V (s) − Q(s, a) < . Proof. See supplementary materials. Large regions of the state space are irrelevant to our objective so we use a heuristic to guide node expansion. Our heuristic is a measure of the sensitivity of values at the root to change in the upper and lower bounds of unexpanded nodes. Similar heuristics have been applied in many settings  [Rivest, 1987, Smith and Simmons, 2004] . In BBVI, these values are the expected discounted visitation rates in the bounding MDPs and are computed as the dual variables from a value iteration LP. The backup procedure is slower than node expansions, so we perform backups in batches. In between backups and heuristic computations, we approximate the state space as a tree and push new states onto the agenda with their parent's heuristic value weighted by the transition probability.  Q + ← SOLVE(BOUNDMDP(E , B, Q + )) Q − ← SOLVE(BOUNDMDP(E , B, Q − )) a * ← argmax a Q − (s 0 , a) end while return -optimal action a *
 EMPRIRICAL EVALUATION We implemented BBVI in Python and use the linear programming solver Gurobi to compute value functions. Our experiments were run on an Intel i7 with 16 GB of RAM. In our first experiments, we compare the scalability of BBVI with that of standard MDP algorithms. In particular we compare the following algorithms: • SPUDD: the baseline factored MDP algorithm in the 2011 IPPC  [Hoey et al., 1999] . • LRTDP(Merge): Labeled Real-Time Dynamic Programming  [Bonet and Geffner, 2003]  with the upper bound from  Singh and Cohn [1998]  as a heuristic. • LRTDP(WI): LRTDP with the Whittle integral as an upper bound. • BBVI: Branch-and-Bound Value Iteration. We evaluate performance on a simple BSP designed to model allocation of research resources to product research and development. Each arm in this problem corresponds to a potential product that our agent could sell in the market. However, before we sell a product we must devote effort to R&D. This can be done in one of two ways, a safe research approach that is slow but reliable, and a risky approach that is fast, but runs a risk of producing a defective product. After at least one product has been researched, the agent can opt to spend a round produce and sell that product or continue research on a different project. Because taking a product to market does not change the state of the BSP, it is straightforward to show that any stationary policy (including π * ) will only ever produce a single product. We define a distribution over these problems by selecting a random market value for each product uniformly from [0, 1] and random durations for the safe and risky research strategies (although we ensure that safe research is at least 3x as slow as risky research). Figure  2  (a) shows the state space for an example arm with typical parameters. Figure  2  (b) shows the results of our comparison. We used a timeout of 1000s and set a memory limit of 4 GB. We ran our experiments in an online setting and running times reflect the amount of time to select an optimal action from scratch. While this is a natural setting for LRTDP and BBVI, it is admittedly a little unfair to SPUDD (which computes a full policy). This is mitigated by the fact that BBVI's improvement over SPUDD, which is a little under 5 orders of magnitude with 10 8 states, is such that an agent would need to be planning over an immense horizon before SPUDD presents a reasonable alternative. The factored MDP bounds from  Singh and Cohn [1998]  are quite ill-suited to this problem. This is because its upper bound (the sum of the independent value for each arm) is very loose and it essentially forces LRTDP to enumerate the state space. In contrast, LRTDP performs quite well when it has good heuristic information. However, its performance degrades faster than that of BBVI, because it does not effectively leverage a lower bound on value. Even for very large problem instances, BBVI computes optimal solutions in well under a second. BBVI's performance in the R&D domain is largely explained by the structure of the arms and the short horizon within each arm. Although there may be a large number of arms, BBVI reduces each individual arm to an equivalent MRP after a small number of node expansions. This means that running time is essentially linear in the number of arms. Note that, in this domain, the solution that solves each MDP independently and executes the corresponding index policy will just opt for conservative research on the most valuable option. This policy is essentially always suboptimal: as long as there are reasonable alternatives it is usually a good idea to try some risky projects. Our next experiment uses a synthetic domain to explore the performance of BBVI as the structure and number of arms changes. In this BSP, each arm is a 20x20 grid world. The actions in this world correspond to moving in the 4 cardinal directions. Rewards are 0 everywhere except at randomly chosen locations and after receiving a reward, the agent transitions to a random location. In this experiment we vary the number of rewards that are available in each arm. Adding rewards will typically cause the optimal policy for an arm to be more sensitive to context. This means that the dominated relaxation will produce a looser upper bound, because it will need to add more actions. We evaluate this difficultly by measuring the convergence criterion of BBVI after 10,000 node expansions. Figure  2  (c) shows the results of this experiment. We can see that BBVI's solution quality decreases with the number of states, but the primary variation comes from changes in the properties of the arms.
 CONCLUSION AND FUTURE WORK In summary, we presented a model of highly decoupled decision making: bandit superprocesses. A BSP presents an agent with the opportunity to act in one of several Markov decision processes. The key constraint is that the agent can only act in a single process at a time. We provided a summary of BSP research, including an upper bound for the value function of a BSP in the form of the Whittle integral. We derived an equivalent relaxation and thus gave a novel proof that the Whittle integral is an upper bound. Finally we presented and evaluated algorithmic solutions for this class of decision problems. In future work, we plan to extend these ideas in three directions. The first is algorithmic improvements for BSPs. A potential direction here is to leverage factored dynamics in the bounding MDPs themselves. A search over sequences of MDPs that only consider states where the current policy stops could be more efficient. Next, also plan to explore generalizations of the bounds used for BSPs to more general cases of global resource constraints (e.g., those considered in  Meuleau et al. [1998] ). Finally, the similarity to sums of games studied in  Conway [2000]  suggests an unexplored connection between BSPs and combinatorial game theory. Figure 1 : 1 Figure 1: (a-b) Retirement process value functions for the reward chains Ex. 1. The slope of the lines is the expected value of the discount parameter at retirement: E [γ τ ]. Flat sections indicate regions of retirement rewardswhere retirement is always suboptimal. The kink in the green curve at ρ = 280 indicates that it has become optimal to retire immediately. The kink in the black curve at ρ = 14 increases the slope to γ 6 < 1; for ρ > 14, it is optimal to retire after collecting a prefix of the reward stream from y 2 . (c) Dominated relaxation of the MDP in Ex. 2. We add additional durative actions to the state space to ensure that a dominating policy exists. Note that the original MDP does not admit a dominating policy.
 Algorithm 2 2 Branch-and-Bound Value Iteration Define:BBVI( M 0 , . . . , M K , s 0 , ) Input: BSP arms, M k ; Initial state, s 0 ; Tolerance, # Lower bound M by fixing a policy for each arm LB k ←toMRP(M k ) Compute critical points for M k , LB k Compute bounds on Q(s 0 , •) with Whittle integrals for M and LB. # Keep track of expanded region of state space E ← ∅ # Keep track of states at boundary of M + , M − B ← {s 0 } a * ← argmax a Q − (s 0 , a) while ∃a = a * s.t Q + (s 0 , a ) ≥ Q − (s 0 , a * ) do s ←pop(B) E ← E ∪ {s} for a ∈ A do for s ∈ successors(s) do Compute upper and lower bounds for Q(s , •) B ← B ∪ {s } end for end for
 Figure 2: (a) the state space for an arm of a R & D BSP. All states accrue 0 reward except for the 'Working Product' state, which collects a fixed reward per time step. (b) shows runtime (log-scale) vs problem size for BBVI and several alternatives on an R & D BSP. SPUDD [Hoey et al., 1999]  and LRTDP (with the heuristic from Singh and Cohn [1998] ) scale poorly compared with algorithms that leverage the Whittle integral. BBVI's additional improvement over LRTDP stems from the use of an efficient, exact lower bound to check convergence. (c) BBVI's bound on suboptimality after 10,000 node expasions (measured in units of discounted reward). Changing the number of rewards in each arm allows us to measure BBVI's performance as arms become more sensitive to context.
			 This result first appears in the literature as an intermediate step in a larger proof from Whittle [1980] . It went unnoticed or unappreciated in the intervening 30+ years.
			 Here we use the fact that the Gittins index of a non-increasing reward sequence is equal to the instantaneous reward.4 AN UPPER BOUND FOR BANDIT SUPERPROCESSES
			 Our result is a small extension on a related result in Brown and Smith [2013] ; it is primarily included to provide intuition.
			 Recall that [ρ−(π), ρ+(π)] is the interval of retirement rewards where π is optimal.

Abstract: A bandit superprocess is a decision problem composed from multiple independent Markov decision processes (MDPs), coupled only by the constraint that, at each time step, the agent may act in only one of the MDPs. Multitasking problems of this kind are ubiquitous in the real world, yet very little is known about them from a computational viewpoint, beyond the observation that optimal policies for the superprocess may prescribe actions that would be suboptimal for an MDP considered in isolation. (This observation implies that many applications of sequential decision analysis in practice are technically incorrect, since the decision problem being solved is often part of a larger, unstated bandit superprocess.) The paper summarizes the state-of-theart in the theory of bandit superprocesses and contributes a novel upper bound on the global value function of a bandit superprocess, defined in terms of a direct relaxation of the arms. The bound is equivalent to an existing bound (the Whittle integral), but is defined constructively, as the value of a related multi-armed bandit. We provide a new method to compute this bound and derive the first practical algorithm to select optimal actions in bandit superprocesses. The algorithm operates by repeatedly establishing dominance relations between actions using upper and lower bounds on action values. Experiments indicate that the algorithm's run-time compares very favorably to other possible algorithms designed for more general factored MDPs.