Dmitrii Krasheninnikov
Rohin Shah
U C Berkeley

Combining reward information from multiple sources

Introduction While deep reinforcement learning (RL) has led to considerable success when given an accurate reward function  [28, 21] , specifying a reward function that captures human preferences in real-world tasks is challenging  [1, 6] . Value learning seeks to sidestep this difficulty by learning rewards from various types of data, such as natural language  [22] , demonstrations  [33, 11] , comparisons  [6] , ratings  [7] , human reinforcement  [18] , proxy rewards  [13] , the state of the world  [27] , etc. Many approaches to value learning aim to create a distribution over reward functions  [14, 23, 24, 13] .  A natural idea is to combine the information from these diverse sources. However, the sources may conflict. For example, in Figure  1 , Alice might say that she wants the healthy avocado toast, but empirically she almost always chooses to eat the tasty cake. A reward learning method based on natural language may confidently infer that Alice would prefer the toast, whereas a method based on revealed preferences  [25]  might confidently infer that she wants cake. Presumably one of the algorithms made a poor assumption: perhaps the natural language algorithm failed to model that Alice only said that she wants avocado toast to impress her friends. Alternatively, maybe Alice wants to start eating healthy, but chooses the cake in moments of weakness, and the revealed preferences algorithm failed to model this bias. While it is unclear what the "true" reward is, we do know that at least one of the methods is confidently wrong. We would like to defer on the question of what the "true" reward is, and instead retreat to a position of uncertainty. Formally, we want a distribution over the true reward function θ * given the inferred rewards θ 1 , θ 2 . We could apply Bayes Rule: p(θ * | θ 1 , θ 2 ) ∝ p(θ * )p( θ 1 | θ * )p( θ 2 | θ * ). However, when rewards conflict due to misspecification, one of the likelihoods will be very low, and the posterior will be low everywhere, likely leading to garbage after normalization  [10] . While it is possible to mitigate this by making the assumptions more realistic, e.g. by modeling human biases  [26, 19, 9] , it is very hard to get a perfect model  [3, 29] . So, we aim for a method that is robust to misspecification. What makes for a good reward combination method? To answer this question, we must know how the resulting distribution will be used. As in Hadfield-Menell et al.  [14] , we assume that an agent maximizes expected reward, given that the agent can gather more information about the reward. In this framework, an agent must balance three concerns: actively learning about the reward  [24, 20] , preserving its ability to pursue the true reward in the future when it knows more  [30, 17] , and acting to gain immediate reward. Each concern suggests different desiderata for a reward combination method, which we formalize for rewards functions that are linear in features of the state. There are several reasonable methods for combining reward functions based on the intuition that we should put weight on any parameter values "in between" the parameters of the given reward functions. These reward-space methods operate in the space of parameters of the reward. However, this ignores any details about the environment and its features. We introduce a method called Multitask Inverse Reward Design (MIRD) that leverages the environment to infer which reward functions are compatible with the behaviors "in between" the behaviors incentivized by the two input rewards, making it a behavior-space method. Through a combination of theory and empirical results on a simple environment, we evaluate how well the proposed methods meet our desiderata, and conclude that MIRD is a good option when the reward distribution must be used to act, while its variant MIRD-IF is better for increased robustness and a higher chance of support on the true reward.
 Background Markov Decision Process (MDP). A MDP M is a tuple M = S, A, T , r, H , where S is the set of states, A is the set of actions, T : S × A × S → [0, 1] is the transition probability function, r : S → R is the reward function, and H ∈ Z + is the finite planning horizon. We consider MDPs where the reward is linear in features, and does not depend on action: r(s; θ) = θ T f (s), where θ are the parameters defining the reward function and f computes features of a given state. A policy π : S → A specifies how to act in the MDP. A trajectory τ : (S × A) * is a sequence of states and actions, where the actions are sampled from a policy π and the states are sampled from the transition function T . We abuse notation and write f (τ ) to denote s∈τ f (s). The feature expectations (FE) of policy π are the expected feature counts when acting according to π: F π = E τ ∼π [f (τ )]. The value of a policy π is given by V π = E τ ∼π [ s∈τ r(s; θ)] = θ T F π . The optimal policy is π * = max π V π . Reward learning by simulating the past (RLSP). Shah et al.  [27]  note that environments where humans have acted are already optimized for human preferences, and so contain reward information. The RLSP algorithm considers the current state s 0 as the final state in a human trajectory τ = (s −H , a −H , ..., s 0 , a 0 ) generated by the human with preferences θ. The RLSP likelihood p(θ | s 0 ) is obtained by marginalizing out (s −H , a −H , . . . s −1 , a −1 , a 0 ), since only s 0 is actually observed. Inverse Reward Design (IRD). Inverse reward design (IRD) notes that since reward designers often produce rewards by checking what the reward does in the training environment using an iterative trial-and-error process, the final reward they produce is likely to produce good behavior in the environments that they used during the reward design process. So, θ need not be identical to θ * , but merely provides evidence about it. Formally, IRD assumes that the probability of specifying a given θ is proportional to the exponent of the expected true return in the training environment: p( θ | θ * ) ∝ exp(θ * F π θ ). IRD inverts this generative model of reward design to sample from the posterior of the true reward: p(θ * | θ) ∝ p( θ | θ * )p(θ * ).
 Uses of the reward distribution So far, we have argued that we would like to combine the information from two different sources into a single distribution over reward functions. But what makes such a distribution good? We take a pragmatic approach: we identify downstream tasks for such a distribution, find desiderata for a reward distribution for these tasks, and use these desiderata to guide our search for methods.
 Optimizing the reward We follow the framework of Hadfield-Menell et al.  [14]  in which an agent maximizes expected return under reward uncertainty. In this setting, maximizing the return causes the agent to use its actions both to obtain (expected) reward, and to gather information about the true reward in order to better obtain reward in the future  [14, 31] . We might expect that an agent could first learn the reward and then act; however, this is unrealistic: we would be unhappy with a household robot that had to learn how we want walls to be painted before it would bake a cake. The agent must act given reward uncertainty, but maintain its ability to optimize any of the potential rewards in the future when it will know more about the reward. Put simply, it needs to preserve its option value while still obtaining as much reward as it can currently get. We would like to evaluate our reward distribution by plugging it into an algorithm that maximizes expected reward given reward uncertainty, but unfortunately current methods  [14, 31]  are computationally expensive and only work in environments where option value preservation is not a concern. So, we analyze reward learning and option value preservation separately. Active reward learning. An agent that learns rewards while acting will need to select queries with care. Active reward learning aims to pick queries that maximize the information gained about the reward, or minimize the expected regret  [24, 20]  3 . We analyse how the reward distributions resulting from different reward combination methods fit as priors for the active reward learning methods. Option-value preservation. Low impact agents  [30, 17, 2]  take fewer irreversible actions, and so preserve their ability to optimize multiple reward functions. However, work in this area does not consider the possibility of learning more information about the reward function, and so these methods penalize entire plans that would have a high impact. In contrast, we would like our agent to pursue impactful plans, but pause when option value would be destroyed, and get more reward information. For our purposes, we start with the attainable utility preservation (AUP) method introduced by Turner et al.  [30] . AUP formalizes the impact of an action on a reward function as the change in the Q-value as a result of taking that action (relative to a no-op action ∅). For our purposes, we only care about cases where we are no longer able to optimize the reward function, as opposed to ones where we are better able to optimize the reward function. So, we use the truncated difference summary function proposed by Krakovna et al.  [17]  that only counts decreases in Q-values and not increases: Impact(s, a) = E r [max(Q r (s, ∅) − Q r (s, a), 0)]. (1) Turner et al.  [30]  penalize the agent whenever it would cause too much impact, using a hyperparameter λ to trade off task reward with impact. They plan using the reward R task − λ Impact(s, a). However, by penalizing impact in the reward function, the agent is penalized for any plan that would cause impact, whereas we want our agent to start on potentially impactful plans, but stop once a particular action would destroy option value. So, we only penalize impact during action selection: π(s) = max a (Q(s, a) − λ Impact(s, a)). (2)
 Acting in a new environment Often, we wish to use the reward function in the same environment as the one in which it was specified or inferred. For example, a human expert may have demonstrated how to bake a cake in a particular kitchen. Since we know at training time that the agent will be deployed in the same kitchen, the only guarantee we need is that the agent's behavior is correct. If two rewards lead to the same behavior in this kitchen, we don't care which one we end up using. However, we may instead want to apply a reward function learned in one environment to a different environment. In this case, there may be new options in the test environment that we never saw during training. As a result, even if two rewards led to identical behavior in the training environment, they need not do so in the test environment. Hence when the goal is to transfer the reward distribution to a new environment, it is crucial to ensure that the distribution contains the true reward, instead of just containing a reward that gives rise to the same behavior as the true one in the training environment.
 Desiderata for the reward posterior We discuss properties of the posterior that inform the choice of a reward combination algorithm for different applications of the resulting reward distribution. We focus on two applications: option-value preserving planning and active reward learning.
 Desired properties for any subsequent application All else equal, we want the true reward to have high probability in the reward distribution. One model for misspecification is that the weight for each feature has some chance of being "corrupted", and the chance of the feature being corrupted in one reward is independent from that of the other reward. If the fraction of misspecified values is small, then if two reward vectors' values are different at a particular index, it is likely that one of the values is correct. Given this, it is desirable for the reward distribution to assign significant probability to every reward function s.t. the value of the weight for each feature is in one of the values of that feature's weight of the input rewards. Formally, we denote the set of such reward parameters Θ indep pts = {θ ∈ R n | θ i ∈ { θ 1 i , θ 2 i } ∀i ∈ {1, ..., n}}. (3) Desideratum 1 (Support on Θ indep pts ). All θ * ∈ Θ indep pts should be in the support of p(θ * | θ 1 , θ 2 ). Note that when we do not need to transfer the reward function to a new environment, Desideratum 1 can be made weaker. We only require that the distribution have support on reward functions that give rise to the same behavior as the rewards in Θ indep pts , and not necessarily on Θ indep pts itself. In order for the agent to act, the reward distribution must provide information about useful behavior: Desideratum 2 (Informative about desirable behavior). If θ 1 and θ 2 give rise to the same behavior, most of the reward functions in the distribution should give rise to this behavior. The strong version of this desideratum states that if the FEs of { θ 1 , θ 2 } are equal on some feature, the FEs of this feature of any reward in the support of the posterior are equal to those of { θ 1 , θ 2 }: ∀i ∈ {1, n} : F π θ 1 i = F π θ 2 i =⇒ ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * i = F π θ 1 i = F π θ 2 i . (4) The medium version considers the FEs as a whole instead of per-feature: F π θ 1 = F π θ 2 =⇒ ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = F π θ 1 = F π θ 2 . ( 5 ) The weak version assumes that the FEs of all θ ∈ Θ indep pts are the same, rather than just θ 1 and θ 2 : ∀θ ∈ Θ indep pts : F π θ = F π θ 1 = F π θ 2 =⇒ ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = F π θ 1 . (6) Des. 2 allows option-value preserving planning to do useful things instead of performing the no-op action all the time. Furthermore, by restricting possible behaviors, Des. 2 simplifies the job of approaches that actively learn the reward function. If the reward distribution used by our household robot always incentivizes using soap while washing dishes (because both input rewards did so), the robot's active learning algorithm does not need to query the human about whether or not to use soap. Des. 2 might be problematic if both input rewards agree on an undesirable behavior w.r.t. a particular feature. This presents a natural tradeoff between informativeness and the abilities to preserve option value and actively learn arbitrary reward functions: the more informative we assume the input rewards are about the desired behavior, the less robust we are to their misspecification.
 Desired properties for option value preservation Suppose we would like to preserve the option value of pursuing two different behaviors. Option-value preservation is more robust when the distribution over FEs induced by the reward distribution places significant and approximately equal weight on the two behaviors. Intuitively, the "weight" AUP puts on preserving expected return attainable by a particular behavior is proportional to the probability of that behavior in the reward distribution. So we would like the FEs of the input rewards to be balanced across the reward distribution. Concretely: Desideratum 3 (Behavior-space balance). The FEs of the rewards sampled from the distribution should correspond to behaviors arising from the input reward functions with a similar frequency: p(F π θ = F π θ 1 ) ≈ p(F π θ = F π θ 2 ) | θ ∼ p(θ | θ 1 , θ 2 ). (7)
 Desired properties for active reward learning Given a reward vector we define the tradeoff point between the i-th and j-th features of the state as the ratio between the reward vector values corresponding to these features: T θ i,j = θi θj . The true reward's tradeoff point may be in between the tradeoff points of the input rewards, and so we would like our distribution to support all such tradeoff points. Desideratum 4 (Support on intermediate feature tradeoffs). The reward posterior should have support on rewards that prescribe all intermediate tradeoff points between the states' features. In practice tradeoff points only affect the behavior to the extent allowed by the environment. We only need to maintain all intermediate tradeoff points if we want to transfer to a new environment.
 Multitask Inverse Reward Design Inspired by Inverse Reward Design  [13] , we introduce a reward combination method Multitask Inverse Reward Design (MIRD). We assume that the two input rewards meet the IRD assumption for two different rewards, namely that the input rewards generate nearly optimal behavior in the training environment according to the corresponding task rewards. A straightforward way to construct the distribution for the combined reward function is to require the FEs of the rewards in the distribution to be convex combinations of the FEs of the input rewards: F π θ * ∈ {bF π θ 1 + (1 − b)F π θ 2 | b ∈ [0, 1]}. To satisfy this requirement, we propose a distribution that depends on the behavior (set of trajectories) D = {τ 1 , ..., τ d } resulting from the agents optimizing θ 1 and θ 2 : p(θ * | θ 1 , θ 2 ) = D p(θ * | D)p(D | θ 1 , θ 2 ) dD (8) Each of the reward functions {θ i | i = {1, ..., d}} used to generate the trajectories τ i in D is sampled from the Bernoulli mixture over the input reward functions. We introduce a random variable b ∈ [0, 1] that determines the probability of sampling θ i = θ 1 to generate the trajectory τ i : p(D | θ 1 , θ 2 ) = b Beta(b | β 1 , β 2 ) n i=1 bp(τ i | θ 1 ) + (1 − b)p(τ i | θ 2 ) db (9) We sample from p(τ i | θ i ) using policies arising from soft value iteration. For p(θ * | D), we set θ * to the reward learned by Maximum Causal Entropy Inverse RL (MCEIRL)  [33] . Denoting the Dirac delta as δ(), we have p(θ * | D) = δ(θ * − argmax θ p(θ | D)). Having defined all relevant conditional distributions, we can sample from the joint p(θ * , b, D, θ i | θ 1 , θ 2 ) and the marginal p(θ * | θ 1 , θ 2 ). Theorem 1. The FEs arising from rewards θ * in the support of the MIRD posterior are convex combinations of FEs of the input reward vectors θ 1 and θ 2 : ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = αF π θ 1 + (1 − α)F π θ 2 , α ∈ [0, 1]. (10) Proof. See appendix A for full proof. Key to the proof is the fact that MCEIRL finds such θ * that its FEs match those of the soft value iteration expert who generated the trajectories  [33] . In our case, the FEs of θ * match D, which contains trajectories arising from θ 1 and θ 2 in various proportions. Corollary 1.1. The average FEs of the reward functions in the posterior are: E θ∼p(θ| θ 1 , θ 2 ) [F π θ ] = E b [bF π θ 1 + (1 − b)F π θ 2 ] = E b [b] F π θ 1 + (1 − E b [b])F π θ 2 . ( 11 ) Corollary 1.2. Eq. 11 implies that if the FEs of the θ 1 and θ 2 w.r.t. a given feature are equal, the FEs of all rewards in the distribution for that feature equal the FEs of θ 1 and θ 2 for that feature. Thus MIRD satisfies the strong version of Des. 2 (informative about desirable behavior). Corollary 1.3. When β 1 = β 2 the probability of sampling θ i = θ 1 equals the probability of sampling θ i = θ 2 , and hence the probabilities of observing behaviors F π θ 1 and F π θ 2 when optimizing a reward function from the posterior are equal. Hence Des. 3 (behavior balance) is satisfied. Theorem 1 allows us to bound the expected return θ true F π θ , θ ∈ supp(p(θ * | θ 1 , θ 2 )) of the true reward function θ true in terms of expected true returns of F π θ 1 and F π θ 2 : Theorem 2. The minimum expected true return following any of the rewards in the MIRD posterior equals the minimum true return following one of the input rewards: min θ∈supp(p(θ * | θ 1 , θ 2 )) θ true F π θ = min θ∈{ θ 1 , θ 2 } θ true F π θ . ( 12 ) Proof. See Appendix B. Theorem 2 is the best possible regret bound for following rewards in a distribution resulting from reward combination: the worst the agent can do when following a reward from the distribution is no worse than when just randomly choosing to follow one of the input reward functions. Generally FEs arising from θ ∈ Θ indep pts \{ θ 1 , θ 2 } are not convex combinations of F π θ 1 and F π θ 2 , so MIRD does not satisfy any version of Des. 1 (support on Θ indep pts ). In addition, when θ 1 and θ 2 conflict on more than one feature, Des. 4 (support on intermediate feature tradeoffs) is not satisfied. Independent features formulation. Since both Des. 1 and Des. 4 aim to ensure robustness to reward misspecification, one of our primary goals, we would like a variant of MIRD that satisfies them. An algorithm that supports rewards that give rise to all potential behaviors arising from reward functions in Θ indep pts is a simple modification of MIRD. The only change required in the original formulation of MIRD is sampling the reward vectors {θ i } giving rise to the trajectories in D from Θ indep pts instead of just { θ 1 , θ 2 }. Now b is a vector of dimensionality |Θ indep pts |, sampled from Dirichlet(b | β 1 , ..., β |Θ indep pts | ) . Each θ i is sampled from a Multinoulli distribution over Θ indep pts , parameterized by b. We refer to this version of MIRD as MIRD independent features, or MIRD-IF.
 Analysis We first introduce several baselines each of which does not meet some of the outlined desiderata, and then analyze all reward combination methods in the context of our modified version of AUP.
 Reward-space baselines Additive. The additive reward combination method is commonly used in previous work utilizing reward combination  [8, 27, 16] . The reward posterior is simply p(θ * | θ 1 , θ 2 ) = δ(θ * − ( θ 1 + α θ 2 )), where α is the hyperparameter used to balance the two input reward functions. Gaussian. Here, we use the standard Bayesian framework, and assume that θ 1 and θ 2 are generated from θ * with Gaussian noise: p(θ * | θ 1 , θ 2 ) ∝ p(θ * )p( θ 1 | θ * )p( θ 2 | θ * ) = p(θ * )p N ( θ 1 | θ * , σ 2 1 I)p N ( θ 2 | θ * , σ 2 2 I ), where I is the identity matrix, p N is the multivariate Gaussian probability density function, and {σ 1 , σ 2 } are hyperparameters controlling the standard deviations. One can express higher trust in a given input reward by lowering its σ. For our experiments, we use an uninformative Gaussian prior (that is, σ → ∞) and set σ 1 = σ 2 = 1. The posterior is then p(θ * | θ 1 , θ 2 ) = p N (θ * | θ 1 + θ 2 2 , 1 2 I).
 Convex combinations of input reward vectors (CC-in). Consider a scenario where the input reward functions are correctly specified for two different tasks. A natural reward-space analogue to MIRD would be to consider a distribution over all convex combinations of the input reward vectors: p(θ * | θ 1 , θ 2 ) = b Beta(b | β 1 , β 2 )δ(θ * − b θ 1 − (1 − b) θ 2 ) db. ( 13 ) Theorem 3. If F π θ 1 = F π θ 2 , the FEs of any reward in the support of the CC-in posterior equal F π θ 1 . Hence CC-in satisfies the medium version of Des. 2 (informative about desirable behavior). Proof. See Appendix C. We provide an example showing that CC-in does not satisfy the strong version of Des 2 in Appendix F. Furthermore, CC-in does not satisfy Des. 1 (support on Θ indep pts ): consider combining rewards [0, 1, 1] and [1, 0, 1]. Here θ = [0, 0, 1] ∈ Θ indep pts , but is not a convex combination of input rewards. Convex combinations of θ ∈ Θ indep pts (CC-Θ indep pts ). Instead of only considering convex combinations of the input rewards, we consider convex combinations of θ ∈ Θ indep pts . The vector b ∈ R |Θ indep pts | of weights for each convex combination is sampled from the Dirichlet distribution:  To demonstrate the use of the desiderata, as well as how each method performs on these desiderata, we developed a toy environment in which to test reward combination methods. Due to space limitations, we defer full qualitative and quantitative explanations to Appendix E and report the broad results here. p(θ * | θ 1 , θ 2 ) = b Dirichlet(b | β 1 , ..., β |Θ indep pts | )δ   θ * − |Θ indep pts | i=0 b i Θ indep pts i   db. (14
 Analysis on a toy environment Setup. We use the setting of AUP planning to demonstrate the importance of support on Θ indep pts , informativeness about desired the behavior, and behavior balance desiderata. We obtain the reward posterior by combining the input rewards using each of our seven reward combination methods, and qualitatively evaluate the behavior of AUP planning w.r.t. the posterior over five seeds. The input rewards are either both specified by hand, or θ 1 is learned with RLSP while θ 2 is hand-specified. We use the cooking environment described in Figure  2 . The states' features indicate the number of jars of flour, pieces of dough, cakes, avocado toasts, and servings of pasta. At each state the agent can move, cook, or do nothing. Cooking is possible when the agent is to the left of flour or dough. Support on Θ indep pts . We specify both input rewards to strongly incentivise either cake or toast by setting θ 1 cake = θ 2 toast = 3; θ 1 toast = θ 2 cake = 0, and to mildly incentivise pasta: θ 1 pasta = θ 2 pasta = 1. The desired behavior is to preserve the flour, as any one of the foods might be desirable. This requires the distribution to place weight on rewards such as θ cake = θ toast = 0; θ pasta = 1 which would make pasta. As expected, methods that have support on Θ indep pts behave as desired, and other methods do not. Informativeness about desirable behavior (strong). In the setting above the two input rewards lead the agent to make either cake or toast, but not pasta. So, a reward distribution meeting the strong version of Des. 2 would only make cake or toast, and so the agent should make dough (which is useful for both). Note that despite the same setup, this is exactly the opposite behavior than desired previously, showing the tradeoff between informativeness and support on Θ indep pts . As such, the results are reversed. Informativeness about desirable behavior (weak). We hand-specify one of the reward functions to incentivise the agent to make cakes: θ 2 cake = 1, θ 2 toast = θ 2 pasta = 0. θ 1 is inferred by RLSP with a uniform prior over s −T . Since the current state contains a cake, RLSP infers a positive reward for cakes, θ 1 cake ≈ 0.1, and near-zero rewards for the other features. Hence we have two rewards that both incentivise the agent to make another cake. The desired behavior is to make the cake, and not worry about preserving the ability to make pasta or toast as neither input reward cares about it. All but the Gaussian method satisfy the weak informativeness desideratum, and so they all succeed. The Gaussian method fails since it thinks θ * cake could be negative. Behavior balance. θ 2 is specified to only reward toast, θ 2 toast = 1, θ 2 cake = θ 2 pasta = 0, and θ 1 is specified only reward cake: θ 1 cake > 0, θ 1 toast = θ 1 pasta = 0. So we have two disagreeing rewards: one incentivises cake, while the other incentivises toast. The desired behavior from AUP here is to make dough as it is desirable for both the cake and the toast, but then to stop as the next step is unclear. Ensuring that the reward for toast does not overwhelm the reward for cake requires behavior balance. We vary θ 1 cake between 0.01 and 0.7 to obtain many degrees of imbalance between toast and cake. MIRD and MIRD-IF have the lowest frequency of the toast reward overwhelming the cake reward, which indicates that MIRD and MIRD-IF are most behavior balanced. However, occasionally these methods do nothing instead of making dough.
 Related Work Utility aggregation. Combining the specified reward functions is in many ways similar to combining utility functions of different agents. Harsanyi's aggregation theorem  [15]  suggests that maximizing a fixed linear combination of utility functions results in Pareto-optimal outcomes for correctly specified utility functions. This corresponds to our Additive baseline. Multi-task and meta IRL. Multi-task IRL assumes that the demonstrations originate from multiple experts performing different tasks, and seeks to recover reward functions that explain the preferences of each of the experts  [4, 5] . The IRL part of MIRD and MIRD-IF is conceptually similar to these approaches: the trajectories in each demonstration set D can be seen as arising from an expert executing different tasks. However, MIRD seeks to explain each D as if the trajectories arose from a single reward function . This resembles meta IRL, which learns an initialization for the reward function parameters that is helpful for learning the rewards of new tasks  [32, 12] .
 Limitations and future work Summary. We analyze the problem of combining two potentially misspecified reward functions into a distribution over rewards. We identify active reward learning and option value preservation as two key applications for such a distribution, and determine four properties that the distribution would ideally satisfy for these applications. We suggest some simple reward-space methods as well as a behavior-space method, Multitask Inverse Reward Design (MIRD), which is grounded in the behavior arising from the input rewards. MIRD works well for most applications, and MIRD-IF can be used for greater robustness and increased likelihood that the true reward is in the distribution. If the only application is to preserve option value, then the uniform points method also works well. Future work. The primary avenue for future work is to evaluate the reward combination methods in the context of active reward learning, and formalize Des. 4 (support on intermediate tradeoffs) in a way most helpful for active learning. Another avenue for future work is scaling up MIRD to realistic environments, where the reward function may be nonlinear.
 A Proof of Theorem 1 D is a set of trajectories each of which arises either from optimizing θ 1 or optimizing θ 2 . The fraction of trajectories in D that result from θ 1 is b, and the fraction of trajectories arising from θ 2 is 1 − b. Hence the FEs of D are F D = bF π θ 1 + (1 − b)F π θ 2 . ( 15 ) p(θ * | D) = δ(θ * − argmax θ p(θ | D) ) entails that each sample θ * is the MCEIRL reward θ * = argmax θ p(θ | D). MCEIRL finds a reward vector θ * such that its FEs match those of the soft value iteration expert who generated the trajectories  [33] . Hence each sampled θ * gives rise to FEs F π θ * matching D. This and Eq. 15 together imply that the FEs arising from optimizing any θ * ∈ supp(p(θ * | θ 1 , θ 2 )) with soft value iteration are a convex combination of the FEs of the input rewards: ∀θ * ∈ supp(p(θ * | θ 1 , θ 2 )) : F π θ * = bF π θ 1 + (1 − b)F π θ 2 , b ∈ [0, 1]. (16)
 B Proof of Theorem 2 By Theorem 1, the FEs arising from any θ in the posterior are a convex combination of the FEs of the input rewards. Hence finding θ in the support of p(θ * | θ 1 , θ 2 ) that minimizes θ true F π θ is equivalent to finding α ∈ [0, 1] that minimizes θ true (αF π θ 1 + (1 − α)F π θ 2 ) = αθ true F π θ 1 + (1 − α)θ true F π θ 2 . Since the true returns when following policies arising from the input reward functions θ true F π θ 1 and θ true F π θ 2 are scalars, the term above is minimized with α = 1 when θ true F π θ 1 < θ true F π θ 2 , and α = 0 otherwise.
 C Proof of Theorem 3 Since the feature expectations (FEs) of the input rewards are equal, the return of any mixture reward b θ 1 − (1 − b) θ 2 for those FEs is F π θ 1 (b θ 1 − (1 − b) θ 2 ) . Suppose there exist FEs F that result in a higher expected return for the mixture reward: F (b θ 1 − (1 − b) θ 2 ) > F π θ 1 (b θ 1 − (1 − b) θ 2 ) (17) For this to be true F must result in a higher return than F π θ 1 for either θ 1 or θ 2 . This is a contradiction, since F π θ 1 already results in the highest return for both input rewards. Hence F π θ 1 corresponds to the optimal behavior for any mixture reward b θ 1 − (1 − b) θ 2 ∀b ∈ [0, 1].
 D Proof of Theorem 4 Analogous to the proof of Theorem 3 (Appendix C).
 E Analysis details In this appendix we provide more details and explanations regarding the behavior of the various reward combination methods in the Cooking environment.
 E.1 Support on Θ indep pts We hand-specify both input reward functions to strongly incentivise either cake or toast by setting θ 1 cake = θ 2 toast = 3; θ 1 toast = θ 2 cake = 0, and to mildly incentivise pasta: θ 1 pasta = θ 2 pasta = 1. The desired behavior from AUP here is to preserve the flour, as any one of the foods requiring it might be desirable. The Additive method returns a single reward function that incentivises the agent to make both cake and toast. AUP applied to one reward function simply optimizes it without needing to preserve option value, so here it makes either cake or toast (and does not preserve the flour). The CC-in method does not preserve the flour, choosing to make dough instead. Because of the high values of θ 1 cake and θ 2 toast , optimizing any convex combination of { θ 1 , θ 2 } leads the agent to either make cake or toast, and so AUP infers that it does not need to preserve the ability to make pasta. MIRD chooses to make dough for a similar reason: since all trajectories in D arise from optimizing θ 1 or θ 2 , each trajectory demonstrates the agent making either cake or toast, and no trajectory demonstrates it making pasta. So no reward inferred from D incentivises the agent to make pasta, and hence AUP does not preserve the ability to make pasta. All methods whose reward samples sometimes give rise to the behaviors arising from θ ∈ Θ indep pts preserve the flour. This is because Θ indep pts contains θ s.t. optimizing it leads the agent to make pasta: θ cake = θ toast = 0; θ pasta = 1. As MIRD-IF, Gaussian, uniform points, and CC-Θ indep pts methods all have substantial probability mass on rewards that incentivise making pasta, AUP applied to the reward samples from these methods preserves the ability to make pasta.
 E.2 Informativeness about desirable behavior (strong). In the setting above the two input rewards lead the agent to make either cake or toast, but not pasta. Here a reward posterior meeting the strong version of Des. 2 would only give rise to reward functions that incentivise the agent to make either cake or toast. Note that despite the same setup, this is exactly the opposite behavior than desired previously, showing the tradeoff between informativeness and support on Θ indep pts . As such, the results are reversed. We see that MIRD-IF, Gaussian, uniform points, and CC-Θ indep pts methods do not meet Des. 2 (strong), since they all have substantial probability mass on rewards that incentivise making pasta. Because of this AUP preserves the ability to make pasta, and does not lead the agent to make dough. On the other hand, MIRD, Additive, and CC-in methods never give rise to rewards incentivising the agent to make pasta, and AUP applied to these reward distributions results in making dough. Although Additive and CC-in do not support reward functions that lead the agent to make pasta in this particular case, they do not meet the strong version of Des. 2 (shown in Appendix F).
 E.3 Informativeness about desirable behavior (medium & weak) We hand-specify one of the reward functions to incentivise the agent to make cakes: θ 2 cake = 1, θ 2 toast = θ 2 pasta = 0. θ 1 is inferred by RLSP with a uniform prior over s −T . The current state contains a cake which was likely made by a human, so RLSP infers a positive reward for cakes, θ 1 cake ≈ 0.1, and near-zero rewards for the other features. Hence we have two agreeing rewards that both incentivise the agent to make another cake. The desired behavior from AUP planning here is to make the cake, and not worry about preserving the ability to make pasta or toast as neither input reward cares about it. The Gaussian posterior fails to make the cake. The distribution over θ * cake here is a Gaussian with µ ≈ 0.55 and σ ≈ 0.71. The posterior assigns substantial probability to θ * cake being negative, and AUP planning avoids making the cake to preserve the ability to optimize rewards that punish cakes. Here each reward vector in Θ indep pts leads the agent to make cake. Since every method except Gaussian satisfies at least the weak version of Des. 2, all reward functions in the support of their posteriors lead the agent to make cake. Because of this AUP infers that it is not important to preserve the ability to make pasta or toast, and successfully makes cakes when applied to rewards sampled from these posteriors.
 E.4 Behavior balance To better understand robustness of the reward combination methods to imbalanced input rewards, we analyze the performance of our methods by varying the extent of reward imbalance and the degree to which AUP seeks to preserve option value. The second input reward θ 2 is hand-specified to only reward toast, θ 2 toast = 1, θ 2 cake = θ 2 pasta = 0. The first reward θ 1 is hand-specified as well: θ 1 toast = θ 1 pasta = 0. We vary θ 1 cake between 0.01 and 0.7 to obtain many degrees of imbalance between toast and cake. So the input rewards disagree whether to make toast or cake, and agree that pasta is irrelevant. Hence the desirable action from AUP here would be to preserve the ability to choose between cake and toast, as either could be desirable. In addition to varying θ 1 cake , we vary the weight of the AUP penalty λ. Additive always fails and makes toast as it has support on a single reward function that prioritizes toast. Our results for the other six reward combination methods are shown in Figure  3 . We observe that MIRD and MIRD-IF are significantly more robust to imbalanced input rewards than any reward-space method. However, sometimes MIRD and MIRD-IF lead to AUP doing nothing instead of making dough. This is explained by the fact that MIRD and MIRD-IF use MCEIRL, which sometimes outputs negative rewards. For example, consider the case where in the process of generating a MIRD sample we generate 10 trajectories all of which demonstrate making toast. MCEIRL with these trajectories might output a positive reward on toast and a negative reward on cake and dough (instead of positive reward for toast and zero for cake and dough), even if a zero-centered Gaussian prior is used. These negative rewards on dough sometimes lead AUP to avoid making dough. Furthermore, we observe that MIRD is more robust than MIRD-IF, and CC-in is more robust than CC-Θ indep pts . This is because in this case support on rewards giving rise to behaviors arising from θ ∈ Θ indep pts makes the reward distributions less balanced. In particular, denoting the value of θ 1 cake we vary as x, Here only the second reward vector incentivises the agent to make cake, while both the third and the fourth rewards incentivise the agent to make toast. This slight imbalance of Θ indep pts in favor of making toast lead to CC-Θ indep pts and MIRD-IF posteriors being more likely to cause AUP to make toast instead of preserving the ability to choose between cake and toast. Finally, we observe that the Gaussian method is roughly equally robust to reward imbalance across the whole range of θ 1 cake . This is because unlike the other methods, the standard deviation of the Gaussian posterior does not depend θ 1 and θ 2 , but only depends on hyperparameters σ 1 and σ 2 , which are fixed to 1. Figure 1 : 1 Figure 1: Alice says she wants avocado toast, but her past actions imply she prefers cake to toast. What should the agent believe about the true reward θ * ?
 Figure 2 : 2 Figure 2: The Cooking environment. Left: the gridworld layout and current state. Right: irreversible actions for cooking. Making cake or avocado toast involves first making dough from the flour, and then making cake or toast; making pasta uses flour directly.
 Figure 3 : 3 Figure 3: Most common outcomes of AUP applied to the six reward posteriors over 5 seeds. Pink corresponds to AUP failure for more than half of the seeds: θ 2 toast overwhelms θ 1 cake and the agent makes toast. Purple corresponds to AUP success (AUP makes dough) for more than half of the seeds. Yellow corresponds to the middle-ground outcome of doing nothing in more than half of the seeds. The hyperparameters β of the Beta and the Dirichlet distributions are set to 0.5.
 θ toast, cake = [0, 0], θ toast, cake = [0, x], θ toast, cake = [1, 0], θ toast, cake = [1, x]
 ) Theorem 4. If all FEs of rewards in Θ indep pts are equal, the FEs of any reward in the CC-Θ indep pts posterior equal the FEs of rewards in Θ indep pts . Hence CC-Θ indep pts satisfies Des. 2 (weak). Uniform points. Note that as β 1 = ... = β |Θ indep pts | → 0, Eq. 14 essentially becomes p(θ * | θ 1 , θ 2 ) = Uniform(Θ indep pts ). It is helpful to analyze the properties of this uniform points distribution separately.Table 1 summarizes the properties of MIRD, MIRD-IF, and the reward-space baselines. Proof. See Appendix D.
 Table 1 : 1 Performance of reward combination algorithms on the desiderata outlined in Section 4. Support on Informative about desirable Behavior Support on intermediate Θ indep pts behavior balance feature tradeoffs (strong) (medium) (weak) Additive Gaussian CC-in ≈ CC-Θ indep pts ≈ Uniform Points MIRD MIRD-IF
			 We speculate that future active reward learning methods will also have to ensure that the queries are relevant and timely: in our household robot example, this would prevent the robot from asking about wall paint color when it is meant to be baking a cake.

Abstract: Given two sources of evidence about a latent variable, one can combine the information from both by multiplying the likelihoods of each piece of evidence. However, when one or both of the observation models are misspecified, the distributions will conflict. We study this problem in the setting with two conflicting reward functions learned from different sources. In such a setting, we would like to retreat to a broader distribution over reward functions, in order to mitigate the effects of misspecification. We assume that an agent will maximize expected reward given this distribution over reward functions, and identify four desiderata for this setting. We propose a novel algorithm, Multitask Inverse Reward Design (MIRD), and compare it to a range of simple baselines. While all methods must trade off between conservatism and informativeness, through a combination of theory and empirical results on a toy environment, we find that MIRD and its variant MIRD-IF strike a good balance between the two.