Joshua Greene
Francesca Rossi
John Tasioulas
Kristen Brent Venable
Brian Williams

Embedding Ethical Principles in Collective Decision Support Systems

Introduction We believe it is important to study the embedding of safety constraints, moral values, and ethical principles in agents, within the context of collective decision making systems in societies of agents and humans. Collective decision making involves a collection of agents who express their preferences over a shared set of possible outcomes, and a preference aggregation rule which chooses one of the options to best satisfy the agents' preferences. However, aggregating just preferences may lead to outcomes that do not follow any ethical principles or safety constraints. To embed such principles/constraints in a collective decision making system, we need to understand how to model them, how to reason with them at the level of a single agent, and how to embed them into collective decision making. Just like individual humans, each agent that operates in a multi-agent context needs to be have an internal representation of moral values and ethical principles, as well as an ethical reasoning engine. Otherwise it would not able to explain its behaviour to others. Copyright c 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. We claim that there is a need to adapt current logicbased modelling and reasoning frameworks, such as soft constraints, CP-nets, and constraint-based scheduling under uncertainty, to model safety constraints, moral values, and ethical principles. More precisely, we study how logicbased preference modelling frameworks can be adapted to model both (explicit) ethical principles and (implicit) moral values, as sophisticated constraints over possible actions. The constraints may be unconditional ("hard") constraints, or soft, overridable if the consequences of an individual bad action can still lead to overall good. We propose to replace preference aggregation with an appropriately developed value/ethics/preference fusion, an operation designed to ensure that agents' preferences are consistent with their moral values and do not override ethical principles For ethical principles, we use hard constraints specifying the basic ethical "laws", plus some form of common-sense morality expressed as sophisticated prioritised and possibly context-dependent constraints over possible actions, equipped with a conflict resolution engine. To avoid reckless behavior in the face of uncertainty, we proposed to bound the risk of violating these ethical laws in the form of chance constraints, and we propose to develop stochastic constraint solvers that propose solutions that respect these risk bounds, based on models of environmental uncertainty. We also propose to replace preference aggregation with an appropriately developed constraint/value/ethics/preference fusion, an operation designed to ensure that agents' preferences are consistent with the system's safety constraints, the agents' moral values, and the ethical principles. We will leverage previous experience in developing single and multi-agent preference/constraint reasoning engines. Today, techniques exist to enable agents to make decisions, such as scheduling activities, while satisfying some safety concerns, e.g. by using techniques from constraintbased optimization. For instance, in many critical scenarios, such as space missions where a malfunction can endanger the whole mission, activities are scheduled in such a way to maximise robustness against possible problems. We believe that these techniques can provide an inspiration to handle ethical concerns. However, we think that a much more explicit model and reasoning engine for ethical principles and moral values is needed in order to deal with them satisfactorily and allow them to evolve over time.
 Which ethical principles for intelligent agents? An intelligent agent should have capability to autonomously make good decisions, based on available data and preferences, even in the context of uncertainty, missing or noisy information, as well as incorrect input, and should be able to learn from past experience or from available historical data. Even more importantly, intelligent agents should have the ability to interact with humans, make decisions together with them, and achieve goals by working together. An agent with these capabilities poses several crucial ethical questions. Ethical principles guide humans' behaviour. They tell us what is regarded as right or wrong. They come from values that we regards as absolute, guiding our whole life. If we want intelligent agents to enhance human capabilities, or to collaborate with humans, or even just to live and act in the same society, we need to embed in them some ethical guidelines, so they can act in their environment following values that are aligned to the human ones. Or maybe we need different values and ethical principles for agents, since they are inherently different from humans? As Issac Asimov famously illustrated in his I, Robot series, explicitly programming ethical behavior is surprisingly challenging. Moral philosophy -the field that has studied explicit ethical principles most extensively -suggests three general approaches, corresponding to the three major schools of Western moral thought. The deontological approach (most closely associated with Immanuel Kant) regards morality as a system of rights and duties. Here the focus is on categories of actions, where different actions are deemed impermissible, permissible, or obligatory based on a set of explicit rules. The consequentialist approach (most closely associated with Jeremy Bentham and John Stuart Mill) aims to produce the best aggregate consequences (minimizing costs and maximizing benefits) according to a pre-specified value function. For example, a classical utilitarian approach aims to maximize the total amount of happiness. The virtue-or character-based approach (most closely associated with Aristotle) regards ethical behavior as the product of an acquired set of behavioral dispositions that cannot be adequately summarized as an adherence to a set of deontological rules (concerning actions) or to as a commitment to maximizing good consequences. These three approaches are well known and have been the starting point for nearly all discussions of machine ethics  (Moor 1985; Bostrom 2014; Wallach and Allen 2008) . Each approach has limitations that are well known. Deontological principles are easily to implement but may be rigid. Consequentialist principles require complex calculations that may be faulty. Virtue is opaque and requires extensive training with an unknown teaching criterion. There is, however, a more general problem faced by all three approaches, which is that implementing them may depend on solving daunting, general computation problems that have not been solved and may not be solved for some time. For example, a "simple" deontological rule such as "don't lie" or "dont kill" is not specified in terms of machine movements. Rather, the machine must understand which acts of communication would constitute lying and which body movements would constitute killing in a given context. A consequentialist system would require a machine to represent all of the actions available to it, and a virtue based system would have to recognize the present situation as one with a variety of features that, together, call for one action rather than another. In other words, all three approaches, when fully implemented, seem to require something like general intelligence, which would enable the machine to represent its current situation in rich conceptual terms. Indeed, this speculation is consistent with recent research on the cognitive neuroscience of moral judgment indicating that moral judgment depends on a variety of neural systems that are not specifically dedicated to moral judgment  (Greene 2014) . This includes systems that enable the general representation of value and the motivation of its pursuit, visual imagery, cognitive control, and the representation of complex semantic representations. Unfortunately for Commander Data, humans have no "ethical subroutine". Real human moral judgment uses the whole brain. What, then, can be done? Here, the human brain may nevertheless offer some guidance  (Shenhav and Greene 2014) . Is it morally acceptable to push someone off of a footbridge in order to save five lives  (Thomson 1985 )? A simple deontological response says no ("Dont kill"). A simple consequentialist response says yes ("Save the most lives"), and most humans are at least somewhat conflicted about this, but err on the side of the deontological response (in this particular case). We now know that the deontological response depends on a classically emotional neural structure known as the amygdala (reflecting emotional salience) and that the application of the consequentialist maximizing principle depends on a classically "cognitive" structure known as the dorsolateral prefrontal cortex. It seems that healthy humans engage both responses and that there is a higher-order evaluation process that depends on the ventromedial prefrontal cortex, a structure that across domains attaches emotional weight to decision variables. In other words, the brain seems to make both types of judgment (deontological and consequentialist) and then makes a higher order judgment about which lower-order judgment to trust, which may be viewed as a kind of wisdom (reflecting virtue or good character). Such a hierarchical decision system might be implemented within an agent, or across agents. For example, some agents may apply simple rules based on action features. Others may attempt to make "limited" cost-benefit calculations. And collectively, the behavior of these agents may be determined by a weighting of these distinct, lower-level evaluative responses. Such as system might begin by following simple deontological rules, but then, either acquire more complex rules through learning, or learn when it can and cannot trust its own cost-benefit calculations. Starting with action-based rules and simple cost-benefit calculations substantially reduces the space of possible responses. Learning to trade-off between these two approaches adds some flexibility, but without requiring intractable cost-benefit calculations or lifelong moral education. We offer this approach as just one example strategy. Of course, if we knew how we were going to solve this problem, there would be no need to bring together people with diverse expertise. What we wish to convey is twofold: First, that we are aware of the scope of the challenge and the strengths and limitations of the extant strategies. Second, that we have some preliminary ideas for hybrid approaches that leverage insights from human moral cognition. Another important aspect of our approach would be to consider the extent to which morality could be reduced to a set of rules that is capable of being applied in a fairly straightforward way to guide conduct , e.g. 'Do not kill', 'Keep one's promises', 'Help those in need', etc. We already know that much of common sense morality is codifiable in this way, thanks to the example of the law. However, even if we could achieve an adequate codification of ordinary moral consciousness, at least within some domain, problems would arise. Two cases are especially worth highlighting: (a) cases where the strict application of a given rule generates an unacceptable outcome, often but not always characterisable as such by reference to some other rule that has been violated in adhering to the first, and (b) cases where the strict application of the given set of rules is unhelpfully 'silent' on the problem at hand, because it involved circumstances not foreseen by the rules. Both phenomena (a) and (b) raise the question of when and how the strict application of a rule needs to be modified or supplemented to resolve the problem of perverse results or gaps. One important source of thinking about these issues is Aristotle's discussion of justice and equity in the Nicomachean Ethics. According to Aristotle, the common sense morality codified in law, although capable of being a generally good guide to action, will nonetheless on occasion breakdown along the lines of (a) and (b). For Aristotle, this means that the virtuous judge will need to possess, in addition to a propensity to follow legal rules, the virtue of equity. This enables the judge to use their independent judgment to correct or supplement the strict application of legal rules in cases of type (a) or (b). A key topic involves the clarification of the notion of equity, with its rule and judgment structure, as a prelude to a consideration of how this might be embedded in autonomous agents.
 Designing ethical agents No matter which approach we will choose to express ethical principles and moral values in intelligent agents, we need to find a suitable way to model it in computational terms, which is expressive enough to be able to represent all we have in mind in its full generality, and which can be reasoned upon with computational efficiency. Ethical principles may seem very similar to the concepts of constraints  (Rossi, Van Beek, and Walsh 2006; Dechter 2003 ) and preferences  (Rossi, Venable, and Walsh 2011) , which have already received a large attention in the AI literature. Indeed, constraints and preferences are a common feature of everyday decision making. They are, therefore, an essential ingredient in many reasoning tools. In an intelligent agent, we need to specify what is not allowed according to the principles, thus some form of constraints, as well as some way to prioritise among different principles, that some form of preference. Representing and reasoning about preferences is an area of increasing theoretical and practical interest in AI. Preferences and constraints occur in real-life problems in many forms. Intuitively, constraints are restrictions on the possible scenarios: for a scenario to be feasible, all constraints must be satisfied. For example, if we have an ethical rule that says we should not kill anybody, all scenarios where people are killed are not allowed. Preferences, on the other hand, express desires, satisfaction levels, rejection degrees, or costs. For example, we may prefer an action that solves reasonably well all medical issues in a patient, rather than another one that solves completely one of them but does not address the other ones. Moreover, in many real-life optimization problems, we may have both constraints and preferences. Preferences and constraints are closely related notions, since preferences can be seen as a form of "relaxed" constraints. For this reason, there are several constraint-based preference modeling frameworks in the AI literature. One of the most general of such frameworks defines a notion of soft constraints  (Meseguer, Rossi, and Schiex 2006) , which extends the classical constraint formalism to model preferences in a quantitative way, by expressing several degrees of satisfaction that can be either totally or partially ordered. The term soft constraints is used to distinguish this kind of constraints from the classical ones, that are usually called hard constraint. However, hard constraints can be seen as an instance of the concept of soft constraints where there are just two levels of satisfaction. In fact, a hard constraint can only be satisfied or violated, while a soft constraint can be satisfied at several levels.When there are both levels of satisfaction and levels of rejection, preferences are usually called bipolar, and they can be modeled by extending the soft constraint formalism  (Bistarelli et al. 2006) . Preferences can also be modeled in a qualitative (also called ordinal) way, that is, by pairwise comparisons. In this case, soft constraints (or their extensions) are not suitable. However, other AI preference formalisms are able to express preferences qualitatively, such as CP-nets  (Boutilier et al. 2004 ). More precisely, CP-nets provide an intuitive way to specify conditional preference statements that state the preferences over the instances of a certain feature, possibly depending on some other features. For example, we may say that we prefer driving slow to driving fast if we are in a country road. CP-nets and soft constraints can be combined, providing a single environment where both qualitative and quantitative preferences can be modeled and handled. Specific types of preferences come with their own reasoning methods. For example, temporal preferences are quantitative preferences that pertain to the position and duration of events in time. Soft constraints can be embedded naturally in a temporal constraint framework to handle this kind of preference. An intuitive way to express preferences consists of providing a set of goals, each of which is a propositional formula, possibly adding also extra information such as priorities or weights. Candidates in this setting are variable assignments, which may satisfy or violate each goal. A weighted goal is a propositional logic formula plus a realvalued weight. The utility of a candidate is then computed by collecting the weights of satisfied and violated goals, and then aggregating them. Often only violated goals count, and their utilities are aggregated with functions such as sum or maximin. In other cases, we may sum the weights of the satisfied goals, or we may take their maximum weight. Any restriction we may impose on the goals or the weights, and any choice of an aggregation function, give a different language. Such languages may have drastically different properties in terms of their expressivity, succinctness, and computational complexity. In the quantitative direction typical of soft constraints, there are also other frameworks to model preferences. The most widely used assumes we have some form of independence among variables, such as mutual preferential independence. Preferences can then be represented by an additive utility function in deterministic decision making, or utility independence, which assures an additive representation for general scenarios. However, this assumption often does not hold in practice since there is usually some interaction among the variables. To account for this, models based on interdependent value additivity have been defined which allows for some interaction between the variables while preserving some decomposability. This notion of independence, also called generalized additive independence (GAI), allows for the definition of utility functions which take the form of a sum of utilities over subsets of the variables. GAI decompositions can be represented by a graphical structure, called a GAI net, which models the interaction among variables, and it is similar to the dependency graph of a CP-net or to the junction graph of a Bayesian network. GAI decompositions have been used to provide CP-nets with utility functions, obtaining the so-called UCP networks.
 Preferences and ethical principles in collective decision making systems If agents and humans will be part of a hybrid collective decision making system, and thus will make collective decisions, based on their preferences over the possible outcomes, can ethical principles for such decision system be modelled just like the preferences of another dummy agent, or should they be represented and treated differently? Are the knowledge representation formalisms that are usually used in AI to model preferences suitable to model values as well, or should we use something completely different? A very simple form of values could be modelled by constraints, so that only feasible outcomes can be the results of a collective decisions process. But values and ethical principles could often take a graded form, thus resembling a kind of preference. Also, should individual and collective ethical principles be modelled differently? We believe that some of the answers to these questions may exploit the existing literature on preference aggregation  (Rossi, Venable, and Walsh 2011) . Indeed, an important aspect of reasoning about preferences is preference aggregation. In multi-agent systems, we often need to combine the preferences of several agents. More precisely, preferences are often used in collective decision making when multiple agents need to choose one out of a set of possible decisions: each agent expresses its preferences over the possible decisions, and a centralized system aggregates such preferences to determine the "winning" decision. Preferences are also the subject of study in social choice, especially in the area of elections and voting theory  (Arrow and amd K. Suzumara 2002) . In an election, the voters express their preferences over the candidates and a voting rule is used to elect the winning candidate. Economists, political theorist, mathematicians, as well as philosophers have invested considerable effort in studying this scenario and have obtained many theoretical results about the desirable properties of the voting rules that one can use. Since the voting setting is closely related to multi-agent decision making, in recent years the area of multi-agent systems has witnessed a growing interest in trying to reuse social choice results in the multi-agent setting. However, it soon became clear that an adaptation of such results is necessary, since several issues, which are typical of multi-agent settings and AI scenarios, usually do not occur, or have a smaller impact, in typical voting situations. In a multi-agent system, the set of candidates can be very large with respect to the set of voters. Usually in social choice it is the opposite: there are many voters and a small number of candidates. Also, in many AI scenarios, the candidates often have a combinatorial structure. That is, they are defined via a combination of features. Moreover, the preferences over the features are often dependent on each other. In social choice, usually the candidates are tokens with no structure. In addition, for multi-issue elections, the issues are usually independent of each other. This combinatorial structure allows for the compact modelling of the preferences over the candidates. Therefore, several formalisms have been developed in AI to model such preference orderings. In social choice, little emphasis is put on how to model preferences, since there are few candidates, so one can usually explicitly specify a linear order. In AI, a preference ordering is not necessarily linear, but it may include indifference and incomparability. Moreover, often uncertainty is present, for example in the form of missing or imprecise preferences. In social choice, usually all preferences are assumed to be present, and a preference order over all the candidates is a linear order that is explicitly given as a list of candidates. Finally, multi-agent systems must consider the computational properties of the system. In social choice this usually has not been not a crucial issue. It is therefore very interesting to study how social choice and AI can fruitfully cooperate to give innovative and improved solutions to aggregating preferences of multiple agents. In our effort, since we intend to deal with ethical issues in collective decision making, we need to understand what modifications to the usual preference aggregation scenario should be done to account for them, and how they can be handled satisfactorily when making collective decisions. Collective decision making in the presence of feasibility constraints is starting to be considered in the literature  (Grandi et al. 2014) . However, ethical principles and safety constraints will be much more complex than just a set of constraints, so we need to understand the computational and expressiveness issues arising in this scenario. 			 Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence

Abstract: The future will see autonomous machines acting in the same environment as humans, in areas as diverse as driving, assistive technology, and health care. Think of self-driving cars, companion robots, and medical diagnosis support systems. We also believe that humans and machines will often need to work together and agree on common decisions. Thus hybrid collective decision making systems will be in great need. In this scenario, both machines and collective decision making systems should follow some form of moral values and ethical principles (appropriate to where they will act but always aligned to humans'), as well as safety constraints. In fact, humans would accept and trust more machines that behave as ethically as other humans in the same environment. Also, these principles would make it easier for machines to determine their actions and explain their behavior in terms understandable by humans. Moreover, often machines and humans will need to make decisions together, either through consensus or by reaching a compromise. This would be facilitated by shared moral values and ethical principles.


Choice Set Misspecification in Reward Inference

Introduction Specifying reward functions for robots that operate in environments without a natural reward signal can be challenging, and incorrectly specified rewards can incentivise degenerate or dangerous behavior  [Leike et al., 2018; Krakovna, 2018] . A promising alternative to manually specifying reward functions is to design techniques that allow robots to infer them from observing and interacting with humans. Figure  1 : Example choice set misspecification: The human chooses a pack of peanuts at the supermarket. They only notice the expensive one because it has flashy packaging, so that's the one they buy. However, the robot incorrectly assumes that the human can see both the expensive flashy one and the cheap one with dull packaging but extra peanuts. As a result, the robot incorrectly infers that the human likes flashy packaging, paying more, and getting fewer peanuts. These techniques typically model humans as optimal or noisily optimal. Unfortunately, humans tend to deviate from optimality in systematically biased ways  [Kahneman and Tversky, 1979; Choi et al., 2014] . Recent work improves upon these models by modeling pedagogy  [Hadfield-Menell et al., 2016] , strategic behavior  [Waugh et al., 2013] , risk aversion  [Majumdar et al., 2017] , hyperbolic discounting  [Evans et al., 2015] , or indifference between similar options  [Bobu et al., 2020b] . However, given the complexity of human behavior, our human models will likely always be at least somewhat misspecified  [Steinhardt and Evans, 2017] . One way to formally characterize misspecification is as a misalignment between the real human and the robot's assumptions about the human. Recent work in this vein has examined incorrect assumptions about the human's hypothesis space of rewards  [Bobu et al., 2020a] , their dynamics model of the world  [Reddy et al., 2018] , and their level of pedagogic behavior  [Milli and Dragan, 2019] . In this work, we identify another potential source of misalignment: what if the robot is wrong about what feedback the human could have given? Consider the situation illustrated in Figure  1 , in which the robot observes the human going grocery shopping. While the grocery store contains two packages of peanuts, the human only notices the more expensive version with flashy packaging, and so buys that one. If the robot doesn't realize that the human was effectively unable to evaluate the cheaper package on its merits, it will learn that the human values flashy packaging. We formalize this in the recent framework of rewardrational implicit choice (RRiC)  [Jeon et al., 2020]  as misspecification in the human choice set, which specifies what feedback the human could have given. Our core contribution is to categorize choice set misspecification into several formally and empirically distinguishable "classes", and find that different types have significantly different effects on performance. As we might expect, misspecification is usually harmful; in the most extreme case the choice set is so misspecified that the robot believes the human feedback was the worst possible feedback for the true reward, and so updates strongly towards the opposite of the true reward. Surprisingly, we find that under other circumstances misspecification is provably neutral: it neither helps nor hurts performance in expectation. Crucially, these results suggest that not all misspecification is equivalently harmful to reward inference: we may be able to minimize negative impact by systematically erring toward particular misspecification classes defined in this work. Future work will explore this possibility.
 Reward Inference There are many ways that a human can provide feedback to a robot: demonstrations  [Ng and Russell, 2000; Abbeel and Ng, 2004; Ziebart, 2010] , comparisons  [Sadigh et al., 2017; Christiano et al., 2017] , natural language  [Goyal et al., 2019] , corrections  [Bajcsy et al., 2017] , the state of the world  [Shah et al., 2019] , proxy rewards  [Hadfield-Menell et al., 2017; Mindermann et al., 2018] , etc. Jeon et al. propose a unifying formalism for reward inference to capture all of these possible feedback modalities, called reward-rational (implicit) choice  (RRiC) . Rather than study each feedback modality separately, we study misspecification in this general framework. RRiC consists of two main components: the human's choice set, which corresponds to what the human could have done, and the grounding function, which converts choices into (distributions over) trajectories so that rewards can be computed. For example, in the case of learning from comparisons, the human chooses which out of two trajectories is better. Thus, the human's choice set is simply the set of trajectories they are comparing, and the grounding function is the identity. A more complex example is learning from the state of the world, in which the robot is deployed in an environment in which a human has already acted for T timesteps, and must infer the human's preferences from the current world state. In this case, the robot can interpret the human as choosing between different possible states. Thus, the choice set is the set of possible states that the human could reach in T timesteps, and the grounding function maps each such state to the set of trajectories that could have produced it. Let ξ denote a trajectory and Ξ denote the set of all possible trajectories. Given a choice set C for the human and grounding function ψ : C → (Ξ → [0, 1]), Jeon et al. define a procedure for reward learning. They assume that the human is Boltzmann-rational with rationality parameter β, so that the probability of choosing any particular feedback is given by: P(c | θ, C) = exp(β • E ξ∼ψ(c) [r θ (ξ)]) c ∈C exp(β • E ξ∼ψ(c ) [r θ (ξ)]) (1) From the robot's perspective, every piece of feedback c is an observation about the true reward parameterization θ * , so the robot can use Bayesian inference to infer a posterior over θ. Given a prior over reward parameters P(θ), the RRiC inference procedure is defined as: P(θ | c, C) ∝ exp(β • E ξ∼ψ(c) [r θ (ξ)] c ∈C exp(β • E ξ∼ψ(c ) [r θ (ξ)]) • P(θ) (2) Since we care about misspecification of the choice set C, we focus on learning from demonstrations, where we restrict the set of trajectories that the expert can demonstrate. This enables us to have a rich choice set, while allowing for a simple grounding function (the identity). In future work, we aim to test choice set misspecification with other feedback modalities as well.
 Choice Set Misspecification For many common forms of feedback, including demonstrations and proxy rewards, the RRiC choice set is implicit. The robot knows which element of feedback the human provided (ex. which demonstration they performed), but must assume which elements of feedback the human could have provided based on their model of the human. However, this assumption could easily be incorrect -the robot may assume that the human has capabilities that they do not, or may fail to account for cognitive biases that blind the human to particular feedback options, such as the human bias towards the most visually attention-grabbing choice in Fig 1. To model such effects, we assume that the human selects feedback c ∈ C Human according to P(c | θ, C Human ), while the robot updates their belief assuming a different choice set C Robot to get P(θ | c, C Robot ). Note that C Robot is the robot's assumption about what the human's choice set is -this is distinct from the robot's action space. When C Human = C Robot , we get choice set misspecification. It is easy to detect such misspecification when the human chooses feedback c / ∈ C R . In this case, the robot observes a choice that it believes to be impossible, which should certainly be grounds for reverting to some safe baseline policy. So, we only consider the case where the human's choice c is also present in C R (which also requires C H and C R to have at least one element in common). Within these constraints, we propose a classification of types of choice set misspecification in Table  1 . On the vertical axis, misspecification is classified according to  the location of the optimal element of feedback C R ⊂ C H C R ⊃ C H C R ∩ C H c * ∈ C R ∩ C H A1 A2 A3 ∈ C R \C H B2 B3 c * = argmax c∈C R ∪C H E ξ∼ψ(c) [r θ * (ξ))]. If c * is available to the human (in C H ), then the class code begins with A. We only consider the case where c * is also in C R : the case where it is in C H but not C R is uninteresting, as the robot would observe the "impossible" event of the human choosing c * , which immediately demonstrates misspecification at which point the robot should revert to some safe baseline policy. If c * / ∈ C H , then we must have c * ∈ C R (since it was chosen from C H ∪ C R ), and the class code begins with B. On the horizontal axis, misspecification is classified according to the relationship between C R and C H . C R may be a subset (code 1), superset (code 2), or intersecting class (code 3) of C H . For example, class A1 describes the case in which the robot's choice set is a subset of the human's (perhaps because the human is more versatile), but both choice sets contain the optimal choice (perhaps because it is obvious).
 Experiments To determine the effects of misspecification class, we artificially generated C R and C H with the properties of each particular class, simulated human feedback, ran RRiC reward inference, and then evaluated the robot's resulting belief distribution and optimal policy.
 Experimental Setup Environment To isolate the effects of misspecification and allow for computationally tractable Bayesian inference, we ran experiments in toy environments. We ran the randomized experiments in the four 20 × 20 gridworlds shown in Fig 2  .  Each square in environment x is a state s x = {lava, goal}. lava ∈ [0, 1] is a continuous feature, while goal ∈ {0, 1} is a binary feature set to 1 in the lower-right square of each grid and 0 everywhere else. The true reward function r θ * is a linear combination of these features and a constant stayalive cost incurred at each timestep, parameterized by θ = (w lava , w goal , w alive ). Each episode begins with the robot in the upper-left corner and ends once the robot reaches the goal state or episode length reaches the horizon of 35 timesteps. Robot actions A R move the robot one square in a cardinal or diagonal direction, with actions that would move the robot off of the grid causing it to remain in place. The transition function T is deterministic. Environment x defines an MDP M x = S x , A R , T, r θ * . Inference While the RRiC framework enables inference from many different types of feedback, we use demonstration feedback here because demonstrations have an implicit choice set and straightforward deterministic grounding. Only the human knows their true reward function parameterization θ * . The robot begins with a uniform prior distribution over reward parameters P(θ) in which w lava and w alive vary, but w goal always = 2.0. P(θ) contains θ * . RRiC inference proceeds as follows for each choice set tuple C R , C H and environment x. First, the simulated human selects the best demonstration from their choice set with respect to the true reward c H = argmax c∈C H E ξ∼ψ(c) [r θ * (ξ))]. Then, the simulated robot uses Eq. 2 to infer a "correct" distribution over reward parameterizations B H (θ) P(θ | c, C H ) using the true human choice set, and a "misspecified" distribution B R (θ) P(θ | c, C R ) using the misspecified human choice set. In order to evaluate the effects of each distribution on robot behavior, we define new MDPs M x H = S x , A R , T, r E[B H (θ)] and M x R = S x , A R , T, r E[B R (θ)] for each environment, solve them using value iteration, and then evaluate the rollouts of the resulting deterministic policies according to the true reward function r θ * .
 Randomized Choice Sets We ran experiments with randomized choice set selection for each misspecification class to evaluate the effects of class on entropy change and regret.
 Conditions The experimental conditions are the classes of choice set misspecification in Table 1: A1, A2, A3, B2 and B3. We tested each misspecification class on each environment, then averaged across environments to evaluate each class. For each environment x, we first generated a master set C x M of all demonstrations that are optimal w.r.  If entropy change is positive, then misspecification induces overconfidence, and if it is negative, then misspecification induces underconfidence. Regret is the difference in return between the optimal solution to M x H , with the correctly-inferred reward parameterization, and the optimal solution to M x R , with the incorrectlyinferred parameterization, averaged across all 4 environments. If ξ * x H is an optimal trajectory in M x H and ξ * x R is an optimal trajectory in M x R , then regret = 1 4 3 x=0 [r θ * (ξ * x H )− r θ * (ξ * x R )] . Note that we are measuring regret relative to the optimal action under the correctly specified belief, rather than optimal action under the true reward. As a result, it is possible for regret to be negative, e.g. if the misspecification makes the robot become more confident in the true reward than it would be under correct specification, and so execute a better policy.
 Biased Choice Sets We also ran an experiment in a fifth gridworld where we select the human choice set with a realistic human bias to illustrate how choice set misspecification may arise in practice. In this experiment the human only considers demonstrations that end at the goal state because, to humans, the word "goal" can be synonymous with "end" (Fig  3a ). However, to the robot, the goal is merely one of multiple features in the environment. The robot has no reason to privilege it over the other features, so the robot considers every demonstration that is optimal w.r.t some possible reward parameterization (Fig  3b ). The trajectory that only the robot considers is marked in blue. We ran RRiC inference using this C R , C H and evaluated the results using the same measures described above.
 Results We summarize the aggregated measures, discuss the realistic human bias result, then examine two interesting results: symmetry between classes A1 and A2 and high regret in class B3.  Regret Regret also varied as a function of misspecification class. Each class had a median regret of 0, suggesting that misspecification commonly did not induce a large enough shift in belief for the robot to learn a different optimal policy. However the mean regret, plotted as green lines in Fig  5 , did vary markedly across classes. Regret was sometimes so high in class B3 that outliers skewed the mean regret beyond of the whiskers of the boxplot. Again, classes A1 and A2 are precisely symmetric. We discuss this symmetry in Section 5.3, then discuss the poor performance of B3 in Section 5.4. Figure  6 : Human feedback and the resulting misspecified robot belief with a human goal bias. Because the feedback that the biased human provides is poor, the robot learns a very incorrect distribution over rewards.
 Effects of Biased Choice Sets The This result shows that we can see an outsized negative impact on robot reward inference with a small incorrect assumption that the human considered and rejected demonstrations that don't terminate at the goal.
 Symmetry Intuitively, misspecification should lead to worse performance in expectation. Surprisingly, when we combine misspecification classes A1 and A2, their impact on entropy change and regret is actually neutral. The key to this is their symmetry -if we switch the contents of C Robot and C Human in an instance of class A1 misspecification, we get an instance of class A2 with exactly the opposite performance characteristics. Thus, if a pair in A1 is harmful, then the analogous pair in A2 must be helpful, meaning that it is better for performance than having the correct belief about the human's choice set. We show below that this is always the case under certain symmetry conditions that apply to A1 and A2. Assume that there is a master choice set C M containing all possible elements of feedback for MDP M , and that choice sets are sampled from a symmetric distribution over pairs of subsets D : 2 C M × 2 C M → [0, 1] with D(C x , C y ) = D(C y , C x ) (where 2 C M is the set of subsets of C M ). Let ER(r θ , M ) be the expected return from maximizing the reward function r θ in M . A reward parameterization is chosen from a shared prior P(θ) and C H , C R are sampled from D. The human chooses the optimal element of feedback in their choice set c C H = argmax c∈C H E ξ∼ψ(c) [r θ * (ξ))]. Theorem 1. Let M and D be defined as above. Assume that ∀C x , C y ∼ D, we have c Cx = c Cy ; that is, the human would pick the same feedback regardless of which choice set she   sees. If the robot follows RRiC inference according to Eq. 2 and acts to maximize expected reward under the inferred belief, then: E C H ,C R ∼D Regret(C H , C R ) = 0 Proof. Define R(C x , c ) to be the return achieved when the robot follows RRiC inference with choice set C x and feedback c, then acts to maximize r E[Bx(θ)] , keeping β fixed. Since the human's choice is symmetric across D, for any C x , C y ∼ D, regret is anti-symmetric: Regret(C x , C y ) = R(C x , c Cx ) − R(C y , c Cx ) = R(C x , c Cy ) − R(C y , c Cy ) = −Regret(C y , C x ) Since D is symmetric, C x , C y is as likely as C y , C x . Combined with the anti-symmetry of regret, this implies that the expected regret must be zero: E Cx,Cy∼D [Regret(C x , C y )] = 1 2 E Cx,Cy [Regret(C x , C y )] + 1 2 E Cx,Cy [Regret(C y , C x )] = 1 2 E Cx,Cy [Regret(C x , C y )] − 1 2 E Cx,Cy [Regret(C x , C y )] = 0 An analogous proof would work for any anti-symmetric measure (including entropy change).
 Worst Case As shown in Table  4 , class B3 misspecification can induce regret an order of magnitude worse than the maximum regret induced by classes A3 and B2, which each differ from B3 along a single axis. This is because the worst case inference occurs in RRiC when the human feedback c H is the worst element of C R , and this is only possible in class B3. In class    The axes represent the weights on the lava and alive features and the space of possible parameterizations lies on the circle where w lava + w alive = 1. The opacity of the gold line is proportional to the weight that P(θ) places on each parameter combination. The true reward has w lava , w alive < 0, whereas the peak of this distribution has w lava < 0, but w alive > 0. This is because C R2 contains shorter trajectories that encounter the same amount of lava, and so the robot infers that c H must be preferred in large part due to its length.
 Discussion Summary In this work, we highlighted the problem of choice set misspecification in generalized reward inference, where a human gives feedback selected from choice set C Human but the robot assumes that the human was choosing from choice set C Robot . As expected, such misspecification on average induces suboptimal behavior resulting in regret. However, a different story emerged once we distinguished between misspecification classes. We defined five distinct classes varying along two axes: the relationship between C Human and C Robot and the location of the optimal element of feedback c * . We empirically showed that different classes lead to different types of error, with some classes leading to overconfidence, some to underconfidence, and one to particularly high regret. Surprisingly, under certain conditions the expected regret under choice set misspecification is actually 0, meaning that in expectation, misspecification does not hurt in these situations. Implications There is wide variance across the different types of choice-set misspecification: some may have particularly detrimental effects, and others may not be harmful at all. This suggests strategies for designing robot choice sets to minimize the impact of misspecification. For example, we find that regret tends to be negative (that is, misspecification is helpful) when the optimal element of feedback is in both C Robot and C Human and C Robot ⊃ C Human (class A2). Similarly, worst-case inference occurs when the optimal element of feedback is in C Robot only, and C Human contains elements that are not in C Robot (class B3). This suggests that erring on the side of specifying a large C Robot , which makes A2 more likely and B3 less, may lead to more benign misspecification. Moreover, it may be possible to design protocols for the robot to identify unrealistic choice set-feedback combinations and verify its choice set with the human, reducing the likelihood of misspecification in the first place. We plan to investigate this in future work. Limitations and future work. In this paper, we primarily sampled choice sets randomly from the master choice set of all possibly optimal demonstrations. However, this is not a realistic model. In future work, we plan to select human choice sets based on actual human biases to improve ecological validity. We also plan to test this classification and our resulting conclusions in more complex and realistic environments. Eventually, we plan to work on active learning protocols that allow the robot to identify when its choice set is misspecified and alter its beliefs accordingly. Figure 2 : 2 Figure 2: The set of four gridworlds used in randomized experiments, with the lava feature marked in red.
 Figure 3 : 3 Figure 3: Human and robot choice sets with a human goal bias. Because the human only considers trajectories that terminate at the goal, they don't consider the blue trajectory in CR.
 Figure 4 : 4 Figure 4: Entropy Change (N=24). The box is the IQR, the whiskers are the range, and the blue line is the median. There are no outliers.
 Figure 5 : 5 Figure 5: Regret (N=24). The box is the IQR, the whiskers are the most distant points within 1.5*the IQR, and the green line is the mean. Multiple outliers are omitted.
 (a) feedback cH (b) P(θ | cH , CR)
 human bias of only considering demonstrations that terminate at the goal leads to very poor inference in this environment. Because the human does not consider the blue demonstration from Fig 3b, which avoids the lava altogether, they are forced to provide the demonstration in Fig 6a, which terminates at the goal but is long and encounters lava. As a result, the robot infers the very incorrect belief distribution in Fig 6b. Not only is this distribution underconfident (entropy change = −0.614), but it also induces poor performance (regret = 0.666).
 Figure 7 : 7 Figure 7: Example human choice set and corresponding feedback.
 Fig 9a shows an example robot choice set C R3 from B3, and Fig 9b shows the inferred P(θ | c H , C R3). Note that the peak of this distribution has w lava , w alive > 0. Since c H is the longest and the highest-lava trajectory in C R3 , and alternative shorter and lower-lava trajectories exist in C R3 , the robot infers that the human is attempting to maximize both trajectory length and lava encountered: the opposite of the truth. Unsurprisingly, maximizing expected reward for this belief leads to high regret. The key difference between B2 and B3 is that c H is the lowest-reward element in C R3 , resulting in the robot updating directly away from the true reward.
 Figure 8 : 8 Figure 8: Robot choice set and resulting misspecified belief in B2.
 Figure 9 : 9 Figure 9: Robot choice set and resulting misspecified belief in B3.
 Table 1 : 1 Choice set misspecification classification, where CR is the robot's assumed choice set, CH is the human's actual choice set, and c * is the optimal element from CR ∪ CH . B1 is omitted because if CR ⊂ CH , then CR\CH is empty and cannot contain c * .
 t. at least one reward parameterization θ. For each experimental class, we randomly generated 6 valid C R , C H tuples, with C R , C H ⊆ C x M . Duplicate tuples, or tuples in which c H / ∈ C R , were not considered.
 Table 2 : 2 Entropy change is symmetric across classes A1 and A2. Class Mean Std Q1 Q3 A1 0.04 0.4906 0.1664 0.0 A2 -0.04 0.4906 0.0 -0.1664
 Table 3 : 3 Regret is symmetric across classes A1 and A2.
 Table 4 : 4 Regret comparison showing that class B3 has much higher regret than neighboring classes.

Abstract: Specifying reward functions for robots that operate in environments without a natural reward signal can be challenging, and incorrectly specified rewards can incentivise degenerate or dangerous behavior. A promising alternative to manually specifying reward functions is to enable robots to infer them from human feedback, like demonstrations or corrections. To interpret this feedback, robots treat as approximately optimal a choice the person makes from a choice set, like the set of possible trajectories they could have demonstrated or possible corrections they could have made. In this work, we introduce the idea that the choice set itself might be difficult to specify, and analyze choice set misspecification: what happens as the robot makes incorrect assumptions about the set of choices from which the human selects their feedback. We propose a classification of different kinds of choice set misspecification, and show that these different classes lead to meaningful differences in the inferred reward and resulting performance. While we would normally expect misspecification to hurt, we find that certain kinds of misspecification are neither helpful nor harmful (in expectation). However, in other situations, misspecification can be extremely harmful, leading the robot to believe the opposite of what it should believe. We hope our results will allow for better prediction and response to the effects of misspecification in real-world reward inference.


Thomas L Griffiths
Frederick Callaway
Michael B Chang
Erin Grant
Paul M Krueger
Falk Lieder
Matthew M Botvinick
Samuel J Gershman

Abstract: Artificial intelligence systems use an increasing amount of computation and data to solve very specific problems. By contrast, human minds solve a wide range of problems using a fixed amount of computation and limited experience. We identify two abilities that we see as crucial to this kind of general intelligence: meta-reasoning (deciding how to allocate computational resources) and meta-learning (modeling the learning environment to make better use of limited data). We summarize the relevant AI literature and relate the resulting ideas to recent work in psychology.