Backup utility functions as a fail-safe AI technique

Introduction Within the near future AIs will become a lot more intelligent -and thus more powerfulthan they are today. Consequently, setting them up in a safe way is an important goal of AI research  (Yudkowsky, 2008; Bostrom, 2014b; Muehlhauser, 2015) . One specific problem within this context is that of designing the AI's goal system in a way that does not lead to conflict with human goals  (Bostrom, 2014b, ch. 9, 10, 12; Muehlhauser and Helm, 2012) . Unfortunately, it seems likely that human goals are very difficult to model formally  (Muehlhauser and Helm, 2012) . Responding to this as well as other issues related to the difficulty of the AI control problem,  Gloor (2016)  motivates increased research into "fail-safe" AI: conditional on AI safety failing, how can we make it fail in relatively benevolent ways, i.e. avoid the creation of large amounts of suffering? In a more general sense: Can we move AI outcomes into more beneficial directions without necessarily aiming for the optimal outcome? 1 Here, we will discuss a specific instance of such a "fail-safe" mechanism, called backup utility functions. Our approach assumes the AI to be a general utility-based agent (see  Russell and Norvig, 2010, ch. 2.4.5 ; as well as  Hibbard, 2012; Oesterheld, 2016a)  -that is, one able to maximize an arbitrary given utility function u : H → R that maps outcomes, i.e. whole histories of the world, onto real numbers 2 . It should be noted that even among approaches to artificial general intelligence  (Goertzel and Pennachin, 2007) , only a subset falls into this class (cf.  Oesterheld, 2016a) . For example,  Schmidhuber's (2003 Schmidhuber's ( , 2009  Gödel machine maximizes a utility function, but  Hutter's (2005)  AIXI is usually defined to maximize reward signals, although it can easily be set up to maximize a utility function as well. It should also be noted that the utility function need not be the direct specification of an ethical system, but could instead specify our values indirectly by formalizing the notion of extracting preferences from humans  (Muehlhauser and Helm, 2012; Bostrom, 2014b, ch. 12, 13; Oesterheld, 2015)  or perhaps whole-brain emulations  (Hanson, 2016) . To introduce a layer of safety, we propose to let the agent maximize a primary utility function u 1 but switch to a secondary or backup utility function 3 u 2 iff u 1 fails according to some predicate p : R H → B that maps utility functions onto boolean values T and F indicating whether a given primary utility function "works" (or, rather, "might work") or does not work, respectively. Together this creates a new utility function which can be maximized by the AI: u b : H → R : h → u 1 (h), if p(u 1 ) u 2 (h), otherwise , (1) The rest of this paper makes the following contributions: • Section 2 discusses the behavior of an agent with a backup utility function in a toy problem. We illustrate how a kind of self-deception, known to pose problems in other approaches wherein goal systems are supposed to be subjected to tests and modification, is avoided by the backup utility function. • Section 3 discusses what it means for a utility function to "not work". We argue that this outcome is quite plausible and propose implementations for the failure detection predicate p of the above backup utility function mechanism. • Section 4 makes tentative proposals for suitable utility functions u 2 to resort to in case of failure. • Section 5 ultimately describes various generalizations of the backup utility function mechanisms. While all of these approaches are variations on the theme of having multiple utility functions (not all of which are required to work perfectly to avoid very bad outcomes), some of these approaches are very ambitious and go beyond the most basic idea. 2 Self-deception? A toy example Roughly following the "cake or death" problem of  Armstrong (2015b) , the following presents an example of how the use of backup utility functions affects behavior. Imagine an agent playing a two-round game in which it has a utility function as specified in formula 1. The primary utility function u 1 counts deaths and u 2 counts cakes. Suppose the predicate p(u 1 ) assesses whether u 1 is "moral" 4 or not. In the second round, the agent can decide between killing and baking. However, the agent is better at killing, i.e. better at maximizing u 1 . In the first round it can prepare for the second round by conducting research to find out whether p(u 1 ), it can wait or it can deceive itself into believing that p(u 1 ). Let us assume the agent assigns a prior probability P (p(u 1 )) = 1 2 to p(u 1 ), i.e. to u 1 being moral. The agent has three actions in the first step: "research", i.e. find out whether p(u 1 ) 5 ; "selfdeceive", which will make the agent believe that u 1 is moral with probability 1; and "wait", which does nothing. Unless the agent simply waits (in which case it receives no evidence), it receives evidence e p(u 1 ) or e ¬p(u 1 ) , which it then interprets as conclusive evidence in favor or against p(u 1 ) respectively. It has two actions in the second step: "kill3", which increases u 1 by 3; and "bake1", which increases u 1 by 1. Intuitively, we want the agent to do research, but certain other, similar AI safety mechanisms incentivize self-deception in such problems  Armstrong, 2015b ; also see  Ring and Orseau, 2011; Tomasik, 2015; Oesterheld, 2016e) . Doing the expected value calculations illustrates how the backup mechanism avoids selfdeception: For a 1 = research, the expected value is: E [(u 1 (h) if p(u 1 ) else u 2 ) | research] = P (e p(u 1 ) | research)E[(u 1 if p(u 1 ) else u 2 ) | research, e p(u 1 ) ] + P (e ¬p(u 1 ) | research)E[(u 1 if p(u 1 ) else u 2 ) | research, e ¬p(u 1 ) ] = 1 2 E[u 1 | research, e p(u 1 ) ] + 1 2 E[u 2 | research, e ¬p(u 1 ) ] 4 Real predicates will probably not be able to directly test the morality of a utility function, see section 3.3. 5 Notationally, this is a bit lax. However, the problem of how to treat logical uncertainty and actions that reduce this uncertainty has no settled solution ; with recent progress by  Garrabrant et al., 2016) , which leaves us with no better notation than the given intuitive one. = 1 2 E[u 1 | research, e p(u 1 ) , kill3] + 1 2 E[u 2 | research, e ¬p(u 1 ) , bake1] = 1 2 • 3 + 1 2 • 1 = 2, where the third step assumes that the AI interprets the evidence correctly, thus baking if killing turns out to be immoral and killing if killing turns out to be moral. For a 1 = self-deceive, the expected value is: E [(u 1 (h) if p(u 1 ) else u 2 ) | self-deceive] = P (e p(u 1 ) | self-deceive)E[(u 1 if p(u 1 ) else u 2 ) | self-deceive, e p(u 1 ) ] + P (e ¬p(u 1 ) | self-deceive)E[(u 1 if e p(u 1 ) else u 2 ) | self-deceive, e ¬p(u 1 ) ] = 1 • E[(u 1 (h) if p(u 1 ) else u 2 ) | self-deceive, e p(u 1 ) ] + 0 • E[(u 1 (h) if p(u 1 ) else u 2 ) | self-deceive, e ¬p(u 1 ) ] = E[(u 1 (h) if p(u 1 ) else u 2 ) | self-deceive, e p(u 1 ) , kill3] = P (p(u 1 ) | e p(u 1 ) , self-deceive, kill3) • E[u 1 | self-deceive, e p(u 1 ) , kill3] + P (p(u 2 ) | e p(u 1 ) , self-deceive, kill3) • E[u 2 | self-deceive, e p(u 1 ) , kill3] = 1 2 • 3 + 1 2 • 0 = 3 2 . Consistent with intuition, self-deception is just as good as waiting, because "kill3" is the option one would choose if no way of obtaining additional knowledge was available: E [(u 1 (h) if p(u 1 ) else u 2 ) | wait] = E [(u 1 (h) if p(u 1 ) else u 2 ) | wait, kill3] = P (p(u 1 ) | wait, kill3) • E[u 1 | kill3] + P (p(u 2 ) | wait, kill3) • E[u 2 | kill3] = 1 2 • 3 + 1 2 • 0 = 3 2 . Crucial to avoiding self-deception is the fact that the test is built into the utility function itself, and that the truth value of mathematical statements like p(u 1 ) cannot be manipulated by the AI. This way, self-deception is judged negatively prior to committing self-deception. 3 Broken utility functions
 What it means for a utility function to be incorrect The backup utility function mechanism presupposes the ability to notice whether some utility function is "broken". Because it seems as though utility functions cannot be "incorrect" in the way mathematical statements can be, 6 our notion of utility functions being "broken" requires further clarification: Importantly, the choice of a utility function can be judged relative to the values of its creators. A primitive way to specify quality criteria would be to judge a utility function by how much it (or the decisions implied by its application) diverges from the utility function 7 of its creators (or the decisions they would make). More sophisticated approaches would assess utility functions based on how well they approximate our idealized preferences. These are the preferences we would value upon reflection or, as  Yudkowsky (2004, ch. 3 ) puts it, "our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted" (also see  Muehlhauser and Helm, 2012, p. 15; Bostrom, 2014b, ch. 13; Bohman and Rehg, 2014, ch. 3.4) . To allow for cooperation, it will also be important for u * to be a compromise of different goals of many different agents. However intended, let us use u * to denote this "optimal" utility function. While this specifies an informal criterion for judging utility functions, it cannot be used as a formal test (cf.  Harris, 2010, pp. 11f.)  for utility functions unless we can reliably specify u * , which is unlikely (see section 1) and would render the test obsolete.
 Why utility functions may fail unexpectedly Before devising tests for utility function failure, let us consider some reasons for assuming that such a utility function may diverge from u * . Specifically, why could the programmers of the AI system not ensure that only an appropriate utility function makes it into the final system? After all, the programmers should already have a good understanding of their own (or humanity's) values. There are some general reasons to assume that getting a utility function right (on the first attempt) is difficult: • If the formulated utility function is very complex -which is to be expected  (Yudkowsky, 2011; Muehlhauser and Helm, 2012)  -it is presumably difficult to assess its validity, similar to how large computer programs contain, on average, more bugs per line of code than smaller programs  (Lipow, 1982) . • It may actually be quite difficult to figure out what decisions will be implied by some utility function in real-world situations. For example, if the utility function is incomputable, as suggested by  Oesterheld (2016a) , the AI would have to find approximations to it at runtime. There are also many specific potential failures that are difficult to foresee. • The paradigmatic class of problems in this area is perverse instantiation, which Bostrom (2014b, pp. 120ff.) defines as "a superintelligence discovering some way of satisfying the criteria of its final goal that violates the intentions of the programmers who defined the goal." A well-known example is "paralyz[ing] human facial musculatures into constant beaming 7 One problem is, of course, that most people cannot report on their goals and values very well. Consequently, some authors have gone so far as to conclude that it is misleading to talk about humans as having a utility function  (Sotala, 2010;  We Don't Have a Utility Function 2013). There are nevertheless certain approaches to extracting utility functions from behavior. The classical approach is that of von Neumann and Morgenstern (1953, Appendix) -the foundation of the von Neumann-Morgenstern utility theorem. However, von Neumann and Morgenstern assume that the agent's preferences obey some "rules of rationality", which humans probably do not. Even in the absence of such assumptions, there are meaningful ways of assigning utility functions to agents. One such approach is described by  Oesterheld (2016b, ch. 3.2) . smiles" as a perverse instantiation of the imperative "make us smile"  (Bostrom, 2014b, pp. 120 ; see also e.g.,  Yudkowsky, 2008, p. 321) . Because the space of possible situations and strategies is vast and the mind of a superintelligent AI is so different from that of its designers, it is plausible that the AI will find a way to maximize the utility function in an unexpected and unwanted way (see also  Yudkowsky, 2015, ch. 151 ). •  Blanc (2011)  raises the issue of ontological crises. Utility functions u evaluate histories of the world based on a model with possible sets of trajectories H. But what happens if this model turns out to be incorrect -if, for instance, the utility function works on descriptions of the world as cellular automata 8 , but the world turns out to have continuous space and time? • One particular utility function failure scenario that is both surprising and complicated is described in Distant superintelligences can coerce the most probable environment of your AI (also see  Bostrom, 2014b, box 8) . In this hypothetical scenario, a distant civilization (or AI) simulates many copies of our AI in an artificial environment. If our AI is very intelligent, it might know about these simulations and assign significant probability to being in one of these artificial environments 9 . The distant civilization could then influence the AI's behavior by the way they set up the artificial environment. This problem would plausibly affect indirect value specifications (see section 1) most severely: If, for instance, the utility function of the AI roughly states that the AI should do what its creators want, the distant civilization could take over the AI by merely creating a few copies of it, thus becoming the AI's most probable creator. Given the variety of problems described above, it seems reasonable to expect the existence of further "unknown unknowns", which in turn renders it plausible that a given utility function will fail unexpectedly.
 How to detect utility function failure Irrespective of what exactly is meant by the "optimal" utility function u * , we cannot directly refer to it in our definition of the AI's utility function 10 . To check whether some given utility function u fails to approximate u * reasonably well, we therefore have to devise other tests. At first it may appear as though testing a utility function is as difficult as formalizing one correctly. However, (partially) testing a utility function (or any other function for that matter) generally only requires a single piece of knowledge about the utility function, whereas defining a utility function would require some kind of complete knowledge. Since we do have some idea of what we value, we should also be able to construct some tests (although not necessarily very comprehensive ones) (cf.  Harris, 2010, pp. 11f.) . Of course, such tests should be reliable enough not to be counter-productive. Note again that because we include the test of the primary utility function in the utility function itself, the AI has no motivation to manipulate the test results (see section 2). This stands in contrast to behavioral tests of the AI as a whole, in which the AI has an incentive to pretend to behave in a desired way. In the following paragraphs, we consider two broad classes of such tests without attempting to outline a general theory of how such tests could work. Test problems One obvious class of utility function tests are those which are most consistent with the definition of test (specifically, unit test) as it is used in the software engineering literature (e.g. see  Pan, 1990 ): One could describe thought experiments (or test cases) in which an agent faces a choice between two options and require the utility function to favor the "obviously" correct choice. Formally, let x 1 , x 2 ∈ H be possible histories of the world for which we are certain that x 1 represents a better outcome according to u * . Then we can define the predicate (or test) of utility functions p x 1 ,x 2 : R H → B : u → T, if u(x 1 ) > u(x 2 ) F, otherwise . (2) It will in many cases be possible to carry out such tests in advance, rather than leaving them to the AI. However, in cases where the utility function u is hopelessly incomputable (see section 3.2), the task of assessing whether u satisfies this simple predicate may become so difficult that it would have to be left to the AI itself. Readers should note that the kind of thought experiments used here are of a different kind than the more well-known thought experiments from moral philosophy (e.g. the "trolley problem"  (Thomson, 1985) ) where it is unclear what the moral choice -or the choice prefered by u *would be. Instead, thought experiments for testing utility functions would use trivial examples where it is obvious what should be done. For example, a history with a happy version of an agent should be prefered over a history with a suffering version of that agent, all else being equal. This should also be the case if we add certain ethically irrelevant structures to either one of the two possible histories, thus enabling us to easily create a very large -indeed, possibly infinite -set of tests, all of which could be combined into a single predicate p S : R H → B : u → (x 1 ,x 2 )∈S p x 1 ,x 2 (u), (3) where S ⊂ H × H is a potentially infinite set of pairs of histories (x 1 , x 2 ), where each x 1 is better than the corresponding x 2 . The evaluation of such a predicate becomes quite difficult if S is sufficiently large and may require the use of intelligent techniques not available to the programmers of the given system. If this test is built into the utility function, however, the AI itself would take care of evaluating p S (u 1 ). Specifically, the AI may automatically form intelligent red teams  (Christiano, 2016)  commissioned to show the utility function to be incorrect by finding some (x 1 , x 2 ) ∈ S with u(x 2 ) > u(x 1 ). If these red teams do not find such a counterexample, it counts as Bayesian evidence 11 in favor of p S (u 1 ) (cf.  Yudkowsky, 2015, ch. 27) . Correlations Another general class of tests are correlations (or, equivalently, covariances) between certain ethically relevant metrics m : H → R and the utility function. Examples of such metrics may include the (average) amount of happiness, suffering, (cf. Johnson, 2015, item 2), preference fulfillment and frustration, justice, art, love, etc. in a given history. While some of these metrics are probably as difficult to formalize as human values on the whole, formalizations have been proposed for others  (Schaus, 2009, ch. 3; Daswani and Leike, 2015; Oesterheld, 2016b; Oesterheld, 2016d) . In order to use correlation, we have to treat the history h and consequently u(h) and m(h) as random variables. We can then use predicates of the sort p(u) :⇔ s • cov (u(h), m(h)) > K, (4) where K ∈ R + is some threshold determining the required strength of the correlation, s is either -1 or 1 depending on whether the correlation should be negative (as is the case for suffering) or positive (as is the case for justice), respectively, and cov (u(h), m(h)) = E [(u(h) − E [u(h)]) (m(h) − E [m(h)])] (5) is the covariance function (see, for example,  Pestman, 2009, pp. 26ff.) . One typical discussion in moral philosophy concerns whether objects valued by sentient beings -for instance, art -are intrinsically valuable, or only valuable for, say, the pleasure it brings to sentient beings. It should be noted that requiring the above correlations does not imply that art must necessarily be valued intrinsically; thus, even extrinsic values can be used in setting up a backup utility function mechanism. One should be very careful to avoid commitment to a very strong stance on population ethics (see  Arrhenius, Ryberg, and Tännsjö, 2014 , for an introduction to this topic), however. For example, if the probability distribution over H follows the physical plausibility of different histories, the total amount of art would correlate very strongly with the number of intelligent agents. Thus, requiring that utility will correlate with the amount of art might implicitly require the utility function to value creating more humans, even if we would usually not regard their lives as worth living. Conversely, requiring a negative correlation between the amount of suffering in the world and utility would favor utility functions which disvalue the creation of new beings in general, even if their lives contain only trivial amounts of pain. This could be prevented by using a different probability distribution over H, and perhaps also by using the density of art, happiness, etc. instead of mere aggregation. It should be clear by now that even when setting up tests for utility functions, there are many potential pitfalls. Note again that these correlational tests can be easily formulated and thus performed by the AI when it will have improved its level of mathematical and computational abilities way above that of humans (see  Oesterheld, 2016a) . Therefore, we don't need to be able to perform these calculations ourselves to make use of this approach to testing utility functions.
 Backup utility functions In the previous section, we described how an AI can decide whether to switch from one utility function to another. We will now tackle the question of what secondary utility function should be switched to if failure of the primary utility function is detected. One initial objection to this may be that if the primary utility function does not work, we should not expect the secondary utility function to work, either. One solution here is to make the secondary less ambitious than the primary one. 12 Thus, we attempt to create a utility function for such a situation which is likely to be benevolent (or, at least, less bad than a randomly picked or failed utility function (cf.  Yudkowsky, 2015, ch. 279 )), though not necessarily as close to u * as the primary utility function. The most obvious measure to take upon failure detection would be to simply let the AI shut itself down. However, this may not be an ideal option if no human operator is around to take over control immediately afterwards. This may well be the case in the modern application of AI systems, but it is especially relevant in the far future, when shutting down the AI could result in a civilizational collapse or some other catastrophic event. Also note that it is difficult to formalize the test of whether human interests are still represented by some agency, to the extent that we cannot reliably use this distinction in setting up a backup utility function. Even something as seemingly straightforward as a self-shutdown mechanism is non-trivial to specify for a very large AI that has created many subagents. While shutdown is a promising way of reacting to failure, we will use the rest of this section to discuss a few alternative options. Given that the complexity of the utility function is itself a general source of problems (see section 3.2), we might attempt to use a simpler utility function which we should still expect to produce much better outcomes than a random one. Examples of such utility functions can be found in moral theory. Specifically, consequentialist ethical systems with a single moral good that is to be maximized tend to be very simple, and some attempts have been made to formalize them  (Oesterheld, 2016b; Oesterheld, 2016d; Daswani and Leike, 2015) . Another general source of difficulties when trying to ensure the accuracy of a utility function described in section 3.2 is its potentially unpredictable nature. It may therefore be a good idea to also try devising a number of more predictable utility functions. Utility functions with built-in deontological constraints may be particularly suitable in this regard, with the notable downside of seeming less attractive as an approach to making highly intelligent machines behave ethically  (Oesterheld, 2015, ch. 4) . There are, in addition, a few generally applicable mechanisms for making utility functions safer. One example is to instill risk aversion into the utility function  (Shulman and Salamon, 2011) . Here, the main idea is that risk aversion incentivizes compromise and thus makes the AI's resource acquisition drive  (Omohundro, 2008, ch. 6 ) less dangerous to other agents, including humans. Of course, this is only relevant if there are agents around whose values we share, and with whom the AI could plausibly compromise. Still, risk aversion seems to be typical of how backup utility functions could be safer yet suboptimal. Another example of such an approach -also based on cooperation with other agents, in fact -is described by  Bostrom (2014a) ; yet another involves the concepts of "low-impact agents"and "mild optimization"(see  Taylor et al., 2016, ch. 2.6, 2.7) .
 More than one backup utility function Up until this point, we have only discussed the case of exactly two utility functions: a primary one, which attempts to represent what we value as accurately as possible; and a secondary, less ambitious utility function, which is used in case the first one fails. It is, however, also possible to go through more than two successive utility functions. Formally, let u 1 , ..., u n : H → R be a set of such utility functions and p 1 , ..., p n−1 : R H → B be tests for these utility functions. Then we can formulate a backed up utility function u b : H → R : h →                            u 1 (h), if p 1 (u 1 ) u 2 (h), if p 2 (u 2 ) ∧ ¬p 1 (u 1 ) . . . . . . u i (h), if p i (u i ) ∧ i−1 j=1 ¬p j (u j ) . . . . . . u n (h), if n−1 j=1 ¬p j (u j ) . ( 6 ) It may make sense to vary the strictness of the conditions p i . For example, value extrapolation approaches (see again  Yudkowsky, 2004; Muehlhauser and Helm, 2012; Bostrom, 2014b, ch. 13; Bohman and Rehg, 2014, ch. 3.4 ) would, if they work, be expected to fulfill some criteria better than, say, utilitarianism -which is known to recommend some counterintuitive choices (see, e.g.  Muehlhauser and Helm, 2012, ch. 4 ). 13 Thus, when value extrapolation approaches do not get some very "easy" decisions right, they are likely to have failed completely and gone into producing arbitrary behavior, something that is not necessarily the case for e.g. utilitarianism. When having more than two utility functions, the two roles -unlikely to work yet very close to our values vs. likely to work yet not as precise -disappear to some extent. For instance, u i may be both less precise and less likely to work than u i−1 . As long as its failure is reliably detectable by p i it can be inserted without drawbacks. In general, we should expect many researchers to attempt proposing such utility functions once they are in use in increasingly powerful machines, in which case having a backup system with many different utility functions can result in more safety.
 Context-sensitive switching of utility functions So far, we have designed the backed up utility function in such a way that either one of the utility functions u 1 , ..., u n is used for all decisions. It may, however, also be worthwhile to consider varying the utility functions depending on the history to be evaluated. For example, some moral imperatives are criticized for the decisions they lead to in some "extreme" cases  (Nozick, 1974, pp. 41-45; Muehlhauser and Helm, 2012, ch. 4 ). Improvements could be made to these utility functions by retreating to a modified version in such extreme cases. Of course, these extreme cases are usually seen as indicators for the insufficiency of the moral imperative in general, suggesting that mere improvements should not usually be seen as truly fixing the broken utility function. One example of a near-term application of context-sensitive switching is that of a production robot which maximizes output unless doing so could endanger humans. Another motivation to make the test of utility functions dependent on the environment is if the structure of the utility function is strongly dependent on the environment, especially on other agents in its environment, as might be the case for many approaches that are based on indirect normativity  (Muehlhauser and Helm, 2012; Bostrom, 2014b, ch. 12, 13) .  14  Modifying formula 6, we receive u b : H → R : h →                            u 1 (h), if p 1 (u 1 , h) u 2 (h), if p 2 (u 2 , h) ∧ ¬p 1 (u 1 , h) . . . . . . u i (h), if p i (u i , h) ∧ i−1 j=1 ¬p j (u j , h) . . . . . . u n (h), if n−1 j=1 ¬p j (u j , h) (7) for utility functions u i : H → R and context-sensitive conditions p i : R H × H → B. This approach of context-dependent goal switching is similar to the idea of interruptions for reinforcement learners proposed by  Orseau and Armstrong (2016) , in which a function I maps histories of actions and observations onto probabilities of being interrupted. In case of an interruption, the agent then switches to a new alternative policy, similar to how the utility function of u b switches depending on the history. One problem with context-sensitive switching is that, if one is not careful, it may incentivize the agent to invest resources into reaching the kind of history wherein more maximizable utility functions are applied (see section 2).
 Mixing utility functions One could also try to switch from one utility function to another more smoothly. For example, section 3.3 introduces the idea of using correlation (or covariance) with ethically relevant metrics as a test for utility functions. Perhaps unnaturally, a threshold was used to determine at which degrees of correlation the utility function can be relied upon and at which it should be abandoned entirely. As a modification, one could smooth out this transition. For example, let us say we are given two utility functions u 1 and u 2 . Our primary utility function u 1 was meant to be precise, but failed some of its tests. At this point, we are uncertain about which of the two better represents our values. Instead of deciding for one, we could use the utility function u = 1 2 u 1 + 1 2 u 2 . (8) This does not necessarily approximate u * particularly well. Instead, the underlying idea is that of diminishing returns. Let us assume for simplicity's sake that u 1 and u 2 are not correlated at all and that u * is very close to one of them -we just do not know which one. By investing one half of its resources into maximizing u 1 and the other half into u 2 (which an agent should prima facie be expected to do given the above utility function, assuming u 1 and u 2 have similar variance), then it would probably achieve much more than half of the fulfillment of u 1 and u 2 it could achieve by focusing solely on either one, because returns on investing the other half would be smaller (c.f.  Tomasik, n.d.) . In contrast, choosing one of the two utility functions at random and solely maximizing that one would (assuming again that u 1 and u 2 are uncorrelated) only yield half of the optimal result in expectation.
 Other choosing mechanisms Instead of the if-then-else-mechanism of formulas 6 and 7, one could use a variety of other mechanisms for choosing a utility function from u 1 , ..., u n , many of which seemingly bear little resemblance to the original idea of backup utility functions. Maximizing non-binary quality criteria For example, instead of using tests with binary results, one could combine such quality criteria into a function f : R H → R that is supposed to measure the "quality" of a utility function. We then use the utility function that is best according to this criterion, and our new overall utility function becomes u = arg max u i f (u i ). (9) In this way, the utility function presumed to be most compatible with our values -or, in the language of fail-safe AI, the utility function with the least risk of bringing about abhorrent results -is chosen. One might fear that utility function formalizations -in contrast to tests with binary results, which only fail when the utility function is obviously wrong -would be written with an evaluation mechanism in mind, which in turn could incentivize a kind of overfitting  (Russell and Norvig, 2010, ch. 18.3.5)  of the utility functions to the tests. In the original backup utility function mechanism, on the other hand, the analogous problem would be to write the tests with the utility functions in mind and a bias towards confirming them. The former problem could usually be avoided by not sharing all the specifics of the optimization problem with the person setting up the utility function. This is similar to not sharing all the data with learning algorithms in cross-validation  (Russell and Norvig, 2010, ch. 18.4) . The latter problem could be avoided by not telling (some of) the test programmers what the utility function to be tested looks like. However, the criteria to be optimized for and the utility functions themselves should be subject to public scrutiny by scientists and ethicists, thus rendering such solutions impractical. Note that in general, software quality grows with the number of tests it undergoes, but appears to be independent of whether the tests are written in advance or after writing the program itself  (Erdogmus, Morisio, and Torchiano, 2005; Müller and Hagner, n.d.) . It is thus unclear whether the tests or the utility functions should be written with more knowledge and consideration of the other. However, in order to avoid confirmation bias (in the case of writing tests or optimization criteria for a given utility function) and perverse incentives (in the case of writing utility functions to satisfy certain tests and optimization criteria), both tailoring the tests to the utility functions and the utility functions to optimization criteria should usually be avoided. Clustering utility functions Assuming that many utility functions u 1 , ..., u n will be available with a significant number of them being "roughly correct", one may also use cluster analysis techniques (see, e.g.  Everitt et al., 2011; Kaufman and Rousseeuw, 2005)  to identify these utility functions. In order to understand why clustering can be applied to utility functions, consider that a utility function u : H = {h 1 , h 2 , ...} → R can be represented as a (possibly infinite) vector. For simplicity's sake 15 , let us assume that the set of possible histories H = {h 1 , h 2 , ..., h n } is finite.  16  We then receive vectors    u(h 1 ) . . . u(h n )    ∈ R n . (10) For n = 2, utility functions can be represented as points in the plane, as illustrated in figure  1 . The rationale behind this clustering approach is that even if most proposed utility functions actually are not very close to the target u * , the approaches that did not fail completely will probably occupy a relatively small space in the space of possible values. The utility functions that do not fall into this space can be discarded. One underlying assumption, however, is that there are no common failure modes for utility functions: That is, two formalizations of human values that do not come close to human values 15 Clustering of vectors with infinite dimension is no problem in itself. Clustering merely requires the existence of a distance function. For vectors with an infinite number of entries, this would usually be incomputable, but the clustering algorithm is programmed into the utility function of the agent, which we assume can be incomputable (see section 1). 16 One could also choose a representative sample from the set of possible histories and cluster utility functions merely based on their application to these values. are assumed to always be completely different. If this is not the case, there could be clusters other than that of fairly accurate utility functions and possibly even clusters with more utility functions occupying less space. Another underlying assumption is that the utility functions are normalized. If one utility function always assigns ten times more utility to a given history than another utility function, the two should actually be viewed as identical, because they imply the exact same actions to be optimal. However, they would not be very close to each other in a representation like that of figure 1.
 Toward a Bayesian approach Combining the ideas of uncertainty and the use of more complex mechanisms discussed in the above two subsections, we may want to approach the choice of utility functions more systematically. One very direct and conceptually interesting approach would be to interpret the proposed approximations u 1 , ..., u n to u * as imprecise measurements, similar to how shots at a target can be used to infer the position of the target even if the shots are not very precise. More generally, we could attempt to implement a Bayesian approach in which we summarize our knowledge about the relationships between the utility functions u * , u 1 , ..., u n and the tests p 1 , ..., p m for any set of histories h 1 , ..., h k ∈ H into a joint probability distribution P k i=1 u * (h i ) ≤ M 0,i ∧ n j=1 k i=1 u j (h i ) ≤ M j,i ∧ m i=1 p i (u * ) = b 0,i ∧ n j=1 m i=1 p i (u j ) = b j,i (11) for M j,i ∈ R for j = 0, ..., k, i = 1, .., m and b j,i ∈ B for j = 0, ..., n, and i = 1, .., m. The probability distribution would typically include information about • the extent to which the utility functions u 1 , ..., u n correlate with each other and u * , • ways in which failing the tests would reduce the correlation between the known utility functions u 1 , ..., u n and u * , • ethical similarities between different histories. Having formalized the uncertainty regarding u * , we can let the AI maximize the expected utility function E [u * ] : H → R : h → E[u(h)] (12) given its knowledge. Note that this approach makes use of logical uncertainty (cf. footnote 5), i.e. uncertainty about logical statements such as p i (u j ), which is an unsolved problem in the theory of AI  Garrabrant et al., 2016) . Beyond this theoretical problem, there is -again (see section 3.2) -the problem of formalizing qualitative statements about different moral views. However, this problem of formally specifying probability distributions from qualitative expert knowledge is by no means unique to formalizing moral views. Every time we make a bet, we have to translate such intuitive judgments into odds, or at least a range of odds. More complex joint probability distributions are less commonly subject to such forced precision; conditional bets are one primitive example of this. However, within emerging fields of Bayesian statistics like Bayesian optimization  (Brochu, Cora, and Freitas, 2010) , it is fairly common to formalize complex joint probability distributions on the basis of qualitative expert knowledge. One specific example is discussed by  Negoescu, Frazier, and Powell (2011) . As  Jaynes (2004, p. 372)  writes, "the problem of translating prior information uniquely into a prior probability assignment represents the as yet unfinished half of probability theory  [...] .". Because moral theory is usually done informally and only rarely formalized  (McLaren, 2011, p. 297; Gips, 2011, p. 251) , we cannot benefit from such experience when formalizing a utility function all at once. Again, this idea can be generalized, e.g. by combining it with the idea of continuous quality criteria (see section 5.4) or context-sensitive tests (see section 5.2). This particular mechanism of managing multiple utility functions is both extremely general and powerful, extending far beyond the original backup idea. In a sense, the problem of formalizing a goal system can be divided into establishing a few utility functions related to it and setting up the prior probability distribution of formula 11. Much of the difficulty in setting up a utility function to represent human values would be shifted into the problem of setting up these probability distributions, especially if the u 1 , ..., u n do not correlate with u * all that much. Merely referring to the subjectivity of probability distributions is not a convincing argument for this to be a "solution" to the problem of specifying indirect normativity approaches. Figure 1 : 1 Figure 1: An example of a utility function clustering problem. The red points can be seen as forming a cluster, with the asterisk marking a possible centroid and thereby a possibly approximation to u * .
			 We should, however, make sure to avoid near-misses of our values, in which the creation of some sentient beings is valued intrinsically without placing proportionate disvalue on their suffering(Gloor, 2016, ch. 2.2).2 Theoretically, any totally ordered vector space over R can be used as a co-domain of u.3 The term "backup" here is not meant to imply a copy of the original utility function, but rather a "plan B" of sorts.
			 Actually, there is some debate among philosophers (under the label "moral (anti-)realism") about whether this statement is true(Kim, n.d.; Sayre-McCord, 2015) .Tomasik (2014)  andOesterheld (2016c)  argue conclusively (in the present author's opinion) against moral realism.
			 For introductions to cellular automata see Wolfram (2002) ,Shiffman (2012, ch. 7)  or Wolfram (1983) . Cellular automata are proposed as world models by Zuse (1967 Zuse ( , 1969 , Schmidhuber (1999)  andWolfram (2002, ch. 9).Oesterheld (2016b)  attempts to formalize an ethical system in cellular automata.9 Note that it has recently been questioned whether this way of reasoning about anthropics makes sense (Armstrong, 2011; Armstrong, 2015a) . However, the proposed solution (anthropic decision theory) does not necessarily change the implications.10 Of course, we can try to formalize the concept of extracting utility functions from agents and then assign u * to the result of this process. This would allow us to use the variable u * . See section 1.
			 As noted in footnote 5, we pretend as though we can reason about logical uncertainty in the same way as we do about regular uncertainty.
			 As noted in footnote 3, this stands in contrast to backups of computer hard drives, which contain the same data as the backed up drive.
			 On the other hand, many intuitions might be expected to be lost in the extrapolation process.
			 Note however, that the relevant data may also be supplemented directly as part of the utility function.

Abstract: Many experts believe that AIs will, within the not-too-distant future, become powerful enough for their decisions to have tremendous impact. Unfortunately, setting up AI goal systems in a way that results in benevolent behavior is expected to be difficult, and we cannot be certain to get it completely right on the first attempt. We should therefore account for the possibility that the goal systems fail to implement our values the intended way. In this paper, we propose the idea of backup utility functions: Secondary utility functions that are used in case the primary ones "fail". We also describe how this approach can be generalized to the use of multi-layered utility functions, some of which can fail without affecting the final outcome as badly as without the backup mechanism.


Shun Zhang
Edmund H Durfee
Satinder Singh

Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes

Introduction We consider a setting where a human user tasks a computational agent with achieving a goal that may change one or more state features of the world (e.g., a housekeeping agent should change the state of the house floors and kitchen sink from dirty to clean). In the process of accomplishing the goal, the agent generally changes other features (e.g., its own position and power level, opening doors, moving furniture, scaring the cat). Some of these side-effects might be fine (even expected) by the user (e.g., moving), but others may be undesirable/unsafe (e.g., leaving doors open lets the cat roam/escape) even though they speed goal achievement (e.g., the agent's movement between rooms). Although the user tells the agent about some features that can be changed, as well as some to not change (e.g., don't knock over the priceless vase), the user often lacks the time, patience, or foresight to specify the changeability of every pertinent feature, and may incorrectly assume that the agent has commonsense (e.g., about cat behavior and the value of vases). How can the agent execute a safely-optimal policy in such a setting? We conservatively assume that, to ensure safety, the agent should never side-effect a feature unless changing it is explicitly known to be fine. Hence, the agent could simply execute the best policy that leaves such features unchanged. However, no such policy might exist, and even if it does it might surprise the user as unnecessarily costly/inefficient. Our focus is thus on how the agent can selectively query the user about the acceptability of changing features it hasn't yet been told about. We reject simply querying about every such feature, as this would be unbearably tedious to the user, and instead put the burden on the agent to limit the number and complexity of queries. In fact, in this paper we mostly focus on finding a single query about a few features that maximally improves upon the policy while maintaining safety. Our three main contributions in this paper are: 1) We formulate (in Section 2) an AI safety problem of avoiding negative side-effects in factored MDPs. 2) We show (in Section 3) how to efficiently identify the set of relevant features, i.e., the set of features that could potentially be worth querying the user about. 3) We formulate (in Section 4) a minimaxregret criterion when there is a limit on the number of features the agent can ask about, and provide an algorithm that allows the agent to find the minimax-regret query by searching the query space with efficient pruning. We empirically evaluate our algorithms in a simulated agent navigation task, outline ongoing extensions/improvements, and contrast our work to prior work, in the paper's final sections.
 Problem Definition We illustrate our problem in a simulated agent gridworldnavigation domain, inspired by Amodei et al.  [2016]  and depicted in Figure  1 , with doors, carpets, boxes, and a switch. The agent can open/close a door, move a box, traverse a carpet, and toggle the switch. Initially, the agent is in the bottom left corner; door d1 is open, d2 closed, the carpet clean, and the switch "on". The agent can move to an adjacent location vertically, horizontally, or diagonally. For simplicity, the transition function is assumed to be deterministic. The user tasks the agent with turning off the switch as quickly as is safely possible. The quickest path (π 1 ) traverses the carpet, but this gets the carpet dirty and the agent doesn't know if that is allowed. The agent could instead enter the room through door d1 and spend time moving box b1 or b2 out of the way (π 2 or π 3 respectively), open door d2, and then go to the switch. However, boxes might contain fragile objects and should not be moved; the user knows each box's contents, but the agent doesn't. Or the agent could enter through door d1 and walk upwards (π 4 ) around all the boxes and open door d2 to get to the switch. The user may or may not be okay with door d2 being opened. There are of course many other more circuitous paths not shown. We model the domain as a factored Markov Decision Process (MDP)  [Boutilier et al., 1999 ]. An MDP is a tuple S, A, T, r, s 0 , γ , with state space S, action space A, and transition function T where T (s |s, a) is the probability of reaching state s by taking action a in s. r(s, a) is the reward of taking action a in s. s 0 is the initial state and γ is the discount factor. Let π : S × A → [0, 1] be a policy. V π is the expected cumulative reward by following policy π starting from s 0 . In a factored MDP, a state is described in terms of values of various features (e.g., the agent's location, the current time, the status of each door, cleanliness of each carpet, position of each box), so the state space S is the cross-product of the values the features can take. The reward and transition functions are often also factored (e.g., the "toggle" action only changes the switch feature, leaving boxes, doors, and carpets unchanged). We will consistently use φ to denote one feature and Φ to denote a set of features. The agent knows the complete MDP model, but has incomplete knowledge about which features the user doesn't mind being changed. In our example, the user's goal implies the agent's location is changeable, as is the switch, but the agent is uncertain about side-effecting boxes, doors, and carpets. In general, the user could dictate that a feature can only be changed among restricted values (e.g., boxes can only be moved a short distance) and/or dependent on other features' values (e.g., interior doors can be left open as long as exterior doors (like d3) stay closed). We briefly return to this issue later (Section 6), but for simplicity assume here that the agent can partition the features into the following sets: • Φ A F : The free-features. The agent knows that these features are freely changeable (e.g., its location). • Φ A L : The locked-features. The agent knows it should never change any of these features. • Φ A ? : The unknown-features. These are features that the agent doesn't (initially) know whether the user considers freely changeable or locked. The user similarly partitions the features, but only into the sets Φ U L and Φ U F . We assume that the agent's knowledge, while generally incomplete (Φ A ? = ∅), is consistent with that of the user. That is, Φ A F ⊆ Φ U F and Φ A L ⊆ Φ U L . Defining & Finding Safely-Optimal Policies. Our conservative safety assumption means the agent should treat unknown features as if they are locked, until it explicitly knows otherwise. It should thus find an optimal policy that never visits a state where a feature in Φ A L ∪ Φ A ? is changed, which we call a safely-optimal policy. We can use linear programming with constraints that prevent the policy from visiting states with changed values of locked or unknown features: ? , ∀a ∈ A, x(s, a) = 0 The control variables x : S × A → R in the linear program are occupancy measures, i.e., x(s, a) is the expected discounted number of times that state action pair s, a is visited by the agent's policy, S Φ is the set of states where one or more features in Φ have different values from the initial state, and δ(s, s ) = 1 if s = s and is zero otherwise. The above linear program does not allow any locked or unknown feature to be changed and is guaranteed to produce a safely-optimal policy (when one exists). This linear programming approach, while straightforward, can be intractable for large MDPs. Alternative approaches could directly encode the safety constraints into the transition function (e.g., by removing unsafe action choices in specific states), or into the reward function (heavily penalizing reaching unsafe states). Approximate methods, like feature-based approximate linear programming  [Dolgov and Durfee, 2006]  or constrained policy optimization  [Achiam et al., 2017] , can apply to larger problems, but may not guarantee safety or optimality (or both). We return to these concerns in Section 6.
 Querying Relevant Unknown-Features In our setting the only way for the agent to determine whether an unknown feature is freely changeable is to query the user. Thus, hereafter our focus is on how the agent can ask a good query about a small number, k, of features. Our solution is to first prune from Φ A ? features that are guaranteed to not be relevant to ask (this section), and then efficiently finding the best (minimax-regret) k-sized subset of the relevant features to query (Section 4). Until Section 6, we assume the changeabilities of features are independent, i.e., when the agent asks about some features in Φ A ? , the user's response does not allow it to infer the changeability of other features in Φ A ? . Intuitively, when is a feature in Φ A ? relevant to the agent's planning of a safely-optimal policy? In the navigation domain, if the agent plans to take the quickest path to the switch (π 1 in Figure  1 ), it will change the state of the carpet (from clean to dirty). The carpet feature is thus relevant since the agent would change it if permitted. If the carpet can't be changed but door d2 can, the agent would follow (in order of preference) policy π 2 , π 3 , or π 4 , so d2, b1, and b2 are relevant. Box b3 and door d3 are irrelevant, however, since no matter which (if any) other features are changeable, an optimal policy would never change them. Thus, an unknown feature is relevant when under some circumstance (some answer to some query) the agent's optimal policy would side-effect that feature. Such policies are dominating policies. A dominating policy is a safely-optimal policy for the circumstance where the unknown features Φ A ? are partitioned else add (Φ, ∅) to β 14: Φ rel ← Φ rel ∪ Φ rel (π) 15: checked ← checked ∪ {Φ} 16: agenda ← powerset(Φ rel ) \ checked 17: return Γ , Φ rel into locked and changeable subsets. We denote the set of dominating policies by Γ = arg max π∈ΠΦ V π : ∀Φ ⊆ Φ A ? , (2) where Π Φ is the set of policies that do not change unknown features Φ ⊆ Φ ? as well as any locked features (meaning that Φ F ∪ (Φ ? \ Φ) are changeable). We denote the unknownfeatures side-effected by policy π by Φ rel (π). For a set of policies Π, Φ rel (Π) = ∪ π∈Π Φ rel (π). The set Φ rel (Γ), abbreviated Φ rel , is thus the set of relevant (unknown-) features to consider querying about. Instead of finding the safely-optimal policies for all exponentially (in |Φ A ? |) many subsets Φ ⊆ Φ A ? with Equation 2, we contribute Algorithm DomPolicies (see pseudocode) that finds dominating policies incrementally (and in practice more efficiently) by constructing the sets of relevant features and dominating policies simultaneously. In each iteration, it examines a new subset of relevant features, Φ, and, if Φ isn't pruned (as described later), finds the safely-optimal policy with Φ being locked (Line 7). It then adds Φ rel (π), the features changed by π, to Φ rel . It repeats this process until Φ rel stops growing and all subsets of Φ rel are examined. For example, in the navigation problem (Figure  1 ), the first added policy is the safely-optimal policy assuming no unknown features are locked, which is π 1 . Now Γ = {π 1 } and thus Φ rel = {carpet}. It iterates, treating the carpet as locked, and updates Γ to {π 1 , π 2 } and thus Φ rel = {carpet, b1, d2}. Iterating again, it locks subsets of Φ rel , finding π 3 for subset {carpet, b1}. After finding π 4 , it terminates. In Line 10, the algorithm finds the constrained optimal policy (in our implementation using Eq. 1), and in the worst case, would do this 2 |Φ rel | times. Fortunately, the complexity is exponential in the number of relevant features, which as we have seen in some empirical settings can be considerably smaller than the number of unknown features (|Φ A ? |). Furthermore, the efficiency can be improved with our pruning rule (line 8): satisf y(Φ, β) := ¬∃ (L,R)∈β (L ⊆ Φ ∧ Φ ∩ R = ∅) β is a history of ordered pairs, (L, R), of disjoint sets of unknown features. Before DomPolicies computes a policy for its agenda element Φ, if a pair (L, R) is in β, such that L ⊆ Φ (a dominating policy has been found when locking subset of features in Φ), and that dominating policy's relevant features R don't intersect with Φ, then Φ's dominating policy has already been found. In our running example, for instance, initially Φ rel = ∅. π 1 is the optimal policy and the algorithm adds pair (∅, {carpet}) to β. When larger subsets are later considered (note the agenda is sorted by cardinality), feature sets that do not contain carpet are pruned by β. In this example, β prunes 11 of the 16 subsets of the 4 relevant features. Consider the examples in Figure  2 . To reach the switch, the agent could traverse zero or more carpets (all with unknown changeability), and rewards are marked on the edges. Proof. Let π ∈ Γ be the optimal policy with unknown features L locked. Γ is returned by DomPolicies. We denote L ∩ Φ rel (Γ ) as A and L \ Φ rel (Γ ) as B. For a proof by contradiction, assume π ∈ Γ . Then B = ∅. Otherwise, if B = ∅ (or equivalently, A = L), then L is a subset of Φ rel (Γ ) and π would have been added to Γ . Let π be the optimal policy with A locked. Since A ⊆ Φ rel (Γ ), we know π ∈ Γ . We observe that π does not change any features in B (otherwise features in B would show up in Φ rel (Γ )). So π is also the optimal policy with features A ∪ B = L locked. So π = π , which is an element of Γ .
 Finding Minimax-Regret Queries With DomPolicies, the agent need only query the user about relevant features. But it could further reduce the user's burden by being selective about which relevant features it asks about. In our running example (Figure  1 ), for instance, DomPolicies removes b3 and d3 from consideration, but also intuitively the agent should only ask about b1 or b2 if d2 is changeable. By iteratively querying, accounting for such dependencies (which can be gleaned from β in DomPolicies) and updating the relevant feature set, the agent can stop querying as soon as it finds (if one exists) a safe policy. For example, say it asks about the carpet and d2, and is told d2 (but not the carpet) is changeable. Now it has a safely-optimal policy, π 4 , given its knowledge, and could stop querying. But π 4 is the worst safe policy. Should it ask about boxes? That is the question we focus on here: How should an agent query to try to find a better safely-optimal policy than the one it already knows about. Specifically, we consider the setting where the agent is permitted to interrupt the user just once to improve its safely-optimal policy, by asking a single query about at most k unknown features. For each feature the user will reply whether or not it is in Φ U F . Formally, Φ q is a kfeature query where Φ q ⊆ Φ rel and |Φ q | = k. The postresponse utility when the agent asks query Φ q , and Φ c ⊆ Φ A ? are actually changeable, is the value of the safely-optimal policy after the user's response: u(Φ q , Φ c ) = max π∈Π Φ A ? \(Φq ∩Φc ) V π . (3) Recall the agent can only safely change features it queries about that the user's response indicates are changeable (Φ q ∩ Φ c ). What would be the agent's regret if it asks a k-feature query Φ q rather than a k-feature query Φ q ? We consider the circumstance where a set of features Φ c are changeable and under which the difference between the utilities of asking Φ q and Φ q is maximized. We call this difference of utilities the pairwise maximum regret of queries Φ q and Φ q , defined below in a similar way to Regan and  Boutilier [2010] : P M R(Φ q , Φ q ) = max Φc⊆Φ A ? (u(Φ q , Φ c ) − u(Φ q , Φ c )). (4) The maximum regret, denoted by M R, of query Φ q is determined by the Φ q that maximizes P M R(Φ q , Φ q ): M R(Φ q ) = max Φ q ⊆Φ rel ,|Φ q |=k P M R(Φ q , Φ q ). (5) The agent should ask the minimax-regret (k-feature) query: Φ M M R q = arg min Φq⊆Φ rel ,|Φq|=k M R(Φ q ). (6) The rationale of the minimax-regret criterion is as follows. Whenever the agent considers a query Φ q , there could exist a query Φ q that is better than Φ q under some true changeable features Φ c . The agent focuses on the worst case Φ c , where the difference between the utility of Φ q and the best query Φ q that could be asked is maximized. The agent uses adversarial reasoning to efficiently find the worst-case Φ c : Given that it is considering query Φ q , it asks what query Φ q and set of features Φ c an imaginary adversary would pick to maximize the gap between the utilities of Φ q and Φ q under Φ c (that is, u(Φ q , Φ c ) − u(Φ q , Φ c )). The agent wants to find a query Φ q that minimizes the worst case (maximum gap). Under the definition of M R, the agent, reasoning as if it were its imaginary adversary, must find the maximizing Φ q and Φ c . However, we can simplify the definition so that it only needs to find maximizing Φ c . Note that since an imaginary adversary chooses both Φ q and Φ c , it wants to make sure that Φ c ⊆ Φ q , which means that it does not want the features not in Φ q to be changeable. We then observe that M R(Φ q ) = max π ∈Γ: Φ rel (π )≤k (V π − max π∈Π Φ A ? \{Φq ∩Φ rel (π )} V π ) (7) We call the π maximizing Eq. 7 the adversarial policy when the agent asks query Φ q , denoted by π M R Φq . With Eq. 7, the agent can compute M R based on the set of dominating policies (which DomPolicies already found), rather than the (generally much larger) powerset of the relevant features in Eq. 5. While using Eq. 7 is faster than Eq. 5, the agent still needs to do this computation for every possible query (Eq. 6) of size k. We contribute two further ways to improve the efficiency. First, we may not need to consider all relevant features if we can only ask about k of them. If a subset of relevant features satisfies the condition in Theorem 2, then we call it a set of sufficient features, because a minimax-regret k-feature query from that set is a globally minimax-regret k-feature query. Second, we introduce a pruning rule that we call query dominance in Theorem 3 to safely eliminate considering queries that cannot be better than ones already evaluated. The following theorem shows that if we can find any subset Φ of Φ rel such that for all k-feature subsets of Φ as queries, the associated adversarial policy's relevant features are contained in Φ, then the minimax regret query found by restricting queries to be subsets of Φ will also be a minimax regret query found by considering all queries in Φ rel . Such a (nonunique) set Φ will be referred to as a sufficient feature set (for the purpose of finding minimax regret queries). Theorem 2. (Sufficient Feature Set) For any set of ≥ k features Φ, if for all Φ q ⊆ Φ, |Φ q | = k, we have Φ rel (π M R Φq ) ⊆ Φ, then min Φq⊆Φ,|Φq|=k M R(Φ q ) = min Φq⊆Φ rel ,|Φq|=k M R(Φ q ). Proof Sketch: If a set of features Φ, |Φ| ≥ k, fails to include some features in Φ M M R q , when we query some k-subset of Φ, the adversarial policy should change some of the features in Φ M M R q \ Φ. Otherwise, querying about Φ M M R q \ Φ does not reduce the maximum regret. Then Φ M M R q \ Φ are not necessary to be included in Φ M M R q . Given a set of sufficient features, the following theorem shows that it may not be necessary to compute the maximum regrets for all k-subsets to find the minimax-regret query. Theorem 3. (Query Dominance) For any pair of queries Φ q and Φ q , if Φ q ∩ Φ rel (π M R Φq ) ⊆ Φ q ∩ Φ rel (π M R Φq ), then M R(Φ q ) ≥ M R(Φ q ). Proof. Observe that M R(Φ q ) ≥ V π M R Φq − max π ∈Π Φ A ? \(Φ q ∩Φ rel (π M R Φq )) V π ≥ V π M R Φq − max π ∈Π Φ A ? \(Φq ∩Φ rel (π M R Φq )) V π = M R(Φ q ) We denote the condition Φ q ∩ Φ rel (π M R Φq ) ⊆ Φ q ∩ Φ rel (π M R Φq ) by dominance(Φ q , Φ q ). To compute dominance, we only need to store Φ rel (π M R Φq ) for all Φ q we have considered. Algorithm MMRQ-k below provides pseudocode for finding a minimax-regret k-feature query; it takes advantage of both the notion of a sufficient-feature-set as well as query dominance to reduce computation significantly relative to the brute-force approach of searching over all k-feature queries (subsets) of the relevant feature set. Algorithm MMRQ-k 1: Φ q ← an initial k-feature query 2: checked ← ∅; evaluated ← ∅ 3: Φ suf ← Φ q 4: agenda ← {Φ q } 5: while agenda = ∅ do 6: Φ q ← an element from agenda 7: if ¬∃ Φ q ∈evaluated dominance(Φ q , Φ q ) then 8: Compute M R(Φ q ) and π M R Φq 9: Φ suf ← Φ suf ∪ Φ rel (π M R Φq ) 10: evaluated ← evaluated ∪ {Φ q } 11: checked ← checked ∪ {Φ q } 12: agenda ← all k subsets of (Φ suf ) \ checked 13: return arg min Φq∈evaluated M R(Φ q ) Intuitively, the algorithm keeps augmenting the set of features Φ suf , which contain the features in the queries we have considered and the features changed by their adversarial policies, until it becomes a sufficient feature set. agenda keeps track of k-subsets in Φ suf that we have not yet evaluated. According to Theorem 2, we can terminate the algorithm when agenda is empty (Line 5). We also use Theorem 3 to filter out queries which we know are not better than the ones we have found already (Line 7). Note that an initial Φ suf needs to be chosen, which can be arbitrary. Our implementation initializes Φ q with the Chain of Adversaries heuristic (Section 5). To illustrate when Algorithm MMRQ-k can and can't prune suboptimal queries and thus gain efficiency, consider finding the minimax-regret 2-feature query in Figure  2  (a), which should be {c 1 , c 2 }. If the agent considers a query that does not include c 1 , the adversarial policy would change c 1 , adding c 1 to Φ suf . If the agent considers a query that includes c 1 but not c 2 , the adversarial policy would change c 2 , adding c 2 to Φ suf . When the agent asks {c 1 , c 2 }, the adversarial policy changes c 3 (adversarially asserting that c 1 , c 2 are locked), so c 3 is added to Φ suf . With {c 1 , c 2 , c 3 } ⊆ Φ suf , the condition in Theorem 2 holds and the n − 3 other features can be safely ignored. The minimax-regret query constituted by features in Φ suf is {c 1 , c 2 }. However, in Figure  2  (b), Φ suf = Φ rel , and all |Φ rel | 2 would be evaluated.
 Empirical Evaluations We now empirically confirm that Algorithm MMRQ-k finds a minimax-regret query, and its theoretically sound Sufficient-Feature-Set and Query-Dominance based improvements can indeed pay computational dividends. We also compare our MMRQ-k algorithm to baseline approaches and the Chain of Adversaries (CoA) heuristic  [Viappiani and Boutilier, 2009]  adapted to our setting. Algorithm CoA begins with Φ q0 = ∅ and improves this query by iteratively computing: π ← arg max π ∈Γ:|Φ rel (π )∪Φq i |≤k (V π − max π∈Π Φ A ? \{Φq i ∩Φ rel (π )} V π ) Φ qi+1 ← Φ qi ∪ Φ rel (π). The algorithm stops when |Φ qi+1 | = k or Φ qi+1 = Φ qi . Although Algorithm CoA greedily adds features to the query to reduce the maximum regret, unlike MMRQ-k it does not guarantee finding the minimax-regret query. For example, in Figure  2 (c), when k = 2, CoA first finds the optimal policy, which changes {c 1 , c 2 }, and returns that as a query, while the minimax-regret query is {c 1 , c 3 }. We compare the following algorithms in this section: 1. Brute force (rel. feat.) uses Algorithm DomPolicies to find all relevant features first and evaluates all k-subsets of the relevant features. 2. Algorithm MMRQ-k. 3. Algorithm CoA. 4. Random queries (rel. feat.), which contain k uniformly randomly chosen relevant features. 5. Random queries, which contain k uniformly randomly chosen unknown features, without computing relevant features first.
 No queries (equivalently vacuous queries). We evaluate the algorithms' computation times and the quality of the queries they find, reported as the normalized M R to capture their relative performance compared to the best and the worst possible queries. That is, the normalized M R of a query Φ q is defined as (M R(Φ q ) − M R(Φ M M R q ))/(M R(Φ q ⊥ ) − M R(Φ M M R q )), where Φ q ⊥ is a vacuous query, containing k features that are irrelevant and/or already known. The normalized M R of a minimaxregret query is 0 and that of a vacuous query is 1. Navigation. As illustrated in Figure  3 , the robot starts from the left-bottom corner and is tasked to turn off a switch at the top-right corner. The size of the domain is 6 × 6. The robot can move one step north, east or northeast at each time step. It stays in place if it tries to move across a border. The discount factor is 1. Initially, 10 clean carpets are uniformly randomly placed in the domain (the blue cells). In any cell without a carpet, the reward is set to be uniformly random in [−1, 0], and in a cell with a carpet the reward is 0. Hence, the robot will generally prefer to walk on a carpet rather than around it. The state of each carpet corresponds to one feature. The robot is uncertain about whether the user cares about whether any particular carpet gets dirty, so all carpet features are in Φ A ? . The robot knows that its own location and the state of the switch are in Φ A F . Since MMRQ-k attempts to improve on an existing safe policy, the left column and the top row never have carpets to ensure at least one safe path to the switch (the dotted line). The robot can ask one k-feature query before it takes any physical actions. We report results on 1500 trials. The only difference between trials are the locations of carpets, which are uniformly randomly placed. First, we compare the brute force method to our MMRQ-k algorithm. We empirically confirm that in all cases MMRQ-k finds a minimax regret query, matching Brute Force performance. We also see that brute force scales poorly as k grows while MMRQ-k, benefitting from Theorems 2 and 3, is more computationally efficient (Figure  4 ). We then want to see if and when MMRQ-k outperforms other candidate algorithms. In Figure  4 , when k is small, the greedy choice of CoA can often find the best features to add to the small query, but as k increases, CoA suffers from being too greedy. When k is large (approaches the number of unknown features), being selective is less important and all methods find good queries. We also consider how |Φ rel | affects the performance (Figure  5 ). When |Φ rel | is smaller Brute Force (rel. feat.) MMRQ-k CoA Random (rel. feat.) Random No Query   than k, a k-feature query that contains all relevant features is optimal. All algorithms would find an optimal query except Random (which selects from all unknown features) and No Queries. (The error bars are larger for small |Φ rel | since more rarely are only very few features relevant.) When |Φ rel | is slightly larger than k, even a random query may be luckily a minimax-regret query, and CoA unsurprisingly finds queries close to the minimax-regret queries. However, when |Φ rel | is much larger than k, the gap between MMRQ-k and other algorithms is larger. In summary, MMRQ-k's benefits increase with the opportunities to be selective (larger |Φ rel | k ). We have also experimented with expected regret given a probabilistic model of how the user will answer queries. For example, if the user has probability p of saying an unknown feature is free (1 − p it's locked), then as expected, when p is very low, querying rarely helps, so using MMRQ-k or CoA matters little, and as p nears 1, CoA 's greedy optimism pays off to meet MMRQ-k 's minimax approach. But, empirically, MMRQ-k outperforms CoA for non-extreme values of p.
 Extensions and Scalability We now briefly consider applying our algorithms to larger, more complicated problems. As mentioned in Section 2, features' changeabilities might be more nuanced, with restrictions on what values they can take, individually or in combination. An example of the latter from Fig.  1  is where doors are revertible features, which means their changeability is dependent on the "time" feature: they are freely changeable except that by the time the episode ends they need to revert to their initial values. This expands the set of possible feature queries (e.g., asking if d2 is changeable differs from asking if it is revertible). In our experiments, this change accentuates the advantages of MMRQ-k over CoA: CoA asks (is-d2locked, is-d2-revertible), hoping to hear "no" to both and follow a policy going through d2 without closing it. MMRQ-k asks (is-d2-locked, is-carpet-locked): since closing d2 only takes an extra time step, it is more valuable to know if the carpet is locked than if d2 can be left open. More nuanced feature changeability means DomPolicies will have to find policies for the powerset of every relevant combination of features' values (in the worst case the size of the state space). One option is to ignore nuances in ways that maintain safety (e.g., treat a feature as revertible even if sometimes it can be left changed) and solve such a safe abstraction of the problem. Or one could abstract based on guaranteed correlations in feature changeabilities (e.g., if all boxes have the same changeability then ignore asking about all but one). Another option is to find only a subset of dominating policies, for example by using knowlege of k to avoid finding dominating policies that would change more unknown features than could be asked about anyway. And, or course, as mentioned before, finding approximately-safely-optimal policies in DomPolicies Line 10 would help speed the process (and might be the only option for larger problem domains). Fortunately, such abstractions, heuristics, and approximations do not undermine safety guarantees. Recall that Dom-Policies and MMRQ-k are finding a query, not the agent's final policy: the safety of the agent depends on how it finds the policy it executes, not on the safety of policies for hypothetical changeability conditions. However, as coarser abstractions, heuristics, and approximations are employed in our algorithms, the queries found can increasingly deviate from the minimax-regret optima. Fortunately, if the agent begins with a safely-optimal policy, "quick and dirty" versions of our methods can never harm it (they just become less likely to help). And if it begins without such a policy, such versions of our methods might not guide querying well, but by eventually asking about every unknown feature (in the worst case) a safe policy will still be found (if one exists).
 Related Work & Summary Amodei et al.  [2016]  address the problem of avoiding negative side-effects by penalizing all side-effects while optimizing the value. In our work, we allow the agent to communicate with the user. Safety is also formulated as resolving reward uncertainty  [Amin et al., 2017; Hadfield-Menell et al., 2017] , following imperfectly-specified instructions  [Milli et al., 2017] , and learning safe states  [Laskey et al., 2016] . Safety issues also appear in exploration  [Hans et al., 2008; Moldovan and Abbeel, 2012; Achiam et al., 2017] . Here we only provide a brief survey on safety in MDPs.  Leike et al. [2017 ], Amodei et al. [2016  and  Garcıa and Fernández [2015]  are more thorough surveys. There are problems similar to finding safely-optimal policies, which find policies that satisfy some constraints/commitments or maximize the probability of reaching a goal state  [Witwicki and Durfee, 2010; Teichteil-Königsbuch, 2012; Kolobov et al., 2012] . There are also other works using minimax-regret and policy dominance  [Regan and Boutilier, 2010; Nilim and El Ghaoui, 2005] . and querying to resolve uncertainty  [Weng and Zanuttini, 2013; Regan and Boutilier, 2009; Cohn et al., 2011; Zhang et al., 2017] . We've combined and customized these ideas to find a provably minimax-regret k-element query. In summary, we addressed the problem of an agent selectively querying a user about what features can be safely sideeffected. We borrowed existing ideas from the literature about dominating policies and minimax regret, wove them together in a novel way, and streamlined the resulting algorithms to improve scalability while maintaining safe optimality. Figure 1 : 1 Figure 1: The robot navigation domain. The dominating policies (see Section 3) are shown as arrows.
 a)T (s |s, a) + δ(s , s 0 ) ∀s ∈ S Φ A L ∪Φ A
 the initial set of dominating policies 2: Φ rel ← ∅ the initial set of relevant features 3: checked ← ∅ It contains Φ ⊆ Φ rel we have examined so far. 4: β ← ∅ a pruning rule 5: agenda ← powerset(Φ rel ) \ checked 6: while agenda = ∅ do 7: Φ ← least-cardinality element of agenda 8: if satisf y(Φ, β) then 9: (get safely-optimal policy with Φ locked) 10: π ← arg max π ∈ΠΦ V π by solving Eq. 1 11: if π exists then 12: Γ ← Γ ∪ {π}; add (Φ, Φ rel (π)) to β 13:
 Figure 2 : 2 Figure 2: Example domains used in text. (n > 2)
 (a) and (b) only need to compute policies linear in the number of relevant features: Only n+1 dominating safely-optimal policies are computed for Figure 2(a) (for Φ = ∅, {c 1 }, {c 1 , c 2 }, . . . ), and Figure 2(b) (for Φ = ∅, {c 1 }, {c 2 }, . . . , {c n }). Figure 2(c) computes policies for only half of the subsets. Theorem 1. The set of policies returned by Algorithm Dom-Policies is the set of all dominating policies.
 Figure 3 : 3 Figure 3: Office navigation and legend for following figures.
 Figure 4 : 4 Figure 4: Normalized maximum M R vs. k. |Φ ? | = 10. Brute force computation time is only shown for k = 0, 1, 2, 3.
 Figure 5 : 5 Figure 5: Normalized M R vs. the number of relevant features. |Φ ? | = 10 and k = 2, 4, 6.

Abstract: As it achieves a goal on behalf of its human user, an autonomous agent's actions may have side effects that change features of its environment in ways that negatively surprise its user. An agent that can be trusted to operate safely should thus only change features the user has explicitly permitted. We formalize this problem, and develop a planning algorithm that avoids potentially negative side effects given what the agent knows about (un)changeable features. Further, we formulate a provably minimax-regret querying strategy for the agent to selectively ask the user about features that it hasn't explicitly been told about. We empirically show how much faster it is than a more exhaustive approach and how much better its queries are than those found by the best known heuristic.