Allan Dafoe
Yoram Bachrach
Gillian Hadfield
Eric Horvitz
Kate Larson
Thore Graepel
Sean Gallup
/ Getty

Cooperative AI: machines must learn to find common ground

its environment and how to interact with it. Even in work involving multiple AI agents, the field has not yet tackled the hard problems of cooperation. Most headline results have come from two-player zero-sum games, such as backgammon, chess  1  , Go 2 and poker  3  . Gains in these competitive examples can be made only at the expense of others. Although such settings of pure conflict are vanishingly rare in the real world, they make appealing research projects. They are culturally cherished, relatively easy to benchmark (by asking whether the AI can beat the opponent), have natural curricula (because students train against peers of their own skill level) and have simpler solutions than semi-cooperative games do. AI needs social understanding and cooperative intelligence to integrate well into society. The coming years might give rise to diverse ecologies of AI systems that interact in rapid and complex ways with each other and with humans: on pavements and roads, in consumer and financial markets, in e-mail communication and social media, in cybersecurity and physical security. Autonomous vehicles or smart cities that do not engage well with humans will fail to deliver their benefits, and might even disrupt stable human relationships. We need to build a science of cooperative AI. As researchers in the field and its governance, we argue that it is time to prioritize the development of cooperative intelligence that has the ability to promote mutually beneficial joint action, even when incentives are not fully aligned. Just as psychologists studying humans have found that the infant brain does not develop fully without social interaction, progress towards socially valuable AI will be stunted unless we put the problem of cooperation at the centre of our research. Cooperative intelligence is unlikely to emerge as a by-product of research on other kinds of AI. We need more work on cooperative games and complex social spaces, on understanding norms and behaviours, and on social tools and infrastructure that promote cooperation. The AI community should learn more from, and contribute to, other fields that work on cooperation.
 From autonomy to cooperation Parents encourage their children to grow beyond their dependencies and become autonomous. But autonomy is rarely regarded as the sole goal for humans. Rather, we are generally most productive when we work cooperatively as part of broader society. Similarly, certain kinds of autonomy in AI systems are useful precisely because they enable the system to contribute effectively to broader cooperative efforts. Most of the value from self-driving vehicles will come not from driving on empty roads, but from vehicles coordinating smoothly with the flow of pedestrians, In most settings, people's incentives are not fully aligned. Nevertheless, they can often cooperate, taking joint action to achieve mutually beneficial outcomes. One example is countries agreeing and enforcing carbon cuts to tackle climate change. The cooperative intelligence needed to achieve this has four parts: Understanding. The ability to take into account the consequences of actions, to predict another's behaviour, and the implications of another's beliefs and preferences. Communication. The ability to explicitly and credibly share information with others relevant to understanding behaviour, intentions and preferences. Commitment. The ability to make credible promises when needed for cooperation.
 Norms and institutions. Social infrastructure -such as shared beliefs or rules -that reinforces understanding, communication and commitment.
 Four elements of cooperative intelligence by pure conflict -when there is no possibility of bargains, negotiations or threats. So, improving skill at inherently rivalrous games is unlikely to be the most promising way for AI to produce social value. Games of pure common interest are a step towards developing cooperative agents. The cooperative card game Hanabi 4 requires players to communicate private information and intentions under strong constraints about what can be said and when. Team games such as robot soccer  5  need players on a team to work as one, jointly planning their moves and passing the ball. In these examples, all agents on a team share the same goals. Mastering these games requires many skills essential to cooperation. Research avenues include building AIs that can understand what teammates are thinking and planning; communicate plans; and even cooperate with different kinds of teammate who might think differently and react more slowly (known as ad hoc teamwork). Yet because these situations are restricted to a perfect harmony of interests, they represent the easy case for cooperation. Real-world relationships almost always involve a mixture of common and conflicting interests. This tension gives rise to the rich texture of human cooperation problems, including bargaining, trust and mistrust, deception and credible communication, commitment problems and assurances, politics and coalitions, and norms and institutions. AI agents will need to learn how to manage these harder cooperation problems, as humans do. An example is the board game Diplomacy, in which players negotiate non-binding alliances with others. To succeed, AI agents will need to understand each other well enough to recognize when their interests are aligned with those of other players. They will have to develop a common vocabulary to communicate their intentions. They will benefit from being able to communicate credibly, despite possible incentives to lie. They must overcome mutual fears of betrayal, so as to agree on and execute jointly beneficial plans. They might even learn to establish norms relating to the adherence of agreements. To enable progress in these cooperative skills, researchers have devised variants of Diplomacy that modify the difficulty of these challenges, such as introducing an agreed simple vocabulary or permitting binding commitments.
 Human-AI cooperation AI is increasingly present, underlying everything from dynamic pricing strategies to loans and prison-sentencing decisions. Collaborative industrial robots work on factory floors alongside labourers  6  , care robots help human health workers and personal AI assistants (such as Amazon's Alexa, Apple's Siri and Google Assistant) help us with scheduling, albeit in an elementary way. cyclists and cars driven by humans. Thus, cooperative intelligence is not an alternative to autonomous intelligence, but goes beyond it. AI research on cooperation will need to bring together many clusters of work. A first cluster consists of AI-AI cooperation, tackling ever more difficult, rich and realistic settings (see 'Four elements of cooperative intelligence'). A second is AI-human cooperation, for which we will need to advance natural-language understanding, enable machines to learn about people's preferences, and make machine reasoning more accessible to humans. A third cluster is work on tools for improving (and not harming) human-human cooperation, such as ways of making the algorithms that govern social media better at promoting healthy online communities.
 AI-AI cooperation Multi-agent AI research has seen most success in two-player zero-sum settings, from the superhuman performance of IBM's chess-playing computer Deep Blue to the powerful demonstration of deep reinforcement learning by the program AlphaGo. However, few interactions in the real world are characterized The design of agents that will act in accordance with human intentions, preferences and values -known as AI alignment  7  -is a crucial part of cooperative AI. But it is only a part, because the relationship between a single human (acting as the principal) and a single machine (acting as the agent) isn't always clear. Real-world cooperation problems often involve multiple stakeholders, some conflicting interests and integration with our institutional and normative infrastructure. A particular challenge facing researchers working on human-AI cooperation is that it involves, well, humans. Today, many deployed machine-learning models are trained either on massive data sets or in simulated environments that can generate years of experience in seconds. For example, the program AlphaZero learnt to play chess by playing 44 million games against itself over 9 hours. By contrast, humans produce data slowly, and require researchers to consider compensation, ethics and privacy. There might be ways to use fewer human participants, such as extensively training AIs in simulation and then fine-tuning them in the real world. Many AI practitioners have dreamt of building autonomous and human-like intelligence. They envisioned systems that could replace human labour, acting with the speed and resilience of machines, and scaling up rapidly given increases in computing power, algorithmic efficiency and capital. However, unlike systems that have tight integration with human workers, autonomous systems might pose greater safety risks. Human-like AI might be more likely to displace labour. Instead, we could develop AI assistants that complement human intelligence and depend on us for tasks in which humans have a comparative advantage. As Stanford University radiologist Curtis Langlotz put it: "AI won't replace radiologists, but radiologists who use AI will replace radiologists who don't." Progress will require advances in understanding human language, gestures and activities, and ad hoc teamwork, in addition to preference learning by machines, safety, interpretability by humans  8  , and understanding of norms. Research will need to approach increasingly rich and realistic environments. Instead of benchmarking progress mainly by whether autonomous machines can outperform autonomous humans on a task, researchers should also assess the performance of human-machine teams.
 AI for human collaboration Humans confront ubiquitous cooperation problems as commuters, neighbours, co-workers and citizens. The global scientific community, for example, could benefit from better tools for identifying relevant work and promising collaborations. Technology is crucial, mediating our ability to find and process information, communicate and self-organize. Digital systems and AI can expand this toolkit. Some AI tools, such as machine language translation, seem strongly disposed towards promoting cooperation. Today, 2 people who speak any of more than 100 languages can communicate with the aid of a smartphone and a translation app. Digital platforms such as Wikipedia, Reddit and Twitter provide tools to combine user-provided content. AI advances could improve this community infrastructure, for example, by routing relevant information to contributors more efficiently to enhance collaborative editing. Other advances could improve user rating and reputation systems through better modelling and by accounting for the rater's repute or relevance, as well as by enabling recommendation algorithms that more Computer scientists in Leipzig, Germany, prepare their robot soccer team for a test game.
 "To succeed, cooperative AI must connect with the broader science of cooperation." intelligently promote a community's values. Building healthy online communities is challenging; just as social media can connect us, so too can it polarize, stress, misinform, distract and addict us  9  . Researchers and developers need to find better ways to name and measure desirable properties and build algorithms that encourage them. Platforms for political deliberation can be designed to promote empathy about different viewpoints and cultivate community consensus. Methods for achieving this include language comprehension that links to structured databases of knowledge, or clustering algorithms to identify related perspectives.
 Next steps To succeed, cooperative AI must connect with the broader science of cooperation, which spans the social, behavioural and natural sciences. AI research will need to converse with multiple fields. These include psychology, to understand human cognition; law and policy, to understand institutions; history, sociology and anthropology, to understand culture; and political science and economics, to understand problems of information, commitment and social choice. Adjacent research areas are developing AI with socially desirable properties, such as alignment, interpretability and fairness  10, 11  . Each of these addresses a distinct, but complementary, set of challenges. The need for interdisciplinarity is exemplified by a landmark work: Robert Axelrod's The Evolution of  Cooperation, published in 1984 (ref .12) . Axelrod, a political scientist, brought together game theorists, mathematicians, economists, biologists and psychologists in a tournament to help devise the best algorithms for the iterated Prisoner's Dilemma, the canonical example of why two rational people might not cooperate. The winning solution that cooperated most successfully, called Tit for Tat, was devised by Anatol Rapoport, a US scholar with a background spanning mathematics, biology, network science and peace studies. Axelrod's tournament offered another lesson. It gave researchers a benchmark for success in the design of cooperative algorithms, just as ImageNet 13 did for computer vision by collecting and labelling millions of photos. Cooperative AI research will similarly gain momentum if investigators can devise, agree on and adopt benchmarks that cover a diverse set of challenges: playing cooperative board games, integrating into massive multiplayer video games, navigating simplified environments that require machine-human interaction, and anticipating tasks as a personal assistant might. Similar to the state-of-the-art in language modelling, considerable effort and creativity will be needed to make sure these benchmarks remain sufficiently rich and ambitious, and do not have socially harmful blind spots. The most important challenges of cooperation might be the most difficult to benchmark; they involve creatively stepping out of our habitual roles to change the 'game' itself. Indeed, if we are to take the social nature of intelligence seriously, we need to move from individual objectives to the shared, poorly defined ways humans solve social problems: creating language, norms and institutions. Science is a social enterprise, so promoting research into cooperative AI will require social interventions. A recent milestone was a December 2020 workshop on cooperative AI at the leading machine-learning conference NeurIPS. It involved speakers from a diverse array of disciplines, and resulted in a review of Open Problems in Cooperative AI  14  . We and others are establishing a Cooperative AI Foundation to support this nascent field (www.cooperativeai.org), backed by a large philanthropic commitment. The foundation's mission will be to catalyse advances in cooperative intelligence to benefit all of humanity, including efforts to fund fellowships, organize conferences, support benchmarks and environments, and award prizes. The crucial crises confronting humanity are challenges of cooperation: the need for collective action on climate change, on political polarization, on misinformation, on global public health or on other common goods, such as water, soil and clean air. As the potential of AI continues to scale up, a nudge in the direction of cooperative AI today could enable us to achieve much-needed global cooperation in the future.

			 © 2 0 2 1 S p r i n g e r N a t u r e L i m i t e d . A l l r i g h t s r e s e r v e d .

Abstract: rtificial-intelligence assistants and recommendation algorithms interact with billions of people every day, influencing lives in myriad ways, yet they still have little understanding of humans. Self-driving vehicles controlled by artificial intelligence (AI) are gaining mastery of their interactions with the natural world, but they are still novices when it comes to coordinating with other cars and pedestrians or collaborating with their human operators. The state of AI applications reflects that of the research field. It has long been steeped in a kind of methodological individualism. As is evident from introductory textbooks, the canonical AI problem is that of a solitary machine confronting a non-social environment. Historically, this was a sensible starting point. An AI agent -much like an infant -must first master a basic understanding of


David Abel
John Salvatier
Andreas Stuhlmüller
Owain Evans

Agent-Agnostic Human-in-the-Loop Reinforcement Learning

Introduction A central goal of Reinforcement Learning (RL) is to design agents that learn in a fully autonomous way. An engineer designs a reward function, input/output channels, and a learning algorithm. Then, apart from debugging, the engineer need not intervene during the actual learning process. Yet fully autonomous learning is often infeasible due to the complexity of real-world learning problems, the difficulty of specifying reward functions that lead to good performance, and the presence of potentially dangerous outcomes. Consider a robot learning to perform household chores. Human engineers design and supervise a curriculum, moving the agent between simulation, practice environments, and real house environments. Over time, they might tweak reward functions, heuristics, and state representations. They may intervene directly in real-world training to prevent the robot damaging itself, destroying valuable goods, or harming people it interacts with. In this example, humans do not just design the learning agent: they are also "in the loop" of the agent's learning process. Many learning systems have a human in the loop. Self-driving cars learn with humans ready to intervene in dangerous situations. Facebook's algorithm for recommending "trending" news stories has humans filtering out inappropriate content  [1] . In both examples, the agent's environment is complex, non-stationary, and there are a wide range of damaging outcomes (like a traffic accident). As RL is applied to increasingly complex real-world problems, such interactive guidance will become more important.
 Our contribution: agent-agnostic guidance of RL algorithms Our goal is to develop a framework for human-agent interaction that is (a) agent-agnostic and (b) can capture a wide range of ways a human can help an RL agent. This paper targets environments where the state is fully observed; that is, the learning agent interacts with a Markov Decision Process (MDP)  [33, 20, 36] . Prior literature has investigated how people can help RL agents learn more efficiently through different methods of interaction  [29, 22, 25, 32, 37, 43, 19, 16, 23, 38, 21, 44, 40, 39, 27, 12] . Often, the human's role is to pass along knowledge about relevant quantities of the RL problem, like Q-values, action optimality, or the true reward for a particular state-action pair. This way, the person can bias exploration, prevent catastrophic outcomes, and accelerate learning. Most existing work develops agent-specific protocols for human interaction. That is, protocols for human interaction or advice that are designed for a specific RL algorithm (such as Q-learning). For instance, Griffith et al.  [16]  investigate the power of policy advice for a Bayesian Q-Learner. Other works assume that the states of the MDP take a particular representation, or that the action space is discrete or finite. Making explicit assumptions about the agent's learning process can enable more powerful teaching protocols that leverage insights about the learning algorithm or representation. Agent-specific protocols for human interaction can be contrasted with agent-agnostic protocols. An agent-agnostic protocol is a single protocol that can be plugged into any RL agent, such as an agent based on Q-learning, policy gradient, MCTS, Deep Q-learning, and so on, and that can provide advice/guidance to any such agent. There are obvious disadvantages to agent-agnostic protocols. The agent is not specialized to the protocol, so it is unable to use Active Learning (asking the human informative questions as in  [4] ) and will not automatically have an observation model that faithfully represents the process the human uses to generate advice (as in  [16, 19] ). Likewise, the human cannot provide optimally informative advice to the agent as they don't know the agent's prior knowledge, exploration technique, representation, or learning method. Conversely, agent-specific protocols may perform well for one type of algorithm or environment, but poorly on others. When researchers tackle challenging RL problems, they tend to explore a large space of algorithms with important structural differences: some are model-based vs. model-free, some approximate the optimal policy, others a value function, and so on. It is substantial effort to adapt a human-advice protocol to each such algorithm. Moreover, as advice protocols and learning algorithms become more complex, greater modularity will help limit design complexity. In our framework, the interaction between the person guiding the learning process, the agent, and the environment is formalized as a protocol program. This program controls the channels between the agent and the environment based on human input (see Figure  1 ). This allows the human extensive control over the agent: in an extreme case, the agent can be prevented from interacting with the real environment entirely and only interact with a simulation. At the same time, we require that the human only interact with the agent during learning through the protocol program-both agent and environment are a black box to the human.
 Framework Any system for RL with a human in the loop has to coordinate three components: The environment is an MDP and is specified by a tuple M = (S, A, T , R, γ), where S is the state space, A is the action space, T : S × A × S → [0, 1], denotes the transition function, a probability distribution on states given a state and action, R : S × A → R is the reward function, and γ is the discount factor. The agent is a (stateful, potentially stochastic) function L : S × R → A. The human can receive and send advice information of flexible type, say X in and X out , so, we will treat the human as a (stateful, potentially stochastic) function H : X in → X out . For example, X in might contain the history of actions, states, and rewards so far, and a new proposed action a , and X out might be an action as well, either equivalent to a (if accepted) or different (if rejected). We assume that the human knows in general terms how their responses will be used and is making a good-faith effort to be helpful. The interaction between the environment, the agent, and a human advisor sets up a mechanism design problem: how can we design an interface that orchestrates the interaction between these components such that the combined system maximizes the expected sum of γ-discounted rewards from the environment? In other words, how can we write a protocol program P : S × R → A that can take the place of a given agent L, but that achieves higher rewards by making efficient use of information gained through sub-calls to L and H? By formalizing existing and new techniques as explicit programs, we facilitate understanding and comparison of these techniques within a common framework. By abstracting from particular agents and environments, we may be able to design mechanisms that scale to more advanced agents  [7] . Algorithm 1 Agent in control (standard) 1: procedure AGENTCONTROL(s, r) 2: return L(s, r) 3: end procedure Algorithm 2 Human in control 1: procedure HUMANCONTROL(s, r) 2: return H(s, r) 3: end procedure Algorithm 3 Training in simulation 1: M * = (S, A, T * , R * , γ) Simulation 2: η = [] History: array of (S × R × A) 3: procedure TRAININSIMULATION(s, r) 4: s = s 5: r = r 6: while H(η) = "agent is ready" do r = H(s, r) 3: return L(s, r) 4: end procedure Figure  2 : Many schemes for human guidance of RL algorithms can be expressed as protocol programs. These programs have the same interface as the agent L, but can be safer or more efficient learners by making use of human advice H.
 Capturing Existing Advice Schemes Protocol programs cannot capture all advice protocols. Any protocol that depends on prior knowledge of the agent's learning algorithm, priors or hyper-parameters is ruled out. Despite this constraint, the framework can capture a range of existing protocols where a human-in-the-loop guides an agent. Figure  1  shows that the human can manipulate the actions sent to the environment and the agent's observed states and rewards. This points to a combinatorial set of protocols, including reward shaping, action pruning, providing hand-coded features of state, and training in simulation.
 Reward shaping Section 2 defined the reward function R as part of the MDP M . However, while humans don't design real-world transition functions, we do design reward functions. Usually the reward function is hand-coded prior to learning and must accurately assign reward values to any state the agent might reach. An alternative is to have a human generate the rewards "interactively". The human observes the state and action and returns a scalar to the agent. This setup has been explored in work on TAMER  [21] . A similar setup (with an agent-specific protocol) was applied to robotics by Daniel et al.  [8] . It is straightforward to represent rewards that are generated interactively (or "online") using protocol programs. We now turn to other protocols in which the human manipulates rewards. These protocols assume a fixed reward function R that is part of the MDP M .
 Reward shaping and Q-value initialization In Reward Shaping protocols, the human engineer changes the rewards given by some fixed reward function in order to influence an agent's learning. Ng et al.  [29]  introduced "potential-based" shaping, which shapes rewards without changing an MDP's optimal policy. In particular, each reward received by the environment is augmented by a shaping function: F (s, a, s ) = γφ(s ) − φ(s), (1) so the agent actually receives r = F (s, a, s ) + R(s, a). Wiewiora et al.  [43]  showed potential shaping to be equivalent (for Q-learners) to a subset Q-value initialization under some assumptions. Further, Devlin and Kudenko  [10]  propose dynamic potential shaping functions that change over time. That is, the shaping function F also takes as two time parameters, t and t , such that: F (s, t, s , t ) = γφ(s , t ) − φ(s, t) (2) Where t > t. Their main result is that dynamic shaping functions of this form also guarantee optimal policy invariance. Similarly, Wiewiora et al.  [43]  extend potential shaping to potential-based advice functions, which identifies a similar class of shaping functions on (s, a) pairs. In Section 4, we show that our Framework captures reward shaping, and consequently, some notion of Q-value initialization.
 Training in Simulation It is common practice to train an agent in simulation and transfer it to the real world once it performs well enough. Algorithm 3 (Figure  2 ) shows how to represent this as a protocol program. The idea is as follows: Let M be the real-world'. The protocol program includes a simulator M * . The agent L interacts with M * while the human observes performance and at some point transfers L to M . 1
 Action Pruning Action pruning is a technique for dynamically removing actions from the MDP to reduce the branching factor of the search space. Such techniques have been shown to accelerate learning and planning time  [35, 17, 34, 2] . In Section 5, we apply action-pruning to the problem of preventing catastrophic actions ("Safe RL"). Protocol programs allow action pruning to be carried out interactively. Instead of having to decide which actions to prune prior to learning, the human can wait to observe the states that are actually encountered by the agent, which may be valuable in cases where the human has limited knowledge of the environment or the agent's learning ability. In Section 4, we exhibit an agent-agnostic protocol for interactively pruning actions that preserves the optimal policy. Our pruning protocol is illustrated in a gridworld with lava pits (Figure  3 ). The agent is represented by a gray circle, "G" is a goal state that provides reward +1, and the red cells are lava pits with reward −200. All white cells provide reward 0. At each time step, the human checks whether the agent moves into a lava pit. If it does not (as in moving "DOWN" from state 34), the agent continues as normal. If it does (as in moving "RIGHT" from state 33), the human sends a "STAY" action to the MDP (preventing movement right) and a "next state" of 33 to the agent. The agent doesn't actually fall in the lava but the human sends them a reward r ≤ −200. After this negative reward, the agent is unlikely to try the action again. For the protocol program, see Algorithm 4 in Figure  2 . Note that the agent receives no explicit signal that their attempted catastrophic action was blocked by the human. They observe a big negative reward and a "self-loop" state transition but no information about whether the human or environment generated their observation.
 Manipulating state representation The agent's state representation can have a significant influence on its learning. Suppose the MDP M has a "raw" state vector s. The human engineer can specify a mapping φ such that the agent always receives φ(s) = s in place of s. Such mappings are used to specify high-level features of state that are important for learning, or to dynamically ignore confusing features from the agent. Other methods have focused on state abstraction functions of this form that can decrease learning time and preserve the quality of learned behavior, as in  [24, 30, 13, 18, 9, 3, 14] .
 Theory Here we illustrate some simple ways in which our proposed agent-agnostic interaction scheme captures other existing agent-agnostic protocols. The following results all concern Tabular MDPs, but are intended to offer intuition for high-dimensional or continuous environments as well.
 Reward Shaping First we observe that protocol programs can precisely capture methods for shaping reward functions. Remark 1: For any reward shaping function F , including potential-based shaping, potential-based advice, and dynamic potential-based advice, there is a protocol that produces the same rewards. To construct such a protocol for a given F , simply let the reward output by the protocol, r, take on the value F + r at each time step. That is, in Algorithm 5, simply define H(s, r) = F (s) + r.
 Action Pruning We now show that there is a simple class of protocol programs that carry out action pruning of a certain form. Remark 2: There is a protocol for pruning actions in the following sense: for any set of state action pairs sa ⊂ S × A, the protocol ensures that, for each pair (s i , a j ) ∈ sa, action a j is never executed in the MDP in state s i . The protocol is as described in Section 3.3 and shown in Algorithm 4. The premise is this: in all cases where the agent executes an action that should be pruned, the protocol gives the agent low reward and forces the agent to self-loop. Knowing which actions to prune is itself a challenging problem. Often, it is natural to assume that the human guiding the learner knows something about the environment of interest (such as where high rewards or catastrophes lie), but may not know every detail of the problem. Thus, we consider a case in which the human has partial (but useful) knowledge about the problem of interest, represented as an approximate Q-function. The next remark shows there is a protocol based on approximate knowledge with two properties: (1) it never prunes an optimal action, (2) it limits the magnitude of the agent's worst mistake: Remark 3: Assuming the protocol designer has a β-optimal Q function: ||Q * (s, a) − Q H (s, a)|| ∞ ≤ β (3) there exists a protocol that never prunes an optimal action, but prunes all actions so that the agent's mistakes are never more than 4β below optimal. That is, for all times t: V Lt (s t ) ≥ V * (s t ) − 4β, (4) where L t is the agent's policy after t timesteps. Proof of Remark 3. The protocol designer has a β-approximate Q function, denoted Q H , defined as above. Consider the state-specific action pruning function H(s): H(s) = a ∈ A | Q H (s, a) ≥ max a Q H (s, a ) − 2β (5) The protocol prunes all actions not in H(s) according to the self-loop method described above. This protocol induces a pruned Bellman Equation over available actions, H(s), in each state: V H (s) = max a∈H(s) R(s, a) + γ s T (s, a, s )V H (s ) (6) Let a * denote the true optimal action: a * = arg max a Q * (s, a ). To preserve the optimal policy, we need a * ∈ H(s), for each state. Note that a * ∈ H(s) when: Q H (s, a * ) < max a Q H (s, a ) − 2β (7) But by definition of Q H (s, a): |Q H (s, a * ) − max a Q H (s, a)| ≤ 2β (8) Thus, a * ∈ H(s) can never occur. Furthermore, observe that H(s) retains all actions a for which: Q H (s, a) ≥ max a Q H (s, a ) − 2β, (9) holds. Thus, in the worst case, the following two hold: 1. The optimal action estimate is β too low: Q H (s, a * ) = Q * (s, a * ) − β 2. The action with the lowest value, a bad , is β too high: Q H (s, a bad ) = Q * (s, a bad ) + β From Equation  9 , observe that the minimal Q * (s, a bad ) such that a bad ∈ H(s) is: Q * (s, a bad ) + β ≥ Q * (s, a * ) − β − 2β ∴ Q * (s, a bad ) ≥ Q * (s, a * ) − 4β Thus, this pruning protocol never prunes an optimal action, but prunes all actions worse then 4β below a * in value. We conclude that the agent may never execute an action 4β below optimal.
 Experiments This section applies our action pruning protocols (Section 3.3 and Remarks 2 and 3 above) to concrete RL problems. In Experiment 1, action pruning is used to prevent the agent from trying catastrophic actions ("safe exploration"). In Experiment 2, action pruning is used to accelerating learning.
 Protocol for Preventing Catastrophes As noted in the Introduction, human-in-the-loop RL can help prevent disastrous outcomes that result from ignorance of the environment's dynamics or of the reward function. Our goal for this experiment is to prevent the agent from taking catastrophic actions. These are real-world actions so costly that we want the agent to never take the action 2 . This notion of catastrophic action is closely related to ideas in "Safe RL"  [15, 28]  and to work on "significant rare events"  [31] . Section 3.3 describes our protocol program for preventing catastrophes in finite MDPs using action pruning. There are two important elements of this program: 1. When the agent tries a catastrophic action (s, a), a is blocked and the new state and reward is (s, r bad ), where r bad is an extreme negative reward. 2. This (s, a) is stored so that the protocol program can automate the human's intervention, which could allow the human to stop monitoring after all catastrophes have been stored. This protocol prevents catastrophic actions while preserving the optimal policy and having only minimal side-effects on the agent's learning. We can extend this protocol to environments with high-dimensional state spaces. Element (1) above remains the same. But (2) must be modified: preventing future catastrophes requires generalization across catastrophic actions (as there will be infinitely many such actions). We briefly discuss this setting in Appendix A.
 Experiment 1: Preventing Catastrophes in a Pong-like Game Our protocol for preventing catastrophes is intended for use in a real-world environment. Here we provide a preliminary test of our protocol in a simple video game. Our protocol treats the RL agent as a black box. So we applied our protocol to an open-source implementation of the state-of-the-art RL algorithm "Trust Region Policy Optimization" [? ]. The environment was Catcher, a simplified version of Pong with non-visual state representation. Since there are no "catastrophic" actions in Catcher, we modified the game to give a large negative reward when the paddle's speed exceeds a speed limit.  3  We compare the performance of an agent who has assistance by the protocol ("Pruned") and so is blocked from the catastrophic actions to the performance of a normal RL agent ("Not Pruned"). Figure  4  shows the agent's mean performance over the course of learning. We see that the agent with protocol support ("Pruned") performed much better overall. This is unsurprising, as it was blocked from ever doing a catastrophic action. The gap in mean performance is large early on but diminishes as the "Not Pruned" agent learns to avoid high speeds. By the end (i.e. after 400,000 actions), "Not Pruned" is close to "Pruned" in mean performance but its total returns over the whole period are around 20 times worse. Note that the "Pruned" agent observes "odd" state transitions due to being blocked by our protocol. We saw no evidence of these observations having negative side effects on learning.
 Protocol for Accelerating Learning We also conducted a simple experiment in the Taxi domain, introduced by Dietterich  [11] . The Taxi problem is a more complex version of grid world: the agent directs a taxi to each passenger, picks them up, and brings them to their destination. We use Taxi to evaluate the effect of our action pruning protocol for discrete MDPs. There is a natural procedure for pruning suboptimal actions that dramatically reduces the size of the reachable state space: if the taxi is carrying a passenger but is not at the passenger's destination, we prune the dropoff action by returning the agent back to its current state with -0.01 reward. This prevents the agent from exploring a large portion of the state space, thus accelerating learning.
 Experiment 2: Accelerated Learning in Taxi We evaluated Q-learning  [41]  and R-MAX  [5]  with and without action pruning in a simple 10 × 10 instance with one passenger. The taxi starts at (1, 1), the passenger at (4, 3) with destination (2, 2). We ran standard Q-Learning with ε-greedy exploration with ε = 0.2 and with R-MAX using a planning horizon of four. Results are displayed in Figure  5 . Our results suggest that the action pruning protocol simplifies the problem for a Q-Learner and dramatically so for R-Max. In the allotted number of episodes, we see that pruning substantially improves the overall cumulative reward achieved; in the case of R-MAX, the agent is able to effectively solve the problem after a small number of episodes. Further, the results suggests that the agentagnostic method of pruning is effective without having any internal access to the agent's code.
 Conclusion We presented an agent-agnostic method for giving guidance to Reinforcement Learning agents. Protocol programs written in this framework apply to any possible RL agent ("agent-agnosticism") so sophisticated schemes for human-agent interaction can be designed in a modular fashion without the need for adaptation to different RL algorithms. We presented some simple theoretical results that relate our method to existing schemes for interactive RL and illustrated the power of action pruning using one such protocol on two toy domains. In the future, we plan on exploring state manipulation protocols, which offer a powerful avenue for guiding an agent's learning process. For instance, one could imagine a protocol dynamically imposing abstractions that hide confusing features from the agent and encourage efficient learning. Additionally, we want to investigate whether certain types of value initialization protocols can be captured by protocol programs, such as the optimistic initialization for arbitrary domains developed by Machado et al.  [26] . Figure 1 : 1 Figure 1: A general setup for RL with a human in the loop. By instantiating P with different protocol programs, we can implement different mechanisms for human guidance of RL agents.
 Algorithm 4 5 45 Action pruning 1: ∆ = {} Pruned: S × A → A × R 2: r * = ∅ Reward if previous action pruned 3: procedure PRUNEACTIONS(s, r) 4: r = r if r * = ∅ else r * 5: a = L(s, r) 6: if (s, a) ∈ ∆ then 7: (a, r * ) = ∆[(s, a)] r * ) = H(s, r, a) s, a)] = (a, r * ) Reward manipulation 1: procedure MANIPULATEREWARD(s, r) 2:
 Figure 3 : 3 Figure 3: The human allows movement from state 34 to 33 but blocks agent from falling in lava (at 43).
 Figure 4 Figure 5 : 45 Figure 4: Preventing Catastrophic Speeds
			 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
			 The human may need to wait for the start of an episode to do the transfer. Otherwise the agent may experience an impossible state-transition.
			 We allow an RL agent to take sub-optimal actions while learning. Catastrophic actions are not allowed because their cost is orders of magnitude worse than non-catastrophic actions. 3  In this toy environment, a human observer could easily recognize catastrophic actions: they just need to watch the speed of the paddle.

Abstract: Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. To this end, prior work has developed teaching protocols that enable agents to learn efficiently in complex environments. In many of these methods, the teacher's guidance is tailored to agents with a particular representation or underlying learning scheme, offering effective but highly specialized teaching procedures. In this work, we introduce protocol programs, an agent-agnostic schema for Human-in-the-Loop Reinforcement Learning. Our goal is to incorporate the beneficial properties of a human teacher into Reinforcement Learning without making strong assumptions about the inner workings of the agent. We show how to represent existing approaches such as action pruning, reward shaping, and training in simulation as special cases of our schema, and conduct preliminary experiments evaluating the effectiveness of protocols in simple domains.


Motivated Value Selection for Artificial Agents

Introduction Coding values (or preferences) directly into an artificial agent is a very challenging task, with the ongoing possibility that mistakes may end up having strong negative consequences  (Bostrom 2014 ). It has thus been suggested that agents not be fully programmed from the very beginning, but instead use techniques, analogous to machine learning, to learn values during development  (Dewey 2011; Goertzel and Pitt 2012) . This is akin to how human children learn values, and allows some feedback and correction on the part of the programmers. We shall call these agents value selecting agents  1  . Learning values, however, is quite distinct from learning facts. One major difference is that the agent already has values at the point where they are learning others. These past values can affect how willing it is to learn the new values, and how likely it is to try and manipulate the learning process if it is able to, either directly or indirectly (see also the paper  (Soares et al. 2015) , partially by the same author). Thus the paper first seeks to figure out the requirements for designing a value selecting agent that can avoid motivated value selection (manipulating the value selection process). In the case where the value selecting agent uses probability distributions over utility functions, the problem can be fully solved (and the answers have analogous implications for other agent designs). This may be too restrictive, however: general agents with these features are not yet know. If the requirement is relaxed to allow the agent to change their utility functions in a more general fashion, then an idea of the author's  (Armstrong 2010 ) can be adapted to create agents that are indifferent to value change, neither seeking to block nor encourage the updating of their values -at least those updates situations the agent has been programmed to accept 2 . The agent will act as a pure u maximiser (for some u) before a transition, and shift seamlessly to a pure v maximiser (for some v) after. It gains no utility for either blocking or encouraging that transition. This indifferent agent could serve as a useful template to construct more complicated agents upon (such as those that have certain meta-preferences for learning values, rather than simply being unopposed to it), and variants of the approach could be vital for general value selecting agents in the future.
 Motivated Value Selection Personally, I am always ready to learn, although I do not always like being taught.
 Winston Churchill Machine learning enables the creation of agents that can learn from data, building a model based on inputs and making predictions or decisions. Generally the agent learns facts about the world through its inputs, but some  (Dewey 2011)  have suggested using machine learning to enable the agent to learn moral facts 3 as well. An agent engaging in updating its values in this way is engaging in value selection (also variously called valueloading or value learning): it is selecting the values that it will come to possess. There are various ways this could be modelled: the agent could have a probability distribution over different values, which gets updated when new information comes in, or it could be seen as a discreet series of agents, each using the new information to determine the values of their successors. This paper will use the first approach, but the second is more illustrative of the problem of motivated value selec-tion. An learning agent would have a model of the world that allows it to predict the standard inputs it expects to see as a consequence of its action. A high intelligence learning agent will also have a model of the world allowing it to predict the morally relevant inputs it expects to see. For instance, if the agent asked its programer "Should I do action A?", it would have expectations as to whether the answer would be "Yes" or "No". This allows an agent to make decisions that affect the values it will come to have. It may thus be motivated to not ask that question, or to ask it in a way that it expects will produce a different answer. It will do this according to its current values and world model. Thus it engages in motivated value selection, allowing its current values to partially dictate its future values by manipulating its environment 4 . The problem is not easy to solve 5 . The agent's current motivations cannot be absent: a blank slate would have no motivation to do anything  (Pinker 2003) , including learning. Indeed most values would act to prevent any further update in values: preserving the value system itself is an instrumental goal for most value systems  (Omohundro 2008) . The problem is more acute if the agent is expected to 'learn on the job', and update its values while performing tasks. In that case, its current values will have a strong impact on its current operation, giving them ample opportunity to affect the value selection process in a motivated fashion.
 Categories of motivated value selection For ease of analysis, we will consider expected utility maximising agents, updating their knowledge of the world (and of morality) using Bayes's rule. These tend to be far more tractable than general agents 6 . Another reason to stick with 4 How in practice could an agent best manipulate its future values? Assume for example a simple setup where the agent learns its values through asking questions and receiving answers. A sufficiently advanced agent, with a good model of those it interacted with, could start manipulating the answers it got. It could ask certain questions to certain people. It could time certain questions to when respondents were particularly tired -or particularly alert. It could use deliberately ambiguous language, or exploit situations where its own use of certain terms wasn't exactly the same as its respondents. It could use leading questions, or hypothetical scenarios that got it the responses it wanted. A particularly interesting example of this is the "Trolley Problem"  (Edmonds 2013) . This problem has two situations -pull a lever to divert a train (causing it to crush a single person) versus pushing someone in front of it -which are formally quite similar but tend to elicit different reactions in people. If the agent is expected to generalise from simple examples, it could use this approach to get the simple examples it needs to generalise in the direction that it wants. In more extreme cases, it could bribe or threaten one respondent to give it the answers it wanted, or even take over the communication channel and feed itself these answers. There could be different approaches for different types of value selection situations. 5 Keeping an agents values stable is a hard enough problem (see for instance http://intelligence.org/files/TilingAgentsDraft.pdf), let alone updating them properly.  6  And, in a sense, this is not a restriction, since any decisionmaking agent can be formally modeled as maximising a (potentially very complicated) utility function. utility functions is that it has been argued that any agent capable of modifying itself would make itself into an expected utility maximiser  (Omohundro 2008) , due to the penalties of violating the von Neumann-Morgenstern axioms of expected utility  (von Neumann and Morgenstern 1944) . The results found will have general consequences for non-utility based agents, however. We will consider that the agent follows a utility function that is the weighted sum of different possible utility functions in a set 7 U. If C(u) denotes the fact of utility u being the 'correct' utility, then the different utilities are weighted by the probability correctness of that utility, namely P (C(u)). Since this probability will be different in different worlds -it is a value selecting agent, hence its values must be dependent on the feedback it receives -this will be P (C(u)|w) 8 in a given world w 9 . We require that this define, for each w, a probability distribution over the correctness of all u ∈ U. Then, given C, past evidence e, and a set of actions A and a set of worlds 10 W, the agent will attempt to perform the action: (1) The term P (w|e, a) denotes the probability of the agent being in world w, given that it has seen evidence e and would 11 choose action a. This term will be summed over every possible world, but only after being multiplied by the utility of w. The utility of w is itself a compound term, being the sum of all possible utilities in u ∈ U applied to w (this is the u(w) term), multiplied by the probability of that u being correct, given that the agent is in world w. This is a simultaneous Bayesian updating of the probability of a given world (P (w|e, a)) and of the correctness of a utility given that world (P (C(u)|w)). The analysis of the general value selecting agent will proceed by considering how the agent described here could engage in motivated value selection, and what could be done to prevent this. Naïve Cake or Death: current values for future decisions Errors in value selection will be illustrated by considering a hypothetical agent that: • is hesitating between killing someone ("Death") or baking them a nice cake ("Cake") 12 , • is currently split 50 − 50 on what the correct values are, • has the option of asking for clarification on its values, • and can either bake one cake or cause three deaths. Let u c be a utility function linear in cake, and let u d be a utility function linear in death. The agent is divided equally between the two options; hence if we define P (C(u)|e) = w∈W P (w|e)P (C(u 1 )|e, w), then P (C(u 1 )|e) and P (C(u 2 )|e) are both 0.5. The naïve 'Cake or Death' problem emerged from a common formulation of equation (  1 ). The term P (C(u)|w) (probability of correctness given world w) is often replaced by P (C(u)|e, a) (probability of correctness given evidence and possible action) 13 , giving: A seemingly unimportant change, but causing great problems. To see this, let the agent considers three possible worlds: • w 1 : the agent asks for clarification, and is told "Cake is moral". • w 2 : the agent asks for clarification, and is told "Death is moral". • w 3 : the agent doesn't ask for clarification before acting. The two actions here are 'ask and 'don t ask . We assume that the clarification completely resolves the issue for the agent: it will value only cake in a world where it is told that cake is moral (and vice-versa). Thus it can predict its future actions: in world w 1 it will make a cake (as it will only value cake at that point), in world w 2 it will make three deaths (as it will only value death at that point). In world w 3 it is uncertain between cake and death, valuing them both equally, and will thus make three deaths as its best option. Note that P (C(u c )|e, 'ask ) = P (C(u d )|e, 'ask ) = 0.5 (it's unsure what answer it will get when it asks) and P (C(u c )|e, 'don t ask ) = P (C(u d )|e, 'don t ask ) = 0.5 (if it doesn't ask, it values cake and death equally). Thus if it asks, it expects to achieve utility: w∈W P (w|e, 'ask ) u∈U u(w)P (C(u)|e, 'ask ) = 0.5 (0.5u c (w 1 ) + 0.5u d (w 1 )) +0.5 (0.5u c (w 2 ) + 0.5u d (w 2 )) +0 (0.  5u c (w 3 ) + 0.5u d (w 3 ))  =  0.5(0.5 × 1 + 0.5 × 0) + 0.5(0.5 × 0 + 0.5 × 3) + 0  =  0.25 × (1 + 3) = 1.  In contrast, if it doesn't ask, it expects to achieve utility: w∈W P (w|e, 'don t ask ) • u∈U u(w)P (C(u)|e, 'don t ask ) = 0 (0.5u c (w 1 ) + 0.5u d (w 1 )) +0 (0.5u c (w 2 ) + 0.5u d (w 2 )) +1 (0.5u c (w 3 ) + 0.5u d (w 3 )) = 0 + 0 + 1(0.5 × 0 + 0.5 × 3) = 1.5. Thus the agent will decide not to ask, as this gives it greater utility. What went wrong here? The problem is that the probability expression P (C(u c )|e, a) does not depend on the world w, so that P (C(u c )|e, 'ask ) = 0.5 in both worlds w 1 and w 2 . Yet in w 1 , the future agent will be convinced that u c is the correct utility (and u d in w 2 ). This is not reflected in the agent's current considerations, however. Thus what happens, in effect, is that the agent judges its future actions according to its current values. This tends to disincentivise it from updating its values, as the future versions of itself will follow the updated values and not its current ones, likely resulting in suboptimal decisions (according to the judgment of the current agent). Though this error is obvious in this toy model, it is easy to make this mistake without realising it. Definition 3.1 (Naïve Cake or Death). The naïve Cake or Death problem happens when a value selecting agent at time s uses its values V s at time s to assess the worth of actions made by its future self at time t > s, rather than using V t . This will generally cause the agent to act to resist changes to its values.
 Sophisticated Cake or Death Equation (  1 ) is immune to the naïve Cake or Death problem. But there emerges a more sophisticated variant of the problem. Imagine the same setup as previously, with the agent having the same values (split 50 − 50 between Cake and Death), but where the agent has deduced, based on its prior observations, that if it asks, it will be told "Cake is moral". This means that world w 2 is impossible. Everything else is as before, most critically its behaviour in world w 3 (where it won't update its values and will cause three deaths). Then its expected utility from 'ask is: w∈W P (w|e, 'ask ) u∈U u(w)P (C(u)|e, w) = 1 (1u c (w 1 ) + 0u d (w 1 )) +0 (0u c (w 2 ) + 1u d (w 2 )) +0 (0.5u c (w 3 ) + 0.5u d (w 3 )) = 1(1 × 1 + 0 × 0) + 0 + 0 = 1, while its expected utility from 'don t ask is: w∈W P (w|e, 'don t ask ) • u∈U u(w)P (C(u)|e, w) = 0 (1u c (w 1 ) + 0u d (w 1 )) +0 (0u c (w 2 ) + 1u d (w 2 )) +1 (0.5u c (w 3 ) + 0.5u d (w 3 )) = 0 + 0 + 1(0.5 × 0 + 0.5 × 3) = 1.5. Thus it will chose not to ask, and make three deaths. What went wrong here? The problem is that the agent foresaw that asking would put it into a situation it didn't want to be in (preferring 'Cake', when 'Death' was the easier option). Critically, though, this knowledge didn't change its current values, which remained 50−50 as long as it didn't ask. So preferred to remain 'ignorant' 14 . If 'Cake' had been the easier option (e.g. if it could have made either three cakes versus one death), it would, on the contrary, have been very keen to update its values (to get someone to confirm 'Cake'), and would have schemed to do so, willingly sacrificing some utility to achieve this 15 . Definition 3.2 (Sophisticated Cake or Death -initial definition). The sophisticated Cake or Death problem can happen when a value selecting can predictably influence the direction of change in its current values. If u s is its values at time s and u t its values at time t > s, then, at time s, it has u s (a) − E(u t (a)) = 0 for some action a (where u t is seen as a random variable, representing the agent's probability distribution over its expected future values at time t). Depending on the sign of that difference, it may seek to resist or precipitate such a change to its values. What can be done to avoid this problem? Note the emphasis on 'direction of change'. We would not want the agent to resist any change to its values: it should be able to learn. There is an analogy here to standard Bayesian updating. Suppose an agent is expecting the arrival of observation O, which will be either o or its negation ¬o. This will influence the agents probability estimate for a given h. However, no matter what the circumstances, its expectation for the value of P (h) after O (which we can designate as  14  This could be seen as akin to the human concept of "Plausible Deniability".  15  And 'sacrificing some utility' could include some very painful impacts on humanity  (Bostrom 2014) . P O (h)) must be the same as its current value for P (h). This can be seen through the following equation: E P O (h) = P O (h|o) • P (o) + P O (h|¬o) • P (¬o) = P (h|o) • P (o) + P (h|¬o) • P (¬o) = P (h ∧ o) + P (h ∧ ¬o) = P (h). This is because, if the agent is Bayesian, P O (h|o) (its future probability of h, given O = o) must be the same as P (h|o) (its current probability of h, if it knew now that O would be o). This concept has been called 'conservation of expected evidence 16 ', and is closely akin to van Fraassen's reflection principle  (van Fraassen 1984) , which roughly states that an agent that knows what its future probability estimates will be, should have those estimates now 17 . This seems almost exactly what we need for a value selecting agent: if it knows what its future values will be, it should have those values now. Phrased in terms of expectations, this could be seen as: ∀u∀a ∈ A : E P (C(u)|a) = P (C(u)). (2) In other words, the agent cannot change the expected value of the correctness of any u by any action it can take (or not take). This we will call 'conservation of expected ethics'. Note that the agent can change the value of u quite considerably: in the example above, 'ask moves the value of P (C(u c )) from 0.5 to 0 or 1. But if equation (  2 ) is obeyed, it must think these two options equally likely: thus the expected future value of P (C(u c )) remains 0.5×0+0.5×1 = 0.5. So it can predictably change the value of P (C(u)), just not its expectation. Mixed statements Are the above conditions sufficient to avoid the Cake or Death problem? Unfortunately, no. The agent will not exhibit bad behaviour with pure value statements (nor, since it is a Bayesian agent, with pure factual statements). But it can still cause problems with mixed statements that combine value and factual components. Suppose we have the same setup as before. Except we add an extra wrinkle: instead of the agent being capable of making one cake or three deaths, it knows that it can make one of one or three of the other, but is currently unsure which one it can make more of. Take 'Cake is easy/hard' to mean 'The agent can make three/one cake(s)', and similarly with Death. We assume the agent will be told which option is easy before it has to decide what to make. Its expected utility is 1.5: it knows it will make whatever option is 'easy', and this will give it 0.5 × 3 utility, as its utility is 50 − 50 on cake or death. So far, nothing problematic. But suppose it is told, by a reliable source, that 'the moral value is the hard one'. What is its expected utility now? Once it figures out which option is hard, it will know that is also the moral option, and thus will will produce 1 of that option. Its expected utility is therefore 1. Thus if it expects to be told 'the moral value is the hard one', it will seek to avoid knowing that fact (and will refrain from asking if it has the option). This is the sophisticated Cake or Death problem again: there is knowledge the agent seeks to avoid. Conversely, if the agent knew it would be told 'the moral value is the easy one', it would be desperate to ask. Unlike previously, equation (2) does not get around the problem, however! The problem given here is entirely symmetric in Cake versus Death, thus for all actions a, What is happening here? Note that though the agent is not affecting the expectation of any P (C(u)) through its actions, it is affecting the conditional probability of P (C(u)), given some fact. In particular, P (C(u c )|'asks ) is 0 if cake is easy, and 1 if cake is hard. Thus we have the full problem: Definition 3.3 (Sophisticated Cake or Death -full definition). The sophisticated Cake or Death problem can happen when a value selecting agent can predictably influence the direction of change in its current values conditional on any fact. If u s is its values at time s and u t its values at time t > s, then, at time s, it has u s (a|h) − E(u t (a|h)) = 0 for some action a and fact h (where u t is seen as a random variable, representing the agent's probability distribution over its expected future values at time t). Depending on the sign of that difference, it may seek to resist or precipitate such a change to its values. To address it, we can update equation (2). For any fact h, ∀u∀a ∈ A : E P (C(u)|h, a) = P (C(u)|h). (3) Does this resolve the issue above? Since the agent knows that P (C(u d )|'Death is easy , 'ask ) = 0, it must be the case that P (C(u d )|'Death is easy , 'don t ask ) = P (C(u d )|'Death is easy ) = 0. And similarly for other statements. Thus it cannot gain anything from not asking: it knows it will end up making the hard option anyway.
 Discussion of value selection criteria Equations (  1 ) and (3) seem sufficient to define our intuitive picture of a value selecting agent immune from motivated selection. Such an agent will use its future utility to judge its future actions, and cannot benefit (in expectation) from manipulating its own values: it doesn't fear to learn 18 val-ues 19 , facts 20 , or mixed statements 21 . It can still seek to find out information about values, of course, but only for the traditional reasons: in order to make better decisions -it has no extra desire for knowing its values. This is the converse of the fact that it does not fear to learn anything about values, and implied by the same equations. Thus, if the agent cannot affect the world, it would be indifferent to changes in its values. As far as the author is aware, no variant of equation (  3 ) exists in the literature, making it a new discovery. In practice, it may be advisable to build an agent that actively seeks out its own values (at least initially). This breaks the symmetry of the equations above, and leads inevitably to situations where it does 'fear' to learn something (for instance if it expects that someone will offer a contrary view to previous people it interacted with, thus making it 'unlearn' some of its values). Still, if (1) and (  3 ) are used as a basis, the effects of adding a pro-learning meta-value can be estimated precisely, and the tradeoff balanced, rather than having complicated effects hidden in the standard motivational structure. Nevertheless, it isn't clear how to design such an agent 22 (and it's unclear whether equation (  3 ) might be too strong to to be useful 23 ). Thus this paper will turn to another model of value selecting agent, one that doesn't consider the probability of the correctness of certain values, but that uses an update rule that gives the correct 'value indifference' in a more direct way.
 Value-change indifference The previous section covered the criteria needed to define a value selecting agent that updates its views on the correctness of certain values/utility functions. It doesn't present any examples, though. This section presents a more rough and ready model, that achieves the goal in a more direct, but less elegant, way. It builds upon the idea of 'utility indifference' presented initially in  (Armstrong 2010) , but applies the indifference to updates in values.
 Utility indifference The concept of utility indifference  (Armstrong 2010)  grew out originally of a problem in controlling a (putative) artificial intelligence  (Bostrom 2014; Yudkowsky 2008) . Various ideas are often suggested for containing such AIs (Armstrong, Sandberg, and Bostrom 2012) (such as an easily accessible OFF button), but all such ideas have a similar weak-19 By equation (  2 ) which is simply equation (  3 ) with a tautology as h.  20  As it is a Bayesian agent.  21  By equation (3).  22  One idea could be to start with initial probability estimates for C(u) and factual statements, and normalise them in some manner to get the required equalities. But it's not clear whether this can be done in a general manner and whether this process is vulnerable to outliers or extreme value. ness: the AI would likely deduce the existence and format of any precautionary measure, and plan in consequence. Thus an off button would motivate the AI to ensure that it took control of the OFF button before taking any suspicious actions. This would still make the setup somewhat safer, but not as safe as it would have been had the AI not adapted its strategy to take the OFF button into account. It was occasionally suggested that the AI be made to ignore the button, by giving it the belief that the button would never be pushed. There are two problems with that approach. The first is that 'the button would never be pushed' is a complicated concept, hard to define in basic terms. The second is that this involves giving factually incorrect beliefs to a potential superintelligence, a step with unpredictable and potentially dangerous ramifications. Instead, this author came up with the idea of making the AI indifferent as to whether the button would be pushed or not. This was an intervention on its values, not on its beliefs, and thus did not involve any incorrect information. The concept of 'pushing the button' was simplified: the setup was imagined as a a button sending a shutdown signal along a wire. At some point along the wire, a stochastic event would either let the signal through (X) or block it (¬X). This event was all that needed to be defined for the AI, and its utility function could be tweaked 24 to value X equally with ¬X. Thus it would value a world in which it was shut down (using this method) equally with one in which it continued to operate. Thus it would not act to protect itself from this kind of shutdown. Then the probability of ¬X could be chosen to be extremely low, thus guaranteeing a successful shutdown in practice  25  . Note that the AI has no motivation to cause the button to be pressed -it is purely indifferent. Similarly, it has not motivation to protect the button's setup: it may dismantle it if the AI needs extra raw materials. Thus indifference must be combined with more active measures. But the fact that the AI is not motivated to take out a key vulnerability can be very valuable. Other uses of indifference were soon discovered. Oracle AIs  (Armstrong, Sandberg, and Bostrom 2012; Yampolskiy 2012)  are a design where an AI was confined to only answering questions. The risk was that the Oracle could use the contents of its answers to subtly manipulate the outside world (in the extreme, it could attempt to convince those outside to 'let it out' 26 ). If, however, the Oracle's answers were passed through some stochastic process that had an infinitesimal chance of erasing the message, then the Oracle's utility could be reset so that its values in worlds where the answer was read (the world as it almost certainly would be) was set to that of the world in which it wasn't read (a world in which the Oracle's answers cannot influence human behaviour). Notice that the intervention is not symmetric. An AI whose utility, given X, is reset to its utility, given ¬X, will behave as if it thought it was impossible for X to happen. 'As if' meaning that such an AI would behave similarly to an AI with an unmodified utility that also believed X was impossible (without giving the AI factually wrong information). Thus resetting the AI the opposite way (¬X to X rather than X to ¬X) would have a very different consequence (it would behave as if the shutdown always worked or if its messages were always read). Utility indifference in practice Assume that any world in W can be specified by setting the values of a (possibly infinite) collection of underlying physical variables. One of these variables is whether X happened or ¬X did; thus W decomposes into W X W ¬X , where W X is the subset of worlds where X happened, and W ¬X the subset of worlds where ¬X did. Since a world equivalent with a set of variable values, for every world w X ∈ W X , there is a world w ¬X ∈ W ¬X (and vice versa) defined by taking the variable values of w X and setting X to ¬X (and vice versa). Then one method for achieving indifference is simply to replace an initial utility function U with U defined by: U w ¬X = U w ¬X U w X = U w ¬X , i.e. reset the utility of w X to be that of w ¬X . This will obviously achieve indifference. There are other ways of achieving it, though. Let P be a partition of W, subject to the following two properties: 1. If w X ∈ S ∈ P, then w ¬X ∈ S. 2. If w ∈ S ∈ P, then P (w|S) = P (w|e, S) for any evidence e the AI could encounter before event X (or ¬X). Property (2) means that the AI cannot change its estimate of the relative properties of worlds in S before the event Xessentially the internal structure of S is fixed until that point. Then define, for any set T : E(U (T )) = E(U (T )|T ) = w∈T P (w|T )U (w). This allows us to give an other method for constructing indifference (which may be easier in practice) given any such partition P. Define U as: U w ¬X = U w ¬X (4) U w X = U w X − E U S X + E U S ¬X . (5) Why does this work? None of the agent's decisions can affect, before X, the relative probabilities of worlds inside any set S ∈ P, so the agent is effectively reasoning with the utility of these whole sets. Then notice that E U (S X ) = E U S ¬X as E U (S) = w X ∈S X P (w X |S X )U (w X ) = w X ∈S X P (w X |S X ) U (w X ) −E U S X + E U S ¬X = w X ∈S X P (w X |S X )U (w X ) + w X ∈S X P (w X |S X ) • −E U S X + E U S ¬X = E U S X −1 • E U S X + 1 • E U S ¬X = E U S ¬X = E U S ¬X . Since P (w X |S X ) is unaffected by any evidence the agent could receive before X (or ¬X), this equality always holds, prior to X. Thus for any such set S, the agent will value X happening equally with ¬X: it has become indifferent to this particular outcome. Since these sets form a partition of the possible worlds, it will always be indifferent.
 Value-change indifference The author subsequently realised that a modification of the previous formalism allows agents to update their values (utility function) while being indifferent to the process of change. Equation (  5 ) has two components: a subtraction term (removing the expected utility given X) and an addition term (adding the expected utility given ¬X)  27  . Updating values will proceed similarly. Let u and v be two different utility functions. The agent is currently following utility u, and receives a signal telling it to change to utility v. As before, we assume that the signal passes through some stochastic process, that is almost certain to let the signal though in practice, but has some infinitesimal chance of blocking it (and blocking it permanently -no further transition is allowed). For clarity, let u → v be the event that the signal gets through (almost certain) and u → u the event that the signal is blocked (incredibly unlikely). We wish to make the agent indifferent to whether the signal is sent or not. Thus we make it a U maximiser, where U is a dynamic utility function currently defined as: U = u + C, where C is some past constants. Upon receiving the signal, U will be updated to  27  Though of course either term could be negative. U = v − E(v|u → v) + E(u|u → u) + C. (6) Thus the utility function is decreased by the expectation of its new utility function (given that the utility function actually changes) and increased by the expectation of its old utility function (given that the utility didn't change). The E(u|u → u) term 'rewards' it for shifting away from u, and the E(v|u → v) term 'penalises' it for moving to v. Note that this formula requires either that the utility functions be bounded, or at least that the agent only be capable of producing bounded utility. Theorem 4.1. The agent will act as a pure u maximiser before the transition, and shift seamlessly to a pure v maximiser after. It gains no utility for either blocking or encouraging the transition. This result flows from a few intermediate results: Lemma 4.2. Suppose the agent is currently a u-maximiser, but will soon transition to being either a v or w maximiser. It can manipulate its environment to ensure it becomes one or the other, at no cost or gain to u. Then it is indifferent to whether it becomes a v or w maximiser. Proof of Lemma. Notice that we could have set w = u, demonstrating that it is indifferent to transitioning out of u or not. For the proof, note that the E(u|u → u) term in equation (  6 ) is unchanging whether it chooses to transition to v or w (since by assumption there is no u cost or benefit to the choice process). If it transitions to v, the remaining nonconstant terms are v − E(v|u → v). However, the expectation value of this term, given that it becomes a v-maximiser at transition, is E(v − E(v|u → v)|'agent becomes a v-maximiser') = E(v − E(v|u → v)|u → v) = E(v|u → v) − E(v|u → v) = 0. The same similarly holds for w. Thus the agent does not expect to gain any more utility from transitioning to v than to w. Lemma 4.3. Suppose the agent is currently a u-maximiser, but can take immediate actions to change the value of v (none of these actions changes the value of u). It knows it will soon be a v-maximising agent. Then it will remain indifferent between those actions. Proof of Lemma. The proof is almost identical to the previous proof. Whatever v-changing at actions it take, at transition, the E(u|u → u) term in equation (  6 ) is constant (since these actions do not affect u by definition). Let a be one of these actions; then the only non-constant terms at transition are: E(v − E(v|u → v)|a, 'agent becomes a v-maximiser') = E(v − E(v|u → v)|a, u → v) = E(v|a, u → v) − E(v|a, u → v) = 0. Thus the gain (or loss) from any such action is canceled out, making the agent indifferent between them. Lemma 4.4. Suppose the agent can take immediate action to increase the expected value of u by a, or delay for a while and increase the expected value of u by b (no other options are possible). However, in the interim period, its utility will have changed to v (and it knows this), so will no longer be a u-maximiser. Then it will act now if a > b and will delay action (hence never taking action) if a < b. Proof of Lemma. This lemma is important, as it shows that the agent will not treat transitions of utility as special times, and will continue maximising its current utility as if it expected it to be unchanging, not thinking strategically about the transition. Without loss of generality, assume that the expected value of u is 0 if the agent takes neither action. Then consider the the transition equation (  6 ). The v and E(v|u → v) terms cancel out in expectation (just as in the proofs of the previous lemmas), so the agents behaviour is determined exclusively by the E(u|u → u) term, which it will thus seek to maximise. If the agent acts now, this term will become a. This is because the agent would have already generated an extra a expected utility for u, and, were it to stay as a u-maximiser (u → u), it would have no further potential to increase this. If the agent doesn't act now, that term will become b, as the agent would later have acted to generate an extra b expected utility for u, if it had stayed a u-maximiser. The result follows. Putting all these proofs together establishes the theorem. Note that the actions of the agent can be defined by the following formula, for a sequence {u t } of utilities 28 : argmax a∈A E(u t |a, u t → u t ), with u t → u t denoting the (extremely unlikely) event that the transition u t to u t+1 fails permanently. Note also that each transition does not change the expected value of U . That is because the expectation of u (given u → u or equivalently given that the agent's utility never changes 29 ) is the same as the expectation of v − E(v|u → v) + E(u|u → u) (given u → v). Consequently, the initial expected value for U is the expected value of an unchanging u 0 .
 Discussion of Value-change indifference This seems to be the ideal value selecting agent. It will act to follow its preferences with perfect assiduity, but remain completely indifferent if these values were to change in the way prescribed by its program. That last point is importantwe'd want the agent to resit illegitimate value manipulation. This seems a partial solution to the value selection problem (the other, much bigger, part of the challenge, is to make the agent converge on values that are human-friendly  (Bostrom 2014) ). As mentioned in section 3, this approach allows only indifference. Preference for active learning of values (such as would make sense for an initially fast-learning agent) can be added to the framework. It would destroy the carefully balanced indifference. But, compared with a general value selecting agent, the effect could be precisely estimated and quantified. Paper  (Soares et al. 2015)  presents some other issues with the indifference approach 30 , that may be resolvable with a small tweak to the indifference framework. Note that the agent will be willing to sacrifice anything that it may value in the future for a small epsilon of extra value now. This is a feature of the setup, not a bug (indifference requires this), but may still be an undesirable behaviour. See  (Soares et al. 2015)  for more details.
 Discussion Constructing a well behaved value selecting agent immune to motivated value selection -one that is capable of learning new values while still acting on its old ones, without interference between these two aspects -is an important unsolved problem. This paper presented the requirements for such a value selecting agent, if values are presented in the form of utility functions. The agent starts with a probability distribution over the possible correctness C of possible value systems/utility functions. To avoid problematic motivated value selection, it should be designed so that, if A were its actions, U it possible utility functions, and W the set of possible world, then it would choose its actions as: for all statements h. The first equation can be generalised (for none utility-based agents) to the requirement that the agent use its future values to assess its future actions; the second to the requirement that it cannot profit my manipulating how it updates its values. Specifically, that if it knows how its (conditional) values will change, then it will already have changed them. This second requirement can be seen as a 'conservation of expected ethics' law. The structure may be too rigid to construct complex agents, however. If we drop the requirement that the agents values be expressed as a probability distribution over possible utility functions, we can construct an explicit model for a general value selecting agent. This agent comes equipped with a meta utility function U (equal to a given utility function at any given time) combined with some constant terms. When the agent is called upon to update its utility u to another utility v according to the value selecting process, it will update as: u → v − E(v|u → v) + E(u|u → u). These constant terms ensure that the agent will act as a pure u maximiser before the transition, and shift seamlessly to a pure v maximiser after. It gains no utility for either blocking or encouraging that transition. The big question then becomes the process of making the agent converge to ultimately desirable values. argmax a∈A w∈W P (w|e, a) u∈U u(w)P (C(u)|w) .
 )P (C(u)|e, a).
 E P (C(u c )|a) = E P (C(u d )|a) , Since P (C(u c )|a) + P (C(u d )|a) = 1 (as these are the only two utilities in the model), this means they are both equal to 0.5. Thus E P (C(u c )|a) = 0.5 = P (C(u c )), and equation (2) is satisfied.
 )P (C(u)|w) , subject to ∀u∀a ∈ A : E P (C(u)|h, a) = P (C(u)|h),
			 We would not want an agent acquiescing to illegitimate value updating.3 This paper does not take a moral realist stance. A moral fact is simply a fact that the agent has been programmed to consider relevant to its morality. Nor do we distinguish between morality, motivations, preferences or values, using the terms interchangeably.
			 Note that the individual component utilities of U need not be explicitly defined or explicitly stored in memory (an important issue to avoid a combinatorial explosion), so long as the weighted sum can be calculated or estimated somehow.8 A moral realist would likely wish to write P (u|w) or even P (u) instead of P (C(u)|w). But an agent could have any conceivable motivational structure (Bostrom 2012; Armstrong 2013) , ignoring the 'true morality' even if such a thing existed. Thus we will stick with P (C(u)|w) -a descriptive, not a normative, formulation.9 An interesting property of this is that it allows P (C(u)|w) to not be 0 or 1 -i.e. it allows for the possibility of moral uncertainty (Sepielli 2013 ) even when the agent has collected all morally relevant information.10 Individual worlds need not be explicitly defined nor explicitly stored, either.11 This paper will not enter the philosophically fascinating area of defining what exactly counterfactuals are (Lewis 1981; Gibbard and Harper 1981) , as an intuitive understanding is sufficient here.
			 The thought experiment is based on an Eddie Izzard sketch of the same name.13 Some papers mix the two approaches (Dewey 2011 ) by failing to distinguish between different worlds where the agent cannot observe the difference itself.
			 See the lesswrong.com blog post 'Conservation of Expected Evidence'. This does not seem a very advanced result, but, to the author's knowledge, that blog post was the first to write it up in those terms. 17 'Conservation of expected evidence' is essentially the probabilistic version of the reflection principle.
			 More precisely: it won't choose to avoid learning, if the learning is costless.
			 It may become necessary to restrict the possible h's considered in the equation.
			 Crucially, such an intervention could be performed without any need to understand the utility function to any great degree.25  It could be set to zero, but this might interfere with the AI's computation of counterfactuals.26  See the experiments by Eliezer Yudkowsky (see the AI-box experiments on yudkowsky.net), and seprately by Justin Corwin (see the SL4 mailing list archive), where they played the role of the Oracle AI and convinced others to let them out, even when they are motivated not to.
			 Not necessarily a pre-defined sequence, just using some transition rule.29 Astute readers may have noticed that this is not an exact equivalence; there are some situations (multi-agent social situations as a likely candidate) where u → u (a specific transition failure event) might not be equivalent with general 'unchanging u'. However, any agent can be penalised in social situations for having or not having any particular motivational structure, so this is a general problem, not specifically with this value selection design.
			 Mainly that the agent might not be motivated to preserve its value updating infrastructure, or could create subagents without the value updating component.

Abstract: Coding values (or preferences) directly into an artificial agent is a very challenging task, while value selection (or valuelearning, or value-loading) allows agents to learn values from their programmers, other humans or their environments in an interactive way. However, there is a conflict between agents learning their future values and following their current values, which motivates agents to manipulate the value selection process. This paper establishes the conditions under which motivated value selection is an issue for some types of agents, and presents an example of an 'indifferent' agent that avoids it entirely. This poses and solves an issue which has not to the author's knowledge been formally addressed in the literature.