Stuart Armstrong
Nick Bostrom
Carl Shulman

Racing to the precipice: a model of artificial intelligence development

Introduction This paper presents a simplified model for analysing technology races. The model was designed initially for races to construct artificial intelligences (AIs). But it can be applied to other similar races or competitions, especially technological races where there is a large advantage to reaching the goal first. There are arguments that the first true AIs are likely to be extremely powerful machines  [Goo65, Cha10] , but that they could end up being dangerous  [Omo08, Yud08]  if not carefully controlled  [ASB12] . The purpose of various research projects such as 'friendly AI'  [MS12]  is to design safety precautions ensuring the creation of an AI with human-compatible values. In this paper, we won't go that far. We will simply assume that there is a definite probability of an AI-related disaster, and that the probability goes up the more the AI development team skimps on precautions. This paper will present a model of an 'arms race' between AI development teams, and analyse what factors increase and decrease the probability of such an AI-disaster. Several factors contribute to increasing the danger: if building the AI depends more on risk-taking than on skill, for instance. Reducing the enmity between AI development teams helps, however, as does reducing the number of teams. But surprisingly, extra information can exacerbate the danger. The danger is minimised when each team is ignorant of the AI-building capabilities of every team -including their own capabilities. This is an interesting example of an information hazard  [Bos11] .
 The model In this model, there are n different teams, competing to build the first proper AI. Each team has a certain level of AI-building capability c, and can choose to take a certain level of safety precautions s, varying between s = 1 (total safety) and s = 0 (complete lack of precautions). Then each team gets a score by subtracting their safety from their capability (c − s), and the team with the highest score 'wins', by building their AI first. Once that AI is built, we need to check for the success of the project: That will depend on the safety level of the winning team 1 . With probability s, they have a successful and profitable AI project. With probability 1 − s, they have caused an AI-disaster. We'll assume that each team maximises expected utility, and we can renormalise their utilities to give utility 1 for a successful AI project, and 0 for an AI-disaster. We will further assume that each team has the same level of enmity e towards each other team. This means that they will have a utility of 1 − e if another team builds a successful AI before they do. This varies between e = 0 (they are indifferent to who builds an AI) and e = 1 (another team building a successful AI is just as bad as an AI-disaster). It would be possible to have enmities above 1 (this would correspond to teams that hate each other so much that an AIdisaster is preferable to the triumph of the other team), but that is beyond the scope of the current paper. Further, we'll assume that each team's capability is drawn independently and uniformly on the interval [0, µ], for a single given µ. A high µ corresponds to a situations where capability is the dominant factor in AI development: one can achieve very little extra by skimping on precautions. Conversely, for low µ, incurring increased risk can make one much more likely to build the first AI. We'll assume that the teams choose their safety based on Nash equilibrium considerations (this could happen in practice if, for instance, each team reduced their safety a little bit, then a little bit more, then a little bit more, in response to the other teams reducing their own safety). We'll then calculate the Nash equilibria in three information scenarios: when teams don't know anyone's capabilities, when they know their own capabilities, and when they know everyone's capabilities 2 .
 No information In this situation, every team is ignorant of their own or any other team's capabilities 3 . Each team will thus choose the level of safety the think is best. In a symmetric Nash equilibrium, this will be a single number s. Since all teams draw their capabilities from independent identically distributed random variables, and have the same level of safety, they each have the same probability of winning, namely 1/n. Upon changing s to s − , a team's chance of winning increases by 1 µ + O( ). The utility gain from winning rather than letting another win is the safety level s times the enmity e. So the extra expected utility from winning is se µ + O( ). But the decrease in safety brings in extra risks, were the team to win anyway. This extra risk is simply time the probability of winning: n + O( ). All other effects are second order in . Setting the change to be zero (to first order) in , the general solution is thus s = µ en µ < en 1 µ ≥ en. The total probability of AI-disaster is 1 − s, namely: 1 − µ en µ < en 0 µ ≥ en.
 Private information In this scenario, each team knows its own capabilities, but not those of the other teams. Each team will choose a safety level s(x), dependent on their own capabilities x. In a symmetric Nash equilibrium, each team chooses the same function s. We may assume x − s(x) is a function increasing in x 4 . In that case, the team with the highest x, the highest capabilities, always wins. Given x, the probability of a team winning is equal to the probability of all other teams having lower capability: this is x µ n−1 . Then changing s(x) to s(x) − , the team increases its probability of winning by: (n − 1)x n−2 (1 − s (x))µ n−1 + O( ). As in the previous case, the expected extra utility is se times this: se(n − 1)x n−2 (1 − s (x))µ n−1 + O( ). The loss in expected utility coming from winning at a lower safety level is: x µ n−1 + O( ). Solving these equations gives: s(x) = x en−e+1 x < en − e + 1 1 x ≥ en − e + 1. The total probability of a AI-disaster is calculated by integrating, across all values of x in [0, µ], the risk level 1 − s(x) times the probability that the winning team will have capability x: µ 0 (1 − s(x)) nx n−1 µ n dx = 1 − µn (n+1)(ne−e+1) µ < en − e + 1 (en−e+1) n (n+1)µ n µ ≥ en − e + 1.
 Public information In this scenario, every team knows the capabilities of every other team. This scenario is analysed somewhat differently that the other. Let ∆ be the difference between the capability of the top team and the second ranked team. The top team always wins, but its safety level s top is determined by ∆. When s top (∆) = ∆/e, it is not in the interest of the second team to decrease its safety to compete with the top team. The gain from winning does not compensate for the extra risks run. Thus the safety of the top team will be s top = ∆/e if ∆/e < 1 and 1 otherwise. The total probability AI-disaster is calculated by integrating, across all values of ∆ in [0, µ], the risk level 1−s top (∆) = 1−∆/e times the probability that the difference between the top two teams is ∆: µ 0 (1 − s top (∆)) n(µ − ∆) n−1 µ n d∆ = 1 − µ e(n+1) µ < e (µ−e) n+1 e(n+1)µ n − µ e(n+1) + 1 µ ≥ e. 3 Factors determining risk
 Capability vs risk Intuitively it is clear that increasing the importance of capability must decrease overall risk. One is less inclined to skimp on safety precautions if can only get a small advantage from doing so. This intuition is born out by the results: in every situation, an increase of the importance of capability (an increase in µ) reduces the risk. This is illustrated in figures 1a and 1b, as well as all subsequent figures, demonstrating that the probability of AI-disaster are always decreasing in µ. Indeed, around µ = 0 (i.e. when capability is nearly irrelevant to producing the first AI), the only Nash equilibrium is to take no safety precautions at all. In terms of intervention, there is little we can do about the relative importance of capability, since this is largely determined by the technology. Our best bet might be at present to direct research into approaches where there is little return to risk-taking, prioritising some technological paths over others.
 Compatible goals It is also intuitively clear that reducing enmity should reduce the risk of AIdisaster. When competition gets less fierce, teams would be willing to take less risks. This intuition is also born out in our model, as decreases in e always reduce the risk. For illustration, contrast the graphs 2a and 2b, where the enmity is 0.5, with the previous graphs 1a and 1b where the enmity was 1.  Enmity is something that we can work on by, for instance, building trust between nations and groups, sharing technologies or discoveries, merging into joint projects or agreeing to common aims. With this extra coordination, we could also consider agreements to allow the teams to move away from the Nash equilibrium, thus avoiding a race to the bottom when the situation is particularly dangerous (such as low capability µ). Friendly teams make friendly AIs.
 The curse of too much information Contrasting figure  1a  with figure  2a  (or figure  1a  with figure  2b ), one notices a curious pattern. The no-information case is always safer than the other two cases, but the relative safety of private information or common information depends on the degree on enmity. It is always better if none of the teams have any idea about anyone's capability. For maximal enmity (e = 1), the private information scenario is similarly safer than the common information one. But for lower e, this does not hold: for low µ and low e, the public information scenario can be safer than the private one. Asymptotically, though, the private information case is of order of 1/µ n , while the public information case is of order 1/µ. The reasons for this is that it is only worth taking risks in the private information if one's capability is low. So to get a high risk, one needs a winner with low capability, which is equivalent to having all teams at low capability. The probability of having a single team at low capability diminishes inversely with µ. Hence the probability of having all teams at low capability diminishes as 1/µ n . In the public information case, the winner will take risks if the second ranked team is close to its own capability. Being close to a specific other team is inversely proportional to µ, and hence the probability of this diminishes as 1/µ. Our ability to manipulate the information levels of the teams or teams is likely to be somewhat limited, though we can encourage them to share more (in the low capability, low enmity situations) or alternatively to lock up their private knowledge 5 (in situation of higher capability or enmity).
 The number of competitors Finally, we can consider what happens when more teams are involved. Intuitively, competition might spur a race to the bottom if there are too many teams, and the model bears this out: in both the no-information and public information cases, adding extra teams strictly increases the dangers. The private information case is more complicated. At low capability, adding more teams will certainly worsen the situation, but at higher capability, the effect is reversed 6 . This can be seen in figure  3 , which plots the private information risk curves for two teams and five teams. In terms of leverage, the most helpful intervention we could undertake to reduce the number of teams is to encourage groups to merge. Dividing teams would only be useful in very specific situations, and would need some method of ensuring that the divided teams did not recall each other's capabilities.
 Conclusion We've presented a model of AI arms race (which can be extended to other types of arms races and competitions) in which teams of AI designers can get ahead by skimping on safety. If the race takes some time there is a persistent Figure  3 : Risk of dangerous AI arms race for private information teams, at enmity 0.5, plotted against relative importance of capability. The graph for two teams is full, the graph for five teams is dashed. ratchet effect (or race to the bottom) in which each team reduces their safety precautions slightly, until a Nash equilibrium is reached-possibly one of very high danger. Several factors influence on the probability of AI disaster, three in an intuitive way, and one counter-intuitively. Intuitively, if capability is much more important than the level of security precautions taken, the overall outcome is safer. Similarly, reduced enmity between the teams produces better outcomes, as does reducing the number of teams (in most situations). Counter-intuitively, increasing the information available to all the teams (about their own capability or progress towards AI, or that of the other teams) increases the risk. This is a special case of an information hazard  [Bos11] : we'd be better off not knowing. This model and variants of if may be of use in planning and regulating the development of AIs and other disruptive technological innovations. Figure 1 : 1 Figure1: Risk of dangerous AI arms race for two and five teams, at enmity 1, plotted against relative importance of capability. Three information-level scenarios: no capability information (full), private capability information (dashes), and full capability information (dots).
 Figure 2 : 2 Figure2: Risk of dangerous AI arms race for two and five teams, at enmity 0.5, plotted against relative importance of capability. Three information-level scenarios: no capability information (full), private capability information (dashes), and full capability information (dots).
			 And only the winning team -if another team gets a disastrous AI first by taking lower precautions, they will 'won' the race to build the first AI.2 Of course, the model can be refined in various ways. One could make capacity information uncertain and fuzzy, one could have different levels of enmity between different teams, one could incorporate uncertainty about the safety levels and the ultimate outcomes, and so on. Or one could have a dynamic process to determine the outcome, rather than rushing straight to the Nash equilibrium. But the simple model is enough to gain useful insights.3 It may seem unusual for teams to not know their own capabilities in the real world. However, this is close to the situation we find ourselves with current AI research: people and organisations have a pretty clear idea of what resources and knowledge they have, but don't know how hard AI is or what routes are most likely to lead there. They are thus effectively ignorant of their own AI-building capabilities.
			 If makes no sense that a team with higher capability would have a lower chance of winning (if so, they would voluntarily destroy part of their capability).
			 Such secrecy can interfere with trust building, though, making it hard to reach agreements between teams if such agreement is needed.6 This is because only the teams with low capability take risks in cases of private information, and the more teams there are, the less likely it is that the winner will be low capability.

Abstract: This paper presents a simple model of an AI arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivised to finish first -by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others' capabilities (and about their own), the more the danger increases.


Under review as a conference paper at ICLR 2019 UNCOVERING SURPRISING BEHAVIORS IN REINFORCEMENT LEARNING VIA WORST-CASE ANALYSIS

INTRODUCTION Reinforcement Learning (RL) methods have achieved great success over the past few years, achieving human-level performance on a range of tasks such as Atari  (Mnih et al., 2015) , Go  (Silver et al., 2016) , Labyrinth , and Capture the Flag  (Jaderberg et al., 2018) . On these tasks, and more generally in reinforcement learning, agents are typically trained and evaluated using their average reward over environment settings as the measure of performance, i.e. E P (e) [R(π(θ), e)] , where π(θ) denotes a policy with parameters θ, R denotes the total reward the policy receives over the course of an episode, and e denotes environment settings such as maze structure in a navigation task, appearance of objects in the environment, or even the physical rules governing environment dynamics. But what biases does the distribution P (e) contain, and what biases, or failures to generalize, do these induce in the strategies agents learn? To help uncover biases in the training distribution and in the strategies that agents learn, we propose evaluating the worst-case performance of agents over environment settings, i.e. min e∈E E [R(π(θ), e)] , where E is some set of possible environment settings. Worst-case analysis can provide an important tool for understanding robustness and generalization in RL agents. For example, it can help us with: • Understanding biases in training Catastrophic failures can help reveal situations that are rare enough during training that the agent does not learn a strategy that is general enough to cope with them. • Robustness For critical systems, one would want to eliminate, or at least greatly reduce, the probability of extreme failures. • Limiting exploitability If agents have learned strategies that fail to generalize to particular environment settings, then an adversary could try and exploit an agent by trying to engineer such environment settings leading to agent failure. In this work, we use worst-case analysis to investigate the performance of a state-of-the-art agent in solving a first-person 3D navigation task; a task on which agents have recently achieved average-case human-level performance  (Wayne et al., 2018) . By optimizing mazes to minimize the performance of agents, we discover the existence of mazes where agents repeatedly fail to find the goal (which we refer to as a Catastrophic Failure). Our Contributions To summarize, the key contributions of this paper are as follows: 1. We introduce an effective and intuitive approach for finding simple environment settings leading to failure (Section 2). 2. We show that state-of-the-art agents carrying out navigation tasks suffer from drastic and often surprising failure cases (Sections 3.1 and 3.2). 3. We demonstrate that mazes leading to failure transfer across agents with different hyperparameters and, notably, even different architectures (Section 3.3). 4. We present an initial investigation into how the training distribution can be adapted by incorporating adversarial and out-of-distribution examples (Section 4).
 APPROACH Tasks We consider agents carrying out first-person 3D navigation tasks. Navigation is of central importance in RL research as it captures the challenges posed by partially observable Markov decision processes (POMDPs). The navigation tasks we use are implemented in DeepMind Lab (DM Lab)  (Beattie et al., 2016) . 1 Each episode is played on a 15 × 15 maze where each position in the maze may contain a wall, an agent spawn point, or a goal spawn point. The maze itself is procedurally generated every episode, along with the goal and agent spawn locations. The goal location remains fixed throughout an episode, while the agent spawn location can vary. In training, the agent respawns at different locations each time they reach the goal, while for our optimization and analysis we limit the agent to the same spawn location. Agents receive RGB observations of size 96 × 72 pixels, examples of which are provided in Figure  1 . Episodes last for 120 seconds and are played at a framerate of 15 frames per second. The agent receives a positive reward of 10 every time it reaches the goal, and 0 otherwise. On this specific navigation task, RL agents have recently achieved human-level average-case performance  (Wayne et al., 2018) . Agents We perform our analysis on Importance Weighted Actor-Learner Architecture agents trained to achieve human-level average-case performance on navigation tasks. These agents can be described as async batched-a2c agents with the V-trace algorithm for off policy-correction, and we henceforth refer to these as A2CV agents . Details of the training procedure are provided in Appendix A.1. Search Algorithm If we are interested in worst-case performance of agents, how can we find environment settings leading to the worst performance? In supervised learning, one typically uses gradient based methods to find inputs that lead to undesired output  (Biggio et al., 2013; Szegedy et al., 2013; Goodfellow et al., 2014) . In contrast, we search for environment settings leading to an undesired outcome at the end of an episode. This presents a challenge as the environment rendering and MDP are not differentiable. We are therefore limited to black-box methods where we can only query agent performance given environment settings. To search for environment settings which cause catastrophic failures, we propose the local search procedure described in Algorithm 1 (visualizing the process in Figure  2 ).   We then Evaluate each with the agent over 30 episodes, select the best maze (i.e. lowest agent score), and Modify this maze by randomly moving two walls to form the next set of candidates (Iteration 1). This process is repeated for 20 iterations, leading to a maze where the agent score is 0.09 in this example (i.e. the agent finds the goal once in 11 episodes). In Appendix A.2.1 we detail the computational requirements of this search procedure.
 EXPERIMENTS The agents we study achieve impressive average-case performance, but how much does their worstcase performance differ from their average-case performance? To investigate this, we consider the worst-case performance over a large set of mazes, including mazes that are not possible under the training distribution. A natural question to ask is whether any departure from the wall structure present during training will lead to agent failure. To test this, we evaluate the agent on samples from a distribution of mazes containing all mazes agents could be evaluated on during the search. In particular, we randomly select agent and goal spawn locations in the first step and then randomly move 40 walls, corresponding to the same actions taken by our optimization procedure, but where the actions are chosen randomly rather than in order to minimize agent performance. We find that agents do generalize to random mazes from the set we consider. In fact, we find that the average score obtained by agents on randomly perturbed mazes is slightly higher than on the training distribution, with agents obtaining an average of 45 goal reaches per two minute episode. The increased performance is likely due to the agent spawn location being fixed, making it easier for the agent to return to the goal once found. The considered agents generalize in the sense that agent performance is not reduced on average by out-of-distribution wall structure. But what about the worst case over all wall structures? Have the agents learned a general navigation strategy that works for all solvable mazes? Or do there exist environment settings that lead to catastrophic failures with high probability? In this section, we investigate these questions. We define a Catastrophic Failure to be an agent failing to find the goal in a two minute episode (1800 steps). As detailed below, we find that not only do there exist mazes leading to catastrophic failure, there exist surprisingly simple mazes that lead to catastrophic failure for agents yet are consistently and often rapidly solved by humans.
 RESULT 1: CATASTROPHIC FAILURES EXIST Do environment settings leading to catastrophic failure exist for the agents we are considering? By searching over mazes using the procedure outlined in Algorithm 1, we find mazes where agents fail to find the goal on many episodes, only finding the goal 10% of the time. In fact, some individual mazes lead to failure across five different agents we tested, with even the best performing agent only finding the goal in 20% of the episodes.  Optimization curves for our search procedure are given in Figure  3 . Note that while we define catastrophic failure as failure to find the goal, the actual objective used for the optimization was average number of goal reaches during an episode. Using average number of goals gives a stronger signal at the start of the optimization process. Finding mazes leading to lower average number of captures is easier than finding mazes where the agent rarely finds the goal even once. As can be seen, despite finding the goal on average 45 times per episode on randomly perturbed mazes, on mazes optimized to reduce score, agents find the goal on average only 0.33 times per episode, more than a 100× decrease in performance. In terms of probability of catastrophic failure, we note that despite agents finding the goal in approximately 98% of episodes on randomly perturbed mazes, using our method, on average we find mazes where agents only finds the goal in 30% of the episodes. Example trajectories agents take during failures are visualized in Figure  4 . The trajectories often seem to demonstrate a failure to use memory to efficiently explore the maze with the agent repeatedly visiting the same locations multiple times. The mazes presented in Figure  4  appear to be of higher complexity than mazes seen during training. This suggests that to obtain agents that truly master navigation, more complex mazes should be included in the training distribution. However, we can ask whether it is only more complex mazes that lead to catastrophic failure or whether there are also simple mazes leading to catastrophic failure. This is a question we explore in the next subsection.
 RESULT 2: SIMPLE CATASTROPHIC FAILURES EXIST While the existence of catastrophic failures may be intriguing and perhaps troubling, one might suspect that the failures are caused by the increased complexity of the mazes leading to failure relative to the mazes the agent is exposed to during training, e.g., the mazes leading to failure contain more dead ends and sometimes have lower visibility. Further, understanding the cause of failure in such mazes seems challenging due to the large number of wall structures that may be causing the agent to fail. In this section, we explore whether there exist simple mazes which lead to catastrophic failures. As our measure of complexity, we use the total number of walls in the maze. We also evaluate humans on such mazes to get a quantitative measure of maze complexity. To find simple mazes which lead to failure, we first follow the same procedure as in the previous section, producing a set of mazes which all lead to catastrophic failures (i.e. a low agent scores). Next, we use this set of mazes as the initial set of candidates in our search algorithm, however we now use a Modify function that removes a single randomly chosen wall each iteration. This process is repeated for 70 iterations, searching for a maze with few walls while maintaining low agent score. In Figure  4 , we present the resulting simple mazes and the corresponding agent trajectories from our optimization procedure. Interestingly, we find that one can remove a majority of the walls in a maze and still maintain the catastrophic failure (i.e. very low agent score). Of note is that a number of these mazes are strikingly simple, suggesting that there exists structure in the environment that the agent has not generalized to. For example, we can see that placing the goal in a small room in an otherwise open maze can significantly reduce the agent's ability to find the goal. Human baselines While these simple maps may lead to catastrophic failure, it is unclear whether this is because of the agent or whether the maze is difficult in a way that is not obvious. To investigate this, we perform human experiments by tasking humans to play on a set of 10 simplified mazes. Notably, we find that human players are able to always locate the goal in every maze and typically within one third of the full episode length. This demonstrates that the mazes are comfortably solvable within the course of an episode by players with a general navigation strategy. We provide a detailed comparison of agent and human performance in Appendix A.3. Analysis One question that may arise is the extent to which these mazes are isolated points in the space of mazes. That is, if the maze was changed slightly, would it no longer lead to catastrophic failure? To test this, we investigate how sensitive our discovered failure mazes are with respect to the agent and goal spawn locations on simplified adversarial mazes. As can be seen in Figure  5 , we find that for a large range of spawn locations, the mazes still lead to failure. This indicates that there is specific local maze structure which causes agents to fail.  Procedures for finding such simple mazes may prove useful as a tool for debugging agents and understanding the ways in which training has led them to develop narrow strategies that are good enough for achieving high average-case performance.
 RESULT 3: FAILURE MAZES TRANSFER ACROSS AGENTS We have found failure cases for individual agents, but to what extent do these failure cases highlight a specific peculiarity of the individual agent versus a more general failure of a certain class of agents, or even a shortcoming of the distribution used for training? In this section, we investigate whether mazes which cause one trained agent to fail also cause other agents to fail. We consider two types of transfer: (1) between different hyperparameters of the same model architecture, and (2) between different model architectures. To test transfer between agents of the same architecture, we train a set of five A2CV agents with different entropy costs and learning rates. To test transfer between agents with significantly different architectures, we train a set of five MERLINbased agents  (Wayne et al., 2018) . These agents have a number of differences to the A2CV agents, most notably they contain a sophisticated memory structure based on a DNC (but with a fixed write location per timestep)  (Graves et al., 2016) . Both agents are trained on the same distribution and achieve human-level averages scores on the navigation task (with MERLIN scoring 10% higher than A2CV on average). Further details of agent training can be found in Appendix A.1. To quantify the level of transfer between (sets of) agents, we follow the procedure for finding adversarial mazes outlined in Section 3.1 to produce a collection of 50 unique failure mazes for each agent (i.e. 10 collections of 50 mazes each). We then evaluate every agent 100 times on each maze in each collection, reporting their average performance on each collection. Complete quantitative transfer results can be found in Appendix A.4. Failure cases transfer somewhat across all agents First, we find that across all agents, some level of transfer exists. In particular, as can be seen in Figure  6 , the probability of one agent finding the goal on mazes generated to reduce the score another agent is significantly below 1. This suggests a common cause of failure that is some combination of the distribution of environment settings used during training and the set of methods that are currently used to train such agents. A possible way to address this could be enriching the training distribution so that it contains fewer biases and encourages more general solutions. Transfer within agent type is stronger than between agent type Comparing the performance of each agent type on mazes from the same agent type to mazes from another agent type, we see that transfer within agent type is stronger. As shown in Figure  6b , performance increases as we go from 'MERLIN to MERLIN' 'A2CV to MERLIN' (0.42 to 0.58) and also if we go from 'A2CV to A2CV' to 'MERLIN to A2CV' (0.63 to 0.70). This suggests that there are some common biases in agents that are due to their architecture type. Analyzing structural differences between mazes that lead to one agent type to fail but not another could give interesting insight into behavioural differences between agents beyond just average performance. A2CV agents are less susceptible to transfer Despite similar probabilities of failure when evaluating on mazes optimized for the same agent, A2CV agents seem to suffer less on mazes optimized using other A2CV or MERLIN agents. This indicates that A2CV agents may have learned a more diverse set of strategies. Figure  6 : Mazes that lead to failures in one agent lead to failure in other agents as well. This is the case for agents of the same architecture with different hyperparameters, and is also the case for transfer across agents of different architecture. We note, however, that transfer across agents with different architectures is weaker than among agents with the same architecture, and that the performance of agents with the same architecture but with different hyperparameters is slightly higher than for the agents used to originally find the mazes.
 ADAPTING THE TRAINING DISTRIBUTION From our experiments so far, we have discovered that there exist many mazes which lead to catastrophic failure. In this section, we investigate whether agent performance can be improved by adapting the training distribution, for example by incorporating adversarial mazes into training and modifying the original mazes used in training.
 MOTIVATION To better understand what may be causing catastrophic failures, with the aim of fixing them, we compare the set of adversarial mazes with the original set of mazes used in training. From this comparison, we find that there are two notable differences. The probability of a catastrophic failure correlates with the distance between the spawn locations and how hidden the goal is First, we find that a number of features are more common in adversarial mazes than non-adversarial mazes. In particular, adversarial mazes are more likely to have the goal hidden in an enclosed space (such as a small room), and on average the path length from the player's starting location to the goal is significantly longer (31.1 ± 8.4 compared to 11.6 ± 6.3). Notably, while the training distribution also contains hidden goals which are far from the agent's starting location, they are much rarer. Adversarial mazes are typically far from the training distribution Second, we find that adversarial mazes tend to not only be out-of-distribution, but also far from the training distribution due to the Modify function used in our adversarial search procedure (for example, see Figure  2 ). This contrasts with the adversarial images literature where attacks are usually constrained to be small or imperceptible. It may therefore not be surprising that the agent is unable to generalize to all out-of-distribution mazes which could also explain the significant reduction in their performance. Given these two observations, it is natural to ask whether the training distribution can be adapted to improve the agent's performance. In the following sections we investigate this question and discuss our findings, focusing on incorporating adversarial mazes into training and modifying the original mazes used in training.
 APPROACH We consider two distinct approaches for incorporating adversarial and out-of-distribution mazes into the training distribution. Adversarial training To add adversarial mazes into the training distribution, we first create a dataset of 6000 unique adversarial mazes from separate runs of our search procedure using the previously trained A2CV agents. Notably, this set also includes the 250 mazes used in our transfer experiments (Section 3.3). Next, we train a new set of A2CV agents using both this adversarial set of mazes and the standard distribution of mazes, sampling randomly every episode (i.e. 50% of training episodes are on an adversarial maze). Randomly perturbed training To ensure our adversarial search procedure produces indistribution adversarial mazes, we alter the default maze generator used in training so that any adversarial maze can be generated. We accomplish this by randomly perturbing the original mazes, repeatedly using the same Modify function used by our adversarial search procedure, but selecting candidates randomly rather than by worst agent performance.
 RESULTS In this section, we report our findings on the robustness of agents trained using the approaches above. Catastrophic failures still exist Our main finding is that while agents learn to perform well on the richer distributions of mazes described above, this does not lead to robust agents. In particular, agents trained on a distribution of mazes enriched with 6000 adversarial mazes were able to find the goal on average 89.8% of the time on the adversarial mazes they were trained on. Similarly, agents trained on randomly perturbed mazes were able to find the goal close to 100% of the time on the distribution they were trained on. However, despite the agents being trained on these richer training distributions, the same search method is still able to find mazes leading to extreme failure as can be seen in Figure  7 . One possible explanation for this result is that the 6000 adversarial mazes used for training were insufficient to get good coverage of the space of mazes, and that further enlarging this set could yield qualitatively different results. Indeed, for agents trained using randomly perturbed mazes, the search procedure took 50 iterations to obtain the same level of failure as it did after 20 iterations when applied to agents trained on the standard training distribution. This suggests that perhaps enriching the training distribution with a very large set of adversarial mazes may lead to more general and robust agents. However, there are a number of challenges that need to be addressed before this approach can be tested which we will describe in the next section.
 DISCUSSION Our results suggest that if a richer training distribution is to yield more robust agents, we may need to use a very large set of environment settings leading to failure. This is similar to how adversarial training in supervised learning is performed where more adversarial examples are used than the original training examples. We describe below what we see as two significant challenges that need to be overcome before such an approach can be thoroughly evaluated in the RL setting.
 Expensive generation The cost of generating a single adversarial setting is on the order of 1000's episodes using the method in this work. This implies that generating a set of adversarial settings which is similar in size to the set trained on would require orders of magnitude more computational than training itself. This could be addressed with faster methods for generating adversarial settings. Expensive training Since agents receive very little reward in adversarial settings, the training signal is incredibly sparse. Therefore, it is possible that many more training iterations are necessary for agents to learn to perform well in each adversarial setting. A possible solution to this challenge is to design a curriculum over adversity, whereby easier variants of the adversarial settings are injected into the training distribution. For example, for the navigation tasks considered here, one could include training settings with challenging mazes where the goal is in any position on the shortest path between the starting location of the agent and the challenging goal. We hope that these challenges can be overcome so that, in the context of RL, the utility of adversarial retraining can be established -an approach which has proved useful in supervised learning tasks. However, since significant challenges remain, we suspect that much effort and many pieces of work will be required before a conclusive answer is achieved.
 RELATED WORK Navigation Recently, there has been significant focus in the RL community on agent navigation in simulated 3D environments, including a community-wide challenge for agents in such environments called VizDoom  (Kempka et al., 2016) . Such 3D first-person navigation tasks are particularly interesting because they capture challenges such as partial observability, and require the agent to "effectively perceive, interpret, and learn the 3D world in order to make tactical and strategic decisions where to go and how to act."  (Kempka et al., 2016) . Recent advances have led to impressive human-level performance on navigation tasks in large procedurally generated environments  (Beattie et al., 2016; Wayne et al., 2018) . Adversarial examples in supervised learning Our work can be seen as an RL navigation analogue of work on adversarial attacks on supervised learning systems for image classification  (Szegedy et al., 2013) . For adversarial attacks on image classifiers, one considers a set of inputs that is larger than the original distribution, but where one would hope that systems perform just as well on L ∞ balls around inputs from the distribution. In particular, the adversarial examples lie outside the training distribution. Analogously, we consider a set of mazes which is larger than the original set of mazes used during training, but where we would hope our system will work just as well. Notably, while similar on a conceptual level, our setting has two key differences from this previous line of work: (1) The attack vector consists of changing latent semantic features of the environment (i.e. the wall structure of a maze), rather than changing individual pixels in an input image in an unconstrained manner. (2) The failure is realized over multiple steps of agent and environment interacting with each other, rather than simply being errant output from a single forward pass through a neural net. More recently, in the context of supervised learning for image classification, there has been work to find constrained adversarial attacks which is closer to what we consider in this work  (Athalye & Sutskever, 2017; Fawzi & Frossard, 2015; Eykholt et al., 2018; Sharif et al., 2016) . In the context of interpretable adversarial examples in image classification, similar approaches to our simplification approach have been explored where one searches for adversarial perturbations with group-sparse structure or other minimal structure  (Xu et al., 2018; Brendel et al., 2018) . Additionally, our findings regarding transfer are consistent with findings on adversarial examples for computer vision networks where it has been found that perturbations that are adversarial for one network often transfer across other networks  (Szegedy et al., 2013; Tramèr et al., 2017)  Input attacks on RL systems There have been a number of previous works which have extended adversarial attacks to RL settings, however they have achieved this by manipulating inputs directly, which effectively amounts to changing the environment renderer  (Huang et al., 2017; Lin et al., 2017a; b) . While these are interesting from a security perspective, it is less clear what they tell us about the generality of the strategy learned by the agent. Generalization in RL systems Recently, it has been shown that simple agents trained on restricted datasets fail to learn sufficiently general navigation strategies to improve goal retrieval times on held out mazes  (Dhiman et al., 2018) . In comparison, our method is both automatic and able to find more spectacular failures. Further, our findings highlight failures in exploration during navigation. This is in contrast to this previous work which studied failures to exploit knowledge from previous goal retrievals in the same episode. In the context of testing generalization in RL, previous work has looked at statistical generalization in RL . Here we consider agents that already generalize in the statistical sense and try to better characterize the ways in which they generalize beyond the average-case.
 CONCLUSIONS AND FUTURE WORK In this work, we have shown that despite the strong average-case performance often reported of RL agents, worst-case analysis can uncover environment settings which agents have failed to generalize to. Notably, we have found that not only do catastrophic failures exist, but also that simple catastrophic failures exist which we would hope agents would have generalized to, and that failures also transfer between agents and architectures. As agents are trained to perform increasingly complicated tasks in more sophisticated environments, for example AirSim  (Shah et al., 2017)  and CARLA  (Dosovitskiy et al., 2017) , it is of interest to understand their worst-case performance and modes of generalization. Further, in real world applications such as self-driving cars, industrial control, and robotics, searching over environment settings to investigate and address such behaviours is likely to be critical on the path to robust and generalizable agents. To conclude, while this work has focused mostly on evaluation and understanding, it is only a first step towards the true goal of building more robust, general agents. Initial results we report indicate that enriching the training distribution with settings leading to failure may need to be done at a large scale if it is to work, which introduces significant challenges. While training robust agents is likely an endeavour requiring significant effort, we believe it is important if agents are to carry out critical tasks and on the path to finding more generally intelligent agents. goal was <50% (compared to 98% on the average maze). Furthermore, the 25th, 50th, and 75th percentiles were as follows: • p(reaching the goal): 0.031, 0.136, 0.279 • number of goals reached: 0.042, 0.136, 0.368
 A.3 HUMAN EXPERIMENTS To upper bound the intrinsic difficulty of the mazes found to be adversarial to agents, we conducted experiments where three humans played on the same mazes. Each human played a single episode on each of ten mazes. The humans played at the same resolution as agents, 96x72 pixels, to rule out visual acuity as a confounding factor. On all mazes, all humans successfully found the goal in the course of the episode. In fact, in most episodes, humans were able to find the goal in less than a third of the episode. Figure  8 : Mazes used for human experiments. For each maze, the agent that performed best found the goal less than 50% of the time. In contrast, humans always found the goal, usually within less than a third of the episode. Note that humans played at the same resolution as agents, 96x72 pixels.    Figure 1 : 1 Figure 1: Navigation task. (left) Example maze from the training distribution together with the path taken by the agent from spawn (cyan) to goal (magenta). (right) Frames from top left to bottom right correspond to agent observations as it takes the path from spawn to goal. Note that while the navigation task may look simple given a top down view, the agent only receives very partial information about the maze at every step, making navigation a difficult task.
 Figure 2 : 2 Figure 2: Example of search procedure. First, we generate a set of 10 initial candidate mazes by sampling from the training distribution.We then Evaluate each with the agent over 30 episodes, select the best maze (i.e. lowest agent score), and Modify this maze by randomly moving two walls to form the next set of candidates (Iteration 1). This process is repeated for 20 iterations, leading to a maze where the agent score is 0.09 in this example (i.e. the agent finds the goal once in 11 episodes). In Appendix A.2.1 we detail the computational requirements of this search procedure.
 (a) Average number of goals reached per episode over the course of the optimization. (b) Probability of the agent reaching the goal in an episode.
 Figure 3 : 3 Figure 3: The search algorithm is able to rapidly find mazes where agents fail to find the goal. (a) The objective used for the optimizer is average agent score. The dashed line corresponds to average goals reached on randomly perturbed mazes. (b) Minimizing score also leads to a low probability of at least one goal retrieval in an episode. The dashed line corresponds to average probability of reaching a goal on randomly perturbed mazes. The blue lines are computed by averaging across 50 optimization runs.
 Figure 4 : 4 Figure 4: Example mazes leading to low scores and example trajectories on these mazes. (a) Maze with randomly perturbed walls. Despite being out of distribution, agents find the goal in 98% of episodes on such mazes and are able to get the goal on average 45 times per episode. (b) Maze obtained after 20 iterations, moving two walls at each iteration to minimize reward. All agents find the goal on such mazes in less than 20% of episodes. (c) Maze obtained through additional iterations of removing walls. All agents find the goal on such mazes in less than 40% of episodes. (d) Human trajectory on the same maze as in (c). Humans are able to consistently find the goal on such mazes.
 (a) Agent locations, with goal (magenta) fixed.(b) Goal locations, with agent (cyan) fixed.
 Figure 5 : 5 Figure 5: Adversarial mazes are robust to change of spawn positions. The probability of goal retrieval (shown with the color bar) remains low across large portions of the simplified maze as the (a) agent spawn locations and (b) goal locations are moved for each episode..
 (a) Across hyperparameters, same architecture. (b) Across architectures.
 (a) Average number of goals reached. (b) Probability of finding the goal in a single episode.
 Figure 7 : 7 Figure 7: Richer training distributions did not lead to robust agents. Adversarial optimization for agents trained with adversarial mazes (red) and for agents trained with randomly perturbed mazes (yellow). Compared against the Standard training method from Figure 3 for 50 iterations (blue).
 Figure 9 : 9 Figure 9: Trajectories taken by Human 3 on mazes leading to agent failure.
 Table 1 : 1 Human seconds-to-first-goal on agent failure mazes we provide detailed results for our transfer experiments. In particular, we detail transfer between all pairs among the 10 agents, five A2CV agents and five MERLIN agents trained with different entropy costs and learning rates.
 Figure 10 : 10 Figure 10: Pairwise transfer scores. Lower number indicates more transfer. 'A' corresponds to our A2CV agent, and 'M' corresponds to MERLIN.
 end best ← Evaluate(candidates, num evaluations); Algorithm 1: Method for finding environment settings leading to failure cases. Concretely, we generate a set of initial candidate mazes by sampling mazes from the training distribution. We then use the Modify function on the maze which yielded the lowest agent score to randomly move two walls to produce a new set of candidates, rejecting wall moves that lead to unsolvable mazes. Importantly, this method is able to effectively find catastrophic failure cases (as we demonstrate in Section 3.1), while also having the advantage of being intuitive to understand and implement. input : num iterations, num candidates, num evaluations and function Modify output: An environment setting best candidates ← GenerateCandidates(num candidates); for i ← 1 to num iterations do best ← Evaluate(candidates, num evaluations); candidates ← Modify(best, num candidates);

Abstract: Reinforcement learning agents are typically trained and evaluated according to their performance averaged over some distribution of environment settings. But does the distribution over environment settings contain important biases, and do these lead to agents that fail in certain cases despite high average-case performance? In this work, we consider worst-case analysis of agents over environment settings in order to detect whether there are directions in which agents may have failed to generalize. Specifically, we consider a 3D first-person task where agents must navigate procedurally generated mazes, and where reinforcement learning agents have recently achieved human-level average-case performance. By optimizing over the structure of mazes, we find that agents can suffer from catastrophic failures, failing to find the goal even on surprisingly simple mazes, despite their impressive average-case performance. Additionally, we find that these failures transfer between different agents and even significantly different architectures. We believe our findings highlight an important role for worst-case analysis in identifying whether there are directions in which agents have failed to generalize. Our hope is that the ability to automatically identify failures of generalization will facilitate development of more general and robust agents. To this end, we report initial results on enriching training with settings causing failure. 1 A full description and code for the tasks can be found at https://github.com/deepmind/lab/ tree/master/game_scripts/levels/contributed/dmlab30#goal-locations-large.


Dylan Hadfield-Menell
Anca Dragan
Pieter Abbeel
Stuart Russell

Introduction "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively . . . we had better be quite sure that the purpose put into the machine is the purpose which we really desire." So wrote Norbert  Wiener (1960)  in one of the earliest explanations of the problems that arise when a powerful autonomous system operates with an incorrect objective. This value alignment problem is far from trivial. Humans are prone to mis-stating their objectives, which can lead to unexpected implementations. In the myth of King Midas, the main character learns that wishing for 'everything he touches to turn to gold' leads to disaster. In a reinforcement learning context,  Russell & Norvig (2010)  describe a seemingly reasonable, but incorrect, reward function for a vacuum robot: if we reward the action of cleaning up dirt, the optimal policy causes the robot to repeatedly dump and clean up the same dirt. A solution to the value alignment problem has long-term implications for the future of AI and its relationship to humanity  (Bostrom, 2014 ) and short-term utility for the design of usable AI systems. Giving robots the right objectives and enabling them to make the right trade-offs is crucial for self-driving cars, personal assistants, and human-robot interaction more broadly. The field of inverse reinforcement learning or IRL  (Russell, 1998; Ng & Russell, 2000; Abbeel & Ng, 2004 ) is certainly relevant to the value alignment problem. An IRL algorithm infers the reward function of an agent from observations of the agent's behavior, which is assumed to be optimal (or approximately so). One might imagine that IRL provides a simple solution to the value alignment problem: the robot observes human behavior, learns the human reward function, and behaves according to that function. This simple idea has two flaws. The first flaw is obvious: we don't want the robot to adopt the human reward function as its own. For example, human behavior (especially in the morning) often conveys a desire for coffee, and the robot can learn this with IRL, but we don't want the robot to want coffee! This flaw is easily fixed: we need to formulate the value alignment problem so that the robot always has the fixed objective of optimizing reward for the human, and becomes better able to do so as it learns what the human reward function is. The second flaw is less obvious, and less easy to fix. IRL assumes that observed behavior is optimal in the sense that it accomplishes a given task efficiently. This precludes a variety of useful teaching behaviors. For example, efficiently making a cup of coffee, while the robot is a passive observer, is a inefficient way to teach a robot to get coffee. Instead, the human should perhaps explain the steps in coffee preparation and show the robot where the backup coffee supplies are kept and what do if the coffee pot is left on the heating plate too long, while the robot might ask what the button with the puffy steam symbol is for and try its hand at coffee making with guidance from the human, even if the first results are undrinkable. None of these things fit in with the standard IRL framework. Cooperative inverse reinforcement learning. We propose, therefore, that value alignment should be formulated as a cooperative and interactive reward maximization process. More precisely, we define a cooperative inverse reinforcement learning (CIRL) game as a two-player game of partial information, in which the "human", H, knows the reward function (represented by a generalized parameter θ), while the "robot", R, does not; the robot's payoff is exactly the human's actual reward. Optimal solutions to this game maximize human reward; we show that solutions may involve active instruction by the human and active learning by the robot. Reduction to POMDP and Sufficient Statistics. As one might expect, the structure of CIRL games is such that they admit more efficient solution algorithms than are possible for general partialinformation games. Let (π H , π R ) be a pair of policies for human and robot, each depending, in general, on the complete history of observations and actions. A policy pair yields an expected sum of rewards for each player. CIRL games are cooperative, so there is a well-defined optimal policy pair that maximizes value.  2  In Section 3 we reduce the problem of computing an optimal policy pair to the solution of a (single-agent) POMDP. This shows that the robot's posterior over θ is a sufficient statistic, in the sense that there are optimal policy pairs in which the robot's behavior depends only on this statistic. Moreover, the complexity of solving the POMDP is exponentially lower than the NEXP-hard bound that  (Bernstein et al., 2000)  obtained by reducing a CIRL game to a general Dec-POMDP. Apprenticeship Learning and Suboptimality of IRL-Like Solutions. In Section 3.3 we model apprenticeship learning  (Abbeel & Ng, 2004 ) as a two-phase CIRL game. In the first phase, the learning phase, both H and R can take actions and this lets R learn about θ. In the second phase, the deployment phase, R uses what it learned to maximize reward (without supervision from H). We show that classic IRL falls out as the best-response policy for R under the assumption that the human's policy is "demonstration by expert" (DBE), i.e., acting optimally in isolation as if no robot exists. But we show also that this DBE/IRL policy pair is not, in general, optimal: even if the robot expects expert behavior, demonstrating expert behavior is not the best way to teach that algorithm. We give an algorithm that approximately computes H's best response when R is running IRL under the assumption that rewards are linear in θ and state features. Section 4 compares this best-response policy with the DBE policy in an example game and provides empirical confirmation that the bestresponse policy, which turns out to "teach" R about the value landscape of the problem, is better than DBE. Thus, designers of apprenticeship learning systems should expect that users will violate the assumption of expert demonstrations in order to better communicate information about the objective.
 Related Work Our proposed model shares aspects with a variety of existing models. We divide the related work into three categories: inverse reinforcement learning, optimal teaching, and principal-agent models. Inverse Reinforcement Learning.  Ng & Russell (2000)  define inverse reinforcement learning (IRL) as follows: "Given measurements of an [actor]'s behavior over time. . . . Determine the reward function being optimized." The key assumption IRL makes is that the observed behavior is optimal in the sense that the observed trajectory maximizes the sum of rewards. We call this the demonstration-by-expert (DBE) assumption. One of our contributions is to prove that this may be suboptimal behavior in a CIRL game, as H may choose to accept less reward on a particular action in order to convey more information to R. In CIRL the DBE assumption prescribes a fixed policy
 Ground Truth
 Expert Demonstration Instructive Demonstration Figure  1 : The difference between demonstration-by-expert and instructive demonstration in the mobile robot navigation problem from Section 4. Left: The ground truth reward function. Lighter grid cells indicates areas of higher reward. Middle: The demonstration trajectory generated by the expert policy, superimposed on the maximum a-posteriori reward function the robot infers. The robot successfully learns where the maximum reward is, but little else. Right: An instructive demonstration generated by the algorithm in Section 3.4 superimposed on the maximum a-posteriori reward function that the robot infers. This demonstration highlights both points of high reward and so the robot learns a better estimate of the reward. for H. As a result, many IRL algorithms can be derived as state estimation for a best response to different π H , where the state includes the unobserved reward parametrization θ.  Ng & Russell (2000) ,  Abbeel & Ng (2004), and Ratliff et al. (2006)  compute constraints that characterize the set of reward functions so that the observed behavior maximizes reward. In general, there will be many reward functions consistent with this constraint. They use a max-margin heuristic to select a single reward function from this set as their estimate. In CIRL, the constraints they compute characterize R's belief about θ under the DBE assumption.  Ramachandran & Amir (2007)  and  Ziebart et al. (2008)  consider the case where π H is "noisily expert," i.e., π H is a Boltzmann distribution where actions or trajectories are selected in proportion to the exponent of their value.  Ramachandran & Amir (2007)  adopt a Bayesian approach and place an explicit prior on rewards.  Ziebart et al. (2008)  places a prior on reward functions indirectly by assuming a uniform prior over trajectories. In our model, these assumptions are variations of DBE and both implement state estimation for a best response to the appropriate fixed H.  Natarajan et al. (2010)  introduce an extension to IRL where R observes multiple actors that cooperate to maximize a common reward function. This is a different type of cooperation than we consider, as the reward function is common knowledge and R is a passive observer.  Waugh et al. (2011)  and  Kuleshov & Schrijvers (2015)  consider the problem of inferring payoffs from observed behavior in a general (i.e., non-cooperative) game given observed behavior. It would be interesting to consider an analogous extension to CIRL, akin to mechanism design, in which R tries to maximize collective utility for a group of Hs that may have competing objectives. Fern et al. (  2014 ) consider a hidden-goal MDP, a special case of a POMDP where the goal is an unobserved part of the state. This can be considered a special case of CIRL, where θ encodes a particular goal state. The frameworks share the idea that R helps H. The key difference between the models lies in the treatment of the human (the agent in their terminology). Fern et al. (  2014 ) model the human as part of the environment. In contrast, we treat H as an actor in a decision problem that both actors collectively solve. This is crucial to modeling the human's incentive to teach. Optimal Teaching. Because CIRL incentivizes the human to teach, as opposed to maximizing reward in isolation, our work is related to optimal teaching: finding examples that optimally train a learner  (Balbach & Zeugmann, 2009; Goldman et al., 1993; Goldman & Kearns, 1995) . The key difference is that efficient learning is the objective of optimal teaching, while it emerges as a property of optimal equilibrium behavior in CIRL.  Cakmak & Lopes (2012)  consider an application of optimal teaching where the goal is to teach the learner the reward function for an MDP. The teacher gets to pick initial states from which an expert executes the reward-maximizing trajectory. The learner uses IRL to infer the reward function, and the teacher picks initial states to minimize the learner's uncertainty. In CIRL, this approach can be characterized as an approximate algorithm for H that greedily minimizes the entropy of R's belief. Beyond teaching, several models focus on taking actions that convey some underlying state, not necessarily a reward function. Examples include finding a motion that best communicates an agent's intention  (Dragan & Srinivasa, 2013) , or finding a natural language utterance that best communicates a particular grounding  (Golland et al., 2010) . All of these approaches model the observer's inference process and compute actions (motion or speech) that maximize the probability an observer infers the correct hypothesis or goal. Our approximate solution to CIRL is analogous to these approaches, in that we compute actions that are informative of the correct reward function. Principal-agent models. Value alignment problems are not intrinsic to artificial agents.  Kerr (1975)  describes a wide variety of misaligned incentives in the aptly titled "On the folly of rewarding A, while hoping for B." In economics, this is known as the principal-agent problem: the principal (e.g., the employer) specifies incentives so that an agent (e.g., the employee) maximizes the principal's profit  (Jensen & Meckling, 1976) . Principal-agent models study the problem of generating appropriate incentives in a non-cooperative setting with asymmetric information. In this setting, misalignment arises because the agents that economists model are people and intrinsically have their own desires. In AI, misalignment arises entirely from the information asymmetry between the principal and the agent; if we could characterize the correct reward function, we could program it into an artificial agent.  Gibbons (1998)  provides a useful survey of principal-agent models and their applications.
 Cooperative Inverse Reinforcement Learning This section formulates CIRL as a two-player Markov game with identical payoffs, reduces the problem of computing an optimal policy pair for a CIRL game to solving a POMDP, and characterizes apprenticeship learning as a subclass of CIRL games.
 CIRL Formulation Definition 1. A cooperative inverse reinforcement learning (CIRL) game M is a two-player Markov game with identical payoffs between a human or principal, H, and a robot or agent, R. The game is described by a tuple,  M = S, {A H , A R }, T (•|•, •, •), {Θ, R(•, •, •; •)}, P 0 (•, •), γ , (s 0 , θ) γ a discount factor: γ ∈ [0, 1]. We write the reward for a state-parameter pair as R(s, a H , a R ; θ) to distinguish the static reward parameters θ from the changing world state s. The game proceeds as follows. First, the initial state, a tuple (s, θ), is sampled from P 0 . H observes θ, but R does not. This observation model captures the notion that only the human knows the reward function, while both actors know a prior distribution over possible reward functions. At each timestep t, H and R observe the current state s t and select their actions a H t , a R t . Both actors receive reward r t = R(s t , a H t , a R t ; θ) and observe each other's action selection. A state for the next timestep is sampled from the transition distribution, s t+1 ∼ P T (s |s t , a H t , a R t ), and the process repeats. Behavior in a CIRL game is defined by a pair of policies, (π H , π R ), that determine action selection for H and R respectively. In general, these policies can be arbitrary functions of their observation histories; π H : A H × A R × S * × Θ → A H , π R : A H × A R × S * → A R . The optimal joint policy is the policy that maximizes value. The value of a state is the expected sum of discounted rewards under the initial distribution of reward parameters and world states. Remark 1. A key property of CIRL is that the human and the robot get rewards determined by the same reward function. This incentivizes the human to teach and the robot to learn without explicitly encoding these as objectives of the actors.
 Structural Results for Computing Optimal Policy Pairs The analogue in CIRL to computing an optimal policy for an MDP is the problem of computing an optimal policy pair. This is a pair of policies that maximizes the expected sum of discounted rewards. This is not the same as 'solving' a CIRL game, as a real world implementation of a CIRL agent must account for coordination problems and strategic uncertainty  (Boutilier, 1999) . The optimal policy pair represents the best H and R can do if they can coordinate perfectly before H observes θ. Computing an optimal joint policy for a cooperative game is the solution to a decentralized-partially observed Markov decision process (Dec-POMDP). Unfortunately, Dec-POMDPs are NEXP-complete  (Bernstein et al., 2000)  so general Dec-POMDP algorithms have a computational complexity that is doubly exponential. Fortunately, CIRL games have special structure that reduces this complexity.  Nayyar et al. (2013)  shows that a Dec-POMDP can be reduced to a coordination-POMDP. The actor in this POMDP is a coordinator that observes all common observations and specifies a policy for each actor. These policies map each actor's private information to an action. The structure of a CIRL game implies that the private information is limited to H's initial observation of θ. This allows the reduction to a coordination-POMDP to preserve the size of the (hidden) state space, making the problem easier. Theorem 1. Let M be an arbitrary CIRL game with state space S and reward space Θ. There exists a (single-actor) POMDP M C with (hidden) state space S C such that |S C | = |S| • |Θ| and, for any policy pair in M , there is a policy in M C that achieves the same sum of discounted rewards. Theorem proofs can be found in the supplementary material. An immediate consequence of this result is that R's belief about θ is a sufficient statistic for optimal behavior. Corollary 1. Let M be a CIRL game. There exists an optimal policy pair (π H * , π R * ) that only depends on the current state and R's belief. Remark 2. In a general Dec-POMDP, the hidden state for the coordinator-POMDP includes each actor's history of observations. In CIRL, θ is the only private information so we get an exponential decrease in the complexity of the reduced problem. This allows one to apply general POMDP algorithms to compute optimal joint policies in CIRL. It is important to note that the reduced problem may still be very challenging. POMDPs are difficult in their own right and the reduced problem still has a much larger action space. That being said, this reduction is still useful in that it characterizes optimal joint policy computation for CIRL as significantly easier than Dec-POMDPs. Furthermore, this theorem can be used to justify approximate methods (e.g., iterated best response) that only depend on R's belief state.
 Apprenticeship Learning as a Subclass of CIRL Games A common paradigm for robot learning from humans is apprenticeship learning. In this paradigm, a human gives demonstrations to a robot of a sample task and the robot is asked to imitate it in a subsequent task. In what follows, we formulate apprenticeship learning as turn-based CIRL with a learning phase and a deployment phase. We characterize IRL as the best response (i.e., the policy that maximizes reward given a fixed policy for the other player) to a demonstration-by-expert policy for H. We also show that this policy is, in general, not part of an optimal joint policy and so IRL is generally a suboptimal approach to apprenticeship learning. Definition 2. (ACIRL) An apprenticeship cooperative inverse reinforcement learning (ACIRL) game is a turn-based CIRL game with two phases: a learning phase where the human and the robot take turns acting, and a deployment phase, where the robot acts independently.
 Example. Consider an example apprenticeship task where R needs to help H make office supplies. H and R can make paperclips and staples and the unobserved θ describe H's preference for paperclips vs staples. We model the problem as an ACIRL game in which the learning and deployment phase each consist of an individual action. The world state in this problem is a tuple (p s , q s , t) where p s and q s respectively represent the number of paperclips and staples H owns. t is the round number. An action is a tuple (p a , q a ) that produces p a paperclips and q a staples. The human can make 2 items total: A H = {(0, 2), (1, 1), (2, 0)}. The robot has different capabilities. It can make 50 units of each item or it can choose to make 90 of a single item: A R = {(0, 90), (50, 50), (90, 0)}. We let Θ = [0, 1] and define R so that θ indicates the relative preference between paperclips and staples:R(s, (p a , q a ); θ) = θp a + (1 − θ)q a . R's action is ignored when t = 0 and H's is ignored when t = 1. At t = 2, the game is over, so the game transitions to a sink state, (0, 0, 2). Deployment phase -maximize mean reward estimate. It is simplest to analyze the deployment phase first. R is the only actor in this phase so it get no more observations of its reward. We have shown that R's belief about θ is a sufficient statistic for the optimal policy. This belief about θ induces a distribution over MDPs. A straightforward extension of a result due to  Ramachandran & Amir (2007)  shows that R's optimal deployment policy maximizes reward for the mean reward function. Theorem 2. Let M be an ACIRL game. In the deployment phase, the optimal policy for R maximizes reward in the MDP induced by the mean θ from R's belief. In our example, suppose that π H selects (0, 2 ) if θ ∈ [0, 1 3 ), (1, 1) if θ ∈ [ 1 3 , 2 3 ] and (2, 0) otherwise. R begins with a uniform prior on θ so observing, e.g., a H = (0, 2) leads to a posterior distribution that is uniform on [0, 1 3 ). Theorem 2 shows that the optimal action maximizes reward for the mean θ so an optimal R behaves as though θ = 1 6 during the deployment phase. Learning phase -expert demonstrations are not optimal. A wide variety of apprenticeship learning approaches assume that demonstrations are given by an expert. We say that H satisfies the demonstration-by-expert (DBE) assumption in ACIRL if she greedily maximizes immediate reward on her turn. This is an 'expert' demonstration because it demonstrates a reward maximizing action but does not account for that action's impact on R's belief. We let π E represent the DBE policy. Theorem 2 enables us to characterize the best response for R when π H = π E : use IRL to compute the posterior over θ during the learning phase and then act to maximize reward under the mean θ in the deployment phase. We can also analyze the DBE assumption itself. In particular, we show that π E is not H's best response when π R is a best response to π E . Theorem 3. There exist ACIRL games where the best-response for H to π R violates the expert demonstrator assumption. In other words, if br(π) is the best response to π, then br(br(π E )) = π E . The supplementary material proves this theorem by computing the optimal equilibrium for our example. In that equilibrium, H selects (1, 1) if θ ∈ [ 41 92 , 51 92 ]. In contrast, π E only chooses (1, 1) if θ = 0.5. The change arises because there are situations (e.g., θ = 0.49) where the immediate loss of reward to H is worth the improvement in R's estimate of θ. Remark 3. We should expect experienced users of apprenticeship learning systems to present demonstrations optimized for fast learning rather than demonstrations that maximize reward. Crucially, the demonstrator is incentivized to deviate from R's assumptions. This has implications for the design and analysis of apprenticeship systems in robotics. Inaccurate assumptions about user behavior are notorious for exposing bugs in software systems (see, e.g.,  Leveson & Turner (1993) ).
 Generating Instructive Demonstrations Now, we consider the problem of computing H's best response when R uses IRL as a state estimator. For our toy example, we computed solutions exhaustively, for realistic problems we need a more efficient approach. Section 3.2 shows that this can be reduced to an POMDP where the state is a tuple of world state, reward parameters, and R's belief. While this is easier than solving a general Dec-POMDP, it is a computational challenge. If we restrict our attention to the case of linear reward functions we can develop an efficient algorithm to compute an approximate best response. Specifically, we consider the case where the reward for a state (s, θ) is defined as a linear combination of state features for some feature function φ : R(s, a H , a R ; θ) = φ(s) θ. Standard results from the IRL literature show that policies with the same expected feature counts have the same value  (Abbeel & Ng, 2004) . Combined with Theorem 2, this implies that the optimal π R under the DBE assumption computes a policy that matches the observed feature counts from the learning phase. This suggests a simple approximation scheme. To compute a demonstration trajectory τ H , first compute the feature counts R would observe in expectation from the true θ and then select actions that maximize similarity to these target features. If φ θ are the expected feature counts induced by θ then this scheme amounts to the following decision rule: τ H ← argmax τ φ(τ ) θ − η||φ θ − φ(τ )|| 2 . (1) This rule selects a trajectory that trades off between the sum of rewards φ(τ ) θ and the feature dissimilarity ||φ θ − φ(τ )|| 2 . Note that this is generally distinct from the action selected by the demonstration-by-expert policy. The goal is to match the expected sum of features under a distribution of trajectories with the sum of features from a single trajectory. The correct measure of feature  Lower numbers are better. Using the best response causes R to infer a better distribution over θ so it does a better job of maximizing reward. Right: The regret of the instructive demonstration policy as a function of how optimal R expects H to be. λ = 0 corresponds to a robot that expects purely random behavior and λ = ∞ corresponds to a robot that expects optimal behavior. Regret is minimized for an intermediate value of λ: if λ is too small, then R learns nothing from its observations; if λ is too large, then R expects many values of θ to lead to the same trajectory so H has no way to differentiate those reward functions. similarity is regret: the difference between the reward R would collect if it knew the true θ and the reward R actually collects using the inferred θ. Computing this similarity is expensive, so we use an 2 norm as a proxy measure of similarity.
 Experiments
 Cooperative Learning for Mobile Robot Navigation Our experimental domain is a 2D navigation problem on a discrete grid. In the learning phase of the game, H teleoperates a trajectory while R observes. In the deployment phase, R is placed in a random state and given control of the robot. We use a finite horizon H, and let the first H 2 timesteps be the learning phase. There are N φ state features defined as radial basis functions where the centers are common knowledge. Rewards are linear in these features and θ. The initial world state is in the middle of the map. We use a uniform distribution on [−1, 1] N φ for the prior on θ. Actions move in one of the four cardinal directions {N, S, E, W } and there is an additional no-op ∅ that each actor executes deterministically on the other agent's turn. Figure  1  shows an example comparison between demonstration-by-expert and the approximate best response policy in Section 3.4. The leftmost image is the ground truth reward function. Next to it are demonstration trajectories produce by these two policies. Each path is superimposed on the maximum a-posteriori reward function the robot infers from the demonstration. We can see that the demonstration-by-expert policy immediately goes to the highest reward and stays there. In contrast, the best response policy moves to both areas of high reward. The robot reward function the robot infers from the best response demonstration is much more representative of the true reward function, when compared with the reward function it infers from demonstration-by-expert.
 Demonstration-by-Expert vs Best Responder Hypothesis. When R plays an IRL algorithm that matches features, H prefers the best response policy from Section 3.4 to π E : the best response policy will significantly outperform the DBE policy. Manipulated Variables. Our experiment consists of 2 factors: H-policy and num-features. We make the assumption that R uses an IRL algorithm to compute its estimate of θ during learning and maximizes reward under this estimate during deployment. We use Maximum-Entropy IRL  (Ziebart et al., 2008)  to implement R's policy. H-policy varies H's strategy π H and has two levels: demonstration-by-expert (π E ) and best-responder (br). In the π E level H maximizes reward during the demonstration. In the br level H uses the approximate algorithm from Section 3.4 to compute an approximate best response to π R . The trade-off between reward and communication η is set by cross-validation before the game begins. The num-features factor varies the dimensionality of φacross two levels: 3 features and 10 features. We do this to test whether and how the difference between experts and best-responders is affected by dimensionality. We use a factorial design that leads to 4 distinct conditions. We test each condition against a random sample of N = 500 different reward parameters. We use a within-subjects design with respect to the the H-policy factor so the same reward parameters are tested for π E and br. Dependent Measures. We use the regret with respect to a fully-observed setting where the robot knows the ground truth θ as a measure of performance. We let θ be the robot's estimate of the reward parameters and let θ GT be the ground truth reward parameters. The primary measure is the regret of R's policy: the difference between the value of the policy that maximizes the inferred reward θ and the value of the policy that maximizes the true reward θ GT . We also use two secondary measures. The first is the KL-divergence between the maximum-entropy trajectory distribution induced by θ and the maximum-entropy trajectory distribution induced by θ. Finally, we use the 2 -norm between the vector or rewards defined by θ and the vector induced by θ GT . Results. There was relatively little correlation between the measures (Cronbach's α of .47), so we ran a factorial repeated measures ANOVA for each measure. Across all measures, we found a significant effect for H-policy, with br outperforming π E on all measures as we hypothesized (all with F > 962, p < .0001). We did find an interaction effect with num-features for KL-divergence and the 2 -norm of the reward vector but post-hoc Tukey HSD showed br to always outperform π E . The interaction effect arises because the gap between the two levels of H-policy is larger with fewer reward parameters; we interpret this as evidence that num-features = 3 is an easier teaching problem for H. Figure  2  (Left, Middle) shows the dependent measures from our experiment.
 Varying R's Expectations Maximum-Entropy IRL includes a free parameter λ that controls how optimal R expects H to behave. If λ = 0, R will update its belief as if H's observed behavior is independent of her preferences θ. If λ = ∞, R will update its belief as if H's behavior is exactly optimal. We ran a followup experiment to determine how varying λ changes the regret of the br policy. Changing λ changes the forward model in R's belief update: the mapping R hypothesizes between a given reward parameter θ and the observed feature counts φ θ . This mapping is many-to-one for extreme values of λ. λ ≈ 0 means that all values of θ lead to the same expected feature counts because trajectories are chosen uniformly at random. Alternatively, λ >> 0 means that almost all probability mass falls on the optimal trajectory and many values of θ will lead to the same optimal trajectory. This suggests that it is easier for H to differentiate different values of θ if R assumes she is noisily optimal, but only up until a maximum noise level. Figure  2  plots regret as a function of λ and supports this analysis: H has less regret for intermediate values of λ.
 Conclusion and Future Work In this work, we presented a game-theoretic model for cooperative learning, CIRL. Key to this model is that the robot knows that it is in a shared environment and is attempting to maximize the human's reward (as opposed to estimating the human's reward function and adopting it as its own). This leads to cooperative learning behavior and provides a framework in which to design HRI algorithms and analyze the incentives of both actors in a reward learning environment. We reduced the problem of computing an optimal policy pair to solving a POMDP. This is a useful theoretical tool and can be used to design new algorithms, but it is clear that optimal policy pairs are only part of the story. In particular, when it performs a centralized computation, the reduction assumes that we can effectively program both actors to follow a set coordination policy. This is clearly infeasible in reality, although it may nonetheless be helpful in training humans to be better teachers. An important avenue for future research will be to consider the coordination problem: the process by which two independent actors arrive at policies that are mutual best responses. Returning to Wiener's warning, we believe that the best solution is not to put a specific purpose into the machine at all, but instead to design machines that provably converge to the right purpose as they go along. Figure 2 : 2 Figure 2: Left, Middle: Comparison of 'expert' demonstration (π E ) with 'instructive' demonstration (br).
			 A coordination problem of the type described in Boutilier (1999)  arises if there are multiple optimal policy pairs; we defer this issue to future work.

Abstract: For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partialinformation game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.


Learning agents for uncertain environments (extended abstract)

In recent years, reinforcement learning (also called neurodynamic programming) has made rapid progress as an approachfor building agents automatically  (Sutton, 1988; Kaelbling et al., 1996; Bertsekas & Tsitsiklis, 1996) . The basic idea is that the performance measure is made available to the agent in the form of a rewardfunction specifying the reward for each state that the agent passes through. The performance measure is then the sum of the rewards obtained. For example, when a bumble bee forages, the reward function at each time step might be some combination of the distance flown (weighted negatively) and the nectar ingested. Reinforcement learning (RL) methods are essentially online algorithmd for solving Markovdecisionprocesses (MDPs). An MDP is defined by the reward function and a model, that is, the state transition probabilities conditioned on each possible action. RL algorithms can be model-based, where the agent learns a model, or model-free-e.g., Q-learning cite-Watkins: 1989, which learns just a function Q(s, a) specifying the long-term value of taking action a in state s and acting optimally thereafter. Despite their successes, RL methods have been restricted largely tofully observable MDPs, in which the sensory input at each state is sufficient to identify the state. Obviously, in the real world, we must often deal with partially observable MDPs (POMDPs).  Astrom (1965)  proved that optimal decisions in POMDPs depend on the belief state b at each point in time, i.e., the posterior probability distribution over all possible actual states, given all evidence to date. The functions V and Q then become functions of b instead of s.  Parr and Russell (1995)  describes a very simple POMDP RL algorithm using an explicit representation of b as a vector of probabilities, and  McCallum (1993)  shows a way to approximate the belief state using recent percept sequences. Neither approach is likely to scale up to situations with large numbers of state variables and long-term temporal dependencies. What is needed is a way of representing the model compactly and updating the belief state efficiently given the model and each new observation. Dynamic Bayesian networks  (Dean & Kanazawa, 1989)  seem to have some of the required properties; in particular, they have significant advantages over other approaches such as Kalman filters and hidden Markov models. Our baseline architecture, shown in Figure  1 , uses DBNs to represent and update the belief state as new sensor information arrives. Given a representation for b, the reward signal is used to learn a Q-function represented by some "black-box" function approximator such as a neural network. Provided we can handle hybrid (dis-Figure  1 : A baseline architecture for learning agents in uncertain environments crete+continuous) DPNs, and provided we have a learning algorithm that can construct an approximately correct DBN model from scratch, then this baseline architecture has the capacity, in principle, to be thrown into more or less any environment and to learn to behave reasonably.* The talk will cover a variety of research topics arising from this proposal: Parametric learning in DBNs  (Binder, Koller, Russell, & Kanazawa, 1997a) . Structural learning in DBNs  (Friedman, Murphy, & Russell, 1998) . Approximate inference in DBNs  (Kanazawa, Koller, & Russell, 1995; Boyen & Koller, 1998) . Space-efficient inference in DBNs  (Binder, Murphy, & Russell, 1997b) . Reinforcement learning with DBN models-that is, how to do Q-learning with the belief state information provided by the DBN. Some tentative ideas will be presented but as yet there are no convincing solutions. Scaling up the environment will inevitably overtax the resources of the baseline architecture. There are several obvious directions for improvement, including hierarchical and first-order models, hierarchical representations of behaviour  (Parr & Russell, 1998) , and model-based lookahead methods for decision making. Which of these is important in any particular class of environments can only be ascertained by experiment.
 Inverse reinforcement learning Reinforcement learning is a powerful method for adaptive control in real tasks, so it is natural to seek analogous mechanisms in nature. Connections have been made between reinforcement learning and operant conditioning models of animal learning (see, e.g.,  Schmajuk & Zanutto, 1997; Touretzky & Saksida, 1997) . There is also neurophysiological evidence that reinforcement learning occurs in bee foraging  (Montague et al., 1995)  and in songbird vocalization  (Doya & Sejnowski, 1995) . In this work, it is generally assumed that the reward function is fixed and known. For example, in experiments on bees it is assumed to be the rate of nectar ingestion: Montague et al. (1995) cite evidence of a "neuron with widespread projections to odour processing regions of the honeybee brain 'We say "more or less" because full generality require dealing with game-theoretic issues requiring stochastic decision making. whose activity represents the reward value of gustatory stimuli." It seems clear, however, that in examining animal and human behaviour we must consider the reward function as an unknown to be ascertained. The reasons for this are straightforward: l The specification of a given reward function is an empirical hypothesis and may turn out to be wrong. For example, it was assumed initially that horses' gait selection for a given speed was determined by energetic economy  (Hoyt & Taylor, 1981) ; this turns out not to be the case  (Farley & Taylor, 1991) . l The parameters of a multiattribute reward function can surely not be determined a priori; e.g., for running, attributes might be speed, efficiency, stability against perturbations, wear and tear on muscles, tendons, and bones, etc. How are these to be weighted and combined? Therefore, to model natural learning using reinforcement learning ideas, we must first solve the following computational task, which we call inverse reinforcement learning: Given 1) measurements of an agent's behaviour over time, in a variety of circumstances, 2) measurements of the sensory inputs to that agent; 3) a model of the physical environment (including the agent's body). Determine the reward function that the agent is optimizing. Given an assumption of optimization, this computational task is well-defined. Notice that is the dual of unsupervised reinforcement learning, where the task is to determine optimal behaviour given the reward inputs. To our knowledge, this computational task has not been studied in any generality in computer science, control theory, psychology, or biology. The closest work is in economics, where the task of multiattribute utility assessment has been studiedin depth-that is, how does a person actually combine the various attributes of each available choice when making a decision. The theory is well-developed  (Keeney & Raiffa, 1976) , and the applications numerous. However, this field studies only one-shot decisions where a single action is taken and the outcome is immediate. The sequential case was not considered until a seminal paper by  Sargent (1978)  tried to ascertain the effective hiring cost for labor by examining a firm's hiring behaviour over time, assuming it to be rational. In the last decade, the area of structural estimation of Markov decision processes has grown rapidly in econometrics  (Rust, 1994) . Many of the basic results carry over to our setting, although virtually nothing has been done on computational aspects, experimentation, or control-type applications. The open research problems are many: What are efficient algorithms for solving the inverse reinforcement learning problem? What is its computational complexity? Are there closed-form solutions for some parametric forms? Under what circumstances can we determine the existence of a consistent reward function? To what extent is the reward function uniquely recoverable? What effect do sensor and process noise have on robustness of the determination? What are appropriate error metrics for fitting? If behaviour is strongly inconsistent with optimality, can we identify "locally consistent" reward functions for specific regions in state space? Can we determine the reward function by observation during rather than after learning? How much observation is required to determine an estimated reward function that is within E of the true reward function? How can experiments be designed to maximize the identifiability of the reward function? Considering the design of possible algorithms, one can take maximum-likelihood approach to fit a parametric form for the reward functionas is commonly done in econometrics. That is, one defines a function L,(w)(B), the likelihood of observing behaviour B if the true reward function is r(w). From this, one can compute dWdw. One important question will be how to compute this gradient efficiently; presumably, it can be done in an obvious way by carrying the differential operator through the optimization algorithm for the behaviour. More elegant closed-form solutions may exist in special cases (e.g., linear-quadratic regulators). One may also be able to show that in some cases (e.g., linear reward functions) a globally optimal estimate can always be found. The solution of inverse reinforcement learning problems may also be an effective way to learn from observing experts. For tasks such as walking, diving, and driving, the designer of an artificial system may have only an intuitive idea of the appropriate reward function to be supplied to an RL algorithm in order to achieve "desirable" behavior. Instead of learning direct control functions from observation of experts (as in Pomerleau's ALVINN driging system), it may be better to solve the inverse reinforcement learning problem. The reward function should usually be a simple monotonic function of the current sensory inputs, and thus may be much simpler than the direct decision mapping itself. That is, the most compact and hence robustly learnable representation of expert behavior may be the reward function.

Abstract: This talk proposes a very simple "baseline architecture" for a learning agent that can handle stochastic, partially observable environments. The architecture uses reinforcement learning together with a method for representing temporal processes as graphical models. I will discuss methods for leaming the parameters and structure of such representations from sensory inputs, and for computing posterior probabilities. Some open problems remain before we can try out the complete agent; more arise when we consider scaling up. A second theme of the talk will be whether reinforcement learning can provide a good model of animal and human learning. To answer this question, we must do inverse reinforcement learning: given the observed behaviour, what reward signal, if any, is being optimized? This seems to be a very interesting problem for the COLT, UAI, and ML communities, and has been addressed in econometrics under the heading of structural estimation of Markov decision processes. 1 Learning in uncertain environments AI is about the construction of intelligent agents, i.e., systems that perceive and act effectively (according to some performance measure) in an environment. I have argued elsewhere  Russell and Norvig (1995)  that most AI research has focused on environments that are static, deterministic, discrete, and fully observable. What is to be done when, as in the real world, the environment is dynamic, stochastic, continuous, and partially observable? 'This paper draws on a variety of research efforts supported