Andrea Bajcsy
Dylan P Losey
Marcia K O'malley
Anca D Dragan

Learning Robot Objectives from Physical Human Interaction

Introduction Imagine a robot performing a manipulation task next to a person, like moving the person's coffee mug from a cabinet to the table (Fig.  1 ). As the robot is moving, the person might notice that the robot is carrying the mug too high above the table. Knowing that the mug would break if it were to slip and fall from so far up, the person easily intervenes and starts pushing the robot's end-effector down to bring the mug closer to the table. In this work, we focus on how the robot should then respond to such physical human-robot interaction (pHRI). Several reactive control strategies have been developed to deal with pHRI  [1, 2, 3] . For instance, when a human applies a force on the robot, it can render a desired impedance or switch to gravity compensation and allow the human to easily move the robot around. In these strategies, the moment the human lets go of the robot, it resumes its original behavior-our robot from earlier would go back to carrying the mug too high, requiring the person to continue intervening until it finished the task (Fig.  1, left ). Although such control strategies guarantee fast reaction to unexpected forces, the robot's return to its original motion stems from a fundamental limitation of traditional pHRI strategies: they miss the fact that human interventions are often intentional and occur because the robot is doing something wrong. While the robot's original behavior may have been optimal with respect to the robot's pre-defined objective function, the fact that a human intervention was necessary implies that this objective function was not quite right. Our insight is that because pHRI is intentional, it is also informative-it provides observations about the correct robot objective function, and the robot can leverage these observations to learn that correct objective. Returning to our example, if the person is applying forces to push the robot's end-effector closer to the table, then the robot should change its objective function to reflect this preference, and complete the rest of the current task accordingly, keeping the mug lower (Fig.  1 , right). Ultimately, human interactions should not be thought of as disturbances, which perturb the robot from its desired behavior, but rather as corrections, which teach the robot its desired behavior. In this paper, we make the following contributions: Formalism. We formalize reacting to pHRI as the problem of acting in a dynamical system to optimize an objective function, with two caveats: 1) the objective function has unknown parameters θ θ θ Figure  1 : A person interacts with a robot that treats interactions as disturbances (left), and a robot that learns from interactions (right). When humans are treated as disturbances, force plots reveal that people have to continuously interact since the robot returns to its original, incorrect trajectory. In contrast, a robot that learns from interactions requires minimal human feedback to understand how to behave (i.e., go closer to the table). θ, and 2) human interventions serve as observations about these unknown parameters: we model human behavior as approximately optimal with respect to the true objective. As stated, this problem is an instance of a Partially Observable Markov Decision Process (POMDP). Although we cannot solve it in real-time using POMDP solvers, this formalism is crucial to converting the problem of reacting to pHRI into a clearly defined optimization problem. In addition, our formalism enables pHRI approaches to be justified and compared in terms of this optimization criterion. Online Solution. We introduce a solution that adapts learning from demonstration approaches to our online pHRI setting  [4, 5] , but derive it as an approximate solution to the problem above. This enables the robot to adapt to pHRI in real-time, as the current task is unfolding. Key to this approximation is simplifying the observation model: rather than interpreting instantaneous forces as noisy-optimal with respect to the value function given θ, we interpret them as implicitly inducing a noisy-optimal desired trajectory. Reasoning in trajectory space enables an efficient approximate online gradient approach to estimating θ. User Study. We conduct a user study with the JACO2 7-DoF robotic arm to assess how online learning from physical interactions during a task affects the robot's objective performance, as well as subjective participant perceptions. Overall, our work is a first step towards learning robot objectives online from pHRI.
 Related Work We propose using pHRI to correct the robot's objective function while the robot is performing its current task. Prior research has focused on (a) control strategies for reacting to pHRI without updating the robot's objective function, or (b) learning the robot's objectives-from offline demonstrationsin a manner that generalizes to future tasks, but does not change the behavior during the current task. An exception is shared autonomy work, which does correct the robot's objective function online, but only when the objective is parameterized by the human's desired goal in free-space. Control Strategies for Online Reactions to pHRI. A variety of control strategies have been developed to ensure safe and responsive pHRI. They largely fall into three categories  [6] : impedance control, collision handling, and shared manipulation control. Impedance control  [1]  relates deviations from the robot's planned trajectory to interaction torques. The robot renders a virtual stiffness, damping, and/or inertia, allowing the person to push the robot away from its desired trajectory, but the robot always returns to its original trajectory after the interaction ends. Collision handling methods  [2]  include stopping, switching to gravity compensation, or re-timing the planned trajectory if a collision is detected. Finally, shared manipulation  [3]  refers to role allocation in situations where the human and the robot are collaborating. These control strategies for pHRI work in real-time, and enable the robot to safely adapt to the human's actions; however, the robot fails to leverage these interventions to update its understanding of the task-left alone, the robot would continue to perform the task in the same way as it had planned before any human interactions. By contrast, we focus on enabling robots to adjust how they perform the current task in real time. Offline Learning of Robot Objective Functions. Inverse Reinforcement Learning (IRL) methods focus explicitly on inferring an unknown objective function, but do it offline, after passively observing expert trajectory demonstrations  [7] . These approaches can handle noisy demonstrations  [8] , which become observations about the true objective  [9] , and can acquire demonstrations through physical kinesthetic teaching  [10] . Most related to our work are approaches which learn from corrections of the robot's trajectory, rather than from demonstrations  [4, 5, 11] . Our work, however, has a different goal: while these approaches focus on the robot doing better the next time it performs the task, we focus on the robot completing its current task correctly. Our solution is analogous to online Maximum Margin Planning  [4]  and co-active learning  [5]  for this new setting, but one of our contributions is to derive their update rule as an approximation to our pHRI problem. Online Learning of Human Goals. While IRL can learn the robot's objective function after one or more demonstrations of a task, online inference is possible when the objective is simply to reach a goal state, and the robot moves through free space  [12, 13, 14] . We build on this work by considering general objective parameters; this requires a more complex (non-analytic and difficult to compute) observation model, along with additional approximations to achieve online performance. 3 Learning Robot Objectives Online from pHRI
 Formalizing Reacting to pHRI We consider settings where a robot is performing a day-to-day task next to a person, but is not doing it correctly (e.g., is about to spill a glass of water), or not doing it in a way that matches the person's preferences (e.g., is getting too close to the person). Whenever the person physically intervenes and corrects the robot's motion, the robot should react accordingly; however, there are many strategies the robot could use to react. Here, we formalize the problem as a dynamical system with a true objective function that is known by the person but not known by the robot. This formulation interprets the human's physical forces as intentional, and implicitly defines an optimal strategy for reacting. Notation. Let x denote the robot's state (its position and velocity) and u R the robot's action (the torque it applies at its joints). The human physically interacts with the robot by applying external torque u H . The robot transitions to a next state defined by its dynamics, ẋ = f (x, u R + u H ), where both the human and robot can influence the robot's motion. POMDP Formulation. The robot optimizes a reward function r(x, u R , u H ; θ), which trades off between correctly completing the task and minimizing human effort r(x, u R , u H ; θ) = θ T φ(x, u R , u H ) − λ||u H || 2 (1) Following prior IRL work  [15, 4, 8] , we parameterize the task-related part of this reward function as a linear combination of features φ with weights θ. Note that we assume the relevant set of features for each task are given, and we will not explore feature selection within this work. Here θ encapsulates the true objective, such as moving the glass slowly, or keeping the robot's endeffector farther away from the person. Importantly, this parameter is not known by the robot-robots will not always know the right way to perform a task, and certainly not the human-preferred way. If the robot knew θ, this would simply become an MDP formulation, where the states are x, the actions are u R , the reward is r, and the person would never need to intervene. Uncertainty over θ, however, turns this into a POMDP formulation, where θ is a hidden part of the state. Importantly, the human's actions are observations about θ under some observation model P (u H |x, u R ; θ). These observations u H are atypical in two ways: (a) they affect the robot's reward, as in  [13] , and (b) they influence the robot's state, but we don't necessarily want to account for that when planning -the robot should not rely on the human to move the robot; rather the robot should consider u H only for its information value. Observation Model. We model the human's interventions as corrections which approximately maximize the robot's reward. More specifically, we assume the noisy-rational human selects an action u H that, when combined with the robot's action u R , leads to a high Q-value (state-action value) assuming the robot will behave optimally after the current step (i.e., assuming the robot knows θ) P (u H | x, u R ; θ) ∝ e Q(x,u R +u H ;θ) (2) Our choice of (2) stems from maximum entropy assumptions  [8] , as well as the Bolzmann distributions used in cognitive science models of human behavior  [16] . Aside. We are not formulating this as a POMDP to solve it using standard POMDP solvers. Instead, our goal is to clarify the underlying problem formulation and the existence of an optimal strategy.
 Approximate Solution Since POMDPs cannot be solved tractably for high-dimensional real-world problems, we make several approximations to arrive at an online solution. We first separate estimation from finding the optimal policy, and approximate the policy by separating planning from control. We then simplify the estimation model, and use maximum a posteriori estimate (MAP) instead the full belief over θ. QMDP. Similar to  [13] , we approximate our POMDP using a QMDP by assuming the robot will obtain full observability at the next time step  [17] . Let b denote the robot's current belief over θ. The QMDP simplifies into two subproblems: (a) finding the robot's optimal policy given b Q(x, u R , b) = b(θ)Q(x, u R , θ)dθ (3) where arg max u R Q(x, u R , b) evaluated at every state yields the optimal policy, and (b) updating our belief over θ given a new observation. Unlike the actual POMDP solution, here the robot will not try to gather information. From Belief to Estimator. Rather than planning with the belief b, we plan with only the MAP of θ. From Policies to Trajectories (Action). Computing Q in continuous state, action, and belief spaces is still not tractable. We thus separate planning and control. At every time step t, we do two things. First, given our current θt , we replan a trajectory ξ = x 0:T ∈ Ξ that optimizes the task-related reward. Let θ T Φ(ξ) be the cumulative reward, where Φ(ξ) is the total feature count along trajectory ξ such that Φ(ξ) = x t ∈ξ φ(x t ). We use a trajectory optimizer  [18]  to replan the robot's desired trajectory ξ t R ξ t R = arg max ξ θt • Φ(ξ) (4) Second, once ξ t R has been planned, we control the robot to track this desired trajectory. We use impedance control, which allows people to change the robot's state by exerting torques, and provides compliance for human safety  [19, 6, 1] . After feedback linearization  [20] , the equation of motion under impedance control becomes M R (q t − qt R ) + B R ( qt − qt R ) + K R (q t − q t R ) = u t H (5) Here M R , B R , and K R are the desired inertia, damping, and stiffness, x = (q, q), where q is the robot's joint position, and q R ∈ ξ R denotes the desired joint position. Within our experiments, we implemented a simplified impedance controller without feedback linearization u t R = B R ( qt R − qt ) + K R (q t − q t R ) (6) Aside. When the robot is not updating its estimate θ, then ξ t R = ξ t−1 R , and our solution reduces to using impedance control to track an unchanging trajectory  [2, 19] . From Policies to Trajectories (Estimation). We still need to address the second QMDP subproblem: updating θ after each new observation. Unfortunately, evaluating the observation model (2) for any given θ is difficult, because it requires computing the Q-value function for that θ. Hence, we will again leverage a simplification from policies to trajectories in order to update our MAP of θ. Instead of attempting to directly relate u H to θ, we propose an intermediate step; we interpret each human action u H via a intended trajectory, ξ H , that the human wants the robot to execute. To compute the intended trajectory ξ H from ξ R and u H , we propagate the deformation caused by u H along the robot's current trajectory ξ R ξ H = ξ R + µA −1 U H ( 7 ) where µ > 0 scales the magnitude of the deformation, A defines a norm on the Hilbert space of trajectories and dictates the deformation shape  [21] , U H = u H at the current time, and U H = 0 at all other times. During experiments we here used a norm A based on acceleration  [21] , but we will explore learning the choice of this norm in future work. Importantly, our simplification from observing human action u H to implicitly observing the human's intended trajectory ξ H means we no longer have to evaluate the Q-value of u R + u H given some θ value. Instead, the observation model now depends on the total reward of the implicitly observed trajectory: P (ξ H | ξ R , θ) ∝ e θ T Φ(ξ H )−λ||u H || 2 ≈ e θ T Φ(ξ H )−λ||ξ H −ξ R || 2 (8 ) This is analogous to  (2) , but in trajectory space-a distribution over implied trajectories, given θ and the current robot trajectory.
 Online Update of the θ Estimate The probability distribution over θ at time step t is P (ξ 0 H , .., ξ t H |θ, ξ 0 R , .., ξ t R )P (θ). However, since θ is continuous, and the observation model is not Gaussian, we opt not to track the full belief, but rather to track the maximum a posteriori estimate (MAP). Our update rule for this estimate will reduce to online Maximum Margin Planning  [4]  if we treat ξ H as the demonstration, and to coadaptive learning  [5] , if we treat ξ H as the original trajectory with one waypoint corrected. One of our contributions, however, is to derive this update rule from our MaxEnt observation model in  (8) . MAP. Assuming the observations are conditionally independent given θ, the MAP for time t + 1 is θt+1 = arg max θ P (ξ 0 H , .., ξ t H | ξ 0 R , .., ξ t R , θ)P (θ) = arg max θ t τ =0 log P (ξ τ H | ξ τ R , θ) + log P (θ) (9) Inspecting the right side of (9), we need to define both P (ξ H |ξ R , θ) and the prior P (θ). To approximate P (ξ H |ξ R , θ), we use (8) with Laplace's method to compute the normalizer. Taking a second-order Taylor series expansion of the objective function about ξ R , the robot's current best guess at the optimal trajectory, we obtain a Gaussian integral that can be evaluated in closed form P (ξ H | ξ R , θ) = e θ T Φ(ξ H )−λ||ξ H −ξ R || 2 e θ T Φ(ξ)−λ||ξ−ξ R || 2 dξ ≈ e θ T Φ(ξ H )−Φ(ξ R ) −λ||ξ H −ξ R || 2 (10) Let θ0 be our initial estimate of θ. We propose the prior P (θ) = e − 1 2α ||θ− θ0 || 2 (11) where α is a positive constant. Substituting  (10)  and (  11 ) into (  9 ), the MAP reduces to θt+1 ≈ arg max θ t τ =0 θ T Φ(ξ τ H ) − Φ(ξ τ R ) − 1 2α ||θ − θ0 || 2 (12) Notice that the λ||ξ H −ξ R || 2 terms drop out, because this penalty for human effort does not explicitly depend on θ. Solving the optimization problem (  12 ) by taking the gradient with respect to θ, and then setting the result equal to zero, we finally arrive at θt+1 = θ0 + α t τ =0 Φ(ξ τ H ) − Φ(ξ τ R ) = θt + α Φ(ξ t H ) − Φ(ξ t R ) (13) Interpretation. This update rule is actually the online gradient  [22]  of (9) under our Laplace approximation of the observation model. It has an intuitive interpretation: it shifts the weights in the direction of the human's intended feature count. For example, if ξ H stays farther from the person than ξ R , the weights in θ associated with distance-to-person features will increase. Relation to Prior Work. This update rule is analogous to two related works. First, it would be the online version of Maximum Margin Planning (MMP)  [4]  if the trajectory ξ t H were a new demon- stration. Unlike MMP, our robot does not complete a trajectory, and only then get a full new demonstration; instead, our ξ t H is an estimate of the human's intended trajectory based on the force applied during the robot's execution of the current trajectory ξ t R . Second, the update rule would be co-active learning  [5]  if the trajectory ξ t H were ξ t R with one waypoint modified, as opposed to a propagation of u t H along the rest of ξ t R . Unlike co-active learning, however, our robot receives corrections continually, and continually updates the current trajectory in order to complete the current task well. Nonetheless, we are excited to see similar update rules emerge from different optimization criteria. Summary. We formalized reacting to pHRI as a POMDP with the correct objective parameters as a hidden state, and approximated the solution to enable online learning from physical interaction. At every time step during the task where the human interacts with the robot, we first propagate u H to implicitly observe the corrected trajectory ξ H (simplification of the observation model), and then update θ via Equation (  13 ) (MAP instead of belief). We replan with the new estimate (approximation of the optimal policy), and use impedance control to track the resulting trajectory (separation of planning from control). We summarize and visualize this process in Fig.  2 .
 User Study We conducted an IRB-approved user study to investigate the benefits of in-task learning. We designed tasks where the robot began with the wrong objective function, and participants phsyically corrected the robot's behavior 1 .
 Experiment Design Independent Variables. We manipulated the pHRI strategy with two levels: learning and impedance. The robot either used our method (Algorithm 1) to react to physical corrections and re-plan a new trajectory during the task; or used impedance control (our method without updating θ) to react to physical interactions and then return to the originally planned trajectory. Dependent Measures. We measured the robot's performance with respect to the true objective, along with several subjective measures. One challenge in designing our experiment was that each person might have a different internal objective for any given task, depending on their experience and preferences. Since we do not have direct access to every person's internal preferences, we defined the true objective ourselves, and conveyed the objectives to participants by demonstrating the desired optimal robot behavior (see an example in Fig.  3(a) , where the robot is supposed to keep the cup upright). We instructed participants to get the robot to achieve this desired behavior with minimal human physical intervention. For each robot attempt at a task, we evaluated the task related and effort related parts of the objective: θ T Φ(ξ) (a cost to be minimized and not a reward to be maximized in our experiment) and t ||u t H || 1 . We also evaluate the total amount of time spent interacting physically with the robot. For our subjective measures, we designed 4 multi-item scales shown in Table  1 : did participants think the robot understood how they wanted to task done, did they feel like they had to exert a lot of effort to correct the robot, was it easy to anticipate the robot's reactions, and how good of a collaborator was the robot. Hypotheses: H1. Learning significantly decreases interaction time, effort, and cumulative trajectory cost. H2. Participants will believe the robot understood their preferences, feel less interaction effort, and perceive the robot as more predictable and more collaborative in the learning condition. Tasks. We designed three household manipulation tasks for the robot to perform in a shared workspace (see Fig.  3 ), plus a familiarization task. As such, the robot's objective function considered two features: velocity and a task-specific feature. For each task, the robot carried a cup from a start to a goal pose with an initially incorrect objective, requiring participants to correct its behavior during the task. During the familiarization task, the robot's original trajectory moved too close to the human. Participants had to physically interact with the robot to get it to keep the cup further away from their body. In Task 1, the robot would not care about tilting the cup mid-task, risking spilling if the cup was too full. Participants had to get the robot to keep the cup upright. In Task 2, the robot would move the cup too high in the air, risking breaking it if it were to slip, and participants had to get the robot to keep it closer to the table. Finally, in Task 3, the robot would move the cup over a laptop to reach it's final goal pose, and participants had to get the robot to keep the cup away from the laptop. Participants. We used a within-subjects design and counterbalanced the order of the pHRI strategy conditions. In total, we recruited 10 participants (5 male, 5 female, aged 18-34) from the UC Berkeley community, all of whom had technical backgrounds. Procedure. For each pHRI strategy, participants performed the familiarization task, followed by the three tasks, and then filled out our survey. They attempted each task twice with each strategy for robustness, and we recorded the attempt number for our analysis. Since we artificially set the true objective for participants to measure objective performance, we showed participants both the original and desired robot trajectory before interaction (Fig.  3 ), so that they understood the objective.
 Results Objective. We conducted a factorial repeated measures ANOVA with strategy (impedance or learning) and trial number (first attempt or second attempt) as factors, on total participant effort, interaction time, and cumulative true cost 2 (see Figure  4  and Figure  5 ). Learning resulted in significantly less interaction force (F (1, 116) = 86.29, p < 0.0001) and interaction time (F (1, 116) = 75.52, p < 0.0001), and significantly better task cost (F (1, 116) = 21.85, p < 0.0001). Interestingly, while trial number did not significantly affect participant's performance with either method, attempting the task a second time yielded a marginal improvement for the impedance strategy, but not for the learning strategy. This may suggest that it is easier to get used to the impedance strategy. Overall, this supports H1, and aligns with the intuition that if humans are truly intentional actors, then using interaction forces as information about the robot's objective function enables robots to better complete their tasks with less human effort compared to traditional pHRI methods. Subjective. Table  1  shows the results of our participant survey. We tested the reliability of our 4 scales, and found the understanding, effort, and collaboration scales to be reliable, so we grouped them each into a combined score. We ran a one-way repeated measures ANOVA on each resulting score. We found that the robot using our method was perceived as significantly (p < 0.0001) more understanding, less difficult to interact with, and more collaborative. However, we found no significant difference between our method and the baseline impedance method in terms of predictability. Participant comments suggest that while the robot adapted quickly to their corrections when learning (e.g. "The robot seemed to quickly figure out what I cared about and kept doing it on its own"), determining what the robot was doing during learning was less apparent (e.g. "If I pushed it hard enough sometimes it would seem to fall into another mode and then do things correctly"). Therefore, H2 was partially supported: although our learning algorithm was not perceived as more predictable, participants believed that the robot understood their preferences more, took less effort to interact with, and was a more collaborative partner.  Cup Table Laptop Task
 Discussion Summary. We propose that robots should not treat human interaction forces as disturbances, but rather as informative actions. We show that this results in robots capable of in-task learningrobots that update their understanding of the task which they are performing and then complete it correctly, instead of relying on people to guide them until the task is done. We test this concept with participants who not only teach the robot to finish its task according to their preferences, but also subjectively appreciate the robot's learning. Limitations and Future Work. Ours is merely a step in exploring learning robot objectives from pHRI. We opted for an approximation closest to the existing literature, but other possible better online solutions are possible. In our user study, we assumed knowledge of the two relevant reward features. In reality, reward functions will have larger feature sets and human interactions may only give information about a certain subset of relevant weights. The robot will thus need to disambiguate what the person is trying to correct, likely requiring active information gathering. Further, developing solutions that can handle dynamical aspects, like preferences about the timing of the motion, would require a different approach to inferring the intended human trajectory, or going back the space of policies altogether. Finally, while we focused on in-task learning, the question of how and when to generalize learned objectives to new task instances remains open. Figure 2 : 2 Figure 2: Algorithm (left) and visualization (right) of one iteration of our online learning from pHRI method in an environment with two obstacles O1, O2. The originally planned trajectory, ξ t R (black dotted line), is deformed by the human's force into the human's preferred trajectory, ξ t H (solid black line). Given these two trajectories, we compute an online update of θ and can replan a better trajectory ξ t+1 R (orange dotted line).
 (a) Task 1 :Figure 3 : 13 Figure 3: Simulations depicting the robot trajectories for each of the three experimental tasks. The black path represents the original trajectory and the blue path represents the human's desired trajectory.
 Figure 4 : 4 Figure4: Learning from pHRI decreases human effort and interaction time across all experimental tasks (total trajectory time was 15s). An asterisk (*) means p < 0.0001.
 Figure 5 : 5 Figure 5: (left) Average cumulative cost for each task as compared to the desired total trajectory cost. An asterisk (*) means p < 0.0001. (right) Plot of sample participant data from laptop task: desired trajectory is in blue, trajectory with impedance condition is in gray, and learning condition trajectory is in orange.
 Table 1 : 1 QuestionsCronbach's α Imped LSM Learn LSM F(1,9) p-value understanding By the end, the robot understood how I wanted it to do the task. Even by the end, the robot still did not know how I wanted it to do the task.The robot learned from my corrections.The robot did not understand what I was trying to accomplish. The robot did not collaborate with me to complete the task. Results of ANOVA on subjective metrics collected from a 7-point Likert-scale survey. 0.94 1.70 5.10 118.56 <.0001
			 For video footage of the experiment, see: https://www.youtube.com/watch?v=1MkI6DH1mcw
			 For simplicity, we only measured the value of the feature that needed to be modified in the task, and computed the absolute difference from the feature value of the optimal trajectory.

Abstract: When humans and robots work in close proximity, physical interaction is inevitable. Traditionally, robots treat physical interaction as a disturbance, and resume their original behavior after the interaction ends. In contrast, we argue that physical human interaction is informative: it is useful information about how the robot should be doing its task. We formalize learning from such interactions as a dynamical system in which the task objective has parameters that are part of the hidden state, and physical human interactions are observations about these parameters. We derive an online approximation of the robot's optimal policy in this system, and test it in a user study. The results suggest that learning from physical interaction leads to better robot task performance with less human effort.


Approval-directed agency and the decision theory of Newcomb-like problems

Introduction In decision theory, there is a large debate about how an instrumentally rational agent, i.e., an agent trying to achieve some goal or maximize some utility function, should decide in Newcomb's problem (introduced by  Nozick 1969 ) and variations thereof (a list is given by  Ledwig 2000, pp. 80-87) . Consequently, different normative theories of instrumental rationality have been developed. The best known ones are evidential (sometimes also called Bayesian) decision theory (EDT)  (Ahmed 2014; Almond 2010; Price 1986; Horgan 1981 ) and causal decision theory (CDT)  (Gibbard and Harper 1981; Joyce 1999; Lewis 1981; Skyrms 1982; Weirich 2016 ), but many have attempted to remediate what they view as failures of the two theories by proposing further alternatives  (Spohn 2003 (Spohn , 2012 Poellinger 2013; Arntzenius 2008; Gustafsson 2011; Wedgwood 2013; Dohrn 2015; Price 2012; Soares and Levinstein 2017) . Because the main goal of artificial intelligence is to build machines that make instrumentally rational decisions  (Russell and Norvig 2010, Sects. 1.1.4, 2.2; Legg and Hutter 2007; Doyle 1992) , this normative disagreement has some bearing on how to build these machines (cf. Soares and Fallenstein 2014a, Sect. 2.2;  Soares and Fallenstein 2014b, Sect. 1; Bostrom 2014b, Chap. 13, Sect. "Decision theory") . The differences between these decision theories are probably inconsequential in most situations  (Ahmed 2014, Sect. 0.5, Chap. 4; Briggs 2017 ), 1 but still matter in some  (Ahmed 2014, Chap. 4-6; Soares 2014a; Bostrom 2014a) . In fact, AI may expose the differences more often. For example, Newcomb's problem and the prisoner's dilemma with a replica  (Kuhn 2017, Sect. 7 ) are easy to implement for agents with copyable source code (cf.  Yudkowsky 2010 pp. 85ff .  Soares and Fallenstein 2014b, Sect. 2; Soares 2014b; Cavalcanti 2010; Sect. 5) . Indeed, the existence of many copies is the norm for (successful) software, including AI-based software. While copies of present-day software systems may only interact with each other in rigid, explicitly pre-programmed ways, future AI-based systems will make decisions in a more autonomous, flexible and goal-driven way. Overall, the decision theory of Newcomb-like scenarios is a central foundational issue which will plausibly become practically important in the longer term. The problem for AI research posed by the disagreement among decision theorists can be divided into two questions: 1. What decision theory do we want an AI to follow? 2. How could we implement such a decision theory in an AI? Or: How do decision theories and AI frameworks or architectures map onto each other? Although it certainly requires further discussion, there already is a large literature related to the first question. 2 In this paper, I would thus like to draw attention to the second question. Specifically, I would like to investigate how approval-directed agents behave in Newcomb-like problems. By an approval-directed agent, I mean an agent that is coupled with an overseer. After the agent has chosen an action, the overseer scores the agent for that action. Rather than, say, trying to bring about particular states in the environment, the agent chooses actions so as to maximize the score it receives from the overseer (cf.  Christiano 2014) . A model of approval-directed agency that allows us to describe Newcomb-like situations is described and discussed in Sect. 2. Approval-directed agency is intended as a model of reinforcement learning agents (see  Sutton and Barto 1998; Russell and Norvig 2010; Chaps. 17, 21 , for introductions to reinforcement learning), for whom the reward function is analogous to the approval-directed agent's overseer. Since reinforcement learning is such a general and commonly studied problem in artificial intelligence  (Hutter 2005, e.g. Chap. 4.1.3; Russell and Norvig 2010, p. 831; Sutton and Barto 1998, Chap. 1) , it is an especially attractive target for modeling. 3 However, because decision theories are usually defined only for single decisions, we will only discuss single decisions whereas reinforcement learning is usually concerned with sequential interactions of agent and environment. However, this decision can also be a policy choice to model sequential decision problems.  4  In addition to limiting our analysis to single decisions, we will not discuss the learning process and simply assume that the agent has already formed some model of the world. If we assume that, after an action has been taken, the overseer rewards the agent based on the expected value of some von Neumann-Morgenstern utility function, the agent is implicitly driven by two decision theories: The overseer can use the regular conditional expectation or the causal expectation to estimate the value of its utility function; and the agent itself can follow CDT or EDT when maximizing the score it receives from the overseer (Sect. 3). We then show how the overall decision theory depends on these two potentially conflicting decision theories. If the overseer bases its expected value calculations on looking only at the world, then the agent's decision theory is decisive. If the overseer Footnote 2 continued  (Meacham 2010; Soares and Fallenstein 2014b, Sect. 3; Yudkowsky 2010, Sect. 2; Greene 2018)  . The same arguments imply that even if one is convinced of CDT or EDT one would not want the AI to use CDT and EDT. That said, one could also leave the self-modification to the AI.  3  Reinforcement learning and approval-directed agency are also common outside of artificial intelligence. For example,  Achen and Bartels (2016, Chap. 4 ) review evidence which shows that electorates often vote retrospectively to punish or reward incumbents.  1  A causal model of an approval-directed agent in a Newcomb-like decision problem. A denotes the agent's action, H the environment history, O the observation on which the overseer bases the reward, R is that reward, and E r is information about the way the reward is computed that is only available to the overseer. The box is used to indicate that H includes the two random variables H p and H f . All of H may have a causal influence on O bases its estimates only on the agent's action, then the overseer's decision (or perhaps rather action evaluation) theory is decisive.
 A H p H f R O E r H Fig.
 Approval-directed agency We first describe a model of approval-directed agency. To be able to apply both CDT and EDT, we will use causal models in  Pearl's (2009)  sense. Consequently, we use Pearl's do-calculus-based version of CDT  (Pearl 2009, Chap. 4 ). We will, throughout this paper, assume that the agent has already formed a (potentially implicit) model of the world 5 -e.g., based on past interactions with the environment. Also, we will only consider single decisions rather than sequential problems of iterative interaction between agent and environment. A causal model of such a one-shot Newcomb problem from the perspective of the approval-directed agent is given in Fig.  1 . In this model, the agent decides to take some action A, which may causally affect some part of the environment history, i.e., the history of states, H . We will call that part of the history the agent's causal future H f . Furthermore, the agent may be causally influenced by some other part of the environment history, which we will call the agent's causal past H p . H may contain information other than H f and H p , which we will assume to be independent of A.  6  The overseer, physically realized by, e.g., some module physically attached to the agent or a human supervisor, observes the agent's action and partially, via some percept O, the state of the world 7 . The overseer then calculates the reward R. To set proper incentives to the agent, we will assume the overseer to know not only the action and observation, but also everything that the agent knows (cf.  Christiano 2016) . The overseer may also have access to some additional piece of information E r about the way the reward is to be calculated. 8 Lastly, we assume that the sets of possible values of A, O and E r are finite. In principle, the overseer could reward the agent in all kinds of ways. E.g., it could reward the agent "deontologically" (Alexander and Moore 2016) for taking a particular action independently of the consequences of taking that action. In this paper, we will assume that the reward estimates the value of some von Neumann-Morgenstern utility function U that only depends on states of the world. I use the capital U to indicate that the utility function, too, is a random variable (in the Bayesian sense). For simplicity's sake, we will, again, assume that the set of possible values of U is finite. We will view U as representing the system designer's preferences over world states. 9 While other ways of assigning the reward are possible, this is certainly an attractive way of getting an approval-directed agent to achieve goals that we want it to achieve. After all, in real-world applications, we will usually care about the outcomes of the agent's decisions, such as whether a car has reached its destination in time or whether a human has been hurt. The standard way of estimating U (H ) (or any quantity for that matter) is the familiar conditional expectation. Thus, the overseer may compute the reward as r = E [U (H ) | e r , a, o] , (1) Footnote 6 continued know your common source code), the dependence persists. We exclude these dependences because such situations cannot be modeled by standard causal graphs. However, we could adapt causal graphs to accomodate for these kinds of dependences. First, we could modify our definition of causality in such a way that dependence does imply causation, as has been proposed by  Spohn (2003 Spohn ( , 2012 ,  Yudkowsky (2010)  and others. For instance, we could model the dependence between the outputs of two instances of an algorithm by introducing a logical node as a common cause of the two. This logical node would then represent the output of the abstract algorithm that the two copies implement. While changes to the concept of causation may affect CDT's implied behavior, the results from this paper can be directly transfered to such modifications. Alternatively, we could extend causal graphs to also include non-causal dependences (cf.  Poellinger 2013) . Such extension necessitates a new CDT formalism, so the proofs from this paper do not directly transfer to this case. That said, I expect our results to generalize given that both EDT and CDT would probably treat non-causal dependences on the action just like they treat causal arrows directed toward the action.  7  Christiano (2014) does not define approval-directed agency formally, but judging from a comment he made at https://medium.com/paulfchristiano/i-agree-that-the-key-feature-of-approval-directed-agents-isthat-the-causal-picture-is-736b4474910e, he considers it crucial to his conception that the overseer only looks at the agent's action and does not observe the action's consequences (cf. the distinction introduced in Sect. 3).  8  One reason for the overseer to have access to such additional information is that some of the human supervisor's values may not be expressible in a way that the approval-directed agent's algorithm can utilize (cf.  Muehlhauser and Helm 2012, Sects. 3, 4, 5.3 ). 9 Some have tried to modify the reward relative to the designer's preferences to make the reinforcement learning problem easier to solve  (Sorg 2011) , although  Sutton and Barto (1998, Sect. 3 .2) explicitly discourage such tricks in their reinforcement learning textbook. where r , a, e r , and o are values of R, A, E r , and O, respectively. 10 A causal decision theorist overseer agrees that after an action a is taken the righthand side of Eq. 1 most accurately estimates how much utility is achieved. She merely thinks that this term should not be used to decide which action a to take in the first place. 11 However, this puts a causal decision theorist overseer in a peculiar situation. Whatever formula she uses to compute the reward will also be used by the rewardmaximizing agent to decide which action to take. A causal decision theorist overseer might therefore worry (rightfully, as we will see) that providing rewards according to Eq. 1 will make the agent EDT-ish. Hence, she either has to incorrectly estimate how much utility was achieved; or live with the agent using an-in her mind-incorrect way of weighing her options. If she prefers the latter, she would reward according to Eq. 1. But arguably getting the agent to choose correctly is the overseer's primary goal. Thus, she might prefer to compute the reward according to r = E [U (H ) | e r , do(a), o] . (2) Here, do(a) refers to Pearl's do-calculus, where conditioning on do(a) roughly means intervening from outside the causal model to set A to a. For an introduction to the do-calculus, see  Pearl (2009) . Although a causal decision theorist overseer may prefer computing rewards according to Eq. 1, we will from now on say "the overseer uses CDT" if rewards are computed according to Eq. 2 and "the overseer uses EDT" if rewards are calculated according to Eq. 1. An approval-directed agent is characterized by maximizing the reward it receives from the overseer. 12 However, decision theory offers us, again, (at least) two different expected values, the regular expected value of EDT E [R | a] , (3) and CDT's causal expected value E [R | do(a)] . ( 4 ) 10 At first sight this may be confusing to some readers, because in reinforcement learning, utility sometimes refers to expected cumulative reward  (Russell and Norvig 2010, Chap. 17, 21) , although others use the term value function instead  (Sutton and Barto 1998, Sect. 3.7 ). Here, U does not refer to utility in that sense but in the decision-theoretical sense of representing intrinsic values. So, in the present case, we have two "layers" of goals: first, the agent maximizes the reward r . Second, the agent as incentivized by the overseer's way of calculating rewards maximizes utility U (H ). One cause of confusion is that in model applications of reinforcement learning, the reward function possesses full knowledge of the world state and thus does not require the use of the expectation operator. 11 If the disagreement in Newcomb's problem is to be about different theories of rational choice (EDT, CDT and so forth) rather than the predictive abilities of "the being", Omega or the psychologist, then after requesting both boxes a proponent of two-boxing has to believe that she will probably receive only $1000. Causal and evidential decision theorists agree that regular conditional expectation is the correct way of updating one's beliefs about the state of the world after an action has been taken (cf. the distinction between "acts" and "actions" in  Pearl 2009 Sect. 4.1.1) . We leave the interesting question of which (if any) decision theory describes the behavior of current reinforcement learning algorithms to future research 13 and in the following assume that the agent is known to implement either CDT or EDT.
 The conflict of the decision theories of agent and overseer When viewed together with the overseer, our agent may now be seen as containing two decision theories, one for computing the reward and one in the algorithm that tries to find the action to maximize that reward. These decision theories may not always be the same. Given this potential discrepancy, the question is which of the two decision theories prevails, i.e., for which configurations of the two decision theories the overall agent acts like a CDT agent and for which it acts like an EDT agent w.r.t. U . As it turns out, the answer to this question depends on the decision problem in question. In particular, it depends on whether the overseer updates its estimate of U (H ) primarily based on the action taken by the agent or on its observation of the environment. For illustration, consider two versions of Newcomb's problem. In both versions, the predictor is equally reliable-e.g., correct with 90% probability-and the potential box contents are the same-e.g., the standard $1K and $1M. As usual, the content of the opaque box cannot be causally influenced by one's decision. In the first version, the overseer eventually sees the payoff, i.e., how much money the agent has made. In this case, as soon as the money is observed, the overseer's estimate of U (H ) becomes independent of the agent's action. More generally, O may tell the overseer so much about U (H ) that it becomes independent of A even if U (H ) is not yet fully observed. That is, E [U (H ) | e r , a, o] = E [U (H ) | e r , o] (5) and E [U (H ) | e r , do(a), o] = E [U (H ) | e r , o] (6) for all e r , a and o. Note that neither of these two implies the other.  14  Intuitively speaking, these two mean that the reward is ultimately determined by U (H ). In the second version of Newcomb's problem, the monetary payoff is not observed but covertly invested into increasing the agent's utility function. Only the agent's choice can then inform the overseer about U (H ). Formally, it is both   E [U (H ) | e r , a, o] = E [U (H ) | e r , a] (7) and E [U (H ) | e r , do(a), o] = E [U (H ) | e r , do(a)] . (8) Intuitively speaking, these two equations mean that the reward is not determined by U (H ) but by what the overseer believes U (H ) will be given a or do(a). Again, we assume that this is known to the agent. An example class of cases is that in which the agent's decisions are correlated with those of agents in far-away parts of the environment (cf.  Treutlein and Oesterheld 2017; Oesterheld 2018b ). The two versions are depicted in Fig.  2 . Of course, these are only the two extremes from the set of all possible situations. In real-world Newcomb-like scenarios, the overseer may also draw some information from both sources. Nonetheless, it seems useful to understand the extreme cases, as this may also help us understand mixed ones. In the following subsections, we will show that in the first type, the decision theory of the agent is decisive, whereas in the second type, the overseer's decision theory is 15 . Roughly, the reason for that is the following: As noted earlier, the reward in the first type depends directly on U (H ). Thus, the agent will try to maximize U (H ) according to its own decision theory. In the second type, the overseer takes the agent's action a and then considers what either a or do(a) says about U (H ). Thus, the agent has to pay careful attention to whether the overseer uses EDT's or CDT's expected value. We prove this formally by considering all possible configurations of the type of the problem, the overseer's decision theory and the agent's decision theory. While we will limit our analysis to EDT and CDT, the results can easily be generalized to variants of these that arise from modifying the causal model or conditional credence distribution (e.g. Yudkowsky 2010; "Disposition-based decision theory"; Spohn 2012;  Dohrn 2015 ). The analysis is summarized in Table  1 .
 First type
 The EDT agent The EDT agent judges its action by E [R | a] . (9) If the overseer calculates regular conditional expectation, then it is E [R | a] = E [E [U (H ) | E r , O, a] | a] (10) = E [U (H ) | a] , ( 11 ) where the last line is due to what is sometimes called the law of total expectation (LTE) or the tower rule (see, e.g.,  Ross 2007, Sect. 3.4; Billingsley 1995, Theorem 34.4 ). Intuitively, you cannot expect that gaining more evidence (i.e., E r and O in addition to a) moves your expectation of U (H ) into any particular direction. Because the overseer knows more than the agent, we will need this rule in all of the following derivations. Its application makes it hard to generalize these results to other decision theories, since LTE does not apply if the two decision theories do not both compute a form of expected utility. Equations 10 and 11 show that if the overseer computes regular expected value and the agent maximizes the reward according to EDT, then the agent as a whole maximizes U according to EDT. If the overseer computes CDT's expected value, it is  E [R | a] = E [E [U (H ) | E r , do(a), O] | a] ( 12 = E [E [U (H ) | E r , a, O] | a] (15) = LTE E [U (H ) | a] (14) (16)
 The CDT agent The CDT agent judges its action by E [R | do(a)] . ( 17 ) If the overseer uses regular expected value (EDT), then E [R | do(a)] = E [E [U (H ) | a, O, E r ] | do(a)] (18) = e r ,o P(e r , o | do(a)) • E [U (H ) | a, o, e r ] (19) = eq. 5 and 6 e r ,o P(e r , o | do(a)) • E [U (H ) | do(a), o, e r ] (20) = E [E [U (H ) | do(a), O, E r ] | do(a)] (21) = LTE E [U (H ) | do(a)] (22) Learning about an intervention do(a) cannot always be treated in the same way as learning about other events. Hence, the application of the law of total expectation is not straightforward. However, P(• | do(x)) is always a probability distribution. Because the law of total expectation applies to all probability distributions, it also applies to ones resulting from the application of the do-calculus. If the overseer uses CDT's expected value, then E [R | do(a)] = E [E [U (H ) | E r , O, do(a)] | do(a)] (23) = LTE E [U (H ) | do(a)] . (24)
 Second type
 The EDT agent The EDT agent judges its actions by E [R | a] . ( 25 ) If the overseer is based on regular conditional expectation (EDT), then it is again E [R | a] = E [E [U (H ) | E r , a] | a] (26) = LTE E [U (H ) | a] . ( 27 ) If the overseer is based on CDT-type expectation, then (33) E [R | a] = E [E [U (H ) | E r , do(a)] | a] ( 28
 The CDT agent The CDT agent judges actions by E [R | do(a)] . ( 34 ) Because of Rule 2 in Theorem 3.4.1 of  Pearl (2009, Sect. 3.4. 2) applied to the causal graph of Fig.  2b , it is E [R | do(a)] = E [R | a] . (35) Thus, the analysis of the CDT agent is equivalent to that of the EDT agent.
 Conclusion In this paper, we have taken a step to map reinforcement learning architectures onto decision theories. We found that in Newcomb-like problems, if the overseer rewards the agent purely on the basis of the agent's action, then the overall system's behavior is determined by the decision theory implicit in the overseer's reward function. If the overseer judges the agent based on looking at the world, however, then the agent's decision theory is decisive. This has implications for how we should design approval-directed agents. For instance, if we would like to leave decision-theoretical judgements to the overseer, we must ensure that the overseer assigns rewards before making new observations about the world state (cf.  Christiano 2014, Sect. "Avoid lock-in") . Of course, this makes the reward less accurate and may thus slow down the agent's learning process. If we want the overseer to look at both the world and the agent's action, then we need to align both the overseer's and the agent's decision theory. Much more research is left to be done at the intersection of decision theory and artificial intelligence. For instance, what (if any) decision theories describe the way modern reinforcement learning algorithms maximize reward? Do the results of this paper generalize to sequential decision problems? Moving away from the reinforcement learning framework, what decision theories do other frameworks in AI implement? What about decision theories other than CDT and EDT? The reward is computed based on ob-(b) The reward is computed based on observing the agent's action.
 Fig. 2 2 Fig. 2 Two different ways in which the overseer can calculate the reward
 r , o | a) • E [U (H ) | e r , do(a), o] (13) = eq. 5 and 6 e r ,o P(e r , o | a) • E [U (H ) | e r , a, o]
 r | a) • E [U (H ) | do(a), e r ] r ) • E [U (H ) | do(a), e r ] r | do(a)) • E [U (H ) | do(a), e r ] (31) = E [E [U (H ) | E r , do(a)] | do(a)] (32) = LTE E [U (H ) | do(a)] .
 Table 1 1 An overview of the results of the calculations in Sect. 3 Type of Newcomb problem Agent's DT Overseer's DT Resulting DT First type CDT EDT CDT CDT CDT EDT EDT EDT CDT EDT Second type CDT EDT EDT CDT CDT EDT EDT EDT CDT CDT
			 In fact, Eells (1981)  has argued that EDT and CDT always behave in the same way, but I disagree with this assessment based on the reasons given byAhmed (2014, Sect. 4.3-4.6) and Price (1986) .2  Of course, the existing literature asks about the right decision theory proper. The answer to that question might differ from the answer to the AI-specific question (cf. Kumar 2017;  Treutlein 2018). After all, even if we have identified the right decision theory for ourselves, we may want to implement a different decision theory in an AI. One reason could be that the main contenders are not self-recommending-it has been pointed out that EDT and CDT both recommend to self-modify into slightly different decision theories
			 This is consistent with what reinforcement learning algorithms usually do-they choose policies rather than individual actions. This is because the utility of a single action usually cannot be evaluated without knowing how the agent will deal with situations that might arise as a result of taking that action. When individual actions can be evaluated in isolation, the ex ante policy choice sometimes differs from the choice of individual actions (see the absent-minded driver, introduced by Piccione and Rubinstein 1997; cf. Aumann et al. 1997 ; the Newcomb-like scenarios discussed by, e.g., Hintze 2014; Soares and Levinstein 2017, Sect. 2; and the problems in anthropics discussed by Armstrong 2011) . While it is rarely discussed in the debate between evidential and causal decision theorists, a few authors regard this discrepancy as crucial and have argued that a proper decision theory should be about optimal policy choices (e.g. Hintze 2014;   Soares and Fallenstein 2014b, Sect. 2.1; Soares and Levinstein 2017, Sect. 2). However, this issue is beyond the scope of the present paper. Further issues in sequential Newcomb-like problems are discussed by Everitt et al. (2015) .
			 There is a broad philosophical literature on whether causal relationships exist and whether they can be inferred in cases where the agent is part of the environment. See, e.g., the edited volume byPrice and Corry  (2007).6  For simplicity, we will ignore dependences not resulting from causation (Arntzenius 2010) . For example, if you play against a copy, there is a logical dependence between your and your copy's decision. Even if you know a set of nodes in the causal graph that d-separates your and your copy's decision (e.g., if you
			 In reinforcement learning, some have proposed alternative optimization targets that incorporate, e.g., risk aversion(García and Fernández 2015, Sect. 3).
			 For preliminary work on this question, see Mayer et al. (2016) , Oesterheld (2018a)  and perhaps Albert and Heiner (2001) .14  We give a brief justification of this claim. If all of a's causal influence on H can be discerned from O, then, of course, a could still be diagnostically relevant for one's estimate of U (H ). The other direction is more complicated. The idea is that Eq. 5 can be true if the causal and non-causal implications of a exactly cancel each other out. An example is a version of Newcomb's problem in which one-boxing ensures with certainty that both boxes contain the same amount of money. Then if O and E r do not contain any information, the expected value of two-boxing and one-boxing is the same and so learning of the action is irrelevant for estimating U (H ). However, two-boxing is causally better than one-boxing, so Eq. 6 is violated.
			 The dominance of the overseer's decision theory in the second type of Newcomb's problem is mentioned (though not proven) byChristiano (2014, Sect. "Avoid lock-in").
			 Synthese (2021) 198 (Suppl 27):S6491-S6504

Abstract:
Decision theorists disagree about how instrumentally rational agents, i.e., agents trying to achieve some goal, should behave in so-called Newcomb-like problems, with the main contenders being causal and evidential decision theory. Since the main goal of artificial intelligence research is to create machines that make instrumentally rational decisions, the disagreement pertains to this field. In addition to the more philosophical question of what the right decision theory is, the goal of AI poses the question of how to implement any given decision theory in an AI. For example, how would one go about building an AI whose behavior matches evidential decision theory's recommendations? Conversely, we can ask which decision theories (if any) describe the behavior of any existing AI design. In this paper, we study what decision theory an approval-directed agent, i.e., an agent whose goal it is to maximize the score it receives from an overseer, implements. If we assume that the overseer rewards the agent based on the expected value of some von Neumann-Morgenstern utility function, then such an approval-directed agent is guided by two decision theories: the one used by the agent to decide which action to choose in order to maximize the reward and the one used by the overseer to compute the expected utility of a chosen action. We show which of these two decision theories describes the agent's behavior in which situations.
 Keywords Reinforcement learning • Causal decision theory • Evidential decision theory • Newcomb's problem • AI safety • Philosophical foundations of AI B Caspar Oesterheld


Off-policy Monte Carlo agents with variable behaviour policies

Off-policy Monte Carlo The Monte Carlo agent is a model-free reinforcement learning agent  [3] . These operate when the environment is a Markov decision process (MDP). In an MDP, the next observation depends only on the current observation -the state -and the current action. The full set of state action pairs is designated by S × A. In each state, an agent chooses its actions according to a (possibly stochastic) policy π, that is assumed to be Markov. Monte Carlo agents operate by computing the Q-values of a policy for each state-action pair (s, a), which is the expected return if the agent choose a in s and subsequently follows π. It is episodic, exploring the same MDP repeatedly to compute Q-values. Episodic MDP's have initial state (where the agent starts an episode) and terminal states (where the agent ends an episode). This paper focuses on computing Q-values, not on updating policy choice in consequence. Assumption 1 In the following, these are always assumed, required for convergence results: (a) The MDP is finite, (b) Whatever policy the agent uses, its expected time until reaching a terminal state is finite. (c) The rewards after each step have finite expectation (consequently the total reward has finite expectation). If an agent is following one policy (e.g., ρ) but wishes to compute the Qvalues of another (e.g., π), it can use the off-policy Monte Carlo algorithm  [3] . In this case ρ is the behaviour policy, and π is the evaluation policy. The algorithm requires that for all state-action pairs (s, a): π(a|s) > 0 ⇒ ρ(a|s) > 0. ( ) 1 Because of this requirement, π(a|s) is zero whenever ρ(a|s) is, and the ratio π(a|s)/ρ(a|s) can be defined as 0 in these cases. The full algorithm is then given in box 1 -there are two variants, ordinary importance sampling using N (s, a) (the number of episodes in which the agent has encountered the state-action pair (s, a)) as the denominator, and weighted importance sampling using W (s, a) (the sum of the weights) instead  [4, 1] . 1. For all s ∈ S and a ∈ As, initialise N (s, a), W (s, a) and W R(s, a) to zero. 2. For all n ≥ 1: (a) Generate an episode history hn by following policy ρ. (b) The episode data will consist of the state, action chosen in the state, the immediate reward experienced by the agent. (c) For each state-action pair (s, a) appearing in the episode history: i. Let t be the first appearance of (s, a) in the history. ii. Let Rn(s, a) be the total subsequent reward. iii. Define  Note that W (s, a) and N (s, a) have the same expectation: let H (s,a) be the set of possible histories in the MDP subsequent to (s, a). For h ∈ H (s,a) , relabel the indexes so that h starts in step zero at (s, a). Use ρ and π to denote the probabilities of certain events conditional on the agent following those policies, and E ρ and E π similarly. Then if I n (s, a) denotes the event that (s, a) appears in the n-the episode, wn(s, a) = k>t π(a k |s k ) ρ(a k |s k ) . (2) E ρ (w(s, a)|I n (s, a)) = h∈H (s,a) ρ(h|I n (s, a)) k>0 π(h a k |h s k ) ρ(h a k |h s k ) = h∈H (s,a) ρ(h|I n (s, a)) π(h|I n (s, a)) ρ(h|I n (s, a)) = h∈H (s,a) π(h|I n (s, a)) = π(H (s,a) |I n (s, a)) = 1. ( ) 3 where h a k denotes the k-th action of history h, and h s k denotes the k-th state of history h. Then simply note that the value of N n (s, a) is simply the sum The same argument as in equation  (3)  shows that E ρ (w n (s, a)R n (s, a)|I n (s, a)) = h∈H (s,a) R h (s, a)ρ(h|I n (s, a)) k>0 π(h a k |h s k ) ρ(h a k |h s k ) = E π (R(s, a)|I n (s, a)), (4) where R h (s, a) is the reward along the history h subsequent to the first (s, a). Thus the expected weighted reward from following policy ρ, is the expected reward from following policy π. Assume that agents following ρ would almost surely explore every stateaction pair infinitely often (equivalently, that N n (s, a) → ∞ almost surely). Then the convergence of ordinary importance sampling is simply a consequence of the law of large numbers applied to every episode that visits (s, a). Convergence of weighted importance sampling is a consequence of Corollary 1 below.
 Varying the behaviour policy The previous proof, however, assumes that policy ρ is fixed. But what if it varies from episode to episode? If π and π are two policies, any p ∈ [0, 1] defines the mixed policy: (1 − p)π + pπ . This is the policy that, in each state, independently chooses whether to follow π, with probability 1 − p, and π , with probability p. Because the decisions are independent, the mixed policy is Markov if π and π are. A general form for all behaviour policies is then: Proposition 1. If ρ and π obey the restriction in equation (  1 ), then there exists a θ ∈ [0, 1) and a policy π such that ρ = (1 − θ)π + θπ . Proof. Let σ be the maximum across S × A of the ratio π(a|s)/ρ(a|s), in all cases where ρ(a|s) = 0. Then define (1 − θ) = 1/σ. Since π(a|s)/ρ(a|s) ≤ σ, (1 − θ) = 1/σ ≤ ρ(a|s)/π(a|s) and hence ρ(a|s) ≥ (1 − θ)π(a|s). Then define π as π (a|s) = 1 θ ρ(a|s) − (1 − θ)π(a|s) . To check that this quantity is less than 1, note that ρ(a|s ) = 1 − b =a ρ(b|s) ≤ 1 − b =a (1 − θ)π(b|s) hence that ρ(a|s) − (1 − θ)π(a|s) ≤ 1 − b (1 − θ)π(b|s) ≤ 1 − (1 − θ) ≤ θ. Then, by construction, ρ(a|s) = (1 − θ)π(a|s) + (θ)π (a|s). For n being the episode number, define ρ n by ρ n = (1 − θ n )π + θ n π n , (5) for some π n and θ n ∈ [0, 1). For convenience, further define the set S ⊂ S, which is the set of states s such that ρ n (s) = π(s). Then neither off-policy Monte Carlo algorithm need converge: Theorem 1. Neither off-policy Monte Carlo algorithm following policy ρ n need converge to the correct Q-values for π, even if the agent generates every possible episode history infinitely often. The proof is given in Section 3. But that convergence failure need not happen for all such ρ n . A sufficient condition for convergence is: Theorem 2. Let σ n = 1/(1−θ n ). Assume that σ n is eventually non-decreasing, and that there exists a δ > 0 such that, for large enough n, (σ n ) √ log(n) < n 1−δ . Assume that π visits each state-action pair with non-zero probability. Then both off-policy Monte Carlo algorithms following policy ρ n will almost surely converge on the correct Q-values for π. This will be proved in Section 4. Note that the requirement for visiting all (s, a) pairs is for the evaluation policy π -the behaviour policy will also do so, as a consequence of the conditions above. One immediate corollary of theorem 2 is: Corollary 1. If θ n is bounded above by κ < 1, both off-policy Monte Carlo algorithms using ρ n will converge on the correct Q-values for π, if π visits each state-action pair with non-zero probability. Proof. This kind of result has been proved before  [5] . The σ n are bounded above by 1/(1 − κ), and thus, for any δ, are less than n 1−δ for sufficiently large n. Then Theorem 2 implies the result. A second corollary is: Corollary 2. If θ n = 1 − 1 log(n) , both off-policy Monte Carlo algorithms using ρ n will converge on the correct Q-values for π, if π visits each state-action pair with non-zero probability. Proof. Set δ = 1/2, note that σ n = log(n), and that log (σ n ) √ log(n) = log(n) log(log(n)) < log(n) 2 = log( √ n) for large enough n. Hence, for large enough n, (σ n ) √ log(n) < √ n = n 1−1/2 . Then Theorem 2 implies the result. And a third corollary is: Corollary 3. Assume π n (a|s) > r (s,a) whenever π(a|s) > 0, for constants r (s,a) > 0. Then both off-policy Monte Carlo algorithms using ρ n will converge on the correct Q-values for π, if π visits each state-action pair with non-zero probability. Proof. Let r be the minimum value of r (s,a) /π(a|s), across all (s, a) where π(a|s) > 0. Then π n can be rewritten as: π n = rπ + (1 − r)π n , for some π n . Similarly ρ n can be rewritten as ρ n = (1 − (1 − r)θ n )π + (θ n )(1 − r)π n . Then note that (θ n )(1 − r) is bounded above by 1 − r, and the result follows by corollary 1. The rest of this paper will be dedicated to proving theorems 1 and 2.
 Failure to converge This Section aims to prove Theorem 1, by constructing a counter-example using the MDP of figure  1 . Set π(a|s 1 ) = 1, π(a|s 2 ) = π(b|s 2 ) = 1/2, S = {s 2 }, π n (a|s 2 ) = 0, π n (b|s 2 ) = 1, and define θ n as The probability of the n-th episode history being h b = (s 1 , a, 0, s 2 , b, −3, s 4 ) is greater or equal to 1/2: this history therefore almost surely gets generated infinitely often. Now consider the episode history h a = (s 1 , a, 0, s 2 , a, −1, s 4 ). It will get generated during episode n with probability θ n = 1 − 1/(n log(n)). 1 2 • (1 − θ n ) = 1 2n log(n) . Each episode is independent and n 1/(2n log(n)) = ∞, so by the converse Borel-Cantelli lemma, the episode is generated infinitely often. Then note that if the history h a is generated during the n-th episode, it is generated with weight 1/(1 − θ n ) = n log(n), while the history h b is generated with weight 1/(1 + θ n ) = 1/(2 − 1/(n log(n))). Split the weight total W n (s 1 , a) as W a n (s 1 , a) + W b n (s 1 , a) (the weight totals due to the histories h a and h b respectively). Then ordinary importance sampling will give Q n as: Q n (s 1 , a) = −W a n (s 1 , a) − 3W b n (s 1 , a) N n (s 1 , a) , (6) while weighted importance sampling gives: Q n (s 1 , a) = −W a n (s 1 , a) − 3W b n (s 1 , a) W a n (s 1 , a) + W b n (s 1 , a) . (7) In state s 1 , both actions are equiprobable and independent, so the law of large numbers implies N n (s 1 , a)/n → 1 almost surely. The W b n (s 1 , a) is the sum of weights less than 1, so W b n (s 1 , a) ≤ N n (s 1 , a). The probability of h b increases to 1, and the weights for that history are larger than 1/2 for n ≥ 1. Thus, almost surely, for sufficiently large n, n/3 and 2n are lower and upper bounds for both N n (s 1 , a) and W b n (s 1 , a). Now pick an n where episode history h a is generated, which must happen infinitely often. The weight of this history is n log(n), so W a n (s 1 , a) ≥ n. In the limit, this must dominate the n-bounded contributions of N n (s 1 , a) and W b n (s 1 , a). Thus, for a large enough n where h a is generated, equation (6) then gives the upper bound Q n (s 1 , a) ≤ −C log(n), for some constant C. Conversely, equation (7) gives an upper bound: Q n (s 1 , a) ≥ −1 − C/ log(n), for some C. The correct Q-value for Q(s 1 , a) under π is clearly (−3−1)/2 = −2, so neither algorithm can converge to the correct values. Ordinary importance sampling clearly cannot find the optimal policy (of choosing a in state s 2 ) either. If b had a reward of 0 instead of a reward of −2, then weighted importance sampling would fail to find the optimal policy on that MDP, so both algorithms can fail to find optimal policies. Remark 1. The lack of convergence can also be proved for θ n = 1 − 1/n, but the proofs are more involved.
 Proof of convergence This Section aims to prove 3 Theorem 2.
 Infinite variance The reason that such mathematical machinery is needed is because, in many cases, the variance of the reward become infinite. Consider the MDP of figure  2 . Two actions are available in s 2 : action a which, with p probability will return the agent to state 1 with a reward of 1, and otherwise send them to state s 3 with no reward. And b, which goes straight to s 3 with no reward. Set π(a|s 1 ) = 1, π n (b|s 2 ) = 1, and θ n = 1 − q. The weights are powers of 1/q, ρ n (a|s 2 ) = q, so the expected weighted reward is the correct ∞ l=0 (l/q l )(pq) l , which is finite. Since for any random variable X, Var(X) = E(X 2 ) − E(X) 2 , the variance of the weighted reward is finite iff the expected squared weighted reward is finite. Then the expected squared weighted reward is: ∞ l=0 (l/q l ) 2 (pq) l = ∞ l=0 l 2 (p/q) l . And this sum diverges for q ≤ p.
 Proving convergence This proof will use ordinary importance sampling, then generalise to weighted importance sampling. Fix any pair (s, a). Since π must visit that pair with finite probability, there is an m < |S × A| and a probability τ > 0 such that an agent following π would  s 3 s 2 s 1 a, 0 a, 0, 1 − q b, 0 a, 1, q (σ n ) m < (σ n ) √ log(n) < n 1−δ < n. Hence, for large enough n, τ (1 − θ n ) m = τ (1/σ n ) > τ /n, so ∞ τ (1 − θ n ) m = ∞. Then since each episode is independent, the converse Borel-Cantelli lemma implies that an agent following ρ will almost surely visit (s, a) infinitely often. How regular will these visits be? The expected number of visits during the episodes up to the n-th is simply n j=1 τ (1 − θ j ) m . For large enough n, σ n < n 1−δ √ log(n) and so eventually τ (1 − θ n ) m > τ n −m 1−δ √ log(n) > n −δ , for any δ > 0. Therefore, for large enough n, the number of visits to (s, a) among the n episodes must be almost surely greater than n j=1 j −δ = O(n 1−δ ). Thus, for large enough n, N n (s, a) > n 1−δ almost surely. Let Q * be the true Q-value of the MDP under π, and let H be the set of possible episode histories, ignoring rewards. Let H l be the set of histories of length l. Write π(h n ) for h ∈ H to designate the probability that episode n has history h if the agent follows the policy π. Note that since π is fixed, π(h n ) is independent of n (unlike ρ n (h n )). Let η be the policy designed to maximise the expected time the agent spends in the MDP. This means that η is a Markov policy, as the policy that maximises the time spent if the agent is in state (s , a ) does not depend on the agent's prior history. Combined with the MDP, η describes a Markov chain, with absorbing final states. Its transition matrix is of the form: P R 0 Id , for P a transition matrix on the non-absorbing state-action pairs, and Id the identity matrix on the absorbing ones. Since any episode must terminate with probability 1, the matrix P has a single maximal real eigenvalue µ < 1  [2] . For large n, the probability that the matrix will not have terminated by the n-th episode is bounded by (µ ) n . Since η is the policy that maximises the expected time spent in the MDP, an agent following π cannot expect to stay in MDP longer than that, so there exists a C such that π(H l ) ≤ C (µ ) l . Fix any µ < µ < 1, then because l 2 (µ ) l must eventually be less than µ l , there exists a C such that lπ(H l ) ≤ l 2 π(H l ) ≤ Cµ l . Let E be the maximal expected reward the agent can generate from a single state-action pair. Let S be the maximal expected squared reward the agent can generate from a single state-action pair. Then if R h is the random variable denoting the reward generated along history h ∈ H l , E (R h |h) ≤ lE Var (R h |h) ≤ E R 2 h |h ≤ l 2 S. Let W R l n denote the random variable that returns 0 if the length of the n-th episode is not l, and the (weighted) reward otherwise, under the assumption that the agent visits (s, a) during episode n. Therefore: W R l n = h∈H l w(h n )R h ρ(h n ). Note that w(h n ) = π(h n )/ρ(h n ) ≤ (σ n ) i h , where i h ≤ l is the number of times that the agent goes through a state in S along h. Then E ρn W R l n = h∈H l w(h n )E (R h |h) ρ n (h n ) = h∈H l E (R h |h) π(h n ), which is the expected reward from episode histories of length l from an agent following π. Therefore ∞ l=1 E ρn W R l n = h E (R h |h) π(h n ) = Q * . The expectation and variance of W R l n can be bounded as: E ρn W R l n = h∈H l E (R h |h) π(h n ) ≤ lEπ(H l ) ≤ ECµ l Var ρn W R l n ≤ E ρn W R l n 2 ≤ h∈H l E (w(h n )R h ) 2 |h ρ(h n ) ≤ h∈H l (σ n ) i h E (R h ) 2 |h π(h n ) ≤ π(H l )(σ n ) l l 2 S ≤ SC(µσ n ) l . (8) Redefine C as max(EC, SC) so that these bounds are Cµ l and C(µσ n ) l respectively. Then define W R <l n = j<l W R j n and W R ≥l n = j≥l W R j n . Equation (8) then implies that E ρn W R ≥l n ≤ C ∞ j=l µ j ≤ Cµ l 1 − µ Var ρn W R <l n ≤ l−1 j=1 Var ρn W R <l n ≤ l−1 j=1 C(µσ n ) j ≤ C(σ n µ) l − 1 σ n µ − 1 ≤ C(σ n ) l σ n µ − 1 Redefine C as C/(1 − µ) so that the first bound is Cµ l . For large enough n, σ n µ − 1 > 1; so, redefining C if needed to cover the finitely many smaller values of n, the second bound is C(σ n ) l : E ρn W R ≥l n ≤ Cµ l Var ρn W R <l n ≤ C(σ n ) l Let I n be the indexing variable that denotes that the agent visited (s, a) during episode n. By assumption, N n (s, a) = n j=1 I n (s, a), and this section has shown that N n (s, a) > n 1−δ for any δ > 0 and large enough n. Define Q ≥l n = 1 N n (s, a) n j=1 I j W R ≥l j Q <l n = 1 N n (s, a) n j=1 I j W R <l j . Then, since σ n are eventually non-decreasing, for any δ > 0 and for large enough n: E ρn Q ≥l n ≤ 1 N n (s, a) n j=1 I j Cµ l ≤ n j=1 I j N n (s, a) Cµ l ≤ Cµ l Var ρn Q <l n ≤ 1 N n (s, a) 2 n j=1 CI j (σ j ) l ≤ C N n (s, a) (σ n ) l ≤ C n 1−δ (σ n ) l . The following is a key Lemma: Lemma 1. Let 1 ≤ m 1 ≤ m 2 . . . be a sequence that is lacunary in that there exists a c > 1 such that m j+1 /m j > c for sufficiently large j. Then Q mj converges to Q * almost surely as j → ∞. Proof. Fix any > 0, and consider the subsequence n j = e √ j . Then since {m j } is eventually an exponentially growing sequence, m j > n j for sufficiently large j. Set l j = 4 √ j and consider Q mj = Q ≥lj mj + Q <lj mj . Then note that if k = 4 √ j, E Q ≥lj mj ≤ Cµ lj , and j 2 2 Cµ lj ≤ k 8 2 Cµ k−1 → 0, as k → ∞. This implies that for large enough j, E Q ≥lj mj ≤ j 2 . Any random variable X with finite expectation has the first moment bound: P (|X| ≥ λ) ≤ E|X| λ . Setting λ = , this implies that P (|Q ≥lj mj | ≥ ) ≤ 1 j 2 for large enough j. Since ∞ j=1 1 j 2 < ∞, the Borel-Cantelli lemma implies that, almost surely, |Q ≥lj mj | ≥ only finitely often. Since Q * = E Q mj = E Q ≥lj mj +E Q <lj mj , this also implies |E Q <lj mj − Q * | ≥ only finitely often. Consider now Var Q <lj mj ≤ C (mj ) 1−δ (σ mj ) lj . Note that l j = 4 √ j ≤ √ j ≤ log(n j ) ≤ log(m j ) . So, by the assumption of Theorem 2, for large enough j, (σ mj ) lj < m 1−δ j for some δ > 0. Fix δ = δ/2. Hence for large enough j, C (mj ) 1−δ/2 (σ mj ) lj < 1 (mj ) δ/2 . Since m δ/2 j > n δ/2 j ≥ e (δ/2) √ j > Cj 2 / 2 for large enough j, eventually Var Q <lj mj ≤ 2 j 2 . Any random variable X with finite variance has the second moment bound: P (|X − E(X)| ≥ λ) ≤ Var(X) λ 2 . Setting λ = , this implies that for large enough j, P (|Q <lj mj − E Q <lj mj | ≥ ) ≤ 1 j 2 . Using the Borel-Cantelli lemma a second time, |Q <lj mj − E Q <lj mj | < except finitely often. Putting together the three bounds completes the proof by showing that, almost surely, except for finitely many j, |Q mj − Q * | < 3 . Since was arbitrary, this completes the proof. The next step is to generalise from lacunary sequences to all sequences. Since all rewards have been assumed positive, nQ n ≤ mQ m for n < m. Then for any 1 > > 0, define the lacunary sequence m j = (1 + ) j . For m j < n < m j+1 , (1 + ) j Q mj ≤ nQ n ≤ Q mj+1 (1 + ) j+1 Q mj+1 . This implies that for large enough j, Q n ≥ Q mj (1/(1 + ) − 2 ) ≥ Q mj (1 − ) and Q n ≤ Q mj ((1 + ) + 2 ) ≤ Q mj (1 + 2 ) , where the 2 term comes from the fact that (1 + ) j need not be an integer. By the Lemma above, the sequence Q mj converge almost surely to Q * , thus, for large enough j and n > (1 + ) j , Q n must be within 3 (Q * + 1) of Q * . Since was arbitrary, this proves that Q n converges to Q * almost surely as n → ∞. This completes the proof for all MDP's with positive rewards. Note that the same proof works for MDP's with negative rewards. Then the general proof is established for by dividing the rewards into positive and negative parts, noting their separate convergence, and noting that the Q-values update process is linear in rewards. Since this proves the convergence of Q n (s, a) to Q * (s, a) for any (s, a), and there are finitely many (s, a) pairs, this proves the almost sure convergence of Q n to Q * in general. It is now necessary to extend the result to weighted importance sampling. To do that, it will suffice to show that, under the conditions above, W n (s, a)/N n (s, a) → 1 almost surely. To see this, change the MDP by setting all rewards to 0, except for the final reward when the agent reaches a terminal state, where they will get 1. This means that the reward along each history is 1, and the weighted reward is just the weight. Thus the new Q n for ordinary importance sampling is Q n (s, a) = W n (s, a)/N n (s, a). By the result we've just proved, these must converge almost surely to the correct Q-values for the modified MDP, i.e., to 1. This proves that the ratios converge almost surely to 1, as required.
 A Note on the independence assumption The policy ρ n , as defined in equation (  5 ), assumes that the agent choose independently between π and π n for each s ∈ S. If this independence is dropped, the situation can get even worse for convergence -the agent may fail to converge to the right values 4 even for fixed θ < 1. Consider the MDP in figure  3 . Define the policy π as choosing randomly amongst the two actions at s 1 , and choosing a otherwise. Define S = {s 2 , s 3 } and π n as choosing b from all states in S . For ρ, the probabilities of choosing π and π n in S are equal (and the same from episode to episode), θ 2 = θ 3 = 0.5, but they are strictly anti-correlated within a given episode. Notice that the episode history (s 1 , a, 0, s 2 , a, 1, s 3 , a, 1, s 4 ) never appears. This is because the uses of π n at s 2 and s 3 are anti-correlated, so the agent cannot avoid both of them. Therefore the agent never experiences a total reward of 2; moreover, the only episodes with rewards are the (equiprobable) iv. Nn(s, a) = Nn−1(s, a) + 1. v. Wn(s, a) = Wn−1(s, a) + wn(s, a). vi. W Rn(s, a) = W Rn−1(s, a) + wn(s, a)R(s, a). vii. Either Qn(s, a) = W Rn(s,a) Nn(s,a) , or Qn(s, a) = W Rn(s,a) Wn(s,a) . (d) Discard the episode data.
 n j=1 I j (s,a) , to see that N n (s, a) and W n (s, a) = n j=1 I j (s,a) w n (s, a) have same expectation. If ρ is fixed, then by sampling only those histories including (s, a), the strong law of large numbers implies that the ratio of N n (s, a) and W n (s, a) converges to 1 almost surely.
 3 a, − 1 Fig. 1 . 311 Fig. 1. An MDP where the off-policy Monte Carlo algorithms can fail to compute the correct Q-values.
 Fig. 2 . 2 Fig. 2. An MDP where the variance of the reward can become infinite within a single episode.
 1 Fig. 3 . 13 Fig. 3. An MDP on which the agent has non-Markov policy choices.
 Table 1 . 1 Off-policy Monte Carlo algorithm
			 The proof will closely mirror the standard proofs of the strong law of large numbers, see for instance https://terrytao.wordpress.com/2008/06/18/the-strong-lawof-large-numbers/
			 For non-Markov policies like this one, the off-policy algorithm has to be adjusted to consider ratios π(h)/ρ(h) for entire histories h (rather the product of state-action pair probabilities), but this is not a large change.

Abstract: This paper looks at the convergence property of off-policy Monte Carlo agents with variable behaviour policies. It presents results about convergence and lack of convergence. Even if the agent generates every possible episode history infinitely often, the algorithm can fail to converge on the correct Q-values. On the other hand, it can converge on the correct Q-values under certain conditions. For instance, if, during the n-th episode, the agent has an independent probability of 1/ log(n) of following the original policy at any given state, then it will converge on the right Q-values for that policy.