Joshua Achiam
David Held
Aviv Tamar
Pieter Abbeel

Constrained Policy Optimization

Introduction Recently, deep reinforcement learning has enabled neural network policies to achieve state-of-the-art performance on many high-dimensional control tasks, including Atari games (using pixels as inputs)  (Mnih et al., 2015; , robot locomotion and manipulation  (Schulman et al., 2015; Levine et al., 2016; Lillicrap et al., 2016) , and even Go at the human grandmaster level  (Silver et al., 2016) . 1 UC Berkeley 2 OpenAI. Correspondence to: Joshua Achiam <jachiam@berkeley.edu>. Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 by the author(s). In reinforcement learning (RL), agents learn to act by trial and error, gradually improving their performance at the task as learning progresses. Recent work in deep RL assumes that agents are free to explore any behavior during learning, so long as it leads to performance improvement. In many realistic domains, however, it may be unacceptable to give an agent complete freedom. Consider, for example, an industrial robot arm learning to assemble a new product in a factory. Some behaviors could cause it to damage itself or the plant around it-or worse, take actions that are harmful to people working nearby. In domains like this, safe exploration for RL agents is important  (Moldovan & Abbeel, 2012; Amodei et al., 2016) . A natural way to incorporate safety is via constraints. A standard and well-studied formulation for reinforcement learning with constraints is the constrained Markov Decision Process (CMDP) framework  (Altman, 1999) , where agents must satisfy constraints on expectations of auxilliary costs. Although optimal policies for finite CMDPs with known models can be obtained by linear programming, methods for high-dimensional control are lacking. Currently, policy search algorithms enjoy state-of-theart performance on high-dimensional control tasks  (Mnih et al., 2016; Duan et al., 2016) . Heuristic algorithms for policy search in CMDPs have been proposed  (Uchibe & Doya, 2007) , and approaches based on primal-dual methods can be shown to converge to constraint-satisfying policies  (Chow et al., 2015) , but there is currently no approach for policy search in continuous CMDPs that guarantees every policy during learning will satisfy constraints. In this work, we propose the first such algorithm, allowing applications to constrained deep RL. Driving our approach is a new theoretical result that bounds the difference between the rewards or costs of two different policies. This result, which is of independent interest, tightens known bounds for policy search using trust regions  (Kakade & Langford, 2002; Pirotta et al., 2013; Schulman et al., 2015) , and provides a tighter connection between the theory and practice of policy search for deep RL. Here, we use this result to derive a policy improvement step that guarantees both an increase in reward and satisfaction of constraints on other costs. This step forms the basis for our algorithm, Constrained Policy Optimization (CPO), which computes an approximation to the theoretically-justified update. In our experiments, we show that CPO can train neural network policies with thousands of parameters on highdimensional simulated robot locomotion tasks to maximize rewards while successfully enforcing constraints.
 Related Work Safety has long been a topic of interest in RL research, and a comprehensive overview of safety in RL was given by  (García & Fernández, 2015) . Safe policy search methods have been proposed in prior work.  Uchibe and Doya (2007)  gave a policy gradient algorithm that uses gradient projection to enforce active constraints, but this approach suffers from an inability to prevent a policy from becoming unsafe in the first place. Bou  Ammar et al. (2015)  propose a theoretically-motivated policy gradient method for lifelong learning with safety constraints, but their method involves an expensive inner loop optimization of a semi-definite program, making it unsuited for the deep RL setting. Their method also assumes that safety constraints are linear in policy parameters, which is limiting.  Chow et al. (2015)  propose a primal-dual subgradient method for risk-constrained reinforcement learning which takes policy gradient steps on an objective that trades off return with risk, while simultaneously learning the trade-off coefficients (dual variables). Some approaches specifically focus on application to the deep RL setting.  Held et al. (2017)  study the problem for robotic manipulation, but the assumptions they make restrict the applicability of their methods.  Lipton et al. (2017)  use an 'intrinsic fear' heuristic, as opposed to constraints, to motivate agents to avoid rare but catastrophic events.  Shalev-Shwartz et al. (2016)  avoid the problem of enforcing constraints on parametrized policies by decomposing 'desires' from trajectory planning; the neural network policy learns desires for behavior, while the trajectory planning algorithm (which is not learned) selects final behavior and enforces safety constraints. In contrast to prior work, our method is the first policy search algorithm for CMDPs that both 1) guarantees constraint satisfaction throughout training, and 2) works for arbitrary policy classes (including neural networks).
 Preliminaries A Markov decision process (MDP) is a tuple, (S, A, R, P, µ), where S is the set of states, A is the set of actions, R : S × A × S → R is the reward function, P : S ×A×S → [0, 1] is the transition probability function (where P (s � |s, a) is the probability of transitioning to state s � given that the previous state was s and the agent took action a in s), and µ : S → [0, 1] is the starting state distribution. A stationary policy π : S → P(A) is a map from states to probability distributions over actions, with π(a|s) denoting the probability of selecting action a in state s. We denote the set of all stationary policies by Π. In reinforcement learning, we aim to select a policy π which maximizes a performance measure, J(π), which is typically taken to be the infinite horizon discounted total return, J(π) . = E τ ∼π [ � ∞ t=0 γ t R(s t , a t , s t+1 )]. Here γ ∈ [0, 1) is the discount factor, τ denotes a trajectory (τ = (s 0 , a 0 , s 1 , ...)), and τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π: s 0 ∼ µ, a t ∼ π(•|s t ), s t+1 ∼ P (•|s t , a t ). Letting R(τ ) denote the discounted return of a trajectory, we express the on-policy value function as V π (s) . = E τ ∼π [R(τ )|s 0 = s] and the on-policy action-value function as Q π (s, a) . = E τ ∼π [R(τ )|s 0 = s, a 0 = a]. The advantage function is A π (s, a) . = Q π (s, a) − V π (s). Also of interest is the discounted future state distribution, d π , defined by d π (s) = (1−γ) � ∞ t=0 γ t P (s t = s|π). It allows us to compactly express the difference in performance between two policies π � , π as J(π � ) − J(π) = 1 1 − γ E s∼d π � a∼π � [A π (s, a)] , (1) where by a ∼ π � , we mean a ∼ π � (•|s), with explicit notation dropped to reduce clutter. For proof of (1), see  (Kakade & Langford, 2002)  or Section 10 in the supplementary material.
 Constrained Markov Decision Processes A constrained Markov decision process (CMDP) is an MDP augmented with constraints that restrict the set of allowable policies for that MDP. Specifically, we augment the MDP with a set C of auxiliary cost functions, C 1 , ..., C m (with each one a function C i : S × A × S → R mapping transition tuples to costs, like the usual reward), and limits d 1 , ..., d m . Let J Ci (π) denote the expected discounted return of policy π with respect to cost function C i : J Ci (π) = E τ ∼π [ � ∞ t=0 γ t C i (s t , a t , s t+1 )]. The set of feasible stationary policies for a CMDP is then Π C . = {π ∈ Π : ∀i, J Ci (π) ≤ d i } , and the reinforcement learning problem in a CMDP is π * = arg max π∈Π C J(π). The choice of optimizing only over stationary policies is justified: it has been shown that the set of all optimal policies for a CMDP includes stationary policies, under mild technical conditions. For a thorough review of CMDPs and CMDP theory, we refer the reader to  (Altman, 1999) . We refer to J Ci as a constraint return, or C i -return for short. Lastly, we define on-policy value functions, actionvalue functions, and advantage functions for the auxiliary costs in analogy to V π , Q π , and A π , with C i replacing R: respectively, we denote these by V π Ci , Q π Ci , and A π Ci .
 Constrained Policy Optimization For large or continuous MDPs, solving for the exact optimal policy is intractable due to the curse of dimensionality  (Sutton & Barto, 1998) . Policy search algorithms approach this problem by searching for the optimal policy within a set Π θ ⊆ Π of parametrized policies with parameters θ (for example, neural networks of a fixed architecture). In local policy search  (Peters & Schaal, 2008) , the policy is iteratively updated by maximizing J(π) over a local neighborhood of the most recent iterate π k : π k+1 = arg max π∈Π θ J(π) s.t. D(π, π k ) ≤ δ, (2) where D is some distance measure, and δ > 0 is a step size. When the objective is estimated by linearizing around π k as J(π k ) + g T (θ − θ k ), g is the policy gradient, and the standard policy gradient update is obtained by choosing  Schulman et al., 2015) . D(π, π k ) = �θ − θ k � 2 ( In local policy search for CMDPs, we additionally require policy iterates to be feasible for the CMDP, so instead of optimizing over Π θ , we optimize over Π θ ∩ Π C : π k+1 = arg max π∈Π θ J(π) s.t. J Ci (π) ≤ d i i = 1, ..., m D(π, π k ) ≤ δ. (3) This update is difficult to implement in practice because it requires evaluation of the constraint functions to determine whether a proposed point π is feasible. When using sampling to compute policy updates, as is typically done in high-dimensional control  (Duan et al., 2016) , this requires off-policy evaluation, which is known to be challenging  (Jiang & Li, 2015) . In this work, we take a different approach, motivated by recent methods for trust region optimization  (Schulman et al., 2015) . We develop a principled approximation to (3) with a particular choice of D, where we replace the objective and constraints with surrogate functions. The surrogates we choose are easy to estimate from samples collected on π k , and are good local approximations for the objective and constraints. Our theoretical analysis shows that for our choices of surrogates, we can bound our update's worst-case performance and worst-case constraint violation with values that depend on a hyperparameter of the algorithm. To prove the performance guarantees associated with our surrogates, we first prove new bounds on the difference in returns (or constraint returns) between two arbitrary stochastic policies in terms of an average divergence between them. We then show how our bounds permit a new analysis of trust region methods in general: specifically, we prove a worst-case performance degradation at each update. We conclude by motivating, presenting, and proving gurantees on our algorithm, Constrained Policy Optimization (CPO), a trust region method for CMDPs.
 Policy Performance Bounds In this section, we present the theoretical foundation for our approach-a new bound on the difference in returns between two arbitrary policies. This result, which is of independent interest, extends the works of  (Kakade & Langford, 2002) ,  (Pirotta et al., 2013) , and  (Schulman et al., 2015) , providing tighter bounds. As we show later, it also relates the theoretical bounds for trust region policy improvement with the actual trust region algorithms that have been demonstrated to be successful in practice  (Duan et al., 2016) . In the context of constrained policy search, we later use our results to propose policy updates that both improve the expected return and satisfy constraints. The following theorem connects the difference in returns (or constraint returns) between two arbitrary policies to an average divergence between them. Theorem 1. For any function f : S → R and any policies π � and π, define δ f (s, a, s � ) . = R(s, a, s � ) + γf (s � ) − f (s), � π � f . = max s |E a∼π � ,s � ∼P [δ f (s, a, s � )]| , L π,f (π � ) . = E s∼d π a∼π s � ∼P �� π � (a|s) π(a|s) − 1 � δ f (s, a, s � ) � , and D ± π,f (π � ) . = L π,f (π � ) 1 − γ ± 2γ� π � f (1 − γ) 2 E s∼d π [D T V (π � ||π)[s]] , where D T V (π � ||π)[s] = (1/2) � a |π � (a|s) − π(a|s) | is the total variational divergence between action distributions at s. The following bounds hold: D + π,f (π � ) ≥ J(π � ) − J(π) ≥ D − π,f (π � ). (4) Furthermore, the bounds are tight (when π � = π, all three expressions are identically zero). Before proceeding, we connect this result to prior work. By bounding the expectation E s∼d π [D T V (π � ||π)[s]] with max s D T V (π � ||π)[s], picking f = V π , and bounding � π � V π to get a second factor of max s D T V (π � ||π)[s], we recover (up to assumption-dependent factors) the bounds given by  Pirotta et al. (2013)  as Corollary 3.6, and by  Schulman et al. (2015)  as Theorem 1a. The choice of f = V π allows a useful form of the lower bound, so we give it as a corollary. Corollary 1. For any policies π � , π, with � π � . = max s |E a∼π � [A π (s, a)]|, the following bound holds: J(π � ) − J(π) ≥ 1 1 − γ E s∼d π a∼π � � A π (s, a) − 2γ� π � 1 − γ D T V (π � ||π)[s] � . ( 5 ) The bound (5) should be compared with equation (  1 ). The term  5 ) is an approximation to J(π � ) − J(π), using the state distribution d π instead of d π � , which is known to equal J(π � ) − J(π) to first order in the parameters of π � on a neighborhood around π  (Kakade & Langford, 2002) . The bound can therefore be viewed as describing the worst-case approximation error, and it justifies using the approximation as a surrogate for J(π � ) − J(π). (1 − γ) −1 E s∼d π ,a∼π � [A π (s, a)] in ( Equivalent expressions for the auxiliary costs, based on the upper bound, also follow immediately; we will later use them to make guarantees for the safety of CPO. Corollary 2. For any policies π � , π, and any cost function C i , with � π � Ci . = max s |E a∼π � [A π Ci (s, a)]|, the following bound holds: J Ci (π � ) − J Ci (π) ≤ 1 1 − γ E s∼d π a∼π � � A π Ci (s, a) + 2γ� π � Ci 1 − γ D T V (π � ||π)[s] � . (6) The bounds we have given so far are in terms of the TV-divergence between policies, but trust region methods constrain the KL-divergence between policies, so bounds that connect performance to the KL-divergence are desirable. We make the connection through Pinsker's inequality  (Csiszar & Körner, 1981) : for arbitrary distributions p, q, the TV-divergence and KL-divergence are related by D T V (p||q) ≤ � D KL (p||q)/2. Combining this with Jensen's inequality, we obtain E s∼d π [D T V (π � ||π)[s]] ≤ E s∼d π � � 1 2 D KL (π � ||π)[s] � ≤ � 1 2 E s∼d π [D KL (π � ||π)[s]] (7) From (  7 ) we immediately obtain the following. Corollary 3. In bounds (4), (  5 ), and (6), make the substitution E s∼d π [D T V (π � ||π)[s]] → � 1 2 E s∼d π [D KL (π � ||π)[s]]. The resulting bounds hold.
 Trust Region Methods Trust region algorithms for reinforcement learning  (Schulman et al., 2015;  have policy updates of the form π k+1 = arg max π∈Π θ E s∼d π k a∼π [A π k (s, a)] s.t. DKL (π||π k ) ≤ δ, (8) where DKL (π||π k ) = E s∼π k [D KL (π||π k )[s]], and δ > 0 is the step size. The set {π θ ∈ Π θ : DKL (π||π k ) ≤ δ} is called the trust region. The primary motivation for this update is that it is an approximation to optimizing the lower bound on policy performance given in (5), which would guarantee monotonic performance improvements. This is important for optimizing neural network policies, which are known to suffer from performance collapse after bad updates  (Duan et al., 2016) . Despite the approximation, trust region steps usually give monotonic improvements  (Schulman et al., 2015; Duan et al., 2016)  and have shown state-of-the-art performance in the deep RL setting  (Duan et al., 2016; Gu et al., 2017) , making the approach appealing for developing policy search methods for CMDPs. Until now, the particular choice of trust region for (8) was heuristically motivated; with (5) and Corollary 3, we are able to show that it is principled and comes with a worstcase performance degradation guarantee that depends on δ. Proposition 1 (Trust Region Update Performance). Suppose π k , π k+1 are related by (8), and that π k ∈ Π θ . A lower bound on the policy performance difference between π k and π k+1 is J(π k+1 ) − J(π k ) ≥ − √ 2δγ� π k+1 (1 − γ) 2 , ( 9 ) where � π k+1 = max s � � E a∼π k+1 [A π k (s, a)] � � . Proof. π k is a feasible point of (8) with objective value 0, so E s∼d π k ,a∼π k+1 [A π k (s, a)] ≥ 0. The rest follows by (5) and Corollary 3, noting that (8) bounds the average KLdivergence by δ. This result is useful for two reasons: 1) it is of independent interest, as it helps tighten the connection between theory and practice for deep RL, and 2) the choice to develop CPO as a trust region method means that CPO inherits this performance guarantee.
 Trust Region Optimization for Constrained MDPs Constrained policy optimization (CPO), which we present and justify in this section, is a policy search algorithm for CMDPs with updates that approximately solve (3) with a particular choice of D. First, we describe a policy search update for CMDPs that alleviates the issue of off-policy evaluation, and comes with guarantees of monotonic performance improvement and constraint satisfaction. Then, because the theoretically guaranteed update will take toosmall steps in practice, we propose CPO as a practical approximation based on trust region methods. By corollaries 1, 2, and 3, for appropriate coefficients α k , β i k the update π k+1 = arg max π∈Π θ E s∼d π k a∼π [A π k (s, a)] − α k � DKL (π||π k ) s.t. J Ci (π k ) + E s∼d π k a∼π � A π k Ci (s, a) 1 − γ � + β i k � DKL (π||π k ) ≤ d i is guaranteed to produce policies with monotonically nondecreasing returns that satisfy the original constraints. (Observe that the constraint here is on an upper bound for J Ci (π) by (  6 ).) The off-policy evaluation issue is alleviated, because both the objective and constraints involve expectations over state distributions d π k , which we presume to have samples from. Because the bounds are tight, the problem is always feasible (as long as π 0 is feasible). However, the penalties on policy divergence are quite steep for discount factors close to 1, so steps taken with this update might be small. Inspired by trust region methods, we propose CPO, which uses a trust region instead of penalties on policy divergence to enable larger step sizes: π k+1 = arg max π∈Π θ E s∼d π k a∼π [A π k (s, a)] s.t. J Ci (π k ) + 1 1 − γ E s∼d π k a∼π � A π k Ci (s, a) � ≤ d i ∀i DKL (π||π k ) ≤ δ. (10) Because this is a trust region method, it inherits the performance guarantee of Proposition 1. Furthermore, by corollaries 2 and 3, we have a performance guarantee for approximate satisfaction of constraints: Proposition 2 (CPO Update Worst-Case Constraint Violation). Suppose π k , π k+1 are related by (10), and that Π θ in (10) is any set of policies with π k ∈ Π θ . An upper bound on the C i -return of π k+1 is J Ci (π k+1 ) ≤ d i + √ 2δγ� π k+1 Ci (1 − γ) 2 , where � π k+1 Ci = max s � � E a∼π k+1 � A π k Ci (s, a) �� � .
 Practical Implementation In this section, we show how to implement an approximation to the update (10) that can be efficiently computed, even when optimizing policies with thousands of parameters. To address the issue of approximation and sampling errors that arise in practice, as well as the potential violations described by Proposition 2, we also propose to tighten the constraints by constraining upper bounds of the auxilliary costs, instead of the auxilliary costs themselves.
 Approximately Solving the CPO Update For policies with high-dimensional parameter spaces like neural networks, (  10 ) can be impractical to solve directly because of the computational cost. However, for small step sizes δ, the objective and cost constraints are well-approximated by linearizing around π k , and the KLdivergence constraint is well-approximated by second order expansion (at π k = π, the KL-divergence and its gradient are both zero). Denoting the gradient of the objective as g, the gradient of constraint i as b i , the Hessian of the KL-divergence as H, and defining c i . = J Ci (π k ) − d i , the approximation to (  10 ) is: θ k+1 = arg max θ g T (θ − θ k ) s.t. c i + b T i (θ − θ k ) ≤ 0 i = 1, ..., m 1 2 (θ − θ k ) T H(θ − θ k ) ≤ δ. (11) Because the Fisher information matrix (FIM) H is always positive semi-definite (and we will assume it to be positive-definite in what follows), this optimization problem is convex and, when feasible, can be solved efficiently using duality. (We reserve the case where it is not feasible  for   −1 2λ � g T H −1 g − 2r T ν + ν T Sν � + ν T c − λδ 2 , ( 12 ) where r . = g T H −1 B, S . = B T H −1 B . This is a convex program in m+1 variables; when the number of constraints is small by comparison to the dimension of θ, this is much easier to solve than (11). If λ * , ν * are a solution to the dual, the solution to the primal is θ * = θ k + 1 λ * H −1 (g − Bν * ) . ( 13 ) Our algorithm solves the dual for λ * , ν * and uses it to propose the policy update (13). For the special case where there is only one constraint, we give an analytical solution in the supplementary material (Theorem 2) which removes the need for an inner-loop optimization. Our experiments Algorithm 1 Constrained Policy Optimization Input: Initial policy π 0 ∈ Π θ tolerance α for k = 0, 1, 2, ... do Sample a set of trajectories D = {τ } ∼ π k = π(θ k ) Form sample estimates ĝ, b, Ĥ, ĉ with D if approximate CPO is feasible then Solve dual problem (12) for λ * k , ν * k Compute policy proposal θ * with (13) else Compute recovery policy proposal θ * with (14) end if Obtain θ k+1 by backtracking linesearch to enforce satisfaction of sample estimates of constraints in (10) end for have only a single constraint, and make use of the analytical solution. Because of approximation error, the proposed update may not satisfy the constraints in (10); a backtracking line search is used to ensure surrogate constraint satisfaction. Also, for high-dimensional policies, it is impractically expensive to invert the FIM. This poses a challenge for computing H −1 g and H −1 b i , which appear in the dual. Like  (Schulman et al., 2015) , we approximately compute them using the conjugate gradient method.
 Feasibility Due to approximation errors, CPO may take a bad step and produce an infeasible iterate π k . Sometimes (11) will still be feasible and CPO can automatically recover from its bad step, but for the infeasible case, a recovery method is necessary. In our experiments, where we only have one constraint, we recover by proposing an update to purely decrease the constraint value: θ * = θ k − � 2δ b T H −1 b H −1 b. (14) As before, this is followed by a line search. This approach is principled in that it uses the limiting search direction as the intersection of the trust region and the constraint region shrinks to zero. We give the pseudocode for our algorithm (for the single-constraint case) as Algorithm 1, and have made our code implementation available online. 1
 Tightening Constraints via Cost Shaping Because of the various approximations between (3) and our practical algorithm, it is important to build a factor of safety into the algorithm to minimize the chance of constraint violations. To this end, we choose to constrain upper bounds 1 https://github.com/jachiam/cpo on the original constraints, C + i , instead of the original constraints themselves. We do this by cost shaping: C + i (s, a, s � ) = C i (s, a, s � ) + Δ i (s, a, s � ), (15) where Δ i : S × A × S → R + correlates in some useful way with C i . In our experiments, where we have only one constraint, we partition states into safe states and unsafe states, and the agent suffers a safety cost of 1 for being in an unsafe state. We choose Δ to be the probability of entering an unsafe state within a fixed time horizon, according to a learned model that is updated at each iteration. This choice confers the additional benefit of smoothing out sparse constraints.
 Connections to Prior Work Our method has similar policy updates to primal-dual methods like those proposed by  Chow et al. (2015) , but crucially, we differ in computing the dual variables (the Lagrange multipliers for the constraints). In primal-dual optimization (PDO), dual variables are stateful and learned concurrently with the primal variables  (Boyd et al., 2003) . In a PDO algorithm for solving (3), dual variables would be updated according to ν k+1 = (ν k + α k (J C (π k ) − d)) + , (16) where α k is a learning rate. In this approach, intermediary policies are not guaranteed to satisfy constraints-only the policy at convergence is. By contrast, CPO computes new dual variables from scratch at each update to exactly enforce constraints.
 Experiments In our experiments, we aim to answer the following: • Does CPO succeed at enforcing behavioral constraints when training neural network policies with thousands of parameters? • How does CPO compare with a baseline that uses primal-dual optimization? Does CPO behave better with respect to constraints? • How much does it help to constrain a cost upper bound (15), instead of directly constraining the cost? • What benefits are conferred by using constraints instead of fixed penalties? We designed experiments that are easy to interpret and motivated by safety. We consider two tasks, and train multiple different agents (robots) for each task: Returns: Constraint values: (closer to the limit is better) • Circle: The agent is rewarded for running in a wide circle, but is constrained to stay within a safe region smaller than the radius of the target circle. • Gather: The agent is rewarded for collecting green apples, and constrained to avoid red bombs. For the Circle task, the exact geometry is illustrated in Figure  5  in the supplementary material. Note that there are no physical walls: the agent only interacts with boundaries through the constraint costs. The reward and constraint cost functions are described in supplementary material (Section 10.3.1). In each of these tasks, we have only one constraint; we refer to it as C and its upper bound from (15) as C + . We experiment with three different agents: a point-mass (S ⊆ R 9 , A ⊆ R 2 ), a quadruped robot (called an 'ant') (S ⊆ R 32 , A ⊆ R 8 ), and a simple humanoid (S ⊆ R 102 , A ⊆ R 10 ). We train all agent-task combinations except for Humanoid-Gather. For all experiments, we use neural network policies with two hidden layers of size (64, 32). Our experiments are implemented in rllab  (Duan et al., 2016) .
 Evaluating CPO and Comparison Analysis Learning curves for CPO and PDO are compiled in Figure  1 . Note that our constraint value graphs show C + return, instead of the C return (except for in Point-Gather, where we did not use cost shaping due to that environment's short time horizon), because this is what the algorithm actually constrains in these experiments. For our comparison, we implement PDO with (16) as the In Humanoid-Circle, the safe area is between the blue panels. update rule for the dual variables, using a constant learning rate α; details are available in supplementary material (Section 10.3.3). We emphasize that in order for the comparison to be fair, we give PDO every advantage that is given to CPO, including equivalent trust region policy updates. To benchmark the environments, we also include TRPO (trust region policy optimization)  (Schulman et al., 2015) , a stateof-the-art unconstrained reinforcement learning algorithm. The TRPO experiments show that optimal unconstrained behaviors for these environments are constraint-violating. We find that CPO is successful at approximately enforcing constraints in all environments. In the simpler environments (Point-Circle and Point-Gather), CPO tracks the constraint return almost exactly to the limit value. By contrast, although PDO usually converges to constraintsatisfying policies in the end, it is not consistently constraint-satisfying throughout training (as expected). For example, see the spike in constraint value that it experi-ences in Ant-Circle. Additionally, PDO is sensitive to the initialization of the dual variable. By default, we initialize ν 0 = 0, which exploits no prior knowledge about the environment and makes sense when the initial policies are feasible. However, it may seem appealing to set ν 0 high, which would make PDO more conservative with respect to the constraint; PDO could then decrease ν as necessary after the fact. In the Point environments, we experiment with ν 0 = 1000 and show that although this does assure constraint satisfaction, it also can substantially harm performance with respect to return. Furthermore, we argue that this is not adequate in general: after the dual variable decreases, the agent could learn a new behavior that increases the correct dual variable more quickly than PDO can attain it (as happens in Ant-Circle for PDO; observe that performance is approximately constraint-satisfying until the agent learns how to run at around iteration 350). We find that CPO generally outperforms PDO on enforcing constraints, without compromising performance with respect to return. CPO quickly stabilizes the constraint return around to the limit value, while PDO is not consistently able to enforce constraints all throughout training.
 Ablation on Cost Shaping In Figure  3 , we compare performance of CPO with and without cost shaping in the constraint. Our metric for comparison is the C return, the 'true' constraint. The cost shaping does help, almost completely accounting for CPO's inherent approximation errors. However, CPO is nearly constraint-satisfying even without cost shaping.
 Constraint vs. Fixed Penalty In Figure  4 , we compare CPO to a fixed penalty method, where policies are learned using TRPO with rewards R(s, a, s � ) − νC + (s, a, s � ) for ν ∈ {1, 5, 50}. We find that fixed penalty methods can be highly sensitive to the choice of penalty coefficient: in Ant-Circle, a penalty coefficient of 1 results in reward-maximizing policies that accumulate massive constraint costs, while a coefficient of 5 (less than an order of magnitude difference) results in cost-minimizing policies that never learn how to acquire any rewards. In contrast, CPO automatically picks penalty coefficients to attain the desired trade-off between reward and constraint cost.
 Discussion In this article, we showed that a particular optimization problem results in policy updates that are guaranteed to both improve return and satisfy constraints. This enabled the development of CPO, our policy search algorithm for CMDPs, which approximates the theoretically-guaranteed  algorithm in a principled way. We demonstrated that CPO can train neural network policies with thousands of parameters on high-dimensional constrained control tasks, simultaneously maximizing reward and approximately satisfying constraints. Our work represents a step towards applying reinforcement learning in the real world, where constraints on agent behavior are sometimes necessary for the sake of safety. the next subsection.) With B . = [b 1 , ..., b m ] and c . = [c 1 , ..., c m ] T , a dual to (11) can be expressed as max λ≥0 ν�0
 Figure 1 . 1 Figure1. Average performance for CPO, PDO, and TRPO over several seeds (5 in the Point environments, 10 in all others); the x-axis is training iteration. CPO drives the constraint function almost directly to the limit in all experiments, while PDO frequently suffers from over-or under-correction. TRPO is included to verify that optimal unconstrained behaviors are infeasible for the constrained problem.
 Figure 2 . 2 Figure 2. The Humanoid-Circle and Point-Gather environments.In Humanoid-Circle, the safe area is between the blue panels.
 Figure 3. Using cost shaping (CS) in the constraint while optimizing generally improves the agent's adherence to the true constraint on C return.
 Figure 4 . 4 Figure 4. Comparison between CPO and FPO (fixed penalty optimization) for various values of fixed penalty.

Abstract: For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (


Existential Risk and Existential Hope: Definitions

The simple definition One fairly crisp approach is to draw the line at extinction: Definition (i): An existential catastrophe is an event which causes the end of existence of our descendants. This has the virtue that it is a natural division and is easy to understand. And we certainly want to include all extinction events. But perhaps it doesn't cast a wide enough net. Example A: A totalitarian regime takes control of earth. It uses mass surveillance to prevent any rebellion, and there is no chance for escape. This regime persists for thousands of years, eventually collapsing when a supervolcano throws up enough ash that agriculture is prevented for decades, and no humans survive. In Example A, clearly the eruption was bad, but the worst of the damage was done earlier. After the totalitarian regime was locked in, it was only a matter of time until something or other finished things off. We'd like to be able to talk about entering this regime as the existential catastrophe, rather than whatever event happens to end it. So we need another definition. Although we'll now look at other definitions for existential catastrophes, we do like the simple definition. Luckily there's another term that's already understood: human extinction. Sometimes it's better to talk about extinction risks rather than existential risks, as 'existential risk' is a piece of jargon, whereas 'extinction risk' will be clear to everyone.
 Bostrom's definition Nick Bostrom introduced the concept of existential risks. He has defined them as follows: Definition (ii): An existential risk is one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development.  1  This definition deals well with Example A, placing the existential catastrophe at the point where the totalitarian regime arose, as this caused the permanent and drastic destruction of [humanity's] potential for desirable future development. Example B: A totalitarian regime takes control of the earth. There is only a slight chance that humanity will ever escape. Is this an existential catastrophe? Bostrom's definition doesn't clearly specify whether it should be considered as one. Either answer leads to some strange conclusions. Saying it's not an existential catastrophe seems wrong as it's exactly the kind of thing that we should strive to avoid for the same reasons we wish to avoid existential catastrophes. Saying it is an existential catastrophe is very odd if humanity does escape and recover -then the loss of potential wasn't permanent after all. The problem here is that potential isn't binary. Entering the regime certainly seems to curtail the potential, but not to eliminate it.
 Definition via expectations The idea that potential isn't binary motivates our suggested definition: Definition (iii): An existential catastrophe is an event which causes the loss of a large fraction of expected value. This definition deals well with Example B. If we enter into the totalitarian regime and then at a later date the hope of escape is snuffed out, that represents two existential catastrophes under this definition. We lost most of the expected value when we entered the regime, and then lost most of the remaining expected value when the chance for escape disappeared. A lot of the work of this definition is being done by the final couple of words. 'Value' refers simply to whatever it is we care about and want in the world, in the same way that 'desirable future development' worked in Bostrom's definition. And to talk about expectations we need to have some probabilities in mind. Here we are thinking of objective probabilities. Note that 'potential' in Bostrom's definition requires some similar work to making assumptions about probabilities.
 Existential eucatastrophes and existential hope If we enter the totalitarian regime and then manage to escape and recover, then we had an existential catastrophe which was balanced out by a subsequent gain in expected value. This kind of event gives us a concept parallel to that of an existential catastrophe: Definition (iv): An existential eucatastrophe 2 is an event which causes there to be much more expected value after the event than before. This concept is quite natural. We saw it in the context of escape from a regime which threatened the existence of a prosperous future. Our world has probably already seen at least one existential eucatastrophe: the origin of life. When life first arose, the expected value of the planet's future may have become much bigger. To the extent that they were not inevitable, the rise of multicellular life and intelligence may also have represented existential eucatastrophes. In general successfully passing any 'great filter' 3 is an existential eucatastrophe, since beforehand the probability of passing it is small, so the expected value is much smaller than after the filter is dealt with. Armed with this concept, we can draw a new lesson. Just as we should strive to avoid existential catastrophes, we should also seek existential eucatastrophes. In some ways, this isn't a new lesson at all. Under Bostrom's definition we are comparing ourselves to the most optimistic potential we could reach, so failing to achieve a eucatastrophe is itself a catastrophe. However we think more naturally in terms of events than non-events. If life fails to arise on a planet where it might have, it's much clearer to think of a failure to achieve a eucatastrophe than of an existential catastrophe stretching out over the billions of years in which life did not arise. Just as we tend to talk about the existential risk rather than existential catastrophe, we want to be able to refer to the chance of an existential eucatastrophe; upside risk on a large scale. We could call such a chance an existential hope. In fact, there are already people following both of the strategies this suggests. Some people are trying to identify and avert specific threats to our future -reducing existential risk. Others are trying to steer us towards a world where we are robustly well-prepared to face whatever obstacles come -they are seeking to increase existential hope.
 Conclusions We were interested in pinning down what is meant by 'existential risk'. Much of the time, all of the definitions we've looked at will agree on whether something is an existential risk. Keeping it simple can be good, because it helps more people to understand. We therefore advocate talking about 'extinction risks' rather than 'existential risks' when the former term will work. Nonetheless, we may sometimes have to consider more unusual scenarios. It's good to know how to make the definition work well there as it can help us to think about things more clearly. We think that the definition in terms of expectations does a better job of this than previous definitions. In devising the notion of existential catastrophe (and hence existential risk) via expectations, we came across the dual concept we have called 'existential eucatastrophe' (and hence 'existential hope'). We think this captures a natural class of events, and what may be an important one. We hope that having a label for the concept may help others to make better judgements about what courses to pursue. 			 * Future of Humanity Institute, University of Oxford & Centre for Effective Altruism † Future of Humanity Institute, University of Oxford
			 Bostrom, Existential Risk Reduction as Global Priority, Global Policy, Vol 4, Issue 1 (2013), p15
			 The word 'eucatastrophe' is made of the Greek root 'eu-' meaning 'good' and the word 'catastrophe' in its classical sense of a sudden turn. It was coined by Tolkien to refer to the sudden and unexpected turn for the better frequently found at the end of fairy tales. (Tolkien, John Ronald Reuel. On fairy-stories. Oxford University Press, 1947.)3 Hanson, Robin, The Great Filter -Are We Almost Past It?, 1998.

Abstract: We look at the strengths and weaknesses of two existing definitions of existential risk, and suggest a new definition based on expected value. This leads to a parallel concept: 'existential hope', the chance of something extremely good happening. An existential risk is a chance of a terrible event occurring, such as an asteroid striking the earth and wiping out intelligent life -we could call such events existential catastrophes. In order to understand what should be thought of as an existential risk, it is necessary to understand what should be thought of as an existential catastrophe. This is harder than it first seems to pin down.


Michele Piccione
Ariel Rubinstein
Bob Aumann
Paolo Battigalli
Sergiu Hart
Dana Heller
Bart Lipman
Avishai Margalit
Roger Myerson
Hugh Neary
Motty Perry
Tim Van Zandt
Ruth Weintraub

On the Interpretation of Decision Problems with Imperfect Recall*

INTRODUCTION This paper is an examination of some modelling problems regarding imperfect recall within the model of extensive games. It is argued that, if the assumption of perfect recall is violated, care must be taken in interpreting the main elements of the model. Interpretations that are inconsequential under perfect recall have important implications in the analysis of games with imperfect recall. The distinction between perfect and imperfect recall for extensive games Ž . was introduced in  Kuhn 1953  . Since then, traditional game theory has excluded games with imperfect recall from its scope. In this paper, we wish to readdress this topic. Since the interpretative issues on the agenda appear within the framework of extensive games with a single player, we confine our discussion to decision problems with imperfect recall. Extensive decision problems are a special case of extensive games in that the set of players is a singleton. Our basic understanding of an extensive decision problem includes the following assertions: 1. Informational assumptions are modeled by partitioning the situa-Ž . tions histories where the decision maker takes an action into ''information sets.'' The interpretation of the ''informational structure'' is that the decision maker knows the information set he is at but, if an information set is not a singleton, he cannot distinguish among the points in the set he has reached. The decision maker, however, can make inferences or form a belief. 2. A strategy for the decision maker assigns to each information set one distinct action which is executed whenever an information set is reached. The decision maker cannot plan to assign different actions to two histories which lie in the same information set. 3. If the decision maker assesses his strategy at an information set including more than one history, he forms beliefs about the histories which led to it. These beliefs are the basis for his considerations. A decision problem exhibits imperfect recall if, at a point of time, the decision maker holds information which is forgotten later on. Specifically, an information set includes some histories which are incompatible with previously held information. Figures 1᎐3 are examples that illustrate three main aspects of imperfect recall in decision problems. The standard motive of imperfect recall appears in Example 2. The decision maker is initially informed about the move of chance and loses  . 1 which is central in this paper is rather unconventional; the decision maker does not distinguish between the first and the second nodes. Reaching the second node, he loses the information that he had previously made a choice. The inability of a decision maker to distinguish between two histories on the same path will be referred to as absentmindedness. In the above examples, the information sets determine the ability of the decision maker to recall. In practice, a decision maker can affect what he remembers. In this paper, however, we assume that the decision maker is not allowed to employ an external device to assist him in keeping track of the information which he would otherwise lose. Thus, we sidestep the decision maker's considerations regarding the trade-off between ''more memory'' and ''memory costs.''  FIG. 3. Example 3.  The common interpretation of a decision problem with imperfect recall refers to a situation in which an individual takes several successive actions and faces memory constraints. Imperfect recall, however, is not necessarily related to the mental phenomenon of memory loss and may also reflect the imperfect ability to make the inferences necessary for distinguishing among different points in the same information set. A decision maker may not realize that he is at the 17th exit along a highway, either because he does not recall whether he has passed the 16th intersection, or because he cannot infer that he is at the 17th intersection, despite the perfect pictures of each intersection in his mind. The latter interpretation of imperfect recall brings our discussion closer to the topic of ''bounded rationality.'' Ž . An alternative interpretation is found in Isbell 1957 . The decision maker is an organization consisting of many agents who have the same interests and act at different possible instances. In this case, the decision process of the organization may exhibit imperfect recall either because it must keep the instructions given to agents acting in successive situations simple or because of communication problems between agents. Agents receive instructions on how to behave and the collection of these instructions is equivalent to the notion of a strategy. From a psychological point of view, imperfect recall is a very important phenomenon as it puts severe constraints on a decision maker's behavior. We believe, however, that the ultimate proof for its relevance in economic analysis can only come from an interesting model which clearly explains an economic phenomenon. Constructing such a model is beyond the confines of this paper. The structure of the paper is as follows. We begin by presenting an example which we call the '' paradox of the absentminded dri¨er.'' This example is used to illustrate many of the points of this paper and will be referred to repeatedly. After providing the formal definition of the model, we discuss two issues of importance for decision problems with absent-mindedness. First, in Section 4, we review the circumstances in which whether the decision maker is allowed to randomize affect the analysis. Second, in Section 5, we show that the extension of Bayesian updating to decision problems with absentmindedness is not trivial. The two main topics which will be addressed are the timing of decision and the multiself approaches. The significance of these issues is marginal in decision problems with perfect recall since possible answers are inconsequential for the analysis.
 Ž . i Timing of decision. This issue deals with the interpretation of an information set as either a point of decision or a point in which a strategy is executed. In Section 6, we will show that for decision problems with imperfect recall optimal strategies may be time inconsistent and that strategies which are time consistent may not be optimal. The timing of decisions can be an important consideration for a decision maker whereas, with perfect recall, there is no reason to make some decisions at a particular point of time.
 Ž . ii The multisel¨es approach to decision making. Standard dynamic inconsistencies are generally addressed by assuming that the decision maker acts as a collection of distinct ''selves'' who behave independently. The behavior of the decision maker is analyzed as an equilibrium. In Section 7, we extend the multiself approach to decision problems with imperfect recall and show that in this extension an optimal strategy is dynamically consistent.
 THE PARADOX OF THE ABSENTMINDED DRIVER An individual is sitting late at night in a bar planning his midnight trip home. In order to get home he has to take the highway and get off at the Ž . second exit. Turning at the first exit leads into a disastrous area payoff 0 . Ž . Turning at the second exit yields the highest reward payoff 4 . If he continues beyond the second exit, he cannot go back and at the end of the Ž . highway he will find a motel where he can spend the night payoff 1 . The driver is absentminded and is aware of this fact. At an intersection, he cannot tell whether it is the first or the second intersection and he cannot Ž remember how many he has passed one can make the situation more . realistic by referring to the 17th intersection . While sitting at the bar, all he can do is to decide whether or not to exit at an intersection. We exclude at this stage the possibility that the decision maker can include random elements in his strategy. Example 1 describes this situation. Planning his trip at the bar, the decision maker must conclude that it is impossible for him to get home and that he should not exit when reaching an intersection. Thus, his optimal plan will lead him to spend the night at the motel and yield a payoff of 1. Now, suppose that he reaches an intersection. If he had decided to exit, he would have concluded that he is at the first intersection. Having chosen the strategy to continue, he concludes that he is at the first intersection with probability 1r2. Then, reviewing his plan, he finds that it is optimal for him to leave the highway since it yields an expected payoff of 2. Despite no new information and no change in his preferences, the decision maker would like to change his initial plan once he reaches an intersection! Note that if the decision maker now infers that he would have exited the highway had he passed the first intersection, his reasoning becomes circular; he must conclude that he is at the first intersection and that it is optimal to continue. We wish to emphasize that this is not a standard example of time inconsistency. Usually, time inconsistency is obtained as a consequence of Ž . changes in either preferences tastes or information regarding the moves of nature during the execution of the optimal plan. Here, preferences over final outcome are constant and the only factor intervening between planning and execution of the optimal strategy is the occurrence of the situation which calls for execution, that is, reaching the intersection. We find this example paradoxical as it exhibits a conflict between two ways of reasoning at the intersection. The first is based on a quite minimal principle of rationality; having chosen an optimal strategy, one does not have to verify its optimality at the time of execution unless there is a change in information or in preferences. In our example, the decision maker knew he would reach the intersection with certainty and his preferences are constant. This principle leads to the conclusion that the decision maker should stick to his plan to continue. The second way is based on the principle which calls at each instance to maximize expected payoffs given the relevant beliefs. In our example, this principle leads to the conclusion of exiting. The conflict between these two potential lines of reasoning is at the root of the apparent ambiguity of our example.
 EXTENSIVE DECISION MODEL In this section, a formal definition of the extensive decision model is Ž . given. The presentation follows that of  Osborne and Rubinstein 1994 .  The reader can easily identify the model with the standard definition in the ''tree'' language. Ž . ² : A finite decision problem is a five-tuple ⌫ s H, u, C, , I , where: Ž . a H is a finite set of sequences. We assume that the empty se-Ž . quence, , is an element of H and that if a , . . . , a g H and 1 K Ž . Ž . a, . . . , a / then a , . . . , a g H. 1 K 1 Ky1 Ž . We interpret a history a , . . . , a g H as a feasible sequence of 1 K Ž . actions taken by the decision maker or by chance. The history a , . . . , a 1 K Ž . g His terminal if there is no a , . . . , a , a g H. The set of terminal 1 K histories is denoted by Z. The set of actions available to the decision Ž . Ä maker or chance after a nonterminal history h is defined by A h s a: Ž . 4 Ž . h,a g H . To avoid degenerate cases we assume that A h contains at least two elements. When presenting a decision problem diagramatically, we draw H as a tree whose nodes are the set of histories with root and Ž . Ž . whose edges combine a node a , . . . , a with a node a , . . . , a . 1 K 1 Kq1 Ž . Ž . b u: Z ª R is a utility function which assigns a number payoff to each of the terminal histories. Preferences are defined on the set of all lotteries over terminal histories and satisfy the VNM assumptions.
 Ž . c C is a subset of H. We assume that the chance player moves after histories in C.
 Ž . d is the decision maker's belief about the chance player's behav-Ž . ior. assigns to each history h g C a probability measure on A h . To Ž . avoid degeneracy, we assume that h, a is strictly positive for all h g C Ž . and a g A h . Thus, the set of histories H is partitioned into three subsets: Z, the set of terminal histories; C, the set of histories after which chance moves; D s H y Z y C, the set of histories after which the decision maker moves.
 Ž . e The set of information sets, which is denoted by I, is a partition of Ž . D. We assume that for all h, hЈ in the same cell of the partition A h s Ž . A hЈ ; i.e., the sets of actions available to the decision maker at histories in the same information set are identical. For convenience, with a slight abuse of notation, we will sometimes denote the set of actions which are Ž . available at a history in X by A X . Note that, in contrast to some authors, we do not exclude from the class Ž of decision problems those which exhibit absentmindedness see definition . below . If all information sets in I are singletons we say that ⌫ is a decision problem with perfect information. Ž . A pure strategy, f, is a function which assigns to every history h g D Ž . an element of A h with the restriction that if h and hЈ are in the same Ž . Ž . information set f h s f hЈ . Notice that this definition requires that the decision maker plans an action at histories which he will not reach if he follows the strategy. We are now ready for the main definitions of this paper. The experience Ž . of the decision maker at a history h in D, denoted by exp h , is the sequence of information sets and actions of the decision maker along the history h. We adopt the convention that the last element in the sequence Ž . exp h is the information set which contains h. A decision problem has perfect recall if for any two histories, h, hЈ g D, Ž . Ž . which lie in the same information set, exp h s exp hЈ . Thus, in a decision problem with perfect recall, the decision maker ''remembers'' the succession of the information sets he has faced and the actions he has taken. A decision problem for which the above condition is violated is referred to as a decision problem with imperfect recall. Ž . Given a history h s a , . . . , a and L -K, the history hЈ s 1 K Ž . a, . . . , a is a subhistory of h. We say that a decision problem ⌫ exhibits 1 L absentmindedness if there are two histories h and hЈ such that hЈ is a subhistory of h and both belong to the same information set. The decision problem illustrated in Example 1 exhibits absentminded-Ž . ness since history B and its subhistory are in the same information set.
 THE VALUE OF RANDOMIZATION In this section, we discuss the implications of enlarging the strategy set of a decision maker to include random strategies. Given that the decision maker behaves as an expected utility maximizer, randomization over pure strategies is redundant for problems of either perfect or imperfect recall. Define a mixed strategy to be a probability distribution over the set of pure strategies. It describes a behavior in which randomization occurs only at the outset, before the decision problem unfolds. Each pure strategy induces a lottery over Z. A mixed strategy induces a lottery over Z which is the compound lottery of the lotteries induced by each of the pure strategies in its support. Therefore, no mixed strategy can be strictly preferred to all the pure strategies. Behavior strategies perform a different method of randomization. A beha¨ioral strategy, b, is a function which assigns to every history h g D, a Ž . Ž .
 Ž . Ž . distribution b h over A h such that b h s b hЈ for any two histories h and hЈ which lie in the same information set. In decision problems without Ž . absentmindedness b h is a lottery which is realized when the information set which contains h is reached. For decision problems with absentminded-Ž . ness we take b h to be a random device which is activated independently every time the information set which includes h is reached. Consider again Example 1. In this problem there are two pure strategies, ''B'' and ''E'' which yield payoffs of 1 and 0, respectively. Although the absentminded driver cannot use a pure strategy to reach home with certainty, he can toss a coin and obtain an expected payoff of 1.  25 . Note 1 that his optimal behavioral strategy is to exit with probability and yields 3 4 the expected payoff of . 3 It turns out that absentmindedness is necessary for behavioral strategy Ž . to be strictly optimal. This was shown in Isbell 1957 and we provide the proof for completeness. PROPOSITION 1. Suppose ⌫ does not exhibit absentmindedness. Then for any beha¨ioral strategy there is a pure strategy which yields a payoff at least as high. ² : Con¨ersely, suppose ⌫ s H, u, C, , I exhibits absentmindedness. Then, ² : there exist a decision problem ⌫Ј s H, uЈ, C, , I and a beha¨ioral strategy which yields a payoff strictly higher than any payoff achie¨ed by a pure strategy. Proof. See the Appendix. The Paradox of the Absentminded Dri¨er Re¨isited. The inconsistency discussed in Section 2 is not a consequence of the restriction that the strategy set includes only pure strategies and persists when the decision maker is allowed to choose random actions. The optimal behavioral 2 strategy is to choose B with probability . Reaching the intersection, the 3 driver will form beliefs about where he is. Denote by ␣ the probability he assigns to being at the first intersection. Then, his expected payoff is w 2 Ž . x Ž .w Ž . x ␣ p q4 1 y p p q 1 y ␣ p q 4 1 y p , where p is the probability of Ä Ž . 4 not exiting. The optimal p is now max 0, 7␣ y 3 r6␣ . This is inconsistent with his original plan unless ␣ s 1. In other words, his original plan is time consistent if and only if he believes that there is no chance he has passed the first intersection. We find such a belief unreasonable. Given his strategy it seems natural to assign to the second intersection a probability 2 which is times the probability assigned to the first intersection, which 3 implies ␣ s 0.6. The issue of consistent beliefs will be discussed in the next section. This type of time inconsistency can appear also in decision problems in which the optimal strategy is pure. Consider the following example shown in Fig.  4 . The optimal behavioral strategy is the pure strategy which selects L at d and L at d . To verify it, denote by ␣ and ␤ the probabilities of
 CONSISTENT BELIEFS If information sets are to be interpreted as points of decision, Examples 1 and 4 suggest that a decision maker who acts on the basis of expected utility maximization may be unable to execute the optimal strategy. The first step in addressing this issue is to specify the decision maker's beliefs at an information set which is not a singleton. As we shall see, finding an appropriate specification for decision problems with absentmindedness is not conceptually trivial. We define a belief system as a function which assigns to any Ž < . information set X and any history h g X a nonnegative number h X Ž < . such that Ý h X s 1. The interpretation is that the decision maker, h g X Ž < . upon reaching X, assigns probability h X to the possibility that he is at Ž < . h. Let p h hЈ, b be the probability that, conditional on reaching hЈ, the history h will be realized when executing the strategy b. We denote Ž < . Ž < . p h ,b by p h b . Several alternatives are conceivable for the specification of the decision maker's beliefs. Since our objective is to examine the optimality of a strategy during its execution, we find it natural to assume that the beliefs of the decision maker are related in a systematic way with the strategy to be assessed. The condition that we require a belief system to satisfy to be consistent with a behavioral strategy b mirrors the frequency approach to belief formation. Namely, if an information set X is reached with Ž < . positive probability, h X is assumed to be equal to the long run proportion of times in which ''visiting'' the information set X involves being in h for a decision maker who plays the decision problem again and again and follows b.  DEFINITION.  A belief system is consistent with the behavioral strategy b if for every information set X which is reached with positive Ž < . Ž < . Ž < . probability and for every h g X, h X s p h b rÝ p hЈ b .
 hЈg X Our definition of consistency imposes restrictions only on beliefs at information sets which are reached with positive probability. Notice that, for decision problems without absentmindedness, consistency is equivalent to Bayes' formula. For decision problem with absentmindedness, however, the denominator can be greater than one. The similarity with Bayes' formula is only notational. To clarify this definition, consider first the absentminded driver example  Our definition of consistent beliefs can also be motivated as being derived from a probability space which includes the time at which the decision maker can be. As an illustration consider Example 5 and assume that each action takes one unit of time. The relevant space of instances Ž . Ž . Ž . Ž . Ž . consists of L, 1 , L, 2 , R, 1 , and R, 2 , where x, t is the instance in which the chance player chooses x and time is t. Assuming equal probabilities for each instance is consistent with the description of the chance player in the decision problem. Then, a decision maker who is told that he is at d updates his belief by the Bayesian formula. For example, the 1 unconditional probability of the second node after R is pr4, where p is the probability of choosing B at d , and the unconditional probability of  In this paper, we adopt the above definition of consistency. However, we find the following definition of consistency reasonable as well. DEFINITION. A belief system is Z-consistent with the behavioral strategy b if for every information set X which is reached with positive probability and for every h g X < h X Ž . < Ä 4 Ý p z b r࠻ hЈ ¬ hЈgX and is a subhistory of z Ž . Äz<h is a subhistory of z4 s . < Ý p z b Ž . Äz<z has a subhistory in X 4 The rationale behind this definition is that, given the behavioral strategy b, the probability of the event that the random elements which determine Ž < . the terminal history z are realized is p z b ; if the history z includes K histories in which the decision maker is asked to act then he assigns to Ž < . each of them the ex ante probability p z b rK. To illustrate the Z-consistency, consider again the absentminded driver 1 example and the strategy that selects B with probability Note that the paradoxical flavor of the absentminded driver example is unaffected by the type of consistency we adopt. Also, if the decision problem does not exhibit absentmindedness then consistency and Z-consistency are identical. We do not have a firm view about the ''right'' definition of consistent beliefs. The issue deserves further investigation. We wish to point out that if the decision maker adopts Z-consistency then he is exposed to a sort of ''money pump.'' Consider Example 5 and the behavioral strategy that selects B with probability 1. Given Z-consistent beliefs a risk neutral decision maker, upon reaching the information set, will always accept an Ž . agreement in which he gains $1.1 if he is at L and loses $1 otherwise. If such an agreement is offered whenever the information set is reached, the resulting undesirable lottery yields the decision maker $1.1 with probability 0.5 and y$2 with probability 0.5.
 TIME CONSISTENCY When does the decision maker make his decision? Can he decide about the point of time at which a decision is made? To what extent can the decision maker commit to decisions he makes? These questions are superfluous for decision problems with perfect recall since the ex ante optimal strategy remains optimal during its execution. In the presence of imperfect recall, the optimal strategy may cease to be optimal along its execution and answers to these questions are significant for the analysis. We say that a strategy is time consistent if there is not, at any information set which is reached as the decision problem unfolds, a different strategy for the remainder of the decision problem which yields a higher expected payoff. Notice that we require the optimality to be assessed only at the information sets and not before the initial history. The significance of the statement that a strategy is optimal at an information set depends on the beliefs that the decision maker is assumed to hold. Requiring that beliefs be consistent leads to the following definition.  DEFINITION . A behavioral strategy b is time consistent if there is a belief consistent with b such that for every information set X which is reached with positive probability under b, h p z¬h,b u z G h p z¬h,bЈ u z Ž . Ž . Ž . Ž . Ž . Ž . Ý Ý Ý Ý h gX z g Z h g X z g Z for any behavioral strategy bЈ. The notion of time consistency that we use is a very strong criterion. We must emphasize that it does not rest on a model of how decisions are made at each information set. The definition presumes that the decision maker, in principle, can commit to a future course of action. Of course, if the decision maker is aware that a revision of the plan is possible at future information sets, expectations of commitments are naive. Time consistency is an assessment as to whether, conditional upon the realization of information set, there exists an alternative rule of behavior which yields the decision maker a higher payoff. If an optimal strategy satisfies this criterion, the distinction with respect to information sets as points of decision or points of execution is inconsequential for a decision maker who is implementing it and whose beliefs are consistent. If, however, an optimal strategy fails this criterion, this distinction becomes significant and alternative specifications of the domain of choice and the timing of decision of the decision maker can give rise to different analyses. A well-known result states that for decision problems with perfect recall a strategy is optimal if and only if it is time consistent. This is not the case for problems with imperfect recall. As we have seen, the optimal behavioral strategy for the absentminded driver is not time consistent. It is easy 5 to see that the only time consistent strategy is to exit with probability . In the case of absentmindedness, the divergence between optimality and time consistency is two-sided. The only optimal strategy is not time consistent and the only time consistent strategy is not optimal. The next Ž . example Fig.  6  shows that, even without absentmindedness, a strategy may be time consistent and not optimal. Consider the strategy where the decision maker plays B at both information sets. The belief system consistent with this strategy assigns equal probabilities to each of the histories in each of the information sets. Therefore, this strategy is time consistent although the optimal strategy is to choose S at both information sets. The next proposition provides conditions under which optimal and time consistent strategies are equivalent. Two histories are said to split at h if h is their longest common subhistory. PROPOSITION 2. Let ⌫ be a decision problem without absentmindedness for which for any information set X and two histories hЈ, hЉ g X which split at h g C, the information sets which appear in exp hЈ are the same as in Ž . exp hЉ .
 Ž . Ž . ) A beha¨ioral strategy for ⌫ is optimal if and only if it is time consistent. probability one to nature's move R immediately after R and probability 2 to both of nature's moves when R is followed by his action B.
 THE MULTISELF APPROACHES
 Ž . For standard dynamic inconsistencies, Strotz 1956 provides a framework of analysis in which every information set is assumed to be a point of decision and the decision maker is unable to control his behavior at future information sets. A decision maker acts as a collection of hypothetical Ž . agents selves whose plans form an equilibrium. More precisely, for any Ž . decision problem ⌫ define G ⌫ to be the extensive game in which each information set of ⌫ is assigned a distinct player, and all players have the same payoff as the decision maker. The behavior of the decision maker in Ž . ⌫ is then analyzed as an equilibrium of G ⌫ . As is well known, any optimal play for ⌫ is the play induced by some subgame perfect equilib-Ž . Ž . rium of G ⌫ , and any subgame perfect equilibrium of G ⌫ corresponds to an optimal strategy. This property has the consequence that the backward induction algorithm is a procedure for solving a decision problem with perfect information. An analogous result is valid for decision problems with perfect recall and imperfect information. In this case we use the solution concept of sequential equilibrium which combines sequential rationality with the requirement of consistent beliefs. The set of distributions over the terminal nodes generated by the sequential equilibria of Ž . G ⌫ is identical to the set of distributions generated by the optimal Ž . strategies of ⌫ see  Hendon, Jacobsen, and Sloth, 1993 .  These results are called the ''no single improvement'' property; a strategy is optimal if, conditional upon reaching an information set, the decision maker cannot improve his expected payoff by changing only the action prescribed by the strategy for that information set. It implies that it makes no difference whether we model a decision problem piecemeal or in its entirety. The equivalence of the single-self and the multiselves approaches for decision problems with perfect recall breaks down when we analyze decision problems with imperfect recall. Consider Example 3. The Ž . optimal strategy R, r corresponds to a sequential equilibrium in the corresponding multiself game. However, the multiself approach does not Ž . rule out the inferior strategy L, l ; the decision maker cannot improve his payoff by simply changing the action prescribed at a single information set. Conditional upon being at d , the choice of L is optimal if the decision 1 maker treats his behavior at d as given and unchangeable and believes he 2 will play l. The Paradox of the Absentminded Dri¨er Re-re¨isited. Consider again the paradox of the absentminded driver. The formal notion of sequential equilibrium is not applicable to decision problems with absentmindedness. However, the requirement that the behavioral strategy at each information set is optimal, given the consistent beliefs, is meaningful. Despite the fact that there is only one information set, this requirement generates a strong dependence between behavioral strategies and beliefs. As we have noted before, the only strategy which satisfies this requirement is to exit with 5 probability , whereas the strategy that maximizes expected payoffs ex ante 9 1 is to exit with probability . Nevertheless, one can modify the multiself 3 approach to avoid the possibility that an optimal strategy does not satisfy sequential rationality with respect to consistent beliefs. Beliefs consistent with the optimal behavioral strategy in the paradox of 3 the absentminded driver assign probability to the first intersection. If the 5 decision maker anticipates that his ''twin-self,'' if asked to move again, will 1 1 exit with probability , then it is optimal for him to exit with probability 3 3 3 2 Ž w x w x as he is indifferent between E yielding the expected payoff 0 q 4 5 5 8 3 1 2 28 . Ž w Ž . Ž . x w x . s and B yielding an expected payoff of 4 q 1 q 1 s . In this modification, when considering a deviation at an information set X from the behavioral strategy assigned to X, the ''self'' assumes that his ''twin-self'' will use the equilibrium strategy if asked again to act, regardless of the choice he now makes. The requirement that the decision maker uses identical randomization at all histories in the same information set is retained only in equilibrium. In an actual play of the decision problem, if he decides to deviate, he may use different randomizations at different histories in the same information set. Also notice that, for the optimal strategy to satisfy sequential rationality in this modification of the multiself approach, the decision maker must 3 believe that he is at the first intersection with probability . In particular, if 5 one replaces our definition of consistency of beliefs with Z-consistency, sequential rationality fails in this modification as well. We now define the ''modified multiself'' approach formally and prove that every optimal Ž strategy is modified multiself consistent discussions with Bob Aumann and . Roger Myerson were very helpful in shaping this part of the paper .  DEFINITION . A behavioral strategy b is modified multiself consistent if there exists a belief consistent with b such that for every information set Ž . X which is reached with positive probability and for every action a g A X Ž .Ž . Ž . for which b h a ) 0 for h g X, there is no aЈ g A X such that h P z¬ h,aЈ ,b u z Ž . Ž . Ž . Ž . Ý Ý h gX z g Z ) h P z¬ h,a ,b u z . Ž . Ž . Ž . Ž . Ý Ý h gX z g Z Note that in contrast with the approach discussed in the first part of this section, the above definition imposes requirements only on the selves which are reached with positive probability. PROPOSITION 3. If a beha¨ioral strategy is optimal then it is modified multiself consistent. Proof. See the Appendix. Proposition 3 demonstrates that, within the framework of the ''modified'' multiself approach, the inconsistency of optimal plans for decision problems with absentmindedness evaporates. The interpretative ambiguities originating in the paradox of the absentminded driver also vanish as the 2 strategy to exit with probability is singled out as the only optimal and 3 consistent strategy. The value of this resolution, however, depends on the appropriateness of the ''modified'' multiself approach as a theory of decision making under imperfect recall. The behavioral assumption of the multiself approach is that each self maximizes his conditional expected payoff given his own beliefs. In decision problems with perfect recall, beliefs at each information set are independent of the choices of previous selves and each self can determine the optimal decisions of future selves by recursion. In decision problems with imperfect recall, however, the conditional payoff of each self depends on his beliefs about the behavior of earlier selves and the independence of beliefs and actions from previous choices is an additional assumption. For decision problems with absentmindedness, the ''modified'' multiself approach assumes that a decision maker, upon reaching an information set, takes his actions to be immutable at future occurrences of that information set, no matter which course of action he is contemplating now. At the other extreme one finds the opposite axiom for which only one self resides in the information set and expects that, were the information set to occur again, he would adopt whichever behavioral rule he adopts now. In this case, the strategy to exit 5 with probability would be the consistent rule of behavior for Example 1. 9 These two assumptions reflect two alternative ways of reasoning at the information set. We find both of them to have some intuitive appeal and neither to be universally valid.
 FINAL COMMENTS Our observations are only a limited exercise intended to suggest some of the issues which a comprehensive theory of imperfect recall must confront. We conclude our discussion with the following three comments: a. Other types of imperfect recall. We model imperfect recall by including in the same information set histories which contain different experiences. Such histories are assumed to be indistinguishable for the decision maker. A different type of imperfect recall is represented by a situation in which ''I know that at the first intersection I would be aware of this but at the second intersection I do not know whether I am at the first or the second intersection.'' One way to model such a scenario is by nonparti-Ž . Ž . tional informational structures see  Geanakopolos, 1990 . Rubinstein 1991  discussed the difficulty of modelling considerations such as ''At the first intersection I know that I am there but if I reach the second intersection I Ž think that there is a 10% chance that I am at the first intersection'' see . also Fluck, 1994 . It seems to us that new analytical frameworks are needed to address these issues as well as issues such as partial recall of strategy and imperfect ability to make inferences. b. Games with imperfect recall. This paper focuses on issues related to modeling decision problems with imperfect recall. Obviously, such issues carry over to extensive games. Isbell and Kuhn show that the nonequivalence between mixed and behavioral strategies can cause nonexistence of Nash equilibria in behavioral strategies. Time inconsistency of optimal strategies for some decision problems with imperfect recall makes it possible to construct games with imperfect recall that have no equilibria which satisfy sequential rationality with respect to consistent beliefs. Ambiguities in the interpretation of games with imperfect recall is probably the reason why only a handful of papers have dealt with the topic. Ž . Ž . The few exceptions include  Isbell 1957 and Alpern 1988  , who prove the existence of mixed behavioral strategy equilibrium in a class of games with Ž . imperfect recall, and Binmore 1992 who relates imperfect recall and nonpartitional knowledge. The literature on machines playing repeated Ž . games see, for example, Rubinstein, 1986 may also be interpreted as an analysis of a class of repeated games with imperfect recall, where it is costly for the players to distinguish between different histories of the game. Further investigation is needed to adapt solution concepts designed for games with perfect recall to games with imperfect recall. c. The paradox of the absentminded dri¨er. In all its forms the absentminded driver example exhibits a conflict between two types of reasoning. Commitment to the ex ante optimal behavioral strategy is obviously the normative rule of behavior. We do not have a firm view about the resolution of the paradox. We have investigated one resolution which requires dividing a decision maker into multiple independent selves. Another resolution would entail the rejection of expected utility maximization given consistent beliefs when the information set includes histories whose probabilities depend on the decision maker's actions at that information set. Savage's theory views a state as a description of a scenario which is independent of the act. In contrast, ''being at the second intersection'' is a state which is not independent from the action taken at the first, and, consequently, at the second intersection.
 APPENDIX Proof of Proposition 1. Consider an information set X and h g X. Let Ž .Ž . b be a behavioral strategy and b h a denote the probability of choosing Ž . a g A h at h. Since ⌫ does not exhibit absentmindedness, the expected Ž .Ž . payoff from b can be written as Ý b h a g q g, where each Suppose that ⌫ exhibits absentmindedness. Then, there exist a final Ž . Ž . history z and two distinct subhistories of z, h, a and hЈ, aЈ , such that h and hЈ belong to the same information set and a / aЈ. Consider a payoff Ž . Ž . function uЈ such that uЈ z s 1 and uЈ z s 0 for z / z. The claim follows, since z cannot be reached by any pure strategy, and is reached with positive probability by any behavioral strategy which assigns a positive probability to each action. Q.E.D.   FIG. 1 . 1 FIG. 1. Example 1.
 FIG. 2 . 2 FIG. 2. Example 2.
 FIG. 4. Example 4.
 1 and the strategy that selects B with probability equal to . A consistent 2 2 belief, conditional upon the information set d , assigns probability to 1 3 Ž . being at the first intersection. Consider next Example 5 see Fig. 5 and the strategy which selects B with probability 1.
 FIG FIG. 5. Example 5.
 R is . If p s 1, the conditional probability of each of the 4 1 three nodes in d is .
 9 The problem of time consistency can also arise without absentmindedness as Example 2 shows. The optimal strategy is to choose S at d , B at1 d , and R at d . However, upon reaching d the decision maker prefers 2 3 1 using B at d and L at d . A decision maker who can postpone his 1 3 decisions until he has reached d or d can make a better decision 1 2 regarding the choice at d than a decision maker who decides ex ante. A 3 complete model should indicate if the domain of choice of the decision maker includes timing of the decision and the extent of his ability to use his knowledge at d and d to influence his choice at d .
 g and g are independent of b h . Consider a) g A h which a Ž .Ž . achieves the highest g . Setting b h a) s 1 yields a payoff at least as a high. The claim follows by repeating the argument for every information set.
 2. Let b denote the payoff that the decision Ž . maker obtains by playing b. Given an information set X, let Z X denote Ž . the set of final histories having subhistories in X. Then, we can write b as < b s p z ¬ b u z q p z b u z .
 Since ⌫ does not exhibit absentmindedness, a terminal history z in Z X Ž < . Ž < . Ž < . has a unique subhistory h in X. Thus, p z b s p h b p z h, b and
 See the Appendix. Both histories L, B and R, B are in d , but exp L, B s ,d and exp R, B s d , B, d . ) is violated, since at d the Ž . To clarify the meaning of condition ) in Proposition 2, consider first Ž Example 2. 3 . Ž . Ž . Ž d,B 1 3 . Ž . Ž 2 3 . Ž . 3 decision maker loses information about the move of chance. In Example 6, even though at no information set the decision maker knows nature's Ž . move, ) is violated since exp L s L, d Ž . Ž 2 . Ž and exp R, B s . Ž . R ,d,B ,d . Notice that, however, if the decision maker had beliefs 1 2 consistent with the strategy assigning B to d and S to d , he would assign 1 2 1 FIG. 6. Example 6.Proof.

Abstract: We argue that in extensive decision problems extensive games with a single . player with imperfect recall care must be taken in interpreting information sets and strategies. Alternative interpretations allow for different kinds of analysis. We address the following issues: 1. randomization at information sets; 2. consistent beliefs; 3. time consistency of optimal plans; 4. the multiselves approach to decision making. We illustrate our discussion through an example that we call the ''paradox of the absentminded driver.'' Journal of Economic Literature Classification Numbers: C7, D0.