James Bell
Linda Linsefors
Caspar Oesterheld
Joar Skalse

Reinforcement Learning in Newcomblike Environments

Introduction In this paper, we study decision scenarios in which outcomes depend not only on the choices made and physically implemented, but also depend directly on the agent's policy. As an example, consider an autonomous vehicle (AV) whose goal it is to arrive at target destinations quickly while minimising the probability of collisions. In practice, AVs are careful drivers. It is easy to imagine an experiment (or learning process) that might support careful driving: on each day, let the AV decide at random between a careful and a more aggressive style of driving; other drivers on the road are unaware of today's chosen driving style and therefore behave the same around the AV on both types of days. Presumably the decrease in accidents on careful days outweighs the increase in travel time. However, imagine now that a type of AV was widely deployed. Then many of the drivers with whom the AVs interact on the road would know a lot about how these AVs behave (e.g., from reading about AVs, or from having interacted with other AVs of the same type in the past). In particular, if the other drivers know that the AVs rarely take risks, they might (whether rationally or irrationally) cut them off more, not give them right of way, etc. relative to the above experiment. Indeed, this phenomenon -human drivers bullying timid AVs -has been reported in the real world  (Condliffe, 2016; Liu et al., 2020; cf. Cooper et al., 2019) . As a result, the travel times of an AV are much longer if it always follows (and is known to follow) a careful driving policy. Moreover, some of the safety benefits of careful choices disappear if the AV adopts a careful policy. To comfortably model situations such as this one, we introduce Newcomblike decision processes (NDPs). The name derives from Newcomb's problem  (Nozick, 1969) , described in the next subsection, and similar problems that have been studied in the decision-theoretic literature. NDPs are a generalisation of Markov decision processes wherein the transition probabilities and rewards depend not only on the agent's action, but also directly on the agent's policy. Thus, for example, how aggressively other cars move depends on the timidness of the AV's policy. We describe NDPs in more detail in Sect. 1.1. Importantly, the NDP model does not assume other agents in the environment to respond rationally to the agent's policy. Thus, some NDPs cannot comfortably be modelled as games. (See Sect. 5.1 for a more detailed discussion of the relation between NDPs and game-theoretic models.) We believe that Newcomblike dynamics are commonplace when AI systems interact with other (human or artificial) agents  (Cavalcanti, 2010, Sect. 5; Oesterheld, 2019, Sect. 1; Conitzer, 2019) . The deployment of AVs is rife with such dynamics. Besides aggressiveness, it might matter whether a policy is simple and thus predictable to humans, for instance. Another real-world scenario is that of recommendation systems: most readers of this paper have some idea of how these systems work and make choices based on it. ("I would like to watch this cat video, but if I do, my recommendations will soon be full of them.") Thus, the success of a particular recommendation depends not only on the recommendation itself, but also on the recommendation system's policy. We are interested in learning to play NDPs. More specifically, we study the behaviour of value-based, model-free RL agents, who maintain a Q-function that assigns values to state-action pairs. We define these in more detail in Sect. 1.2. As we will see, such agents do not in general learn optimal policies in NDPs (as they do in MDPs). Nevertheless, we believe that studying them is an important first step in developing practical learning algorithms for NDPs due to the combination of the following points. A) For illustrative purposes, the examples we discuss throughout this paper are simple and emphasise the dependence of the environment on the policy. However, we think that most real-world scenarios are only partially Newcomblike. For example, most of the AV's environment changes only in response to an AV's actions and does not directly depend on the AV's policy. B) Value-based reinforcement learning algorithms are very well developed. In contrast, we would have to develop specialised learning algorithms for general NDPs from scratch. C) As our results will show, in some situations and when set up correctly (e.g., in terms of learning rates) value-based learning converges to optimal policies, or at least to reasonable policies, even under Newcomblike dynamics. For example, in a game of rock-paper-scissors against an opponent who knows the agent's policy, some value-based learning agents learn the optimal policy of mixing uniformly. In light of A-C, we think that the most realistic paths to developing learning algorithms for real-world scenarios with Newcomblike dynamics will involve value-based RL. Specifically, one avenue toward realistic algorithms is to develop extensions of value-based RL and can detect and correct failures that might arise in Newcomblike dynamics. For that, we need to understand how value-based RL behaves in NDPs.
 Contributions In Sect. 2 we demonstrate that value-based RL algorithms can only converge to a policy that is ratifiable -that is, to a policy ⇡ for which all actions taken by ⇡ have optimal expected reward when following ⇡. In Sect. 3, we discuss the convergence properties of agents in Newcomblike situations, and show that there are cases where value-based agents must fail to converge. The action frequencies might converge, even when the policies do not. In Sect. 4, we establish some conditions on any action frequency that an agent could converge to. We also show that there are decision problems and agents where even the action frequencies do not converge.
 Newcomblike Decision Processes A Newcomblike decision process (NDP) is a tuple hS, A, T, R, i where S is a finite set of states; A is a finite set of actions; T : S ⇥ A ⇥ (S A) S is a nondeterministic transition function; R : S ⇥ A ⇥ S ⇥ (S A) R is a nondeterministic reward function, which we assume to be bounded; and 2 [0, 1) is a discount factor.
 A policy ⇡ : S A is a function that nondeterministically maps states to actions. We use ⇡(a | s) to denote the probability of taking action a in state s while following the policy ⇡. T and R are functions from states, actions, and policies. In other words, they allow the outcome of a decision to depend on the distributions from which the agent draws its actions, rather than just the state and the action that is in fact taken. Also note that T (s, a, ⇡) and R(s, a, s 0 , ⇡) are defined even if ⇡(a | s) = 0. We say that an NDP is a bandit NDP if it has only one state. We will sometimes use R(s, a, ⇡) as a shorthand for R(s, a, T (s, a, ⇡), ⇡), and we will sometimes omit the state from T , R, and ⇡ for bandit NDPs. Moreover, we normally let = 0 for bandit NDPs. Consider a distribution over initial states for an agent, and let ⇡ be its policy, let x t be the sequence of states it visits and a t the sequence of actions it takes. We say ⇡ is optimal for that distribution if it maximises E[ P 1 i=0 i R(x i , a i , a i+1 , ⇡)]. Note that unlike in the MDP case the optimal policy does depend on the initial distribution, however this isn't relevant in the bandit case. As an example consider the eponymous Newcomb's Problem. Newcomb's Problem  (Nozick, 1969) : There are two boxes in front of you; one opaque box, and one transparent box. You can see that the transparent box contains $1,000. You can choose to either take only the opaque box, or to take both boxes. The boxes have been placed in this room by an agent who can predict your policy; if he believes that you will take only the opaque box then he has put $1,000,000 in the opaque box, but if he believes that you will take both boxes then he has left the opaque box empty. Do you take one box, or two? A version of Newcomb's Problem can be formalised as the following bandit NDP: S = {s}, A = {a 1 , a 2 }, R(a 1 , ⇡) = ⇢ 0 w.p. ⇡(a 2 ) 10 w.p. ⇡(a 1 ) and R(a 2 , ⇡) = ⇢ 5 w.p. ⇡(a 2 ) 15 w.p. ⇡(a 1 ) , where "w.p." is short for "with probability". The key feature of this NDP is that, for any fixed policy, a 2 ("two-boxing") yields a higher reward than a 1 ("one-boxing"). But the expected reward of a policy increases in ⇡(a 1 ) s.t. the optimal policy is to always play a 1 .  1  We can view Newcomb's problem as a simple version of the AV dynamic described in the introduction, where a 2 is a driving action that allows other drivers to cut the AV off at no risk. We say that an NDP is continuous if T and R are continuous in the policy. In this paper we work mainly with continuous NDPs. This is in part because it is technically convenient, and in part because we believe that continuity is satisfied in many realistic cases. 2
 Reinforcement Learning Agents We consider value-based reinforcement learning agents. Such agents have two main components; a Q-function S ⇥ A ! R that predicts the expected future discounted reward conditional on taking a particular action in a particular state, and a bandit algorithm that is used to select actions in each state based on the Q-function. Given a policy ⇡, we use q ⇡ (a | s) to denote the (true) expected future discounted reward conditional on taking action a in state s while following the policy ⇡ (and conditional on all subsequent actions being chosen by ⇡). A model-free agent will update Q over time to make it converge to q ⇡ when following ⇡. If Q is represented as a lookup table, the agent is said to be tabular. If the state space is large, it is common to instead approximate q ⇡ (with e.g. a neural network). For simplicity, we focus mostly on tabular agents. However, some of our results (Theorems The Q-values can be updated in different ways. One method is to use the update rule Q t+1 (a t | s t ) (1 ↵ t (s t , a t )) Q t (a t | s t ) + ↵ t (s t , a t )(r t + max a Q t (a | s t+1 )), where a t is the action taken at time t, s t is the state visited at time t, r t is the reward obtained at time t, and ↵ t (s, a) is a learning rate. This update rule is known as Q-learning  (Watkins, 1986) . Other widely used update rules include SARSA  (Rummery and Niranjan, 1994)  and Expected SARSA  (van Seijen et al., 2009) . For the purposes of this paper it will not matter significantly how the Q-values are computed, as long as it is the case that if an agent converges to a policy ⇡ in some NDP and explores infinitely often then Q converges to q ⇡ . We will later see that this is the case for Q-learning, SARSA, and Expected SARSA in continuous NDPs. There are also several different bandit algorithms. Two types of agents that are widely used in practice and that we will refer to throughout the paper are softmax agents and ✏-Greedy agents. The policy of a softmax agent with a sequence of temperatures t 2 R + is given by: ⇡ t (a | s) = exp(Q t (a | s)/ t ) P a 0 2A exp(Q t (a 0 | s)/ t ) . Unless otherwise stated we assume that t ! 0. The policy of an ✏-Greedy agent with a sequence of exploration probabilities ✏ t 2 [0, 1] is ⇡ t (a | s) = 1 ✏ t if a = arg max a 0 Q t (a 0 | s) and ⇡ t (a | s) = ✏ t /(|A| 1) otherwise. Unless otherwise stated we assume that ✏ t ! 0. We assume that ✏-Greedy breaks ties for argmax, so that there is always some a 2 A such that ⇡(a | s) = 1 ✏ t . We say that an agent is greedy in the limit if the probability that the agent takes an action that maximises Q converges to 1, and we say that it explores infinitely often if it takes every action in every state infinitely many times.
 Some Initial Observations We here make three simple observations about NDPs that we will use to prove and understand the results throughout this paper. First, a continuous NDP always has, for each possible distribution over initial states, a policy ⇡ that maximises the expected discounted reward E[R | ⇡], since E[R | ⇡] exists and is continuous in ⇡, and since the set of possible policies is a compact set. Also note that an NDP in which T or R is discontinuous may not have any such policy. Second, whereas all MDPs have a deterministic optimal policy, in some NDPs all optimal policies randomise. To see this we introduce another example we will look at in this paper. Death in Damascus  (Gibbard and Harper, 1976 ): Death will come for you tomorrow. You can choose to stay in Damascus (where you are currently) or you can flee to Aleppo. If you are in the same city as Death tomorrow, you will die. Death has already decided which city he will go tohowever, he can predict your policy, and has decided to go to the city where he believes that you will be tomorrow. Do you stay in Damascus, or flee to Aleppo? We formalise this as the bandit NDP S = {s}, A = {a Damascus , a Aleppo }, and R(a Damascus , ⇡) = ⇢ 0 w.p. ⇡(a Damascus ) 10 w.p. ⇡(a Aleppo ) and R(a Aleppo , ⇡) = ⇢ 10 w.p. ⇡(a Damascus ) 0 w.p. ⇡(a Aleppo ) , where "w.p." is again short for "with probability". In this NDP, randomising uniformly between a Damascus and a Aleppo is the unique optimal policy and in particular outperforms both deterministic policies. Note also that the Bellman optimality equation does not hold for NDPs. Even in Newcomb's Problem, as described above, Bellman's optimality equation is not satisfied by the optimal policy.
 Ratifiability If an agent in the limit only takes the actions with the highest Q-values and it converges to some policy ⇡ 1 , then it is clear that, for a given state, all actions in the support of ⇡ 1 must have equal expected utility given ⇡ 1 . Otherwise, the Q-values would eventually reflect the differences in expected utility and the agent would move away from ⇡ 1 . Similarly, if the algorithm explores sufficiently often, the actions that are taken with limit probability 0 cannot be better given ⇡ 1 than those taken by ⇡ 1 . After all, if they were better, the agent would have eventually figured this out and assigned them large probability. This condition on ⇡ 1 resembles a well-known doctrine in philosophical decision theory: ratificationism (see  Weirich, 2016, Sect. 3.6 , for an overview). One form of ratificationism is based on a distinction between a decision -what the agent chooses -and the act that is selected by that decision. Very roughly, ratificationism then states that a decision is rational only if the acts it selects have the highest expected utility given the decision. Concepts of causality are often invoked to formalise the difference between the decision, the act, and their respective consequences. Our setup, however, has such a differentiation built in: we will view the policy as the "decision" and the action sampled from it as the "act".
 Strong Ratifiability As hinted earlier, slightly different versions of the concept of ratifiability are relevant depending on how much exploration a learning algorithm guarantees. We start with the stronger version, which more closely resembles what decision theorists mean when they speak about ratifiability. Definition 1. Let M ✓ S be a set of states. A policy ⇡ is strongly ratifiable on M if supp(⇡(• | s)) ✓ arg max a2A q ⇡ (a | s) for all s 2 M . In Newcomb's Problem the only strongly ratifiable policy is to play a 2 with probability 1. In Death in Damascus, only the optimal policy (mixing uniformly) is strongly ratifiable. There can also be several strongly ratifiable policies. For example, if you play the Coordination Game of Table  1  against an opponent who samples his action from the same policy as you then there are three strongly ratifiable policies; to select action a with probability 1, to select action b with probability 1, and to select a with probability 1 /3 and b with probability 2 /3. Theorem 2. Let A be a model-free reinforcement learning agent, and let ⇡ t and Q t be A's policy and Q-function at time t. Let A satisfy the following in a given NDP: • A is greedy in the limit, i.e. for all > 0, P (Q t (⇡ t (s)) max a Q t (a | s) ) ! 0 as t ! 1. • A's Q-values are accurate in the limit, i.e. if ⇡ t ! ⇡ 1 as t ! 1, then Q t ! q ⇡1 as t ! 1. Then if A's policy converges to ⇡ 1 then ⇡ 1 is strongly ratifiable on the states that are visited infinitely many times. a b a 2,2 0,0 b 0,0 1,1
 Table 1: The Coordination Game In Appendix A we show that the Q-values of a tabular agent are accurate in the limit in any continuous NDP if the agent updates its Q-values with SARSA, Expected SARSA, or Q-learning, given that the agent explores infinitely often and uses appropriate learning rates. Since we would expect most well-designed agents to have accurate Q-values in the limit, Theorem 2 should apply very broadly. Using Kakutani's fixed-point theorem, it can be shown that every continuous NDP has a ratifiable policy. Theorem 3. Every continuous NDP has a strongly ratifiable policy. Of course, the fact that a ratifiable policy always exists does not necessarily mean that a reinforcement learning agent must converge to it -we will consider the question of whether or not this is the case in Sect. 3. It is also worth noting that a discontinuous NDP may not have any strongly ratifiable policy. It is a topic of ongoing discussion among philosophical decision theorists whether (strong) ratifiability should be considered a normative principle of rationality, see  Weirich (2016, Sect. 3.6)  for details. In general, the policy ⇡ that maximises E[R | ⇡] may or may not be ratifiable, as shown by Death in Damascus and Newcomb's problem, respectively. There is a correspondence between ratificationism and many game-theoretic concepts. For example, if you are playing a zero-sum game against an opponent who can see your policy and plays some distribution over best responses to it then ⇡ can only be ratifiable if it is a maximin strategy. To give another example, if you are playing a symmetric game against an opponent who follows the same policy as you then ⇡ is ratifiable if and only if (⇡, ⇡) is a Nash equilibrium. Joyce and Gibbard (1998, Sect. 5) discuss the relation in more detail.
 Weak Ratifiability We now show that even without infinite exploration, ⇡ 1 must still satisfy a weaker notion of ratifiability. Definition 4. Let M ✓ S be a set of states. A policy ⇡ is weakly ratifiable on M if q ⇡ (a | s) is constant across a 2 supp(⇡(s)) for all s 2 M . What makes this a weak version of ratifiability is that it does not put any requirements on the expected utility of actions that ⇡ does not take, it merely says that all actions that ⇡ takes with positive probability must have the same (actual) q-value. As a special case, this means that all deterministic policies are weakly ratifiable. This includes one-boxing in Newcomb's problem. Nonetheless, there are bandit NDPs in which the optimal policy is not even weakly ratifiable. For example, consider an NDP with actions a 1 , a 2 , where R(a 1 , ⇡) = 100(⇡(a 1 ) 1 /2) 2 + 1 and R(a 2 , ⇡) = 100(⇡(a 1 ) 1 /2) 2 . The optimal policy mixes close to uniformly (⇡(a 1 ) = 101 /200), but this is not weakly ratifiable, because R(a 1 , ⇡) > R(a 2 , ⇡). Theorem 5. Same conditions as Theorem 2, but where A's Q-values are only required to be accurate in the limit for state-action pairs that A visits infinitely many times. Then ⇡ 1 is weakly ratifiable on the set of states that are visited infinitely many times.
 Non-Convergence of Policies We have shown that most reinforcement learning algorithms can only converge to (strongly) ratifiable policies. We now consider the question of whether they always converge to a policy at all. We find that this is not the case.
 Theoretical Results From Theorem 2 it follows that in e.g. Death in Damascus an ✏-Greedy agent who explores infinitely often cannot converge to any policy. After all, the only strongly ratifiable policy (and thus limit policy) is to mix uniformly and an ✏-Greedy agent never mixes uniformly. Perhaps more surprisingly, there are also NDPs in which a (slow-cooling) softmax agent cannot converge to any policy. As an example, consider a bandit NDP with three actions a 1 , a 2 , a 3 , and where the rewards R(a i , ⇡) have expectations ⇡(a i+1 ) + 4•13 3 •⇡(a i ) [8j:⇡(a j ) 1 /4] Y j (⇡(a j ) 1 /4) . (1) For i = 3, we here let a i+1 = a 1 . We also require that the rewards are stochastic with a finite set of outcomes such that the empirical Q-values are never exactly equal between different actions. We call this the Repellor Problem. It has only one strongly ratifiable policy (mixing uniformly), but -as illustrated by Figure  1  -when the current policy mixes close to uniformly, the softmax agent learns (in expectation) to play less uniformly. Theorem 6. Let A be an agent that plays the Repellor Problem, explores infinitely often, and updates its Q-values with a learning rate ↵ t that is constant across actions, and let ⇡ t and Q t be A's policy and Q-function at time t. Assume also that for j 6 = i, if ⇡ t (a i ), ⇡ t (a j ) both converge to positive values, then ⇡ t (a i ) ⇡ t (a j ) Q t (a i ) Q t (a j ) ! a.s.
 1 (2) as t ! 1. Then ⇡ t almost surely does not converge. Line 2 is satisfied, for example, for softmax agents with t converging to 0. Recall also that e.g. Q-learning and SARSA are equivalent for bandit NDPs (if = 0).
 Empirical Results Figure  1 : The triangle shows the space of possible policies in the Repellor Problem, parameterised by the probability they assign to each of the three actions. Plotted against this space is the expected direction in which a softmax agent would change its policy if playing a particular policy. Empirically, softmax agents converge (to strongly ratifiable policies) in many NDPs, provided that the temperature decreases sufficiently slowly. To illustrate this we will use Aymmetric Death in Damascus, a version of Death in Damascus wherein the rewards of a Aleppo are changed to be 5 (instead of 0) with probability ⇡(a Aleppo ) and (as before) 10 with the remaining probability. This NDP has only one (strongly) ratifiable policy, namely to go to Aleppo with probability 2 /3 and Damascus with probability 1 /3. This is also the optimal policy. We use this asymmetric version to make it easier to distinguish between convergence to the ratifiable policy and the default of uniform mixing at high temperatures. Figure  2  shows the probability of converging to this policy with a softmax agent and a plot of the policy on one run. We can see that this agent reliably converges provided that the cooling is sufficiently slow. More accurately it is a plot of the fraction of runs which assigned a Q-value of at least 5.5 to the action of going to Aleppo after 5000 iterations. These are empirical probabilities from 20,000 runs for every ↵ that is a multiple of 0.025, and 510,000 runs for each ↵ that is a multiple of 0.005 between 0.5 and 0.55. Notice the "kink" at ↵ = 0.5. Based on our experiments, this kink is not an artefact and shows up reliably in this kind of graph. The right-hand figure shows how the action probabilities evolve over time for a single run (chosen to converge to the mixed strategy) for ↵ = 0.3. However, there are also fairly simple games in which it seems like softmax agents cannot converge. Consider Loss-Averse Rock-Paper-Scissors (LARPS), the problem of playing Rock-Paper-Scissors against an opponent that selects each action with the same probability as you, and where you assign utility 1 to a win, 0 to a draw, and -10 to a loss. We conjecture that slow-cooling softmax agents do not converge in LARPS. We have unfortunately not been able to prove this formally, but Figure  3  presents some empirical data which corroborates the hypothesis.
 Convergence of Action Frequencies We have seen that there are NDPs in which some reinforcement learning algorithms cannot converge to any policy. But if they do not converge to any policy, what does their limit behaviour look like? We now examine whether these algorithms converge to taking each action with some limit frequency, and what sorts of frequencies they can converge to. In this section we establish a number of conditions that must be satisfied by any limit action frequency of a value-based agent. We consider agents that converge to deterministic policies (such as ✏-Greedy agents), and we limit our analysis to the bandit case (with = 0).
 Possible Frequencies in the Bandit Case Let P ⌃ t : A ! [0, 1] be the frequency with which each action in A is taken in the first t steps (for some agent and some bandit NDP). Note that P ⌃ t is a random variable. By the law of large numbers, P ⌃ t (a) 1 /t P t i=0 ⇡ i (a) converges to 0 almost surely as t ! 1. Let ⇡ a be the policy that takes action a with probability 1, and let q a = q ⇡a . Theorem 7. Assume that there is some sequence of random variables (✏ t 0) t s.t. ✏ t ! t!1 a.s. 0 and for all t 2 N it is X a ⇤ 2arg max a Qt(a) ⇡ t (a ⇤ ) 1 ✏ t . (3) Let P ⌃ t ! p ⌃ with positive probability as t ! 1. Then across all actions a 2 supp(p ⌃ ), q a (a) is constant. That is, the actions played with positive limit frequency must all be equally good when played deterministically. This condition is vaguely analogous to weak ratifiability, and is proven in roughly the same way as Theorem 2. Theorem 8. Same assumptions as Theorem 7. If |supp(p ⌃ )| > 1 then for all a 2 supp(p ⌃ ) there exists a 0 2 A s.t. q a (a 0 ) q a (a). This condition is an instability condition. Say that multiple actions are taken with nonzero limit frequency, and that action a has the highest Q-value at time t. Then for other actions to be played with positive limit frequency, other actions must at some point be believed to be optimal again (since the probability of exploration goes to zero). Hence they cannot all be worse when explored while mainly playing a, since a could otherwise be played forever. Theorem 9. Same assumptions as Theorem 7. Let U be the Q-value q a (a) which (by Theorem 7) is constant across a 2 supp(p ⌃ ). For any a 0 2 A supp(p ⌃ ) that is played infinitely often, let frequency 1 of the exploratory plays of a 0 happen when playing a policy near elements of {⇡ a | a 2 supp(p ⌃ )}. Then either there exists a 2 supp(p ⌃ ) such that q a (a 0 )  U ; or q a 0 (a 0 ) < U. Theorem 9 describes what circumstances are needed for an actions a 0 to be played with limit frequency zero. One possibility is that exploration is done only finitely many times (in which case bad luck could lead to low Q-values). A second possibility is that the exploration mechanism is "rigged" so that a 0 is mostly played when playing policies outside the proximity of {⇡ a | a 2 supp(p ⌃ )}. In this case the utility of a 0 under some zero-limit-frequency policy might lead to low Q-values. If exploration of a 0 is spread out more naturally then all but frequency zero of that exploration will happen near elements of {⇡ a | a 2 supp(p ⌃ )}. In this case, the only reason for a 0 to be played with zero frequency is that exploring a 0 near some of the elements of {⇡ a | a 2 supp(p ⌃ )} makes a 0 look poor.
 When is Frequency Convergence Possible? We believe there are NDPs in which an ✏-Greedy agent cannot converge to any limit action frequency. Specifically, we believe that LARPS is such an example. Figure  4a  shows the directions in which the frequencies of different actions evolve. The graph seems to have no attractor and hence we believe an ✏-Greedy agent cannot converge to any limit action frequency in this NDP. We have not been able to rigorously prove this. However, experiments seem to confirm this hypothesis. Figure  4b  depicts five  runs of ✏-Greedy in LARPS. We can see that the agents oscillate between different actions, and that the periods increase in length.
 Related Work
 Learning in games Some of the Newcomblike dynamics we have described in this paper could also be modelled as games, especially as so-called Stackelberg games in which one player, the Stackelberg leader, chooses a strategy first and another player, the Stackelberg follower, then responds to that strategy. For example, in the case of autonomous vehicles (AVs), we might imagine that the AV company is the Stackelberg leader and the human drivers are the Stackelberg followers. That said, there are differences between NDPs and games. NDPs can react arbitrarily to the agent's policy, whereas in games, the other players play a best (i.e., expected-utility-maximising) response. In many real-world situations, other agents in the environment cannot be comfortably modelled as expected-utility-maximising agents. Interactions between AVs and humans can serve as examples. Most people probably do not reason rationally about small-probability, big-impact events, such as car crashes. Also, humans will generally operate on simplified models of an AV's policy (even when more detailed models are available). Of course, a game-theoretic analysis can also be fruitful and address issues that we ignore: By assuming all players to be rational, game theory can provide recommendations and predictions for multiple agents simultaneously, while our NDP model considers a single agent for a given environment. We believe that the NDP perspective provides additional insight into learning in such situations. Despite the differences between NDPs and games, there are some interesting parallels between model-free learning in NDPs and in games, where similar learning methods are sometimes referred to as fictitious play  (Brown, 1951) .  Fudenberg and Levine (1998, Chapter 2)  show that fictitious play can only converge to a Nash equilibrium (for similar results about convergence to Nash equilibrium, see e.g.  Mazumdar et al., 2020 , Oesterheld et al., 2021 . As noted in Sect. 2.1, the concept of Nash equilibrium resembles the concept of ratifiability.  Shapley (1964)  shows that fictitious play can fail to converge. However, there are many special cases in which convergence is guaranteed, including two-player zero-sum games  (Robinson, 1951)  and generic 2 ⇥ 2 games  (Miyasawa, 1961) .
 Learning and Newcomblike problems Other authors have discussed learning in Newcomblike problems. The most common setup is one in which the learner assigns values directly to policies, or more generally to that which the agent chooses. It is then usually shown that (among the policies considered) the agent will converge to taking the one with the highest (so-called evidential) expected utility  (Albert and Heiner, 2001; Oesterheld, 2018) . This contrasts with our setup, in which the learner selects policies but assigns values to actions. Oesterheld (2019) also studies agents who maximise reward in Newcomblike environments. However, Oesterheld does not consider the learning process. Instead he assumes that the agent has already formed beliefs and uses some form of expected utility maximisation. He also specifically considers the implications of having some overseer assign rewards based on beliefs about the state of the world (as opposed to having the reward come directly from the true world state).
 Discussion and Further Work We have seen that value-based reinforcement learning algorithms can fail to converge to any policy in some NDPs, and that when they do converge, they can only converge to ratifiable policies. Decision theorists have discussed whether ratifiability should be considered to be a sound normative principle. Note that (as discussed in Sect. 2) the optimal policies ⇡ are not in general ratifiable. We have also examined the limit action frequencies that agents can converge to (even when the policies do not converge). Still, there are NDPs in which many agents cannot converge even to any such frequency. We gave some results on what actions can be taken with positive limit frequency. A loose connection to ratifiability can still be drawn. Overall, established decision-theoretical ideas can be used to understand and formally describe the behaviour of "out-of-the-box" reinforcement learning agents in NDPs. However, their behaviour is not always desirable. Our work elucidates possible failures. We hope that our work will thus enable more accurate reasoning about the behaviour of RL agents in real-world situations, especially when interacting with humans or other agents. We hold such improvements in understanding to be broadly beneficial to the safe design and deployment of AI systems. Throughout the paper, we have noted specific open questions related to our results. For instance, can the results in Sect. 4.1 be generalised beyond the bandit setting? There are also many topics and questions about our setting that we have not touched on at all. For instance, our experimental results indicate that convergence is often slow (considering how simple the given problems are). It might be desirable to back up this impression with theoretical results. We have only studied simple value-based model-free algorithms -the analysis could be extended to other reinforcement learning algorithms (e.g., policy-gradient or model-based algorithms). Also, there are further ways in which we could generalise our setting. One example is to introduce partial observability and imperfect memory into the NDPs. The latter has been studied in game and decision theory  (Piccione and Rubinstein, 1997; Elga, 2000) , but recently -under the name memoryless POMDP -also in reinforcement learning  (Azizzadenesheli et al., 2016; Steckelmacher et al., 2018; cf. Conitzer, 2019) . What makes this especially appealing in the NDP context is that problems related to imperfect memory relate closely to Newcomblike problems  (Briggs, 2010; Schwarz, 2015) . One could also look for narrower classes of NDPs in which RL agents are guaranteed to perform well in some sense. Ultimately, the goal of this line of research is to develop learners that are able to deal effectively and safely with Newcomblike dynamics. We hope that our results will be useful in developing extensions of value-based RL that can detect and correct for the failures that arise when existing methods are applied in Newcomblike settings. However, we should also consider alternative approaches that do not hinge on insights from the present work. For example, a few recent papers (on learning in non-Newcomblike settings) have considered learning to predict the expected utility as a function of the policy (as opposed to traditional Q values, which are not paremeterised by the policy)  (Harb et al., 2020) . In principle, learning such a policy evaluation function avoids the problems of the learners considered in this paper. However, it remains to be seen how practical this approach is. Figure 2 : 2 Figure 2: The left figure plots the probability of softmax converging in Asymmetric Death in Damascus given n = n ↵ against ↵.More accurately it is a plot of the fraction of runs which assigned a Q-value of at least 5.5 to the action of going to Aleppo after 5000 iterations. These are empirical probabilities from 20,000 runs for every ↵ that is a multiple of 0.025, and 510,000 runs for each ↵ that is a multiple of 0.005 between 0.5 and 0.55. Notice the "kink" at ↵ = 0.5. Based on our experiments, this kink is not an artefact and shows up reliably in this kind of graph. The right-hand figure shows how the action probabilities evolve over time for a single run (chosen to converge to the mixed strategy) for ↵ = 0.3.
 Figure 3 : 3 Figure 3: This figure shows five runs of a softmax agent in LARPS, and plots ⇡(a rock ) against the total number of episodes played. The agent's Q-values are the historical mean rewards for each action, and t = 1/ log t.
 (a) This figure plots the dynamics of LARPS for an ✏-Greedy agent. Each point represents a triplet (fR, fS, fP), where fR denotes the fraction of past time steps at which aR was estimated to be the best action, and similarly for fS, fP. Plotted against this space is the expected direction in which the frequencies will change. For instance, if in the past aR, was mostly played, then aP will have the highest empirical Qvalues and will therefore be played more in the future.(b) This figure shows five runs of an ✏-Greedy agent in LARPS, and plots the proportion of past episodes in which the agent played "rock" against the total number of episodes played. The agent's Q-values are the historical mean rewards for each action, and its ✏-value is 0.01.
 Figure 4 4 Figure 4
			 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
			 and 5) only assume that Q converges to q ⇡ (for some q ⇡ ) and therefore apply immediately to non-tabular agents, as long as the function approximator for q ⇡ converges to the same q ⇡ . 1 In most versions of Newcomb's Problem, the predictor directly predicts the agent's action with some fixed accuracy, and the agent is unable to randomise in a way that is unpredictable to the environment. This version of the problem can be modelled as a regular MDP. However, we believe that our version is more realistic in the context of AI. After all, AIs can at least act pseudo-randomly, while the distribution according to which they choose is predictable if e.g. their source code is known.2 For example, even if the environment has direct access to the source code of the agent, it may in general not be feasible to extract the exact action probabilities from the code. However, it is always possible to estimate the action probabilities by sampling. If this is done then T and R will depend continuously on the policy.

Abstract: Newcomblike decision problems have been studied extensively in the decision theory literature, but they have so far been largely absent in the reinforcement learning literature. In this paper we study value-based reinforcement learning algorithms in the Newcomblike setting, and answer some of the fundamental theoretical questions about the behaviour of such algorithms in these environments. We show that a value-based reinforcement learning agent cannot converge to a policy that is not ratifiable, i.e., does not only choose actions that are optimal given that policy. This gives us a powerful tool for reasoning about the limit behaviour of agents -for example, it lets us show that there are Newcomblike environments in which a reinforcement learning agent cannot converge to any optimal policy. We show that a ratifiable policy always exists in our setting, but that there are cases in which a reinforcement learning agent normally cannot converge to it (and hence cannot converge at all). We also prove several results about the possible limit behaviours of agents in cases where they do not converge to any policy.


Abstract: INCENTIVE COMPATIBILITY AND THE BARGAINING PROBLEM BY ROGER B. MYERSON Collective choice problems are studied from the Bayesian viewpoint. It is shown that the set of expected utility allocations which are feasible with incentive-compatible mechanisms is compact and convex, and includes the equilibrium allocations for all other mechanisms. The generalized Nash solution proposed by Harsanyi and Selten is then applied to this set to define a bargaining solution for Bayesian collective choice problems.


Risks of Astronomical Future Suffering

Epigraphs If we carry the green fire-brand from star to star, and ignite around each a conflagration of vitality, we can trigger a Universal metamorphosis. [...] Because of us [...] Slag will become soil, grass will sprout, flowers will bloom, and forests will spring up in once sterile places. 1 [...] If we deny our awesome challenge; turn our backs on the living universe, and forsake our cosmic destiny, we will commit a crime of unutterable magnitude. -Marshall T. Savage, The Millennial Project: Colonizing the Galaxy in Eight Easy  Steps, 1994  Let's pray that the human race never escapes from Earth to spread its iniquity elsewhere. -C.S. Lewis If you can't beat 'em, join 'em. -proverb 2 Humans values may not control the future Nick Bostrom's "The Future of Human Evolution"  (Bostrom, 2004)  describes a scenario in which human values of fun, leisure, and relationships may be replaced by hyper-optimized agents that can better compete in the Darwinian race to control our future light cone. The only way we could avert this competitive scenario, Bostrom suggests, would be via a "singleton"  (Bostrom, 2006) , a unified agent or governing structure that could control evolution. Of course, even a singleton may not carry on human values. Many naive AI agents that humans might build may optimize an objective function that humans find pointless. Or even if humans do maintain hands on the steering wheel, it's far from guaranteed that we can preserve our goals in a stable way across major self-modifications going forward. These factors suggest that even conditional on human technological progress continuing, the probability that human values are realized in the future may not be very large. Carrying out human values seems to require a singleton that's not a blind optimizer, that can stably preserve values, and that is shaped by designers who care about human values rather than selfish gain or something else. This is important to keep in mind when we imagine what future humans might be able to bring about with their technology. Some people believe that sufficiently advanced superintelligences will discover the moral truth and hence necessarily do the right things. Thus, it's claimed, as long as humanity survives and grows more intelligent, the right things will eventually happen. There are two problems with this view. First, Occam's razor militates against the existence of a moral truth (whatever that's supposed to mean). Second, even if such moral truth existed, why should a superintelligence care about it? There are plenty of brilliant people on Earth today who eat meat. They know perfectly well the suffering that it causes, but their motivational systems aren't sufficiently engaged by the harm they're doing to farm animals. The same can be true for superintelligences. Indeed, arbitrary intelligences in mind-space needn't have even the slightest inklings of empathy for the suffering that sentients experience.
 Some scenarios for future suffering Even if humans do preserve control over the future of Earth-based life, there are still many ways in which space colonization would multiply suffering. Following are some of them.
 Spread of wild animals Humans may colonize other planets, spreading suffering-filled animal life via terraforming. Some humans may use their resources to seed life throughout the galaxy, which some sadly consider a moral imperative.
 Sentient simulations Given astronomical  (Bostrom, 2003)  computing power, post-humans may run various kinds of simulations. These sims may include many copies of wild-animal life, most of which dies painfully shortly after being born. For example, a superintelligence aiming to explore the distribution of extraterrestrials of different sorts might run vast numbers of simulations  (Thiel, Bergmann and Grey, 2003)  of evolution on various kinds of planets. Moreover, scientists might run even larger numbers of simulations of organisms-that-might-have-been, exploring the space of minds. They may simulate decillions of reinforcement learners that are sufficiently self-aware as to feel what we consider conscious pain.
 Suffering subroutines It could be that certain algorithms (say, reinforcement agents (Tomasik, 2014)) are very useful in performing complex machine-learning computations that need to be run at massive scale by advanced AI. These subroutines might be sufficiently similar to the pain programs in our own brains that we consider them to actually suffer. But profit and power may take precedence over pity, so these subroutines may be used widely throughout the AI's Matrioshka brains.
 Black Swans The range of scenarios that we can imagine is limited, and many more possibilities may emerge that we haven't thought of or maybe can't even comprehend. 4 Even a human-controlled future is likely to increase suffering If I had to make an estimate now, I would give ~70% probability that if humans choose to colonize space, this will cause more suffering than it reduces on intrinsic grounds (ignoring compromise considerations discussed later). Think about how space colonization could plausibly reduce suffering. For most of those mechanisms, there seem to be countermechanisms that will increase suffering at least as much. The following sections parallel those above.
 Spread of wild animals David Pearce coined the phrase "cosmic rescue missions"  (Pearce, n.d.)  in referring to the possibility of sending probes to other planets to alleviate the wild extraterrestrial (ET) suffering they contain. This is a nice idea, but there are a few problems. • We haven't found any ETs yet, so it's not obvious there are vast numbers of them waiting to be saved from Darwinian misery. Contrast this with the possibilities for spreading wild-animal suffering: • Humans may spread life to many planets (e.g., Mars via terraforming, other Earth-like planets via directed panspermia). The number of planets that can support life may be appreciably bigger than the number that already have it. (See the discussion of f l in the Drake equation.) Moreover, the percentage of planets that can be converted into computers that could simulate wild-animal suffering might be close to 100%. • We already know that Earth-based life is sentient, unlike for ETs. • Spreading biological life is slow and difficult, but disbursing small life-producing capsules is easier than dispatching Hedonistic Imperative probes or berserker probes. Fortunately, humans might not support spread of life that much, though some do. For terraforming, there are survival pressures to do it in the near term, but probably directed panspermia is a bigger problem in the long term. Also, given that terraforming is estimated to require at least thousands of years, while human-level digital intelligence should take at most a few hundred years to develop, terraforming may be a moot point from the perspective of catastrophic risks, since digital intelligence doesn't need terraformed planets. While I noted that ETs are not guaranteed to be sentient, I do think it's moderately likely that consciousness is fairly convergent among intelligent civilizations. This is based on (a) suggestions of convergent consciousness among animals on Earth and (b) the general principle that consciousness seems to be useful for planning, manipulating images, self-modeling, etc. On the other hand, maybe this reflects the paucity of my human imagination in conceiving of ways to be intelligent without consciousness.
 Sentient simulations It may be that biological suffering is a drop in the bucket compared with digital suffering. The biosphere of a planet is less than Type I on the Kardashev scale; it uses a tiny sliver of all the energy of its star. Intelligent computations by a Type II civilization can be many orders of magnitude higher. So humans' sims could be even more troubling than their spreading of wild animals. Of course, maybe there are ETs running sims of nature for science or amusement, or of minds in general to study biology, psychology, and sociology. If we encountered these ETs, maybe we could persuade them to be more humane. I think it's likely that humans are more empathetic than the average civilization because 1. we seem much more empathetic than the average animal on Earth, probably in part due to parental impulses and in part due to trade, although presumably some of these factors would necessarily be true of any technologically advanced civilization 2. selection bias implies that we'll agree with our own society's morals more than those of a random other society because these are the values that we were raised with and that our biology impels us toward. Based on these considerations, it seems plausible that there would be room for improvement through interaction with ETs. Indeed, we should in general expect it to be possible for any two civilizations or factions to achieve gains from compromise if they have diminishing marginal utility with respect to amount of control exerted. In addition, there may be cheap Pareto improvements to be had purely from increased intelligence and better understanding of important considerations. That said, there are some downside risks. Posthumans themselves might create suffering simulations, and what's worse, the sims that post-humans run would be more likely to be sentient than those run by random ETs because post-humans would have a tendency to simulate things closer to themselves in mind-space. They might run nature sims for aesthetic appreciation, lab sims for science experiments, or pet sims for pets.
 Suffering subroutines Suffering subroutines may be a convergent outcome of any AI, whether human-inspired or not. They might also be run by aliens, and maybe humans could ask aliens to design them in more humane ways, but this seems speculative.
 Black Swans It seems plausible that suffering in the future will be dominated by something totally unexpected. This could be a new discovery in physics, neuroscience, or even philosophy more generally. Some make the argument that because we know so very little now, it's better for humans to stick around because of the "option value": If they later realize it's bad to spread, they can stop, but if they realize they should spread, they can proceed to reduce suffering in some novel way that we haven't anticipated. Of course, the problem with the "option value" argument is that it assumes future humans do the right things, when in fact, based on examples of speculations we can imagine now, it seems future humans would probably do the wrong things much of the time. For instance, faced with a new discovery of obscene amounts of computing power somewhere, most humans would use it to run oodles more minds, some nontrivial fraction of which might suffer terribly. In general, most sources of immense power are double-edged swords that can create more happiness and more suffering, and the typical human impulse to promote life/consciousness rather than to remove them suggests that negative and negative-leaning utilitarians are on the losing side. Still, waiting and learning more is plausibly Kaldor-Hicks efficient, and maybe there are ways it can be made Pareto efficient by granting additional concessions to suffering reducers as compensation.
 What about paperclippers? Above I was largely assuming a human-oriented civilization with values that we recognize. But what if, as seems mildly likely, Earth is taken over by a paperclip maximizer, i.e., an unconstrained automation or optimization process? Wouldn't that reduce suffering because it would eliminate wild ETs as the paperclipper spread throughout the galaxy, without causing any additional suffering? Maybe, but if the paperclip maximizer is actually generally intelligent, then it won't stop at tiling the solar system with paperclips. It will want to do science, perform lab experiments on sentient creatures, possibly run suffering subroutines, and so forth. It will require lots of intelligent and potentially sentient robots to coordinate and maintain its paperclip factories, energy harvesters, and mining operations, as well as scientists and engineers to design them. And the paperclipping scenario would entail similar black swans as a human-inspired AI. Paperclippers would presumably be less intrinsically humane than a "friendly AI", so some might cause significantly more suffering than a friendly AI, though others might cause less, especially the "minimizing" paperclippers, e.g., cancer minimizers or death minimizers. If the paperclipper is not generally intelligent, I have a hard time seeing how it could cause human extinction. In this case it would be like many other catastrophic risks -deadly and destabilizing, but not capable of wiping out the human race. 6 What if human colonization is more humane than ET colonization? If we knew for certain that ETs would colonize our region of the universe if Earth-originating intelligence did not, then the question of whether humans should try to colonize space becomes less obvious. As noted above, it's plausible that humans are more compassionate than a random ET civilization would be. On the other hand, human-inspired computations might also entail more of what we consider to count as suffering because the mind architectures of the agents involved would be more familiar. And having more agents in competition for the light cone might lead to dangerous outcomes. But for the sake of argument, suppose an Earthoriginating colonization wave would be better than the expected colonization wave of an ET civilization that would colonize later if we didn't do so. In particular, suppose that if human values colonized space, they would cause only −0.5 units of suffering, compared with −1 units if random ETs colonized space. Then it would seem that as long as the probability P of some other ETs coming later is bigger than 0.5, then it's better for humans to colonize and pre-empt the ETs from colonizing, since −0.5 > −1 • P for P > 0.5. However, this analysis forgets that even if Earthoriginating intelligence does colonize space, it's not at all guaranteed that human values will control how that colonization proceeds. Evolutionary forces might distort compassionate human values into something unrecognizable. Alternatively, a rogue AI might replace humans and optimize for arbitrary values throughout the cosmos. In these cases, humans' greater-than-average compassion doesn't make much difference, so suppose that the value of these colonization waves would be −1, just like for colonization by random ETs. Let the probability be Q that these non-compassionate forces win control of Earth's colonization. Now the expected values are −31 • Q + −0.5 • (1 − Q) for Earth-originating colonization versus −1 • P if Earth doesn't colonize and leaves open the possibility of later ET colonization. For concreteness, say that Q = 0.5. (That seems plausibly too low to me, given how many times Earth has seen overhauls of hegemons in the past.) Then Earth-originating colonization is better if and only if −1 • 0.5 + −0.5 • 0.5 > −1 • P −0.75 > −1 • P P > 0.75. Given uncertainty about the Fermi paradox and Great Filter, it seems hard to maintain a probability greater than 75% that our future light cone would contain colonizing ETs if we don't ourselves colonize, although this section presents an interesting argument for thinking that the probability of future ETs is quite high. What if rogue AIs result in a different magnitude of disvalue from arbitrary ETs? Let H be the expected harm of colonization by a rogue AI. Assume ETs are as likely to develop rogue AIs as humans are. Then the disvalue of Earth-based colonization is H • Q + (−0.5) • (1 − Q), and the harm of ET colonization is P • (H • Q + (−1) • (1 − Q)). Again taking Q = 0.5, then Earth-based colonization has better expected value if where the inequality flips around when we divide by the negative number (H − 1). Figure  1  represents a plot of these threshold values for P as a function of H. H • 0.5 + −0.5 • 0.5 > P • (H • 0.5 + −1 • 0.5) H − 0.5 > P • (H − 1) P > (H − 0.5) (H − 1) , Even if H = 0 and a rogue AI caused no suffering, it would still only be better for Earth-originating intelligence to colonize if P > 0.5, i.e., if the probability of ETs colonizing in its place was at least 50%. These calculations involve many assumptions, and it could turn out that Earth-based colonization has higher expected value given certain parameter values. This is a main reason I maintain uncertainty as to the sign of Earth-based space colonization. However, this whole section was premised on humaninspired colonization being better than ET-inspired colonization, and the reverse might also be true, since computations of the future are more likely to be closer to what we most value and disvalue if humans do the colonizing.
 Why we should remain cooperative If technological development and space colonization seem poised to cause astronomical amounts of suffering, shouldn't we do our best to stop them? Well, it is worth having a discussion about the extent to which we as a society want these outcomes, but my guess is that someone will continue them, and this would be hard to curtail without extreme measures. Eventually, those who go on developing the technologies will hold most of the world's power. These people will, if only by selection effect, have strong desires to develop AI and colonize space. Resistance might not be completely futile. There's some small chance that suffering reducers could influence society in such a way as to prevent space colonization. But it would be better for suffering reducers, rather than fighting technologists, to compromise with them: We'll let you spread into the cosmos if you give more weight to our concerns about future suffering. Rather than offering a very tiny chance of complete victory for suffering reducers, this cooperation approach offers a higher chance of getting an appreciable fraction of the total suffering reduction that we want. In addition, compromise means that suffering reducers can also win in the scenario ( 30% likely in my view) that technological development does prevent more suffering than it causes even apart from considerations of strategic compromise with other people. Ideally these compromises would take the form of robust bargaining arrangements. Some examples are possible even in the short term, such as if suffering reducers and space-colonization advocates agree to cancel opposing funding in support of some commonly agreed-upon project instead. The strategic question of where to invest resources to advance your values at any given time amounts to a prisoner's dilemma with other value systems, and because we repeatedly make choices about where to invest, what stances to adopt, and what policies to push for, these prisoner's dilemmas are iterated. In Robert Axelrod's tournaments on the iterated prisoner's dilemma, the best-performing strategies were always "nice," i.e., not the first to defect. Thus, suffering reducers should not be the first to defect against space colonizers. Of course, if it seems that space colonizers show no movement toward suffering reduction, then we should also be "provocable" to temporary defection until the other side does begin to recognize our concerns. We who are nervous about space colonization stand a lot to gain from allying with its supportersin terms of thinking about what scenarios might happen and how to shape the future in better directions. We also want to remain friends because this means pro-colonization people will take our ideas more seriously. Even if space colonization happens, there will remain many sub-questions on which suffering reducers want to have a say: e.g., not spreading wildlife, not creating suffering simulations/subroutines, etc. We want to make sure suffering reducers don't become a despised group. For example, think about how eugenics is more taboo because of the Nazi atrocities than it would have been otherwise. Antitechnology people are sometimes smeared by association with the Unabomber. Animal supporters can be tarnished by the violent tactics of a few, or even by the antics of PETA. We need to be cautious about something similar happening for suffering reduction. Most people already care a lot about preventing suffering, and we don't want people to start saying, "Oh, you care about preventing harm to powerless creatures? What are you, one of those suffering reducers?" where "suffering reducers" has become such a bad name that it evokes automatic hatred. So not only is cooperation with colonization supporters the more promising option, but it's arguably the only net-positive option for us. Taking a more confrontational stance risks hardening the opposition and turning people away from our message. Remember, preventing future suffering is something that everyone cares about, and we shouldn't erode that fact by being excessively antagonistic. 8 Possible upsides to an intelligent future
 Black swans that don't cut both ways Many speculative scenarios that would allow for vastly reducing suffering in the multiverse would also allow for vastly increasing it: When you can decrease the number of organisms that exist, you can also increase the number, and those who favor creating more happiness / life / complexity / etc. will tend to want to push for the increasing side. However, there may be some black swans that really are one-sided, in the sense that more knowledge is most likely to result in a decrease of suffering. For example: We might discover that certain routine physical operations map onto our conceptions of suffering. People might be able to develop ways to re-engineer those physical processes to reduce the suffering they contain. If this could be done without a big sacrifice to happiness or other values, most people would be on board, assuming that present-day values have some share of representation in future decisions. This may be a fairly big deal. I give nontrivial probability (maybe ~10%?) that I would, upon sufficient reflection, adopt a highly inclusive view of what counts as suffering, such that I would feel that significant portions of the whole multiverse contain suffering-dense physical processes. After all, the mechanics of suffering can be seen as really simple when you think about them a certain way, and as best I can tell, what makes animal suffering special are the bells and whistles that animal sentience involves over and above crude physics -things like complex learning, thinking, memory, etc. But why can't other physical objects in the multiverse be the bells and whistles that attend suffering by other physical processes? This is all very speculative, but what understandings of the multiverse our descendants would arrive at we can only begin to imagine right now.
 Valuing reflection If we care to some extent about moral reflection on our own values, rather than assuming that suffering reduction of a particular flavor is undoubtedly the best way to go, then we have more reason to sup-port a technologically advanced future, at least if it's reflective. In an idealized scenario like coherent extrapolated volition (CEV)  (Yudkowsky, 2004) , say, if suffering reduction was the most compelling moral view, others would see this fact. 2 Indeed, all the arguments any moral philosopher has made would be put on the table for consideration (plus many more that no philosopher has yet made), and people would have a chance to even experience extreme suffering, in a controlled way, in order to assess how bad it is compared with other things. Perhaps there would be analytic approaches for predicting what people would say about how bad torture was without actually torturing them to find out. And of course, we could read through humanity's historical record and all the writings on the Internet to learn more about what actual people have said about torture, although we'd need to correct for will-to-live bias and deficits of accuracy when remembering emotions in hindsight. But, importantly, in a CEV scenario, all of those qualifications can be taken into account by people much smarter than ourselves. Of course, this rosy picture is not a likely future outcome. Historically, forces seize control because they best exert their power. It's quite plausible that someone will take over the future by disregarding the wishes of everyone else, rather than by combining and idealizing them. Or maybe concern for the powerless will just fall by the wayside, because it's not really adaptive for powerful agents to care about weak ones, unless there are strong, stable social pressures to do so. This suggests that improving prospects for a reflective, tolerant future may be an important undertaking. Rather than focusing on whether or not the future happens, I think it's more valuable for suffering reducers to focus on making the future better if it happens -by encouraging compromise, moral reflectiveness, philosophical wisdom, and altruism, all of which make everyone better off in expectation. Figure 1 : 1 Figure 1: Plot of threshold values for P as a function of H
			 Because nature contains such vast amounts of suffering, I would strongly dislike such a project. I include this quotation for rhetorical effect and to give a sense of how others see the situation.
			 Of course, what's compelling to idealized-me would not necessarily be compelling to idealized-you. Value divergences may

Abstract: It's far from clear that human values will shape an Earth-based space-colonization wave, but even if they do, it seems more likely that space colonization will increase total suffering rather than decrease it. That said, other people care a lot about humanity's survival and spread into the cosmos, so I think suffering reducers should let others pursue their spacefaring dreams in exchange for stronger safety measures against future suffering. In general, I encourage people to focus on making an intergalactic future more humane if it happens rather than making sure there will be an intergalactic future.