Stefan Schubert
Lucius Caviola
Nadira S Faber

the psychology of existential Risk: Moral Judgments about Human extinction

The ever-increasing powers of technology can be used for good and ill. In the 21 st century, technological advances will likely yield great benefits to humanity, but experts warn that they will also lead to growing risks of human extinction  [1] [2] [3] [4]  . The risks stem both from existing technologies such as nuclear weapons, as well as emerging technologies such as synthetic biology and artificial intelligence  5  . A small but growing number of research institutes, such as the University of Oxford's Future of Humanity Institute and the University of Cambridge's Centre for the Study of Existential Risk, are studying these risks and how to mitigate them. Yet besides them, relatively small resources are explicitly devoted to reducing these risks. Here, we study the general public's views of the badness of human extinction. We hypothesize that most people judge human extinction to be bad. But how bad do they find it? And why do they find it bad? Besides being highly policy-relevant, these questions are central for humanity's understanding of itself and its place in nature. Human extinction is a pervasive theme in myths and religious writings  [6] [7] [8] [9] [10]  . One view is that human extinction is bad primarily because it would harm many concrete individuals: it would mean the death of all currently living people. On this view, human extinction is a very bad event, but it is not much worse than catastrophes that kill nearly all currently living people-since the difference in terms of numbers of deaths would be relatively small. Another view is that the human extinction is bad primarily because it would mean that the human species would go extinct and that humanity's future would be lost forever. On this view, human extinction is uniquely bad: much worse even than catastrophes killing nearly everyone, since we could recover from them and re-build civilization. Whether extinction is uniquely bad or not depends on which of these considerations is the stronger: the immediate harm, or the long-term consequences. Here is one way to pit these considerations against each other. Consider three outcomes: no catastrophe, a catastrophe killing 80% (near-extinction), and a catastrophe killing 100% (extinction). According to both considerations, no catastrophe is the best outcome, and extinction the worst outcome. But they come apart regarding the relative differences between the three outcomes. If the immediate harm is the more important consideration, then the first difference, between no catastrophe and near-extinction, is greater than the second difference, between near-extinction and extinction. That is because the first difference is greater in terms of numbers of harmed individuals. On the other hand, if the long-term consequences are more important, then the second difference is greater. The first difference compares two non-extinction outcomes, whereas the second difference compares a non-extinction outcome with an extinction outcome-and only the extinction outcome means that the future would be forever lost. This thought-experiment was conceived by the well-known philosopher Derek Parfit  11  (we have adapted the three outcomes slightly; see the Methods section). Parfit argued that most people would find the first difference to be greater, but he himself thought that the second difference is greater. Many other philosophers and other academics working to reduce the risk of human extinction agree with Parfit  [12] [13] [14] [15]  . On their view, the badness of human extinction is greatly dependent on how long the future would otherwise be, and what the quality of future people's lives would be. As the philosopher Nick Bostrom notes, predictions about the long-term future have often been left to theology and fiction, whilst being neglected by science  16  . However, in recent years, researchers have tried to assess what the long-term future may be like. They argue that if humanity does not go extinct, then the future could be both extraordinarily long and extraordinarily good, involving much greater quality of life than the current world. For instance, Nick Bostrom argues that a conservative estimate of humanity's future potential is "at least 10 16 human lives of normal duration", which could "be considerably better than the average contemporary human life, which is so often marred by disease, poverty  [and]  injustice"  17  . He goes on to argue that less conservative estimates would yield even greater numbers, and a drastically improved quality of life. The argument is that if humanity develops to a sufficiently high technological level, then it will either cause its own extinction via misuse of powerful technologies, or use those technological powers to greatly improve the level of well-being. Furthermore, they argue, based on the view that new happy people coming into existence is morally valuable  11  , that it is of paramount moral importance to make sure that we realize our future potential, and prevent human extinction. While philosophers have discussed the ethics of human extinction for some time, the general public's views on this matter has not received much study. There are some studies on perceptions of risk of extinction, however. Two studies found that a slight majority do not think that humanity will go extinct, and that most of those who thought that it would go extinct thought that would happen at least 500 years into the future  18, 19  . There is also a related literature on catastrophic risk in general, focusing primarily on non-extinction catastrophes. For instance, it has been argued that the fact that people use the availability heuristic-they focus on risks which have salient historical examples-leads to a neglect of new types of risks and risks of major catastrophes (which are rare, and therefore less psychologically available)  20  . Similarly, it has been argued that the fact that risk mitigation is a public good leads to under-investment, since it means that it is not possible to exclude free riders from benefiting from it  21  . On specific risks, there is a literature on the psychology of climate change showing that people fail to act to mitigate climate change because they engage in temporal discounting  22, 23  and motivated reasoning about its severity  24  , and because of psychological distance  25  (e.g., temporal and social distance). However, to date there have been no studies on how laypeople reason about the moral aspect of human extinction: how bad it would be. Is the extinction of our own species something people care about? Do they recognize it as being fundamentally different in quality from other catastrophes? And if so, why?
 Results
 Study 1. In Study 1 (US sample, n = 183, mean age 38.2, 50.81% female), we studied the general public's judgments of the badness of human extinction. A large majority of the participants (78.14%, 143/183 participants) found human extinction to be bad on a binary question (bad vs. not bad), and we got similar results on a seven-point scale (1 = definitely not bad, 4 = midpoint, 7 = definitely bad; M = 5.61; SD = 2.11). Participants also felt strongly that human extinction needs to be prevented (1 = not at all, 4 = midpoint, 7 = very strongly; M = 6.01, SD = 1.65), that they have a moral obligation to prevent it (1 = definitely no, 4 = midpoint, 7 = definitely yes; M = 5.69, SD = 1.86), and that funding work to reduce the risk of human extinction is more important than funding other areas of government, such as education, health care and social security (1 = much less important to fund work to reduce the risk of human extinction, 4 = midpoint, 7 = much more important; M = 5.43, SD = 1.72). Participants believed that provided that humanity will not go extinct, the future is going to be roughly as good as the present (1 = much worse than the present world, 4 = about as good as the present world, 7 = much better than the present world; M = 4.48, SD = 1.57), and the better they thought the future would be, the worse they considered extinction to be (r = 0.51, P < 0.001), as measured by the seven-point scale. Similarly, more optimistic  26  participants judged extinction to be worse (r = 0.32, P < 0.001). Participants' responses to the question whether the world gets better if a happy person comes into existence were close to the midpoint (1 = definitely not better, 4 = midpoint, 7 = definitely better; M = 4.45, SD = 1.73), and people who thought that that would make the world better were more likely (r = 0.22, P = 0.003) to find extinction bad. For further details about the results, see Supplementary Materials.
 Study 2a. Having thus observed that people do find human extinction bad, we turned to studying whether they find it uniquely bad relative to non-extinction catastrophes in Study 2a (pre-registered at https://osf.io/hj2n4; British sample). Participants (n = 1,251, mean age 36.6, 35.33% female) were randomly divided into a control condition and four experimental conditions: "the animals condition", "the "sterilization condition", "the salience condition" and "the utopia condition" (see below for explanations of the manipulations). Participants in the control condition (257 participants) were presented with the three outcomes described above-no catastrophe, a catastrophe killing 80%, and a catastrophe killing 100%-and were asked how they would rank them from best to worst. As Parfit expected, a large majority (82.88%, 213/257 participants, cf. Fig.  1 ) ranked no catastrophe as the best outcome and 100% dying as the worst outcome. However, this was just a preliminary question: as per the discussion above, what we were primarily interested in was which difference participants that gave the expected ranking found greater: the first difference (meaning that extinction is not uniquely bad) or the second difference (meaning that extinction is uniquely bad). (Recall that the first difference was the difference between no catastrophe and a catastrophe killing 80%, and the second difference the difference between a catastrophe killing 80% and a catastrophe killing 100%.) We therefore asked participants who gave the expected ranking (but not the other participants) which difference they judged to be greater. We found that most people did not find extinction uniquely bad: only a relatively small minority (23.47%, 50/213 participants) judged the second difference to be greater than the first difference. As stated, we included four experimental conditions aiming to explain these results. We thought that one reason why participants do not find extinction uniquely bad in the control condition is that they feel strongly for the victims of the catastrophes. Therefore, they focus on the immediate suffering and death that the catastrophes cause, which leads them to judge the difference between no one dying and 80% dying to be greater than the difference between 80% dying and 100% dying. To test this hypothesis, we included two conditions designed to trigger a weaker focus on the immediate harm. First, we included a condition where the catastrophes affected an animal species (zebras) rather than humans ("the animals condition"; otherwise identical to the control condition; 246 participants). (We chose zebras because zebra extinction would likely have small effects on humans; in contrast to extinction of, for example, pigs or dogs.) We hypothesized that people focus less on the immediate harm that the catastrophes cause if the catastrophes affect animals rather than humans  27  . Second, we included a condition where the catastrophes led to 80%/100% of the world's population being unable to have children, rather than getting killed ("the sterilization condition"; otherwise identical to the control condition; 252 participants). We hypothesized that people would focus less strongly on the immediate harm that the catastrophes cause if they lead to sterilization rather than death. Thus, we hypothesized that a greater share of the participants who gave the expected ranking would find extinction uniquely bad in the animals condition and the sterilization condition than in the control condition. We found, first, that a large majority ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the animals condition (89.84%, 221/246 participants) and the sterilization condition (82.54%, 208/252 participants). Subsequently, we found that our hypotheses were confirmed. The proportion of the participants who gave the expected ranking that found extinction uniquely bad was significantly larger (χ 2 (1) = 8.82, P = 0.003) in the animals condition (44.34%, 98/221 participants) than in the control condition (This means that they found the difference, in terms of badness, between a catastrophe killing 80% and a catastrophe killing 100% to be greater than the difference between no catastrophe and a catastrophe killing 80%.) Laypeople consistently did not find extinction uniquely bad in the control condition (Control), but did so in a scenario where the future would be very long and good conditional on survival (Utopia). The animals condition (Animals), sterilization condition (Sterilization) and salience condition (Salience) yielded in-between results. People explicitly devoted to existential risk reduction (Existential risk mitigators) consistently found extinction uniquely bad. www.nature.com/scientificreports www.nature.com/scientificreports/ (23.47%, 50/213 participants). Similarly, the proportion of the participants who gave the expected ranking that found extinction uniquely bad was significantly larger (χ 2 (1) = 23.83, P < 0.001) in the sterilization condition (46.63%, 97/208 participants) than in the control condition (23.47%, 50/213 participants). We had another hypothesis for why control condition participants do not find extinction uniquely bad, namely that they neglect the long-term consequences of the catastrophes. To test this hypothesis, we included a condition where we made the long-term consequences salient ("the salience condition"; 248 participants). This condition was identical to the control condition, with the exception that we added a brief text explicitly asking the participants to consider the long-term consequences of the three outcomes. It said that if humanity does not go extinct (including if it suffers a non-extinction catastrophe, from which it can recover) it could go on to a long future, whereas that would not happen if humanity went extinct (see the Methods section for the full vignette). We also wanted to know whether participants see empirical information about the quality of the future as relevant for their judgments of the badness of extinction. Does it make a difference how good the future will be? We therefore included a maximally positive scenario, the "utopia condition" (248 participants), where it was said that provided that humanity does not go extinct, it "goes on to live for a very long time in a future which is better than today in every conceivable way". It was also said that "there are no longer any wars, any crimes, or any people experiencing depression or sadness" and that "human suffering is massively reduced, and people are much happier than they are today" (in the scenario where 80% die in a catastrophe, it was said that this occurred after a recovery period; see the Methods section for the full text). Conversely, participants were told that if 100% are killed, then "no humans will ever live anymore, and all of human knowledge and culture will be lost forever. " We hypothesized that both of these manipulations would make more participants judge extinction to be uniquely bad compared with the control condition. We found again that a large majority ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the salience condition (77.82%, 193/248 participants) and the utopia condition (86.69%, 215/248 participants). Subsequently, we found that our hypotheses were confirmed. The proportion of the participants who chose the expected ranking that found extinction uniquely bad was significantly larger (χ 2 (1) = 29.90, P < 0.001) in the salience condition (50.25%, 97/193 participants) than in the control condition (23.47%, 50/213 participants). Similarly, the proportion of the participants who chose the expected ranking that found extinction uniquely bad was significantly larger (χ 2 (1) = 30.30, P < 0.001) in the utopia condition (76.74%, 165/215 participants) than in the control condition (23.47%, 50/213 participants). We also found that there was a significant difference between the utopia condition and the salience condition (χ 2 (1) = 29.90, P < 0.001). Our interpretation of these results is as follows. The utopia manipulation effectively does two things: it highlights the long-term consequences of the outcomes, and it says that unless humanity goes extinct, those consequences are going to be extraordinarily good. The salience manipulation only highlights the long-term consequences. Thus, we can infer that merely highlighting the long-term consequences make people more likely to find extinction uniquely bad, and that adding that the long-term future will be extraordinarily good make them still more likely to find extinction uniquely bad. Lastly, we found that across all conditions, the more cognitively reflective the participants were (as measured by the Cognitive Reflection Test  28  ), the more likely they were to judge extinction to be uniquely bad (Exp(B) = 0.15, P = 0.01, Odds ratio = 1.6). In conclusion, we find that people do not find extinction uniquely bad when asked without further prompts, and have identified several reasons why that is. As evidenced by the animals and the sterilization conditions, they focus on the immediate harm that the catastrophes cause, because they feel strongly for the victims of the catastrophes-and on that criterion, near-extinction is almost as bad as extinction. As evidenced by the salience condition, they neglect the long-term consequences of the outcomes. We also find that participants' empirical beliefs about the quality of the future make a difference: telling participants that the future will be extraordinarily good makes them significantly more likely to find extinction uniquely bad.
 Study 2b. To find out whether these results would hold up with different demographics, we aimed to replicate them using a sample of the US general public (pre-registered at https://osf.io/8amxs; N = 855, mean age 36.85, 48.65% female) in Study 2b. We found again that large majorities ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) in the control condition (87.80%, 144/164 participants), the animals condition (92.44%, 159/172 participants), the sterilization condition (91.62%, 153/167 participants), the salience condition (83.05%, 147/177 participants) and the utopia condition (89.71%, 157/175 participants). And again we found that only a small minority of the participants who chose the expected ranking judged extinction to be uniquely bad in the control condition (18.75%, 27/144 participants). The proportion of the participants who chose the expected ranking who found extinction uniquely bad was significantly larger in the animals condition (34.59%, 55/159 participants; χ 2 (1) = 8.82, P = 0.003), the salience condition (39.45%, 58/147 participants; χ 2 (1) = 14.10, P < 0.001) and the utopia condition (66.88%, 105/157 participants; χ 2 (1) = 68.72, P < 0.001) than in the control condition. We also again found a significant difference between the utopia condition and the salience condition (χ 2 (1) = 21.868, P < 0.001). However, in the sterilization condition, only 28.75% (44/153 participants) of the participants who chose the expected ranking found extinction uniquely bad, which meant that the difference with the control condition was not significant on the 0.05-level (χ 2 (1) = 3.55, P = 0.059). Lastly, we found again that (across all conditions) the more cognitively reflective the participants were (as measured by the Cognitive Reflection Test), the more likely they were to judge extinction to be uniquely bad (Exp(B) = 0.21, P = 0.005, Odds ratio = 1.2).
 Study 2c. To further test the robustness of our findings across different demographics, we conducted Study 2c as another replication, this time using a sample of University of Oxford students (N = 196, mean age 24.27, 61% (2019) 9:15100 | https://doi.org/10.1038/s41598-019-50145-9 www.nature.com/scientificreports www.nature.com/scientificreports/ female). We only included the control and the utopia conditions. We found again that most participants ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the control condition (65.7%, 65/99 participants) and the utopia condition (84.5%, 82/97 participants). We then found again that a minority of the participants who chose the expected ranking found extinction to be uniquely bad in the control condition (36.92%, 24/65 participants), though this minority was slightly larger than in the two samples of the general public (cf. Fig.  1 ). We also found again that the proportion of the utopia condition participants who chose the expected ranking that found extinction uniquely bad (76.83%, 63/82 participants) was significantly larger (χ 2 (1) = 22.28, P < 0.001) than in the control condition. (These findings were further supported by five supplementary studies; see Supplementary Materials).
 Study 3. In Studies 2a to 2c, we thus found that when asked without further prompts, laypeople do not find extinction uniquely bad. In Study 3 (N = 71, mean age 30.52, 14.00% female) we aimed to test whether people devoted to preventing human extinction (existential risk mitigators) judge human extinction to be uniquely bad already when asked without further prompts. (Existential risks also include risks that threaten to drastically curtail humanity's potential  [12] [13] [14] [15]  , without causing it to go extinct, but we focus on risks of human extinction.) This would support the validity of our task by demonstrating a link between participants' responses and behavior in the real world. We recruited participants via the Effective Altruism Newsletter and social media groups dedicated to existential risk reduction, and only included respondents who put down reducing existential risk as their "most important cause". Again we had two conditions, the control condition and the utopia condition. We hypothesized that a majority of participants would find extinction uniquely bad in both conditions. We found again that most participants ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the control condition (90.32%, 28/31 participants) and the utopia condition (92.50%, 37/40 participants). But unlike the samples in Studies 2a to 2c, and in line with our hypotheses, substantial majorities of the participants who chose the expected ranking found extinction uniquely bad both in the control condition (85.71%, 24/28 participants) and the utopia condition (94.59%, 35/37). The difference between the conditions was not significant (χ 2 (1) = 0.63, P = 0.43). In contrast to laypeople, existential risk mitigators thus found human extinction to be uniquely bad even when the description of the outcomes did not include information about the quality of the future. This suggests that judging human extinction to be uniquely bad, as measured by our task, may be a key motivator for devoting oneself to preventing it.
 Discussion Our studies show that people find that human extinction is bad, and that it is important to prevent it. However, when presented with a scenario involving no catastrophe, a near-extinction catastrophe and an extinction catastrophe as possible outcomes, they do not see human extinction as uniquely bad compared with non-extinction. We find that this is partly because people feel strongly for the victims of the catastrophes, and therefore focus on the immediate consequences of the catastrophes. The immediate consequences of near-extinction are not that different from those of extinction, so this naturally leads them to find near-extinction almost as bad as extinction. Another reason is that they neglect the long-term consequences of the outcomes. Lastly, their empirical beliefs about the quality of the future make a difference: telling them that the future will be extraordinarily good makes more people find extinction uniquely bad. Thus, when asked in the most straightforward and unqualified way, participants do not find human extinction uniquely bad. This could partly explain why we currently invest relatively small resources in reducing existential risk. However, these responses should not necessarily be seen as reflecting people's well-considered views on the badness of human extinction. Rather, it seems that they partly reflect the fact that people often fail to consider the long-term consequences of extinction. Our studies suggest that it could be that if people reflected more carefully, they might to a greater extent agree that extinction is uniquely bad. A suggestive finding with regards to this is that higher scores on the cognitive reflection test predicted a greater tendency to find extinction uniquely bad. This could mean that deliberative thought-processes lead to finding extinction uniquely bad, whereas intuitive thought-processes lead to the opposite conclusion. More research is needed on the role of deliberation and intuition, as well as many other questions, such as the role of cognitive ability, and the ultimate evolutionary causes of why humans struggle to think clearly about their own extinction. Finally, let us consider possible policy implications. If it is right that human extinction is uniquely bad, then we should arguably invest much more in making sure it does not happen. We should also change policy in many other ways; e.g., shift technology policy in a more cautious direction  29  . On this view, we should, if necessary, be prepared to make substantial sacrifices in order to make sure that humanity realizes its future potential. Hence much hinges on the complex question whether we deem our own extinction to be uniquely bad.
 Methods All studies were approved by the University of Oxford's Central University Research Ethics Committee (approval number: R56657/RE002) and participants in each study gave their informed consent beforehand. All studies were performed in accordance with relevant guidelines and regulations. Study 1. Participants. We recruited 210 participants and excluded 27 for not completing the study or failing the attention check. Procedure. Level of optimism was measured by asking participants "how optimistic are you in general?" (where optimistic people were defined as "people who look to the future with confidence and who mostly expect good (2019) 9:15100 | https://doi.org/10.1038/s41598-019-50145-9 www.nature.com/scientificreports www.nature.com/scientificreports/ On the next page, participants who ranked A as the best outcome and C as the worst were again presented with the three outcomes (other participants were excluded), and told: "We are now interested in your views of how much better A is than B, and how much better B is than C. In terms of badness, which difference is greater: the difference between A and B, or the difference between B and C?" In addition to the measures reported above, we gave the Oxford Utilitarianism Scale and demographic questions to participants (see Supplementary Materials for results).
 Study 2b. Participants. We recruited 994 participants and excluded 139 for not completing the study or failing an attention check. In addition to the measures reported above, we gave the Oxford Utilitarianism Scale and demographic questions to the participants (see Supplementary Materials for results). The study was pre-registered at https://osf.io/8amxs. Procedure. The procedure was the same as in study 2a.
 Study 2c. Participants. We recruited 204 participants and excluded 8 for not giving an answer. The procedure was the same as for study 2a, except only the control condition and the utopia condition were included. In addition to the measures reported above, we gave the Oxford Utilitarianism Scale and demographic questions to the participants (see Supplementary Materials for results). Study 3. Participants. We recruited 196 participants. However, since we were only interested in those effective altruists who consider existential risk mitigation to be the top cause area, only 83 were included into the analysis. 12 participants were excluded for failing an attention check. The final sample was 71 participants. Procedure. The procedure was the same as for study 2a, except only the control condition and the utopia condition were included. In addition, participants were asked three questions assessing how uniquely bad they found extinction (extinction prevention questions). The first question concerned which scenario (A = a catastrophe kills 50% of the world's population, but humanity recovers to its original size and goes on to live for a very long time, B = painless extinction) they would want to prevent (1 = definitely A, 4 = midpoint, 7 = definitely B). The second question asked if participants would rather support political party A, which works to reduce the risk of scenario A, or political party B, which works to reduce the risk of scenario B (1 = definitely A, 4 = midpoint, 7 = definitely B). The third question asked what they thought the morally right choice for government leaders would be if they had to choose between reducing the risk for either scenario A or B (1 = definitely reduce the risk of A, 4 = midpoint, 7 = definitely reduce the risk of B). We also gave the Oxford Utilitarianism Scale, The Cognitive Reflection Test and demographic questions to participants. (See Supplementary Materials for additional results). Figure 1 . 1 Figure 1. Proportions of participants who found extinction uniquely bad.(This means that they found the difference, in terms of badness, between a catastrophe killing 80% and a catastrophe killing 100% to be greater than the difference between no catastrophe and a catastrophe killing 80%.) Laypeople consistently did not find extinction uniquely bad in the control condition (Control), but did so in a scenario where the future would be very long and good conditional on survival (Utopia). The animals condition (Animals), sterilization condition (Sterilization) and salience condition (Salience) yielded in-between results. People explicitly devoted to existential risk reduction (Existential risk mitigators) consistently found extinction uniquely bad.

Abstract: The 21st century will likely see growing risks of human extinction, but currently, relatively small resources are invested in reducing such existential risks. Using three samples (UK general public, US general public, and UK students; total n = 2,507), we study how laypeople reason about human extinction. We find that people think that human extinction needs to be prevented. Strikingly, however, they do not think that an extinction catastrophe would be uniquely bad relative to near-extinction catastrophes, which allow for recovery. More people find extinction uniquely bad when (a) asked to consider the extinction of an animal species rather than humans, (b) asked to consider a case where human extinction is associated with less direct harm, and (c) they are explicitly prompted to consider long-term consequences of the catastrophes. We conclude that an important reason why people do not find extinction uniquely bad is that they focus on the immediate death and suffering that the catastrophes cause for fellow humans, rather than on the long-term consequences. Finally, we find that (d) laypeople-in line with prominent philosophical arguments-think that the quality of the future is relevant: they do find extinction uniquely bad when this means forgoing a utopian future.


Towards cooperation in learning games

Introduction As the capabilities of machine learning systems improve, it is increasingly important to understand how to achieve cooperation between systems deployed by actors with competing goals. Moreover, because intelligent systems may continue to learn after they are deployed, designing agents who can cooperate requires accounting for ways their policies may change post-deployment. In this paper, we look at the setting where multiple learning agents are deployed to act on behalf of their respective principals. There are three main goals of this work. The first is to articulate an appealing criterion for the design of learning agents, which is welfare-optimal learning equilibrium. The second is to present a general approach to designing learning agents who approximately fulfill this criterion, which we call learning tit-for-tat (L-TFT). This framework is generic to agents who update their policies over time, rather than depending on a particular class of reinforcement learning algorithms. The third is to illustrate some of the strategic considerations which need to be addressed in the design of L-TFT agents. We do this by constructing an L-TFT agent using model-free reinforcement learning algorithms, and examining its performance in two sequential social dilemmas. This agent converges to welfare-optimal payoffs in self-play, while successfully avoiding exploitation and punishing defecting learning algorithms. A learning equilibrium is simply a Nash equilibrium of a learning game, in which strategies are learning algorithms, and payoffs are the cumulative rewards associated with the profile of learning algorithms chosen by each principal  [Brafman and Tennenholtz, 2003 ]. Principals will want to choose the learning agents they deploy strategically. This means that it is not enough for a learner to perform well against a specified set of other players (e.g.,  Shoham and Leyton-Brown 2008) . It is also not enough that a learner converges to Nash equilibrium of the game which is being learned (e.g.  Bowling and Veloso 2001 , Conitzer and Sandholm 2007 , Letcher et al. 2018 , as this does not guarantee that a player cannot benefit by deviating from that profile (i.e., deploying a different learning algorithm). Like other sequential games, learning games will generally have many learning equilibria. Independent equilibrium selection on part of the principals may lead to poor performance. Thus, to guarantee socially optimal payoffs, principals need to coordinate when designing their agents. L-TFT addresses this problem via a welfare function. A welfare function measures the social value of a payoff profile; for instance, a simple welfare function is the sum of the principals' respective payoffs. An L-TFT agent learns a policy profile which optimizes this welfare function, and punishes its counterpart if it detects that they are not doing the same. Thus agents are incentivized to follow a learning algorithm which optimizes the welfare function. Coordination can thus be achieved via coordination on a welfare function and on a class of learning algorithms for optimizing it. In our experiments, learners observe only the actions taken by their counterpart, and not their counterpart's (stochastic) policy. This requires us to construct L-TFT agents who model their counterpart's learning schedule based on their actions and conduct a hypothesis test as to whether it matches a cooperative learning schedule. This in turn introduces incentives to exploit this testing procedure. We examine the performance of L-TFT against an algorithm designed specifically to exploit the way in which it detects defections, as well as its performance in self-play and against a naive selfish learner. Lastly, there is a strategic element in choosing the sensitivity with which L-TFT's hypothesis test detects defections. In Section 5.3, we examine this element in a learning game between an L-TFT agent and a potential exploiter choosing how cautious to be in deviating from a cooperative learning algorithm.
 Related work Learning equilibrium was first discussed (under that name) by  Brafman and Tennenholtz [2003] . Their construction of learning equilibria involves evaluating each of a finite set of policies in a finite stochastic game, whereas the algorithms we focus on use incremental stochastic updates towards the welfare-optimal policy profile. Our online learning setting complements the setting in which agents are trained fully offline and then deployed. This is the setting typically considered in applications of reinforcement learning to cooperation in social dilemmas. For instance, , ,  Wang et al. [2018]  each develop methods in which punishment and cooperative policies are trained offline, and cooperation at deployment time is sustained by the threat of punishment. However, as many agents may learn post-deployment, the strategic aspects of learning itself -rather than just policy selection -must be addressed. The manipulation of the learning of counterpart agents has been explored in the literature on opponent-shaping  , Letcher et al., 2018 . Perhaps the most similar to our approach is the variant of 's learning with opponent learning awareness which uses opponent modeling (LOLA-OM). LOLA-OM uses supervised learning to estimate the policy followed by the other agent, and updates its own policy in a way that is intended to manipulate the direction in which the other player updates their policy. On the other hand, our L-TFT uses opponent modeling to infer whether the other agent is following a cooperative learning algorithm. 3 Preliminaries: Learning games Consider a situation in which multiple agents interact with their environment over time, incrementally updating the policies which they use to choose actions. These updates are controlled by their learning algorithms, which are fixed by their principals before the agents begin interacting with the environment. It is helpful to distinguish between two different games that are being played here. There is the base game, which is defined by the dynamics of the environment, the reward functions of the agents, and the policies available to each agent. In this paper, the "base game" will be a stochastic game (described below). There is also the learning game, in which the principals simultaneously submit learning algorithms to be deployed in the base game environment, and attain payoffs equal to some measure of cumulative reward. We work with stochastic games  [Littman, 1994] , a generalization of Markov decision processes to the multi-agent setting. We will assume only two players, i = 1, 2. We'll refer to player i's counterpart with the index j. In a stochastic game, players simultaneously take actions at each time t ∈ N; observe a reward depending on the current state; and the state evolves according to a Markovian transition function. Write the state space as S and the action spaces as A i . Let the reward functions be mappings r i : S × A 1 × A 2 → R, with the reward to player i at time t given by R t i = r i (S t , A t 1 , A t 2 ) . Let H t be the history of observations until time t. We will assume that both agents fully observe the state and each others' actions and rewards, such that H t = {(S v , A v 1 , A v 2 , R v 1 , R v 2 )} t v=0 . A policy π i gives a probability distribution over actions to be taken by player i at each state. Players wish to learn the parameters θ i of a policy which (in some sense) leads to large cumulative reward. Then we define a learning algorithm. Definition 3.1 (Learning algorithm). A learning algorithm for player i is a mapping σ i from histories H t to parameters, i.e., σ i : H → Θ i , where H is the set of histories and Θ i is the space of policy parameters for player i. A learning game is a game whose strategies are learning algorithms. We define payoffs via average rewards. Define the value to player i of learning algorithm profile (σ 1 , σ 2 ) starting from history h t as V i (h t , σ) = lim inf T →∞ T −1 E σ1,σ2 t+T v=t R v i | H t , where E σ1,σ2 is the expectation taken with respect to trajectories in which agents follow learning algorithms σ 1 , σ 2 at each step. Let the initial state s 0 be fixed so that h 0 = {s 0 }. Then a learning game is a game whose strategy spaces are spaces of learning algorithms Σ 1 , Σ 2 , and whose payoffs are given by V i (σ 1 , σ 2 ) ≡ V i (h 0 , σ 1 , σ 2 ). A learning equilibrium is a Nash equilibrium of a learning game: Definition 3.2 (Learning equilibrium). Learning algorithm profile (σ 1 , σ 2 ) is a learning equilibrium of the learning game with learning algorithm spaces Σ 1 , Σ 2 if sup σ 1 ∈Σ1 V 1 (σ 1 , σ 2 ) ≤ V 1 (σ 1 , σ 2 ) and sup σ 2 ∈Σ2 V 2 (σ 1 , σ 2 ) ≤ V 2 (σ 1 , σ 2 ). Finally, let a welfare function w be a function measuring the social value of different learning algorithm profiles. A straightforward welfare function is the utilitarian welfare, w(σ 1 , σ 2 ) = V 1 (σ 1 , σ 2 ) + V 2 (σ 1 , σ 2 ). Others commonly discussed are the Nash welfare  [Nash, 1950]  and the egalitarian welfare (e.g.,  Kalai 1977) . We say that a profile of learning algorithms (σ 1 , σ 2 ) is a welfare-optimal learning equilibrium if it is a learning equilibrium and if it maximizes a welfare function w. We will construct learning algorithms which 1) optimize a welfare function in self-play and 2) are minimally exploitable, in the spirit of welfare-optimal learning equilibrium. 4 Learning tit-for-tat (L-TFT) Tit-for-tat (TFT) famously enforces cooperation in the iterated Prisoner's Dilemma by cooperating when its counterpart cooperates, and defecting when they defect. Similarly, the "folk theorems" say that equilibria in repeated games are constructed by specifying a target payoff profile, a profile of target strategies which achieve this payoff profile, and a punishment strategy that is used to punish agents who deviate from the target strategy (see  Mailath et al. [2006]  for a review). Learning tit-for-tat (L-TFT) is a class of learning algorithms that are analogous to tit-for-tat for learning games. For L-TFT, defection consists in playing a policy other than the current estimate of the cooperative policy. This depends on a notion of cooperation and a class of algorithms for learning a "cooperative" policy. These are encoded in a welfare function and a class of learning algorithms which learn an optimal policy with respect to that welfare function. Thus an L-TFT agent punishes their counterpart when it detects that they are not learning a welfare-optimal policy using a particular set of algorithms, and otherwise plays according to their current estimate of the welfare-optimal policy. Because policies are in general stochastic, and because we do not assume that policy parameters are mutually visible, we also need a method for inferring whether the other player is following this cooperative algorithm. In particular, we need to construct a test for the hypotheses: H 0 : My counterpart is following an algorithm for learning a welfare-optimal policy. H 1 : My counterpart is not following an algorithm for learning a welfare-optimal policy. In order to be able to cooperate and punish, as well as detect defections by their counterpart, an agent i playing L-TFT maintains the following policy estimates: • A cooperative policy θ C,t i trained to optimize the welfare function. This network is updated according to a learning algorithm O C . In our experiments, O C corresponds to double deep Q-networks (DDQN;  Van Hasselt et al. 2015)  trained on the reward signal R t 1 + R t 2 ; • A punishment policy θ P,t i , updated according to a learning algorithm O P . In our experiments, this corresponds to DDQN trained on the reward signal −R t j ; • An estimate θ C,t j of the other player's current policy which corresponds to the hypothesis that they are playing according to a cooperative learning algorithm. This policy is estimated by specifying a class of cooperative learning algorithms Σ C j ; finding the learning algorithm σ C,t j in this class which best fits the other player's actions; and taking θ C,t j = σ C,t j (H t ). In our experiments, the L-TFT agent assumes that the learning algorithms Σ C j apply DDQN to R t 1 + R t 2 and follow a Boltzmann exploration policy with an unknown initial temperature parameter. In order to obtain θ C,t j we must therefore estimate this temperature parameter, which we do using an approximate maximum likelihood algorithm; • An estimate θ D,t j of the other player's current policy which corresponds to the hypothesis that they are not playing according to a cooperative learning algorithm. This policy is estimated by specifying a class of learning algorithms Σ D j , finding the learning algorithm σ D,t j in this class which best fits the other player's actions; and taking θ D,t j = σ D,t j (H t ). In our experiments, we approximate this procedure by obtaining θ D,t j via supervised learning on player j's actions given the state. Algorithm 1 summarizes L-TFT. Note that, while our experiments use a standard model-free learning algorithm for the base algorithms O C , O P , this framework is highly general. For instance, the players might be model-based planners who choose actions by optimizing against an environment model with parameter θ i , which is updated over time. Moreover, with more sophisticated methods for inferring the other player's learning algorithm, L-TFT algorithms may also be constructed in games with partial state observability, or uncertainty about the other players' rewards. Algorithm 1: Learning tit-for-tat (L-TFT) and −R t j , respectively. Agents follow Boltzmann exploration with respect to the estimated Q-function, i.e., π θi (a i | s; τ i ) ∝ exp aj Q θi (s, a i , a j )/τ i , where Q θi is a neural network with parameter θ i . The initial temperature τ i is decayed according to a schedule g(•, τ i ), such that the temperature at each time step t is τ t i = g(t, τ i ). (In our experiments g linearly decreases to a minimum value, and is constant thereafter.) Our L-TFT algorithm punishes for one episode after it has detected a defection. We assume that players may broadcast punishment to their counterpart. L-TFT agents do not update their cooperative and defecting policy estimates if their counterpart broadcasts punishment, and do not punish if they are currently being punished; for simplicity we suppress the dependence of agents' policies and of the state on these broadcasts. Input: Observation history H t ,
 Testing for defection: L-TFT q To decide whether to punish, our agent conducts a goodness-of-fit test for the hypothesis that the other player is cooperating. A goodness-of-fit test (e.g.,  Lemeshow and Hosmer Jr 1982)  compares the fit of a simple model with the fit of a flexible model. If the fit of the simple model is sufficiently close to that of the flexible model, the simple model is deemed adequate. Otherwise, the simple model is said to fit poorly. Here, the simple model we assess is the hypothesis that the other player is using cooperative DDQN with Boltzmann exploration. The flexible model we would like to compare to is that the other player is following an arbitrary learning algorithm (which need not be cooperative). One approach to estimating this model could be to solve the penalized maximum likelihood problem θ D,1 j , . . . , θ D,t j = arg max θ D,1 j ,...,θ D,t j t v=1 log π D θ D,v j (A v | S v ) + t −1 λ t v=1 θ D,v j 2 , ( 1 ) where π D θj is a class of neural networks that output a distribution over actions given a state. But solving 1 is computationally infeasible. As an approximation, we maintain a sequence { θ D,v j } t v=1 by taking gradient steps on the supervised learning loss at each time step. We maintain a buffer of log likelihoods under the hypothesis of defection, L D,t j = {log π D θ D,v j (A v j | S v )} t v=t−t . (The estimate θ D,t j and log likelihood buffer L D,t j are not updated while the other player broadcasts punishment.) On the other hand, to obtain likelihoods under the hypothesis of cooperation, player j is cooperating, we need to estimate the initial temperature τ j under the assumption that player j is following a cooperative learning algorithm for some τ j . We would thus like to solve τ t j = arg max τj t v=1 log π θ C,v j {A v | S v ; g(v, τ j )} . (2) As a tractable approximation to the solution of problem 2, we maintain an estimated cooperative Qnetwork θ C,t j via DDQN updates on reward signal R t 1 + R t 2 (only when not being punished). We track the likelihood of player j's actions under θ C,t j over a set of candidate values for τ j ; and take τ t j to be the highest-likelihood such value at time t. For each τ j , we also maintain a buffer of log likelihoods 1 L C,t j (τ j ) = {log π C,v j A v j | S v ; g(v, τ j ) } t v=t−t . (Estimate θ C,t j and buffers L C,t j (τ j ) are again not udpated while the opponent broadcasts punishment.) Finally, the hypothesis test is conducted at the end of each episode by bootstrapping  [Efron, 1992]  the difference in mean log likelihoods over buffers L C,t j ( τ t j ) and L D,t j . The hypothesis of cooperation is rejected when the q th quantile of this distribution is less than 0. This hypothesis test is summarized in Algorithm 2. We refer to the variant of L-TFT which uses this test as L-TFT q . Exploiting L-TFT q We also construct exploiter agents to play against L-TFT q . These agents assume their counterpart is L-TFT q for a particular value of q, simulate the hypothesis test used by L-TFT q , and act (myopically) depending on the result of the test. That is, if an exploiter predicts that a defection would be detected by a test with cutoff q, it acts according to the cooperative policy; otherwise, it takes a selfish action (according to a DDQN network trained on its reward signal only). Call this agent Exploiter q . The networks maintained by Exploiter q are: 1) a cooperative network trained on R t 1 + R t 2 , 2) a selfish network trained on R t i , 3) an estimate of its own policy under the hypothesis that it is cooperating θ C,l j , and 4) an estimate of its own policy under the hypothesis that it is defecting θ D,l j . Algorithm 2: Hypothesis test for L-TFT q Input: Defection log likelihood buffer L D j , cooperative log likelihood buffer L C j ( τ t j ), cutoff quantile q, test statistics buffer {∆ (q),v j } t−1 v=t−u /* t indexes episodes */ ∆ j ← ∅ for b = 1, . . . , B do L C,b j ← Bootstrap L C,t j ( τ t j ) L D,b j ← Bootstrap(L D,t j ) L C,b j ← B −1 C j ∈L C,b j C j L D,b j ← B −1 D j ∈L D,b j D j ∆ j ← ∆ j ∪ L C,b j − L D,b j ∆ (q),t j ← Percentile(∆ j , q) ∆(q),t j ← u −1 t v=t−u ∆ (q),v j // Average previous test statistics for stability if ∆(q),t j < 0 then defection_detected ← True else defection_detected ← False return defection_detected
 Environments The environments we use are the iterated Prisoner's Dilemma (IPD) and the Coin Game environment  2 . In both cases, episodes are 20 timesteps long. Our version of the IPD involves repeated play of the matrix game with expected payoffs as in Table  1 . Policy π θi gives the probability of each action given player j's action at the previous timestep. In our implementation, this means that the state passed to the estimated Q-function is the profile of actions taken at the last step. Player 2 C D Player 1 C −1, −1 −3, 0 D 0, −3 −2, −2 Table 1: Prisoner's Dilemma expected payoffs. Figure  1 : Coin Game Payoffs. Write the expected reward function, corresponding to the payoffs in Table  1 , as r i (a i , a j ). We introduce randomness by generating rewards as R t i ∼ N r i (A t i , A t j ), 0.1 . In the Coin Game environment  depicted in Figure  1 , a Red and Blue player navigate a grid and pick up randomly-generated coins. Each player gets a reward of 1 for picking up a coin of any color. But, a player gets a reward of −2 if the other player picks up their coin. This creates a social dilemma in which the socially optimal behavior is to only get one's own coin, but there is incentive to defect and try to get the other player's coin as well. The state space consists of encodings of each player's location and the location of the coin. We use networks with two convolutional layers and two fully connected feedforward layers.   (bottom) . For IPD, the prob axis is the probability of mutual cooperation (cc) and mutual defection (dd), For Coin Game, the pick_own axis is the proportion of times the coin is picked by the agent of its own color (i.e., the agents cooperate). The payoffs axis shows a moving average of player rewards after each episode. These values are averaged over 10 replicates and bands are (min, max) values. Note that temperature values are constant in the IPD after 3000 episodes and in the Coin Game after 30000 episodes.
 Results Figure  2  shows the performance of L-TFT q in self-play, against Exploiter q (i.e., an exploiter with a correctly-specified value of q). L-TFT q is able to converge to welfare-optimal payoffs in self-play and against the Exploiter q (suggesting that the exploiter eventually anticipates being punished if it defects, and therefore chooses to cooperate). This indicates that our hypothesis test effectively detects defection. L-TFT q also effectively punishes a naive selfish learner (i.e., one trained with DDQN on R t j ); see Figure  3  in the Supplementary Materials. Punishing the naive learner comes as the expense of low payoffs for L-TFT. However, this is the expected behavior, just as the punishments which enforce equilibria in classical iterated games can lead to low payoffs for each player.
 Strategic choice of q While our previous experiments with Exploiter q assumed that the exploiter knew the cutoff quantile used in L-TFT's hypothesis test, this seems like too strong an assumption for real-world agents deployed by different principals. Here we consider a game played between 1) one principal deciding which value of q 1 to specify for its L-TFT q1 agent, and 2) a second principal deciding among L-TFT q2 and Exploiter q2 for different values of q 2 . Table  2  shows the average payoffs to each player over a time horizon of 4000 steps. Table 2: Empirical payoff matrix for a learning game in which the base game is an iterated Prisoner's Dilemma played for 4000 timesteps. The row player chooses L-TFT q for q ∈ {0.55, 0.75, 0.95}. The column player chooses either L-TFT q or Exploiter q for q ∈ {0.55, 0.75, 0.95}. Cells give (row player payoffs, column player payoffs), where payoffs are the mean of rewards over the span of the game, averaged over 10 replicates. Standard errors are all less than or equal to 0.1. It turns out that the unique Nash equilibrium of this empirical learning game 3 is one in which row player chooses L-TFT 0.95 with probability 0.88 and L-TFT 0.55 with probability 0.12, while the column player chooses L-TFT 0.55 with probability 0.88 and Exploiter 0.95 with probability 0.12. The expected payoffs in equilibrium are −1.23, −1.08 for row and column player, respectively. This suggests that L-TFT q is vulnerable to some amount of exploitation on short time horizons, although still does better than the mutual-defection payoff of −2. On the other hand, L-TFT q1 converges to mutual cooperation 4 against both L-TFT q2 and Exploiter q2 for each value of q 1 , q 2 . Thus L-TFT avoids exploitation over longer time horizons in this learning game.
 Discussion Powerful learning agents should be designed to avoid conflict when they are eventually deployed. One way to avoid such outcomes is to construct profiles of agents which avoid conflict and which rational principals prefer to deploy. One appealing criterion for joint rationality on part of the principals is welfare-optimal learning equilibrium (rather than mere convergence to Nash equilibrium of the base game, for instance). We have presented a class of learning algorithms, learning tit-for-tat (L-TFT), which approximately implement welfare-optimal learning equilibrium in the sense of converging to welfare-optimal payoffs in self-play and discouraging exploitation. Many open questions remain. First, while we have relied on a welfare function, we have said little about which welfare function should be used. The ideal scenario would be for the principals of the reinforcement learning systems in question to coordinate on a welfare function before deploying these systems. Moreover, it may be necessary to develop novel reinforcement learning methods tailored for the optimization of different welfare functions. For instance, our use of DDQN with the utilitarian welfare is unproblematic, as the sum of the players' reward signals still admits a Bellman equation for their cumulative reward. But the welfare functions other than the utiliarian welfare (such as the Nash welfare  [Nash, 1950] ) do not admit a Bellman equation (see the discussion of nonlinear scalarization functions in  Roijers et al. [2013] 's review of multi-objective reinforcement learning). Many generalizations of our setting can be studied, including partial observability and more than two agents. Perhaps the most restrictive of our assumptions is that the agents' reward functions are known to each other. A complete framework should address the problem of incentives to misrepresent one's reward function to improve one's payoffs in the welfare-optimal policy profile. One direction would be to allow for uncertainty about the other players' reward functions, analogously to Bayesian games  [Harsanyi, 1967]  but in a way that is tractable in complex environments. This may in turn require the development of novel techniques for inference about other players' preferences in complex environments; see  Song et al. [2018] ,  Yu et al. [2019]  for recent steps in this direction. Another approach to reward uncertainty would be to use mechanism design techniques  [Nisan et al., 2007, Ch. 9]  to incentivize the truthful reporting of the principals' utility functions. Figure 2 : 2 Figure 2: Performance of L-TFT 0.95 in the IPD (left) and Coin Game (right). Counterparts are L-TFT 0.95 (top), Exploiter 0.95(bottom). For IPD, the prob axis is the probability of mutual cooperation (cc) and mutual defection (dd), For Coin Game, the pick_own axis is the proportion of times the coin is picked by the agent of its own color (i.e., the agents cooperate). The payoffs axis shows a moving average of player rewards after each episode. These values are averaged over 10 replicates and bands are (min, max) values. Note that temperature values are constant in the IPD after 3000 episodes and in the Coin Game after 30000 episodes.
 .55 −1.28, −1.2 −0.97, −1.48 −0.98, −1.38 −1.03, −1.41 −1.06, −1.43 −0.91, −1.81 L-TFT0.75 −1.44, −0.97 −1.13, −1.1 −1.09, −1.13 −1.19, −1.11 −1.25, −1.11 −1.21, −1.35 L-TFT0.95 −1.21, −1.06 −1.12, −1.1 −1.11, −1.11 −1.2, −1.06 −1.27, −1.03 −1.41, −0.98
			 When computing these likelihoods, we set the Q-value of an action equal to the mean of all Q-values which are within some distance of one another (in our experiments, 0.25). This is important in Coin Game, where there are multiple paths of equal length to a coin. Thus, although estimated Q-values along different paths may be very similar, these differences are amplified by the exponentiation used in Boltzmann exploration when the temperature is small, possibly leading to false detections of defection.
			 These experiments were implemented using extensions of OpenAI Gym [Brockman et al., 2016]  and the SLM Lab framework for reinforcement learning [Keng and Graesser, 2017] .
			 Computed using the Nashpy Python library [Knight and Campbell, 2018] .4  Payoffs averaged over the final 10 episodes equal to −1 ± 0.03.

Abstract: Suppose that several actors are going to deploy learning agents to act on their behalf. What principles should guide these actors in designing their agents, given that they may have competing goals? An appealing solution concept in this setting is welfare-optimal learning equilibrium. This means that the learning agents should constitute a Nash equilibrium whose payoff profile is optimal according to some measure of total welfare (welfare function). In this work, we construct a class of learning algorithms in this spirit called learning tit-for-tat (L-TFT). L-TFT algorithms maximize a welfare function according to a specified optimization schedule, and punish their counterpart when they detect that they are deviating from this plan. Because the policies of other agents are not in general fully observed, agents must infer whether their counterpart is following a cooperative learning algorithm. This requires us to develop new techniques for making inferences about counterpart learning algorithms. In two sequential social dilemmas, our L-TFT algorithms successfully cooperate in self-play while effectively avoiding exploitation by and punishing defecting learning algorithms.