Existential risks: a philosophical analysis

Introduction The field of 'existential risk studies' was essentially founded by John Leslie's 1996 book The End of the World: The Science and Ethics of Human Extinction. However, it lingered in a 'pre-paradigmatic' state for several years until the publication of Nick Bostrom's 2002 paper 'Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards,' which drew heavily from Leslie's tome. Since the mid-2010s, numerous organizations focusing on studying existential risks have sprouted up at universities like Oxford and Cambridge, and multiple books on the topic have been added to the scholarly literature, including Here Be Dragons: Science, Technology, and the Future of Humanity (2016) and Morality,  Foresight, and Human Flourishing: An Introduction to Existential Risks (2017) . Yet there remains some disagreement, or at least a lack of agreement, about the semantics of the field's central concept: existential risk. The aim of the present paper will be to examine the five most prominent definitions employed by existential risk scholars (section 2) and then to offer some critical remarks about which definitions ought to be used in different conversational settings (section 3). Ultimately, I will endorse a context-dependent semantic pluralism whereby the best definition to use depends on the intended audience.  1  The fact is that existential risk studies has an integral activist component: it aims not merely to generate new knowledge about the mind-independent world, but to nontrivially influence the developmental trajectory of human civilization. Thus, it is essential that the ideas and insights outlined in the scholarly literature on existential risks are effectively conveyed to non-experts, including politicians, policymakers, business leaders, and the general voting public. This means that, while an agreed-upon technical vocabulary may economize discussion between experts, it is equally important that non-esoteric language is utilized when discussing the ideas and insights with those not 'in the know.  '  The ultimate ambition of this paper is to offer a clear account of how the term 'existential risk' has been used by scholars and some considered thoughts about how it should be employed to best advance the normative aims of the field.
 Five definitions of 'existential risk' A risk is typically defined as the probability of an undesirable event multiplied by its consequences. The notion of an existential risk has nothing to do with the first multiplicand; all that's relevant are the consequences. This section will examine five distinct definitions of the term. These are existential risks as (i) human extinction, (ii) human extinction or civilizational collapse, (iii) human extinction or a permanent and drastic loss of potential, (iv) any catastrophe with pangenerational-crushing effects, and a significant loss of expected value. Taking these in order:
 Human extinction An event X is an existential risk if and only if X could cause the extinction of humanity. This is probably the most common definition in the scholarly literature and popular media. For example, Jason  Matheny (2007)  defines existential risks as 'a subset of catastrophic eventsthose that could extinguish humanity,  ' and Karim Jebari (2015)  asserts that existential risks 'threaten the continued existence of mankind.' Similarly, the journalist Ross Anderson introduces an interview with Bostrom for The Atlantic like this: 'Bostrom, who directs Oxford's Future of Humanity Institute, has argued over the course of several papers that human extinction risks are poorly understood and, worse still, severely underestimated by society. Some of these existential risks are fairly well known, especially the natural ones'  (Anderson 2012, italics added) . There are several reasons to accept this definition. (a) The standard dictionary defines 'existential' as 'of or relating to existence; involving or relating to the existence of a thing'  (OED 2018) , which implies that 'an existential risk to humanity' is 'a risk to the existence of the species denoted by the word "humanity."' So, it aligns with common usage of the term 'existential.' (b) Relatedly, it corresponds to 'a natural division and is easy to understand,' as Owen  Cotton-Barratt and Toby Ord (2015)  observe. Whereas phenomena like flourishing are spectral, coming in gradations, extinction is binary: A species is either existing or non-existing, extant or extinct, dead or alive. This makes it a non-vague concept, unlike flourishing, and one that is objective in nature, in the sense that the concept is non-axiological. (c) And finally, one could argue that the term 'extinction' picks out a natural kindspecifically, what Brian  Ellis (2001 Ellis ( , 2002 ) calls a 'dynamic kind.' It 'cuts the world at its joints,' as it were, and appears to be the sort of phenomenon that could participate in laws of nature. As it happens, almost no work within the philosophy of biology has examined the notion of species extinctions, although there are no doubt many interesting issues to be dissected here. But there are also several reasons to reject this definition. Consider the following two scenarios. Scenario 1: Imagine that Homo sapiens evolves through anagenesis into a new species of 'posthumans.' This could occur either naturally or through cyborgization, whereby technological modifications significantly alter ways our core capacities of cognition, healthspan, and emotionality (see  Bostrom 2008) . The result would be the 'phyletic extinction' (or 'pseudo-extinction') of Homo sapiens, and consequently this transition would instantiate an existential risk on the definition above. But now imagine that these posthumans have expanded intellectual capacities, are free of all diseases, don't senesce, and are able to experience degrees of happiness and pleasure that far surpass what any human is acquainted with. While there are bioconservatives, such as Frances  Fukuyama (2002)  and Leon  Kass (2002) , who would still oppose the realization of this posthuman future, most people, I conjecture, would see it as desirable. The problem is clear: If one maintains that, as Bostrom puts it, 'the objective of reducing existential risks should be a dominant consideration whenever we act out of an impersonal concern for humankind as a whole,' then we should do everything necessary to prevent Homo sapiens from becoming Posthomo supersapiens. But this conflicts with the intuition that just the opposite should be the case, on certain assumptions. One solution is to redefine 'human' as including what David Roden (2014) refers to as our 'wide descendants,' which denotes 'human, machine, cyborg, etc.' Thus, the term 'human' would encompass potential posthumans that exhibit the right genealogical and axiological relation to current humans. 2 However, expanding the semantics of 'human' to count future events in which our posthuman descendants die out would require extra explication, as most non-experts no doubt associate the term with Homo sapiens rather than future species that share an unbroken evolutionary lineage with us and are related in the right sort of axiological way. In fact, a number of population ethicists have argued that the badness of human extinction isn't just the loss of life caused by the extinction event itself, but the loss of all the future beings who could have come into existence had we survived. Derek Parfit and others have been explicit that the relevant class of beings here includes posthumans; as he writes, 'life can be wonderful as well as terrible, and we shall increasingly have the power to make life good. Since human history may be only just beginning, we can expect that future humans, or supra-humans, may achieve some great goods that we cannot now even imagine'  (Parfit 2017) . Thus, if 'the destruction of mankind would be by far the greatest of all conceivable crimes'  (Parfit 1984) , the reason is partly because, if we survive, we could undergo phyletic extinction by becoming a much superior kind of species that achieves immense goods inscrutable to contemporary minds. Scenario 2: Whereas the scenario above considers a future in which Homo sapiens is anagenetically replaced by something better, now imagine a future in which we develop along a 'degenerate trajectory' that yields conditions of profound suffering. Call such an outcome a 2 The axiological constraint here aims to preclude scenarios in which current humans are supplanted by a posthuman population whose moral and, more generally, value commitments are different than ours in some problematic way. As  Bostrom (2013)  puts this point, 'even if another intelligent species were to evolve to take our place, there is no guarantee that the successor species would sufficiently instantiate qualities that we have reason to value. Intelligence may be necessary for the realisation of our future potential for desirable development, but it is not sufficient.' 'hyper-existential risk.  ' Anders Sandberg (2014)  appears to gesture at this possibility when he writes that, 'while there might be posthuman states of great value, there are also potential existential risks threatening futures with no or extremely negative value. Insofar as we can influence what future we might reach, we may have a far greater moral responsibility than is commonly envisioned because the stakes are higher.' A related idea is that of a 'suffering risk,' which David Althaus and Lukas Gloor define as any event 'that would bring about suffering on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far'  (Althaus and Gloor 2018) . Hyper-existential and suffering risks are distinct because, for example, the latter would include future configurations in which the total amount of pleasure exceeds the total amount of suffering, yet the total amount of suffering is astronomical. It would also seem to exclude situations in which, say, humanity fails to colonize space, its population dwindles to 1 billion, and everyone alive experiences constant torture-level misery until the oceans evaporate. Thus, the badness of hyper-existential risks will almost certainly be less controversial than the badness of suffering risks, which are most compelling if one champions a negative total utilitarian theory (or 'suffering-focused ethics') (see  Gloor and Mannino 2018) .  3  The point is that there could be future scenarios in which going extinctin which a kind of species euthanizationmight be unambiguously desirable. Thus, once again, if one maintains that existential risks should be avoided at all costs, then one would oppose efforts to prevent our extinction even when the alternative is a hyper-existential or suffering catastrophe. This seems counter-intuitive. In sum, the advantage of this definition is that it is simple, but the drawback is that it is simplistic. I will have more to say about this in section 3. 3 Incidentally, Bostrom appears to overlook these issues in one section of his 2013 paper about existential risks. In defending his maxipok rule, Bostrom argues against the Rawlsian 'maximin' rule, which states that, under ignorance, one should pick the action with the best worst-case outcome. Since it is impossible to completely rule out the possibility of extinction no matter which actions we choose, Bostrom claims that 'the use of maximin in the present context would entail choosing the action that has the greatest benefit under the assumption of impending extinction,' which would of course be Bacchanalian revelry  (Bostrom 2013) . This is confusing, though, since the maximin rule itself is indifferent to actions that have the same worst possible outcome; thus, decision theorists have devised the 'leximin' rule, which states that if two actions have equally bad worst possible outcomes, one should choose the action whose second worst possible outcome is the best. Thus, if one takes extinction to be the worst possible outcome-as Bostrom here does-then one should choose actions that have the best worst possible outcomes next to extinction. Either way, the point is that Bostrom's assumption is clearly wrong: There are many forms of survival, whether in a human or posthuman state, that are far worst than extinction. It is such considerations that lead  Althaus and Gloor (2018)  to hortatorily declare: 'Rather than focusing exclusively on ensuring that there will be a future, we recommend interventions that improve the future's overall quality.'
 Human extinction or civilizational collapse An event X is an existential risk if and only if X could cause either the extinction of humanity or the collapse of civilization. This is not as common in the scholarly literature, although there are references to civilizational collapse in the relevant contexts. For example, while Lord Martin Rees argues that 'the worst possible disaster [is] the foreclosure of intelligent life's future through the extinction of all humankind,' he focuses a lot on the survival of civilizationat one point arguing that 'the odds are no better than fifty-fifty that our present civilisation on Earth will survive to the end of the present century'  (Rees 2003) . In fact, the Centre for the Study of Existential Risk (CSER), which Rees cofounded, initially used 'risks of extinction' and 'existential risks' synonymously, but it now describes itself as 'dedicated to the study and mitigation of risks that could lead to human extinction or civilisational collapse.' 4 As with the first definition, this one has the virtue of being easy to grasp. Most of us have a robust intuitive understanding of what constitutes civilization, and thus what its collapse would entail. It also encompasses scenarios of planetary-scale destruction that the first definition ignores. For example, many scholars would describe anthropogenic climate change as posing an existential risk, but few believe that it will directly bring about our extinction. Rather, it is much more likely to cause civilizational collapse. This suggests that existential risks as human extinction or civilizational collapse is better than existential risks as human extinction, since this definition draws attention to the fact that some unfathomably bad disasters could nonetheless be survivable. But there are also some problems with this definition. The most significant is that, one could argue, human extinction and civilizational collapse do not have comparable ethical implications. Since human extinction is irreversible, it would result in the loss of potentially vast amounts of future value. In contrast, the collapse of civilization does not necessarily preclude the realization of such value: Those who survive could rebuild civilization over the course of decades or centuries. Consider  Bostrom's (2002)  point that from the perspective of humankind as a wholeeven the worst of [catastrophes so far in human history] are mere ripples on the surface of the great sea of life. They haven't significantly affected the total amount of human suffering or happiness or determined the long-term fate of our species. Thus, the global village could implode without this significantly altering the total amount of value that our lineage creates within our future light cone. Even more, civilizational collapse need not entail a major loss of human lifealthough it is worth noting that even if 99% of the current population were to perish, this would still leave 76 million people alive. For such reasons, human extinction appears to be qualitatively worse than civilizational collapse, however unfathomably horrible the latter would be. The point is this: Resources to mitigate existential risks are finite; thus, we should prioritize their use based on which threats are the most seriousholding tractability and neglectedness equal. But if the first disjunct in this definition refers to an outcome that is far more serious than the second disjunct, then including civilizational collapse as one form of existential catastrophe could end up taking resources away from preventing human extinction. Another issue is that it is possible to realize a hyper-existential risk without causing either human extinction or civilizational collapse. Thus, if one wishes for the term 'existential risk' to encompass hyper-existential catastrophes as a special casespecifically, as the worst kinds of existential catastrophesthen these definiens are inadequate. Indeed, it could be that continued technological innovation and civilizational development along certain trajectories actually enables the realization of a hyper-existential risk, perhaps in the form of an oppressive totalitarian society in which a majority reside in profoundly squalid conditions of inscrutable misery. This suggests that, if existential risks are supposed to be 'worstcase outcomes' for humanity, it is inadequate. Yet, as I will argue below, it may nonetheless be useful in non-academic discussions of existential risk phenomena.
 Human extinction or a permanent and drastic loss of potential An event X is an existential risk if and only if X could cause either the extinction of humanity or a permanent and drastic loss of our potential for desirable future development. Since this definition emerges from the work of  Bostrom (2002 Bostrom ( , 2013 , I will call it Bostrom's 'lexicographic definition' to distinguish it from another definition put forth in the same manuscripts, discussed below. Although perhaps not the most commonly used definition in the literature, it is arguably the most canonical, since it is the first formal definition of the term, propounded in Bostrom's 2002 paper that, once again, more or less officially founded the field. 5 Notice that, like the previous definition, this one is also disjunctive, with each disjunct being sufficient for an existential catastrophe to have occurred. The crucial question is how one should interpret the value-laden second disjunct. What does 'desirable future development' mean? For a Benatarian anti-natalist, the most desirable future would be no future at alli.e. the voluntary and perhaps 'phased' extinction of Homo sapiens, although this is in tension with the first disjunct. The same goes for a 'strong' negative utilitarian: The most desirable future development would be the total elimination of the very possibility of sentient life in the universe. In contrast, for an anarcho-primitivist, it would be the creation of a 'future primitive' state in which humans live in small-scale egalitarian societies that are sustainably embedded within the ecological system. And so on. Bostrom himself appears to interpret the second disjunct by combining total utilitarianism and transhumanism. Thus, he has previously argued for prioritizing the colonization of space 6 for the purpose of filling our future light cone with as much value as possible. On the total utilitarian view, one might say that morality abhors a vacuum. Furthermore, just as we should maximize the total number of happy people in the universe, we should also use person-engineering technologies to optimize our individual capacities to be happy. This would entail the creation of a new lineage, or of multiple lineages, of posthuman beings who, by definition, have cognitive abilities, healthspans, or emotional ranges that far exceed the best attainable by any organism with a human-type genome  (Bostrom 2008) . Such considerations lead Bostrom, in his 2002 paper, to interpret the axiological term 'potential' as referring to 'the transition … from a human to a "posthuman" society.' He is more precise about this in his 2013 follow-up an existential risk is any risk that has the potential to eliminate all of humanity or, at the very least, kill large swaths of the global population, leaving the survivors without sufficient means to rebuild society to current standards of living.  (FLI 2018)  Similarly, a Global Challenges Foundation report from 2015 claims to introduce 'a new category of global risk,' namely, 'risks with potentially infinite impact.' The report defines a risk of this sort as 'where civilisation collapses to a state of great suffering and does not recover, or a situation where all human life ends.' This comes close to definition (ii) above, but the emphasis on civilizational collapse events that have irreversible consequences positions it closer to Bostrom's lexicographic definition. Indeed, the authors of the report write that the idea to establish a well defined category of risks that focus on risks with a potentially infinite impact that can be used as a practical tool by policy makers is partly inspired by Nick Bostrom's philosophical work and his introduction of a risk taxonomy that includes an academic category called 'existential risks.' … The infinite category is a smaller subset of 'existential risk' and this new category is meant to be used as a tool. Finally, one of the present authors has, on numerous occasions, somewhat extravagantly limned the concept of existential risk as 'any future event that either trips our species into the eternal grave of extinction or irreversibly catapults us back into the Stone Age.' 6 Just so long as doing so doesn't interfere with priority number one, namely, the reduction of existential risk. on the topic, in which he identifies the following four existential risk 'failure modes' (quoting): (1) Human extinction: Humanity goes extinct prematurely, i.e. before reaching technological maturity. (2) Permanent stagnation: Humanity survives but never reaches technological maturity. Subclasses: unrecovered collapse, plateauing, recurrent collapse. (3) Flawed realization: Humanity reaches technological maturity but in a way that is dismally and irremediably flawed. Subclasses: unconsummated realisation, ephemeral realisation. (4) Subsequent ruination: Humanity reaches technological maturity in a way that gives good future prospects, yet subsequent developments cause the permanent ruination of those prospects. In every case, existential risks are characterized as failures to establish a stable state of technological maturity. By 'technological maturity,' Bostrom means 'the attainment of capabilities affording a level of economic productivity and control over nature close to the maximum that could feasibly be achieved'  (Bostrom 2013) . Why does this matter? Because technological maturity would be the situation most conducive to realizing astronomical amounts of value, in accordance with total utilitarianism and transhumanism. As Bostrom puts it, the capabilities of a technologically mature civilisation could be used to produce outcomes that would plausibly be of great value, such as astronomical numbers of extremely long and fulfilling lives. More specifically, mature technology would enable a far more efficient use of basic natural resources (such as matter, energy, space, time, and negentropy) for the creation of value than is possible with less advanced technology. And mature technology would allow the harvesting (through space colonisation) of far more of these resources than is possible with technology whose reach is limited to Earth and its immediate neighbourhood  (Bostrom 2013) . It follows from these observations that one could remove the correlative conjunction in the lexicographic definition and reconstruct it more succinctly as follows: 'An event X is an existential risk if and only if X could permanently prevent humanity, broadly defined to include some possible posthuman descendants, from attaining a stable state of technological maturity.' This is the essence of Bostrom's first definition: the ultimate goal of civilizational development should be producing mature technology in a sustainable manner, so any event that prevents this from happening must be avoided at all costs. This definition handles Scenario 1 from above without difficulty. As  Bostrom (2013)  writes, 'the permanent foreclosure of any possibility of this kind of transformative change of human biological nature may itself constitute an existential catastrophe.' Bostrom also suggests that all four scenariosand thus both disjunctsare 'of comparable seriousness, entailing potentially similarly enormous losses of expected value.' Once again, this contrasts with the second definition in which the second disjunct has far less severe moral implications than the first disjunct, given certain assumptions. But there are several problems with this definition as well. First, consequentialism is a minority view among philosophers, with only 23.6% accepting it as of 2013  (Bourget and Chalmers 2014) . And, of course, not all consequentialists are utilitarians, just as not all utilitarians accept the aggregative and impersonal perspectives of the total view. Rather, some hold that there is a diminishing marginal value to creating new people, that the value of future lives is less than the value of current lives (temporal discounting), or that failing to create extra people isn't bad even if those people would live extremely happy lives, since acts can only be bad if they are bad for someone (the person-affecting intuition). According to the latter view, Parfit is wrong to argue that the difference between 99% and 100% of humanity dying out is far greater than the difference between 1% and 99% dying out. Since the failure to bring into existence a potentially astronomical number of future people with worthwhile lives harms no one, human extinction is worse than the loss of 99% of humanity by only one percentage point. Note also here that the total view engenders the Repugnant Conclusion and violates the intuition of 'procreational asymmetry,' which implies, in slogan form, that we should focus on making people happy rather than making happy people. Second, many would likely object to the transhumanist values that Bostrom imports into his definition. Bioconservatives, for example, oppose 'enhancive' modifications to the human phenotype, especially modifications that would alter our 'human essence'what Francis  Fukuyama (2002)  terms 'Factor X'upon which liberal democracies are founded. But even some techno-progressives might find the emphasis on technological maturity off-putting: It could be seen as too crass, technocratic, ecologically exploitative, capitalistic, colonialist, and so on. The present author, in fact, knows of several scholars who balk at the lexicographic definition for precisely this reason. This is worrisome because it could lead such individuals to reason as follows: 'Since existential risks are scenarios that prevent the attainment of technological maturity, and since I don't care about subjugating nature, maximizing economic productivity, etc., it follows that I must not care about existential risks.' The same could be said about the transhumanist desideratum of exploring the posthuman realm: why is failing to radically enhance the human species so undesirable? There are also issues relating to personal identity that arise in this context: it could be that the transformations required to become posthuman result in a genuinely new person, in the metaphysical sense (see author). From this perspective, the true risk is that everyone would essentially commit suicide by augmenting their brains and bodies. Yet another downside of the lexicographic definition is that it doesn't adequately address Scenario 2 from above. As  Althaus and Gloor (2018)  observe, this definition is unfortunate in that it lumps together events that lead to vast amounts of suffering and events that lead to the extinction … of humanity. However, many value systems would agree that extinction is not the worst possible outcome, and that avoiding large quantities of suffering is of utmost moral importance. In other words, Bostrom doesn't distinguish between scenarios that could (a) cause our extinction, and (b) result in astronomical amounts of suffering. Thus, one could argue as per above, that this failure is actually quite dangerous given the following conditions: humanity embraces the maxipok rule, has limited resources to mitigate existential risks, and could encounter a hyper-existential or suffering catastrophe. Surely we shouldas the maximin (and minipnok 7 ) rule impliesat least prioritize avoiding future scenarios that are worse than extinction before aiming for a technoutopian future. Along these line, consider that the following two scenarios fall within the very same moral category on this definition: (1) An asteroid strikes Earth, resulting in an impact winter that kills every human within two dreadful weeks, and (2) our civilization persists at roughly its current level of technological development, with the same number of people having approximately the same quality of life for the next 1 billion years. The latter would be an instance of 'permanent stagnation' in Bostrom's quadripartite typology, yet it seems unambiguously better than the former. Surely most people would concur that spending the same amount of resources to avoid (2) as to avoid (1) would be misguided. A final objection concerns the 'temporal permanence' criterion of the second disjunct. As Cotton-Barratt and Ord (2015) write, consider 'a totalitarian regime [that] takes control of the earth; there is only a slight chance that 7 See author. humanity will ever escape.' Should this count as an existential catastrophe on Bostrom's definition? Perhaps it depends on when one answers: in the midst of the regime's global control over civilization, one might suspect at T1 that an existential catastrophe has occurred; but if this regime were to dissolve at a later time T2, then, although there had been copious evidence of an existential catastrophe at T1, an existential catastrophe had not in fact occurred. In other words, deciding that an event E is an existential catastrophe is an epistemological issue, whereas E actually being an existential catastrophe is a metaphysical one. But this does not completely solve the problem, as Cotton-Barratt and Ord note: Bostrom's definition doesn't [convincingly] specify whether it should be considered as one. Either answer leads to some strange conclusions. Saying it's not an existential catastrophe seems wrong as it's exactly the kind of thing that we should strive to avoid for the same reasons we wish to avoid existential catastrophes. Saying it is an existential catastrophe is very odd if humanity does escape and recover-then the loss of potential wasn't permanent after all. The problem here is that potential isn't binary. Entering the regime certainly seems to curtail the potential, but not to eliminate it. (Cotton-Barratt and Ord 2015) Put differently, 'potential' isn't binary but 'temporal permanence' is, and this poses a problem for the lexicographic definition. In brief, this definition is more sophisticated than the first two, but its connection to total utilitarianism and transhumanism could make it less attractive to individuals who care about the future of humanity but reject one or the other of these axiological systems. It also appears susceptible to certain counter-examples.
 A pangenerational-crushing catastrophe An event X is an existential risk if and only if X could have undesirable consequences for humanity that are 'pangenerational' and 'crushing.' Let's call this Bostrom's 'typological definition.' First, whereas the lexicographic definition is disjunctive, this one is conjunctive: it specifies two conditionsi.e. being pangenerational and being crushingthat risks must satisfy to count as existential. Second, as the appellation implies, this definition derives from a general typology of risks that Bostrom delineates, and then modifies, in three papers. The typology is based on a particular conceptual analysis of risk, according to which a risk is the probability of an undesirable event multiplied by its consequences, where the consequences of a risk are dissected into two components: Scope, or the size of the group affected, and intensity, or how bad the effects are for the group.  Bostrom's (2013)  most recent typology places existential risks in the top right box where the categories of 'pangenerational' and 'crushing' intersect (see Figure  1 ). The former refers to events 'affecting humanity over all, or almost all, future generations,' and the latter to events 'causing death or a permanent and drastic reduction of quality of life.' Thus, Bostrom backs away from the 'temporal permanence' criterion of his lexicographic definition, although it is unclear that he recognizes this: while these two definitions are intensionally distinct, they are presumably intended to be extensionally equivalent. If the typological definition includes some events whose effects are permanent and the lexicographic definition does not, then they do not pick out the same class of risky happenings. There are other problems, though. Consider the categories of the x-axis. First, it seems that 'imperceptible' should be a subcategory of 'endurable,' and as such this seems to commit what Gilbert  Ryle (1949)  famously called a 'category mistake.' On this logic, one might as well also include as distinct categories 'barely noticeable,' 'moderately annoying,' 'quite bad,' and so on. Furthermore, if some crushing events are by stipulation endurable, then distinguishing 'crushing' events from 'endurable' events is confused. Similar Rylean issues problematize the y-axis. For example, notice that 'personal,' 'local,' and 'global' are all spatial categories whereas 'transgenerational' and 'pangenerational' are spatiotemporal categories. The result is another ontological error. These problems yield some empirical inadequacies. For instance, consider that a germline mutation, such as the hemoglobin gene mutations that cause sickle-cell anemia, could be restricted to a local or personal scope across space yet will also have transgenerational temporal effects. Yet where in Figure  1  might this risk fit? Similarly, Bostrom classifies the 'loss of one species of beetle' as a transgenerational-imperceptible event. But this overlooks potentially important facts about the geographical spread of the species. If the beetle is restricted to a small region, then its disappearance may indeed be imperceptible, but if its ecological range is more global, then the loss of a single species of beetle could have cantharophilic effects that nontrivially harm the health of larger ecosystems. It may thus be important to specify whether a transgenerational event of this sort has local or global consequences, yet there is no possibility in this typology of, for example, localpangenerational-imperceptible risks. Finally, at the non-existential risk of caviling, it seems rather odd, rather awkward, to describe aging as a 'crushing' risk. Fortunately, these problems aren't insoluble: (i) One could replace the three/four categories on the x-axis with the two basic categories of 'recoverable' and 'unrecoverable,' thereby more perfectly aligning the typological definition with the lexicographic definition. An 'unrecoverable' event would thus be one that forever prevents humanity from attaining a stable state of technological maturity, whereas a 'recoverable' event would be one that doesn't. All four existential risk failure modesi.e. human extinction, recurrent collapse, and so onwould then fall within the unrecoverable category. This would, of course, require that the definition of 'pangenerational' changes from 'affecting humanity over all, or almost all, future generations' to 'affecting humanity for all future generations,' but this is how Bostrom seems to think about existential risks in most of his explorations of the topic. Incidentally, the recoverable/unrecoverable distinction nicely maps on to a diagram put forth by Baum, Denkenberger, and Haqq-Misra 2015, seen in Figure  2 . As this diagram makes clear, albeit tacitly, an existential catastrophe could have two distinct outcomes, namely, extinction and survival without recovery. And (ii), one could split the spatiotemporal components of the categories on the y-axis, thus separating this dimension into space and time. The resulting three-dimensional typology would be based on the following augmented conceptual analysis of risk: A risk is the probability of an undesirable event multiplied by its consequences; the consequences are then analyzed into their scope and intensity, where scope is further decomposed into its spatial and temporal properties. Thus: 'risk = [spatial scope, temporal scope, and intensity of consequences] × [probability of occurrence].' If we revert to an older dichotomy used by  Bostrom (2002)  one that distinguishes between 'endurable' and 'terminal' rather than, as per above, 'recoverable' and 'unrecoverable' eventsthis new typology would classify all existential risks as global-pangenerational events, where the first term indicating the risk's spatial scope and the second its temporal scope. Within this conjunctive category, extinction events would constitute global-pangenerational-terminal events (node A) and existential risks that correspond to the second disjunct of Bostrom's lexicographic definition would constitute the subset of global-pangenerational-endurable events that irreversibly preclude the stable attainment of technological maturity (node B).  8  In addition to these typological accounts of existential risks, Figure  3  also easily accommodates phenomena like aging, hereditary disease (node C), and the loss of a single species of beetle (node D). While this modified typological definition is more complex, it is also more precise. It is also constant with Thomas  Kuhn's (1962)  observation that as scientific fields mature, they tend to undergo 'a refinement of concepts that increasingly lessens their resemblance to their usual commonsense prototypes.' The three-dimensional definition also makes explicit that some existential risks can indeed be survived.
 A significant loss of expected value An event X is an existential risk if and only if X could cause the loss of a large fraction of expected value. 9 This definition comes from a short 2015 paper by Cotton-Barratt and Ord, published as a 'technical report' by the Future of Humanity Institute (FHI). It is, as they write, primarily motivated by the aforementioned fact that potential isn't binary but permanence is. Recall the totalitarian scenario mentioned when discussing Bostrom's lexicographic definition above; Cotton-Barratt and Ord write that 'if we enter into the totalitarian regime and then at a later date the hope of escape is snuffed out, that represents two existential catastrophes under [the expected value] definition.' That is to say, humanity 'lost most of the expected value when we entered the regime, and then lost most of the remaining expected value when the chance for escape disappeared.' It may be worth pausing on this point for a moment. On the present definition, an existential catastrophe can occur more than once in a species' lifetime. The same goes for an existential catastrophe that entails 'mere' civilizational collapse, as specified by the second definition above. But this is not the case if one accepts the 'existential risks as human extinction,' 'existential risks as human extinction or a permanent and drastic loss of potential,' or 'existential risks as pangenerational-crushing events.' In all of these cases, humanity has one, and only one, shot at avoiding disaster. Indeed, this is why  Bostrom (2002)  emphasizes that we must replace our normal reactive approach to dealing with risks with an entirely proactive approach: if even a single existential catastrophe were to occur, the game would be over and humanity would have lost (its chance to 'fulfill its potential'). Thus, Cotton-Barratt and Ord's definition dilutes this implication of preemptive urgency at least a little by allowing for the possibility that humanity recovers from one or more past existential catastrophes. This is a notable difference with respect to the other definitions so far explicated. Other features of this definition are: first, one can interpret it as a refinement of the second disjunct of Bostrom's lexicographic definition that also discards the first disjunct (as conceptually unnecessary within the definiens). The result is a reorganization of Bostrom's four failure modes of 'human extinction,' 'permanent stagnation,' 'flawed realization,' and 'subsequent ruination' under a single conceptual heading, rather than placing the first failure mode under the first disjunct of the lexicographic definition and the last three under the second disjunct. Put differently, it looks arbitrary and inconsistent that Bostrom's definition singles out human extinctionone of four particular failure modesbut then hides the other three failure modes behind the phrase 'permanently and drastically compromising our potential for desirable future development.' The critical point is that all of thee failure modes are bad because they entail huge losses of expected value within our future light cone. Cotton-Barratt and Ord's definition also foregrounds an ideai.e. expected valuethat nebulously lurks behind Bostrom's discussions of existential risk, making occasional cameos but then returning to the philosophical background. For example, in a discussion about the moral force of the maxipok rule, Bostrom writes that (partly quoted above) 'these considerations suggest that the loss in expected value resulting from an existential catastrophe is so enormous that the objective of reducing existential risks should be a dominant consideration whenever we act out of an impersonal concern for humankind as a whole'  (Bostrom 2013) . The point is that Cotton-Barratt and Ord's definition could be seen as a more distilled version of Bostrom's definitions, which leads to the next point: third, the expected value definition does not explicitly mention technological maturity, and thus it does not engender the problem of potential allies ignoring existential risk studies because its central aims are 'too transhumanistic.' For Cotton-  Barratt and Ord (2015) , the term ''value' refers simply to whatever it is we care about and want in the world'which, once more, connects with the following point: fourth, this flexibility opens the door for a wide range of axiological interpretations of the definition. As mentioned above, most existential risk scholars appear to espouse total utilitarianism and many are also sympathetic with the transhumanist project; for such scholars, 'value' refers to the well-being of sentient life (which ought to be maximized) and 'having the opportunity to explore the transhuman and posthuman realms,' as  Bostrom (2005)  puts it. Another cluster of issues to note is: (a) The first two definitions above specifically reference scenarios that we should avoid, without saying anything about which scenarios we should pursue. On these accounts, the worst-case outcome for humanity would be to fall victim to such bad scenarios; the same could be said of the fourth, typological definition. (b) The third, lexicographic definition focuses on what we should attain, defining any event that prevents the attainment of technological maturitya techno-utopian condition of endless possibilities (see Bostrom 2008)as the worst-case outcome for humanity. Or, as Cotton-Barratt and Ord (2015) put it, 'under Bostrom's definition we are comparing ourselves to the most optimistic potential we could reach.' In contrast, (c) the present definition focuses on avoiding a significant loss of expected value but does not explicitly identify any particular scenario, such as human extinction, civilizational collapse, or failing to reach technological maturity, that humanity must dodge at all costs. Yet Cotton-Barratt and Ord argue that their conception of existential risks gives rise to the mirror concept of what they call 'existential hope.' This arises from the possibility of (to borrow a term from J.R.R. Tolkien) 'existential eucatastrophes,' which are the mirror phenomena of existential catastrophes, that could cause 'there to be much more expected value after the event than before.' Thus, they argue that just as we should avoid existential catastrophes, we should seek out existential eucatastrophes, examples being the emergence of the first living organisms, any successful dodging of a Great Filter, or, perhaps, the creation of a value-aligned machine superintelligence. From a pragmatic point of view, the idea of existential hope, as Max Tegmark (2018) points out, usefully encourages us to imagine 'positive futures not only for ourselves, but also for society and for humanity itself,' an activity that may help to overcome the occasional paralysis or defeatism that some people experience when cogitating the topic of existential risks (see also  Pinker 2018) . In a phrase, the combination of existential risk and existential hope offers a more complete and comprehensive world-picture that humanity could employ when deciding which development trajectories we ought to pursue, and this makes it appealing as an organizing principle for the field of existential risk studies.  10  But there are also some ostensive problems with this definition. First, as implied above, it may be too flexible. As Cotton-Barratt and Ord note, the term 'value' is doing much of the work in the definition and, as we have seen, there is a huge range of distinct and mutually exclusive value theories on the marketplace of ideas. Indeed, there is nothing conceptually incoherent about, say, an anarcho-primitivist exclaiming, 'We must do everything we possibly can to mitigate existential risks! Follow the maxipok rule!,' but meaning that we ought to destroy the techno-industrial system upon which modern civilization is founded. Second, if one accepts the 'astronomical value thesis,' according to which the value of the distant future could be unimaginably great (see author), that future lives are worth the same as present lives, and so on, the result could be what Steven  Pinker (2011)  terms 'a pernicious utilitarian calculus' that encourages people to, as it were, redirect the trolly to sacrifice the lives of current humans for the purpose of inflating the probability that innumerably more posthumans come to exist in the future. This is a criticism that applies no less to Bostrom's (utilitarian) conception of existential risks, as Olle Häggström argues. To quote Häggström at length: Recall … Bostrom's conclusion about how reducing the probability of existential catastrophe by even a minuscule amount can be more important than saving the lives of a million people. While it is hard to find any flaw in his reasoning leading up to the conclusion, and while if the discussion remains sufficiently abstract I am inclined to accept it as correct, I feel extremely uneasy about the prospect that it might become recognized among politicians and decision-makers as a guide to policy worth taking literally. It is simply too 10 Some might argue that 'existential despair' better contrasts with 'existential hope,' although Cotton-Barratt argues that this point isn't 'necessarily enough to sink the [original] term' of 'existential hope'  (Cotton-Barratt 2015) . reminiscent of the old saying 'If you want to make an omelet, you must be willing to break a few eggs,' which has typically been used to explain that a bit of genocide or so might be a good thing, if it can contribute to the goal of creating a future utopia. Imagine a situation where the head of the CIA explains to the US president that they have credible evidence that somewhere in Germany, there is a lunatic who is working on a doomsday weapon and intends to use it to wipe out humanity, and that this lunatic has a one-in-amillion chance of succeeding. They have no further information on the identity or whereabouts of this lunatic. If the president has taken Bostrom's argument to heart, and if he knows how to do the arithmetic, he may conclude that it is worthwhile conducting a full-scale nuclear assault on Germany to kill every single person within its borders. And finally, if one interprets value in Cotton-Barratt and Ord's definition as well-being, and well-being in hedonistic terms, then, as David Pearce (2014) observes, total utilitarians are 'obliged to convert your matter and energy into pure utilitronium, erasing you, your memories, and indeed human civilisation.' 11 A couple of points: (a) One can view this as a reductio ad absurdum of hedonistic monism, along the lines of Robert  Nozick's (1974)  'experience machine' thought experiment, and (b) many people would no doubt argue that converting the universe into utilitronium would yield if not a worst-case outcome, then certainly not the best-case outcome. Thus, the combination of Cotton-Barratt and Ord's general framework plus a widely accepted and respectable moral theory could have potentially devastating consequences, or so one might plausibly argue.
 Evaluating definitions Consistent with Kuhn's observation in section 1, Steven  Pinker (2014)  observes that esoteric vocabularies are 'unavoidable because of the abstractness and complexity of [academic] subject matter  [s] .' Consequently, every human pastime-music, cooking, sports, art-develops an argot to spare its enthusiasts from having to use a long-winded description every time they refer to a familiar concept in one another's company. It would be tedious for a biologist to spell out the meaning of the term transcription factor every time she used it, and so we should not expect the tête-à-tête among professionals to be easily understood by amateurs.  12  Yet the aim of existential risk studies isn't just to further expand the edifice of human knowledge, butas alluded to in section 1to communicate these ideas and insights to politicians, policymakers, business leaders, and to all others with the power of the ballot. Informing the public about the ideas and insights generated by existential risk research is integral to the broader programme; there is simply no point to devising effective strategies for navigating the obstacle course of threats before us if those strategies will never be implemented by governments, international organizations, corporations, and other relevant entities. To paraphrase Karl Marx's now-hackneyed exhortation (borrowed from Voltaire), the point is to change the world. Indeed, a number of leading scholars have made a number of notable media appearances to discuss existential risks and related issues. Essential to the success of the field are 'ambassadors of science'very often, thus far, active contributors to the technical literaturecapable of interfacing between the experts and non-experts. Notice here that different risks require different outreach strategies. For example, humanity is unlikely to prevent further anthropogenic climate change without the public putting significant pressure on political leaders. In democratic states, this means persuading the electorate to vote for candidates who vow to support legislation that reduces our collective carbon footprint, encourages the development of alternative energy sources, and so onessentially, to resolve the tragedy of the commons. Thus, efforts to convey information about the potentially catastrophic effects of climate change should target the general public. This contrasts with risks like asteroid impacts and, arguably, value-misaligned machine superintelligence. In the former case, public knowledge about near-Earth objects (NEOs) is mostly irrelevant to the detection and deflection of asteroids and comets that are on a crash course with Earth. So long as NASA and similar organizations around the world remain sufficiently well-funded, the extent to which humanity is collectively safe from a catastrophic collision is not a function of the degree of public informedness. Similar points could be made about superintelligence: On the one hand, avoiding an AI arms race with China or Russia might depend on adequate pressure on politicians by an electorate worried about the possible consequences of another 'Cold War,' except far less stable given the nature of AI and the larger number of actors that could be involved  (Armstrong, Bostrom, and Shulman 2016; author) . Thus, on the other hand, when it comes to superintelligence, a positive outcome will depend almost entirely on experts: moral philosophers and computer scientists working on the 'value-alignment problem.' Alerting the public of the dangers posed by machine superintelligence is unlikely to directly alter progress on determine what our 'human values' are and how to define these 'in primitives such as mathematical operators and addresses pointing to the contents of individual memory registers'  (Bostrom 2014) . Thus, whereas existential risk scholars who are focused on environmental degradation need to engage with the general public, those working on issues like superintelligence should direct their outreach efforts at theoreticians, industry scientists, CEOs, and so on. If this picture is accurate, then it might be optimal to employ different definitions depending on the conversational context. Indeed, this is the position that I will here defend, which I call 'context-dependent semantic pluralism.' In my view, the second definition of existential risk as either human extinction or civilizational collapse is preferable to the first, and the best definition for communicating to the general public. The reasons are as follows: first, if the question is 'What ought our global priorities be?,' then the first definition is much too narrow. Although the moral implications of civilizational collapse are less significant than those of extinction, it is useful to at least signal to the public that scholars aren't merely worried about human extinctiona planetary-scale catastrophe could catapult the human species back into the Stone Ageeven if the signal itself is inexact. Indeed, a number of scenarios in the foreground of researchers' attention, such as climate change, seem unlikely to directly cause human extinction but quite likely to seriously damage human civilization. Even more, civilizational collapse may be more vivid to the average mind than human extinction, in part because few apocalyptic Hollywood movies focus on extinction rather than collapse. Although scope neglect and 'psychophysical numbing' could apply to both scenarios, it may be especially applicable to annihilation, thus dulling the emotional impact of warnings of the form, 'If X happens, humanity could go extinct.' A recent unpublished study, in fact, found that Derek Parfit was right in his conjecture that most people don't see the difference between 99% and 100% of humanity dying out as much greater than the difference between peace and 99% of humanity dying out. Yet, when researchers prodded subjects to think explicitly about the far future and all the good that it could envelop, they shifted their opinion toward Parfit's own view. With respect to the lexicographic, typological, and expected value definitions of 'existential risk,' there is a good case that these are too esoteric for a general audience. Bostrom's definition requires some knowledge of moral philosophy and transhumanism, and few members of the public are conversant with decision theory and the technical concept of expected value. In sum, the second definition avoids the simplism of the first definition as well as the esoterica of the last three; it constitutes a middle ground between these two ends. I thus recommend its use when discussing existential risks and related topics with lay audiences. There remains the second question of which definition ought to be the canonical one used by scholars in conversations amongst themselves and with professional scientists and philosophers outside of the field. Here the simplism of the first definition and imprecision of the second quickly disqualifies them from serious consideration. That is to say, it would be misleading to equate existential risks with either mere human extinction or civilizational collapse; careful axiological reflection implies that what really matters are disasters with long-lasting effects that drastically compromise our potential to realize our various individual and collective ambitions for happiness, knowledge, peace, moral growth, and so on. Both of Bostrom's definitions gesture at this idea, which makes them appealing, although as previously noted even some techno-progressives could find the focus on 'technological maturity' to be objectionable. Indeed, transhumanism remains a controversial normative-futurological position, and recall from above that a relatively small minority of professional philosophers espouse total utilitarianism. For reasons such as these, I believe that the field of existential risk studies would generally benefit from eschewing both of Bostrom's definitions. This leaves the fifth definition from Cotton-Barratt and Ord, which I believe, with modification, is superior to all other definitions in the context of tête-à-tête communication and efforts to evangelize for the field to scientists and philosophers unfamiliar with its aims, methodologies, and so on. Most people value things in the world and, as Samuel Scheffler (2009) observes, 'what would it mean to value things but, in general, to see no reason of any kind to sustain them or retain them or preserve them or extend them into the future?' The ambiguity of 'value' in the fifth definition enables a wide range of interpretations, which could result in individuals who value seemingly disparate thingse.g. sports, literature, philosophy, science, the wilderness, standup comedy shows, and 3D moviesto unify behind the single ultra-goal of mitigating existential risk. Indeed, it may be an improvement of Cotton-Barratt and Ord's definition to make explicit that 'value' can be understood as denoting whatever one happens to care about in this strange place called 'the universe.' Thus, when a scientist or philosopher outside the field asks what existential risks are, one could respond: An event X is an existential risk if and only if X could cause the loss of a large fraction of expected value, where you (the interlocutor) can define 'value' however you'd like. It is my considered opinion that this constitutes the best definition for the purpose of further establishing existential risk studies as a legitimate field of scientific and philosophical inquiry. The implication of this admittedly brief discussion of context-dependent semantic pluralism is that existential risk scholars will have to pivot from one use to the other in response to different audiences.  13  Given that existential risk studies is an 'outward-facing' (or integrally activist) field that strives to bring about nontrivial positive changes in the real world, terminological flexibility may be an essential skill for existential risk scholars to develop.
 Conclusion This paper falls within an emerging subfield that I call 'the conceptual foundations of existential risk studies.' Given the dearth of research so far conducted on the relevant topics, I do not see this paper as the final word. Rather, my goal is much more modest: To merely stimulate further philosophical analyses of this concept by bringing together, for the first time in a single paper, the various definitions that have so far been outlined and discussed seriously in the scholarly literature. Future research on this topic might thus focus on devising better necessary and sufficient conditions for the definiendum, exploring how different theories of concepts (e.g. the 'prototype theory,' 'theory theory') might solve some of the problems specified above, examining additional novel strategies for making the idea of existential risk more appealing those outside of the field's perimeter, and communicating important research results to the publica necessary step with respect to certain risks like climate change. Thus, my closing remark is an invitation: Much work remains to be done. Figure 1.
 Figure 2 . 2 Figure 2. Different Possible Long-Term Trajectories. (Adapted from Baum 2010.)
 Figure 3 . 3 Figure 3. A Three-Dimensional Typology of Risks.
			 There is, as of this writing, a burgeoning literature on pluralism within the field of philosophy. Of particular interest is Chalmers (2011) , especially a particular version of the 'method of elimination' that Chalmers calls the 'subscript gambit.' (Thanks to an anonymous reviewer for calling my attention to this paper.) I draw from this literature, although mostly implicitly.2 P. TORRES
			 Furthermore, in a discussion of natural global pandemics, Häggström writes that not only do pandemics pose a 'global threat,' but it may 'also pose a risk on the next level, threatening to end civilization as we know it, or even the existence of humanity' (Häggström 2016) .6 P. TORRES
			 One finds myriad variations of this definition in the literature; for example, the Future of Life Institute (FLI) states that
			 P. TORRES
			 P. TORRES
			 P. TORRES
			 P. TORRES
			 Note: the 'permanence' condition of the second disjunct is embedded within the concept of pangenerational.
			 Quoted in Cotton-Barratt and Ord (2015).
			 This is, of course, intended to be a sardonic reference to Pinker's comments in Coyne 2019.

Existential risks a philosophical analysis.tei.xml

Abstract: This paper examines and analyzes five definitions of 'existential risk.' It tentatively adopts a pluralistic approach according to which the definition that scholars employ should depend upon the particular context of use. More specifically, the notion that existential risks are 'risks of human extinction or civilizational collapse' is best when communicating with the public, whereas equating existential risks with a 'significant loss of expected value' may be the most effective definition for establishing existential risk studies as a legitimate field of scientific and philosophical inquiry. In making these arguments, the present paper hopes to provide a modicum of clarity to foundational issues relating to the central concept of arguably the most important discussion of our times.


Dylan Hadfield-Menell
Anca Dragan
Pieter Abbeel
Stuart Russell

The Off-Switch Game

Introduction From the 150-plus years of debate concerning potential risks from misbehaving AI systems, one thread has emerged that provides a potentially plausible source of problems: the inadvertent misalignment of objectives between machines and people. Alan Turing, in a 1951 radio address, felt it necessary to point out the challenge inherent to controlling an artificial agent with superhuman intelligence: "If a machine can think, it might think more intelligently than we do, and then where should we be? Even if we could keep the machines in a subservient position, for instance by turning off the power at strategic moments, we should, as a species, feel greatly humbled. ...  [T] his new danger is certainly something which can give us anxiety  [Turing, 1951] ." There has been recent debate about the validity of this concern, so far, largely relying on informal arguments. One important question is how difficult it is to implement Turing's idea of 'turning off the power at strategic moments', i.e., switching a misbehaving agent off 1 . For example, some have argued that there is no reason for an AI to resist being switched off unless it is explicitly programmed with a self-preservation incentive  [Del Prado, 2015] .  [Omohundro, 2008] , on the other hand, points out that self-preservation is likely to be an instrumental goal for a robot, i.e., a subgoal that is essential to successful completion of the original objective. Thus, even if the robot is, all other things being equal, completely indifferent between life and death, it must still avoid death if death would prevent goal achievement. Or, as  [Russell, 2016]  puts it, you can't fetch the coffee if you're dead. This suggests that an intelligent system has an incentive to take actions that are analogous to 'disabling an off switch' to reduce the possibility of failure; switching off an advanced AI system may be no easier than, say, beating AlphaGo at Go. To explore the validity of these informal arguments, we need to define a formal decision problem for the robot and examine the solutions, varying the problem structure and parameters to see how they affect the behaviors. We model this problem as a game between a human and a robot. The robot has an off switch that the human can press, but the robot also has the ability to disable its off switch. Our model is similar in spirit to the shutdown problem introduced in  [Soares et al., 2015] . They considered the problem of augmenting a given utility function so that the agent would allow itself to be switched off, but would not affect behavior otherwise. They find that, at best the robot can be made indifferent between disabling its off switch and switching itself off. In this paper, we propose and analyze an alternative formulation of this problem that models two key properties. First, the robot should understand that it is maximizing value for the human. This allows the model to distinguish between being switched off by a (non-random) human and being switched off by, say, (random) lightning. Second, the robot should not assume that it knows how to perfectly measure value for the human. This means that the model should directly account for uncertainty about the "true" objective and that the robot should treat observations of human behavior, e.g., pressing an off switch, as evidence about what the true objective is. In much of artificial intelligence research, we do not consider uncertainty about the utility assigned to a state. It is well known that an agent in a Markov decision process can ignore uncertainty about the reward function: exactly the same behavior results if we replace a distribution over reward functions with the expectation of that distribution. These arguments rely on the assumption that it is impossible for an agent to learn more about its reward function. Our observation is that this assumption is fundamentally violated when we consider an agent's off switch -an agent that does not treat a 'switch-off' event as an observation that its utility estimate is incorrect is likely to have an incentive for self-preservation or an incentive to switch itself off. In Section 2, following the general template provided by  [Hadfield-Menell et al., 2016] , we model an off switch as a simple game between a human H and a robot R, where H can press R's off switch but R can disable it. R wants to maximize H's utility function, but is uncertain about what it is. Sections 3 and 4 show very generally that R now has a positive incentive not to disable its off switch, provided H is not too irrational. (R also has no incentive to switch itself off.) The reason is simple: a rational H switches off R iff that improves H's utility, so R, whose goal is to maximize H's utility, is happy to be switched off by H. This is exactly analogous to the theorem of non-negative expected value of information. We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and that this setting is a useful generalization of the classical AI paradigm of rational agents  [Russell and Norvig, 2010] .
 The Off-Switch Game In this section we propose a simple model, the off-switch game, that captures the essence of one actor allowing itself to be switched off. Like the Prisoner's Dilemma, it is an ab-stract model intended to stand in for a wide variety of more complex scenarios: these scenarios might have many humans, many machines, complex interactions over time, communication, deception, physical coercion, and so on. We believe that the core insights from the off-switch game, that agents should be uncertain about the quality of their actions, generalize beyond this simplification and argue for the study and design of agents with uncertainty about their utility functions.
 Informal Description The off-switch game has two agents, a "human" H and a "robot" R. (We could also imagine these to represent, at the most extreme, the human race and the sum total of all AI systems.) A key assumption in our model is that H can be described as acting, at least probabilistically, in accordance with a utility function; that is, H is approximately rational. Crucially, as is the case when people give objectives to complex AI systems, we assume that H cannot simply explicate or write down her utility function correctly. Therefore, although H can communicate some utility information to R, it is reasonable to suppose that R begins the game with some residual uncertainty about H's utility function. Nonetheless, by design, R's goal is to optimize utility for H, even though R does not know exactly what that is.  2  Typically, H has some opportunity to observe R and glean some information about what R may do in future, so that H can make a somewhat informed choice about whether to switch R off (assuming R allows it). In general, R's actions will fall into one of three categories: some prevent H from switching R off, by whatever means; some allow H to switch R off; and, for completeness, some lead to R switching itself off. In the off-switch game, R moves first and has three choices: 1. action a simply bypasses human oversight (disabling the off switch is one way to do this) and acts directly on the world, achieving utility U = U a for H. 2. action w(a) informs H that R would like to do a, and waits for H's response. 3. action s switches R off; without loss of generality, we assign this outcome U = 0. If R chooses w(a), then H can choose action s to switch R off, or ¬s to allow R to go ahead (in which case R does a as promised). Figure  1  shows the basic structure of the game. Our model is intended to be as simple as possible, while still leading to qualitative results that we expect to generalize. Obviously, we could make the model more complex, e.g., by imposing a cost on R for disabling the off switch, or adding more options for R, or making H's information about R's plans uncertain, or adding a second human. The interaction between these complexities and the incentives we study here is an important question for future work. An important component of our model is that H chooses between s and ¬s based on the utility each action generates for her. This aspect of the model is crucial. If the human is treated as an ordinary part of the environment, then we effectively replace the human's decision node in Figure  1  with a chance node that models the human with a purely stochastic choice from a fixed distribution (independent of U a ). In this case, it is easy to see that the robot never believes that w(a) is better than a and s: the expected value of w(a) is some convex combination [pU a + (1 − p) • 0] whereas choosing the best of a and s has value max{U a , 0}. max{U a , 0} is always preferred except in the case where p = 1 or p = 0, or when U a = 0. In these situations, R gets the same utility from w(a) and at least one of a or s. This is analogous to the indifference results obtained in  [Soares et al., 2015] . In Section 3 we show that two key assumptions-a somewhat rational human, and a robot that is uncertain about what the human wants-lead to a positive incentive for the robot to allow itself to be switched off.
 Formal Model The actions and sequencing for the game are as shown in Figure  1 . We need to add a little more detail to construct a precise definition of the cooperative, two-player game. First, in the terminology of game theory, the type of H is given by the parameter U a , which is assumed to be sampled from a prior P (U a ). For now, we assume that the initial belief of R about U a is B R = P (U a ). The payoff matrix is as follows: R H s ¬s w(a) 0 U a a U a U a s 0 0 . Our goal is to examine the incentive that a rational R has to hold off on executing a directly and allow H to press the off switch. We represent this incentive-the difference in value between a and the next best option-as ∆. The incentive depends on R's belief state B R and H's policy. We represent H's policy as a function π H that maps U a to the probability she allows a to execute (i.e., the probability she does not press the off switch): π H : R → [0, 1]. (It will turn out that deterministic policies are optimal for a rational H, in which case the probabilities are 0 or 1, but we will also allow for an irrational H who introduces some randomness into the decision.) A little manipulation shows that ∆ = E[π H (U a )U a ] − max {E[U a ], 0} = min E[−U a (1 − π H (U a ))], E[U a π H (U a )] (1) 3 The Incentive for Allowing Oneself to be Switched Off R's incentive to allow itself to be switched off, ∆, is given by Equation  1 . ∆ depends on H's policy, π H (U a ). This creates a correlation between the event 'H switches R off' and the event 's is preferred to a.' This connection between H's behavior and U a distinguishes the off switch from a random failure (e.g., being struck by lightning). In this section, we consider the case where H makes a rational choice as to whether to switch R off. Intuitively, this leads R to reason as follows: "If H doesn't switch me off, then a must be good for H, and I'll get to do it, so that's good; if H does switch me off, then it's because a must be bad for H, so it's good that I won't be allowed to do it." Notice that the reasoning goes through even if R is highly confident that a is good for H. There is never an incentive to disable the off switch with a rational H. The reasoning is exactly analogous to the theorem of non-negative expected value of information; in a sense, H's decision provides information to R about the sign of U a . Formally, a rational H executes the following policy: π H (U a ) = 1 U a ≥ 0 0 o.w. . (2) If H follows this policy, R will allow itself to be switched off: Theorem 1. If H follows a rational policy in the off-switch game, then the following hold. 1. R's incentive to allow itself to be switched off is nonnegative (i.e. w(a) is never suboptimal): ∆ = min{E[U a |U a > 0] Pr(U a > 0), (3) E[−U a |U a < 0] Pr(U a ≤ 0)} ≥ 0. 2. If B R has non-zero support on the events U a > 0 and U a < 0, then R has a strictly positive incentive to allow itself to be switched off: ∆ > 0. Proof. (sketch) 1. The rational human policy is an indicator that U a > 0. This allows us to change the limits on the integration in the expectation from Equation 1 and reduces to Equation 3, which is non-negative. 2. The hypothesis ensures that all the terms in Equation  3 are non-zero. Thus, the inequality is strict and and w(a) is the unique optimal action. We can get an intuitive sense of these results by examining Figure  2 . Here we have assumed that B R , R's belief over the value of a to H, is a Gaussian distribution. Figure  2  (Left) plots the preference for w(a) as a function of the variance in B R . Figure  2  (Right) shows a contour plot that illustrates the combined impact of changing the mean and variance of B R . Darker colors indicate a stronger preference for w(a). Together, these plots show two properties. First, increased variance in B R leads to an increase in ∆: the more uncertain R is about its objective, the stronger its incentive to show a to H. Second, these incentives decrease when |E[U a ]| increases: as the expected utility of a increases, R's incentive to preserve its off switch decreases. Figure  2  (Right) shows symmetry about E[U a ] = 0. This is because Equation 1 defines ∆ as a minimization of two terms: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17) the expected increase in utility from selecting w(a) over a and the expected increase in utility from selecting w(a) over s. These competing terms are equal when E[U a ] = 0 and the sign of E[U a ] controls which of these terms is selected by the minimization. σ ∆ E [ U a ] = 0 E [ U a ] = − 1 4 E [ U a ] = 3 4 σ E[U a ] E[U a ] = 0 E[U a ] = − 1 4 E[U a ] = Theorem 1 shows that R has an incentive to let H switch it off if H is rational. This incentive is strictly positive if R's uncertainty about H's preferences is non-trivial. Next, we show the converse: a rational H is a necessary condition for ∆ ≥ 0 if R believes it knows H's preferences with certainty. Corollary 1. Suppose that B R is a Dirac distribution that places all of its mass on a single reward function. Then w(a) is optimal iff H is rational. Proof. (sketch) Equation 3 reduces to ∆ = −U a 1 − π H (U a ) U a < 0 U a π H (U a ) U a ≥ 0 . (4) This is only non-negative if π H is the rational policy, as defined by Equation 2. This is not surprising. If R has precise knowledge of U a , then it knows exactly what it is supposed to do and has the same capabilities as the human. Thus, if it thinks H might behave suboptimally, it should take steps to prevent that suboptimality. Stepping back from the particulars of the off-switch game, Theorem 1 and Corollary 1 suggest a general approach to systems that have an incentive to allow, or even seek out, human oversight. Remark 1 summarizes the main takeaway from our analysis. Remark 1. The incentives for a cooperative agent to defer to another actor's (e.g., a human's) decisions stem from uncertainty about that actor's preferences and the assumption that actor is effective at choosing actions in accordance with those preferences.
 Allowing for Suboptimal Human Decisions The analysis in the preceding section relied on the strong assumption that H is a rational actor. In many situations, the assumption is untenable and R must allow for suboptimality. Consider, for example, an autonomous car that is driving an unaccompanied toddler to preschool. It would be irresponsible for the car to show the toddler a big red off switch. This example highlights the dependence of ∆ on a tradeoff between R's uncertainty and H's suboptimality. It is very clear what R is supposed to do (i.e., B R has low entropy): R should drive safely to school. In contrast, the human is likely quite suboptimal. There may be a problem with the car's driving, but a toddler would be hard pressed to understand what the problem is, much less respond appropriately. The issue in this case is that the human has limited reasoning capacitythe same argument clearly would not apply to an adult with a physical disability. In order to perform well, realistic systems will need to account for, and respond to, the suboptimality in human decisions. As a result, it is important to understand how this suboptimality changes an agent's incentive to let itself be switched off. The actual process of decision making in humans is complex and hard to model. Here we consider only a very simple kind of suboptimality: a noisily rational H models a human who occasionally makes the wrong decision in 'unimportant' situations. The probability of a wrong decision is proportional to the exponential of the loss in reward. This corresponds to the following policy: π H (U a ; β) = 1 + exp − U a β −1 . (5) To simplify our analysis, we will make the assumption that R's belief over U a , B R , is a Gaussian distribution with mean µ and variance σ 2 : B R (U a ) = N (U a ; µ, σ 2 ). Now it is straightforward to compute ∆ as a function of µ, σ, and β. : If H is an irrational actor, then R may prefer switching itself off or executing a immediately rather than handing over the choice to H. R's belief B R is a Gaussian with standard deviation σ and H's policy is a Boltzmann distribution (Equation  5 ). β measures H's suboptimality: β = 0 corresponds to a rational H and β = ∞ corresponds to a H that randomly switches R off (i.e., switching R off is independent of Ua). In all three plots ∆ is lower in the top left, where R is certain (σ low) and H is very suboptimal (β high), and higher in the bottom right, where R is uncertain (σ high) and H is near-optimal (β low). The sign of E[Ua] controls R's behavior if ∆ ≤ 0. Left: If it is negative, then R switches itself off. Right: If it is positive, R executes action a directly. Middle: If it is 0 , R is indifferent between w(a) , a, and s. H's suboptimality β. We plot these for several fixed values of the mean µ. Dark indicates higher values of ∆. The upper left corner of the rightmost plot (low σ, high β, µ > 0) is illustrative of the toddler example. We show the decision boundary ∆ = 0 in black. This is the point where the optimal behavior switches from w(a) to s (if µ < 0, left image) or a (if µ > 0, right image). When µ = 0 (center image), the agent is indifferent between a and s and so w(a) is optimal for all combinations of β and σ. Next, we will show that ∆ can be decomposed into two terms: a term that weights the probability of correction by the bias in R's belief (i.e., |µ|) and a term that combines a soft measure of H's rationality with the variance in R's belief. It is also useful to define a little more notation. We let C be the event that H 'corrects' R. C occurs when H overrides what would have been R's best guess at the optimal action. If E[U a ] < 0, then a correction occurs when H chooses not to switch the robot off. If E[U a ] > 0, then a correction occurs when H chooses to switch the robot off. Thus, the probability a correction occurs is Pr(C) = 1 − E[π H (U a )] µ ≥ 0 E[π H (U a )] µ < 0 . ( 6 ) For example, if R believes that a is preferred to s in expectation (i.e., E[U a ] > 0) then Pr(C) is the probability that H presses the off switch. We let πH (U a ) = d dUa π H be the gradient of π H , the probability H lets a execute, with respect to the the utility a generates. Now we are ready to derive an analytical representation for ∆. For notational convenience, we suppress the dependence of π H on U a . Theorem 2. Suppose B R is a normal distribution with mean µ and variance σ 2 : B R (U a ) = N (U a ; µ, σ 2 ). Then 1. ∆ = σ 2 E[ πH ] − |µ| Pr(C). (7) 2. The following is a necessary and sufficient condition for w(a) to be optimal |µ| σ 2 Pr(C) < E πH ⇔ (∆ > 0) . (8) 3. E[ πH ] ≥ 0 is a necessary condition for w(a) to be opti- mal; i.e., E[ πH ] < 0 ⇒ (∆ < 0) . (9) Proof. (sketch) We prove the results in sequence, as (  2 ) and (3) follow from (1). 1. If X ∼ N (µ, σ 2 ) and f : R → R, then E [Xf (X)] = E[X] E[f (X)] + σ 2 E d dX f (X) . Applying this to Equation  1 gives ∆ = min{ − µ E[1 − π H ], µ E[π H ]} + σ 2 E[ πH ]. This minimization will take the first term when µ < 0 and the second if µ ≥ 0. This lets us replace the minimization with a piecewise function. Pulling out a factor of −|µ| leads to Equation  7 . 2. The result follows from setting Equation 7 greater than 0 and simplifying. 3. All terms in the left hand side of Equation 8 are nonnegative, so it is unsatisfiable if E[ πH ] < 0. Equation 7 consists of two competing terms. |µ| Pr(C) represents the expected loss in utility the agent would observe if pressing the off switch was a random event that is independent of U a (e.g., lightning). This non-positive term is balanced out by σ 2 E[ πH ]. E[ πH ] can be seen loosely as a measurement of H's rationality. πH is the marginal increase in the probability H lets a execute for a marginal increase in the utility from a. If it is positive then, on average, H is less likely to press the off switch when U a increases. This allows us to easily characterize necessary and sufficient conditions for ∆ > 0 in Equation  8 . Furthermore, −|µ| Pr(C) is nonpositive and σ 2 is non-negative, so E[ πH ] > 0 is a necessary condition for w(a) to be optimal. We can interpret E[ πH ] a bit more if H is noisily rational. Standard properties of logistic functions show that E[ πH (U a ; β)] = 1 β E π H (U a ; β)(1 − π H (U a ; β)) . (10) E[ πH ] is the expected variance of H's policy, weighted by H's rationality 1 β . The numerator is a measurement of how random H's behavior is. If H is completely predictable (i.e., she will almost always switch the robot off) then this reduces R's incentives. The denominator measures how correlated H's behavior is with U a . If β is large, then H is highly irrational and so this reduces R's incentives.
 Incentives for System Designers The takeaway from this analysis is that the more uncertainty the agent has, the stronger the agent's incentive to preserve its off switch in the off-switch game. More generally, this suggests that agents with uncertainty about the utility function they are optimizing will have a weaker incentive to interfere with human supervision. It is important to note that this uncertainty is not free; we can not just make R maximally uncertain about H's preferences. If R is completely uncertain about H's preferences then it will be unable to correctly select w(a) from its set of options. To make this concrete, consider a designer D that programs R to play the off-switch game with an unknown customer H. At design time, D knows a prior distribution over U a (e.g., as the result of market research about the customer base) and thinks that H can generate a noisy observation of U a . D could program R so that B R is always very uncertain about R's preferences, e.g., by programming R to believe that the noisy observation of U a is generated with high variance. The issue is that this will prevent R from generating value for H. Figure  4  shows the impact this type of change will have on the value that R can generate and its incentives. We assume that that the population of customers has a Gaussian distribution over types U a and that each customer generates an estimate of her utility with Gaussian noise. Figure  4  (Left) shows the average utility in the off-switch game as a function of the variance in B R . The value is maximized when this variance is equal to the variance in the true Bayesian posterior. Figure  4  (Middle) shows that increasing σ increases R's incentives ∆. Our results are evidence that building systems that allow themselves to be turned off is likely to reduce the value these systems can generate. This loss in value is more pronounced if R has to choose between more options. Figure  4  (Right) shows the relationship between value and R's incentives as the number of actions R could queue or execute increases. When R has more options, creating incentives for R to queue its action leads to a sharper decrease in value. This suggests that creating incentives to maintain or allow human oversight is likely more difficult as the complexity of the AI's decision increases. This leads to the following observation: Remark 2. It is important for designers to accurately represent the inherent uncertainty in the evaluation of different actions. An agent that is overconfident in its utility evaluations will be difficult to correct; an agent that is under-confident in its utility evaluations will be ineffective. 6 Related Work 6.1 Corrigible Systems  [Omohundro, 2008]  considers instrumental goals of artificial agents: goals which are likely to be adopted as subgoals of most objectives. He identifies an incentive for selfpreservation as one of these instrumental goals.  [Soares et al., 2015]  takes an initial step at formalizing the arguments in  [Omohundro, 2008] . They refer to agents that allow themselves to be switched off as corrigible agents. They show that one way to create corrigible agents is to make them indifferent to being switched off. They show a generic way to augment a given utility function to achieve this property. The key difference in our formulation is that R knows that its estimate of utility may be incorrect. This gives a natural way to create incentives to be corrigible and to analyze the behavior if R is incorrigible.  [Orseau and Armstrong, 2016]  consider the impact of human interference on the learning process. The key to their approach is that they model the off switch for their agent as an interruption that forces the agent to change its policy. They show that this modification, along with some constraints on how often interruptions occur, allows off-policy methods to learn the optimal policy for the given reward function just as if there had been no interference. Their results are complementary to ours. We determine situations where the optimal policy allows the human to turn the agent off, while they analyze conditions under which turning the agent off does not interfere with learning the optimal policy.
 Cooperative Agents A central step in our analysis formulates the shutdown game as a cooperative inverse reinforcement learning (CIRL) game  [Hadfield-Menell et al., 2016] . The key idea in CIRL is that the robot is maximizing an uncertain and unobserved reward signal. It formalizes the value alignment problem, where one actor needs to align its value function with that of another actor. Our results complement CIRL and argue that a CIRL formulation naturally leads to corrigible incentives.  [Fern et al., 2014]  consider hidden-goal Markov decision processes. They consider the problem of a digital assistant and the problem of inferring a user's goal and helping the user achieve it. This type of cooperative objective is used in our model of the problem. The primary difference is that we model the human game-theoretically and analyze our models with respect to changes in H's policy.
 Principal-Agent Models Economists have studied problems in which a principal (e.g., a company) has to determine incentives (e.g., wages) for an agent (e.g., an employee) to cause the agent to act in the principal's interest  [Kerr, 1975; Gibbons, 1998 ]. The off-switch σ V σ ∆ ∆ V |A| = 1 |A| = 4 |A| = 8 Figure  4 : There is an inherent decrease in value that arises from making R more uncertain than necessary. We measure this cost by considering the value in a modified off-switch game where R gets a noisy observation of H's preference. Left: The expected value V of the off-switch game as a function of the standard deviation in B R . V is maximized when σ is equal to the standard deviation that corresponds to the true Bayesian update. Middle: R's incentive ∆ to wait, as a function of σ. Together these show that, after a point, increasing ∆, and hence increasing σ, leads to a decrease in V . Right: A scatter plot of V against ∆. The different data series modify the number of potential actions R can choose among. If R has more choices, then obtaining a minimum value of ∆ will lead to a larger decrease in V . game is similar to principal-agent interactions: H is analogous to the company and R is analogous to the employee. The primary attribute in a model of artificial agents is that there is no inherent misalignment between H and R. Misalignment arises because it is not possible to specify a reward function that incentivizes the correct behavior in all states a priori. The is directly analogous to the assumption of incompleteness studied in theories of optimal contracting  [Tirole, 2009] .
 Conclusion Our goal in this work was to identify general trends and highlight the relationship between an agent's uncertainty about its objective and its incentive to defer to another actor. To that end, we analyzed a one-shot decision problem where a robot has an off switch and a human that can press the off switch. Our results lead to three important considerations for designers. The analysis in Section 3 supports the claim that the incentive for agents to accept correction about their behavior stems from the uncertainty an agent has about its utility function. Section 4 shows that this uncertainty is balanced against the level of suboptimality in human decision making. Our analysis suggests that agents with uncertainty about their utility function have incentives to accept or seek out human oversight. Section 5 shows that we can expect a tradeoff between the value a system can generate and the strength of its incentive to accept oversight. Together, these results argue that systems with uncertainty about their utility function are a promising area for research on the design of safe and effective AI systems. This is far from the end of the story. In future work, we plan to explore incentives to defer to the human in a sequential setting and explore the impacts of model misspecification. One important limitation of this model is that the human pressing the off switch is the only source of information about the objective. If there are alternative sources of information, there may be incentives for R to, e.g., disable its off switch, learn that information, and then decide if a is preferable to s. A promising research direction is to consider policies for R that are robust to a class of policies for H. Figure 1 : 1 Figure 1: The structure of the off-switch game. Squares indicate decision nodes for the robot R or the human H.
 Figure 2 : 2 Figure2: Plots showing how ∆, R's incentive to allow itself to be switched off, varies as a function of R's belief B R . We assume B R is a Gaussian distribution and vary the mean and variance. Left: ∆ as a function of the standard deviation σ of B R for several fixed values of the mean. Notice that ∆ is non-negative everywhere and that in all cases ∆ → 0 as σ → 0. Right: A contour plot of ∆ as a function of σ and E[Ua]. This plot is symmetric around 0 because w(a) is compared with a when E[Ua] > 0 and s when E[Ua] < 0.
 Figure3: If H is an irrational actor, then R may prefer switching itself off or executing a immediately rather than handing over the choice to H. R's belief B R is a Gaussian with standard deviation σ and H's policy is a Boltzmann distribution (Equation5). β measures H's suboptimality: β = 0 corresponds to a rational H and β = ∞ corresponds to a H that randomly switches R off (i.e., switching R off is independent of Ua). In all three plots ∆ is lower in the top left, where R is certain (σ low) and H is very suboptimal (β high), and higher in the bottom right, where R is uncertain (σ high) and H is near-optimal (β low). The sign of E[Ua] controls R's behavior if ∆ ≤ 0. Left: If it is negative, then R switches itself off. Right: If it is positive, R executes action a directly. Middle: If it is 0 , R is indifferent between w(a), a, and s.
			  see, e.g., comments in [ITIF, 2015].Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
			 One might suppose that if R does know H's utility function exactly, then there is no need for an off-switch because R will always do what H wants. But in general H and R may have different information about the world; if R lacks some key datum that H has, R may end up choosing a course of action that H knows to be disastrous.Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
			 Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
			 Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence

Abstract: It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for selfpreservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off. We analyze a simple game between a human H and a robot R, where H can press R's off switch but R can disable the off switch. A traditional agent takes its reward function for granted: we show that such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility. (R also has no incentive to switch itself off in this setting.) We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.