A peer-reviewed electronic journal published by the Institute for Ethics and Emerging Technologies

A new subfield The field of existential risk studies, or existential riskology, can be traced back to a 1996 book by the philosopher John  Leslie. 1  In the early 2000s, the field emerged as a more formal discipline of active scholarship, led primarily by transhumanists  (Bostrom 2005) . Numerous institutions dedicated to understanding the greatest threats to our collective future have since been founded, such as the Future of Life Institute, the Centre for the Study of Existential Risks (Cambridge), and the Future of Humanity Institute (Oxford). Despite these signs of progress, the field remains in something like a "pre-paradigmatic" stage, whereby a comprehensive research program has yet to be firmly established. A particularly problematic gap in the scholarship stems from the failure of existential riskologists to take seriously the range of agents who might use advanced technologies to initiate a catastrophe. One finds only occasional references in the literature to "psychopaths," "hate groups," "terrorists," and "malevolent governments," typically without any further details about the unique properties of these entities. This mistake is tantamount to asserting, "Future technologies -I'll refrain from saying which ones, how they might be used, the properties that make them dangerous, and so on -could annihilate humanity." Just as it's crucial to study the properties of advanced technologies, so too is it crucial to study the properties of agents. The present paper aims to rectify this shortcoming: in effect, it establishes a new subfield of agential riskology. The paper is organized as follows: the next section establishes some basic terminology. Sections 3 and 4 examine the phenomena of agential terror and agential error, respectively. The penultimate section then argues that we should expect the threat of ecoterrorism and apocalyptic terrorism to increase nontrivially in the coming decades.
 Definitions An "existential risk" is an event that results in either total annihilation or a permanent and severe reduction in our quality of life  (Bostrom 2002 ). Let's refer to the definiens' first disjunct as an "extinction risk" and the second disjunct as a "stagnation risk." Extinction risks are terminal for our species, but stagnation risks are survivable, although they entail an irreversible state of significant deprivation, perhaps resulting in the life opportunities of contemporary North Koreans or our ancestors from the Paleolithic. From a transhumanist perspective, both scenarios would prevent us from reaching a posthuman state in which one or more of our "core capacities" are augmented beyond their natural limits  (Bostrom 2008 ). I use the term "existential risk" to reference either scenario, while "extinction risk" and "stagnation risk" refer to specific existential circumstances. Existential risks are defined by their consequences, not their probability or etiology. With respect to the latter, we can identify three broad categories of existential risk types. First, there are risks posed by nature, such as supervolcanic eruptions, global pandemics, asteroid/comet impacts, supernovae, black hole explosions or mergers, galactic center outbursts, and gamma-ray bursts. These form our cosmic risk background and they have no direct, immediate connection to human activity -that is, except insofar as advanced technologies could enable us to neutralize them. For example, we could deflect an incoming asteroid with a spacecraft or devise a vaccine to contain a deadly pathogen that might otherwise cause a global outbreak of infection. Second, there are anthropogenic risks like climate change and biodiversity loss. These are the accidental byproducts of industrial civilization. As elaborated below, both are slow-motion catastrophes that will almost certainly lower the "conflict thresholds" that ensure peace between state and nonstate actors. They will, in other words, exacerbate existing geopolitical tensions and introduce entirely new struggles. Climate change and biodiversity loss could thus be considered "context risks" whose most significant effects are to modulate the dangers posed by virtually every other existential risk facing humanity -including those from nature.  2  Other anthropogenic risks include physics disasters (such as the Large Hadron Collider) and accidentally contacting hostile aliens through Active SETI projects. The third category subsumes risks that arise from the misuse and abuse of advanced "dual-use" technologies. The property of "dual usability" refers to the moral ambiguity of such technologies, which can be used for either good or bad.  3  The very same centrifuges that can enrich uranium for nuclear power plants can also enrich uranium for nuclear bombs, and the very same technique (such as CRISPR/Cas9) that might enable scientists to cure diseases could also enable terrorists to synthesize a designer pathogen. According to many existential riskologists, advanced technologies constitute the greatest threat to our collective future. Not only are many of these technologies becoming exponentially more powerful -thereby making it possible to manipulate and rearrange the physical world in unprecedented new ways -but some are becoming increasingly accessible to groups and individuals as well. Consequently, the total number of token agents capable of inflicting harm on society is growing. While the existential risk literature offers many sophisticated accounts of how such tools could be used to cause a catastrophe, almost no one has examined the various agents (with one notable exception) who might want to do this and why.  4  Let's define a "tool" as any technology that an agent could use to achieve its ends, and an "agent" as any entity, independent of its material substrate, with the capacity to choose its own actions in the world. This lacuna is problematic because the "risk potential" of advanced technologies can be realized only by a complete "agent-tool coupling." In other words, a tool without an agent isn't going to destroy the world. Engineered pandemics require engineers, just as a nuclear missile launch requires a nuclear missile launcher. Thus, it's crucial to study the various properties special to every type of agent. Without a careful examination of both sides of the agent-tool coupling, existential risk scholars could leave humanity vulnerable to otherwise avoidable catastrophes. To illustrate this point, consider a world X in which a large number of species-annihilating technologies exist, and another world Y in which only a single such technology exists. Now imagine that world X contains a single dominant species of peaceable, compassionate beings who almost never resort to violence. How dangerous is this world? If one looks only at the tools, it appears to be extremely dangerous. But if one considers the agents too, it appears to be extremely safe. Now imagine that world Y contains a species of bellicose, warmongering organisms. Again, if one looks only at the tools, then Y appears far safer than X. But when the complete agent-tool complex comes into view, Y is clearly more likely to selfannihilate. A final distinction needs to be made before moving on to the next section, namely that between error and terror. Note that this distinction is agential in nature. It concerns the agential intentions behind a catastrophe independent of the catastrophes' consequences. Thus, an error could, no less than terror, bring about an existential disaster. In the case of world X, one might argue that an error is most likely to cause an existential catastrophe, whereas in Y the greatest threat stems from terror. The error/terror distinction is important in part because there appear to be far more token agents who might induce an extinction or stagnation disaster by accident than are likely to bring about such an outcome on purpose. The next two sections discuss agential terror and agential error in turn.
 Agential terror Many existential riskologists identify terror involving advanced technologies as the most significant threat to our prosperity and survival. But upon closer examination, there are fewer types of agents who would want to cause an existential catastrophe than one might suspect. Consider another thought experiment: imagine a future world in which there exists a profusion of "doomsday buttons" that are accessible to every citizen of Earth. The question then arises: what sort of individual would intentionally push this button? What kind of agent would purposively cause an existential catastrophe? If the intention were to actualize an extinction risk, the agent would need to exhibit at least two properties. First, it would need to devalue its own post-catastrophe survival. In other words, the agent would have to be suicidal. This immediately disqualifies a large number of entities as potential agential risks, since states and political terrorists tend to value their own survival. Neither North Korea nor al-Qaeda, for example, is suicidal. Their goal, in each case, is to change rather than destroy humanity. Even in the case of suicide bombers and kamikaze pilots, the aim is to ensure group survival through the altruistic sacrifice of one's own life. And second, the agent would need to want every other human on the planet to perish. In other words, he or she would have to be omnicidal. (We can coin the term "true omnicide" to refer to circumstances that combine both suicide and omnicide, as just defined, resulting in the irreversible termination of our evolutionary lineage.) In contrast, if the aim were to actualize a stagnation risk, the agent could be suicidal, omnicidal, or neither, but not both (according to the above definitions). A terrorist could, for example, attempt to permanently cripple modern civilization without harming anyone, including him or herself. Alternatively, a terrorist could attempt to cripple civilization through a suicide attack or an attack directed at others. Either way, the relevant agent would be motivated by an ideology that is incompatible with our species reaching a posthuman state. In the following discussion, we will consider extinction and stagnation possibilities separately.
 A typology of agential risks With these properties in mind, let's examine five categories of agents that, when coupled with sufficiently destructive tools, might purposively bring about an existential catastrophe. (1) Superintelligence. This is one of the most prominent topics of current existential risk studies, although it's typically conceptualized -on my reading of the literature -as a technological risk rather than an agential risk. To be clear, a variety of agent types could use narrow AI systems as a tool to achieve their ends. But once an AI system acquires human-level intelligence or beyond, it becomes an agent in its own right, capable of making its own decisions in pursuance of its own goals. Many experts argue that superintelligence is the greatest long-term threat to human survival, and I concur. On the one hand, a superintelligence could be malevolent rather than benevolent. Call this the amityenmity conundrum. Roman Yampolskiy (2015) delineates myriad pathways that could lead to humanunfriendly superintelligences. For example, human programmers could intentionally program a superintelligence to prefer enmity over amity. (The relevant individuals could thus be classified as agential risks as well, even though they wouldn't be the proximate agential cause of an existential catastrophe.) A malevolent superintelligence could also arise as a result of a philosophical or technical failure to program it properly  (Yudkowsky 2008) , or through a process of recursive self-improvement, whereby a "seed AI" augments its capacities by modifying its own code. But it's crucial to note that a superintelligence need not be malevolent to pose a major existential risk. In fact, it appears more likely that a superintelligence will destroy humanity simply because our species happens to be somewhere between it and its goals. Consider two points: first, the relevant definition of "intelligence" in this context is "the ability to acquire the means necessary to achieve one's ends, whatever those ends happen to be." This definition, which is standard in the cognitive sciences, is roughly synonymous with the philosophical notion of instrumental rationality. And since it focuses entirely on an agent's means rather than its ends, it follows that an intelligence could have any number of ends, including ones that we wouldn't recognize as intelligible or moral. Scholars call this the "orthogonality thesis"  (Bostrom 2012) . For example, there's nothing incoherent about a superintelligent machine that believes it must purify Earth of humanity because God wills it to do so. Nor is there anything conceptually problematic about a superintelligent machine whose ultimate goal is to manufacture as many paperclips as possible. This goal may sound benign, but upon closer inspection it appears just as potentially catastrophic as an AI that wants us dead. Consider the fact that to create paperclips, the superintelligence would need a source of raw materials: atoms. As it happens, this is precisely what human bodies are made out of. Consequently, the superintelligence could decide to harvest the atoms from our bodies, thereby causing our extinction. As Eliezer Yudkowsky puts it, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else"  (Yudkowsky 2008) . Scholars categorize resource acquisition, along with self-preservation, under the term "instrumental convergence." Even more, our survival could be at risk in situations that initially appear favorable. For example, imagine a superintelligence that wants to eliminate human sadness from the world. The first action it might take is to exterminate Homo sapiens, because human sadness can't exist without humans. Or it might notice that humans smile when happy, so it could try to cover our faces with electrodes that cause certain muscles to contract, thereby yielding a "Botox smile." Alternatively, it might implant electrodes into the pleasure centers of our brains. The result could be a global population of euphoric zombies too paralyzed by pleas-ure to live meaningful lives  (Bostrom 2014, 146-48) . All of these outcomes would, from a certain perspective be undesirable. The point is that there's a crucial difference between "Do what I say" and "Do what I mean," and figuring out how to program a superintelligence to behave according to the latter is a formidable task. Making matters worse, a superintelligence whose material substrate involves the propagation of electrical potentials rather than action potentials would be capable of processing information orders of magnitude faster than humans. Call this a quantitative superintelligence. As Yudkowsky observes, if the human brain were sped up a million times, "a subjective year of thinking would be accomplished for every 31 physical seconds in the outside world, and a millennium would fly by in eight-and-a-half hours"  (Yudkowsky 2008) . A quantitative superintelligence would thus have a huge speed advantage over humanity. In the amount of time that it takes our biological brains to process the thought, "This AI is going to slaughter us," the AI could already be halfway done the deed. Another possibility concerns not speed but capacity. That is, an AI with a different cognitive architecture could potentially think thoughts that lie outside of our species-specific "cognitive space." This is based on the following ideas: (a) to understand a mind-independent feature of reality, one must mentally represent it, and (b) to mentally represent that feature, one must generate a concept whose content consists of that feature. Thus, if the mental machinery supplied to us by nature is unable to generate the relevant concept, the corresponding feature of reality will be unknowable. Just as a chipmunk can't generate the concepts needed to understand a boson or the stock market, so too are the concept-generating mechanisms of our minds limited by their evolutionary history. The point is that a qualitative superintelligence could come to understand phenomena in the universe that are permanently beyond our epistemic reach. This could enable it to devise ways of manipulating the world that would appear to us as pure magic. In other words, we might observe changes in the world that we simply can't understand -that are as mysterious as the science behind cellphones or the atomic bomb is to a chipmunk scientist. In sum, not only would a quantitative superintelligence's speed severely disadvantage humanity, but a qualitative superintelligence could also discover methods for "commanding nature," as it were, that would leave us utterly helpless. As with the other agents below, superintelligence itself doesn't pose a direct threat to our species. But it could pose a threat if coupled to any of the tools previously mentioned, including nuclear weapons, biotechnology, synthetic biology, and nanotechnology. As Bostrom writes, if nanofactories don't yet exist at the time, a superintelligence could build them to produce "nerve gas or target-seeking mosquito-like robots [that] might then burgeon forth simultaneously from every square meter of the globe"  (Bostrom 2014) . A superintelligence could also potentially gain control of automated processes in biology laboratories to synthesize a designer pathogen, exploit narrow AI systems to disrupt the global economy, and generate false signals in early-warning systems to provoke a nuclear exchange between states. A superintelligence could press a preexisting doomsday button or create its own button. According to a recent poll, a large majority of AI researchers believe that an artificial general intelligence (AGI) will be created this century  (Müller and Bostrom 2016) . But such predictions have been notoriously inaccurate in the past. This fact is immaterial to the present thesis, though, which merely states that if, independent of when, humans build a superintelligence, it could pose a major, unpredictable agential risk. (2) Idiosyncratic actors. This category includes individuals or groups who are driven by idiosyncratic motives to destroy humanity or civilization. History provides several examples of the mindset that would be required for such an act of terror. First, consider Eric Harris and Dylan Klebold, the adolescents behind the 1999 Columbine High School massacre. Their aim was to carry out an attack as spectacular as the Oklahoma City bombing, which occurred four years earlier. They converted propane tanks into bombs, built 99 improvised explosive devices, and equipped themselves with several guns. By the end of the incident, 12 students and one teacher were dead, while 21 others were injured. (Although if the propane bombs had exploded, which they didn't, all 488 students in the cafeteria at the time could have perished.) This was the deadliest school shooting in US history until Adam Lanza killed 20 children and 6 adults at Sandy Hook Elementary School in 2012 before committing suicide. This leads to the question: what if Harris and Klebold had generalized their misanthropic hatred from their high school peers to the world as a whole? What if certain future anticipated technologies had been available at the time? In other words, what if they'd had access to a doomsday button? Would they have pushed it? The plausible answer is, "Yes, they would have pushed it." If revenge on school bullies was the deeper motive behind their attack, as appears to be the case,  5  then what better way to show others "who's boss" than to "go out with the ultimate bang"? If people like Harris and Klebold, with their dual proclivities for homicide and suicide, get their hands on advanced technologies in the future, the result could be true omnicide. History also provides a model of someone who might try to destroy civilization without intentionally killing anyone. Consider the case of Marvin Heemeyer, a Colorado welder who owned a muffler repair shop. After years of a zoning dispute with the local town and several thousand dollars in fines for property violations, Heemeyer decided to take revenge by converting a large bulldozer into a "futuristic tank." It was covered in armor, mounted with video cameras, and equipped with three gun-ports. On June 4, 2004, he climbed inside the tank and headed into town. With a top speed of a slow jog and numerous police walking behind him during the incident, Heemeyer proceeded to destroy one building after another. Neither a flash-bang grenade thrown into the bulldozer's exhaust pipe nor 200 rounds of ammunition succeeded in stopping him. After more than two hours of relentless destruction, the bulldozer became lodged in a basement, at which point Heemeyer picked up a pistol and shot himself. The motivation of this attack was also a form of bullying, that is, as perceived by Heemeyer. A significant difference between Heemeyer's rampage and the Columbine massacre is that, according to some residents sympathetic with Heemeyer, he went out of his way not to injure anyone. Indeed, he was the only person to die in the attack.  6  It's also worth pointing out that Heemeyer saw himself as God's servant. As he put it, "God blessed me in advance for the task that I am about to undertake. It is my duty. God has asked me to do this. It's a cross that I am going to carry and I'm carrying it in God's name." Again, we can ask: what if a delusional person like Heemeyer were to someday hold a grudge not against the local town, but civilization as a whole? What if a future person feels abandoned or "screwed over" by society and wants to retaliate for perceived injustices? In the past, lone wolves with idiosyncratic grievances were unable to wreak havoc on society because of the limited means available to them. This will almost certainly change in the future, as advanced technologies become increasingly powerful and accessible.  7  This category is especially worrisome moving forward, since it is arguably the type with the most potential tokens. Perhaps future advances in psychology, or brain-decoder technologies and other surveillance systems, will enable us to identify agents at risk of engaging in violence of this kind. (3) Ecoterrorists. Imagine for a moment that scientists found a 52 per cent reduction in the global population of wild vertebrates between 1970 and 2010. Imagine that, on even the most optimistic assumptions, the biosphere had entered the sixth mass extinction event in life's 3.8 billion year history. Imagine further than a single species were almost entirely responsible for this environmental crisis, namely Periplaneta americana, the American cockroach. What would our response be? To exterminate the culprit, of course, thereby saving Earth from a planetary catastrophe. It's this basic line of reasoning that could lead a group of radical environmentalists or a lone wolf activist to attempt to relocate humanity from the category of "extant" to "extinct." In fact, the claims made above are scientifically accurate: the total population of wild mammals, birds, reptiles, amphibians, and fish really did halve in forty years, and we really are in the beginning stages of a new mass extinction event (see WWF 2014;  Ceballos et al. 2015) . But the culprit isn't the American cockroach, it's humanity. To date, the vast majority of environmentalist movements have been peaceful, despite the FBI classifying some affiliates of the Earth Liberation Front (ELF) and the Animal Liberation Front (ALF) as constituting "one of the most serious domestic terror threats in the US"  (Flannery 2016) . Even groups that believe Gaia would be better off without Homo sapiens trampling the planet, such as the Voluntary Human Extinction Movement (VHEMT), reject violence or coercion as legitimate means for achieving their ideological aims. Nonetheless, there are exceptions. On September 1, 2010, James Lee terrorized a Discovery Channel building in DC with a gun and explosives. During several hours of negotiations, Lee explained to law enforcement officials that he wanted new programming on the Discovery Channel "to convey how to improve the natural world and reverse human civilization." He also wanted humans to stop procreating. As Lee wrote, "Children represent FUTURE catastrophic pollution whereas their parents are current pollution. NO MORE BABIES!" (quoted in  Flannery 2016) . Around 4:48pm, Lee aimed his gun at a hostage in the building and SWAT snipers killed him. Once more, the question is: if Lee had access to a doomsday button that, say, could have sterilized every human on the planet (thereby causing an extinction catastrophe), would he have pushed it? The answer is, "Yes, probably." The radical environmental movement is founded on the philosophy of biocentrism, or the view that "humans are no more intrinsically valuable than any other creature." Some scholars have even suggested that this ideology and its following should be seen as a "new religious movement," which Bron Taylor calls the "deep green religion"  (Taylor 2009) . Along these lines, Ted Kaczynski, discussed below, advocated what he called a "wilderness religion." Given the fanaticism of this stance, it's not hard to envisage a group emerging in the future that attempts to bring about human extinction through the use of advanced technologies in an effort to "save" the planet. This scenario is made even more plausible by the fact that the largest demographic of Earth Liberation Front members consists of "well educated" and "technologically literate" males  (Flannery 2016) . Thus, if synthetic biology techniques were to enable the synthesis of a designer pathogen that infects only Homo sapiens, a radical environmentalist could try to spread this germ around the globe. Or they could design and release self-replicating nanobots that selectively target Homo sapiens by recognizing genetic signatures unique to our DNA. Such nanobots could annihilate our species while leaving the biosphere more or less unharmed. There might also be radical environmentalists who aren't motivated by a death wish for humanity, but by a destruction wish for civilization. Ted Kaczynski, also known as the Unabomber, provides a compelling example. His objective was neither suicide nor omnicide, but the dismantlement of technological civilization, according to the slogan "Back to the Pleistocene"  (Flannery 2016) . Although Kaczynski's own terrorist bombings weren't intended to achieve his ambitious aims, if a doomsday button had been available at the time, Kaczynski probably would have pushed it, thereby initiating a stagnation catastrophe. While people might die in the process, this wouldn't be the goal. As Kaczynski declares, "We therefore advocate a revolution against the industrial system. This revolution may or may not make use of violence; it may be sudden or it may be a relatively gradual process spanning a few decades. … Its object will be to overthrow not governments but the economic and technological basis of the present society" (quoted in Flannery 2016). This type of agential risk should be carefully studied moving forward, for reasons explicated in Section 5. If advanced technologies become sufficiently powerful and accessible, people under the spell of the deep green religion could inflict unprecedented harm on civilization. (4) Religious terrorists. Terrorists motivated by nationalist, separatist, anarchist, Marxist, and other political ideologies are unlikely to cause an existential catastrophe because their goals are typically predicated on the continued existence of civilization and our species. They want to change the world, not destroy it. But this is not the case for some terrorists motivated by religious ideologies. For them, what matters isn't this life, but the afterlife; the ultimate goal isn't worldly, but otherworldly. These unique features make religious terrorism especially dangerous, and indeed it has proven to be both more lethal and indiscriminate than past forms of "secular" terrorism.  8  According to the Global Terrorism Index, religious extremism is now the primary driver behind global terrorism, and there are reasons (see Section 5) for expecting this to remain the case moving forward  (Arnett 2014) . The most worrisome form of religious terrorism is apocalyptic terrorism. As Jessica Stern and J.M. Berger observe, apocalyptic groups aren't "inhibited by the possibility of offending their political constituents because they see themselves as participating in the ultimate battle." Consequently, they are "the most likely terrorist groups to engage in acts of barbarism"  (Stern and Berger 2015) . The apocalyptic terrorist sees humanity as being engaged in a cosmic struggle at the very culmination of world history, and the only acceptable outcome is the complete decimation of God's enemies. These convictions, when sincerely held, can produce a grandiose sense of moral urgency that apocalyptic warriors can use to justify virtually any act of cruelty and violence, no matter how catastrophic. To borrow a phrase from the former Director of the CIA, James Woolsey, groups of this sort "don't want a seat at the table, they want to destroy the table and everyone sitting at it"  (Lemann 2001) . There are two general types of active apocalyptic groups. First, there are movements that have advocated something along the lines of omnicide. History provides many striking examples of movements that maintained -with the unshakable firmness of faith -that the world must be destroyed in order to be saved. For example, the Islamic State of Iraq and Syria believes that its current caliph, or leader, is the eighth of twelve caliphs in total before the apocalypse. This group's adherents anticipate an imminent battle between themselves and the "Roman" forces (the West) in the small northern town of Dabiq, in Syria. After the Romans are brutally defeated, one-third of the victorious Muslim army will supernaturally conquer Constantinople (now Istanbul), after which the Antichrist will appear, Jesus will descend above the Umayyad Mosque in Damascus, and various other eschatological events will occur. In the end, those who reject Islam will be judged by God and cast into hellfire, and the Islamic State sees itself as playing an integral role in getting this process started (Torres 2016a). Another example comes from the now-defunct Japanese cult Aum Shinrikyo. This group's ideology was a syncretism of Buddhist, Hindu, and Christian beliefs. From Christianity, the group imported the notion of Armageddon, which it believed would constitute a Third World War whose consequences would be "unparalleled in human history." Only those "with great karma" and "those who had the defensive protection of the Aum Shinrikyo organization" would survive  (Juergensmeyer 2003) . In 1995, Aum Shinrikyo attempted to knock over the first domino of the apocalypse by releasing the chemical sarin in the Tokyo subway, resulting in 12 deaths and sickening "up to 5,000 people." This was the biggest terrorist attack in Japanese history, and it was perpetrated by a religious cult that was explicitly motivated by an active apocalyptic worldview. Other contemporary examples include the Eastern Lightning in modern-day China, which believes that it's in an apocalyptic struggle with the communist government, and the Christian Identity movement in the US, which believes that it must use catastrophic violence to purify the world before the return of Jesus. Second, there are multiple groups that have advocated mass suicide. The Heaven's Gate cult provides an example. This group is classified as a millenarian UFO religion, led by Marshall Applewhite and Bonnie Nettles. They believed that, as James Lewis puts it, ancient "aliens planted the seeds of current humanity millions of years ago, and have to come to reap the harvest of their work in the form of spiritual evolved individuals who will join the ranks of flying saucer crews. Only a select few members of humanity will be chosen to advance to this transhuman state"  (Lewis 2001) . The world was about to be "recycled," and the only possible "way to evacuate this Earth" was to leave their bodies behind through collective suicide. Members believed that, once dead, they would board an alien spacecraft that was trailing the Hale-Bopp comet as it swung past Earth in 1997. To fulfill this eschatological prediction, they drank phenobarbital, along with applesauce and vodka. Between March 24-26, 39 members of the cult committed suicide. Other examples could be adduced, such as The Movement for the Restoration of the Ten Commandments of God in Uganda, which slaughtered 778 people after unrest among members following a failed apocalyptic prophesy  (New York Times 2000) . But the point should be sufficiently clear. With respect to extinction risks, there are (quite intriguingly) no notable groups that have combined these two tendencies of suicide and omnicide. No major sect has said, "We must destroy the world, including ourselves, to save humanity." But this doesn't mean that such a group is unlikely to emerge in the future. The ingredients necessary for a truly omnicidal ideology to take shape are already present in our culture. Perhaps, for reasons discussed below, societal conditions in the future will push religious fanatics to even more extreme forms of apocalypticism, thereby yielding a group that believes God's will is for everyone to perish. Whether this happens or not, apocalyptic groups also pose a significant stagnation risk. For example, what if Aum Shinrikyo had somehow been successful in initiating an Armageddon-like Third World War? What might civilization look like after such a catastrophe? Could it recover? Or, what if the Islamic State managed to expand its caliphate across the entire world? How might this affect humanity's long-term prospects? Zooming out from our focus on apocalyptic groups, there are numerous less radical groups that would like to reorganize society in existentially catastrophic ways. One of the ultimate goals of al-Qaeda, for example, is to implement Sharia law around the world. If this were to happen, it would destroy the modern secular values of democracy, freedom of speech and the press, and open scientific inquiry. The imposition of Sharia law on civilization is also the aim of non-jihadist Islamists, who comprise roughly 7 per cent of the Muslim community  (Flannery 2014 ). Similarly, "dominionist" Christians in the US, a demographic that isn't classified as "terrorist," believe that God commands Christians to control society and govern it based on biblical law. If a state run by dominionists were to become sufficiently powerful and global in scope, it could induce an existential catastrophe of the stagnation variety. (5) Rogue states. As with political terrorists, states are unlikely to intentionally cause an extinction catastrophe because they are generally not suicidal. Insofar as they pursue violence, it's typically to defend or expand their territories. The total annihilation of Homo sapiens would interfere with these ends. But defending and expanding a state's territories could cause a catastrophe of the stagnation variety. For example, if North Korea were to morph into a one-world government with absolutist control over the global population until Earth became unlivable, the result would be an existential catastrophe. Alternatively, a benevolent one-world government could emerge from institutions like the United Nations or the European Union. Once in place, a malevolent demagogue could climb to the power ladder and seize control over the system, converting it into a tyrannical dictatorship. Again, the outcome would be a stagnation catastrophe. Of all the agential risk types here discussed, historians, sociologists, philosophers, and other scholars have studied state-level polities and governmental systems the most thoroughly.
 Agential error The discussion to this point has focused on agential terror. But what about the other side of the error/terror coin? The danger posed by agential error depends in part on how accessible future technologies become. For example, if even a small percentage of the human population in 2050, which is projected to be 9.3 billion, were to acquire "biohacker" laboratories, the chance that someone might accidentally re-lease a pathogen into the environment could be unacceptably high  (Pew Research Center 2015a) . After all, a significant number of mistakes have happened in highly regulated government laboratories over the years  (Torres 2016a) . The 2009 swine flu pandemic may have occurred because of a laboratory mistake made in the 1970s, and "a CDC lab accidentally contaminated a relatively benign flu sample with a dangerous H5N1 bird flu strain that has killed 386 people since 2003"  (Zimmer and Burke 2009; McNeil 2014) . If such problems occur among professionals, imagine the potential dangers of hobbyists around the world -perhaps hundreds of millions -handling pathogenic microbes with almost no regulatory oversight. The exact same logic applies to technologies that are becoming more accessible, such as nanotechnology, robotics, AI systems, and possibly nuclear weapons, not to mention future artifacts that currently lie hidden beneath the horizon of our technological imaginations. There could also be malicious agents that want to cause an existential catastrophe, but nonetheless end up doing this by accident rather than design. For example, in preparing for the "big day," a doomsday cult could accidentally release a deadly pathogen or self-replicating nanobot into the environment, resulting in an unplanned disaster. This scenario could involve ecoterrorists, idiosyncratic actors, and states as well. Or an agent with no desire to cause an existential catastrophe could push a "catastrophe button" that inadvertently brings about an existential disaster. For example, a rogue state that hopes to gain regional or global power through the use of nuclear missiles must answer the following question: exactly how many nuclear missiles are required to bring the world's governments to their knees without causing an extinction-inducing nuclear winter? The same question could be asked with respect to biological weapons, nanoweapons, and weaponized AI systems. How sure can one be that there won't be unintended consequences that catapult humanity back into the Stone Age? (As Albert Einstein once said, "I do not know how the Third World War will be fought, but I can tell you what they will use in the Fourth -rocks!" Calaprice 2005.) The unpredictability and uncertainty inherent in global catastrophe scenarios could make it easy for non-existential terror to slide into existential error. Finally, a special case of agential error worth examining on its own involves superintelligence. A genuinely superintelligent agent (coupled with advanced technologies) would wield extraordinary power in the world. This fact would make humanity especially vulnerable to any error made by such an agent. Even a single mistake could be sufficiently devastating to cause an existential catastrophe. One might respond by asserting that a superintelligence is surely less likely to make a mistake, given its superior intelligence  (Bostrom 2014) . But I would challenge this assumption. Consider that humans have the most developed neocortex and the highest encephalization quotient in the Animal Kingdom. Yet it is our species, rather than our intellectually "inferior" relatives, that is responsible for the environmental catastrophes of climate change and biodiversity loss. Even more, our species has greatly increased the total number of existential risks from a small handful of improbable natural threats to a dizzying array of anthropogenic and agent-tool dangers. Was this a mistake? In a sense, yes: we certainly didn't intend for this to happen. Historically speaking, human ingenuity and the threat of existential annihilation have risen together. This suggests that there isn't a strong connection between higher intelligence and the capacity to avoid errors. Call this the "orthogonality thesis of fallibility"  (Torres 2016a ). If our own history is a guide to the future, we might expect the creation of a superintelligence to further increase the total number of existential risks, perhaps in ways that are either now, or permanently, inscrutable to us. The point is that even if we were to solve the "control problem" and create a friendly superintelligence, it could nudge us over the precipice of disaster on accident, rather than push us on purpose. What can we say? It's only superhuman.
 The future of agential risks Neutralizing the threats posed by agential risks requires understanding not only their synchronic properties, but also how these properties might evolve diachronically. There are two sets of factors relevant to this task, which we can organize into external and internal categories, depending on whether they originate from outside or within an agent's motivating ideology.
 External factors As previously mentioned, climate change and biodiversity loss are "context risks" that will frame, and therefore modulate, virtually every other threat facing humanity. According to our best current science, these phenomena -appropriately dubbed "threat multipliers" -will become more severe in the coming decades, and their effects will "extend longer than the entire history of human civilization"  (Clark et al. 2016 ). This will significantly elevate the probability of future struggles and conflicts between state and nonstate actors. A simple thought experiment illustrates the point. In which of the following two worlds are wars more likely: one beset by megadroughts, extreme weather, scorching heat waves, desertification, sea-level rise, and the spread of infectious disease, or one without these tragedies? In which of the following two worlds are terrorist attacks more likely: a world in which food supply disruptions, mass migrations, social upheaval, economic collapse, and political instability are widespread, or one in which they're not? One could even ask, in which of the following two worlds is a malevolent superintelligence more likely to emerge: one crushed by environmental catastrophes or one in which civilization is functioning properly? Environmental degradation could also increase the likelihood of incidents involving idiosyncratic agents, in part because it could increase the prevalence of "bullying"-type behavior. When people are desperate, moral considerations tend to be occluded by the instinctual drive to meet our biological needs. Even more, climate change and biodiversity loss could significantly fuel ecoterrorism. To quote Flannery, "As the environmental situation becomes more dire, eco-terrorism will likely become a more serious threat in the future"  (Flannery 2016) . Not only will the deleterious effects of industrial society on the natural world become more salient, but sudden changes in the environment's stability could prod activists to consider more aggressive, even violent, tactics. Scientists have, for example, argued that Earth could be approaching an abrupt, irreversible, catastrophic collapse of the global ecosystem. A planetary-scale "state shift" of this sort could unfold on the timescale of decades and cause "substantial losses of ecosystem services required to sustain the human population." The result would be "widespread social unrest, economic instability, and loss of human life," and these phenomena could inspire fierce rebellions against civilization. There are also reasons for expecting climate change and biodiversity loss to nontrivially increase the size and frequency of apocalyptic movements in the future  (Torres 2016b; Juergensmeyer, forthcoming) . In fact, we already have at least one example of this happening, according to a 2015 study published in the Proceedings of the National Academy of Sciences. This study argues that one can draw a straight line of causation from anthropogenic climate change to the record-breaking 2007-2010 Syrian drought to the 2011 Syrian civil war  (Kelly et al. 2015) . And the Syrian civil war was the Petri dish in which the Islamic State consolidated its forces to become the wealthiest and most powerful terrorist organization in human history. The link between environmental havoc and terrorism has also been confirmed by the current Director of the CIA, John Brennan, the former US Defense Secretary, Chuck Hagel, and the US Department of Defense  (Torres 2016c) . As Mark Juergensmeyer (forthcoming) observes in detail, apocalyptic ideologies tend to arise during periods of extreme societal stress. When a group's basic identity and dignity is threatened, when losing one's cultural identity is unthinkable to those in the group, and when the crisis isn't solvable through ordinary human means, people often turn to supernatural frameworks to make sense of their suffering and give them hope for the future. In Juergensmeyer's words, "The presence of any of these three characteristics increases the likelihood that a real-world crisis may be conceived in cosmic terms," and "cosmic terms" form the language of apocalyptic activism. Because of climate change and biodiversity loss, these are precisely the conditions we can expect in the future, as societies inch toward the brink of collapse. It's also worth noting that floods, earthquakes, droughts, famines, and disease are prophesied by many religions as harbingers of the end. Consequently, environmental degradation could actually reinforce people's prior eschatological convictions, or even lead nonbelievers to convert. 9 There is, in fact, a strong preexisting base of widespread apocalyptic belief within the Abrahamic traditions. For example, a 2010 Pew poll finds that 41 per cent of Americans believe that Jesus will either or "probably" return by 2050 (Pew Research Center 2010), and a 2012 Pew poll reports that 83 per cent of people in Afghanistan, 72 per cent in Iraq, 68 per cent in Turkey, and 67 per cent in Tunisia believe that the Mahdi, Islam's end-of-days messianic figure, will return in their lifetime  (Pew Research Center 2012) . One should expect these percentages to rise moving forward. There are two additional reasons for anticipating more apocalyptic movements in the future. First, a statistical point. According to a 2015 Pew study, the percentage of nonbelievers is projected to shrink in the coming decades, despite the ongoing secularization of Europe and North America (Pew Research Center 2015a; see chapter 3, "Articles of Faith"). By 2050, more than 60 per cent of humanity will identify as either Christian or Muslim, in roughly equal proportion. As Alan Cooperman puts it, "You might think of this in shorthand as the secularizing West versus the rapidly growing rest"  (Pew Research Center 2015b) . This is disconcerting because religion normalizes bad epistemological habits, and thinking clearly about big-picture issues is the only hope our species has of navigating the wilderness of existential risks before us. In addition, not only is superstition rising as advanced technologies become more powerful, but if the relative proportion of extremists at the fringe remains fixed, the absolute number of religious fanatics will undergo a growth spurt. This alone suggests that the future will contain a historically anomalous number of terrorists (although we should note that it will contain more "good guys" as well). Furthermore, the inchoate GNR (genetics, nanotech, and robotics) Revolution will result in a wide range of fundamental changes to society. It could introduce new forms of government -or, as Benjamin  Wittes and Gabriella Blum (2015)  argue, undercut the social contract upon which modern states are foundedand even challenge our notion of what it means to be human. These changes could be profound, pervasive, and quite rapid, given the exponential rate of innovation. If this is the case, it could also fulfill the conditions specified by Juergensmeyer, thereby fueling apocalyptic extremists to declare an imminent end to the world. (In a sense, this might be true, since the transition from the human era to a posthuman era would mark a watershed moment in our evolutionary history.) The fact is that past technological revolutions have inspired religious fanaticism, and by nearly all accounts the GNR Revolution will be far more disruptive than any previous revolution. As Juergensmeyer puts it, "radical change breeds radical religion," and radical change is exactly what we should expect.  10  Tying this all together: a confluence of environmental degradation, demographic shifts, and disruptive technologies could significantly exacerbate the threat of apocalyptic terrorism, as well as idiosyncratic agents and ecoterrorists, in the future. The recent unrest in the Middle East is, arguably, only a preview of what's to come.
 Internal factors But there are also factors internal to the ideologies espoused by different agents that are no less important for existential riskologists to study. For example, the year 2076 will likely see a spike in apocalyptic fervor within the Islamic world  (Cook 2008 (Cook , 2011 . One can only know this, and therefore prepare appropriately, if one understands the relevant Islamic traditions. The reason 2076 will be especially dangerous is that it roughly corresponds to 1500 in the Islamic calendar (AH), and eschatological enthusiasm has risen in the past at the turn of the century. Consider the fact that the Iranian Revolution, which was widely seen as an "apocalyptic occurrence" by Shi'ites, happened in 1979  (Cook 2011) . So did the Grand Mosque seizure, during which a group of 500 insurgents took approximately 100,000 worshipers hostage. This group claimed to have the Mahdi with them and believed that the Last Hour was imminent. The point is that 1979 corresponds to 1400AH, a date that fueled the apocalypticism behind these events. Scholars should also keep an eye on 2039, since it is the 1200th anniversary of the Mahdi's occultation in the Twelver Shia tradition. As Cook writes, "the 1000-year anniversary of the Mahdi's occultation was a time of enormous messianic disturbance that ultimately led to the emergence of the Bahai faith. … [A]nd given the importance of the holy number 12 in Shiism, the twelfth century after the occultation could also become a locus of messianic aspirations." He adds: In one scenario, either a messianic claimant could appear or, more likely, one or several movements hoping to "purify" the Muslim world (or the entire world) in preparation for the Mahdi's imminent revelation could develop. Such movements would likely be quite violent; if they took control of a state, they could conceivably ignite a regional conflict.  (Cook 2011)  Looking forward, who knows what powerful technologies might exist by 2039 or, even more, 2076? If a messianic movement with violent proclivities were to arise in the future, it could have existential implications for humanity. Another example involves apocalyptic US militias influenced by Christian Identity teachings. On April 19, 1995, Timothy McVeigh pulled up to the Alfred P. Murrah Federal Building in Oklahoma City and detonated a bomb that killed 168 people. As Flannery notes, this event unfolded "just as the Christian Identity affiliated Covenant, Arm, and the Sword (CSA) militia had planned a decade earlier while training 1,200 recruits in the Endtime Overcomer Survival Training School." The date of April 19 "was no accident." Exactly two years earlier, the government ended its confrontation with the Branch Davidians in their Waco, Texas compound, resulting in 74 deaths. And exactly 8 years before this event, there was a similar standoff between the government and the Covenant, Arm, and the Sword. And centuries before, in 1775, the Battles of Lexington and Concord that inaugurated the American Revolutionary War against Great Britain took place on April 19. Consequently, the date of "April 19 has come to resonate throughout a constructed history of the radical Right as a day of patriotic resistance"  (Flannery 2016, 144) . More generally, some experts refer to April as the beginning of "the killing season." While Harris and Klebold reportedly planned their massacre on April 19 (being inspired by McVeigh), they ended up delaying it one day to coincide with Adolf Hitler's birthday  (Rosenwald 2016) . Another date to watch is April 15, the deadline for income tax filings in the United States. This has meaning to certain antigovernment groups. As the Anti-Defamation League (2005) warns, April is a month that looms large in the calendar of many extremists in the United States, from racists and anti-Semites to anti-government groups. Some groups organize events to commemorate these April dates. Moreover, there is always a certain threat that one or more extremists may choose to respond to these anniversaries with some sort of violent act. It adds: "Because of these anniversaries, law enforcement officers, community leaders and school officials should be vigilant." Existential risk scholars too should be especially mindful of heightened risks in April. If a doomsday or catastrophe button were to become available to Christian Identity terrorists motivated by an active apocalyptic ideology, April 19 might be the day on which they would decide to press it.
 Conclusion Most states and terrorists are unlikely to intentionally cause an existential catastrophe, although certain high-impact instances of catastrophic violence could accidentally realize an extinction or stagnation risk. The primary danger posed by states and terrorists concerns their capacity to press a catastrophe button, if it were to become available. There are, however, at least five types of agents who could be motivated by various goals to bring about a cataclysm of existential proportions. I do not intend for this list to be exhaustive. Indeed, the agential threat horizon could expand, shift, or transmogrify in unanticipated ways as humanity is thrust forward by the invisible hand of time. It's nonetheless important to specify a typology of agential risks based on the best current research -a task that no one has yet attempted -because the agents of each category have their own unique properties, and must therefore be studied as unique threats in exactly the same way that nuclear weapons, biotechnology, and molecular manufacturing are studied separately. While much of the existential risk literature focuses on the various tools, both present and anticipated, that could bring about a secular apocalypse, we must give the agents equal consideration. A key idea of this paper is that, first, advanced technologies will provide malicious agents with bulldozers, rather than shovels, to dig mass graves for their enemies. And second, the risk potential of these technologies cannot be realized without a complete agent-tool coupling. This is why the field of existential risk studies desperately needs a subfield of agential riskology, which this paper aims to establish. No doubt much of what I have said above will need to be refined, but such is to be expected when there are few shoulders upon which to stand. Notes 1. See Leslie 1986. 2. For example, a world thrown into chaos by environmental degradation might be less prepared to deflect an incoming asteroid or coordinate on stopping a global pandemic. 3. Originally, "dual-use" referred to entities with both civilian and military uses, but the term has acquired a more promiscuous signification in recent scholarship. 4. With the exception of superintelligence, discussed below. 5. According to one study, over 66 per cent of premeditated school shootings have been shown to be connected to bullying  (Boodman 2006 ). 6. Although luck might be partly to blame, as Heemeyer fired his rifle at propane tanks that could have killed someone in the vicinity if they had exploded 7. As I've written elsewhere about similar examples: these may seem too anecdotal to be scientifically useful. But drawing this conclusion would be wrong. Given the immense power of anticipated future technologies, single individuals or groups could potentially wield sufficient power to destroy the world. The statistically anomalous cases of omnicidal lone wolves or terrorist groups are precisely the ones we should be worried about, and therefore should study. 8. See, for example, Hoffman 1993. 9. History provides numerous examples of natural disasters leading to a spike in religious belief, such as the Plague of Cyprian.

Abstract: The greatest existential threats to humanity stem from increasingly powerful advanced technologies. Yet the "risk potential" of such tools can only be realized when coupled with a suitable agent who, through error or terror, could use the tool to bring about an existential catastrophe. While the existential risk literature has provided many accounts of how advanced technologies might be misused and abused to cause unprecedented harm, no scholar has yet explored the other half of the agent-tool coupling, namely the agent. This paper aims to correct this failure by offering a comprehensive overview of what we could call "agential riskology." Only by studying the unique properties of different agential risk types can one acquire an accurate picture of the existential danger before us.


Challenges of Aligning Artificial Intelligence with Human Values

Introduction The rapid development of artificial intelligence (AI) has enabled a wide range of beneficial applications in various contexts, from healthcare to security and governance. Fields or abilities formerly thought to be exclusively "human" are now being tackled by a machine-learning approach  (Arnold et al., 2017, p. 81) . As AI becomes more powerful, more attention is being paid to ways of maximising its societal benefit. The growing capabilities of AI and machine learning and the development of countless new applications also pose questions of risk and trust  (Sutrop, 2019) . AI could be a threat to humanity either through its malicious use by humans  (Brundage et al., 2018) , unintended consequences  (Amodei et al., 2016) , or autonomous acts by the artificial system  (Farquhar et al., 2017) . A joint report, The Malicious Use of  AI: Forecasting, Prevention, and Mitigation (2018) , issued by the leading AI research centres in the UK and the US  (Brundage et al., 2018)  has recently analysed the landscape of potential security threats from malicious uses of AI technologies and explicitly stated that if adequate defences are not developed soon, AI will threaten our digital, physical and political security. Currently, AI systems are narrowly dedicated to specific tasks such as speech, face or object recognition, spam filters or autonomous driving, and they are not capable of setting their own goals or choosing the best courses of action across domains. Researchers predict that AI will outperform humans in translating languages by 2024, writing high-school essays by 2026, driving a truck by 2027, or working as a surgeon by 2053; they believe there is a 50 per cent chance of AI outperforming humans in all tasks within 45 years and of automating all human jobs in 120 years  (Grace et al., 2018) . As robots become increasingly autonomous and able to make decisions on their own, the risk increases that humans will lose control over AI systems. Wendell  Wallach and Colin Allen (2009)  have suggested that as robots take on more and more responsibility, they must be programmed with moral decision-making abilities, for the sake of our own safety. What is AI? AI has been described in various ways. It can refer to certain humandesigned systems, or to a scientific discipline that encompasses several approaches and techniques, such as machine learning, machine reasoning, and robotics. In this paper, I will discuss AI systems according to the definition of the European Commission High Level Expert Group on Artificial Intelligence as follows: AI systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information derived from this data and deciding the best action(s) to take to achieve the given goal. AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behaviour by analysing how the environment is affected by their previous actions.  (European Commission, 2019a)  Visionary scenarios consider the possibility that intelligence equal to our own, or even exceeding it, will already be created within the current century. An AI equipped with human-level or even greater ability, which is able to find a solution when presented with an unfamiliar task, is referred to as Artificial General Intelligence (AGI). Indeed, freed from biological constraints such as limited memory and slow biochemical processing speeds, machines may eventually become more intelligent than we are-with profound implications for us all. It has been hypothesised that if humans can create AGI at a human level of intelligence, such an innovation could give rise to higher and higher intelligence; in combination with recursive self-improvement of the AGI, this would eventually lead to Superintelligence (SAI)  (Kurzweil, 2005; Tegmark, 2017) . SAI is "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest"  (Bostrom, 2014, p. 26) . According to a high-level survey of expert opinion, AI specialists estimate that systems will move to superintelligence by the end of this century, and that "the chance is about one in three that this development turns out to be 'bad' or 'extremely bad' for humanity"  (Müller & Bostrom, 2016) . Even if we are still far from humanlevel AI and AI advancements may be overhyped, several research programs have been launched on risk assessment of AI systems, attempting to mitigate threats of rapid and uncontrollable AI development. The AI developers themselves worry about how to ensure that an AI does not escape its confines and cause damaging effects  (Yudkowsky, 2004) . Computer scientists warn, "[a]s autonomous systems become more prevalent in society, it becomes increasingly important that they robustly behave as intended"  (Russell et al., 2016) . Ethicists stress, "[w]hen independent of the humans that created them, their true 'intelligence' is tested in terms of their status as 'moral beings'"  (Iphofen & Kritikos, 2019) . Since the potential threats are great, policymakers have emphasised the need to develop ethically-aligned AI which will respect our ethical principles and values. A good overview of various guidelines for ethical, rights-respecting and beneficial AI has been provided in the recent report 'Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-based Approaches to Principles for The research program of designing AI that conforms to human values is called 'value alignment'. AI researchers explain this as follows: "As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value alignment problem"  (Fisac et al., 2018, p. 1) . 'Value alignment' is defined as a property of an intelligent agent that allows it only to pursue goals and activities which are beneficial to humans  (Soares & Fallenstein, 2014; Soares, 2015; Russell et al., 2016; Arnold et al., 2017; Russell, 2019) . Identifying the purposes we really desire remains an open question. The opposite of 'value alignment', 'value misalignment' can be explored from two perspectives, either by focusing on the behavior of individual tech artefacts (individual level) or on the functioning of sociotechnical systems (system-level)  (Osoba et al., 2020, p. 332) . The AI 'value alignment' problem faces two interrelated kinds of challengestechnical and normative  (Gabriel, 2020) . The technical challenge deals with the question of "how to encode human values in artificial intelligence." The normative challenge is associated with two questions: "which values or whose values should artificial intelligence align with?" My concern is that AI developers and computer scientists underestimate the difficulty of answering the normative question. They hope that if we find out "what we want, what we really, really want"  (Russell, 2015) , we can create a beneficial AI which is "humble, altruistic, and committed to pursue our objectives, not theirs". But how are we to decide which objectives or values to induce in AI, given that there is a plurality of values and moral principles and that our everyday life is full of moral disagreements? In the first part of my paper I will demonstrate how technical and normative challenges of value alignment are interrelated. I will provide a brief overview of the ways in which AI researchers have tried to solve the 'value alignment problem'. I will explain why it is problematic that in value alignment programs the word 'value' is used interchangeably with goals or preferences. In the second part of my paper I will focus on normative challenges of value alignment. Several research papers, AI guidelines and manuals discuss the need to take into account cultural differences, while also respecting universal principles. I will suggest how this apparent inconsistency can be removed. My thesis is that although it is not realistic to reach an agreement on what we humans really want, given that people value different things and seek different ends, it may be possible to agree on what we do not want to happen, considering the possibility that intelligence can be created that is equal to our own, or even exceeding it. I agree with Isaiah Berlin's view as expressed in 'The Pursuit of the Ideal' that even if our ends and moral principles are plural, they are not infinitely many, since they have to remain "within the human horizon"  (Berlin, 2013 (Berlin, [1947 ). I will argue for pluralism (and not for relativism!) which is compatible with objectivism. Despite the fact that there is no uniquely best solution to every moral problem, it is still possible to identify which answers are wrong. And this is where we should begin the value alignment of AI.
 A technical challenge: how to align AI with values? The answer to the question how to align AI with values depends, on the one hand, on the technical possibilities and, on the other hand, on what we want AI to align with. Theories of value alignment vary on what they mean by 'value'. In the context of value alignment, Iason  Gabriel (2020)  notes that the notion of 'value' can serve as a place-holder for many things: AI could be designed to align with expressed or intended instructions, revealed preferences, informed preferences or desires, interest or well-being, or moral values as defined by the individual or society. In view of these different approaches, value alignment turns out to be an umbrella for various attempts to ensure that general artificial intelligence will remain beneficial to humans. Through what means do AI researchers and designers hope to reach value alignment? There seem to be two basic-albeit opposite-approaches, either feeding the artificial agent the right principles or applying (cooperative inverse) reinforcement learning. Both approaches-feeding the artificial intelligent with principles and reinforcement learning-have their proponents and critics. With progress in the area of machine learning, the idea of training a system on data (either supervised or unsupervised) has drawn increasing attention. Thus there seems to be a turn away from traditional machine ethics with its topdown articulation of rules, norms or models of behaviour and toward learning values either through imitation or by identifying the preferences of humans. Nevertheless, some critics doubt that beneficial AI can be achieved in this way, arguing that AI can be truly beneficial only if it can become a moral agent which acts on ethical principles or has moral virtues. Iason  Gabriel (2020)  has recently argued for a principle-based approach. His view is that instead of looking for "true" moral principles for AI, the real task is "to identify fair principles for alignment that receive reflective endorsement despite widespread variation in people's moral beliefs"  (Gabriel, 2020) . The idea behind reinforcement learning is that desirability of any state sequence can be expressed as a sum of immediate rewards associated with each state in the sequence. The objectives that the reinforcement learning agent is supposed to follow can be defined by specifying reward function. In the case of inverse reinforcement learning (IRL), an AI system infers the underlying utility function of an agent by observing its behavior.  Hadfield-Menell et al. (2016)  argue that the value alignment problem can only be solved through cooperative inverse reinforcement learning (CIRL) which they describe as "a cooperative, partialinformation game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is." Mark O.  Riedl and Brent Harrison (2016)  have argued that given potentially infinite undesirable outcomes in an open world, if values cannot be easily enumerated by human programmers, they can be learned by reinforcement. They hypothesise that an artificial intelligence that can read and understand stories can learn the values tacitly held by the culture from which the stories originate.  Riedl and Harrison (2016)  describe their preliminary work on using stories to generate a value-aligned reward signal for reinforcement learning agents that preempts psychotic-appearing behavior. They argue, Value alignment in a reinforcement learning agent theoretically can be achieved by providing the agent with a reward signal that encourages it to solve a given problem and discourages it from performing any actions that would be considered harmful to humans. A value-aligned reward signal will reward the agent for doing what a human would do in the same situations when following social and cultural norms (for some set of values in a given society and culture) and penalise the agent if it performs actions otherwise. A reinforcement learning agent will learn that it cannot maximise reward over time unless it conforms to the norms of the culture that produced the reward signal.  (Riedl & Harrison, 2016)  How can a value-aligned reward signal be learned?  Riedl and Harrison's (2016)  answer is that, "[a] value-aligned reward signal is produced from the crowdsourced stories in a two-stage process". When humans tell stories to other humans, they skip over many details and events. By crowdsourcing the narratives, one can get better coverage of all steps involved in a situation and extract the most reliable pattern of events. Riedl and Harrison describe the reinforcement learning process as follows: in the first step, the plot graph learning process "aligns" the crowdsourced example narratives to extract the most reliable patterns of events; in the second step, the plot graph is translated into a "trajectory tree" which is used to produce a reward signal. If the reinforcement learning agent performs an action that is a successor of the current node in the trajectory tree, it receives a reward; if it fails, it receives a small punishment. The aim is to make the reinforcement learning agent solve some problem in the most human-like fashion. The problem is that in many stories humans violate social and cultural norms. The authors' hope is that the most typical human behaviour that emerges from crowdsourced stories is compliant to the norms. The proposal to achieve value alignment through reinforcement learning has been criticised by Thomas Arnold, Daniel Kasenberg and Matthias Scheutz (2017) as ethically inadequate and insufficient to guide artificial agents in decision making. They point out a number of problematic areas associated with reinforcement learning, including data bias, generalisation issues, and the adequacy of reward functions to represent temporally complex norms  (Arnold et al., 2017, p. 81) .  Arnold et al. (2017)  claim that even if there is agreement that top-down approaches of giving AI systems ethics would be intractable, IRL on its own cannot solve the problem and train such an agent to be moral. They point out that ethical behavior depends not only on the acts themselves, but on intentions, reasons, norms, and counterfactuals. The question IRL faces is what kind of model could truly learn ethical practices. Alternatively, value alignment of AI can take the form of imitation learning. In their article entitled 'The virtuous machine-old ethics for new technology' Nicolas  Berberich and Klaus Diepold (2018)  draw inspiration from Aristotle to demonstrate that virtue ethics can provide a solution to the value alignment problem. Instead of focusing on the formulation of duties and maxims (as deontologists do) or on the identification of desirable and maximisable consequences of actions (i.e., happiness), Berberich and Diepold argue that one should concentrate on moral actions performed by virtuous moral agents. "With respect to machines this changes the primary question from 'By what kind of algorithm can a machine choose the right action?' to 'How can we build a machine that, owing to its constitution, acts appropriately in arbitrary situations?'"  (Berberich & Diepold, 2018, p. 4 ). The article concludes by stating, 1. Virtue ethics fits nicely with modern artificial intelligence research and is a promising moral theory. 2. Taking the virtue ethics route to building moral machines allows for a much broader approach than simple decision-theoretic judgement of possible actions. Instead, it takes other cognitive functions into account like attention, emotions, learning and actions.  (Berberich & Diepold, 2018, p. 23)  Berberich and Diepold believe that since humans have for centuries learned virtues from imitating the behaviour of virtuous moral agents, virtue theory should be the guiding moral theory for building truly moral machines. They point out that since virtues are an integral part of one's character, the AI would not have the desire of changing its virtue of temperance  (Berberich & Diepold, 2018, pp. 23-24) . The difficulty in this argument is that virtue ethics has a hard time giving reasons for actions, and this means one cannot claim responsibility. The most influential approach to the problem of AI value alignment has been provided by Stuart Russell, whose ideas are summarised in his recent book, Human Compatible: AI and the Problem of Control (2019).  Russell's (2019)  central idea is that in order to maintain control, we have to design beneficial AI which is "humble, altruistic, and committed to pursue our objectives, not theirs". In order to guide AI researchers and developers in thinking about how to create beneficial AI systems, Stuart Russell formulates three principles: 1. The machine's only objective is to maximise the realization of human preferences. 2. The machine is initially uncertain about what those preferences are. 3. The ultimate source of information about human preferences is human behavior.  (Russell, 2019, pp. 172-173)  Russell believes that there is plenty of data about human actions (what has been written, filmed or observed) and about attitudes to these actions. In other words, rather than having a detailed ethical taxonomy programmed into them, AI systems should infer human values by observing and emulating our behavior. It is noteworthy that in his earlier writings and interviews, Russell has spoken about 'human values' which have now been replaced by the expression 'human preferences'. In an interview from 2015, Russell stated: To the extent that human values are shared, machines can and should share what they learn about human values. […] by assigning very broad priors over what human values might be, and by making the AI system risk-averse, it ought to be possible to induce exactly the behavior one would want: before taking any serious action affecting the world, the machines engage in an extended conversation with us and an extended exploration of our literature and history to find out what we want, what we really, really want.  (Russell, 2015)  In his 2019 book, Russell replaces the expression 'human values' with 'human preferences'. He also provides explanations as to why he has dropped the word 'value'. Russell points out that since the word 'value' is often understood as 'moral value', use of this term can be confusing. He does not want to give an impression of attempting to solve moral dilemmas like stops on a trolley route. In his view, such an approach is not worth considering, as in true dilemmas there are good arguments for both sides, and artificial agents cannot cause more harm than humans even if they take a wrong decision. Russell's concern is how to avoid the worst catastrophes and to ensure that we never lose control of machines more intelligent than we are. In addition, Russell insists that one should not equate value alignment with the wish to instill in machines a single ideal value system which may raise the question whose values should be encoded or who has been given the right to align AI with such values. He says that he uses the word 'value' as a technical term, "synonymous with utility, which measures the degree of desirability of anything from pizza to paradise"  (Russell, 2019, p. 178) . He also explains what he means by 'preferences': these cover everything one might care about. Furthermore, in order to be beneficial, the purely altruistic machine should aim to maximise the realisation of human preferences. Instead of "putting in values" or preferences, the machines "should learn to predict better, for each person, which life that person would prefer, all the while being aware that the predictions are highly uncertain and incomplete"  (Russell, 2019, p. 178 ). Russell believes that, in principle, an AI could learn billions of different predictive preference models. With respect to the issue of how the machine should trade off the preferences of multiple humans,  Russell (2019, p. 174 ) refers to the utilitarian principle of "the greatest happiness for the greatest numbers". The crucial question is how the machine might learn about human preferences. His answer is, again, very simple: by observing human choices which reveal information about human preferences. In Russell's opinion, the real complications arise because humans are not perfectly rational and the machine must take such an imperfection into account  (Russell, 2019, p. 177) . It turns out that, in essence, what Russell means by 'value' is in reality a 'preference'. However, the same problem crops up with the concept of 'preference' as it did with the concept of 'value'. Both have multiple meanings, and their use in different disciplines is highly variable. In its most general meaning, a value is that which is worthy, that which is worth having, getting or doing, or that which possesses some property or properties that make it so. Values are action-guiding; they are taken into account when making decisions and when planning activities. In the most general terms, preferences are evaluations; they concern matters of value. Preferences are subjective in that the evaluation is typically attributed to an agent. Russell uses the term as subjective comparative evaluation, of the form 'Agent A prefers X to Y'. What one prefers is not necessarily something objectively valuable, but rather something subjectively valuable from one's personal point of view. But this is where the problem begins. We can imagine the situation where the designer of a self-driving car asks potential customers whether they would prefer a car which will protect the persons riding in it or those on the street. Or, even worse, whether they would like a car which violates traffic rules, ignores traffic lights and brings the owner quickly and safely to the destination. Should such preferences be fulfilled by artificial intelligence? By leaving aside moral values and concentrating only on human preferences, it is difficult to understand how situations in which AI might fulfil the preferences of immoral agents can be avoided. As there are more non-democratic countries than democratic ones in the world, we should be careful not to allow AI to be programmed to do something devastating. Somewhere there will always be efforts underway to create an AI that will be destructive, for example, for military purposes. Some experts have questioned the use of robots in military combat, especially if such machines were to be given some degree of autonomous functions, e.g., being able to independently choose targets to attack with weapons. What is the solution? AI should be programmed not to fulfil immoral preferences and refuse to do immoral actions but they should be able to distinguish between moral and immoral actions. Thus, besides values we should also provide them with the capacity of moral deliberation to decide when a certain moral principle applies, and to be able to rank values in different contexts. It seems dangerous to let AI become an autonomous decision-maker without its becoming a moral agent. To be a moral agent entails far more criteria than adopting ethical values. Amitai  Etzioni and Oren Etzioni (2018)  note that to be a moral agent requires a certain set of attributes: (a) Self-consciousness. If the agent is not aware of itself in any given situation, and of the alternative courses that might be followed, then no moral decisions can be rendered. (b) The agent must be aware that she can affect the situation. (c) The agent must be able to understand the moral principles to be employed in arriving at a particular moral choice. (d) The agent must have a motive to act morally. This involves having passions, as otherwise moral preferences are merely intellectual preferences with nothing to fuel the moral action. Some scholars also add that a will or intention is required…  (Etzioni & Etzioni, 2018, pp. 239-240) . This argumentation indicates that if we want to avoid the scenario that AI can satisfy both moral and immoral preferences, we need to integrate ethics into AI, not only align it with values. The research programme of AI value alignment should ideally be interdisciplinary, besides computer scientists also involving philosophers, lawyers, economists, and cognitive scientists, psychologists and pedagogical scientists. Its success will depend on how much the AI designers will be able to integrate the knowledge provided by these disciplines in the design of AI. So far, discussions on value alignment have not actively included ethicists. As a result, there is a danger that AI ethics will undergo simplification, reduction or dogmatism. In several working groups which aim to secure beneficial AI, ethics has been boiled down to law or reduced to some specific principle (fairness, transparency, equality or accountability) without providing any reasons why one or another principle should be prioritised. The analysis of the policy documents on ethical AI  (Fjeld et al., 2020)  showed that the most common governance regime suggested for dealing with AI is international human rights law-64 per cent of all documents contained a reference to human rights. Annette  Zimmermann and Bendert Zevenbergen (2019)  have recently issued warnings concerning seven traps into which AI ethics can fall: reductionism, oversimplification, the relativism trap, value-alignment trap, dichotomy; the myopia trap; and the trap of the rule of law. For the purposes of this paper, it is especially interesting how the authors present the relativism trap and the valuealignment trap as two unacceptable options. The relativism trap is associated with the tendency to conclude that since pervasive moral disagreements exist, ethics is relative. Should we argue that relativism is wrong, we risk falling into the value-alignment trap according to which there must be one morally right answer. If we cannot reach an agreement, we will have failed. I am quite inclined to agree with the authors' view that AI ethics should welcome value pluralism without collapsing into extreme value relativism. We should accept that it may not always be possible to determine "the one right answer", thereby avoiding moral disagreement. However, it is often possible to clarify that at least some paths of action are clearly wrong and some paths of action are comparatively better. The normative challenge: which values should AI align with? In order to answer the normative question 'Which values should AI align with?' one must first answer the question of whether values are objective or subjective; universal or culturally relative. Are there absolute values or prima facie values? Several regulatory bodies that write guidelines about ethical AI stress that one has to be sensitive to cultural differences while at the same time respecting universal principles. Thus the Ethics Guidelines for Trustworthy AI (European Commission, 2019b) assert that "AI systems should respect the plurality of values and choices of individuals", thereafter claiming that "certain fundamental rights and principles, such as human dignity, are absolute and cannot be subject to a balancing exercise" (European Commission, 2019b, p. 13). In order to make it possible for European citizens to reap the benefits of AI, it needs to be "aligned with our foundational values of respect for human rights, democracy, and rule of law" (European Commission, 2019b, p. 4). IBM's manual entitled Everyday Ethics for Artificial Intelligence states: "Care is required to ensure sensitivity to a wide range of cultural norms and values. As daunting as it may seem to take value systems into account, the common core of universal principles is that they are a cooperative phenomenon." (IBM, 2019) Another important document, Ethically Aligned Design. A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, published under the auspices of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems poses the rhetorical question: "If machines engage in human communities as autonomous agents, then those agents will be expected to follow the community's social and moral norms. A necessary step in enabling machines to do so is to identify these norms. But whose norms?" (IEEE, 2019) Again, it seems that in spite of stressing the variety of social and moral norms, this document emphasises that values-based design of AI should respect universal ethical principles and internationally recognised human rights. Indeed, the document states that […] whether our ethical practices are Western (e.g., Aristotelian, Kantian), Eastern (e.g., Shinto, School of Mo, Confucian), African (e.g., Ubuntu), or from another tradition, honoring holistic definitions of societal prosperity is essential versus pursuing one-dimensional goals of increased productivity or gross domestic product (GDP). Autonomous and intelligent systems should prioritize and have as their goal the explicit honoring of our inalienable fundamental rights and dignity as well as the increase of human flourishing and environmental sustainability.  (IEEE, 2019, p. 1)  It seems to me that all these documents point in the same direction-namely, that we should accept plurality of values while at the same time prioritising certain specific values, and doing both of these things without necessarily providing explanations of how pluralism of values and objectivity of ethics are compatible. In the following I will try to explain how pluralism can accommodate both persistent moral disagreements and the objectivity of ethics. Pluralism differs from both relativism and subjectivism. Unlike the relativist, who believes that rightness depends on the culture or society, and unlike the subjectivist, who believes that rightness depends on the individual, the pluralist holds that in some cases the question of what is right lacks a determinate answer. Rightness is indeterminate, as there is irreducibility of values. As shown by Susan Wolf in the article 'Two levels of pluralism  ' (1992) , this irreducibility can happen on two levels of decision: on the level of values and on the level of moral systems. Wolf says that on the first level we constrain moral judgements within a moral system, i.e., we have to decide which value, for example, equality or liberty, is more important and we have to admit that there is no right answer to this question. On the second level, we constrain moral assessments of the systems themselves. Here we have to decide which moral system, with its different understanding of moral rightness, we should prefer (for example, Kantianism or utilitarianism or virtue ethics).  Wolf (1992)  shows that there is a possibility that after a careful assessment it may turn out that these value systems are all incommensurably good, and living according to any of these moral systems will lead to a morally good life. Also Isaiah Berlin supports pluralism which is compatible with objectivism. Pluralism means for him that "there are many different ends that men may seek and still be rational, fully men, capable of understanding each other"  (Berlin 2013 (Berlin [1947 , p. 11). In fact, we do have shared values which form bridges between people and make intercommunication between cultures in time and space even possible. "But our values are ours, and theirs are theirs"  (Berlin 2013 (Berlin [1947 , p. 11). We might criticise the values of other cultures, but we have to understand where they come from, what it means to live the way they do. Here Berlin emphasises people's empathy and capacity for imagination, their need to put themselves in the shoes of a representative of another era or culture. Pluralism in Berlin's view does not exclude a world of objective values or ends that men pursue for their own sake. Ends and moral principles are plural, but they are not infinitely many, since they have to remain "within the human horizon"  (Berlin 2013 (Berlin [1947 , p. 12). We should be able to distinguish between objectively good and bad life forms and condemn slavery, sexism, racism, or Nazism. Given that there are people who argue from an egoistic position of power, with the intention of preserving their preferred state or that some lack imagination and empathy for what it would mean to be in another's situation (in cases such as slavery, Nazism, female circumcision, racial or gender hatred, disparagement of minorities, abuse of animals), we need to have the opportunity to say that such things are unacceptable. This remains so even if we favor value pluralism. We can get a better idea of how pluralism can be compatible with objectivity by reading John Kekes's book The Morality of  Pluralism (1993) . Kekes makes a distinction between primary and secondary values. Primary values are the same for everyone, for they constitute the minimum requirements of good lives and are derivable from the facts of human nature. For example, we share the same physiological and psychological needs and capacities. Thus, the same things, goods pertaining to the self, intimacy, and social order are needed by everybody to live a good life. At the same time, the identity of secondary values depends on variable social and personal circumstances. For example, whereas killing is universally considered to be evil, what sort of killing counts as murder may be dependent on context. Kekes explains that secondary values are contingent on primary ones; either they are particular forms of expression of these values (for instance, the primary value of intimacy will take the form of certain sexual practices) or they are genuinely new goods, the realisation of which depends on the presence of primary values (for instance, in order to derive pleasure from creative participation in art or science, one's elementary needs for food and shelter need to be satisfied)  (Kekes, 1993, pp. 42-43) . Primary and secondary values can be either moral or non-moral. While moral values (both primary and secondary) affect others, nonmoral values (both primary and secondary) affect the agent or occur naturally. Primary values (both moral and non-moral) are universal requirements of all good lives, while secondary values count as benefits or harms because a tradition or a conception of a good life makes them so  (Kekes, 1993, p. 45) . Kekes explains that pluralists differ from relativists in holding primary values to be objectively justifiable by referring to our common nature. He admits that even monists must not reject plurality of values, provided there is some overriding value which allows authoritative ranking of the plural values. Kekes points out that there is a great variety of monist conceptions of what the overriding value or some principle of ranking values may be: the utilitarian ideal of the greatest happiness for the greatest number of people, the welfare economists' goal of preference satisfaction, the Kantian principle of the categorical imperative, the contractarian list of fundamental human rights, or something else  (Kekes, 1993, p. 47) . Thus, the main difference between monists and pluralists is that for the former there is one overriding value or principle, while for the latter values are manifold and contextual. Kekes explains that we usually aim to construct a conception of the good life out of the primary and secondary values that we ourselves favor and that our tradition supplies. Although he argues that for pluralists there is no authoritative way of settling the value conflict, there still seems to be at least one overriding principle for comparing and ranking different values. To my mind, by arguing that primary values originate in our common human nature, Kekes is also suggesting an authoritative overriding principle which helps to compare and rank values. For instance, he proposes that in value conflicts one should try to reduce secondary values to primary values (which have their source in human nature), by making it explicit that in spite of all differences, the fact that situatedness in various historically conditioned traditions and individual life-situations has led to differentiation in our conceptions of a good life, we continue to have similar human needs and capacities. Although Kekes himself has not recognised that his own appeal to common human nature is monistic in a similar way as the appeal to human rights framework, categorical imperative or general happiness principle, it is still possible that he ultimately speaks for pluralism and not monism. I think that as long as one accepts that there can be a plurality of ultimate values, all of which can be equally justified, one is a pluralist. Thus, indeed, pluralism really is compatible with objectivism.
 Conclusions I have tried to show that it is good that AI researchers have identified the importance of ensuring that AI will remain beneficial and that therefore AI should align with human values. However, since the term 'value' only functions as a placeholder and in reality one could limit oneself to making AI fulfil human preferences, the problem may arise that AI will not necessarily act morally, but potentially also satisfy immoral preferences. Thus, in order to secure ethical AI one should compel it to follow moral principles or obtain moral values. In the course of the argument the question emerged whether there are universal values or whether AI should reckon with cultural differences. I hope I have shown that value pluralism can accommodate both moral disagreements caused by diversity of values and the objectivity of ethics, accepting that there are no uniquely correct answers to moral questions. Moral ambiguity does not necessarily imply relativism. Accepting plurality does not mean that anything goes. We can identify wrong answers as long as we share our expectations of morality. Indeed, the aim of morality is to regulate our life in society, enabling us to live together peacefully and allowing human beings to satisfy their various needs: physiological, psychological, social, cultural and intellectual.  Thus, value alignment should begin with seeking agreement on negative valueswhat we do not want to lose and what we do not want to happen if it becomes possible to create artificial intelligence with intellectual capacities equal to or beyond our own. Our efforts to define what is immoral should keep in mind that morality's function is to satisfy the basic human needs which we all share.Value plurality and objectivity can be joined if we accept that there are many moral principles by which one can justify and rank moral values. Different moral theories can prefer different ultimate principles. One can live a moral life according to all of these. Nevertheless, there are fewer choices about what is morally wrong, and thus we should seek agreement on what is clearly immoral.

Abstract: As artificial intelligence (AI) systems are becoming increasingly autonomous and will soon be able to make decisions on their own about what to do, AI researchers have started to talk about the need to align AI with human values. The AI 'value alignment problem' faces two kinds of challenges-a technical and a normative one-which are interrelated. The technical challenge deals with the question of how to encode human values in artificial intelligence. The normative challenge is associated with two questions: "Which values or whose values should artificial intelligence align with?" My concern is that AI developers underestimate the difficulty of answering the normative question. They hope that we can easily identify the purposes we really desire and that they can focus on the design of those objectives. But how are we to decide which objectives or values to induce in AI, given that there is a plurality of values and moral principles and that our everyday life is full of moral disagreements? In my paper I will show that although it is not realistic to reach an agreement on what we, humans, really want as people value different things and seek different ends, it may be possible to agree on what we do not want to happen, considering the possibility that intelligence, equal to our own, or even exceeding it, can be created. I will argue for pluralism (and not for relativism!) which is compatible with objectivism. In spite of the fact that there is no uniquely best solution to every moral problem, it is still possible to identify which answers are wrong. And this is where we should begin the value alignment of AI.