A model of pathways to artificial superintelligence catastrophe for risk and decision analysis

Introduction An important issue in the study of future artificial intelligence (AI) technologies is the possibility of AI becoming an artificial superintelligence (ASI), meaning an AI significantly more intelligent than humans in all respects. Computers have long outperformed humans in specific domains, such as multiplication. As AI advances, humans are losing in more and more domains, as seen in the recent Jeopardy! victory of IBM's Watson computer. These AI can greatly help society, and also pose some risks, but they do not fundamentally change the status quo of humans being in ultimate control of the planet. However, some scholars propose that AIs could be designed that significantly exceed human capabilities, changing the status quo, with results that could be either fantastically good or catastrophically bad for the whole of humanity, depending on how the initial 'seed' AI is programmed  (Bostrom, 2014; Good, 1965; Yudkowsky, 2008) . This proposition is controversial, 1 but if true, the implications are enormous. Given the stakes, the proposition should be taken seriously even if it only has a small chance of being true  (Ćirković, 2012) . In this paper, we present a graphical model that depicts major pathways to catastrophe caused by ASI seizing control from humanity. We call the model the Artificial Superintelligence Pathways Model or ASI-PATH. The model's pathways involve AIs that recursively self-improve, i.e. AIs that create successively more intelligent AIs via improvements to their hardware or software.  2  The model includes both the human process of building the AI and some aspects of the AI itself. The model uses established modelling paradigms from risk and decision analysis, namely fault trees and influence diagrams, in order to depict combinations of events and conditions that could lead to ASI catastrophe, as well as interventions that could avoid ASI catastrophe. Model structures are derived from published literature on ASI catastrophe risk. no attempt is made to quantify model parameters beyond the use of Boolean variables because many of the parameters are highly uncertain; quantifying them rigorously would require significant research beyond the scope of this paper. More generally, given that the broader study of ASI is young and in flux, we expect the ASI-PATH model to evolve in future work. The model presented here can thus be treated as ASI-PATH 1.0. even without quantified parameters, the models can serve as communication tools and guides for actions to reduce ASI catastrophe risk. Furthermore, if parameter values are quantified, then the models can be used for quantitative risk analysis, technology forecasting and decision analysis. By characterising the space of possible AI futures, the models help show, amongst other things: what could happen; where key points of uncertainty are; which short-term and medium-term precursor events to monitor; and what types of interventions could help avoid harmful outcomes and promote beneficial outcomes. The model presented here thus offers a foundation for rigorous quantitative evaluation and decision-making on the risk of ASI catastrophe. Focusing on ASI catastrophe scenarios, we do not mean to imply that scenarios involving positive ASI outcomes are less important. A global-scale ASI catastrophe would clearly be important, but a global-scale positive ASI event could be similarly important, including because it could help address other global catastrophic risks. That said, much of the modelling presented in this paper can readily be extended to positive ASI scenarios. The following section provides detailed background on the ASI scenarios, ASI risk-reduction interventions, and the modelling paradigms used in the paper. Section 3 presents the ASI-PATH model by showing a series of interconnected model sections. Section 4 concludes. An Appendix compiles definitions of select technical terms.
 Conceptual background
 AI catastrophe scenarios We focus on scenarios in which AI becomes significantly more intelligent and more capable than humans, resulting in an ASI causing a major global catastrophe. Specifically, we focus on scenarios involving the following steps: (1) Humans create a seed AI that is able to undergo recursive self-improvement in either hardware or software.  (2)  The seed AI undergoes recursive self-improvement, resulting in a takeoff of successively more intelligent AIs.  (3)  The takeoff results in one or more ASIs. (4) The ASI(s) gain decisive strategic advantage over humanity, meaning 'a level of technological and other advantages sufficient to enable it [the AI] to achieve complete world domination'  (Bostrom, 2014, p. 78 ). (  5 ) The ASI(s) uses their decisive strategic advantage to cause a major global catastrophe, such as (but not necessarily limited to) human extinction.  3  Whilst other AI scenarios may also be able to bring great benefits or harms to humanity, this type of scenario has been the subject of considerable scholarly and popular attention, enough to merit its own dedicated treatment. We remain intentionally mute -with one important exception -on what qualifies as a major global catastrophe. The nature of major global catastrophe is a thorny moral issue, touching on issues such as the value of distant future generations.  4  The ASI scenarios raise additional moral issues, such as whether the ASI itself could be inherently morally valuable. How these moral issues are resolved affects how specific seed AIs are to be evaluated, but it does not affect the modelling presented in this paper. The one exception -the one position this paper takes -concerns the mere existence of superintelligence. We assume, for purposes of this paper and without committing our own views, that the mere existence of an ASI with decisive strategic advantage over humanity does not itself qualify as a major global catastrophe. In other words, this paper takes the position that it is not inherently catastrophic for humanity to lose control of its own destiny. We recognise that this is a contestable position, and in using this position in this paper we do not mean to advocate for it. Instead, we use this position in this paper because much discussion of superintelligence effectively takes this position and because it makes for a reasonable starting point for analysis. Similarly, we have no comment on the relative merits of various concepts for safe (or otherwise desirable) seed AI that have been proposed. These concepts have been discussed using terms such as 'Friendliness'  (Yudkowsky, 2001 (Yudkowsky, , 2012 , 'safe AI scaffolding  ' (omohundro, 2012, 2014) , 'AI safety engineering'  (Yampolskiy & Fox, 2013)  and 'motivation selection'  (Bostrom, 2014) . Distinguishing between these is also an important task but again beyond the scope of this paper. Instead, we use the generic term 'safe AI' throughout the paper without reference to what qualifies as a safe AI and without intending preference for any particular 'safe AI' concept. The modelling presented here is of sufficient generality that it could be applied to a wide range of conceptions of desirable outcomes and desirable seed AI.
 Risk reduction interventions Models like ASI-PATH can help people make better decisions towards ASI safety -that is, towards reducing the risk of ASI catastrophe. each model component suggests opportunities to reduce this risk. The following bulleted list describes the risk reduction intervention options used in ASI-PATH. These are some (but not all) potential risk reduction intervention options, drawing heavily on  Sotala and Yampolskiy (2015) . The interventions can be made by a range of actors, including governments, companies, universities, funding bodies, civil society and AI researchers (including both ASI researchers and their colleagues in other branches of AI). • ASI research risk review boards. review boards would evaluate the riskiness of specific ASI research projects in order to steer research in less risky directions. The review boards could steer research, for example, by disallowing certain projects, directing funds away from riskier projects and towards safer ones and establishing guidelines for publication. Successful review boards would need to correctly evaluate the riskiness of AI research, which could be difficult  (Sotala & Yampolskiy, 2015) . review boards might also be limited in the scope of research they review. A board set up by one country may not be able to review research conducted in other countries, though this could be addressed via international treaty  (Wilson, 2013) . review boards could also overlook ASI projects that operate secretly  (Sotala & Yampolskiy, 2015) . that limit the AI's ability to affect the rest of the world so that it does not gain decisive strategic advantage.  5  Confinement can be applied to the seed AI, the recursively self-improving AIs or the ensuing ASI(s). For example, AI software could be designed such that the AI can only answer questions; this is known as 'oracle AI'  (Armstrong, Sandberg, & Bostrom, 2012) . Another type of confinement would keep AI completely disconnected from the world, for example to test the AI's safety  (Yampolskiy & Fox, 2013 ). In such scenarios, the first AI(s) must be designed to conduct enforcement.
 Fault tree and influence diagram modelling ASI-PATH uses conventions from two established modelling paradigms: fault trees and influence diagrams. Fault trees offer a simple but powerful method for modelling sets of interrelated possible scenarios. Fault trees have been used in modelling risks in a wide variety of contexts, including nuclear power plant safety  (rasmussen, 1981) , homeland security risk  (Cheesebrough, 2010) , nuclear war  (Barrett, Baum, & Hostetler, 2013; Davis, 1989)  and other kinds of systems  (Bedford & Cooke, 2001; Kumamoto & Henley, 1996) . Fault trees represent the ways that events and conditions could combine to be at fault for causing a particular outcome, in this case an ASI catastrophe. each node in the tree represents a particular event, such as an attempted creation of a seed AI, or a condition, such as the presence of some technology keeping the AI safe. The 'top event' node in the tree represents the scenario outcome. Below the top node, the tree branches out with additional nodes corresponding to different ways that the top node can be reached. each layer in the tree represents the combination of events and conditions that could lead to the outcome in the layer directly above it. nodes are connected by Boolean logic gates, such as or, AnD and noT gates. The tree thus shows a set of possible scenarios that could result in the top event. Influence diagrams represent the ways that certain events can influence subsequent events. The events can be random variables, determined by factors external to the model, or decision variables, determined by decisions informed by the model. Influence diagrams thus can be thought of as Bayesian networks with the addition of decision nodes that represent variables whose values can be chosen instead of having a random value  (Pearl, 1988) . Influence diagrams have been used for decades in decision analysis to inform business, government and other policy decisions under conditions of uncertainty  (Clemen & reilly, 2001; edwards, Miles, & von Winterfeldt, 2007) . ASI-PATH uses influence diagrams to model the risk reduction interventions described in Section 2.2. The figures in this paper use standard visual conventions for fault trees and influence diagrams. Connector lines are used to connect nodes representing events and conditions. Vertical connectors with no arrows are used for fault trees; the lower node is at fault for causing the upper node. Horizontal connectors with arrows are used for influence diagrams; the left node influences the right node. Intervention options appear as unrounded rectangles with white text on black background; in influence diagram terminology, these are called decision nodes. other (non-intervention) events appear as rounded rectangles with black text on white background; these are called random-variable nodes. Some figures use both fault tree and influence diagram conventions, as well as both decision nodes and random-variable nodes. 7 The third layer in Figure  1  contains more model detail: six conditions that must all be in place in order for ASI catastrophe to occur. If any of these conditions are missing, then there will be no AI takeoff, or the resulting ASI will not cause catastrophe. Starting on the left side: in order for a seed AI to be created and have the capability to gain decisive advantage, three conditions must all be met:
 The artificial superintelligence pathways model
 Model overview
 Superintelligence from AI takeoff is possible There is some debate about whether it could ever be physically possible for ASI to be created via recursive self-improvement. There is also debate on whether humans are capable of creating such ASI. For example,  Bringsjord, Bringsjord, and Bello (2012, pp. 403-406)  argue that it is far from certain that humans would ever be able to create AI with human-level intelligence, or that such an AI would be able to recursively create more intelligent AI to arrive at superintelligence.
 Seed AI is created For ASI catastrophe to occur, there must be a project in which humans build a seed AI capable of becoming superintelligent via recursive self-improvement. Such an AI is presumably not going to spontaneously materialise out of thin air. There are a variety of ways in which ASI projects could build a seed AI (Section 3.2).
 Containment fails If containment fails, then the ASI gains decisive strategic advantage. Containment measures could include takeoff limits, such that the seed AI does not begin recursive self-improvement, or such that successive AIs do not continue recursive self-improvement so as to become superintelligent. other containment measures could prevent a post-takeoff ASI from gaining decisive strategic advantage, for example via confinement, human enhancement or AI enforcement (Section 2.2). Moving to the right side of Figure  1 , third layer: in order for an ASI to have unsafe goals -in order for it to not use its decisive strategic advantage to cause catastrophe -three conditions must all be met:
 Human attempts to make AI goals safe fail Goal safety can potentially be built into an ASI by the AI designers. Indeed, some people are currently calling for ASI developers to make their AIs' goals safe  (Bostrom, 2014; Yudkowsky, 2012) , and there is ongoing research into various ways to make ASI goals safe  (Goertzel & Pitt, 2012; Yampolskiy & Fox, 2013) . There is some debate about how to design safe ASI goals -more on this below.
 AI does not make its own goals safe Several people have suggested that AIs could make their own goals safe even if the goals were not designed to be safe  (Hall, 2011; Waser, 2008) . one possible reason could be that 'minds which are intelligent enough will, due to game-theoretical and other considerations, become altruistic and cooperative'  (Sotala & Yampolskiy, 2015, p. 21) . others have disputed this point, arguing that without significant dedicated effort, catastrophe is likely to result (e.g.  Bostrom, 2014; omohundro, 2008 Yampolskiy & Fox, 2013; Yudkowsky, 2001 Yudkowsky, , 2012 . For example, omohundro (2008) argues that almost any simple goal, such as playing chess well, could be unsafe. The AI would cause catastrophe in its relentless pursuit of that goal, such as by hoarding the world's resources to play a better game of chess.
 ASI is not deterred from acting unsafely To deter is to persuade another agent to refrain from taking some action by convincing the agent that the action is not in their own interest. even if an ASI has decisive strategic advantage and unsafe goals, it potentially could still be deterred from using its advantage to cause catastrophe. However,  Bostrom (2014)  argues that an ASI with decisive strategic advantage would be undeterrable. We now fill in some additional model details with additional fault tree layers, first for the possibility of limits to AI takeoff, then for human efforts to make the AI safe. The subsequent figures show some major relationships between decision nodes (rounded rectangles, black text on white background) and random-variable nodes (unrounded rectangles, white text on black background). The selection of relationships presented is not intended to be comprehensive. The decision nodes can influence other random-variable nodes besides those shown in the figures. There can also be other decision nodes representing other interventions not modelled in this paper. What is shown is intended only to be illustrative of how decision nodes can be used to incorporate risk reduction interventions in the ASI-PATH model.
 Seed AI creation A critical question is how and when an AI research and development project could build a seed AI. When the seed AI is created matters for several reasons. earlier creation could make takeoff slower and easier to contain (Section 3.3), but earlier creation also gives less opportunity to design a safe seed AI. Figure  2  presents additional detail on seed AI creation, extending the 'Seed AI is created' node from Figure  1 . Figure  2  shows two major types of seed AI: AI of novel design and whole brain emulation. Figure  2  also has a catch-all 'other seed AI' branch for any types of seed AI that do not fit into the two major types. This section of the model draws heavily on  Bostrom (2014, ch. 2) . AI of novel design refers to seed AI whose design is at least novel relative to the human brain. In other words, the seed AI design is at least in part not based on the human brain. Four different pathways to seed AI of novel design are modelled: (1) Artificial general intelligence  (AGI)    better AIs, an AI at least as smart as humans in all respects would also be able to programme better AIs, thereby beginning a recursive self-improvement process  (Bostrom, 2014; Sotala, 2012) . The AGI model branch contains a lower layer to model one pathway to AGI that has been identified, which comes from networks or organisations of AIs, for example if 'the Internet might one day "wake up"'  (Bostrom, 2014, p. 49) . The lower layer also includes a catch-all 'other AGI' node. (2) Self-improvement specialist is seed AI that is 'sub-human in most respects, but with a domain-specific talent for coding and AI research'  (Bostrom, 2014, p. 29) . In other words, the AI is a specialist in recursive self-improvement. For example,  Sotala (2012)  and  Bostrom (2014)  discuss ways in which superintelligence could be achieved by AI with initially subhuman general intelligence but greater than human ability to take control of other networked systems, make copies of itself and to coordinate between copies of itself. (3) Neuromorphic AI is seed AI that is based in part, but not entirely, on the human brain: a partial brain emulation. For example, neuromorphic AI could be based on human patterns of cognition whilst lacking human motivations; the difference in motivations could make it harder to achieve goal safety. (  4 ) Other novel design, a catch-all node for any pathways to novel seed AI designs that do not fit into the three identified pathways. Moving on to the second major branch of Figure  2  -the second major type of seed AI: Whole brain emulation is seed AI that emulates the entirety of the human brain  (Alstott, 2014; Sandberg & Bostrom, 2008) . Because the human brain is capable of designing AIs, a whole brain emulation AI presumably could also design AIs. A whole brain emulation AI could also self-improve using more and faster hardware, something that biological human brains cannot readily do. By imitating the human brain, whole brain emulation may require less intellectual breakthrough than seed AI of novel design -it 'relies less on theoretical insight and more on technological capability'  (Bostrom, 2014, p. 33) . Whole brain emulation requires each of the three steps to occur: (1) Scanning, in sufficiently detail, of one or more whole human brains. The brain scan creates an image of the whole brain to be emulated. (2) Translation, using data analysis and image processing, to construct digital representation(s) of the brains neuronal network and/or any other relevant information. The translation thus converts the scanned image into a format that can be emulated. (3) Simulation, which creates a whole brain emulation seed AI based on the whole brain translation. Figure  2  shows the decision node 'review Boards' (i.e. ASI research risk review boards) influencing each of the model section bottom nodes. review boards could evaluate the safety of each type of seed AI development project. For projects found to be unsafe, review boards could recommend that the projects stop their work or that funders stop funding the projects. Such a board would be analogous to, for example, the review of risky biomedical research ('dual use research of concern' or DurC) by the united States Department of Health and Human Services (HHS; see e.g. oBA, 2012;  Patterson, Tabak, Fauci, Collins, & Howard, 2013) . The DurC review helps determine whether HHS (which includes the national Institutes of Health) funds the research and what safety procedures the research must follow. Figure  2  shows separate arrows pointing to each of the bottom nodes instead of one arrow pointing to the top node ('Seed AI is created') because the influence of review Boards can be different for each bottom node. That is, review Boards could have different influence on different types of seed AI project. For example, perhaps review Boards are more effective for neuromorphic AI, which could require large dedicated projects, and less effective for networked AI, which could emerge from entrenched infrastructure (e.g. the Internet). The use of separate arrows illustrates how this model section would be coded, with different influence relations for different bottom nodes.
 AI containment Containment is defined in this paper as any measure that restricts an AI such that it does not result in an ASI gaining decisive strategic advantage. The restriction can be on seed AI, self-improving AI or ASI. Figure  3  models some key details about containment, extending the 'Containment fails' node in Figure  1 . Working down from the top of Figure  3 , containment fails if takeoff limits fail or if superintelligence containment fails. In other words, an ASI gains decisive strategic advantage if a seed AI takes off, becoming superintelligent, and if the resulting ASI is not contained. Starting on the left branch of Figure  3 , AI takeoff limits fail if either hard takeoff limits fail or soft takeoff limits fail, simply because these are the two types of takeoff: (1) Hard takeoff limits fail. A hard takeoff proceeds rapidly, with no opportunity for human intervention. Figure  3     2 ) Soft takeoff limits fail. A soft takeoff proceeds slowly, permitting significant human intervention in the takeoff process. Figure  3  models three soft takeoff scenarios (i.e. failures of soft takeoff limits): (1) hardware quantity, in which the AI takes off due to gradually increasing its amount of available hardware; 10 (2) Hardware quality, in which AI takes off due to gradually develops faster hardware; and (3) software quality, in which the AI gradually develops smarter AI software. Takeoff speed is important because it strongly affects how much opportunity there will be to manage the recursive self-improvement process. If a hard takeoff is more likely, then it is especially important to design the seed AI so as to achieve desirable outcomes because there will be no opportunity to adjust the AI once takeoff begins.  Shulman and Sandberg (2010)  and  Goertzel and Pitt (2012)  suggest that a soft takeoff would be more likely if the seed AI is created earlier because the hardware, programming languages, etc. available for the AI would tend to result in slower seed AI. Thus, to the extent that the seed AI creation model section (Section 3.2) can indicate when seed AI may be created, this can inform the success of containment. Figure  3  shows the decision node 'enhance Humans' (i.e. enhance human capabilities) influencing the three random-variable nodes below 'Soft takeoff limits fail' . In principle, enhancing human capabilities could help with a wide range of AI safety measures, but it could be especially useful for limiting soft takeoff. This is because soft takeoff involves AIs gradually gaining capabilities; enhanced humans may be able to keep up with the AI during much of the takeoff, ensuring that no unsafe ASI gains decisive strategic advantage. In comparison, a hard takeoff would by definition occur too quickly for even enhanced humans to keep up. As with Figures  2 and 3  show separate arrows pointing to each of the bottom nodes instead of one arrow pointing to the top node ('Soft takeoff limits fail') because the influence of enhance Humans can be different for each bottom node -for each type of soft takeoff. For example, perhaps enhance Humans is more effective for containing hardware quality soft takeoff than for containing software quality soft takeoff because enhanced humans could control the AI's access to the physical equipment necessary for hardware production. As with Figure  2 , the use of separate arrows illustrates how this model section would be coded, with different influence relations for different bottom nodes. Moving on to the right branch of Figure  3 , superintelligence containment fails if there is no superintelligence containment or if the AI escapes whatever superintelligence containment is used: (1) ASI is created without superintelligence containment. Some ASI development projects might not use any superintelligence containment. This could happen for example if the developers have no ability to use superintelligence containment, or if they believe the superintelligence containment would not work, or if they want the ASI to gain decisive strategic advantage. Whatever the reason, ASI cannot be confined if there is no superintelligence containment. Figure  3  shows the decision node 'review Boards' (i.e. ASI research risk review boards) influencing the node 'ASI is created without superintelligence containment' . review boards could monitor ASI r&D projects and require them to have some sort of containment. ASI escapes superintelligence containment. Superintelligence containment might not be designed and implemented well, and even world-class superintelligence containment may fail to contain some ASIs.  Bostrom (2014, pp. 129-131, 145-148, 283)  discusses many potential ways an ASI could escape containment, including persuading or tricking any gatekeepers who can remove containment, using methods of interacting with the outside world that containment designers failed to anticipate, or pretending to malfunction so that the containment is at least temporarily disabled. Figure  3  shows the decision nodes 'AI confinement' and 'AI enforcement' influencing the random-variable node 'AI escapes superintelligence containment' . AI confinement and AI enforcement are two types of superintelligence containment, so this influence relation follows from basic logic.
 Human attempts at making ASI goals safe Modelling ASI goal safety is a subtle and challenging task because there is no clear consensus on what it takes to make ASI goals safe. The study of ASI goal safety is, we think it is fair to say, in early stages, with a lot of uncertainty and disagreement. Because of this, one cannot model ASI goal safety with much specificity, or at least not with much confidence about the specifics. Therefore, the models in this section are intended as exploratory, in order to illustrate some of what is known and to promote open discussion with an eye towards future work. We expect to change these models pending such discussion and pending results of ongoing research on ASI goal safety.
 General failure types Figures 4-7 model general types of failure of human attempts at making ASI goals safe. Figure  4  models two major lines of current thinking about how humans could make ASI goals safe. The two lines of thinking are the left and centre nodes of layer 2, with the right node being a catch-all category for any other lines of thinking: (1) Failure of attempts involving mathematical proof. Some people involved in ASI goal safety research have called for goal safety measures to be built into the seed AI, and furthermore for the correctness of these measures to be mathematically proven before the AI is launched. For example, ASI developers could build in goal stability, such that the AIs goals would not change as it undergoes recursive self-improvement. It may be possible to mathematically prove goal stability prior to launch. 11 Mathematical proofs may be able to give AI designers and other observers a high confidence that the ASI's goals would be safe, though it would not guarantee safety with 100% probability  (Brundage, 2014; Muehlhauser, 2013) . requiring mathematical proofs before launch could also delay launch, by creating an additional hurdle that must be cleared before the seed AI is launched. Given that the failure of ASI goal safety could result in major global catastrophe, requiring mathematical proofs before launching is thus in the spirit of the precautionary principle, a major policy paradigm for risky emerging technologies.  12  Finally, note that this line of thinking does not exclude goal safety measures that occur after seed AI launch. (2) Failure of attempts involving AI biasing during takeoff but not involving mathematical proofs. other people in ASI goal safety research have argued against mathematical proofs prior to seed AI launch, and instead argue for biasing the AI's goals towards safety as the AI recursively self-improves. This line of thinking argues that the AI's goals could be biased through human interaction with the AI as it grows and learns, using reward and punishment from a trainer/ biaser in a fashion somewhat analogous to what human children experience from their parents  (Goertzel & Pitt, 2012) . Such AI biasing requires a soft takeoff; otherwise human interaction during takeoff would be impossible. Because mathematical proofs before launch are not sought, this line of thinking could enable earlier launch of the seed AI. There could be reasons for favouring earlier launch, for example in the context of a race between competing AI projects. However, supporters of requiring mathematical proofs have argued that the belief that proofs are unnecessary might give AI designers a dangerous misperception that an AI's goals would remain safe outside its training environment  (Sotala & Yampolskiy, 2015, pp. 21-23) . (  3 ) Failure of other types of attempts. This node is a catch-all for any attempt at ASI goal safety that does not involve either mathematical proofs or biasing during takeoff. Whilst other types of attempts are, to our knowledge, not prominent in ASI goal safety literature, there may nonetheless be other types of attempts that could achieve ASI goal safety.
 Y each of the three nodes in the second layer of Figure  4  has a lower box labelled x. The x box is a model module, essentially an abbreviation for a repeating model section. The abbreviation x is used to conserve space in the diagram, with no implications for the underlying model logic. We refer to this as a model module by analogy to modular computer programming. If the model is written in computer code, then modular programming could be used for x. The details of x are shown in Figure  5 . Figure  5  shows that the x module has five nodes connected by an or gate, each with a new module Y beneath it. The five nodes are: No attempt, meaning no attempt is made at ASI goal safety using the types of attempt shown in Figure  4 . Theory failure, meaning a failure to achieve ASI goal safety due to some flaw in the theory of the type of goal safety attempted. Theory failure could be due, for example, to a philosophical mistake, such as selecting a goal that turns out to be unsafe, as in an ASI that causes catastrophe in the relentless pursuit of better chess  (omohundro, 2008) . Implementation failure, meaning a failure to correctly implement the ASI goal safety technique in the actual AI. Implementation failure could be due to such things as a coding error or a hardware glitch. Adversary failure, meaning the ASI goals become unsafe due to interference by some sort of adversary. The adversary could be a rogue member of the AI r&D team, an external party such as a competing ASI team or a misguided governing body, or another AI, as in AI enforcement. Other failure, a catch-all for failure modes not covered in the other four nodes. each of the five Figure  5  nodes has a module Y beneath it. Module Y is shown in Figure  6 . Module Y contains two nodes connected by an or gate: failure before and during takeoff. Some ASI goal safety measures must be taken before takeoff, such as those involved in designing and coding the seed AI. other safety measures must be taken after takeoff, such as those involving interacting with the AI so that its self-improvement proceeds in a safe direction. Figure  6  shows the decision node 'encourage Safety' (i.e. encourage research into AI safety) influencing the bottom random-variable nodes in Figure  6 . These influence relations indicate that encourage Safety could influence all types of human attempts at making ASI goals safe. ASI goal safety is not the only type of AI safety that one could encourage, but it is a notable type. Active efforts exist to promote ASI goal safety research, accompanied by arguments that this is an especially important task for improving ASI outcomes (e.g.  Muehlhauser & Bostrom, 2014) . The use of separate arrows pointing to each of the bottom nodes in Figure  6  is analogous to the use of separate arrows in Figures  2 and 3 : to indicate that the influence can be different for different bottom nodes. Because of the x and Y modules, the bottom nodes in Figure  6  are each repeated 15 times, meaning that there are 30 different encourage Safety influence relations. Code of this model section could have different relations for each of the 30 bottom nodes. For example, perhaps it is relatively effective to encourage safety for 'Failure of attempts involving mathematical proof' -'no attempt' -'Failure before takeoff' because before takeoff, it is relatively easy to encourage ASI developers to make at least some attempt at mathematical proofs, and perhaps it is relatively ineffective to encourage safety for 'Failure of attempts involving AI biasing during takeoff but not involving mathematical proofs' -'Theory failure' -'Failure during takeoff' because after takeoff, there might not be enough time to develop adequate theory.
 Specific safety measures Figure  7  presents a different format for modelling human attempts to make ASI goals safe. Instead of organising the attempts by lines of thinking and general failure types, as in Figures  4-7  organises the attempts by the specific safety measures the attempts could involve. Figure  7  thus aims to model the details of what goal safety requires. The goal safety measures are grouped into two categories: measures built into the seed AI before its launch and measures taken during takeoff. This categorisation assumes that no human attempts could succeed after takeoff, since then the AI would be superintelligent and presumably impervious to human interventions. Figure  7  includes four types of safety measures that could be built into the seed AI before takeoff: (1) goal safety, meaning safety measures that result in the post-takeoff superintelligence having safe goals; 13 (2) Goal stability, meaning the AI keeps the same goals throughout recursive self-improvement; (3) openness to training/bias during takeoff, such as to permit 'attempts involving AI biasing during takeoff but not involving mathematical proofs' (Figure  3  and above); 14 and (  4 ) other seed AI measures, a catch-all category for any other safety measures that could be built into the seed AI. Figure  7  uses two 'AnD/or?' logic gates. The reason for this unusual logic structure is because it is not clear whether the logic gates should be AnDs or ors, and indeed they potentially could be either. The 'AnD/or?' gates are thus a representation of model uncertainty, analogous to the use of probability distributions for parameter uncertainty. likewise, the gates can be coded by considering each combination of AnD and or, perhaps with a probability assigned to each combination. For the top logic gate: • An AnD gate would be used if, in order for the top node failure to occur, it is necessary for both layer 2 node failures to occur. That would mean that ASI goal safety could be achieved either via building measures into the seed AI or by taking measures during takeoff. For example, if the seed AI can be made safe before launch, then additional measures during takeoff might not be necessary. Indeed, if the takeoff is hard, then there may not be opportunity for additional measures during takeoff, and goal safety would depend on safety measures built into the seed AI. • An or gate would be used if, in order for the top node failure to occur, it is sufficient for either of the layer 2 node failures to occur. That would mean that AI goal safety would require measures built into the seed AI and measures taken during takeoff. For example, for an AI to be biased towards safe goals during takeoff, it might need certain safety measures built into the seed AI, such as openness to training/biasing during takeoff. For the bottom logic gate: • An AnD gate would be used if, in order for the top node failure to occur, it is necessary for all four bottom node failures to occur. That would mean that pre-takeoff ASI goal safety could be achieved via any one of the four safety measures. For example, if the seed AI is open to bias/training during takeoff, then it may not need to have safe or stable goals before takeoff, as its goals can be adjusted during takeoff. • An or gate would be used if, in order for the top node failure to occur, it is sufficient for any of the four bottom node failures to occur. That would mean that AI goal safety would require all four safety measures to be built into the seed AI before takeoff. For example, goal safety may be useless if the goals are not also stable because the goals would become unsafe during takeoff.
 Conclusion It is far from certain that ASI will ever be built, or that, if it is built, it will result in catastrophe. But given the stakes -given the extreme severity of the catastrophe that could result -ASI catastrophe risk merits careful analysis and management. The ASI-PATH model presented in this paper is one step towards ASI risk analysis and risk management decision-making. ASI-PATH uses fault trees and influence diagrams, which are established risk and decision analysis methodologies with a long history of success at informing risk management decisions. This history, and the experience it has generated, is leveraged in ASI-PATH for the analysis of ASI risk. one might think that ASI is such a novel risk that established methodologies would not be suitable. ASI risk does indeed have some significant novel characteristics, in particular the possibility of humans losing control to the ASI. However, core aspects of ASI risk are broadly similar to other risks. ASI risk involves sets of pathways, events and conditions involving interactions between humans and the technologies they develop. Analysis of ASI catastrophe pathways is thus not fundamentally different from analysis of pathways to other technology catastrophes. This is good news because it makes analysis of ASI catastrophe risk an easier endeavor. That said, ASI catastrophe risk remains poorly understood. These are early days for the study of ASI risk. The ASI-PATH model presented here and the understanding of ASI risk it represents are both likely to evolve as the field matures. This is to say that the ASI-PATH model is wrong -but then again, all models are wrong. Models like ASI-PATH can still be useful by helping people better understand the underlying risk, including the major points of uncertainty. For example, the modelling in this paper suggests that one key uncertainty is the combination(s) of measures needed to make ASI goals safe, hence the model uncertainty ('AnD/or?' gates) depicted in Figure  7 . To the extent that future research could reduce the uncertainty, it would be of value for understanding ASI risk and informing ASI risk management decisions. An important task for future work is the quantification of ASI-PATH parameters using real-number values instead of Boolean logic. Doing so would enable the evaluation of ASI risk and specific risk reduction interventions, towards the goal of taking the most effective actions to reduce ASI risk. Quantifying ASI-PATH parameters is a delicate task due to the significant uncertainty that exists about parameter values, but this only means that quantification should proceed carefully, not that it should not proceed. Indeed, given the stakes associated with ASI catastrophe and the corresponding opportunities for reducing ASI catastrophe risk, the entire study of ASI catastrophe risk should be a significant research priority.
 Notes 7. The use of white text on black background is not a standard influence diagram visual convention. We use it to further distinguish decision nodes from random-variable nodes in our figures. 8. The model can be adapted for analyzing safe ASI by switching the layer 2 right node to 'ASI actions are safe'; much of the lower layer modelling (discussed below) will be similar. 9. The term AGI can refer to more than just seed AI. Indeed, an ASI built from a seed AGI would likely also be an AGI, as the AI is unlikely to lose general intelligence as it undergoes recursive self-improvement. 10. one might argue that a 'hardware quantity' soft takeoff is unlikely on grounds that if an AI is able to access more hardware than its designers intended, it would likely be able to access much or all of the entire world's hardware all at once, causing a hard takeoff. Whilst quantifying scenario probabilities is beyond the scope of this paper, such probabilities can be plugged directly into the model. 11. For discussion of this and related ideas, see for example  Yampolskiy and Fox (2013 ), omohundro (2012 ,  Soares and Fallenstein (2014), and russell, Dewey, and Tegmark (2015) . Details of failure modes for these ASI goal safety measures could be modelled in extensions of Figure  3 . 12. on the precautionary principle in the context of catastrophic risk, see for example  Posner (2004) ;  Sunstein (2009) . 13. Goal safety does not require the seed AI or subsequent pre-superintelligence AIs to have safe goals -thus included here are ideas like 'coherent extrapolated volition'  (Yudkowsky, 2004)  in which AI acquires safe goals during takeoff. 14. See also  Soares (2015)  for a characterisation of openness to training: 'corrigibility' , which an AI system would possess 'if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences' . Figure 1 1 Figure1presents the first three layers of ASI-PATH. The first two layers of Figure1should be fairly intuitive: an ASI catastrophe occurs if a seed AI takes off, resulting in a superintelligence with decisive strategic advantage over humanity, AnD if the ASI's actions are unsafe, such that it uses the decisive strategic advantage to cause catastrophe.8   The third layer in Figure1contains more model detail: six conditions that must all be in place in order for ASI catastrophe to occur. If any of these conditions are missing, then there will be no AI takeoff, or the resulting ASI will not cause catastrophe. Starting on the left side: in order for a seed AI to be created and have the capability to gain decisive advantage, three conditions must all be met:
 Figure 2 . 2 Figure 2. Detail on seed ai creation.
 Figure 3 . 3 Figure 3. Detail on containment failure.
 Figure 4 . 4 Figure 4. two lines of thinking on human attempts at making ai goals safe.
 Figure 6 .Figure 5 . 65 Figure 6. Explanation of the Y module used in figure 4.
 Human enhancement in this context refers to anything that could help humans keep up with AI, such that the AI does not gain decisive strategic advantage even during or after takeoff. Human capabilities could be enhanced, for example, via brain-computer interfaces or mind uploading. Human enhancement could reduce some ASI risks, but it could increase other ASI risks (e.g. if enhanced humans create unsafe ASI) or cause other problems (e.g. moral problems associated with modifying human nature). Additionally, human enhancement could require advanced technology that may not be developed before the seed AI, in which case human enhancement would not help with ASI safety. • AI confinement. AI confinement measures are restrictions built into the AI's hardware or software • Encourage research into ASI safety. AI safety research can be on any aspect of ASI software or hard- ware, or on the humans involved in ASI development. ASI safety research could be encouraged, for example, by increasing funding for ASI safety research, creating open source platforms for safe ASI development or cultivating norms of responsibility amongst ASI researchers. encouraging research can be an easier intervention to perform than something like review boards because encour- aging research can be done informally and outside institutional procedures. However, informal encouragement can sometimes be less effective when it lacks institutional clout. As with research risk review boards, encouraging research requires knowing whether a particular activity would increase or decrease ASI risks. • Enhance human capabilities.
 6  • AI enforcement. AI enforcement occurs when one or more AIs prevent other AI(s) from gaining decisive strategic advantage. As with confinement, AI enforcement can be applied to seed AI, self-improving AIs or ASIs; the enforcement can also be performed by any of these AIs, though more intelligent AIs may tend to be more effective enforcers. For AI enforcement to work, the AIs with strategic advantage must be motivated and able to stop other AIs. For scenarios with multiple AIs, there could be a first-mover advantage, such that the first AI(s) gain and keep strategic advantage.

Abstract: An artificial superintelligence (ASI) is an artificial intelligence that is significantly more intelligent than humans in all respects. Whilst ASI does not currently exist, some scholars propose that it could be created sometime in the future, and furthermore that its creation could cause a severe global catastrophe, possibly even resulting in human extinction. Given the high stakes, it is important to analyze ASI risk and factor the risk into decisions related to ASI research and development. This paper presents a graphical model of major pathways to ASI catastrophe, focusing on ASI created via recursive self-improvement. The model uses the established risk and decision analysis modelling paradigms of fault trees and influence diagrams in order to depict combinations of events and conditions that could lead to AI catastrophe, as well as intervention options that could decrease risks. The events and conditions include select aspects of the ASI itself as well as the human process of ASI research, development and management. Model structure is derived from published literature on ASI risk. The model offers a foundation for rigorous quantitative evaluation and decision-making on the long-term risk of ASI catastrophe.


An AI Race for Strategic Advantage: Rhetoric and Risks

Introduction Global leadership in different areas of fundamental and applied research in artificial intelligence has emerged as a strategic priority, both for major companies and nation states. The last two years have seen heightened competition between industry research groups for talented researchers and startups  (Metz 2017 , CB Insights 2017), and the release of strategic plans aimed at establishing research leadership in AI by the United States (NSTC, 2016), China  (Fa, 2017) , and other research-leading nations  (Hall and Prescenti 2017 , NEDO 2017 , CIFAR 2017 . As the applications of AI research become more lucrative, and the impacts of AI on economic, scientific and military applications more transformative, this competition can be expected to intensify. A narrative has begun to emerge, in both policy and public contexts (Simonite 2017, Allen and Husain 2017, Stewart 2017, Allen and Kania 2017, Allen and Chan 2017), framing future trajectories relating to the development of AI in terms typically associated with a race for technological superiority. The China State Council's 2017 "A Next Generation Artificial Intelligence Development Plan" includes in its aims to seize the major strategic opportunity for the development of AI, to build China's first-mover advantage in the development of AI. The United States National Science and Technology Council's Research and Development Strategic Plan highlights that the US no longer leads the world in numbers of deep learning-related publications. Strong public statements have been made by political and technology leaders: notable examples include Russia's President Vladimir Putin stating that "whoever becomes the leader in this sphere will become the ruler of the world" (RT 2017) and Open-AI cofounder Elon Musk tweeting that "competition for AI superiority at national level [is the] most likely cause of WW3"  (Musk 2017 ). These developments reflect a perception that global leadership in AI could confer scientific, infrastructural and economic advantages to frontrunners. Such comparative advantage could also be self-perpetuating, if it helps to recruit the best research talent, or leads to greater wealth and spending power, or to the accumulation of better computing and data resources, or if the application of AI to the process of scientific research itself leads to further breakthroughs. Similarly, although many breakthroughs are likely to be local in application, the perception that some fundamental advances can be applied to a broad range of research challenges further fuels the idea that a winner could take it all. Such notions of 'winner takes all' or the importance of technological leadership, foster the framing of AI development as a race. We will refer to this framing as that of the 'race for technological superiority' in AI. This kind of race is the primary concern of this paper. But at the same time, it is important to note that there are also concerns about a specifically military AI arms race. Although this is not the primary concern of this paper, there are some noteworthy interrelations with the broader race for technological superiority: most significantly, if AI fulfils its promise as a widely-applicable and transformational technology, then general superiority in AI will to a significant degree imply also military superiority. Many aspects of AI progress are likely to have dual-use relevance to both civilian and military applications (for example, advances enabling greater autonomy and a wider range of capabilities in autonomous vehicles) not least given the deep connection between AI research and the defence community  (Geist 2016) . And if systems approach general intel-ligence or superintelligence, they are likely to confer a very significant strategic advantage that could encompass, but go well beyond, conventional weaponry, with radical implications for security and geopolitics (Allen and Chan 2017). The central concern of this paper is what might happen if the rhetoric noted above around first mover advantage or the pursuit of AI superiority becomes a widespread or even dominant framing of AI development. Our concerns arise both from use of the language of a race for technological superiority (independently of whether such a race is happening) and from such a race becoming a reality.
 The Dangers of an AI Race for Technological Advantage: In Rhetoric and in Reality What is so bad about framing the development of AI in terms of a race for technological advantage? After all, it is widely agreed that AI brings enormous potential benefits across many sectors. One recent report estimated that it could add £232 billion by 2030 to the UK economy alone, with healthcare one of the sectors most enhanced, potentially bringing faster, better service to consumers (PwC 2017). There is a widespread belief that competition and other market forces are central to such innovation. So in as much as a race to develop AI technology means these kinds of benefit come sooner, we have reason to view it positively. But at the same time, the development of such a potentially powerful new technology will need to be steered if it is to be as beneficial as possible while minimising the risks it might pose (Crawford and Calo 2016). In the words of the Future of Life Institute's open letter on 'Research Priorities for Robust and Beneficial Artificial Intelligence', signed by over 8,000 people including many leading figures in AI, work should focus "not only on making AI more capable, but also on maximizing the societal benefit of AI." (Future of Life Institute, 2017a). The danger of an AI race is that it makes exactly this thoughtful steering towards broadly beneficial outcomes more difficult.
 Three Sets of Risks We want to distinguish between three sets of risks: i) The dangers of an AI 'race for technological advantage' framing, regardless of whether the race is seriously pursued; ii) The dangers of an AI 'race for technological advantage' framing and an actual AI race for technological advantage, regardless of whether the race is won; iii) The dangers of an AI race for technological advantage being won.
 (i) Risks Posed by a Race Rhetoric Alone It is possible that the trend towards 'race for technological advantage' terminology in AI, including suggestions such as Vladimir Putin's that the winner of such a race "will become the ruler of the world," could pose risks even if the race is not pursued in earnest, let alone won. We perceive two main risks here: (i.a) The kind of thoughtful consideration of how to achieve broadly beneficial AI, as mentioned above, will require subtle, inclusive, multi-stakeholder deliberation over a prolonged peri-od. The rhetoric of the race for technological advantage, with its implicit or explicit threat that dire consequences will follow if some other group wins that race, is not likely to be conducive to such deliberation. Indeed, rhetoric around technological superiority (such as the 'arms race' rhetoric used in the Cold War in the US), played into what has been called a politics of fear, or a politics of insecurity --that is, a political climate that discourages debate in favour of unquestioning support for a prescribed agenda  (Griffith 1987 ). In The Politics of Insecurity, Jef Huysmans argues that use of the language of security (by which he means militarised language, which would include 'arms race' and related rhetoric) "is a particular technique of framing policy questions in logics of survival with a capacity to mobilize politics of fear in which social relations are structured on the basis of distrust"  (Huysmans 2006 ). (i.b) Second, if the rhetoric of a competitive, 'winner takes all' AI race is used in the absence of an actual race, it could contribute to sparking such a race. (ii) Risks Posed by a Race Emerging If the rhetoric of a race for technological advantage became an actual race to develop sophisticated AI, the risks increase further: (ii.a) First, there is the risk that racing to achieve powerful AI would not be conducive to taking the proper safety precautions that such technology will require (Armstrong, Bostrom, and Shulman 2016). We mentioned above the need for broad consultation about the role AI should play in the life of a community. This might help address important considerations such as avoiding biased systems, or maximising fairness. But in addition to these goals, serious attention must also be given to ensuring humans do not lose control of systems. Such considerations become particularly important if AI approaches general intelligence or superintelligence  (Bostrom 2014 ), but also long before, particularly when AI systems are performing critical functions. The risk is that as the perceived benefit to winning the race increases, so correspondingly does the incentive to cut corners on these safety considerations. (ii.b) Second, a 'race for technological advantage' could increase the risk of competition in AI causing real conflict (overt or covert). Huysmans argues that militarised language such as this has "a specific capacity for fabricating and sustaining antagonistic relations between groups. In the case of the race for technological advantage, it encourages us to see competitors as threats or even enemies. The belief that a country intends in earnest to win an AI race, and that this would result in technological dominance, could, for example, prompt other countries to use aggression to prevent this (akin to the cyberattacks made against Iranian nuclear facilities attributed to the US and Israel) (Nakashima 2012), or motivate the targeting of key personnel (precedents --though specific to their historical context --might include Operation Paperclip, during which over 1,600 German scientists and engineers who had worked on military technology were taken to the US (Jacobsen 2014), or the apparently ongoing operations to encourage the defection of nuclear scientists between nations)  (Golden 2017) . Such scenarios would also in-crease the risk that a general race for technological superiority became increasingly a military AI arms race.
 (iii) Risks Posed by Race Victory The third category of risks of an AI race for technological superiority are those that would arise if a race were won. We will not explore these in detail here --and the forms they take will anyway depend on the precise nature of the technology in question. But as an example, these risks include the concentration of power in the hands of whatever group possesses this transformative technology. If we survey the current international landscape, and consider the number of countries demonstrably willing to use force against others, as well as the speed with which political direction within a country can change, and the persistence of non-state actors such as terrorist groups, we might conclude that the number of groups we would not trust to responsibly manage an overwhelming technological advantage exceeds the number we would.
 Choices for the Research Community Given the dangers noted above of talking about AI development as a competitive race, one could argue that it would be better for the community of researchers considering AI and its impacts to avoid this terminology altogether. In this way, the language of the race for dominance would be seen as (in Nick Bostrom's terms) an information hazard --perhaps either an idea hazard (a general idea whose dissemination could increase risk) or an attention hazard (if we consider that the dangerous idea already exists, but is as yet largely unnoticed)  (Bostrom 2011 ). But if we believe that the idea is already being disseminated by influential actors, such as states, including major powers, then the information hazard argument is weakened. It might still apply to particularly influential researchers --those whose thoughts on the future of AI can become headline news around the world. But even in their case, and particularly in the case of lesser-known figures, there is a strong countervailing argument: that if the potentially dangerous idea of an AI race is already gaining currency, then researchers could make a positive impact by publicly drawing attention to these dangers, as well as by pursuing research dedicated to mitigating the risks of such a framing. Of course, many leading researchers are already speaking out against an AI arms race in the sense of a race to develop autonomous weapons --see for example the Future of Life Institute's open letter on this, signed by over three thousand AI researchers and over fifteen thousand others (Future of Life Institute 2015b). We believe this community could also usefully direct its attention to speaking out against an AI race in this other sense of a competitive rush to develop powerful general-purpose AI as fast as possible. Both media and governments are currently giving considerable attention to AI, yet are still exploring ways of framing it and its impacts. We believe that there is therefore an opportunity now for researchers to influence this framing 1 . One of the principles on AI agreed at the 2017 Asilomar conference offers a precedent on which to build --it states: Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards (Future of Life Institute 2017).
 Alternatives to a Race Approach If we are not to pursue (and talk about pursuing) AI development as a race, then how should we pursue (and talk about pursuing) such development? This is too big a topic to consider thoroughly here. But we note these promising directions for alternative narratives around progress in, and benefits of, AI:
 AI Development as a Shared Priority for Global Good As advances in AI find application to an ever-wider range of scientific and societal challenges, there is a burgeoning discussion around harnessing the benefits of AI for global benefit. This reflects a widespread view among AI scientists and a growing number of policymakers that AI presents tremendous opportunities for making progress on global societal ills, and aiding in tackling some of the biggest challenges we face in the coming century --among them climate change and clean energy production, biodiversity loss, healthcare, global poverty and education. Emphasising these benefits could counteract a race approach in a number of ways: First, if there is global scientific consensus that some of the key aims of AI should be to benefit humanity in these ways, then it becomes less important in which companies or countries key breakthroughs occur. Second, it makes clear that cooperation on the development of AI stands to result in faster progress on these pressing challenges. Lastly, if the aims of the field are to benefit humanity worldwide, then the global community represent stakeholders in the process of AI development; this narrative therefore promotes inclusive and collaborative development and deployment of AI.
 Cooperation on AI as it is Applied to Increasingly Safety-Critical Settings Globally The next decade will see AI applied in an increasingly integral way to safety-critical systems; healthcare, transport, infrastructure to name a few. In order to realise these benefits as quickly and safely as possible, sharing of research, datasets, and best practices will be critical. For example, to ensure the safety of autonomous cars, pooling expertise and datasets on vehicle performances across as wide as possible a range of environments and conditions (including accidents and near-accidents) would provide substantial benefits for all involved. This is particularly so given that the research, data, and testing needed to refine and ensure the safety of such systems before deployment may be considerably more costly and time-consuming than the research needed to develop the initial technological capability. Promoting recognition that deep cooperation of this nature is needed to deliver the benefits of AI robustly may be a powerful tool in dispelling a 'technological race' narrative; and a 'cooperation for safe AI' framing is likely to become increasingly important as more powerful and broadly capable AI systems are developed and deployed.
 Responsible Development of AI and Public Perception AI is the focus for a growing range of public concerns as well as optimism (Ipsos MORI 2017, Fast and Horvitz 2017). Many stakeholders, including in industry, recognise the importance of public trust in the safety and benefits offered by AI if it is to be deployed successfully  (Banavar 2017) . It is the view of the authors that a narrative focused on global cooperation and safe, responsible development of AI is likely to inspire greater public confidence than a narrative focused more on technological dominance or leadership. Other powerful new technologies, such as genetically modified organisms and nuclear power, have in the past proven controversial, with significant communities arguing for a cautious, safety-first approach, to which the rhetoric of the race is antithetical.
 Recent Narratives There have been encouraging developments promoting the above narratives in recent years. 'AI for global benefit' is perhaps best exemplified by the 2017's ITU summit on AI for Global Good (Butler 2017), although it also features prominently in narratives being put forward by the IEEE's Ethically Aligned Design process (IEEE 2016), the Partnership on AI, and programmes and materials put forward by Microsoft, DeepMind and other leading companies. Collaboration on AI in safetycritical settings is also a thematic pillar for the Partnership on AI 2 . Even more ambitious cooperative projects have been proposed by others, for example the call for a 'CERN for AI' from Professor Gary Marcus, through which participants "share their results with the world, rather than restricting them to a single country or corporation" (Marcus 2017). Finally, the overall narrative of cooperation was clearly expressed in a statement issued by a Beneficial AI conference in Tokyo 3 : The challenges of ensuring that [AI] is beneficial are challenges for us all. We urge that these challenges be addressed in a spirit of cooperation, not competition.  (Beneficial AI Tokyo, 2017) .
 Conclusion Although artificial intelligence has been discussed for decades, only recently has it received serious and sustained attention from governments, industry and the media. Among the various ways in which this technology is framed, we have highlighted one that we consider to be potentially dangerous: that of the race for technological advantage. Partly, we believe that a general race to develop AI would be dangerous because it would also encompass --given the dual use of this technology --a literal, military arms race. But even if this were not the case, we believe there would be risks -e.g. from corner-cutting in safety and consultation. Although some might argue that the research community should altogether avoid using AI 'race' terminology for fear of giving it currency, we believe that the idea is already current enough to justify interventions that draw attention to the dangers. Much work remains to be done in understanding the dynamics of a possible race, and in developing alternative 			 For example, few countries have published formal strategies or legislation on AI, but a number have commissioned reviews that have sought expert opinion, eg, the UK Government-commissioned report on AI(Hall and Prescenti, 2017)  and the White House report (NSTC, 2016).
			 See https://www.partnershiponai.org/thematic-pillars/ 3  In which both authors were involved. framings for AI development --but there are encouraging examples on which to build.

Abstract: The rhetoric of the race for strategic advantage is increasingly being used with regard to the development of artificial intelligence (AI), sometimes in a military context, but also more broadly. This rhetoric also reflects real shifts in strategy, as industry research groups compete for a limited pool of talented researchers, and nation states such as China announce ambitious goals for global leadership in AI. This paper assesses the potential risks of the AI race narrative and of an actual competitive race to develop AI, such as incentivising corner-cutting on safety and governance, or increasing the risk of conflict. It explores the role of the research community in responding to these risks. And it briefly explores alternative ways in which the rush to develop powerful AI could be framed so as instead to foster collaboration and responsible progress.


Do the desires of rational agents converge?

1. The plausibility of the claim that the desires of all agents will converge after proper deliberation hinges crucially on how one characterizes such deliberation. One could simply claim that a person only counts as having deliberated properly if she reaches certain approved conclusions. This path would assure Smith's first thesis at the cost of invoking a substantive, nonproceduralist conception of proper deliberation. The interest in Smith's claims stems from his willingness to invoke an understanding of proper deliberation which is not conceptually tied to the deliberator arriving at any particular motivations. In this paper I will say that a conception of normative reasons for action is Humean iff (1) it claims that one's rational desires give one normative reasons (hereafter, reasons), and (2) it invokes a proceduralist notion of proper deliberation in which what makes a desire rational is that it would be arrived at if one successfully followed the procedure, and (3) this procedure is specified such that, at least in principle, any particular option could be rationally desired after successfully following the procedure, and (4) it is relativist in the sense that different agents might arrive in the right way at different desires and hence have divergent reasons. This last condition shows that, for the Humean, the normativity for A of the desires that A would arrive at in the approved way is not undermined should other agents arrive in the approved way at other desires. Smith clearly rejects the fourth thesis. He must also be rejecting the first or second thesis. Perhaps he would want to say that successfully following the procedure makes one's resultant desires rational, but that (contra thesis 1) rational desires only give one reasons when all rational agents converge in their desires. Alternatively, Smith might want to say that one's desires are only rational if everyone would reach them after proper deliberation (contra thesis 2). Smith finds himself in substantial agreement with Bernard Williams's account of fully rational deliberation, complaining only that Williams's account requires supplementation. 2 (158) Thus Smith and Williams largely agree about what it is for an agent to arrive at a desire in the right way. Smith glosses Williams's position in this way: in order for a deliberator to counts as fully rational '(i) the agent must have no false beliefs, (ii) the agent must have all relevant true beliefs, (iii) the agent must deliberate correctly.' (  156 ) Conditions (i) and (ii) claim that we can only be fully rational deliberators if we get the facts right. Smith's requirement of true belief shows that he is not trying to capture the notion of proper deliberation given one's limited time and access to the truth. Thus while we might have been rational in light of the information available to us (because, for example, the scientific community had not discovered certain facts) or that it was reasonable for us to collect, we were not fully rational deliberators in Smith's sense for we did not know all the facts. One sense of 'rational deliberation' is to make proper use of the information one has (or could reasonably be expected to get). Thus a person might have been rational in this sense to leave the building after the fire alarm went off even though it turns out that the building was not really on fire. Smith is clearly after another notion of rational deliberation, one perhaps well designed to determine what it is advisable to do or what will get us what we really want. Smith's targeted notion might be called 'ideally rational deliberation.' Several philosophers, myself included, have recently argued that this notion of ideally rational deliberation involves serious conceptual problems, but I want to focus on other issues here.  3  Condition (iii) adds possibilities for practical reasoning such as these: the agent adequately exercises her imagination in considering the options available, the agent makes no mistakes of means-end reasoning, the agent engages in constitutive reasoning 'such as deciding what would make for an entertaining evening, granted that one wants entertainment.'  (Williams 1981 : 104) Smith's main additions to Williams's above criteria are that the agent is not suffering from depression, apathy, or other debilitating attitudes, and the agent has a concern that her overall desire pattern be coherent and justified. (158-61) Crucially these possibilities for practical reason do not amount to the claim that it is criterial of proper deliberation that one reach a particular specified motivational profile. 2. Before considering Smith's arguments for his two theses we must better understand what we mean when we speak of the desires of all ideally rational agents converging. This is more complicated than it might seem. Different things might be thought to constitute convergence. A weak understanding of convergence would be that all fully rational agents agree about every claim of this form: A has normative reason to O in circumstances C. If a Humean account of reasons were correct and this was accepted, then, presumably, we would all agree, when fully rational, about all such reason claims. In fact, the most likely way to achieve such agreement would be for all rational agents to agree on an account of practical rationality; that is, agree about what makes it the case that A has reason to O in circumstances C. This kind of agreement is compatible with different agents having arbitrarily divergent ideally rational desires. Call this 'convergence about each agent's reasons.' This, it turns out, is not the notion of convergence that Smith is pointing to. Or rather, such agreement would seem necessary but not sufficient for the kind of convergence that Smith has in mind. Smith's understanding of convergence would be for all rational agents to converge on desires that have the same de se content. (169-70) If we both want you rather than me to get the larger slice of cake there is a sense in which we converge and a sense in which we do not. We both want the same state of the world to obtain, but we do not both want the cake for ourselves. If we both wanted cake for ourselves our desires would have the same de se content. A simple case of this sort of convergence would be if all fully rational agents desired only their own pleasure. In this case rational agents agree about what kind of things are desirable. Convergence in the objects of desire can be thicker or thinner. We might both want to listen to the Rolling Stones album Exile on Main Street very loud, or we might both want to listen to music. The former agreement in desire is thicker in that the shared object of appropriation is specified more specifically. It is likely that some very thin convergence can always be found between two people or perhaps between all rational agents. Perhaps all rational agents desire to be rational, to live a good life, or to avoid mistakes. If the convergence that Smith has in mind is convergence in the objects of desire, then it is crucial to know what level of generality he intends. Thick convergence among all rational agents would be very surprising and, perhaps, entail moral agreement, whereas very thin convergence would not be very interesting and would not entail moral agreement. Further, if we are not given a specific level of generality at which rational agents are supposed to converge, then one could worry that Smith could, in the face of any seeming disagreement, simply move in each such case to thinner and thinner areas of convergence. There could also be varying degrees of the extent of convergence. Rational agents might agree (at a fixed level of generality) about all, most, or just a few objects of desire. Smith's claim is that there is full convergence in desire amongst rational agents. A problem for the thesis that rational agents will converge on the objects of desire thickly described is that on matters of mere tastes agreement does not seem forthcoming. It is not plausible that every rational deliberator will like chocolate ice cream more than vanilla or Seinfeld more than Friends. Our desires diverge over such issues. Yet in many cases our divergent desires seem for all the world informed in the relevant sense and well insulated from consistency or justificatory pressures from other areas of our motivational make-up. There are strong grounds for thinking such divergence in tastes will survive rational deliberation. Smith concedes this point, but tries to show that this would not prevent the kind of full convergence he has in mind.  4  Smith admits that his preference for beer over wine provides him a reason to get beer rather than wine where our opposite preference provides us reason to get wine instead. Smith claims that this is no threat to full convergence because the preference is 'a relevant feature of our circumstances....' (171) This can be seen, Smith tells us, 'from the fact that I can quite happily agree with you that if I were in your circumstances -if I preferred wine to beer -then the fact that the local wine bar sells very good wine would constitute a reason for me to go there as well ….' (171) Smith continues: 'even if an agent's preferences may enter into a specification of the circumstances that she faces it may still be the case that whether or not she is rationally justified in taking her own preferences into account, and the way in which she is justified in taking them into account if she is, depends on whether fully rational agents would all converge on a desire which makes the preferences she has relevant to her choice.' (171) Smith is here claiming that rational agents might, despite their divergent tastes, nonetheless converge on desires of the following form: if I were in circumstances C (which includes having a certain preference profile P), then I want myself to choose O. At first sight such convergence might appear to be nothing more than convergence on instrumental rationality itself. It appears that the thought is simply this: whatever desires I find myself having, I want myself to satisfy those desires. But things are not so simple. Allan Gibbard notes that 'Ideals differ from tastes: I dislike spinach but think it a matter of taste; that means in part that although I dislike spinach I am willing to eat it if I like it. The norms I accept endorse eating it if I like it and not otherwise. I oppose cruelty unconditionally: I want myself not to be cruel even if, hypothetically, I should want to be.'  (Gibbard, 1990: 167)  Thus, for those that hold ideals in Gibbard's sense (and moral claims seem paradigmatically to be such ideals), what one would want for oneself if one had an altered preference ranking need not simply be maximally to satisfy those altered desires. Smith's claim seems to be that all rational agent's might converge in their de se desires about what they would want for themselves if they were in a particular circumstance (which includes having a certain preference profile). There are at least two different ways in which such convergence might be achieved. The first way would be for all rational agents to treat every preference as a matter of taste and never as an ideal. Such rational agents might agree that for any situation they find themselves in, they would want to maximally satisfy the preferences that they have in that situation. Convergence of this sort, I take it, would be compatible with a thoroughgoing Humean account of an agent's reasons for action. Thus I suppose it must not be this method of reaching convergence that Smith has in mind for he calls his proposal an 'anti-humean theory of normative reasons.' Smith, it would seem, must therefore have in mind a picture in which fully rational agents converge in their desires in a way that crucially involves them having ideals. How might such a picture go? The second way such convergence might occur, in this case involving agents with ideals, would be for all rational agents to agree fully about what counts as a matter of mere taste and what counts as a matter of ideals. It would not be important that the rational agents agree in their tastes, rather only that they agree about which of their preferences are to be treated as mere tastes. For where we treat our preferences as mere matters of taste, we presumably agree about what we want in situations in which we have this taste or that; namely, indulge our tastes as we find them. In this way Smith has found a nice way of acknowledging that fully rational agents would not converge on preferring chocolate ice cream to vanilla, yet still holding out the prospect that they might fully converge in their desires in an interesting and important sense. It would seem however that the rational agents would have to converge on their ideals in order for the full kind of convergence in desires that Smith envisions to occur. When two agents disagree in their ideals, they will sometimes disagree in their wants for themselves in circumstances C (which includes having a certain preference profile P). Thus for the kind of convergence that Smith has in mind to occur all rational agents must both (1) agree about what counts as a mere matter of taste and what counts as an ideal and (2) agree in their ideals. Imperfectly rational agents clearly disagree about what are matters of mere taste and what are ideals. Some want themselves to dress fashionably even in the situation in which they do not want to look fashionable, others treat this as a matter of taste. It must be claimed that this kind of disagreement would be extinguished upon becoming fully rational. It is worth noting that for two people to agree in their ideals they must agree about significantly more than just morality. For as the above example of fashion shows, a person can hold a preference as a personal ideal even when they think others would not be immoral or irrational for failing to share that ideal. Gibbard called such ideals 'existential: it is a choice of what kind of person to be, in a fundamental way, come what might, which the chooser does not take to be dictated by considerations of rationality.' (ibid.: 168) Thus it would seem that to vindicate Smith's conception fully rational agent's would have to converge fully not only about morality, but also about personal ideals. This would seem to necessitate that the process of becoming fully rational extinguish all existential ideals, or at least make false the claim that the holder of the existential ideal makes: namely that commitment to this ideal is not dictated by rationality. And as holding false beliefs is supposed to be incompatible with being fully rational, it would seem that Smith must hold that existential ideals must go if convergence is to take place. Let me here parenthetically acknowledge a clear and important virtue of Smith's attempt at an 'anti-Humean Theory of Normative Reasons.' It deals as successfully with matters of mere taste as does the Humean theory and this to my mind had been one of the clearest advantages of the do the desires of rational agents converge? 143 Humean account. Smith shows us how the anti-Humean can, perhaps effectively, avoid conceding this front. 3. Why must proper deliberation lead to convergence if we are to have reasons at all? Smith's conclusion here is remarkably strong. He claims the very concept of reasons demands convergence. Thus, apparently, the dominant Humean understanding of reason is, in a straightforward way, conceptually confused. I find Smith's clearest arguments for this to be the following: (1) such an assumption is necessary to underwrite the platitude that 'we are … potentially in agreement or disagreement with each other about what constitutes a reason and what doesn't.' (172), and (2) absent this sort of convergence I must see my reasons as being entirely dependent on the fact that my actual desires are as they are. But the shape of my actual desires (which, by hypothesis, will determine the shape of my desires after rational deliberation), Smith tells us, is an 'entirely arbitrary matter, one without any normative significance of its own.' (172) I will consider these arguments in turn. Consider first Smith's case that convergence is necessary to vindicate the thought that we can dispute reasons claims. He writes there is … a sense in which we can talk about rational justification or desirability simpliciter. When you and I talk about the reasons that there are for acting [in this sense], we are therefore talking about the same thing …. On the relative conception, however, matters are quite different. For in order to give a truth condition to the schematic claim 'It is desirable that p in c' we need to know from whose perspective the truth of the claim is to be assessed ….  [Thus]  if I say to you 'There is a reason for O-ing', and you deny this, we are therefore potentially talking about different things …. (166-67) All of this is true. But it does not support either the claim that on the Humean view there could not be a common subject matter when we discuss reasons, or that 'rational justification [concerning reasons] is itself a relative matter; that really there is only rational-justification-relative-tothis-person or rational-justification-relative-to-that.' (171) In fact these two claims are false. Smith claims that on Williams's Humean view when we speak of reasons 'it typically means reason me out of my mouth, reason you out of your's, reason her out of her's and so on.' (172) But, of course, we might both be talking about the reasons of the same person. We might be debating if A has reason to O in circumstances C. On the Humean account when we debate whether A has reason to O in circumstances C we are debating what A's motivations would be like if she were deliberating properly. The Humean thus holds that we can be right or wrong in such claims and we have a common subject matter when we discuss them. Someone who claimed that a person had an intrinsic reason to O, when that person had nothing in her motivational set that O answered to, would just be wrong. Thus the Humean framework allows us to vindicate the platitude that we have a common subject matter when we speak of a particular agent's reasons. Perhaps it is the case that, on the Humean view, we cannot find a common subject matter for other kinds of reason claims. Likely the Humean cannot vindicate certain kinds of reason claims that we might have hoped to vindicate (e.g., that there are reasons, such as moral reasons, which apply to all rational agents as such). But even if this is granted there is no reason here to suppose, as Smith does, that the Humean is forced to abandon the very notion of reasons altogether. Further, the way to justify reason claims such as A has reason to O in circumstances C is not relative to each individual. The Humean holds that there is a fact of the matter about this which is determined by the agent's motivations in counterfactual situations (which will admittedly be very difficult to determine) just as there is a fact of the matter about the effects of red wine on the body. And we justify believing the former sort of facts in the same non-person relative way that we justify the latter sort of facts. Smith's second argument claims that unless desires converged amongst fully rational deliberators, one's rational desires would be determined by what one (perhaps idiosyncratically) happened to find attractive. Smith claims that if this were so one's rational desires would depend on 'an entirely arbitrary matter' and 'arbitrariness is precisely a feature of a consideration that tends to undermine any normative significance it might initially appear to have.' (172-73) Smith could mean one of two things. First, he could mean that as a psychological matter an agent cannot take her own desires seriously, cannot herself find them to be a source of reasons for her, if they are arbitrary in this way. This is clearly false. The prevalence of sincere Humeans demonstrates this. The Humean may be wrong about reasons for action, but it must be allowed that she really believes her theory. The second interpretation of Smith's claim is that there is no justification for treating one's desires that are arbitrary in this way as creating reasons. But this is exactly what is at issue between the Humean and Smith. My rationally arbitrary preference (that is, a preference I could be rational without) for chocolate ice cream gives me a reason to choose it. What is incoherent about all my reasons being such? Smith has baldly claimed that there is a conceptual confusion in the notion of rationally arbitrary desires by themselves creating reasons. The predominant neo-Humean view holds that this is precisely what creates reasons. Thus Smith will have to do better. I myself have trouble seeing anything conceptually confused or problematically arbitrary in the thought that my rationally optional projects and commitments create reasons for me and not for you. Before moving on I want to briefly consider the plight of the agent that deliberates by successfully following the procedure that Smith recommends but unhappily discovers that other rational agents arrived at divergent desires. Smith tells us that this agent has no reason to satisfy her desires, in fact she has no reasons (recall I use 'reason' to refer to Smith's 'normative reason') at all for there are no reasons without convergence. Thus a lack of convergence, according to Smith, would make it the case that I had no reason to avoid pain or choose a flavour of ice cream I like rather than one I find disgusting. I find this suggestion difficult to take seriously. Does Smith really think the existence of such reasons is contingent on universal convergence? Perhaps Smith should only claim that an agent has a reason to O iff all rational agents would converge on a desire for O in the relevant circumstances rather than claim that for an agent to have any reasons all rational agents must converge on all desires. I would still disagree with this claim, but it would not flaunt what I regard as the absurdities mentioned in the previous paragraph. In some cases, like the preference for chocolate rather than vanilla ice cream, our bare liking provide us with a reason to choose one thing rather than another. Even if all rational agents do not converge in their desires, I like chocolate ice cream better than vanilla, and this is all I need to have a reason to choose it. It would seem that Smith must either claim that, in such cases, my getting chocolate ice cream does not benefit me more than getting vanilla or that I have no reason whatsoever in such a case to seek my benefit rather than my detriment. Neither claim strikes me as plausible. 4. Smith argues that convergence in desires amongst rational deliberators is quite plausible. Smith's main argument here is that 'the empirical fact that moral argument tends to elicit the agreement of our fellows gives us reason to believe that there will be a convergence in our desires under conditions of full rationality. For the best explanation of that tendency is our convergence upon a set of extremely unobvious a priori moral truths.' (187) Smith bolster this argument with three supplementary arguments: (1) often by focusing only on moral disagreement we ignore substantial agreement and the fact that 'we' share thick evaluative language which entails a fair degree of agreement in attitude, (2) although current moral disputes appear at times deadlocked, 'we must remember that in the past similarly entrenched disagreements were removed inter alia via a process of moral argument' and (3) some moral disagreement certainly can be explained as clearly arising from lack of rational deliberation. (  188 ) But much of the historical moral argumentation that has actually managed to produce consensus has been (1) factually and logically imperfect, (2) addressed to those poorly positioned to object, (3) addressed to those who share substantial common moral vocabulary, moral education, and cultural identification, and (4) offered by those who are persuasive for reasons other than the cogency of their position. Any historical tendency towards convergence that results from some combination of these four causes (and no doubt others) will not constitute evidence that rational agents will converge in their moral views. Since such causes undoubtedly played a very significant role in producing convergence, we cannot simply point towards a general historical tendency towards convergence (even if there were one) and claim it constitutes inductive evidence for Smith's case. Thus, as we might have suspected, Smith's historical case about a tendency towards convergence will have to be genuinely historical. It will have to persuade us of the crucial role of facts, logic, and reason is explaining the history of convergence and the secondary role of force, guile, and a shared thick moral vocabulary. The sort of convergence Smith holds out real hope for is convergence amongst people that start off as different in their desires, moral vocabulary, and moral education as we can imagine. The convergence in the objects of desire that Smith needs is not simply convergence amongst all actual humans (as if we could make ourselves right by killing dissenters) but rather amongst all possible agents no matter how initially divergent. But what is the history of convergence amongst agents with radically divergent desires, moral vocabularies, and moral educations when they are confronted with factually informed and logically impeccable argumentation? Certainly Smith will have to remind us of such cases. Smith needs a string of cases in which (1) convergence occurred, (2) it occurred for the right reasons, and (3) it occurred between radically different cultures. Smith relies quite plainly and inappropriately on convergence within an extended community. Smith points to convergence within a community that shares a common moral vocabulary of thick evaluative language. (188) This commitment to a common thick evaluative language, Smith rightly points out, reveals considerable moral agreement within the community. (188) But just because this shared thick evaluative language presupposes and reveals considerable moral agreement, convergence amongst those that share thick evaluative language is of very limited use in demonstrating the power of rational argumentation to itself manufacture this agreement. Rational argumentation has some power to lead people who actually agree about morality to see that they agree. This is not do the desires of rational agents converge? 147 doubted. But Smith needs to show that history provides a good inductive case that all fully rational agents, no matter how initially divergent they are prior to becoming fully rational, will agree on moral matters. Smith finds himself 'in substantial agreement with Derek Parfit' (214) when Parfit claims that Belief in God, or in many gods, prevented the free development of moral reasoning. Disbelief in God, openly admitted by a majority, is a very recent event, not yet completed. Because this event is so recent, Non-Religious Ethics is at a very early stage. We cannot yet predict whether, as in Mathematics, we will all reach agreement.  (Parfit 1984: 454)  But betraying such hopes undermines Smith's inductive historical argument for it suggests that the factually informed deliberators will not predicate their moral opinions on religious authority. And surely an overwhelming amount of historical ethical argument that has produced convergence has relied on religious beliefs and authority. It might be true, as Parfit says, that absent such a religious basis we lack any reason to suppose that there will not be convergence. But surely we must therefore also admit that past moral practices do not provide sufficient inductive grounds to suppose that such agreement will be forthcoming. Thus I conclude that Smith's arguments in support of the plausibility of extensive convergence are unconvincing and that he has failed to show that such convergence is necessary for the existence of reasons.  5  Bowling Green State University Bowling  Green,   			 Smith, 1994.  All otherwise unattributed references are to this work.
			 Williams 1981  spells out his neo-Humean account of instrumental rationality.
			  See Velleman 1988 , Sobel 1994 , Rosati 1995 , and Loeb 1995 Interestingly all these critiques are offered against the full information account of well-being rather than the remarkably comparable full information account of reasons for action that Williams and Smith are developing. These two theories, for all their similarity, seem to have developed and be discussed in isolation from each other.
			 Seemingly, Smith might have conceded that the case of mere tastes shows that there would not be full convergence on the objects of desire after fully rational deliberation, while continuing to insist that we have reason to be 'optimistic about the possibility of an agreement about what is right and wrong being reached under more idealized conditions of reflection and discussion.' (189)
			 I would like to thank David Copp, Michael Smith, Sara Worley and an anonymous referee for Analysis for their help with this paper.

Abstract: Michael Smith, in his impressive book The Moral Problem , 1 claims that (1) 'convergence in the hypothetical desires of fully rational creatures is required for the truth of normative reason claims' (  173 ) and (2) we have reason to 'have some confidence [… that] there will be a convergence in our desires under conditions of full rationality.' (187). These claims play a crucial role in supporting Smith's distinctive and interesting meta-ethical position. However, I will here ignore the context of Smith's broader ambitions and focus on these two theses. I will argue that Smith does not give us adequate justification for either claim. This paper will have four sections. In §1 I will explain Smith's conception of fully rational deliberation. In §2 I discuss what it would be for desires to converge as Smith suggests. In §3 I consider Smith's conceptual claim that such convergence is necessary for there to be normative reasons at all. Finally, I address Smith's reasons for optimism about convergence. The first three sections consider Smith's first thesis above while the fourth section considers his second thesis.