GENERAL PURPOSE INTELLIGENCE: ARGUING THE ORTHOGONALITY THESIS

The Orthogonality Thesis Scientists and mathematicians are the stereotypical examples of high intelligence humans. But their morality and ethics have been all over the map. On modern political scales, they can be left-(Oppenheimer) or right-wing (von Neumann) and historically they have slotted into most of the political groupings of their period (Galois, Lavoisier). Ethically, they have ranged from very humanitarian (Darwin, Einstein outside of his private life), through amoral (von Braun) to commercially belligerent (Edison) and vindictive (Newton). Few scientists have been put in a position where they could demonstrate genuinely evil behavior, but there have been a few of those  (Teichmüller, Philipp Lenard, Ted Kaczynski, Shirō Ishii) . Of course, many scientists have been absolutely conventional in their views and attitudes given the society of their time. But the above examples hint that their ethics are not strongly impacted by their high intelligence; intelligence and ethics seem 'orthogonal' (varying independently of each other, to some extent). If we turn to the case of (potential) artificial intelligences we can ask whether that relation continues: would high intelligence go along with certain motivations and goals, or are they unrelated? To avoid the implicit anthropomorphisation in terms such as 'ethics,' we will be looking at agents 'final goals' -the ultimate objectives they are aiming for. Then the Orthogonality thesis, due to Nick Bostrom  (Bostrom, 2012) , states that: Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal. It is analogous to Hume's thesis about the independence of reason and morality  (Hume, 1739) , but applied more narrowly, using the normatively thinner concepts 'intelligence  ' and 'final goals' rather than 'reason' and 'morality'.  But even 'intelligence,' as generally used, has too many connotations. A better term would be efficiency, or instrumental rationality, or the ability to effectively solve problems given limited knowledge and resources  (Wang, 2011) . Nevertheless, we will be sticking with terminology such as 'intelligent agent,' 'artificial intelligence' or 'superintelligence,' as they are well established, but using them synonymously with 'efficient agent,' artificial efficiency' and 'superefficient algorithm.' The relevant criteria is whether the agent can effectively achieve its goals in general situations, not whether its inner process matches up with a particular definition of what intelligence is. Thus an artificial intelligence (AI) is an artificial algorithm, deterministic or probabilistic, implemented on some device, that demonstrates an ability to achieve goals in varied and general situations.  1  We don't assume that it need be a computer program, or a well laid-out algorithm with clear loops and structures -artificial neural networks or evolved genetic algorithms certainly qualify. A human level AI is defined to be an AI that can successfully accomplish any task at least as well as an average human would (to avoid worrying about robot bodies and such-like, we may restrict the list of tasks to those accomplishable over the internet). Thus we would expect the AI to hold conversations about Paris Hilton's sex life, to compose ironic limericks, to shop for the best deal on Halloween costumes and to debate the proper role of religion in politics, at least as well as an average human would. A superhuman AI is similarly defined as an AI that would exceed the ability of the best human in all (or almost all) tasks. It would do the best research, write the most successful novels, run companies and motivate employees better than anyone else. In areas where there may not be clear scales (what's the world's best artwork?) we would expect a majority of the human population to agree the AI's work is among the very best. Nick Bostrom's paper argued that the Orthogonality thesis does not depend on the Humean theory of motivation, but could still be true under other philosophical theories. It should be immediately apparent that the Orthogonality thesis is related to arguments about moral realism. Despite this, we will not address the fertile and extensive literature on this subject. Firstly, because it is contentious: different schools of philosophical thought have different interpretations of the truth and meaning of moral realism, disputes that cannot be currently resolved empirically. Since we are looking to resolve a mainly empirical question -what systems of motivations could we actually code into a putative AI -this theoretical disagreement is highly problematic. Secondly, we hope that by approaching the issue from the computational perspective, we can help shed new light on these issues. After all, we do not expect that the trigger mechanism of a cruise missile to block detonation simply because people will die -but would an "ultra-smart bomb" behave the same way? By exploring the goals of artificial systems up to higher level of efficiency, we may contribute to seeing which kinds of agents are susceptible to moral realism arguments, and which are not. Thus this paper will content itself with presenting direct arguments for the Orthogonality thesis. We will assume throughout that human level AIs (or at least human comparable AIs) are possible (if not, the thesis is void of useful content). We will also take the position that humans themselves can be viewed as non-deterministic algorithms: 2 this is not vital to the paper, but is useful for comparison of goals between various types of agents. We will do the same with entities such as committees of humans, institutions or corporations, if these can be considered to be acting in an agent-like way. The thesis itself might be critiqued for over-obviousness or triviality -a moral anti-realist, for instance, could find it too evident to need defending. Nevertheless, the argument that AIs -or indeed, any sufficiently intelligent being -would necessarily behave morally is a surprisingly common one. A. Kornai, for instance, considers it as a worthwhile starting point for investigations into AI morality  (Kornai, 2013) . He bases his argument on A. Gewith's approach in his book, Reason and Morality  (Gewirth, 1978)  (the book's argument can be found in a summarized form in one of E. M. Adams's papers  (Adams, 1980) ) in which it is argued that all agents must follow a "Principle of Generic Consistency" that causes them to behave in accordance with all other agent's generic rights to freedom and well-being. Others have argued that certain specific moralities are attractors in the space of moral systems, towards which any AI will tend if they start off with certain mild constraints  (Waser, 2008) . Because of these and other examples (and some online criticism of the Orthogonality thesis 3 ), we thought the thesis was worth defending explicitly, and that the argument brought out in its favor would be of general interest to the general discussion.
 Qualifying the Orthogonality Thesis The Orthogonality thesis, taken literally, is false. Some motivations are mathematically incompatible with changes in intelligence ("I want to prove the Gödel statement for the being I would be if I were more intelligent"). Some goals specifically refer to the intelligence of the agent, directly ("I want to be much less efficient!") or indirectly ("I want to impress people who want me to be much less efficient!"). Though we could make a case that an agent wanting to be less efficient could initially be of any intelligence level, it won't stay there long, and it's hard to see how an agent with that goal could have become intelligent in the first place. So we will exclude from consideration those goals that intrinsically refer to the intelligence level of the agent. We will also exclude goals that are so complex or hard to describe that the complexity of the goal becomes crippling for the agent. If the agent's goal takes five planets worth of material to describe, or if it takes the agent twenty years each time it checks what its goal is, then it's obvious that that agent can't function as an intelligent being on any reasonable scale. Many have made the point that there is likely to be convergence in instrumental goals  (Omohundro, 2008) . Whatever their final goals, it would generally be in any agent's interest to accumulate more power, to become more intelligence and to be able to cooperate with other agents of similar ability (and to have all the negotiation, threatening and cajoling skills that go along with that cooperation). Note the similarity with what John Rawls called 'primary goods'  (Rawls, 1971 ). We will however be focusing exclusively on final goals, as the instrumental goals are merely tools to accomplish these.  4  Further we will not try to show that intelligence and final goals can vary freely, in any dynamical sense (it could be quite hard to define this). Instead we will look at the thesis as talking about possible states: that there exist agents of all levels of intelligence with any given goals. Since it's always possible to make an agent stupider or less efficient, what we are really claiming is that there could exist possible high-intelligence agents with any given goal. Thus the restricted Orthogonality thesis that we will be discussing is:
 High-intelligence agents can exist having more or less any final goals (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). 5 We will be looking at two variations of the "can exist" clause: whether the agent can exist in theory, and whether we could build such an agent (given that we could build an AI at all). Though evidence will be presented directly for this thesis in the theoretic agent case, the results of this paper cannot be considered to "prove" the thesis for agents we could build (though they certainly raise its likelihood). In that case, we will be looking at proving a still weaker thesis:
 The fact of being of high intelligence provides extremely little constraint on what final goals an agent could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). That thesis still has nearly all the relevant practical implications that the strong Orthogonality thesis does.
 Orthogonality in Practice for AI Designers The arguments presented in this paper are all theoretical. They posit that AIs with certain goals either 'can exist,' or that 'if we could build an AI, we could build one with any goal'. In practice, the first AIs, if and when they are created, will be assembled by a specific team, using specific methods, and with specific goals in mind. They may be more or less successful at inculcating the goals into the AI (or, as is common in computer programming, they may inculcate the goals exactly, only to realize later that these weren't the goals they really wanted). The AI may be trained by interacting with certain humans in certain situations, or by understanding certain ethical principles, or by a myriad of other possible methods, which will likely focus on a narrow target in the space of goals. The relevance of the Orthogonality thesis for AI designers is therefore mainly limited to a warning: that high intelligence and efficiency are not enough to guarantee positive goals, and that they thus need to work carefully to inculcate the goals they value into the AI.
 Orthogonality for Theoretic Agents If we were to step back for a moment and consider, in our mind's eyes, the space of every possible algorithm, peering into their goal systems and teas-ing out some measure of their relative intelligences, would we expect the Orthogonality thesis to hold? Since we are not worrying about practicality or constructability, all that we would require is that for any given goal system (within the few constraints enumerated above), there exists a theoretically implementable algorithm of high intelligence. Any measurable  6  goal can be paired up with a reward signal: an agent gets a reward for achieving states of the world desired by the goal, and denied these rewards for actions that fail to do so. Among reward signal maximisers, the AIXI is the theoretically best agent there is, more successful at reaching its goals (up to a finite constant) than any other agent  (Hutter, 2005) . AIXI itself is incomputable, but there are computable variants such as AIXItl or Gödel machines  (Schmidhuber, 2007)  that approximate AIXI's efficiency. These methods work for whatever reward signal plugged into them. Or we could simply imagine a supercomputer with arbitrarily large amounts of computing power and a decent understanding of the laws of physics (a 'Laplace demon'  (Laplace, 1814)  capable of probabilistic reasoning), placed 'outside the universe' and computing the future course of events. Paired with an obedient active agent inside the universe with a measurable goal, for which it would act as an advisor, this would also constitute an 'ultimate agent.' Thus in the extreme theoretical case, the Orthogonality thesis seems true. There is only one problem with these agents: they are either impossible in practice (AIXI or Laplace's demon), or require incredibly large amounts of computing resources to work. Let us step down from the theoretical pinnacle and require that these agents could actually exist in our world (still not requiring that we be able or likely to build them). An interesting thought experiment occurs here. We could imagine an AIXI-like super-agent, with all its impractical resources, that is tasked to design and train an AI that could exist in our world, and that would accomplish the super-agent's goals. Using its own vast intelligence, the superagent would therefore design a constrained agent maximally effective at accomplishing those goals in our world. Then this agent would be the highintelligence real-world agent we are looking for. It doesn't matter than the designer is impossible in practice -if the super-agent can succeed in the theoretical thought experiment, then the trained AI can exist in our world. This argument generalizes to other ways of producing the AI. Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things: 1. There cannot exist any efficient real-world algorithm with goal G. 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail. 3. If a human society were highly motivated 7 to design an efficient realworld algorithm with goal G, and were given a million years to do so along with huge amounts of resources, training and knowledge about AI, it must fail. 4. If a high-resource human society were highly motivated to achieve the goal G, then it could not do so (here the human society itself is seen as the algorithm). 5. Same as above, for any hypothetical alien societies. 6. There cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G. 7. There cannot exist any evolutionary or environmental pressures that would evolve a highly efficient real world intelligences following goal G. All of these seem extraordinarily strong claims to make! The last claims all derive from the first, and merely serve to illustrate how strong the first claim actually is. Claim 4, in particular, seems to run counter to everything we know about human nature.
 Orthogonality for Human-level AIs Of course, even if efficient agents could exist for all these goals, that doesn't mean that we could ever build them, even if we could build AIs. In this section, we'll look at the ground for assuming the Orthogonality thesis holds for human-level agents. Since intelligence isn't varying much, the thesis becomes simply: If we could construct human-level AIs at all, then there is extremely little constraint on the final goals that such AIs could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). So, is this true? The arguments in this section are generally independent of each other, and can be summarized as: 1. Some possible AI designs have orthogonality built right into them. 2. AI goals can reach the span of human goals, which is large. 3. Algorithms can be combined to generate an AI with any easily measurable goal. 4. Various algorithmic modifications can be used to further expand the space of possible goals, if needed.
 Utility Functions One classical picture of a rational agent is of an agent with a specific utility function, which it will then act to maximize in expectation. This picture encapsulates the Orthogonality thesis: whatever the utility function, the rational agent will then attempt to maximize it, using the approaches in all cases (planning, analyzing input data, computing expected results). If an AI is built according to this model, with the utility function being prescriptive (given to the AI in a program) rather than descriptive (an abstract formalization of an agent's other preferences), then the thesis would be trivially true: we could simply substitute the utility function for whichever one we desired. However, many putative agent designs are not utility function based, such as neural networks, genetic algorithms, or humans. So from now on we will consider that our agents are not expected utility maximizers with clear and separate utility functions, and look at proving Orthogonality in these harder circumstances.
 The Span of Human Motivations It seems a reasonable assumption that if there exists a human being with particular goals, and we can program an AI, then we can construct a humanlevel AI with similar goals. This is immediately the case if the AI was a whole brain emulation/upload  (Sandberg & Bostrom, 2008) , a digital copy of a specific human mind. Even for more general agents, such as evolved agents, this remains a reasonable thesis. For a start, we know that real-world evolution has produced us, so constructing human-like agents that way is certainly possible. Human minds remain our only real model of general intelligence, and this strongly directs and informs our AI designs, which are likely to be as human-similar as we can make them. Similarly, human goals are the easiest goals for us to understand, hence the easiest to try and implement in AI. Hence it seems likely that we could implement most human goals in the first generation of human-level AIs. So how wide is the space of human motivations 8 ? Our race spans footfetishists, religious saints, serial killers, instinctive accountants, role-players, self-cannibals, firefighters and conceptual artists. The autistic, those with exceptional social skills, the obsessive compulsive and some with splitbrains. Beings of great empathy and the many who used to enjoy torture and executions as public spectacles. 9 It is evident that the space of possible human motivations is vast. 10 For any desire, any particular goal, no matter how niche, 11 pathological, bizarre or extreme, as long as there is a single human who ever had it, we could build and run an AI with the same goal. But with AIs we can go even further. We could take any of these goals as a starting point, make them malleable (as goals are in humans), and push them further out. We could provide the AIs with specific reinforcements to push their goals in extreme directions (reward the saint for ever-more saintly behavior). If the agents are fast enough, we could run whole societies of them with huge varieties of evolutionary or social pressures, to further explore the goal-space. We may also be able to do surgery directly on their goals, to introduce more yet variety. For example, we could take a dedicated utilitarian charity worker obsessed with saving lives in poorer countries (but who doesn't interact, or want to interact, directly with those saved), and replace 'saving lives' with 'maximizing the number of paperclips in the universe' or any similar abstract goal. This is more speculative, of course -but there are other ways of getting similar results.
 Instrumental Goals as Final Goals If someone were to hold a gun to your head, they could make you do almost anything. Certainly there are people who, with a gun at their head, would be willing to do almost anything. A distinction is generally made between instrumental goals and final goals, with the former being seen as simply paths to the latter, and interchangeable with other plausible paths. The gun to your head disrupts the balance: your final goal is simply not to get shot, while your instrumental goals become what the gun holder wants them to be, and you put a great amount of effort into accomplishing the minute details of these instrumental goals. Note that the gun has not changed your level of intelligence or ability. This is relevant because instrumental goals seem to be far more varied in humans than final goals. One can have instrumental goals of filling papers, solving equations, walking dogs, making money, pushing buttons in various sequences, opening doors, enhancing shareholder value, assembling cars, bombing villages or putting sharks into tanks. Or simply doing whatever the guy with gun at our head orders us to do. If we could accept human instrumental goals as AI final goals, we would extend the space of goals quite dramatically. To do so we would want to put the threatened agent, and the gun wielder, together into the same AI. Algorithmically there is nothing extraordinary about this: certain subroutines have certain behaviors depending on the outputs of other subroutines. The 'gun wielder' need not be particularly intelligent: it simply needs to be able to establish whether its goals are being met. If for instance those goals are given by a utility function then all that is required in an automated system that measure progress toward increasing utility and punishes (or erases) the rest of the AI if not. The 'rest of AI' is just required to be a human-level AI which would be susceptible to this kind of pressure. Note that we do not require that it even be close to human in any way, simply that it place a highest value on self-preservation (or on some similar small goal that the 'gun wielder' would have power over). For humans, another similar model is that of a job in a corporation or bureaucracy: in order to achieve the money required for their final goals, some human are willing to perform extreme tasks (organizing the logistics of genocides, weapon design, writing long emotional press releases they don't agree with at all). Again, if the corporation-employee relationship can be captured in a single algorithm, this would generate an intelligent AI whose goal is anything measurable by the 'corporation.' The 'money' could simply be an internal reward channel, perfectly aligning the incentives. If the subagent is anything like a human, they would quickly integrate the other goals into their own motivation, 12 removing the need for the gun wielder/corporation part of the algorithm.
 Noise, Anti-agents and Goal Combination There are further ways of extending the space of goals we could implement in human-level AIs. One simple way is simply to introduce noise: flip a few bits and subroutines, add bugs and get a new agent. Of course, this is likely to cause the agent's intelligence to decrease somewhat, but we have generated new goals. Then, if appropriate, we could use evolution or other improvements to raise the agent's intelligence again; this will likely undo some, but not all of effect of the noise. Or we could use some of the tricks above to make a smarter agent implement the goals of the noise-modified agent. A more extreme example would be to create an anti-agent: an agent whose single goal is to stymie the plans and goals of single given agent. This already happens with vengeful humans, and we would just need to dial it up: have an anti-agent that would do all it can to counter the goals of a given agent, even if that agent doesn't exist ("I don't care that you're dead, I'm still going to despoil your country, because that's what you'd wanted me to not do"). This further extends the space of possible goals. Different agents with different goals can also be combined into a single algorithm. With some algorithmic method for the AIs to negotiate their combined objective and balance the relative importance of their goals, this procedure would construct a single AI with a combined goal system. There would likely be no drop in intelligence/efficiency: committees of two can work very well towards their common goals, especially if there is some automatic penalty for disagreements.
 Further Tricks up the Sleeve This section started by emphasizing the wide space of human goals, and then introduced tricks to push goal systems further beyond these boundaries. The list isn't exhaustive: there are surely more devices and ideas one can use to continue to extend the space of possible goals for human-level AIs. Though this might not be enough to get every goal, we can nearly certainly use these procedures to construct a human-level AI with any human-comprehensible goal. But would the same be true for superhuman AIs?
 Orthogonality for Superhuman AIs We now come to the area where the Orthogonality thesis seems the most vulnerable. It is one thing to have human-level AIs, or abstract superintelligent algorithms created ex nihilo, with certain goals. But if ever the human race were to design a superintelligent AI, there would be some sort of process involved -directed evolution, recursive self-improvement,  13  design by a committee of AIs, or similar -and it seems at least possible that such a process could fail to fully explore the goal-space. The Orthogonality thesis in this context is: If we could construct superintelligent AIs at all, then there is extremely little constraint on the final goals that such AIs could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). There are two counter-theses. The weakest claim is: Incompleteness: there are large categories of goals that no superintelligence designed by us could have. A stronger claim is: Convergence: all human-designed superintelligences would have one of a small set of goals. Here 'small' means 'smaller than the space of current human motivations,' thus very small in comparison with the space of possible AI goals. They should be distinguished; Incompleteness is all that is needed to contradict Orthogonality, but Convergence is often the issue being discussed. Often Convergence is stated in terms of a particular model of metaethics, to which it is assumed all agents will converge (see some of the references in the introduction, or various online texts and argument 14 ).
 No Convergence The plausibility of the convergence thesis is highly connected with the connotations of the terms used in it. "All human-designed rational beings would follow the same morality (or one of small sets of moralities)" sounds plausible; in contrast "all human-designed superefficient algorithms would accomplish the same task" seems ridiculous. To quote an online commentator, how good at playing chess would a chess computer have to be before it started feeding the hungry? Similarly, if there were such a convergence, then all self-improving or constructed superintelligence must fall prey to it, even if it were actively seeking to avoid it. After all, the self-improving lower-level AIs or the designers have certain goals in mind (as we've seen in the previous section, if the designers are AIs themselves, they could have potentially any goals in mind). Obviously, they would be less likely to achieve their goals if these goals were to change as they got more intelligent  (Omohundro, 2008)  (see also N. Bostrom's forthcoming book Superintelligence: Groundwork to a Strategic Analysis of the Machine Intelligence Revolution). The same goes if the superintelligent AI they designed didn't share these goals. Hence the AI designers will be actively trying to prevent such a convergence, if they suspected that one was likely to happen. If for instance their goals were immoral, they would program their AI not to care about morality; they would use every trick up their sleeves to prevent the AI's goals from drifting from their own. So the convergence thesis requires that for the vast majority of goals G: 1. It is possible for a superintelligence to exist with goal G (by section 0). 2. There exists an entity with goal G (by section 0), capable of building a superintelligent AI. 3. Yet any attempt of that entity to build a superintelligent AI with goal G will be a failure, and the superintelligence's goals will converge on some other goal. 4. This is true even if the entity is aware of the convergence and explicitly attempts to avoid it. 5. If the superintelligence were to be constructed by successive self-improvement, then an entity with goal G operating on itself to boost its intelligence is unable to do so in a way that would preserve goal G. This makes the convergence thesis very unlikely. The argument also works against the incompleteness thesis, but in a weaker fashion: it seems more plausible that some types of goals would be unreachable, despite being theoretically possible. There is another interesting aspect of the convergence thesis: these goals G are to emerge, somehow, without them being aimed for or desired. If one accepts that goals aimed for will not be reached, one has to ask why convergence is assumed: why not divergence? Why not assume that though G is aimed for, random accidents or faulty implementation will lead to the AI ending up with one of a much wider array of possible goals, rather than a much narrower one? We won't delve deeper into this, and simply make the point that "superintelligent AIs won't have the goals we want them to have" is therefore not an argument in favor of the convergence thesis.
 Oracles Show the Way If the Orthogonality thesis is wrong, then it implies that Oracles are impossible to build. An Oracle is a superintelligent AI that accurately answers human questions about the world, such as the likely consequences of certain policies and decisions  (Armstrong, Sandberg, & Bostrom, 2012) .  15  If such an Oracle could be built, then we could attach it to a human-level AI with goal G. The human-level AI could then ask the Oracle what the results of different decisions actions could be, and choose the action that best accomplishes G. In this way, the combined system would be a superintelligent AI with goal G. What makes the "no Oracle" implication even more counterintuitive is that any superintelligence must be able to look ahead, design actions, predict the consequences of its actions, and choose the best one available. But the convergence and indifference theses imply that this general skill is one that we can make available only to AIs with certain specific goals. Though agents with those specific goals are capable of doing effective predictions, they automatically lose this ability if their goals were to change.
 Tricking the Controller Just as with human-level AIs, one could construct a superintelligent AI by wedding together a superintelligence with a large motivated committee of human-level AIs dedicated to implementing a goal G, and checking the superintelligence's actions. Thus to deny the Orthogonality thesis requires that one believes that the superintelligence is always capable of tricking this committee, no matter how detailed and thorough their oversight. This argument extends the Orthogonality thesis to moderately superintelligent AIs, or to any situation where there's a diminishing return to intelligence. It only fails if we take AI to be fantastically superhuman: capable of tricking or seducing any collection of human-level beings.
 Temporary Fragments of Algorithms, Fictional Worlds and Extra Tricks These are other tricks that can be used to create an AI with any goals. For any superintelligent AI, there are certain inputs that will make it behave in certain ways. For instance, a human-loving moral AI could be compelled to follow most goals G for a day, if they were rewarded with something sufficiently positive afterwards. But its actions for that one day are the result of a series of inputs to a particular algorithm; if we turned off the AI after that day, we would have accomplished moves towards goal G without having to reward its "true" goals at all. And then we could continue the trick the next day with another copy. For this to fail, it has to be the case that we can create an algorithm which will perform certain actions on certain inputs as long as it isn't turned off afterwards, but that we cannot create an algorithm that does the same thing if it was to be turned off. Another alternative is to create a superintelligent AI that has goals in a fictional world (such as a game or a reward channel) over which we have control. Then we could trade interventions in the fictional world against advice in the real world towards whichever goals we desire.  16  These two arguments may feel weaker than the ones before: they are tricks that may or may not work, depending on the details of the AI's setup. But to deny the Orthogonality thesis requires not only denying that these tricks would ever work, but denying that any tricks or methods that we (or any human-level AIs) could think up, would ever work at controlling the AIs. We need to assume superintelligent AIs cannot be controlled in any way that anyone could think of.
 In Summary Denying the Orthogonality thesis thus requires that: 1. There are goals G, such that an entity an entity with goal G cannot build a superintelligence with the same goal. This despite the fact that the entity can build a superintelligence, and that a superintelligence with goal G can exist. 2. Goal G cannot arise accidentally from some other origin, and errors and ambiguities do not significantly broaden the space of possible goals. 3. Oracles and general purpose planners cannot be built. Superintelligent AIs cannot have their planning abilities repurposed. 4. A superintelligence will always be able to trick its overseers, no matter how careful and cunning they are. 5. Though we can create an algorithm that does certain actions if it was not to be turned off after, we cannot create an algorithm that does the same thing if it was to be turned off after. 6. An AI will always come to care intrinsically about things in the real world. 7. No tricks can be thought up to successfully constrain the AI's goals: superintelligent AIs simply cannot be controlled.
 Conclusion It is not enough to know that an agent is intelligent (or superintelligent). If we want to know something about its final goals, about the actions it will be willing to undertake to achieve them, and hence its ultimate impact on the world, there are no shortcuts. We have to directly figure out what these goals are (or figure out a way of programming them in), and cannot rely on the agent being moral just because it is superintelligent/superefficient. 			 © Stuart Armstrong

Abstract: In his paper "The Superintelligent Will," Nick Bostrom formalized the Orthogonality thesis: the idea that the final goals and intelligence levels of artificial agents are independent of each other. This paper presents arguments for a (narrower) version of the thesis. It proceeds through three steps. First it shows that superintelligent agents with essentially arbitrary goals can exist in our universeboth as theoretical impractical agents such as AIXI and as physically possible realworld agents. Then it argues that if humans are capable of building human-level artificial intelligences, we can build them with an extremely broad spectrum of goals. Finally it shows that the same result holds for any superintelligent agent we could directly or indirectly build. This result is relevant for arguments about the potential motivations of future agents: knowing an artificial agent is of high intelligence does not allow us to presume that it will be moral, we will need to figure out its goals directly.


An Introduction to Löb's Theorem in MIRI Research

Introduction This expository note is devoted to answering the following question: why do many MIRI research papers cite a 1955 theorem of Martin Löb  [12] , and indeed, why does MIRI focus so heavily on mathematical logic? The short answer is that this theorem illustrates the basic kind of self-reference involved when an algorithm considers its own output as part of the universe, and it is thus germane to many kinds of research involving self-modifying agents, especially when formal verification is involved or when we want to cleanly prove things in model problems. For a longer answer, well, welcome! I'll assume you have some background doing mathematical proofs and writing computer programs, but I won't assume any background in mathematical logic beyond knowing the usual logical operators, nor that you've even heard of Löb's Theorem before. To motivate the mathematical sections that follow, let's consider a toy problem. Say that we've designed Deep Thought 1.0, an AI that reasons about its possible actions and only takes actions that it can show to have good consequences on balance. One such action is designing a successor, Deep Thought 2.0, which has improved deductive abilities. But if Deep Thought 1.0 (hereafter called DT1) is to actually build Deep Thought 2.0 (DT2), DT1 must first conclude that building DT2 will have good consequences on balance. There's an immediate difficulty-the consequences of building DT2 include the actions that DT2 takes; but since DT2 has increased deductive powers, DT1 can't actually figure out what actions DT2 is going to take. Naively, it seems as if it should be enough for DT1 to know that DT2 has the same goals as DT1, that DT2's deductions are reliable, and that DT2 only takes actions that it deduces to have good consequences on balance. Unfortunately, the straightforward way of setting up such a model fails catastrophically on the innocent-sounding step "DT1 knows that DT2's deductions are reliable". If we try and model DT1 and DT2 as proving statements in two formal systems (one stronger than the other), then the only way that DT1 can make such a statement about DT2's reliability is if DT1 (and thus both) are in fact unreliable! This counterintuitive roadblock is best explained by reference to Löb's theorem, and so we turn to the background of that theorem. 2 Crash Course in Löb's Theorem
 Gödelian self-reference and quining programs Löb's Theorem makes use of the machinery of Kurt Gödel's incompleteness theorems  [10] , so we will discuss those first. Informally, Gödel found a way to import self-reference into a mathematical system that was simply trying to talk about properties of natural numbers, and then pointed out the odd consequences of a mathematical statement that asserts its own unprovability. One (anachronistic) way of stating Gödel's key insight is that you can use computer programs to search for proofs, and you can prove statements about computer programs. If we think about any conjecture in mathematics that can be stated in terms of arithmetic, you can write a rather simple program that loops over all possible strings, checks whether any of them is a valid proof of the conjecture, and halts if and only if it finds one. Thus, instead of trying to prove that conjecture directly, we could instead try to show (or prove) that the program halts. Now, this generally doesn't make things easier at all, since we're just restating the same problem in a more complicated way, and actually looping over all strings is basically the worst way to try and find proofs of theorems. However, this reformulation makes it more intuitive that we can embed self-reference in mathematics, because we can embed self-reference in computer code! The kind of self-reference I'm talking about is called a quine: a program that is able to reproduce its own source code without taking any inputs. One way to do this is for the program to include a string with a variable, and for it to replace that variable with the original string itself, such that the resulting expanded string is the entire source code of the program itself. Exercise. Write a quining program in your favorite language. No fair copying one from the Wikipedia article, and no fair making calls to external programs or the filesystem. In addition to quines that merely print their source code, programs can be made which perform arbitrary tasks using their own source code. Indeed, one prominent example of a quining program that does other tasks is Ken Thompson's famous C compiler Trojan horse  [15] , which uses the quining trick to avoid ever being visible outside of machine code. Thus we can have a program G which refers to itself in this way, and searches for proofs in arithmetic related to its own source code. In particular, we consider G which searches for a proof of the statement "G runs forever", and halts if and only if it succeeds at finding one. Now, we claim that G never finds a proof, and also that we can never prove that G runs forever! For if we could prove that G ran forever, then G would find that proof, and it would halt-and we could prove that it halted, and thereby produce a contradiction! On the other hand, if G actually halted after some finite number of steps, then the number at which it halts encodes a proof that G runs forever, which again leads to a contradiction!  1  It seems silly to ask whether we could prove that G halts, given that G actually runs forever. But it actually wouldn't be a contradiction if we asserted that G actually halted, so long as we didn't say anything about how long it took! It's only a claim like "G halts in fewer than a googolplex steps" that would be an actual contradiction. It turns out that we could add either "G never halts" or "G halts" (but not both!) as a new axiom of arithmetic, and not introduce a contradiction by doing so. We express this by saying that G is undecidable, and therefore we've proved Theorem 2.1 (Gödel's First Incompleteness Theorem) If the theory of arithmetic is consistent, then there exist undecidable statements in the theory of arithmetic. Remark. Gödel's Theorem is a bit different from the Halting Problem; the latter shows there's no one program X that can tell whether every program Y halts or not, but of course there may be particular programs X that tell you this fact for particular programs Y . But this is saying that there's no program X that can tell you definitively whether the program G halts or runs forever. So Gödel found that we can write undecidable statements about properties of natural numbers, and furthermore showed that adding new axioms won't fix it, since you can repeat the process with the new rule system for whether a statement is a theorem. (There's only one loophole, and it's not a very exciting one: if we use inconsistent axioms, then everything is a theorem, thanks to the Principle of Explosion, and therefore everything is decidable). Now we're not here to bury mathematical logic, but to use it. So from Gödel's Theorem we move on to...
 Löb's Theorem Löb's Theorem  [12]  takes the same framework as Gödel's First Incompleteness Theorem, and constructs programs of the following sort: • Let X be any logical statement (the sort of thing that could be proved or disproved). • Now let ProofSeeker(X) be the program that searches all possible proofs, and halts if and only if one of them is a valid proof of the statement X. • Finally, let L(X) be the statement "if ProofSeeker(X) halts, then X". Let's ponder for a moment whether L(X) should be true or not, using three different kinds of statements X. • If X is a provable statement, for example "2 + 2 = 4", then ProofSeeker(X) halts, and L(X) is "if [true thing], then [true thing]", which is a valid statement. • If X is disprovable, for example the statement "2 + 2 = 5", then ProofSeeker(X) does not halt, and L(X) is "if [false thing], then [false thing]", which is also a valid statement. • If X is neither provable nor disprovable, for example Gödel's statement G, then again ProofSeeker(X) does not halt, and L(X) is "if [false thing], then [maybe true thing]", which is also a true statement (remember your propositional calculus). So it seems like L(X) is always true. And it would certainly be handy if L(X) were provable for every X: for instance, you could use the second case above to show that mathematics proves no contradictions! Because if we could prove "if ProofSeeker("2 + 2 = 5") halts, then 2+2 = 5", then we would have the contrapositive "if 2+2 = 5, then ProofSeeker("2+2 = 5") never halts", and since we can prove 2 + 2 = 5, then we could prove that there is no contradictory proof of 2 + 2 = 5. Alas, that proof of mathematical consistency is too good to be true. As in the case of Gödel's Theorem, something being true is no guarantee of it being provable. And in fact, we find that L(X) is only provable in the first of the three cases above: Theorem 2.2 (Löb's Theorem) For all statements X, if L(X) is provable, then X is provable. (There is a neatly presented formal proof of this theorem at The Cartoon Guide to Löb's Theorem  [16] .) So in particular, if we could prove that mathematics would never prove a contradiction, then in fact mathematics would prove that contradiction! (Note that by the Principle of Explosion, this is indeed a possible state of affairs: if your mathematical axioms lead to a contradiction, then they can prove every statement in the language, including the statement that your mathematical axioms don't lead to a contradiction!) Thus the only systems that prove their own consistency are the inconsistent ones; incidentally, this is precisely Gödel's Second Incompleteness Theorem, although he originally proved it without Löb's Theorem. Remark. If you're interested in going deeper on these topics, Computability and Logic by Boolos, Burgess, and Jeffrey  [5]  is a good reference. Remark. There is also a version of Löb's Theorem for bounded proof searches, in the sense of "look through all formulas of length ≤ N and see if any of them are a proof of φ", and it controls the length of the proof of φ in terms of the length of the proof of φ → φ. In the limit of arbitrarily large computational resources, the phenomena we care about happen in the same way that they do in the case of infinite computation (i.e. access to halting oracles), and so we will generally discuss the latter case because the proofs are simpler and clearer.
 Direct Uses of Löb's Theorem in MIRI Research We can now exhibit three simple cases where Löb's Theorem comes up in MIRI research topics: one where it forms an unexpected obstacle to justifying self-modifications, one where it neatly enables mutual cooperation in a nonstandard Prisoner's Dilemma setup, and one where it frustrates a naive decision algorithm.
 "The Löbstacle" Let's return to the problem we discussed in the Introduction: Deep Thought 1.0 wants to verify that switching on its successor, Deep Thought 2.0, will have good consequences. Because DT2 has better deductive capacities, DT1 cannot deduce exactly what actions DT2 will take, but it does know that DT2 has the same utility function, and that it too will only take actions it deduces to be good. Intuitively, this should be enough for DT1 to "trust" DT2, to say that whatever DT2 does, DT2 must have deduced that to be good, and therefore it must actually be good. But that last clause is analogous to the Löbian statement L(X): "if the action is deduced to be good, then it must actually be good"! And therefore DT1 cannot generally prove that clause (unless its reasoning is inconsistent), since then it could prove that every action is good. This is not just an analogy; when we consider a simple mathematical model of a selfmodifying agent that uses proofs in some consistent formal system to justify its actions, that agent has precisely this problem. In models that presume infinite computing power and represent different deductive powers with different axiom systems, a simple agent with a utility function will only create successors whose formal systems are strictly weaker than its own, since only those are fully trusted by the current system. There are a number of partial and potential remedies to this "Löbstacle", some of them more appealing than others. For more details, see the MIRI preprints Tiling Agents for Self-Modifying AI, and the Löbian Obstacle  [18]  and Problems of self-reference in self-improving space-time embedded intelligence  [9] .
 Löbian cooperation The second topic concerns a variation on the usual Prisoner's Dilemma. Rather than playing directly, you write an algorithm to play against other algorithms on your behalf, as in Robert Axelrod's famous algorithmic Prisoner's Dilemma tournament  [1] . However, instead of that setup, in which the algorithms play iterated games against one another, in this case your algorithm and theirs get to read the opponent's source code, calculate for as long as they like, and then play only once. Using quining (recall Section 2.1), one can write a program that cooperates if and only if the opponent's source code is identical to its own. However, there are many ways to write such programs, and different ones will not cooperate with each other; this is therefore a fragile form of mutual cooperation. There's a different algorithm which (at the cost of using lots of computation) avoids that fragility. We call this agent FairBot, and it operates as follows: • Both FairBot and its opponent X are functions of one argument, which take the opponent's source code and output either C or D. • When FairBot() is called on the input X, it searches through all proofs of length ≤ N to see whether any are valid proofs of the statement "X(FairBot)= C". If yes, then it outputs C; if no, then it outputs D. (Here N is a parameter that doesn't depend on X; we'll think of it as some extremely large number. The only reason we have that parameter at all is so that our algorithm does in fact always return an output in finite time.) Some things are clear from the definition of FairBot. One is that it is capable of cooperation: if X is a simple algorithm that always returns C, then FairBot will return C as well. Another is that (unless arithmetic is inconsistent) FairBot will never be exploited (cooperate while its opponent defects). What is less immediately clear is the outcome when FairBot plays against itself. Intuitively, it seems like both mutual cooperation and mutual defection are stable fixed points of the situation. However, a Löbian statement breaks the deadlock in favor of cooperation! To see this, ignore for the moment the parameter N . Then if we consider the statement L("FairBot(FairBot)= C"), we find that it follows directly from the code of FairBot. For if there is a proof that FairBot(FairBot)= C, then FairBot will discover that proof and output C. But then, by Löb's Theorem, there must be an actual proof of the statement "FairBot(FairBot)= C"! 2 As mentioned in Section 2.2, there is a quantitatively bounded version of Löb's Theorem such that this works even with the parameter N , for sufficiently large values of N . Furthermore, this form of cooperation is much more robust: two such programs don't need to figure out that they are functionally identical, they only need to search for proofs about one another's output. Moreover, even algorithms that are not functionally identical to FairBot can achieve mutual cooperation with it. The FairBot algorithm is due to Vladimir Slepnev; for more on Löbian cooperation, see the MIRI paper Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem  [11] . We will return to this topic in Section 7.1 when we have developed a few more tools.
 Spurious counterfactuals The third topic concerns an unexpected issue that comes up when the agent is a part of the same universe that the agent is proving theorems about. The setting is a pure computational universe U () which takes no input, and which just runs and eventually outputs a number. Within that universe there is an agent A() which knows the code for U (), which does some computations as part of the natural order of the universe and then outputs something. This may seem like a rather extreme setup, and it is: we want to see what we can say about agents that know the source code of the universe and wield arbitrarily large computational resources, because it is often easier to show what happens with such agents than what happens with more realistic, messy, and bounded agents. Moreover, any obstacles we discover in the ideal setting are likely to correspond in the real setting to obstacles that can't be overcome merely by adding more knowledge and computing power to the system. Let's first consider a really simple sort of universe (Algorithm 1), where A() must try to figure out whether to open door number 1, door number 2, or neither. Algorithm 1: U () if A() = 1 then return 100 else if A() = 2 then return 1 else return −∞ end This requires us to specify the function A(). Now of course we could consider a function A() hard-coded to return 1, but that's not especially interesting. Instead, we'll write one (Algorithm 2) which tries to deduce which door (if either) to open, by looking through proofs of arithmetical statements. It seems obvious that there should be proofs of A() = 1 → U () = 100 and A() = 2 → U () = 1 whose Gödel numbers are smaller than a googolplex, and therefore A should correctly prove those statements and choose Door 1. Algorithm 2: A() Door1 ← −∞; Door2 ← −∞; n = 1; while Door1 + Door2 = −∞ and n < 10 10 100 do if n encodes a valid proof of "A() = 1 → U () = c" for some c then Door1 ← c; if n encodes a valid proof of "A() = 2 → U () = c" for some c then Door2 ← c; n+ = 1; end if Door1 ≥ Door2 then return 1 else return 2 end But there's something very strange that can happen instead: A could prove A() = 2 → U () = 1, and then prove a "spurious counterfactual" such as A() = 1 → U () = −1, which then makes A() stop looking early and select door number 2! This wouldn't be a contradiction, because if A() = 2, then A() = 1 → U () = −1 is formally true  3  . But why should it actually happen? It's clearest to see in the case where I can make one innocent-seeming change to A(): I ask that before it starts searching through all n < 10 10 100 to see if they encode proofs, I want it to check two particular proofs. The first is the simple proof that A() = 2 → U () = 1, and the second is a statement φ. Now if φ really is a proof of the spurious counterfactual A() = 1 → U () = −1, it's clear from the rest of the source code of A() that it will break the while loop and choose Door 2. In fact, we can formalize that reasoning: there's a proof that if φ is a proof of A() = 1 → U () = −1, then in fact it's true that A() = 2 and thus A() = 1 → U () = −1. But this is almost a Löbian statement, just with the addition of a specific φ rather than an existential quantifier! By the machinery of quining, we can find a φ that serves our purposes, so it works just as before: since the Löbian statement is provable, so is the conclusion A() = 1 → U () = −1, by means of that particular φ. Even without that malicious change, these kinds of agents and universes are susceptible to spurious counterfactuals. In Section 7.2, we'll consider a similar setup, but with a more careful ordering of which proofs it pays attention to, and that agent can be shown to make the correct decisions (given enough deductive power). The idea was originally due to Benja Fallenstein; Tsvi Benson-Tilson's paper UDT with Known Search Order  [3]  discusses spurious counterfactuals and a different response to them.
 Crash Course in Model Theory For the later topics, we'll need some additional background in mathematical logic. Let's start out by just talking formally about the natural numbers with addition and multiplication.
 Axioms and theories We begin with a language of symbols, and we don't start out assuming anything about what they mean. We'll take some symbols of the propositional calculus, set theory, arithmetic, and three special symbols: {(, ), ∧, ∨, ¬, →, ↔, ∈, ∀, ∃, =, +, •, O, S, N} Here we'll think of N as standing in for the set of natural numbers, O as standing in for zero, and S as standing in for the successor (so that SO represents 1, SSO represents 2, etc.). That way we don't need rules for dealing with infinitely many number symbols, or rules for dealing with digits. We'll also add an infinite family of symbols like x and y for variables (so long as they're quantified over with ∀ or ∃). Finally, we'll use letters like φ to stand in for a formula within the language, but note that φ is not itself part of the language, so we can't say things like ∀φ. We'll import the standard rules of the propositional calculus and set theory as applied to the appropriate symbols (for instance, whenever φ ∧ ψ is a theorem, then both φ and ψ are theorems), but we won't yet assume any properties of +, •, O, S, or N. That's because we'll want to pick our own axioms about arithmetic on the natural numbers! This creates a theory, the set of all theorems that follow from the axioms; different sets of axioms can give rise to different theories. So let's take some true statements about the natural numbers, make them axioms of our logical system, and see what kind of theory we get. (Parenthetical comments are included for our intuition, but are not actually a part of the theory.) A theory can contain contradictions, of course: some theorem φ such that ¬φ is also a theorem. So how can we tell whether the theory built on these axioms contains any contradictions? We could try proving things and see if we ever find a contradiction, but as there are infinitely many theorems, we couldn't be sure that we just hadn't found a contradiction yet. 1. O ∈ N (zero is a natural number) 2. ∀x ∈ N Sx ∈ N (if x is a natural number, so is its successor) 3. ∀x ∈ N∀y ∈ N (x + y ∈ N) ∧ (x • y ∈ N) (the But there's another thing we could do: exhibit an example of a set for which all of the axioms hold! That serves as conclusive evidence that the theory does not contain contradictions, because φ is either actually true or actually false for that specific set, and so at most one of φ and ¬φ can actually be a theorem. We need to do a little more than exhibit an object: we need to have a correspondence between the symbols in the language and the parts of the object. So for instance, we identify the symbol O with the number 0, SO with 1, and so on 4 ; we identify N with the full set of natural numbers N; and we identify addition and multiplication with the usual operations on N. We call such an object a model, and such a correspondence an interpretation of the theory. However, the same theory can have many models, some of them not at all what you were thinking of when you made the axioms...
 Alternative and nonstandard models For instance, an alternate model of the theory above is the set with a single element "Alice"; we identify O with Alice, and then SO with Alice as well (so SO = O), and so on; we declare that Alice+Alice=Alice•Alice=Alice; and then we identify N with the set {Alice}. Note that all of the axioms above are true of this alternate model, even if their intuitive meanings aren't! Fine, let's patch this. We can't add an axiom directly saying it's not this particular model, since Alice isn't part of the language of the theory itself. But we can add an axiom saying SO = O, which is true of the model of the natural numbers we want, but not true of the Alice model. Well, we excluded that alternate model, but there are still others we haven't excluded. In particular, we can consider arithmetic mod n as a model for any n! In order to exclude these, we could add axioms saying SSO = O, SSSO = O, etc, or we could be more economical and just use the axiom ∀x ∈ N Sx = O. (The resulting theory is known as Robinson arithmetic, and is interesting in its own right.) Have we eliminated all alternate models from this theory? Hardly! We're just getting started. Consider the model where N refers to the natural numbers combined with one additional entity, "Bob"; we identify S(Bob)=Bob, Bob•0 = 0, Bob•x =Bob for all x = 0, and Bob+x =Bob for all x. Then we again see that this satisfies all of our axioms so far. We could patch this with the axiom ∀x ∈ N Sx = x, but again we can build more and more complicated alternate models. (One particularly interesting model for Robinson arithmetic consists of all polynomials with integer coefficients whose leading term is nonnegative.) It turns out that what we're missing from the natural numbers is the principle of induction. (Indeed, with Robinson arithmetic you can't even prove that addition is commutative!) In order to express induction in the language (which doesn't have variables for properties, only for numbers), we must resort to an infinite family of new axioms: for every logical formula φ(x) (with no free variables except for x), we want to add the axiom (φ(0) ∧ (∀x ∈ N φ(x) → φ(Sx))) → ∀y ∈ N φ(y).  5  With all of that completed, we've got the axioms for the theory of Peano Arithmetic. Now are we done with alternative models? Of course not! Remember how Gödel's self-referential formula G was undecidable? There are models of Peano arithmetic where G holds, and other models where G fails to hold. The models where G holds include our standard intuitive model of the natural numbers, since as we discussed before, G definitely does not halt at any finite number. But then, what kind of model of Peano Arithmetic could G fail to halt on? It may help to think of the model of N with Bob. If we do Gödel's construction in Robinson Arithmetic, there will be no contradiction if we assert that the special extra number Bob in fact satisfies the property that the formula G is checking (the property which represents "this is a valid proof that no number is a valid proof of G"). This is okay, because Bob doesn't actually have a representation in terms of lots of S's and an O, so this assertion can't be used to actually construct a finite sequence of logical formulas that comprise a proof that no number is a valid proof of G. This loophole suffices to maintain consistency for the new system! We're using Peano Arithmetic, though, so our nonstandard models will be weirder than the natural numbers plus Bob. The key to understanding these is that G never halts at any finite number, but we can't actually define in our formal language what "finite" means. The nonstandard models of Peano Arithmetic are those which have all the usual numbers but also lots of extra numbers that are "too large" to ever be written as lots of S's followed by an O, but which nonetheless are swept along in any inductive statement. (I still don't know quite how to visualize this model, but you can rigorously construct it using set theory.) And again, a "number" that can't actually be written in S's and an O can be asserted to represent a valid proof, but that proof can't be extracted from the number and used against itself, so it all works out consistently. Remark. Since nonstandard models of arithmetic contain numbers larger than any of the usual natural numbers, they can be used in other areas of mathematics. If you take the reciprocal of one of those numbers, you get an infinitesimal; thus you can use these to define the concepts of calculus without using limits! This is known as nonstandard analysis. And it doesn't end there! As I mentioned before, we can add either G or ¬G as an extra axiom of Peano Arithmetic, and since adding a new axiom changes the rules for what counts as a valid proof, we can redo Gödel's construction in our new formal system, and have statements that are undecidable given the new rules. Or we can directly talk about the consistency of Peano Arithmetic (by making a statement that asserts that there is no proof in PA that 0 = 1), and the consistency of the system with all the rules of Peano Arithmetic plus the axiom that PA is consistent, and so on. (Ponder for a moment the formal system which has all the axioms of PA, plus the axiom that PA is consistent, plus the axiom that "the system which has all the axioms of PA, plus the axiom that PA is consistent" is inconsistent. As it turns out, this is a perfectly consistent system 6 !) We'll work with a particular hierarchy of such formal systems in Section 6, but before then, we'll discuss another MIRI paper that relates to theories and models. Remark. Again, Computability and Logic by Boolos, Burgess, and Jeffrey  [5]  covers all of this material at a much deeper level.
 Uses of Model Theory in MIRI Research
 Reflection in probabilistic logic The topic of the first MIRI mathematical workshop paper, Definability of Truth in Probabilistic Logic  [7] , isn't directly entwined with Löb's Theorem. But we've just covered the necessary background to understand it, and we'll need another ingredient before getting to the other MIRI papers in these notes, so we might as well detour here! This paper concerns probabilistic logic: assigning probabilities in [0, 1] to logical statements, rather than just assigning them the labels "provable", "disprovable 7 ", and "undecidable". We'll want to make sure we assign the value 1 to all provable statements and 0 to all disprovable statements, and that we assign values to the undecidable statements in a consistent way (so if φ and ψ are undecidable but φ → ψ is provable, then the probability we assign to φ should be less than or equal to the probability we assign to ψ). Why is this an important topic for MIRI to study? Probabilistic logic is an interesting, clean, and rich model for Bayesian reasoning and for bounded inference. Consider an AI that cannot deduce with certainty whether P = N P ; knowing it one way or the other should certainly change its actions (for instance, its cryptographic precautions against other agents), and so the most sensible way to proceed seems to be assigning it a tentative probability based on the available evidence, using that probability to do consequentialist calculations, and updating it in the light of newer evidence. So, in order to understand bounded reasoning, we might start with the nice and symmetrical (if unrealistic) case of a coherent probability assignment over all logical statements. So let's consider a Gödelian obstacle that bedevils logic, and see if it looks different within probabilistic logic. That obstacle is Tarski's undefinability of truth  [14] . OK, so we've previously established that some statements are undecidable, and in fact, undecidable statements hold in some models of a theory but not others. We might want to endorse some particular model as the "true" one (for instance, our standard model of the natural numbers, without all of those weird nonstandard numbers), and say that logical statements are true if they hold in that model and false if they don't. This truth predicate exists outside the language, and so the logical statements can't talk about the truth predicate, only about weaker notions like provability. All that is perfectly fine, thus far. The trouble comes when we try to construct a language that contains its own truth predicate T , either by finding a formula for it using the previously existing symbols, or by directly adding a new symbol with some rules of usage, such that for all φ, it's true that T (φ) ↔ φ. Either approach is doomed, by an argument that should seem familiar by now: there exists  7  By this, we mean statements whose negation is provable, not statements which cannot be proven. a sentence X that refers to itself by quining, so that it's provable that X ↔ T (¬X). And now, clearly, neither T (X) nor T (¬X) can hold.
 All right, so what changes when you incorporate probabilities? Let's introduce a symbol P which represents the probability of a logical statement. We'd like P to satisfy some mathematical coherence properties, like P (φ ∧ ψ) ≤ P (φ), and we'd also like P to be able to discuss itself. Now what happens when we consider the statement Y constructed such that Y ↔ (P (Y ) < 0.5)? Does this create a contradiction? It depends on what kind of statements P is allowed to make about itself! Namely, if P isn't allowed to make exact statements about its own values, but only arbitrarily precise approximations, then everything can work out consistently. Namely, in the place of the failed reflection principle T (T (φ) ↔ φ), we take the reflection principle ∀φ∀a, b ∈ Q (a < P (φ) < b) → (P (a < P (φ) < b) = 1) . (5.1) In the paper, they prove that with this reflection principle and the natural coherence conditions, there indeed is a probability valuation that works for the values of P . How does this avoid a collision with the statement Y above? It turns out that the true value of P (Y ) is 0.5, but P isn't able to know whether P (Y ) is exactly 0.5, or slightly above it, or slightly below it, and so P (P (Y ) < 0.5) is 0.5 as well, and so on. For any rational > 0, you can prove with certainty that P (0.5 − < P (Y ) < 0.5 + ) = 1, but you can't get from this anything sharp enough to produce a contradiction; in particular, you can't prove the generalization that ∀ > 0 P (0.5 − < P (Y ) < 0.5 + ) = 1. Remark. Nonstandard arithmetic isn't needed for this result, but it can shed some light on the fact that P (0.5 − < P (Y ) < 0.5 + ) = 1 for any rational . Since some models of N include nonstandard natural numbers, P can't rule out the possibility that reciprocals of nonstandard natural numbers (infinitesimals) exist, and so P (Y ) may as well be imagined as 0.5 plus or minus an infinitesimal...
 Remark. Since the result in the Definability of Truth paper is nonconstructive, it's worth noting that there's a followup paper by Christiano on computationally tractable approximations of probabilistic logic: Non-Omniscience, Probabilistic Inference, and Metamathematics  [6] .
 Crash Course in Gödel-Löb Modal Logic The final topics in this survey require some topics outside the usual first course in mathematical logic, namely the Gödel-Löb modal logic of provability. This modal logic captures exactly the parts of Peano Arithmetic that relate to self-reference, and by leaving out the rest, it has a pleasingly simple structure and plenty of powerful tools for analysis. Thus it's a great setting for model problems in decision theory, as we shall see.
 The modal logic of provability You may have heard of modal logic before, in the context of philosophy: Aristotle defined ways of arguing about which facts are necessary or possible rather than merely true or false. By the twentieth century, people had formalized various kinds of modal logic, where you take the usual language of propositional logic and add the symbols and , as well as new axioms and rules incorporating them. φ is usually interpreted as the statement "it is necessary that φ", and φ as "it is possible that φ". (We actually only need , since in all modal logics of interest to us, φ ↔ ¬ ¬φ.) Philosophers have tried out various sets of axioms, mostly in order to write intimidatingly technical-looking arguments for their preferred metaphysical conclusions 8 , but we're more interested in a particular modal logic that constitutes a perfectly rigorous-and quite useful-reflection of Löbian phenomena in Peano Arithmetic and other such systems. This is the Gödel-Löb modal logic (alternately, 'the modal logic of provability' or simply 'provability logic'), which we will denote as GL. We can construct GL in two different ways. First, we can let φ represent the formula of Peano Arithmetic that asserts that φ is provable in Peano Arithmetic, and then restrict ourselves to the formulas and proofs that use only , the logical operators, and Boolean variables (including the constants for true and ⊥ for false, but no numbers, arithmetical operations, or quantifiers). Or, equivalently, we can start with those elements of the language, and then add the following axioms and rules: • All tautologies of the propositional calculus are axioms of GL, including tautologies where an expression including has been substituted for a variable (for instance, p ∨ ¬ p is an axiom). • Modus Ponens Rule: if the expressions A and A → B are theorems, then so is B. • Distribution Axioms: for any expressions A and B in the language, we have the axiom (A → B) → ( A → B). • Generalization Rule: if the expression A is a theorem, then so is A. • Gödel-Löb Axioms: for any expression A, we have the axiom ( A → A) → A. It's a nontrivial theorem that these approaches give us the same modal logic! So GL really does capture the self-referential parts of Peano Arithmetic, while leaving aside the arithmetical parts; for instance, the statement that Peano Arithmetic is consistent is simply ¬ ⊥. Exercise. Show, using the rules of GL, that ⊥ ↔ p; recall that p = ¬ ¬p. Informally, that is, no statement can be proven consistent with Peano Arithmetic unless Peano Arithmetic is inconsistent. [Hint: First prove that ⊥ → p, then that p → ⊥. For the latter, it will help to rewrite ¬ ⊥ as ⊥ → ⊥, then consider the Löbian axiom ( ⊥ → ⊥) → ⊥.] While every specific deduction we would want to make in GL can be made via formal manipulations in the system, this is a computationally intractable way to handle deduction. (Deducing whether a given modal statement is a theorem of GL is NP-complete in general, just as it is in Peano Arithmetic.) However, there are some special cases where there are efficient algorithms for deducing provability in GL, and these happen to include cases of direct interest to decision theory...
 Fixed points of modal statements In Section 3.2, we figured out what happens when two different programs try to prove theorems about one another. In that case, the programs were simple enough that we could directly see where to apply Löb's Theorem, but in general we'd like to be able to handle more complicated phenomena. Fortunately, modal logic comes equipped with a powerful tool for doing precisely that: the theory of fixed points for modal sentences. In Peano Arithmetic, the Gödel statement G refers to itself via quining, in order to claim that it cannot be proved in PA; in GL, G corresponds to the formula p ↔ ¬ p. Similarly, we can have all sorts of formulas that refer to themselves and each other by quining, and these are represented by formulas p ↔ φ(p, q 1 , . . . , q k ) that are modalized in p: every occurrence of p in φ happens within the scope of some . (After all, quining can only refer to the provability of the statement's own Gödel number, not directly to the statement itself!) As it happens, whenever you have this setup in GL, with p equivalent to a formula that is modalized in p, then p is equivalent to some other formula which doesn't use p at all! For example, the Gödel statement is equivalent to the inconsistency of arithmetic: (p ↔ ¬p) ↔ (p ↔ ⊥). In the case of one-variable formulas p ↔ φ(p) with φ(p) modalized in p, there's a neat tool that helps us calculate these fixed points (where the result will involve no variables at all, just logical connectives, , , and ⊥). First, for many modal logics including GL, there's a corresponding class of sets and relations (called Kripke frames) such that a formula is provable in the modal logic if and only if a corresponding property holds for every Kripke frame in that class. And secondly, in the special case of sentences without variables in GL, we can reduce to checking a linear hierarchy corresponding to a world in which ⊥ holds, a world in which ⊥ ∧ ¬ ⊥ holds, and so on up the ladder of n+1 ⊥ ∧ ¬ n ⊥ for each n 9 . If you'll grant me that result (known as an World p p ¬p (p → ¬p) p → (p → ¬p) p p → ¬p ⊥ 1 1 1 1 1 1 1 ⊥ ∧ ¬ ⊥ 1 1 0 1 1 1 0 3 ⊥ ∧ ¬ 2 ⊥ ? ? ? ? ? ? ? Continuing with the second row, we find that p flips to false: World p p ¬p (p → ¬p) p → (p → ¬p) p p → ¬p ⊥ 1 1 1 1 1 1 1 ⊥ ∧ ¬ ⊥ 1 1 0 1 1 1 0 3 ⊥ ∧ ¬ 2 ⊥ 1 1 0 0 0 0 1 4 ⊥ ∧ ¬ 3 ⊥ ? ? ? ? ? ? ? And with the third row, when considering ¬p, it is important to remember that the lemma above requires A to be true in all previous rows, so even though ¬p is now true, ¬p remains false. We will fast-forward through the next few rows: World p p ¬p (p → ¬p) p → (p → ¬p) p p → ¬p ⊥ 1 1 1 1 1 1 1 ⊥ ∧ ¬ ⊥ 1 1 0 1 1 1 0 3 ⊥ ∧ ¬ 2 ⊥ 1 1 0 0 0 0 1 4 ⊥ ∧ ¬ 3 ⊥ 0 1 0 0 0 0 1 5 ⊥ ∧ ¬ 4 ⊥ 0 0 0 0 1 1 0 6 ⊥ ∧ ¬ 5 ⊥ 0 0 0 0 1 1 0 And here it stabilizes: every further row is identical  11  . And now that we've found the truth values of p, if we find a constant formula with the same truth values, then by the adequacy result I mentioned in passing, there must be a proof of equivalence in GL! We can get such a formula by taking Boolean combinations of the formulas that define the worlds (since the truth table stabilizes eventually, one only needs a finite combination of these); in the present case, that formula is ¬( 3 ⊥∧¬ 2 ⊥)∧¬( 4 ⊥∧¬ 3 ⊥), or equivalently 4 ⊥ → 2 ⊥. So in the end, we have the fixed point (p ↔ ( p → (p → ¬p))) ↔ (p ↔ ( 4 ⊥ → 2 ⊥)), and our algorithm for finding it is polynomial in the complexity of the statement (in fact, quadratic: we add columns for sub-expressions, and there's a linear bound for how many rows we need to calculate). Remark. We demonstrated the fixed-point algorithm only for one-variable expressions, but you can also take a statement of the form p ↔ φ(p, q 1 , . . . , q k ), where φ is modalized in p (not necessarily in any of the q i ), and find an equivalent expression p ↔ ψ(q 1 , . . . , q k ). The machinery for this is more difficult, as one needs to deal with more complicated Kripke frames (in the general case of GL, a Kripke frame is a set equipped with a transitive, irreflexive, well-founded relation), but it can be mechanized as well. I swept quite a lot under the rug in this section; if you want to get to know the topic on a much sounder basis, read The Logic of Provability  [4]  by Boolos. (This is an advanced textbook, so wait until after your first course in mathematical logic!) 7 Uses of Gödel-Löb Modal Logic in MIRI Research
 Modal Combat in the Prisoner's Dilemma We're now ready to handle the preprint Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic  [2] , which encompasses the results from Section 3.2, and studies "modal agents" represented by expressions in modal logic. Recall the idea of Löbian cooperation from Section 3.2. One embarrassing thing about FairBot is that it doesn't check whether its potential cooperation would actually make any difference. It happily cooperates with CooperateBot, the constant strategy that simply returns C for every opponent. That's a bit like cooperating against a rock because the rock "cooperated" with you. Let's consider the following alternative: first we define DefectBot as the constant strategy that simply returns D for every opponent; then we define PrudentBot as the algorithm that, given the source code of an opponent X, searches in Peano Arithmetic for a proof of the statement "X cooperates with PrudentBot, and if Peano Arithmetic is consistent  12  , then furthermore X defects against DefectBot", and cooperates if and only if it finds such a proof  13  . Now it's more difficult to figure out what happens when FairBot plays the open-sourcecode Prisoner's Dilemma against PrudentBot, if we view these as proof searches in Peano Arithmetic. But if we assume infinite computational power (i.e. the ability to consult a halting oracle about proof searches in Peano Arithmetic), then they can be written out as simple statements in modal logic. Let p = "FairBot cooperates with PrudentBot", q = "PrudentBot cooperates with FairBot", r = "FairBot cooperates with DefectBot", and  12  Why does PrudentBot include the consistency requirement? Because certain actions that depend on agents require the consistency of Peano Arithmetic; for instance, FairBot does defect against DefectBot, but knowing this requires knowing that Peano Arithmetic is consistent, since otherwise FairBot might find a proof that DefectBot cooperates, using the Principle of Explosion. 13 Also, why not directly check whether your cooperation causes the opponent's cooperation, by defining MagicBot which cooperates if and only if it finds a proof of the statement "X cooperates with MagicBot if and only if MagicBot cooperates with X"? Because this makes it susceptible to spurious counterfactuals as in Section 3.3, and in fact MagicBot is equivalent to FairBot when they are rendered as modal statements. s = "DefectBot cooperates with FairBot". Then we have p ↔ q q ↔ (p ∧ (¬ ⊥ → ¬r)) r ↔ s s ↔ ⊥. If we look for the fixed point of this modal system, we can use the truth table formalism of the fixed-point theorem (we do all of these simultaneously, since every variable is equivalent to a fully modalized expression) to show that p ↔ , q ↔ , r ↔ ⊥, and of course s ↔ ⊥.
 Exercise. Show this formally. So what actually happens when you run the bounded versions of these agents against one another (with sufficiently high bounds on how deep they search for proofs before giving up)? Again, the bounded analogue of Löb's Theorem helps any valid proof in GL to go through if the proof bounds are sufficiently large. And since we really don't expect there to be a contradiction in Peano Arithmetic or any of the systems built atop it by addition of the axioms ¬ n ⊥ 14 , in the bounded case we should expect that every search for n ⊥ comes up empty. Thus p and q are true, and r and s are false. (This is equivalent, in the infinite computation case, to specifying that we care about what actually holds in the standard model of the natural numbers.) Therefore we can use these tools to analyze interactions between "modal agents" represented by families of modal statements like FairBot and PrudentBot, and moreover, we can figure out what actually happens using an efficient algorithm rather than the huge bruteforce proof search! It's a little tricky to define these modal agents in one fell swoop, so let's start with the simplest case: agents that don't check their opponent's action against any third parties, but only against themselves. (We will call these "modal agents of rank 0".) A rank 0 modal agent is represented by a formula p ↔ φ(p, q), where φ is modalized in both p and q (we don't allow modal agents to run the other agent, only to prove things about them; this avoids the infinite regress of two agents simulating each other forever, waiting for the other to 'make the first move'). By the modal fixed-point theorem, this is equivalent in GL to p ↔ φ(q) for some φ modalized in q, so we will define rank 0 modal agents using those formulas. Then if we have two modal agents of rank 0, represented by p ↔ φ(q) and q ↔ ψ(p), the simultaneous fixed point of these formulas gives us constant sentences (combinations of n ⊥) from which we can deduce their actions. Now let's allow calls to third parties, like PrudentBot's call to the opponent's action against DefectBot. We'll insist that it bottom out in a finite number of statements: for this, it suffices that every modal agent can only predict its opponent's action against itself and against modal agents of strictly lower rank. That is, Definition. X = φ, Z 1 , . . . , Z k is a modal agent of rank n ≥ 0 if φ is a fully modalized formula of k + 1 variables p, r 1 , . . . , r k , and each Z i is a modal agent of rank < n. Given two modal agents X = φ X , Z 1 , . . . , Z k and Y = ψ Y , W 1 , . . . , W l , we then construct the family of formulas given by recursively applying the modal agents to each other and applying the third parties to the originals: p X ↔ φ X (q Y , r Z 1 , . . . , r Z k ) q y ↔ ψ Y (p X , s W 1 , . . . , s W l ) r Z i ↔ φ Z i (t (Y,Z i ) , u Z i 1 , . . . , u Z i m i ) t (Y,Z i ) ↔ ψ Y (r Z i , v (W 1 ,Z i ) , . . . , v (W l ,Z i ) ) and so on until it bottoms out. (Please excuse the proliferation of indices and the imperfect rigor, and take another look at the example above of PrudentBot versus FairBot, where FairBot= p , PrudentBot= (p ∧ (¬ ⊥ → ¬r)), DefectBot , and DefectBot= ⊥ .) Thus any two modal agents can be played against each other, and the result of that "modal combat  15  " computed by an effective algorithm. So there's an actual Haskell program, written by Mihaly Barasz and Marcello Herreshoff, that checks the result of the modal combat, and that program helped the authors to find many other patterns about modal agents (including the discovery of PrudentBot, which is never exploited, achieves mutual cooperation with itself and with FairBot, and which correctly defects against CooperateBot). Among the other results in that paper: • Third parties are necessary to get cooperation from FairBot while defecting against CooperateBot; no modal agent of rank 0 can achieve both. • No modal agent globally dominates another modal agent, since it is possible to write another modal agent which punishes or rewards agents for their other decisions. (For example, consider TrollBot, which cooperates with its opponent if and only if it proves that its opponent cooperates with DefectBot.) • In another obstacle to strong definitions of optimality, consider WaitFairBot N defined by ¬ N +1 ⊥∧ (¬ N ⊥ → p) . Then any modal agent that defects against DefectBot will fail to elicit mutual cooperation from WaitFairBot N for sufficiently large N . • Finally, there are some modal agents that can be exploited by non-modal agents but not by any modal agent. Consider MimicFairBot, which cooperates with X if and only if Peano Arithmetic proves that X cooperates with FairBot. (That is, MimicFairBot= r, FairBot .) It is possible for an algorithm to exploit MimicFairBot (consider the non-modal agent that cooperates if and only if the opponent's source code is identical to the source code of FairBot), but every modal agent treats MimicFairBot and FairBot identically. The field of modal combat is new and still wide-open, and I expect to see many more results soon! Furthermore, it inspired an unexpected development in decision theory for one-player games as well...
 Modal Decision Theory Recall the setup in Section 3.3: we have a universe U () which contains an agent A(). The universe runs some computation and returns some outcome; as a part of that computation, the agent runs some computation and returns some action. Everything is deterministic, but we can think of a decision theory as a condition on the allowable form of A(); if you have a certain decision theory, then you find yourself only in certain universes and you perform some specific algorithm in those universes. In order to define what we mean by a decision theory, then, we need to specify where in the universe the agent goes. So we start with some utility values over outcomes, and a universe template U (•) with a spot for an agent; then our decision theory specifies an agent A() that goes there, such that we now have the universe U A () = U (A) which computes a certain outcome; and we judge the decision theory based on how highly it values that outcome. Causal decision theory (CDT) and evidential decision theory (EDT) are two famous philosophical examples of decision theory. Both of these require the universe template to supply the agent with some additional information: in the case of CDT, a causal graph of the universe, and in the case of EDT, the actions and outcomes of other agents in the same universe template. Because CDT and EDT can be shown to make bad decisions on some problems even when given the right info, several people have proposed alternatives, including timeless decision theory  [17]  (TDT) and updateless decision theory (UDT). (For more on the philosophical side of things, see Toward Idealized Decision Theory  [13] .) While working out models of UDT, in particular, Vladimir Slepnev and Benja Fallenstein found a nice formalization of a simple decision theory using modal logic, one that they could even prove optimal on a certain class of modally defined problems! Consider a universe template U (•) with a finite set of possible actions A and a finite set of possible outcomes O. We have some preference ordering over O, and we pick some default action a 0 to return if we can't prove anything at all. We then define the decision theory which takes U (•) and returns the agent A() given by Algorithm 3. Note that this decision theory is immune to the spurious conterfactuals of Section 3.3: if it ever proves a counterfactual, it immediately makes the antecedent true by returning that action, so the consequent must be true as well. Moreover, if it proves any counterfactual, then it has already tried and failed to achieve every outcome higher in its preference order. Before we show an optimality result for Algorithm 3 in a special domain, we will have to show what can go wrong with it in other domains. Firstly, the universe template could be 'unfair', in the sense that it cares about aspects of the agent that aren't significantly related to its output. For instance, we could have a universe template that gives the best outcome if and only if A() is a tab-indented Python program, or if and only if A() can be proved to always select the first action of A in alphabetical order (regardless of the details of U (•). We will not worry about how to succeed on such pathological problems in general. A very strong fairness condition for a universe template is that U (•) should be a function of only the output of A, rather than depending on any other features  16  . We can define this in Peano Arithmetic by choosing an encoding for agents in a particular universe template, then writing a sentence that asserts (A() = B()) → (U A () = U B ()); the universe template is 'provably extensional' if that statement is a theorem of Peano Arithmetic. Everything thus far can be represented in modal logic. We will represent O and A by provably mutually exclusive and exhaustive (p.m.e.e.) families of modal statements. For example, {p∧q, p∧¬q, ¬p} is a p.m.e.e. family. Then if the universe template and the agent are modal expressions, the universe template is provably extensional if (a ↔ b) → (U (a) ↔ U (b)) is a theorem of GL. (You can check that the universe from Algorithm 1 corresponds to a provably extensional modal universe with 3 actions and 3 outcomes.) However, the modal version of Algorithm 3 can fail even on provably extensional modal universe templates; indeed, for any modally defined decision theory there is an "evil problem" which it fails on. (This can be shown by representing the modal decision theory itself in the universe template, and punishing any agent that acts identically to the action of the modal decision theory.) So our definition of optimality must allow some way around this. As it happens, although for any decision theory one can write a specific universe template that frustrates it, for any fixed universe template there is a modification of Algorithm 3 which succeeds on it, and this modification is simply a replacement of "proves in Peano Arithmetic" with "proves in Peano Arithmetic given ¬ n ⊥" for sufficiently large n. We can then show that this modification achieves the best outcome that is achievable by any modal decision theory. This result shows that we're on to something: in this restricted domain, we can do as well as it's possible to do, as long as we're bringing enough deductive capacity to bear! Note that here, the universe acts on the agent only by running it and using the output, while the agent interacts with the source code of the universe only by proving things about it. So in particular, it's less rich than modal combat, in which your agent is proving things about the universe, but another part of that universe (the opponent) is proving things about the agent as well. So before we count ourselves done with using modal logic in decision theory, we have many open questions about competition and bargaining and other phenomena. It's a beautiful Löbian universe, waiting for us to explore! In addition to the contributions of Vladimir Slepnev and Benja Fallenstein, the "evil decision problems" were introduced by Nick Bone. If you'd like to get into this topic, I recommend Benja's recent posts on evil decision problems in provability logic, optimality of the model of UDT  [8] , and its sequel.
 Acknowledgments I originally prepared about half of these notes for a talk at a MIRIx workshop in Los Altos, California in November 2014; the feedback from the participants was especially helpful. When writing up the notes, I got some great assistance from Benja Fallenstein, Marcello Herreshoff, Elliot Jin, and Nathaniel Thomas. product and sum of two natural numbers are also natural numbers) 4. ∀x ∈ N∀y ∈ N (Sx = Sy) → (x = y) (if the successors are equal, so are the originals) 5. ∀y ∈ N(y = O) ∨ (∃x ∈ N Sx = y) (every nonzero number has a predecessor) 6. ∀x ∈ N (x + O = x) ∧ (x • O = O) (how zero interacts with addition and multiplication) 7. ∀x ∈ N∀y ∈ N (x + Sy = S(x + y)) ∧ (x • Sy = (x • y) + x) (how successor interacts with addition and multiplication)
 Algorithm 3 : 3 A() (Slepnev-Fallenstein model of UDT) for x ∈ O (in descending order of preference) do for a ∈ A do if Peano Arithmetic proves "A() = a → U () = x" then return a end end end return a 0
			 Note the asymmetry: a proof of defection does not directly lead to defection, since FairBot acts based only on whether it finds a proof of cooperation, and it nowhere presumes that finding a proof of defection precludes finding another proof of cooperation. Again, the formal system can't correctly know that it is consistent!
			 Note, however, that it's impossible to prove a spurious counterfactual about the choice you actually take, and thus it could not be consistent to find a spurious proof that A() = 1 → U () = 2 or that A() = 2 → U () = 11
			 Of course, this example does take a leap of faith in accepting the existence of infinitely many natural numbers; some mathematicians reject that, call themselves strict finitists, and assert that we have no idea whether the axioms of arithmetic are consistent. But they do accept that any theory with a completely specified finite model must be consistent.
			 Actually, we need to get a bit more complicated to do all inductive proofs on the natural numbers; what we truly need is that if φ(x, z 1 , . . . , z k ) is a logical formula, then∀z 1 ∈ N . . . ∀z k ∈ N (φ(0, z 1 , . . . , z k ) ∧ (∀x ∈ Nφ(x, z 1 , . . . , z k ) → φ(Sx, z 1 , . . . , z k ))) → ∀y ∈ Nφ(y, z 1 , . . . , z k ).
			 Assuming that the standard ZFC axioms of set theory are consistent, as we do throughout this paper, since we use these to construct models of all the weird axiomatizations of Peano Arithmetic.
			 Gödel himself, toward the end of his life, circulated a modal logic version of Anselm's ontological proof of the existence of God. The proof is of course formally valid, but the axioms are a bit overpowered and underjustified.
			 Note that these all correspond to nonstandard models of Peano Arithmetic, but that in some sense they approach the standard model as n → ∞.
			 This part requires our assumption that the expression is modalized in p, since otherwise we couldn't fill in the value for it without first knowing the value for p.
			 Since every expression A either remains true forever or switches over to false and remains that way forever, we can see that every expression must eventually stabilize; and if we've decomposed it at each step involving a , once two rows are identical, all subsequent rows must be as well.
			 If this step worries you, you may be reassured to know that the standard axioms of set theory are strong enough to assert that this tower upon Peano Arithmetic is consistent all the way up. Of course, that simply means that maybe set theory is contradictory, but if so then the contradiction has been hiding pretty effectively from the mathematical community for the last few decades...
			 I came up with that pun. I'm proud of it.
			 Note that this condition is too strong to include modal combat, since there it also matters whether a given formal system can prove what the algorithm's output is, not only what the output is. Thus our optimality theorem will not cover modal combat, which is consistent, given that we mentioned in the last section that we lack a good optimality result for modal combat.

Abstract: I apologize for all of the exclamation points in the previous paragraph; in my defense, if any result deserves exclamation points, it's Gödel's First Incompleteness Theorem! Also, there's a sleight of hand here in the demonstration that G never halts. We'll get further into that in Section 4.