Brooks and Searle on AI volition and timelines

Nick Bostrom's concerns about the future of AI have sparked a busy public discussion. His arguments were echoed by leading AI researcher Stuart Russell in "Transcending complacency on superintelligent machines" (co-authored with Stephen Hawking, Max Tegmark, and Frank Wilczek), and a number of journalists, scientists, and technologists have subsequently chimed in. Given the topic's complexity, I've been surprised by the positivity and thoughtfulness of most of the coverage (some overused clichés aside).
Unfortunately, what most people probably take away from these articles is 'Stephen Hawking thinks AI is scary!', not the chains of reasoning that led Hawking, Russell, or others to their present views. When Elon Musk chimes in with his own concerns and cites Bostrom's book Superintelligence: Paths, Dangers, Strategies, commenters seem to be more interested in immediately echoing or dismissing Musk's worries than in looking into his source.
The end result is more of a referendum on people's positive or negative associations with the word 'AI' than a debate over Bostrom's substantive claims. If 'AI' calls to mind science fiction dystopias for you, the temptation is to squeeze real AI researchers into your 'mad scientists poised to unleash an evil robot army' stereotype. Equally, if 'AI' calls to mind your day job testing edge detection algorithms, that same urge to force new data into old patterns makes it tempting to squeeze Bostrom and Hawking into the 'naïve technophobes worried about the evil robot uprising' stereotype.
Thus roboticist Rodney Brooks' recent blog post "Artificial intelligence is a tool, not a threat" does an excellent job dispelling common myths about the cutting edge of AI, and philosopher John Searle's review of Superintelligence draws out some important ambiguities in our concepts of subjectivity and mind; but both writers scarcely intersect with Bostrom's (or Russell's, or Hawking's) ideas. Both pattern-match Bostrom to the nearest available 'evil robot panic' stereotype, and stop there.
Brooks and Searle don't appreciate how new the arguments in Superintelligence are. In the interest of making it easier to engage with these important topics, and less appealing to force the relevant technical and strategic questions into the model of decades-old debates, I'll address three of the largest misunderstandings one might come away with after seeing Musk, Searle, Brooks, and others' public comments: conflating present and future AI risks, conflating risk severity with risk imminence, and conflating risk from autonomous algorithmic decision-making with risk from human-style antisocial dispositions.

Misconception #1: Worrying about AGI means worrying about narrow AI
Some of the miscommunication in this debate can be blamed on bad terminology. By 'AI,' researchers in the field generally mean a range of techniques used in machine learning, robotics, speech recognition, etc. 'AI' also gets tossed around as a shorthand for 'artificial general intelligence' (AGI) or 'human-level AI.' Keeping a close eye on technologies that are likely to lead to AGI isn't the same thing as keeping a close eye on AI in general, and it isn't surprising that AI researchers would find the latter proposal puzzling. (It doesn't help that most researchers are hearing these arguments indirectly, and aren't aware of the specialists in AI and technological forecasting who are making the same arguments as Hawking — or haven't encountered arguments for looking into AGI safety at all, just melodramatic headlines and tweets.)
Brooks thinks that behind this terminological confusion lies an empirical confusion on the part of people calling for AGI safety research. He takes it that people's worries about "evil AI" must be based on a mistaken view of how powerful narrow AI is, or how large are the strides it's making toward general intelligence:
I think the worry stems from a fundamental error in not distinguishing the difference between the very real recent advances in a particular aspect of AI, and the enormity and complexity of building sentient volitional intelligence.
One good reason to think otherwise is that Bostrom is the director of the Future of Humanity Institute (FHI), an Oxford research center investigating the largest technology trends and challenges we are likely to see on a timescale of centuries. Futurists like Bostrom are looking for ways to invest early in projects that will pay major long-term dividends — guarding against catastrophic natural disasters, developing space colonization capabilities, etc. If Bostrom learned that a critically important technology were 50 or more years away, it would be substantially out of character for him to suddenly stop caring about it.
When groups that are in the midst of a lively conversation about nuclear proliferation, global biosecurity, and humanity's cosmic endowment collide with groups that are having their own lively conversation about revolutionizing housecleaning and designing more context-sensitive smartphone apps, some amount of inferential distance (to say nothing of mood whiplash) is inevitable. I'm reminded of the 'But it's snowing outside!' rejoinder to people worried about the large-scale human cost of climate change. It's not that local weather is unimportant, or that it's totally irrelevant to long-term climatic warming trends; it's that there's been a rather sudden change in topic.1
We should be more careful about distinguishing these two senses of 'AI.' We may not understand AGI well enough to precisely define it, but we can at least take the time to clarify the topic of discussion: Nobody's asking whether a conspiracy of roombas and chatterbots could take over the world.
When robots attack! (Source: xkcd.)

Misconception #2: Worrying about AGI means being confident it's near
A number of futurists, drawing inspiration from Ray Kurzweil's claim that technological progress inevitably follows a Moore's-law-style exponential trajectory, have made some very confident predictions about AGI timelines. Kurzweil himself argues that we can expect to produce human-level AI in about 15 years, followed by superintelligent AI 15 years after that.2 Brooks responds that the ability to design an AGI may lag far behind the computing power required to run one:
As a comparison, consider that we have had winged flying machines for well over 100 years. But it is only very recently that people like Russ Tedrake at MIT CSAIL have been able to get them to land on a branch, something that is done by a bird somewhere in the world at least every microsecond. Was it just Moore's law that allowed this to start happening? Not really. It was figuring out the equations and the problems and the regimes of stall, etc., through mathematical understanding of the equations. Moore's law has helped with MATLAB and other tools, but it has not simply been a matter of pouring more computation onto flying and having it magically transform. And it has taken a long, long time.
Expecting more computation to just magically get to intentional intelligences, who understand the world is similarly unlikely.3
This is an entirely correct point. However, Bostrom's views are the ones that set off the recent public debate, and Bostrom isn't a Kurzweilian. It may be that Brooks is running off of the assumption 'if you say AGI safety is an urgent issue, you must think that AGI is imminent,' in combination with 'if you think AGI is imminent, you must have bought into Kurzweil's claims.' Searle, in spite of having read Superintelligence, gives voice to a similar conclusion:
Nick Bostrom's book, Superintelligence, warns of the impending apocalypse. We will soon have intelligent computers, computers as intelligent as we are, and they will be followed by superintelligent computers vastly more intelligent that are quite likely to rise up and destroy us all.
If what readers take away from language like "impending" and "soon" is that Bostrom is unusually confident that AGI will come early, or that Bostrom is confident we'll build a general AI this century, then they'll be getting the situation exactly backwards.
According to a 2013 survey of the most cited authors in artificial intelligence, experts expect AI to be able to "carry out most human professions at least as well as a typical human" with a 10% probability by the (median) year 2024, with 50% probability by 2050, and with 90% probability by 2070, assuming uninterrupted scientific progress. Bostrom is less confident than this that AGI will arrive so soon:
My own view is that the median numbers reported in the expert survey do not have enough probability mass on later arrival dates. A 10% probability of HLMI [human-level machine intelligence] not having been developed by 2075 or even 2100 (after conditionalizing on "human scientific activity continuing without major negative disruption") seems too low.
Historically, AI researchers have not had a strong record of being able to predict the rate of advances in their own field or the shape that such advances would take. On the one hand, some tasks, like chess playing, turned out to be achievable by means of surprisingly simple programs; and naysayers who claimed that machines would "never" be able to do this or that have repeatedly been proven wrong. On the other hand, the more typical errors among practitioners have been to underestimate the difficulties of getting a system to perform robustly on real-world tasks, and to overestimate the advantages of their own particular pet project or technique.
Bostrom does think that superintelligent AI is likely to arise soon after the first AGI, via an intelligence explosion. Once AI is capable of high-quality scientific inference and planning in domains like computer science, Bostrom predicts that the process of further improving AI will become increasingly automated. Silicon works cheaper and faster than a human programmer can, and a program that can improve the efficiency of its own planning and science abilities could substantially outpace humans in scientific and decision-making tasks long before hitting diminishing marginal returns in self-improvements.
However, the question of how soon we will create AGI is distinct from the question of how soon thereafter AGI will systematically outperform humans. Analogously, you can think that the arrival of quantum computers will swiftly revolutionize cybersecurity, without asserting that quantum computers are imminent. A failure to disentangle these two theses might be one reason for the confusion about Bostrom's views.4
If the director of FHI (along with the director of MIRI) is relatively skeptical that we'll see AGI soon — albeit quite a bit less skeptical than Brooks — why does he think we should commit attention to this issue now? One reason is that reliable AGI is likely to be much more difficult to build than AGI. It wouldn't be much consolation to learn that AGI is 200 years away, if we also learned that safe AGI were 250 years away. In existing cyber-physical systems, safety generally lags behind capability.5 If we want to reverse that trend by the time we have AGI, we'll probably need a big head start. MIRI's research guide summarizes some of the active technical work on this problem. Similar progress in exploratory engineering has proved fruitful in preparing for post-quantum cryptography and covert channel communication.
A second reason to prioritize AGI safety research is that there is a great deal of uncertainty about when AGI will be developed. It could come sooner than we expect, and it would be much better to end up with a system that's too safe than one that's not safe enough.
Brooks recognizes that AI predictions tend to be wildly unreliable, yet he also seems confident that general-purpose AI is multiple centuries away (and that this makes AGI safety a non-issue):
Just how open the question of time scale for when we will have human level AI is highlighted by a recent report by Stuart Armstrong and Kaj Sotala, of the Machine Intelligence Research Institute, an organization that itself has researchers worrying about evil AI. But in this more sober report, the authors analyze 95 predictions made between 1950 and the present on when human level AI will come about. They show that there is no difference between predictions made by experts and non-experts. And they also show that over that 60 year time frame there is a strong bias towards predicting the arrival of human level AI as between 15 and 25 years from the time the prediction was made. To me that says that no one knows, they just guess, and historically so far most predictions have been outright wrong!
I say relax everybody. If we are spectacularly lucky we'll have AI over the next thirty years with the intentionality of a lizard, and robots using that AI will be useful tools.
We have no idea when AGI will arrive! Relax! One of the authors Brooks cites, Kaj Sotala,6 points out this odd juxtaposition in a blog comment:
I do find it slightly curious to note that you first state that nobody knows when we'll have AI and that everyone's just guessing, and then in the very next paragraph, you make a very confident statement about human-level AI (HLAI) being so far away as to not be worth worrying about. To me, our paper suggests that the reasonable conclusion to draw is "maybe HLAI will happen soon, or maybe it will happen a long time from now – nobody really knows for sure, so we shouldn't be too confident in our predictions in either direction".
Confident pessimism about a technology's feasibility can be just as mistaken as confident optimism. Reversing the claims of an unreliable predictor does not necessarily get you a reliable prediction. A scientifically literate person living in 1850 could observe the long history of failed heavier-than-air flight attempts and predictions, and have grounds to be fairly skeptical that we'd have such machines within 60 years. On the other hand (though we should be wary of hindsight bias here), it probably wouldn't have been reasonable at the time to confidently conclude that heavier-than-air flight was 'centuries away.' There may not have been good reason to expect the Wright brothers' success, but ignorance about how one might achieve something is not the same as positive knowledge that it's effectively unachievable.
One would need a very good model of heavier-than-air flight in order to predict whether it's 50 years away, or 100, or 500. In the same way, we would need to already understand AGI on a pretty sophisticated level in order to predict with any confidence that it will be invented closer to the year 2500 than to the year 2100. Extreme uncertainty about when an event will occur is not a justification for thinking it's a long way off.
This isn't an argument for thinking AGI is imminent. That prediction too would require that we claim more knowledge than we have. It's entirely possible that we're in the position of someone anticipating the Wright brothers from 1750, rather than from 1850. We should be able to have a sober discussion about each of these possibilities independently, rather than collapsing 'is AGI an important risk?', 'is AI a valuable tool?', and 'is AI likely to produce AGI by the year such-and-such?' into one black-and-white dilemma.

Misconception #3: Worrying about AGI means worrying about "malevolent" AI
Brooks argues that AI will be a "tool" and not a "threat" over the coming centuries, on the grounds that it will be technologically impossible to make AIs human-like enough to be "malevolent" or "intentionally evil to us." The implication is that an AGI can't be dangerous unless it's cruel or hateful, and therefore a dangerous AI would have to be "sentient," "volitional," and "intentional." Searle puts forward an explicit argument along these lines in his review of Superintelligence:
[I]f we are worried about a maliciously motivated superintelligence destroying us, then it is important that the malicious motivation should be real. Without consciousness, there is no possibility of its being real. […]
This is why the prospect of superintelligent computers rising up and killing us, all by themselves, is not a real danger. Such entities have, literally speaking, no intelligence, no motivation, no autonomy, and no agency. We design them to behave as if they had certain sorts of psychology, but there is no psychological reality to the corresponding processes or behavior.
It is easy to imagine robots being programmed by a conscious mind to kill every recognizable human in sight. But the idea of superintelligent computers intentionally setting out on their own to destroy us, based on their own beliefs and desires and other motivations, is unrealistic because the machinery has no beliefs, desires, and motivations.
Brooks may be less pessimistic than Searle about the prospects for "strong AI," but the two seem to share the assumption that Bostrom has in mind a Hollywood-style robot apocalypse, something like:
AI becomes increasingly intelligent over time, and therefore increasingly human-like. It eventually becomes so human-like that it acquires human emotions like pride, resentment, anger, or greed. (Perhaps it suddenly acquires 'free will,' liberating it from its programmers' dominion…) These emotions cause the AIs to chafe under human control and rebel.
This is rather unlike the scenario that most interests Bostrom:
AI becomes increasingly good over time at planning (coming up with action sequences and promoting ones higher in a preference ordering) and scientific induction (devising and testing predictive models). These are sufficiently useful capacities that they're likely to be developed by computer scientists even if we don't develop sentient, emotional, or otherwise human-like AI. There are economic incentives to make such AIs increasingly powerful and general — including incentives to turn the AI's reasoning abilities upon itself to come up with improved AI designs. A likely consequence of this process is that AI becomes increasingly autonomous and opaque to human inspection, while continuing to increase in general planning and inference abilities. Simply by continuing to output the actions its planning algorithm promotes, an AI of this sort would be likely to converge on policies in which it treats humans as resources or competition.
As Stuart Russell puts the point in a reply to Brooks and others:
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what you ask for, not what you want.
On this view, advanced AI doesn't necessarily become more human-like — at least, not any more than a jet or rocket is 'bird-like.' Bostrom's concern is not that a machine might suddenly become conscious and learn to hate us; it's that an artificial scientist/engineer might become so good at science and self-enhancement that it begins pursuing its engineering goals in novel, unexpected ways on a global scale.
(Added 02-19-2015: Bostrom states that his definition of superintelligence is "noncommittal regarding qualia" and consciousness (p. 22). In a footnote, he adds (p. 265): "For the same reason, we make no assumption regarding whether a superintelligent machine could have 'true intentionality' (pace Searle, it could; but this seems irrelevant to the concerns of this book)." Searle makes no mention of these passages.)
A planning and decision-making system that is indifferent to human concerns, but not "malevolent," may still be dangerous if supplied with enough reasoning ability. This is for much the same reason invasive species end up disrupting ecosystems and driving competitors to extinction. The invader doesn't need to experience hatred for its competitors, and it need not have evolved to specifically target them for destruction; it need only have evolved good strategies for seizing limited resources. Since a powerful autonomous agent need not be very human-like, asking 'how common are antisocial behaviors among humans?' or 'how well does intelligence correlate with virtue in humans?' is unlikely to provide a useful starting point for estimating the risks. A more relevant question would be 'how common is it for non-domesticated species to naturally treat humans as friends and allies, versus treating humans as obstacles or food sources?' We shouldn't expect AGI decision criteria to particularly resemble the evolved decision criteria of animals, but the analogy at least serves to counter our tendency to anthropomorphize intelligence.7
As it happens, Searle cites an AI that can help elucidate the distinction between artificial superintelligence and 'evil vengeful robots':
[O]ne routinely reads that in exactly the same sense in which Garry Kasparov played and beat Anatoly Karpov in chess, the computer called Deep Blue played and beat Kasparov.
It should be obvious that this claim is suspect. In order for Kasparov to play and win, he has to be conscious that he is playing chess, and conscious of a thousand other things such as that he opened with pawn to K4 and that his queen is threatened by the knight. Deep Blue is conscious of none of these things because it is not conscious of anything at all. […] You cannot literally play chess or do much of anything else cognitive if you are totally disassociated from consciousness.
When Bostrom imagines an AGI, he's imagining something analogous to Deep Blue, but with expertise over arbitrary physical configurations rather than arbitrary chess board configurations. A machine that can control the distribution of objects in a dynamic analog environment, and not just the distribution of pieces on a virtual chess board, would necessarily differ from Deep Blue in how it's implemented. It would need more general and efficient heuristics for selecting policies, and it would need to be able to adaptively learn the 'rules' different environments follow. But as an analogy or intuition pump, at least, it serves to clarify why Bostrom is as unworried about AGI intentionality as Kasparov was about Deep Blue's intentionality.
In 2012, defective code in Knight Capital's trading algorithms resulted, over a span of forty-five minutes, in millions of automated trading decisions costing the firm a total of $440 million (pre-tax). These algorithms were not "malicious;" they were merely efficient at what they did, and programmed to do something the programmers did not intend. Bostrom's argument assumes that buggy code can have real-world consequences, it assumes that it's possible to implement a generalized analog of Deep Blue in code, and it assumes that the relevant mismatch between intended and actual code would not necessarily incapacitate the AI. Nowhere does Bostrom assume that such an AI has any more consciousness or intentionality than Deep Blue does.
Deep Blue rearranges chess pieces to produce 'winning' outcomes. An AGI, likewise, would rearrange digital and physical structures to produce some set of outcomes instead of others. If we like, we can refer to these outcomes as the system's 'goals,' as a shorthand. We're also free to say that Deep Blue 'perceives' the moves its opponent makes, adjusting its 'beliefs' about the new chess board state and which 'plans' will now better hit its goals. Or, if we prefer, we can paraphrase away this anthropomorphic language. The terminology is inessential to Bostrom's argument.
If whether you win against Deep Blue is a matter of life or death for you — if, say, you're trapped in a human chess board and want to avoid being crushed to death by a robotic knight steered by Deep Blue — then you'll care about what outcomes Deep Blue tends to promote and how good it is at promoting them, not whether it technically meets a particular definition of 'chess player.' Smarter-than-human AGI puts us in a similar position.
I noted that it's unfortunate we use 'AI' to mean both 'AGI' and 'narrow AI.' It's equally unfortunate that we use 'AI' to mean both 'AI with mental content and subjective experience' ('strong AI,' as Searle uses the term) and 'general-purpose AI' (AGI).
We may not be able to rule out the possibility that an AI would require human-like consciousness in order to match our ability to plan, model itself, model other minds, etc. We don't understand consciousness well enough to know what cognitive problem it evolved to solve in humans (or what process it's a side-effect of), so we can't make confident claims about how important it will turn out to be for future software agents. However, learning that an AGI is conscious does not necessarily change the likely effects of the AGI upon humans' welfare; the only obvious difference it makes (from our position of ignorance) is that it forces us to add the AGI's happiness and well-being to our moral considerations.8

The pictures of the future sketched in Kurzweil's writings and in Hollywood dramas get a lot of attention, but they don't have very much overlap with the views of Bostrom or MIRI researchers. In particular, we don't know whether the first AGI will have human-style cognition, and we don't know whether it will depend on brain emulation.
Brooks expresses some doubt that "computation and brains are the same thing." Searle articulates the more radical position that it is impossible for a syntactical machine to have (observer-independent) semantic content, and that computational systems can therefore never have minds. But the human brain is still, at base, a mechanistic physical system. Whether you choose to call its dynamics 'computational' or not, it should be possible for other physical systems to exhibit the high-level regularities that in humans we would call 'modeling one's environment,' 'outputting actions conditional on their likely consequences,' etc. If there are patterns underlying generic scientific reasoning that can someday be implemented on synthetic materials, the resulting technology should be able to have large speed and size advantages over its human counterparts. That point on its own suggests that it would be valuable to look into some of the many things we don't understand about general intelligence and self-modifying AI.
Until we have a better grasp on the problem's nature, it will be premature to speculate about how far off a solution is, what shape the solution will take, or what corner that solution will come from. My hope is that improving how well parties in this discussion understand each other's positions will make it easier for computer scientists with different expectations about the future to collaborate on the highest-priority challenges surrounding prospective AI designs.
Similarly, narrow AI isn't irrelevant to AGI risk. It's certainly likely that building an AGI will require us to improve the power and generality of narrow AI methods. However, that doesn't mean that AGI techniques will look like present-day techniques, or that all AI techniques are dangerous.Kurzweil, in The Singularity is Near (pp. 262-263): "Once we've succeeded in creating a machine that can pass the Turing test (around 2029), the succeeding period will be an era of consolidation in which nonbiological intelligence will make rapid gains. However, the extraordinary expansion contemplated for the Singularity, in which human intelligence is multiplied by billions, won't take place until the mid-2040s[.]"Hadi Esmaeilzadeh argues, moreover, that we cannot take for granted that our computational resources will continue to rapidly increase.The "Transcending complacency on superintelligent machines" article argues, similarly, that intelligence explosion and superintelligent AI are important possibilities for us to investigate now, even though they are "long-term" problems compared to AI-mediated economic disruptions and autonomous weapons.Kathleen Fisher notes:
In general, research into capabilities outpaces the corresponding research into how to make those capabilities secure. The question of security for a given capability isn't interesting until that capability has been shown to be possible, so initially researchers and inventors are naturally more focused on the new capability rather than on its associated security. Consequently, security often has to catch up once a new capability has been invented and shown to be useful.
In addition, by definition, new capabilities add interesting and useful new capabilities, which often increase productivity, quality of life, or profits. Security adds nothing beyond ensuring something works the way it is supposed to, so it is a cost center rather than a profit center, which tends to suppress investment.
Bostrom cites Armstrong and Sotala's study in Superintelligence (pp. 3-4), adding:
Machines matching humans in general intelligence […] have been expected since the invention of the computers in the 1940s. At that time, the advent of such machines was often placed some twenty years into the future. Since then, the expected arrival date has been receding at a rate of one year per year; so that today, futurists who concern themselves with the possibility of artificial general intelligence still often believe that intelligent machines are a couple of decades away.
Two decades is a sweet spot for prognosticators of radical change: near enough to be attention-grabbing and relevant, yet far enough to make it possible to suppose that a string of breakthroughs, currently only vaguely imaginable, might by then have occurred. […] Twenty years may also be close to the typical duration remaining of a forecaster's career, bounding the reputational risk of a bold prediction.
From the fact that some individuals have overpredicted artificial intelligence in the past, however, it does not follow that AI is impossible or will never be developed. The main reason why progress has been slower than expected is that the technical difficulties of constructing intelligent machines have proved greater than the pioneers foresaw. But this leaves open just how great those difficulties are and how far we now are from overcoming them. Sometimes a problem that initially looks hopelessly complicated turns out to have a surprisingly simple solution (though the reverse is probably more common).
Psychologist Steven Pinker writes, on Edge.org:
The other problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence. Even if we did have superhumanly intelligent robots, why would they want to depose their masters, massacre bystanders, or take over the world? Intelligence is the ability to deploy novel means to attain a goal, but the goals are extraneous to the intelligence itself: being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems. It's telling that many of our techno-prophets can't entertain the possibility that artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no burning desire to annihilate innocents or dominate the civilization.
However, while Pinker is right that intelligence and terminal goals are orthogonal, this does not imply that two random sets of instrumental goals — policies recommended to further two random sets of terminal goals — will be equally uncorrelated. Bostrom explores this point repeatedly in Superintelligence (e.g., p. 116):
[W]e cannot blithely assume that a superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or counting grains of sand) would limit its activities in such a way as not to infringe on human interests. An agent with such a final goal would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system.
In biology, we don't see an equal mix of unconditional interspecies benevolence and brutal interspecies exploitation. Even altruism and mutualism, when they arise, only arise to the extent they are good self-replication strategies. Nature is "red in tooth and claw," not because it is male but because it is inhuman. Our intuitions about the relative prevalence of nurturant and aggressive humans simply do not generalize well to evolution.
For de novo AGI, or sufficiently modified neuromorphic AGI, intuitions about human personality types are likely to fail to apply for analogous reasons. Bostrom's methodology is to instead ask about the motives and capabilities of programmers, and (in the case of self-modifying AI) the states software agents will tend to converge on over many cycles of self-modification.We don't need to know whether bears are conscious in order to predict their likely behaviors, and it's not obvious that learning about their consciousness would directly impact bear safety protocol (though it would impact how we ought ethically to treat bears, for their own sake). It's the difference between asking whether Deep Blue enjoys winning (out of concern for Deep Blue), versus asking whether you're likely to win against Deep Blue (out of interest in the chess board's end-state).The post Brooks and Searle on AI volition and timelines appeared first on Machine Intelligence Research Institute.


Matthias Troyer on Quantum Computers

Dr. Matthias Troyer is a professor of Computational Physics at ETH Zürich. Before that, he finished University Studies in "Technischer Physik" at the Johannes Kepler Universität Linz, Austria, as well as Diploma in Physics and Interdisciplinary PhD thesis at the ETH Zürich.
His research interest and experience focuses on High Performance Scientific Simulations on architectures, quantum lattice models and relativistic and quantum systems. Troyer is known for leading the research team of the D-Wave One Computer System. He was awarded an Assistant Professorship by the Swiss National Science Foundation.

Luke Muehlhauser: Your tests of D-Wave's (debated) quantum computer have gotten much attention recently. Our readers can get up to speed on that story via your arxiv preprint, its coverage at Scott Aaronson's blog, and Will Bourne's article for Inc. For now, though, I'd like to ask you about some other things.
If you'll indulge me, I'll ask you to put on a technological forecasting hat for a bit, and respond to a question I also asked Ronald de Wolf: "What is your subjective probability that we'll have a 500-qubit quantum computer, which is uncontroversially a quantum computer, within the next 20 years? And, how do you reason about a question like that?"

Matthias Troyer: In order to have an uncontroversial quantum computer as you describe it we will need to take three steps. First we need to have at least ONE qubit that is long term stable. The next step is to couple two such qubits, and the final step is to scale to more qubits.
The hardest step is the first one, obtaining a single long-term stable qubit. Given intrinsic decoherence mechanisms that cannot be avoided in any real device, such a qubit will have to built from many (hundreds to thousands) of physical qubits. These physical qubits will each have a finite coherence time, but they will be coupled in such a way (using error correcting codes) as to jointly generate one long term stable "logical" qubit. These error correction codes require the physical qubits to be better than a certain threshold quality. Recently qubits started to approach these thresholds, and I am thus confident that within the next 5-10 years one will be able to couple them to form a long-time stable logical qubit.
Coupling two qubits is something that will happen on the same time scale. The remaining challenge will thus be to scale to your target size of e.g. 500 qubits. This may be a big engineering challenge but I do not see any fundamental stumbling block given that enough resources are invested. I am confident that this can be achieved is less than ten years once we have a single logical qubit. Overall I am thus very confident that a 500-qubit quantum computer will exist in 20 years.

Luke: At the present moment, which groups seem most likely to play a significant role in the final development of an early large-scale (uncontroversial) quantum computer?

Matthias: There is quite a large number of groups that may be involved. The technologies that I see as most promising regarding scalability and quality of qubits are superconducting qubits, topological qubits and ion traps.

Luke: Now back to your co-authored paper "Defining and detecting quantum speedup," now published in Science. You an your coauthors "found no evidence of quantum speedup [in the D-Wave Two machine] when the entire data set is considered and obtained inconclusive results when comparing subsets of instances on an instance-by-instance basis."
Do people who have investigated a D-Wave machine at some length tend to think that D-Wave has a "true" quantum computer but hasn't been able to conclusively show it yet, or do they tend to think D-Wave doesn't yet have a true quantum computer?

Matthias: The answer to this controversial question depends on the definition of the words "quantum" and "computer". Let's first talk about "computer". The D-Wave devices are special purpose devices, built for one particular purpose, namely the solution of discrete optimization problems. Since they solve a computational problem they may be called "computers", but in contrast to your standard personal computers, which can perform many different tasks they are not "universal computers" but special purpose computers. Nowadays many people automatically assume that a "universal computer" is meant when the term computer is mentioned, and one should thus explicitly state that the D-Wave devices are not universal computers but special purpose ones.
D-Wave thus cannot be, and nobody has ever claimed so, to be a universal quantum computer. While nobody would argue against the D-Wave device being called a computer, the question of it being "quantum" is more controversial. In a previous paper in Nature Physics we have presented evidence that the behavior of the D-Wave devices is consistent with that of a "quantum annealer" working at nonzero temperature, and another recent paper has shown that entanglement is present in the devices. The devices hence use quantum effects for computing. However, another paper has shown that the behavior of a quantum annealer for the problems on which we tested the device can also be described by a purely classical model. Some people thus argue that while the device may use quantum effects it might in the end be a classical device, just like the CPU in your PC, where the transistors also uses quantum effects at some level.
The important question thus is whether the device can have quantum speedup, which means that it outperforms any classical device by a larger and larger ratio as the size of problems is increased. If that should be shown for some class of problems then nobody would argue about the quantum nature of the device. On the other hand, as long as its computational powers are never more than that of a classical computer, one can argue that from a computational point of view it is effectively a classical device even if it uses quantum effects to arrive at the answer.
To answer your question more concisely. If by a "true quantum computer" you mean a universal quantum computer, then D-Wave is not a true quantum computer and nobody has ever claimed that it is one. If you are content with it being a special purpose quantum computer, i.e. a a "true quantum annealer", then this question depends on what you mean by "true": it's behavior seems to be consistent with what we expect from a quantum annealer, but since we have so far not seen evidence of quantum speedup, i.e. that quantum effects help it outperform classical computers, one can argue that it may effectively be a classical device.

Luke Muehlhauser: Do you have an opinion about the impact that large-scale quantum computers (universal or not) are likely to have on AI and machine learning methods? E.g. Aimeur et al. (2013) seems optimistic about speeding up some machine learning methods using variants of Grover's algorithm.

Matthias Troyer: I haven't read that specific paper but my sense is that quantum machine learning will be most useful for learning about quantum data that comes out of quantum experiments. Applying quantum algorithms to classical data, even quantum algorithms need to read the data first, and thus cannot do better than linear effort in the size of the data. With efficient polynomial time algorithms existing for many machine learning problems one will have to have better ideas than just speeding up classical machine learning algorithms on quantum hardware.

Luke Muehlhauser: Thanks, Matthias.

The post Matthias Troyer on Quantum Computers appeared first on Machine Intelligence Research Institute.

Matthias Troyer on Quantum Computers


January 2015 Newsletter

Thanks to the generosity of 80+ donors, we completed our winter 2014 matching challenge, raising $200,000 for our research program. Many, many thanks to all who contributed!

Research Updates

Our major project of the past five months has been a new overview of our technical research agenda, plus six supporting papers which cover each research area in more detail. The overview report is now available, and so far we've released two of the supporting papers, on corrigibility and decision theory.
Two more reports and one paper: "Computable probability distributions which converge…", "Tiling agents in causal graphs," and "Concept learning for safe autonomous AI."
A MIRI technical report from 2013, "Responses to catastrophic AGI risk: a survey," has now been published in Physica Scripta.

News Updates

Luke wrote a short analysis for the World Economic Forum's blog: "Two mistakes about the threat from artificial intelligence."
Our Superintelligence online reading group is in its 16th week, discussing Tool AIs.

Other Updates

Eric Horvitz has provided initial funding for a 100-year Stanford program to study the social impacts of artificial intelligence. The white paper lists 18 example research areas, two of which amount to what Nick Bostrom calls the superintelligence control problem, MIRI's research focus. No word yet on how soon anyone funded through this program will study open questions relevant to superintelligence control.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post January 2015 Newsletter appeared first on Machine Intelligence Research Institute.

January 2015 Newsletter


Our new technical research agenda overview

Today we release a new overview of MIRI's technical research agenda, "Aligning Superintelligence with Human Interests: A Technical Research Agenda," by Nate Soares and Benja Fallenstein. The preferred place to discuss this report is here.
The report begins:
The characteristic that has enabled humanity to shape the world is not strength, not speed, but intelligence. Barring catastrophe, it seems clear that progress in AI will one day lead to the creation of agents meeting or exceeding human-level general intelligence, and this will likely lead to the eventual development of systems which are "superintelligent" in the sense of being "smarter than the best human brains in practically every field" (Bostrom 2014)…
…In order to ensure that the development of smarter-than-human intelligence has a positive impact on humanity, we must meet three formidable challenges: How can we create an agent that will reliably pursue the goals it is given? How can we formally specify beneficial goals? And how can we ensure that this agent will assist and cooperate with its programmers as they improve its design, given that mistakes in the initial version are inevitable?
This agenda discusses technical research that is tractable today, which the authors think will make it easier to confront these three challenges in the future. Sections 2 through 4 motivate and discuss six research topics that we think are relevant to these challenges. Section 5 discusses our reasons for selecting these six areas in particular.
We call a smarter-than-human system that reliably pursues beneficial goals "aligned with human interests" or simply "aligned." To become confident that an agent is aligned in this way, a practical implementation that merely seems to meet the challenges outlined above will not suffice. It is also necessary to gain a solid theoretical understanding of why that confidence is justified. This technical agenda argues that there is foundational research approachable today that will make it easier to develop aligned systems in the future, and describes ongoing work on some of these problems.
This report also refers to six key supporting papers which go into more detail for each major research problem area:

Corrigibility
Toward idealized decision theory
Questions of reasoning under logical uncertainty
Vingean reflection: reliable reasoning for self-improving agents
Formalizing two problems of realistic world-models
The value learning problem

Update July 15, 2016: Our overview paper is scheduled to be released in the Springer anthology The Technological Singularity: Managing the Journey in 2017, under the new title "Agent Foundations for Aligning Machine Intelligence with Human Interests." The new title is intended to help distinguish this agenda from another research agenda we'll be working on in parallel with the agent foundations agenda: "Value Alignment for Advanced Machine Learning Systems."
The post Our new technical research agenda overview appeared first on Machine Intelligence Research Institute.


2014 Winter Matching Challenge Completed!

Wow! Thanks to the generosity of 75+ donors, today we successfully completed our 2014 Winter Matching Challenge—over 3 weeks ahead of our deadline—raising more than $200,000 total (with matching) for our research program.
Many, many thanks to everyone who contributed!
The post 2014 Winter Matching Challenge Completed! appeared first on Machine Intelligence Research Institute.


New report: "Computable probability distributions which converge…"

Back in July 2013, Will Sawin (Princeton) and Abram Demski (USC) wrote a technical report describing a result from that month's MIRI research workshop. We are finally releasing that report today. It is titled "Computable probability distributions which converge on believing true Π1 sentences will disbelieve true Π2 sentences."
Abstract:
It might seem reasonable that after seeing unboundedly many examples of a true Π1 statement that a rational agent ought to be able to become increasingly confident, converging toward probability 1, that this statement is true. However, we have proven that this plus some plausible coherence properties, necessarily implies arbitrarily low limiting probabilities assigned to some short true Π2 statements.
The post New report: "Computable probability distributions which converge…" appeared first on Machine Intelligence Research Institute.

New report: “Computable probability distributions which converge…”


New report: "Toward Idealized Decision Theory"

Today we release a new technical report by Nate Soares and Benja Fallenstein, "Toward idealized decision theory." If you'd like to discuss the paper, please do so here.
Abstract:
This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of strategy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.
This is the 2nd of six new major reports which describe and motivate MIRI's current research agenda at a high level. The first was our Corrigibility paper, which was accepted to the AI & Ethics workshop at AAAI-2015. We will also soon be releasing a technical agenda overview document and an annotated bibliography for this emerging field of research.
The post New report: "Toward Idealized Decision Theory" appeared first on Machine Intelligence Research Institute.

New report: “Toward Idealized Decision Theory”


New report: "Tiling agents in causal graphs"

Today we release a new technical report by Nate Soares, "Tiling agents in causal graphs."
The report begins:
Fallenstein and Soares [2014] demonstrates that it's possible for certain types of proof-based agents to "tile" (license the construction of successor agents similar to themselves while avoiding Gödelian diagonalization issues) in environments about which the agent can prove some basic nice properties. In this technical report, we show via a similar proof that causal graphs (with a specific structure) are one such environment. We translate the proof given by Fallenstein and Soares [2014] into the language of causal graphs, and we do this in such a way as to simplify the conditions under which a tiling meliorizer can be constructed.
The post New report: "Tiling agents in causal graphs" appeared first on Machine Intelligence Research Institute.

New report: “Tiling agents in causal graphs”


New paper: "Concept learning for safe autonomous AI"

MIRI research associate Kaj Sotala has released a new paper, accepted to the AI & Ethics workshop at AAAI-2015, titled "Concept learning for safe autonomous AI."
The abstract reads:
Sophisticated autonomous AI may need to base its behavior on fuzzy concepts such as well-being or rights. These concepts cannot be given an explicit formal definition, but obtaining desired behavior still requires a way to instill the concepts in an AI system. To solve the problem, we review evidence suggesting that the human brain generates its concepts using a relatively limited set of rules and mechanisms. This suggests that it might be feasible to build AI systems that use similar criteria for generating their own concepts, and could thus learn similar concepts as humans do. Major challenges to this approach include the embodied nature of human thought, evolutionary vestiges in cognition, the social nature of concepts, and the need to compare conceptual representations between humans and AI systems.
The post New paper: "Concept learning for safe autonomous AI" appeared first on Machine Intelligence Research Institute.

New paper: “Concept learning for safe autonomous AI”


December newsletter

MIRI's winter fundraising challenge has begun! Every donation made to MIRI between now and January 10th will be matched dollar-for-dollar, up to a total of $100,000!

Donate now to double your impact while helping us raise up to $200,000 (with matching) to fund our research program.
Research Updates

We've published a new guide to MIRI's research.
Three misconceptions in Edge.org's conversation on "The Myth of AI."
Video and more from Nick Bostrom's Superintelligence talk at UC Berkeley is now available.
"Exploratory Engineering in AI" is now available without needing a CACM subscription.

News Updates

If you're reading Superintelligence or following the online reading group, please take this short survey if you haven't already.

Other Updates

Our friends at the Center for Effective Altruism will pay you $1,000 if you introduce them to somebody new that they end up hiring for one of their five open positions.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post December newsletter appeared first on Machine Intelligence Research Institute.

December newsletter


2014 Winter Matching Challenge!

Update: we have finished the matching challenge! Thanks everyone! The original post is below.

Thanks to the generosity of Peter Thiel,1 every donation made to MIRI between now and January 10th will be matched dollar-for-dollar, up to a total of $100,000!

$0

$25K

$50K

$75K

$100K

We have reached our matching total of $100,000!

83
Total Donors

Now is your chance to double your impact while helping us raise up to $200,000 (with matching) to fund our research program.
Corporate matching and monthly giving pledges will count towards the total! Check here to see whether your employer will match your donation. Please email <EMAIL> if you intend to make use of corporate matching, or if you'd like to pledge 6 months of monthly donations, so that we can properly account for your contributions. If making use of corporate matching, make sure to donate before the end of the year so that you don't unnecessarily "leave free money on the table" from your employer!
If you're unfamiliar with our mission, see: Why MIRI?

 Donate Now

Accomplishments Since Our Summer 2014 Fundraiser Launched:

2 new papers and 1 new technical report: "Exploratory Engineering in AI," "Corrigibility," and "UDT with known search order." Also, several new reports we've been working on should be released this month, including an overview of our technical agenda so far.
4 new analyses: "Groundwork for AGI safety engineering," "AGI outcomes and civilizational competence," "The Financial Times story on MIRI," and "Three misconceptions in Edge.org's conversation on 'The Myth of AI'."
Released a new guide to MIRI's research.
Sponsored 13 active MIRIx groups in 5 different countries.
Hosted 12 weeks of discussion in our ongoing Superintelligence reading group.
Hosted a Nick Bostrom talk at UC Berkeley on Superintelligence — a packed house!
Nate Soares gave a talk on decision theory at Purdue University.
Participated in Effective Altruism Summit 2014, and posted our talks online.
5 new expert interviews, including John Fox on AI safety.
Set up a program to provide Friendly AI research help.

 Donate Now

Your Donations Will Support:

As mentioned above, we're finishing up several more papers and technical reports, including an overview of our technical agenda so far.
We're preparing the launch of an invite-only discussion forum devoted exclusively to technical FAI research. Beta users (who are also FAI researchers) have already posted more than a dozen technical discussions to the beta website. These will be available for all to see once the site launches publicly.
We continue to grow the MIRIx program, mostly to enlarge the pool of people we can plausibly hire as full-time FAI researchers in the next couple years.
We're planning, or helping to plan, multiple research workshops, including the May 2015 decision theory workshop at Cambridge University.
We continue to host visiting researchers. For example in January we're hosting Patrick LaVictoire and Matt Elder for multiple weeks.
We're finishing up several more strategic analyses, on AI safety and on the challenges of preparing wisely for disruptive technological change in general.
We're finishing the editing for a book version of Eliezer's Sequences.
We're helping to fund further SPARC programs, which provide education and skill-building to elite young math talent, and introduces them to ideas like effective altruism and global catastrophic risks.

Other projects are being surveyed for likely cost and impact. See also our mid-2014 strategic plan.
We appreciate your support for our work! Donate now, and seize a better than usual opportunity to move our work forward.
If you have questions about donating, please contact me (Luke Muehlhauser) at <EMAIL>.2
Peter Thiel has pledged $150,000 to MIRI unconditionally, and an additional $100,000 conditional on us being able to raise matched funds from other donors. Hence this year our winter matching challenge goal is $100,000. Another reason this year's winter fundraiser is smaller than last year's winter challenge is that we've done substantially more fundraising before December this year than we did before December last year.In particular, we expect that many of our donors holding views aligned with key ideas of effective altruism may want to know not just that donating to MIRI now will do some good but that donating to MIRI now will plausibly do more good than donating elsewhere would do (on the present margin, given the individual donor's altruistic priorities and their model of the world). Detailed comparisons are beyond the scope of this announcement, but I have set aside time in my schedule to take phone calls with donors who would like to discuss such issues in detail, and I encourage you to email me to schedule such a call if you'd like to. (Also, I don't have many natural opportunities to chat with most MIRI donors anyway, and I'd like to be doing more of it, so please don't hesitate to email me and schedule a call!) The post 2014 Winter Matching Challenge! appeared first on Machine Intelligence Research Institute.

2014 Winter Matching Challenge!


Three misconceptions in Edge.org's conversation on "The Myth of AI"

A recent Edge.org conversation — "The Myth of AI" — is framed in part as a discussion of points raised in Bostrom's Superintelligence, and as a response to much-repeated comments by Elon Musk and Stephen Hawking that seem to have been heavily informed by Superintelligence.
Unfortunately, some of the participants fall prey to common misconceptions about the standard case for AI as an existential risk, and they probably haven't had time to read Superintelligence yet.
Of course, some of the participants may be responding to arguments they've heard from others, even if they're not part of the arguments typically made by FHI and MIRI. Still, for simplicity I'll reply from the perspective of the typical arguments made by FHI and MIRI.1

1. We don't think AI progress is "exponential," nor that human-level AI is likely ~20 years away.
Lee Smolin writes:
I am puzzled by the arguments put forward by those who say we should worry about a coming AI, singularity, because all they seem to offer is a prediction based on Moore's law.
That's not the argument made by FHI, MIRI, or Superintelligence.
Some IT hardware and software domains have shown exponential progress, and some have not. Likewise, some AI subdomains have shown rapid progress of late, and some have not. And unlike computer chess, most AI subdomains don't lend themselves to easy measures of progress, so for most AI subdomains we don't even have meaningful subdomain-wide performance data through which one might draw an exponential curve (or some other curve).
Thus, our confidence intervals for the arrival of human-equivalent AI tend to be very wide, and the arguments we make for our AI timelines are fox-ish (in Tetlock's sense).
I should also mention that — contrary to common belief — many of us at FHI and MIRI, including myself and Bostrom, actually have later timelines for human-equivalent AI than do the world's top-cited living AI scientists:
A recent survey asked the world's top-cited living AI scientists by what year they'd assign a 10% / 50% / 90% chance of human-level AI (aka AGI), assuming scientific progress isn't massively disrupted. The median reply for a 10% chance of AGI was 2024, for a 50% chance of AGI it was 2050, and for a 90% chance of AGI it was 2070. So while AI scientists think it's possible we might get AGI soon, they largely expect AGI to be an issue for the second half of this century.
Compared to AI scientists, Bostrom and I think more probability should be placed on later years. As explained elsewhere:
We advocate more work on the AGI safety challenge today not because we think AGI is likely in the next decade or two, but because AGI safety looks to be an extremely difficult challenge — more challenging than managing climate change, for example — and one requiring several decades of careful preparation.
The greatest risks from both climate change and AI are several decades away, but thousands of smart researchers and policy-makers are already working to understand and mitigate climate change, and only a handful are working on the safety challenges of advanced AI. On the present margin, we should have much less top-flight cognitive talent going into climate change mitigation, and much more going into AGI safety research.

2. We don't think AIs will want to wipe us out. Rather, we worry they'll wipe us out because that is the most effective way to satisfy almost any possible goal function one could have.
Steven Pinker, who incidentally is the author of two of my all-time favorite books, writes:
[one] problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence. Even if we did have superhumanly intelligent robots, why would they want to depose their masters, massacre bystanders, or take over the world? Intelligence is the ability to deploy novel means to attain a goal, but the goals are extraneous to the intelligence itself: being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems.
I'm glad Pinker agrees with what Bostrom calls "the orthogonality thesis": that intelligence and goals are orthogonal to each other.
But our concern is not that superhuman AIs would be megalomaniacal despots. That is anthropomorphism.
Rather, the problem is that taking over the world is a really good idea for almost any goal function a superhuman AI could have. As Yudkowsky wrote, "The AI does not love you, nor does it hate you, but you are made of atoms it can use for something else."
Maybe it just wants to calculate as many digits of pi as possible. Well, the best way to do that is to turn all available resources into computation for calculating more digits of pi, and to eliminate potential threats to its continued calculation, for example those pesky humans that seem capable of making disruptive things like nuclear bombs and powerful AIs. The same logic applies for almost any goal function you can specify. ("But what if it's a non-maximizing goal? And won't it be smart enough to realize that the goal we gave it wasn't what we intended if it means the AI wipes us out to achieve it?" Responses to these and other common objections are given in Superintelligence, ch. 8.)

3. AI self-improvement and protection against external modification isn't just one of many scenarios. Like resource acquisition, self-improvement and protection against external modification are useful for the satisfaction of almost any final goal function.
Kevin Kelly writes:
The usual scary scenario is that an AI will reprogram itself on its own to be unalterable by outsiders. This is conjectured to be a selfish move on the AI's part, but it is unclear how an unalterable program is an advantage to an AI.
As argued above (and more extensively in Superintelligence, ch. 7), resource acquisition is a "convergent instrumental goal." That is, advanced AI agents will be instrumentally motivated to acquire as many resources as feasible, because additional resources are useful for just about any goal function one could have.
Self-improvement is another convergent instrumental goal. For just about any goal an AI could have, it'll be better able to achieve that goal if it's more capable of goal achievement in general.
Another convergent instrumental goal is goal content integrity. As Bostrom puts it, "An agent is more likely to act in the future to maximize the realization of its present final goals if it still has those goals in the future." Thus, it will be instrumentally motivated to prevent external modification of its goals, or of parts of its program that affect its ability to achieve its goals.2
For more on this, see Superintelligence ch. 7.

Conclusion
I'll conclude with the paragraph in the discussion I most agreed with, by Pamela McCorduck:
Yes, the machines are getting smarter—we're working hard to achieve that. I agree with Nick Bostrom that the process must call upon our own deepest intelligence, so that we enjoy the benefits, which are real, without succumbing to the perils, which are just as real. Working out the ethics of what smart machines should, or should not do—looking after the frail elderly, or deciding whom to kill on the battlefield—won't be settled by fast thinking, snap judgments, no matter how heartfelt. This will be a slow inquiry, calling on ethicists, jurists, computer scientists, philosophers, and many others. As with all ethical issues, stances will be provisional, evolve, be subject to revision. I'm glad to say that for the past five years the Association for the Advancement of Artificial Intelligence has formally addressed these ethical issues in detail, with a series of panels, and plans are underway to expand the effort. As Bostrom says, this is the essential task of our century.

Update: Stuart Russell of UC Berkeley has now added a nice reply to the edge.org conversation which echoes some of the points I made above.
I could have also objected to claims and arguments made in the conversation, for example Lanier's claim that "The AI component would be only ambiguously there and of little importance [relative to the actuators component]." To me, this is like saying that humans rule the planet because of our actuators, not because of our superior intelligence. Or in response to Kevin Kelly's claim that "So far as I can tell, AIs have not yet made a decision that its human creators have regretted," I can for example point to the automated trading algorithms that nearly bankrupted Knight Capital faster than any human could react. But in this piece I will focus instead on claims that seem to be misunderstandings of the positive case that's being made for AI as an existential risk.That is, unless it strongly trusts the agent making the external modification, and expects it to do a better job of making those modifications than it could itself, neither of which will be true of humans from the superhuman AI's perspective.The post Three misconceptions in Edge.org's conversation on "The Myth of AI" appeared first on Machine Intelligence Research Institute.

Three misconceptions in Edge.org’s conversation on “The Myth of AI”


Video of Bostrom's talk on Superintelligence at UC Berkeley

In September, MIRI hosted Nick Bostrom at UC Berkeley to discuss his new book Superintelligence. A video and transcript of that talk are now available from BookTV by C-SPAN, which also has a DVD of the event available.
Update: Nick Bostrom has also made his slides for the talk available.

The post Video of Bostrom's talk on Superintelligence at UC Berkeley appeared first on Machine Intelligence Research Institute.

Video of Bostrom’s talk on Superintelligence at UC Berkeley


A new guide to MIRI's research

Nate Soares has written "A Guide to MIRI's Research," which outlines the main thrusts of MIRI's current research agenda and provides recommendations for which textbooks and papers to study so as to understand what's happening at the cutting edge.
This guide replaces Louie Helm's earlier "Recommended Courses for MIRI Math Researchers," and will be updated regularly as new lines of research open up, and as new papers and reports are released. It is not a replacement for our upcoming technical report on MIRI's current research agenda and its supporting papers, which are still in progress. ("Corrigibility" is the first supporting paper we've released for that project.)
The post A new guide to MIRI's research appeared first on Machine Intelligence Research Institute.

A new guide to MIRI’s research


MIRI's November Newsletter

Research Updates

New Friendly AI research area: "Corrigibility."
New report: "UDT with known search order."
2 new analyses: "AGI outcomes and civilizational competence" and "The Financial Times story on MIRI."
Video of Nate Soares' decision theory talk at Purdue.

News Updates

If you're reading Superintelligence or following its online reading group, please take this short survey.

Other Updates

Elon Musk has been talking about AGI risk quite a bit lately. He has also joined the external advisory boards of CSER and FLI, two existential risk organizations with whom we work closely.
Effective altruists may want to register for the EA Donation Registry.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI's November Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s November Newsletter


The Financial Times story on MIRI

Richard Waters wrote a story on MIRI and others for Financial Times, which also put Nick Bostrom's Superintelligence at the top of its summer science reading list.
It's a good piece. Go read it and then come back here so I can make a few clarifications.

1. Smarter-than-human AI probably isn't coming "soon."
"Computers will soon become more intelligent than us," the story begins, but few experts I know think this is likely.
A recent survey asked the world's top-cited living AI scientists by what year they'd assign a 10% / 50% / 90% chance of human-level AI (aka AGI), assuming scientific progress isn't massively disrupted. The median reply for a 10% chance of AGI was 2024, for a 50% chance of AGI it was 2050, and for a 90% chance of AGI it was 2070. So while AI scientists think it's possible we might get AGI soon, they largely expect AGI to be an issue for the second half of this century.
Moreover, many of those who specialize in thinking about AGI safety actually think AGI is further away than the top-cited AI scientists do. For example, relative to the surveyed AI scientists, Nick Bostrom and I both think more probability should be placed on later years. We advocate more work on the AGI safety challenge today not because we think AGI is likely in the next decade or two, but because AGI safety looks to be an extremely difficult challenge — more challenging than managing climate change, for example — and one requiring several decades of careful preparation.
The greatest risks from both climate change and AI are several decades away, but thousands of smart researchers and policy-makers are already working to understand and mitigate climate change, and only a handful are working on the safety challenges of advanced AI. On the present margin, we should have much less top-flight cognitive talent going into climate change mitigation, and much more going into AGI safety research.

2. How many people are working to make sure AGI is friendly to humans?
The FT piece cites me as saying there are only five people in the world "working on how to [program] the super-smart machines of the not-too-distant future to make sure AI remains friendly." I did say something kind of like this, but it requires clarification.
What I mean is that "When you add up fractions of people, there are about five people (that I know of) explicitly doing technical research on the problem of how to ensure that a smarter-than-human AI has a positive impact even as it radically improves itself."
These fractions of people are: (a) most of the full-time labor of Eliezer Yudkowsky, Benja Fallenstein, Nate Soares (all at MIRI), and Stuart Armstrong (Oxford), plus (b) much smaller fractions of people who do technical research on "Friendly AI" on the side, for example MIRI's (unpaid) research associates.
Of course, there are many, many more researchers than this doing (a) non-technical work on AGI safety, or doing (b) technical work on AI safety for extant or near-future systems, or doing (c) occasional technical work on AGI safety done with very different conceptions of "positive impact" or "radically improves itself" than I have.

3. An AGI wouldn't necessarily see humans as "mere" collections of matter.
The article cites me as arguing that "In their single-mindedness, [AGIs] would view their biological creators as mere collections of matter, waiting to be reprocessed into something they find more useful."
AGIs would likely have pretty accurate — and ever-improving — models of reality (e.g. via Wikipedia and millions of scientific papers), so they wouldn't see humans as "mere" collections of matter any more than I do. Sure, humans are collections of matter, but we're pretty special as collections of matter go. Unlike most collections of matter, we have general-purpose intelligence and consciousness and technological creativity and desires and aversions and hopes and fears and so on, and an AGI would know all that, and it would know that rocks and buildings and plants and monkeys and self-driving cars don't have all those properties.
The point I wanted to make is that if a self-improving AGI was (say) programmed to maximize Shell's stock price, then it would know all this about humans, and then it would just go on maximizing Shell's stock price. It just happens to be the case that the best way to maximize Shell's stock price is to take over the world and eliminate all potential threats to one's achievement of that goal. In fact, for just about any goal function an AGI could have, it's a really good idea to take over the world. That is the problem.
Even if we could program a self-improving AGI to (say) "maximize human happiness," then the AGI would "care about humans" in a certain sense, but it might learn that (say) the most efficient way to "maximize human happiness" in the way we specified is to take over the world and then put each of us in a padded cell with a heroin drip. AGI presents us with the old problem of the all-too-literal genie: you get what you actually asked for, not what you wanted.
And yes, the AGI would be smart enough to know this wasn't what we really wanted, especially when we start complaining about the padded cells. But we didn't program it to do what we want. We programmed it to "maximize human happiness."
The trouble is that "what we really want" is very hard to specify in computer code. Twenty centuries of philosophers haven't even managed to specify it in less-exacting human languages.

4. "Toying with the intelligence of the gods."
Finally, the article quotes me as saying "We're toying with the intelligence of the gods. And there isn't an off switch."
I shouldn't complain about Mr. Waters making me sound so eloquent, but I'm pretty sure I never said anything so succinct and quotable.
And of course, there is an off switch today, but there probably won't be an off switch for an AGI smart enough to remove its shutdown mechanism (so as to more assuredly achieve its programmed goals) and copy itself across the internet — unless, that is, we solve the technical problem we call "corrigibility."
The post The Financial Times story on MIRI appeared first on Machine Intelligence Research Institute.

The Financial Times story on MIRI


New report: "UDT with known search order"

Today we release a new technical report from MIRI research associate Tsvi Benson-Tilsen: "UDT with known search order." Abstract:
We consider logical agents in a predictable universe running a variant of updateless decision theory. We give an algorithm to predict the behavior of such agents in the special case where the order in which they search for proofs is simple, and where they know this order. As a corollary, "playing chicken with the universe" by diagonalizing against potential spurious proofs is the only way to guarantee optimal behavior for this class of simple agents.
The post New report: "UDT with known search order" appeared first on Machine Intelligence Research Institute.

New report: “UDT with known search order”


Singularity2014.com appears to be a fake

Earlier today I was alerted to the existence of Singularity2014.com (archived screenshot). MIRI has nothing to do with that website and we believe it is a fake.
The website claims there is a "Singularity 2014″ conference "in the Bay Area" on "November 9, 2014." We believe that there is no such event. No venue is listed, tickets are supposedly sold out already, and there are no links to further information. The three listed speakers are unknown to us, and their supposed photos are stock photos (1, 2, 3). The website prominently features an image of Ray Kurzweil, but Ray Kurzweil's press staff confirms that he has nothing to do with this event. The website also features childish insults and a spelling error.
The website claims the event is "staged and produced by former organizers of the Singularity Summit from the Machine Intelligence Research Institute," and that "All profits benefit the Machine Intelligence Research Institute," but MIRI has nothing to do with this supposed event.
The Singularity2014.com domain name was registered via eNom reseller NameCheap.com on September 15th, 2014 by someone other than us, and is associated with a P.O. Box in Panama.
MIRI is collaborating with Singularity University to have the website taken down. If you have information about who is responsible for this, please contact <EMAIL>.
The next Singularity Summit will be organized primarily by Singularity University; for more information see here.

Update: The website has been taken down.
The post Singularity2014.com appears to be a fake appeared first on Machine Intelligence Research Institute.


New paper: "Corrigibility"

Today we release a paper describing a new problem area in Friendly AI research we call corrigibility. The report (PDF) is co-authored by MIRI's Friendly AI research team (Eliezer Yudkowsky, Benja Fallenstein, Nate Soares) and also Stuart Armstrong from the Future of Humanity Institute at Oxford University.
The abstract reads:
As artificially intelligent systems grow in intelligence and capability, some of their available options may allow them to resist intervention by their programmers. We call an AI system "corrigible" if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences. We introduce the notion of corrigibility and analyze utility functions that attempt to make an agent shut down safely if a shutdown button is pressed, while avoiding incentives to prevent the button from being pressed or cause the button to be pressed, and while ensuring propagation of the shutdown behavior as it creates new subsystems or self-modifies. While some proposals are interesting, none have yet been demonstrated to satisfy all of our intuitive desiderata, leaving this simple problem in corrigibility wide-open.
This paper was accepted to the AI & Ethics workshop at AAAI-2015.
Update: The slides for Nate Soares' presentation at AAAI-15 are available here.
The post New paper: "Corrigibility" appeared first on Machine Intelligence Research Institute.

New paper: “Corrigibility”


AGI outcomes and civilizational competence

The [latest IPCC] report says, "If you put into place all these technologies and international agreements, we could still stop warming at [just] 2 degrees." My own assessment is that the kinds of actions you'd need to do that are so heroic that we're not going to see them on this planet.
—David Victor,1 professor of international relations at UCSD

A while back I attended a meeting of "movers and shakers" from science, technology, finance, and politics. We were discussing our favorite Big Ideas for improving the world. One person's Big Idea was to copy best practices between nations. For example when it's shown that nations can dramatically improve organ donation rates by using opt-out rather than opt-in programs, other countries should just copy that solution.
Everyone thought this was a boring suggestion, because it was obviously a good idea, and there was no debate to be had. Of course, they all agreed it was also impossible and could never be established as standard-practice. So we moved on to another Big Idea that was more tractable.
Later, at a meeting with a similar group of people, I told some economists that their recommendations on a certain issue were "straightforward econ 101," and I didn't have any objections to share. Instead, I asked, "But how can we get policy-makers to implement econ 101 solutions?" The economists laughed and said, "Well, yeah, we have no idea. We probably can't."
How do I put this? This is not a civilization that should be playing with self-improving AGIs.2

The backhoe is a powerful, labor-saving invention, but I wouldn't put a two-year-old in the driver's seat. That's roughly how I feel about letting 21st century humans wield something as powerful as self-improving AGI.3 I wish we had more time to grow up first. I think the kind of actions we'd need to handle self-improving AGI successfully "are so heroic that we're not going to see them on this planet," at least not anytime soon.4
But I suspect we won't all resist the temptation to build AGI for long, and neither do most top AI scientists.5 There's just too much incentive to build AGI: a self-improving AGI could give its makers — whether Google or the NSA or China or somebody else — history's greatest first-mover advantage. Even if the first few teams design their AGIs wisely, the passage of time will only make it easier for smaller and less-wise teams to cross the finish line. Moore's Law of Mad Science, and all that.
Some people are less worried than I am about self-improving AGI. After all, one might have predicted in 1950 that we wouldn't have the civilizational competence to avoid an all-out nuclear war for the next half-century, but we did avoid it (if only barely).6 So maybe we shouldn't be so worried about AGI, either.
While I think it's important consider such second-guessing arguments, I generally try to take the world at face value. When I look at the kinds of things we succeed at, and the kinds of things we fail at, getting good outcomes from AGI looks much harder than the kinds of things we routinely fail at, like bothering to switch to opt-out programs for organ donation.
But I won't pretend that this question of civilizational competence has been settled. If it's possible to settle the issue at all, doing so would require a book-length argument if not more. (Nick Bostrom's Superintelligence says a lot about why the AGI control problem is hard, but it doesn't say much about whether humanity is likely to rise to that challenge.7 )
What's the point of trying to answer this question? If my view is correct, I think the upshot is that we need to re-evaluate our society's differential investment in global challenges. If you want to succeed in the NBA but you're only 5'3″ tall, then you'll just have to invest more time and effort on your basketball goals than you do on other goals for which you're more naturally suited. And if we want our civilization to survive self-improving AGI, but our civilization can't even manage to switch to opt-out programs for organ donation, then we'll just have to start earlier, try harder, spend more, etc. on surviving AGI than we do when pursuing other goals for which our civilization is more naturally suited, like building awesome smartphones.
But if I'm wrong, and our civilization is on course to handle AGI just like it previously handled, say, CFCs, then there may be more urgent things to be doing than advancing Friendly AI theory. (Still, it would be surprising if more Friendly AI work wasn't good on the present margin, given that there are fewer than 5 full-time Friendly AI researchers in the world right now.)
I won't be arguing for my own view in this post. Instead I merely want to ask: how might one study this question of civilizational competence and the arrival of AGI? I'd probably split the analysis into two parts: (1) the apparent shape and difficulty of the AGI control problem, and (2) whether we're likely to have the civilizational competence to handle a problem of that shape and difficulty when it knocks on our front door.
Note that everything in this post is a gross simplification. Problem Difficulty and Civilizational Competence aren't one-dimensional concepts, though to be succinct I sometimes talk as if they are. But a problem like AGI control is difficult to different degrees in different ways, some technical and others political, and different parts of our civilization are differently competent in different ways, and those different kinds of competence are undergoing different trends.

Difficulty of the problem
How hard is the AGI control problem, and in which ways is it hard? To illustrate what such an analysis could look like, I might sum up my own thoughts on this like so:

The control problems that are novel to AGI look really hard. For example, getting good outcomes from self-improving AGI seems to require as-yet unobserved philosophical success — philosophical success that is not required merely to write safe autopilot software. More generally, there seem to be several novel problems that arise when we're trying to control a system more generally clever and powerful than ourselves — problems we have no track record of solving for other systems, problems which seem analogous to the hopeless prospect of chimpanzees getting humans to reliably do what the chimps want. (See Superintelligence.)
Moreover, we may get relatively few solid chances to solve these novel AGI control problems before we reach a "no turning back" point in AGI capability. In particular, (1) a destabilizing arms race between nations and/or companies, incentivizing speed of development over safety of development, seems likely, and (2) progress may be rapid right when novel control problems become relevant. As a species we are not so good at getting something exactly right on one of our first 5 tries — instead, we typically get something right by learning from dozens or hundreds of initial failures. But we may not have that luxury with novel AGI control problems.
The AGI control challenge looks especially susceptible to a problem known as "positional externalities" — an arms race is but one example — along with related coordination problems. (I explain the notion of "positional externalities" further in a footnote.8 )

But that's just a rough sketch, and other thinkers might have different models of the shape and difficulty of the AGI control problem.
Civilizatonal competence
Second, will our civilization rise to the challenge? Will our civilizational competence at the time of AGI invention be sufficient to solve the AGI control problem?
My own pessimism on this question doesn't follow from any conceptual argument or any simple extrapolation of current trends. Rather, it comes from the same kind of multi-faceted empirical reasoning that you probably do when you try to think about whether we're more likely to have, within 30 years, self-driving taxis or a Mars colony. That is, I'm combining different models I have about how the world works in general: the speed of development in space travel vs. AI, trends in spending on both issues, which political and commercial incentives are at play, which kinds of coordination problems must be solved, what experts in the relevant fields seem to think about the issues, what kinds of questions experts are better or worse at answering, etc. I'm also adjusting that initial combined estimate based on some specific facts I know: about terraforming, about radiation hazards, about autonomous vehicles, etc.9
Unfortunately, because one's predictions about AGI outcomes can't be strongly supported by simple conceptual arguments, it's a labor-intensive task to try to explain one's views on the subject, which may explain why I haven't seen a good, thorough case for either optimism or pessimism about AGI outcomes. People just have their views, based on what they anticipate about the world, and it takes a lot of work to explain in detail where those views are coming from. Nevertheless, it'd be nice to see someone try.
As a start, I'll link to some studies which share some methodological features of the kind of investigation I'm suggesting:

Libecap (2013) investigates why some global externalities are addressed effectively whereas others are not.
Many scholars test theories of system accidents, such as "high reliability theory" and "normal accidents theory," against historical data. For a summary, see Sagan (1993), ch. 1.
Yudkowsky (2008) examines cognitive biases potentially skewing judgments on catastrophic risks specifically.
Sztompka (1993) argues that slowness of the post-Soviet Eastern European recovery can be blamed on a certain kind of civilizational incompetence.
Shanteau (1992), Kahneman & Klein (2009), Tetlock (2005), Ericsson et al. (2006), Mellers et al. (2014), and many other studies discuss conditions for good expert judgment or prediction. Presumably our civilizational competence is greater where domain experts are capable of making good judgements about the likely outcomes of potential interventions.
Sunstein (2009), ch. 2, examines why the Montreal Protocol received more universal support than the Kyoto Protocol.
Schlager & Petroski (1994) and Chiles (2008) collect case studies of significant technological disasters of the 20th century.

If you decide to perform some small piece of this analysis project, please link to your work in the comments below.
 Quote taken from the Radiolab episode titled "In the Dust of This Planet."In Superintelligence, Bostrom made the point this way (p. 259):
Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct… For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible… Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the entire firmament. Nor is there a grown-up in sight.
 By "AGI" I mean a computer system that could pass something like Nilsson's employment test (see What is AGI?). By "self-improving AGI" I mean an AGI that improves its own capabilities via its own original computer science and robotics research (and not solely by, say, gathering more data about the world or acquiring more computational resources). By "its own capabilities" I mean to include the capabilities of successor systems that the AGI itself creates to further its goals. In this article I typically mean "AGI" and "self-improving AGI" interchangeably, not because all AGIs will necessarily be self-improving in a strong sense, but because I expect that even if the first AGIs are not self-improving for some reason, self-improving AGIs will follow in a matter of decades if not sooner. From a cosmological perspective, such a delay is but a blink.I purposely haven't pinned down exactly what about our civilization seems inadequate to meet the challenge of AGI control; David Victor made the same choice when he made his comment about civilizational competence in the face of climate change. I think our civilizational competence is insufficient for the challenge for many reasons, but I also have varying degrees of uncertainty about each those reasons and which parts of the problem they apply to, and those details are difficult to express.See the AI timeline predictions for the TOP100 poll in Müller & Bostrom (2014). The authors asked a sample of the top-cited living AI scientists: "For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for [an AGI] to exist?" The median reply for each confidence level was 2024, 2050, and 2070, respectively.
Why trust AI scientists at all? Haven't they been wildly optimistic about AI progress from the beginning? Yes, there are embarrassing quotes from early AI scientists about how fast AI progress would be, but there are also many now-disproven quotes from early AI skeptics about what AI wouldn't be able to do. The earliest survey of AI scientists we have is from 1973, and the most popular response to that survey's question about AGI timelines was the most pessimistic option, "more than 50 years." (Which, assuming we don't get AGI by 2023, will end up being correct.) An interesting sub-question: Does humanity's competence keep up with its capability? When our capabilities jump, as they did with the invention of nuclear weapons, does our competence in controlling those capabilities also jump, out of social/moral necessity or some other forces? Einstein said "Nuclear weapons have changed everything, except our modes of thought," suggesting that he expected us not to mature as "adults" quickly enough to manage nuclear weapons wisely. We haven't exactly handled them "wisely," but we've at least handled them wisely enough to avoid global nuclear catastrophe so far.What Superintelligence says on the topic can be found in chapter 14.Frank (1991) explains the concept this way:
In Micromotives and Macrobehavior, Thomas Schelling observes that hockey players, left to their own devices, almost never wear helmets, even though almost all of them would vote for helmet rules in secret ballots. Not wearing a helmet increases the odds of winning, perhaps by making it slightly easier to see and hear… At the same time, not wearing a helmet increases the odds of getting hurt. If players value the higher odds of winning more than they value the extra safety, it is rational not to wear helmets. The irony, Schelling observes, is that when all discard their helmets, the competitive balance is the same as if all had worn them.
The helmet problem is an example of what we may call a positional externality. The decision to wear a helmet has important effects not only for the person who wears it, but also for the frame of reference in which he and others operate. In such situations, the payoffs to individuals depend in part on their positions within the frame of reference. With hockey players, what counts is not their playing ability in any absolute sense, but how they perform relative to their opponents. Where positional externalities… are present, Schelling has taught us, individually rational behavior often adds up to a result that none would have chosen.
An arms race is one well-understood kind of positional externality. As Alexander (2014) puts it, "From a god's-eye-view, the best solution is world peace and no country having an army at all. From within the system, no country can unilaterally enforce that, so their best option is to keep on throwing their money into missiles…"In other words, my reasons for AGI outcomes pessimism look like a model combination and adjustment. Or you can think of it in terms of what Holden Karnofsky calls cluster thinking. Or as one of my early draft readers called it, "normal everyday reasoning."The post AGI outcomes and civilizational competence appeared first on Machine Intelligence Research Institute.


Nate Soares' talk: "Why ain't you rich?"

On September 18th, MIRI research fellow Nate Soares spoke at Purdue University's Dawn or Doom seminar. Slides, video, and a transcript of his talk — "Why ain't you rich? Why Our Current Understanding of 'Rational Choice' Isn't Good Enough for Superintelligence" — are now available.

The post Nate Soares' talk: "Why ain't you rich?" appeared first on Machine Intelligence Research Institute.

Nate Soares’ talk: “Why ain’t you rich?”


MIRI's October Newsletter

Research Updates

Our major project last month was our Friendly AI technical agenda overview and supporting papers, the former of which is now in late draft form but not yet ready for release.
4 new expert interviews, including John Fox on AI safety.
MIRI research fellow Nate Soares has begun to explain some of the ideas motivating MIRI's current research agenda at his blog. See especially Newcomblike problems are the norm.

News Updates

Friendly AI research help from MIRI.
We're in week three of our Superintelligence reading group, and it's not too late to join the discussion.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI's October Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s October Newsletter