MIRI's Approach

MIRI's mission is "to ensure that the creation of smarter-than-human artificial intelligence has a positive impact." How can we ensure any such thing? It's a daunting task, especially given that we don't have any smarter-than-human machines to work with at the moment. In the previous post I discussed four background claims that motivate our mission; in this post I will describe our approach to addressing the challenge.
This challenge is sizeable, and we can only tackle a portion of the problem. For this reason, we specialize. Our two biggest specializing assumptions are as follows:
We focus on scenarios where smarter-than-human machine intelligence is first created in de novo software systems (as opposed to, say, brain emulations).
This is in part because it seems difficult to get all the way to brain emulation before someone reverse-engineers the algorithms used by the brain and uses them in a software system, and in part because we expect that any highly reliable AI system will need to have at least some components built from the ground up for safety and transparency. Nevertheless, it is quite plausible that early superintelligent systems will not be human-designed software, and I strongly endorse research programs that focus on reducing risks along the other pathways.
We specialize almost entirely in technical research.
We select our researchers for their proficiency in mathematics and computer science, rather than forecasting expertise or political acumen. I stress that this is only one part of the puzzle: figuring out how to build the right system is useless if the right system does not in fact get built, and ensuring AI has a positive impact is not simply a technical problem. It is also a global coordination problem, in the face of short-term incentives to cut corners. Addressing these non-technical challenges is an important task that we do not focus on.
In short, MIRI does technical research to ensure that de novo AI software systems will have a positive impact. We do not further discriminate between different types of AI software systems, nor do we make strong claims about exactly how quickly we expect AI systems to attain superintelligence. Rather, our current approach is to select open problems using the following question:
What would we still be unable to solve, even if the challenge were far simpler?
For example, we might study AI alignment problems that we could not solve even if we had lots of computing power and very simple goals.
We then filter on problems that are (1) tractable, in the sense that we can do productive mathematical research on them today; (2) uncrowded, in the sense that the problems are not likely to be addressed during normal capabilities research; and (3) critical, in the sense that they could not be safely delegated to a machine unless we had first solved them ourselves. (Since the goal is to design intelligent machines, there are many technical problems that we can expect to eventually delegate to those machines. But it is difficult to trust an unreliable reasoner with the task of designing reliable reasoning!)
These three filters are usually uncontroversial. The controversial claim here is that the above question — "what would we be unable to solve, even if the challenge were simpler?" — is a generator of open technical problems for which solutions will help us design safer and more reliable AI software in the future, regardless of their architecture. The rest of this post is dedicated to justifying this claim, and describing the reasoning behind it.

1. Creating a powerful AI system without understanding why it works is dangerous.
A large portion of the risk from machine superintelligence comes from the possibility of people building systems that they do not fully understand.
Currently, this is commonplace in practice: many modern AI researchers are pushing the capabilities of deep neural networks in the absence of theoretical foundations that describe why they're working so well or a solid idea of what goes on beneath the hood. These shortcomings are being addressed over time: many AI researchers are currently working on transparency tools for neural networks, and many more are working to put theoretical foundations beneath deep learning systems. In the interim, using trial and error to push the capabilities of modern AI systems has led to many useful applications.
When designing a superintelligent agent, by contrast, we will want an unusually high level of confidence in its safety before we begin online testing: trial and error alone won't cut it, in that domain.
To illustrate, consider a study by Bird and Layzell in 2002. They used some simple genetic programming to design an oscillating circuit on a circuit board. One solution that the genetic algorithm found entirely avoided using the built-in capacitors (an essential piece of hardware in human-designed oscillators). Instead, it repurposed the circuit tracks on the motherboard as a radio receiver, and amplified an oscillating signal from a nearby computer.
This demonstrates that powerful search processes can often reach their goals via unanticipated paths. If Bird and Layzell were hoping to use their genetic algorithm to find code for a robust oscillating circuit — one that could be used on many different circuit boards regardless of whether there were other computers present — then they would have been sorely disappointed. Yet if they had tested their algorithms extensively on a virtual circuit board that captured all the features of the circuit board that they thought were relevant (but not features such as "circuit tracks can carry radio signals"), then they would not have noticed the potential for failure during testing. If this is a problem when handling simple genetic search algorithms, then it will be a much larger problem when handling smarter-than-human search processes.
When it comes to designing smarter-than-human machine intelligence, extensive testing is essential, but not sufficient: in order to be confident that the system will not find unanticipated bad solutions when running in the real world, it is important to have a solid understanding of how the search process works and why it is expected to generate only satisfactory solutions in addition to empirical test data.
MIRI's research program is aimed at ensuring that we have the tools needed to inspect and analyze smarter-than-human search processes before we deploy them.
By analogy, neural net researchers could probably have gotten quite far without having any formal understanding of probability theory. Without probability theory, however, they would lack the tools needed to understand modern AI algorithms: they wouldn't know about Bayes nets, they wouldn't know how to formulate assumptions like "independent and identically distributed," and they wouldn't quite know the conditions under which Markov Decision Processes work and fail. They wouldn't be able to talk about priors, or check for places where the priors are zero (and therefore identify things that their systems cannot learn). They wouldn't be able to talk about bounds on errors and prove nice theorems about algorithms that find an optimal policy eventually.
They probably could have still gotten pretty far (and developed half-formed ad-hoc replacements for many of these ideas), but without probability theory, I expect they would have a harder time designing highly reliable AI algorithms. Researchers at MIRI tend to believe that similarly large chunks of AI theory are still missing, and those are the tools that our research program aims to develop.
2. We could not yet create a beneficial AI system even via brute force.
Imagine you have a Jupiter-sized computer and a very simple goal: Make the universe contain as much diamond as possible. The computer has access to the internet and a number of robotic factories and laboratories, and by "diamond" we mean carbon atoms covalently bound to four other carbon atoms. (Pretend we don't care how it makes the diamond, or what it has to take apart in order to get the carbon; the goal is to study a simplified problem.) Let's say that the Jupiter-sized computer is running python. How would you program it to produce lots and lots of diamond?
As it stands, we do not yet know how to program a computer to achieve a goal such as that one.
We couldn't yet create an artificial general intelligence by brute force, and this indicates that there are parts of the problem we don't yet understand.
There are a number of AI tasks that we could brute-force. For example, we could write a program that would be really, really good at solving computer vision problems: if we had an indestructible box that produced pictures and questions about them, waited for answers, scored the answers for accuracy, and then repeated the process, then we know how to write the program that interacts with that box and gets very good at answering the questions. (The program would essentially be a bounded version of AIXI.)
By a similar method, if we had an indestructible box that produced a conversation and questions about it, waited for natural-language answers to the questions, and scored them for accuracy, then again, we could write a program that would get very good at answering well. In this sense, we know how to solve computer vision and natural language processing by brute force. (Of course, natural-language processing is nowhere near "solved" in a practical sense — there is still loads of work to be done. A brute force solution doesn't get you very far in the real world. The point is that, for many AI alignment problems, we haven't even made it to the "we could brute force it" level yet.)
Why do we need the indestructible box in the above examples? Because the way the modern brute-force solution would work is by considering each Turing machine (up to some complexity limit) as a hypothesis about the box, seeing which ones are consistent with observation, and then executing actions that lead to high scores coming out of the box (as predicted by the remaining hypotheses, weighted by simplicity).
Each hypothesis is an opaque Turing machine, and the algorithm never peeks inside: it just asks each hypothesis to predict what score the box will output if it executes a certain action chain. This means that if the algorithm finds (via exhaustive search) a plan that maximizes the score coming out of the box, and the box is destructible, then the opaque action chain that maximizes score is very likely to be the one that pops the box open and alters it so that it always outputs the highest score. But given an indestructible box, we know how to brute force the answers.
In fact, roughly speaking, we understand how to solve any reinforcement learning problem via brute force. This is a far cry from knowing how to practically solve reinforcement learning problems! But it does illustrate a difference in kind between two types of problems. We can (imperfectly and heuristically) divide AI problems up as follows:
There are two types of open problem in AI. One is figuring how to solve in practice problems that we know how to solve in principle. The other is figuring out how to solve in principle problems that we don't even know how to brute force yet.
MIRI focuses on problems of the second class.1
What is hard about brute-forcing a diamond-producing agent? To illustrate, I'll give a wildly simplified sketch of what an AI program needs to do in order to act productively within a complex environment:

Model the world: Take percepts, and use them to refine some internal representation of the world the system is embedded in.
Predict the world: Take that world-model, and predict what would happen if the system executed various different plans.
Rank outcomes: Rate those possibilities by how good the predicted future is, then execute a plan that leads to a highly-rated outcome.2

Consider the modeling step. As discussed above, we know how to write an algorithm that finds good world-models by brute force: it looks at lots and lots of Turing machines, weighted by simplicity, treats them like they are responsible for its observations, and throws out the ones that are inconsistent with observation thus far. But (aside from being wildly impractical) this yields only opaque hypotheses: the system can ask what "sensory bits" each Turing machine outputs, but it cannot peek inside and examine objects represented within.
If there is some well-defined "score" that gets spit out by the opaque Turing machine (as in a reinforcement learning problem), then it doesn't matter that each hypothesis is a black box; the brute-force algorithm can simply run the black box on lots of inputs and see which results in the highest score. But if the problem is to build lots of diamond in the real world, then the agent must work as follows:

Build a model of the world — one that represents carbon atoms and covalent bonds, among other things.
Predict how the world would change contingent on different actions the system could execute.
Look inside each prediction and see which predicted future has the most diamond. Execute the action that leads to more diamond.

In other words, an AI that is built to reliably affect things in the world needs to have world-models that are amenable to inspection. The system needs to be able to pop open the world model, identify the representations of carbon atoms and covalent bonds, and estimate how much diamond is in the real world.3
We don't yet have a clear picture of how to build "inspectable" world-models — not even by brute force. Imagine trying to write the part of the diamond-making program that builds a world-model: this function needs to take percepts as input and build a data structure that represents the universe, in a way that allows the system to inspect universe-descriptions and estimate the amount of diamond in a possible future. Where in the data structure are the carbon atoms? How does the data structure allow the concept of a "covalent bond" to be formed and labeled, in such a way that it remains accurate even as the world-model stops representing diamond as made of atoms and starts representing them as made of protons, neutrons, and electrons instead?
We need a world-modeling algorithm that builds multi-level representations of the world and allows the system to pursue the same goals (make diamond) even as its model changes drastically (because it discovers quantum mechanics). This is in stark contrast to the existing brute-force solutions that use opaque Turing machines as hypotheses.4
When humans reason about the universe, we seem to do some sort of reasoning outwards from the middle: we start by modeling things like people and rocks, and eventually realize that these are made of atoms, which are made of protons and neutrons and electrons, which are perturbations in quantum fields. At no point are we certain that the lowest level in our model is the lowest level in reality; as we continue thinking about the world we construct new hypotheses to explain oddities in our models. What sort of data structure are we using, there? How do we add levels to a world model given new insights? This is the sort of reasoning algorithm that we do not yet understand how to formalize.5
That's step one in brute-forcing an AI that reliably pursues a simple goal. We also don't know how to brute-force steps two or three yet. By simplifying the problem — talking about diamonds, for example, rather than more realistic goals that raise a host of other difficulties — we're able to factor out the parts of the problems that we don't understand how to solve yet, even in principle. Our technical agenda describes a number of open problems identified using this method.
3. Figuring out how to solve a problem in principle yields many benefits.
In 1836, Edgar Allen Poe wrote a wonderful essay on Maelzel's Mechanical Turk, a machine that was purported to be able to play chess. In the essay, Poe argues that the Mechanical Turk must be a hoax: he begins by arguing that machines cannot play chess, and proceeds to explain (using his knowledge of stagecraft) how a person could be hidden within the machine. Poe's essay is remarkably sophisticated, and a fun read: he makes reference to the "calculating machine of Mr. Babbage" and argues that it cannot possibly be made to play chess, because in a calculating machine, each steps follows from the previous step by necessity, whereas "no one move in chess necessarily follows upon any one other".
The Mechnical Turk indeed turned out to be a hoax. In 1950, however, Claude Shannon published a rather compelling counterargument to Poe's reasoning in the form of a paper explaining how to program a computer to play perfect chess.
Shannon's algorithm was by no means the end of the conversation. It took forty-six years to go from that paper to Deep Blue, a practical chess program which beat the human world champion. Nevertheless, if you were equipped with Poe's state of knowledge and not yet sure whether it was possible for a computer to play chess — because you did not yet understand algorithms for constructing game trees and doing backtracking search — then you would probably not be ready to start writing practical chess programs.
Similarly, if you lacked the tools of probability theory — an understanding of Bayesian inference and the limitations that stem from bad priors — then you probably wouldn't be ready to program an AI system that needed to manage uncertainty in high-stakes situations.
If you are trying to write a program and you can't yet say how you would write it given an arbitrarily large computer, then you probably aren't yet ready to design a practical approximation of the brute-force solution yet. Practical chess programs can't generate a full search tree, and so rely heavily on heuristics and approximations; but if you can't brute-force the answer yet given arbitrary amounts of computing power, then it's likely that you're missing some important conceptual tools.
Marcus Hutter (inventor of AIXI) and Shane Legg (inventor of the Universal Measure of Intelligence) seem to endorse this approach. Their work can be interpreted as a description of how to find a brute-force solution to any reinforcement learning problem, and indeed, the above description of how to do this is due to Legg and Hutter.
In fact, the founders of Google DeepMind reference the completion of Shane's thesis as one of four key indicators that the time was ripe to begin working on AGI: a theoretical framework describing how to solve reinforcement learning problems in principle demonstrated that modern understanding of the problem had matured to the point where it was time for the practical work to begin.
Before we gain a formal understanding of the problem, we can't be quite sure what the problem is. We may fail to notice holes in our reasoning; we may fail to bring the appropriate tools to bear; we may not be able to tell when we're making progress. After we gain a formal understanding of the problem in principle, we'll be in a better position to make practical progress.
The point of developing a formal understanding of a problem is not to run the resulting algorithms. Deep Blue did not work by computing a full game tree, and DeepMind is not trying to implement AIXI. Rather, the point is to identify and develop the basic concepts and methods that are useful for solving the problem (such as game trees and backtracking search algorithms, in the case of chess).
The development of probability theory has been quite useful to the field of AI — not because anyone goes out and attempts to build a perfect Bayesian reasoner, but because probability theory is the unifying theory for reasoning under uncertainty. This makes the tools of probability theory useful for AI designs that vary in any number of implementation details: any time you build an algorithm that attempts to manage uncertainty, a solid understanding of probabilistic inference is helpful when reasoning about the domain in which the system will succeed and the conditions under which it could fail.
This is why we think we can identify open problems that we can work on today, and which will reliably be useful no matter how the generally intelligent machines of the future are designed (or how long it takes to get there). By seeking out problems that we couldn't solve even if the problem were much easier, we hope to identify places where core AGI algorithms are missing. By developing a formal understanding of how to address those problems in principle, we aim to ensure that when it comes time to address those problems in practice, programmers have the knowledge they need to develop solutions that they deeply understand, and the tools they need to ensure that the systems they build are highly reliable.
4. This is an approach researchers have used successfully in the past.
Our main open-problem generator — "what would we be unable to solve even if the problem were easier?" — is actually a fairly common one used across mathematics and computer science. It's more easy to recognize if we rephrase it slightly: "can we reduce the problem of building a beneficial AI to some other, simpler problem?"
For example, instead of asking whether you can program a Jupiter-sized computer to produce diamonds, you could rephrase this as a question about whether we can reduce the diamond maximization problem to known reasoning and planning procedures. (The current answer is "not yet.")
This is a fairly standard practice in computer science, where reducing one problem to another is a key feature of computability theory. In mathematics it is common to achieve a proof by reducing one problem to another (see, for instance, the famous case of Fermat's last theorem). This helps one focus on the parts of the problem that aren't solved, and identify topics where foundational understanding is lacking.
As it happens, humans have a pretty good track record when it comes to working on problems such as these. Humanity hasn't been very good at predicting long-term technological trends, but we have reasonable success developing theoretical foundations for technical problems decades in advance, when we put sufficient effort into it. Alan Turing and Alonzo Church succeeded in developing a robust theory of computation that proved quite useful once computers were developed, in large part by figuring out how to solve (in principle) problems which they did not yet know how to solve with machines. Andrey Kolmogorov, similarly, set out to formalize intuitive but not-yet-well-understood methods for managing uncertainty; and he succeeded. And Claude Shannon and his contemporaries succeeded at this endeavor in the case of chess.
The development of probability theory is a particularly good analogy to our case: it is a field where, for hundreds of years, philosophers and mathematicians who attempted to formalize their intuitive notions of "uncertainty" repeatedly reasoned themselves into paradoxes and contradictions. The probability theory at the time, sorely lacking formal foundations, was dubbed a "theory of misfortune." Nevertheless, a concerted effort by Kolmogorov and others to formalize the theory was successful, and his efforts inspired the development of a host of useful tools for designing systems that reason reliably under uncertainty.
Many people who set out to put foundations under a new field of study (that was intuitively understood on some level but not yet formalized) have succeeded, and their successes have been practically significant. We aim to do something similar for a number of open problems pertaining to the design of highly reliable reasoners.

The questions MIRI focuses on, such as "how would one ideally handle logical uncertainty?" or "how would one ideally build multi-level world models of a complex environment?", exist at a level of generality comparable to Kolmogorov's "how would one ideally handle empirical uncertainty?" or Hutter's "how would one ideally maximize reward in an arbitrarily complex environment?" The historical track record suggests that these are the kinds of problems that it is possible to both (a) see coming in advance, and (b) work on without access to a concrete practical implementation of a general intelligence.
By identifying parts of the problem that we would still be unable to solve even if the problem was easier, we hope to hone in on parts of the problem where core algorithms and insights are missing: algorithms and insights that will be useful no matter what architecture early intelligent machines take on, and no matter how long it takes to create smarter-than-human machine intelligence.
At present, there are only three people on our research team, and this limits the number of problems that we can tackle ourselves. But our approach is one that we can scale up dramatically: it has generated a very large number of open problems, and we have no shortage of questions to study.6
This is an approach that has often worked well in the past for humans trying to understand how to approach a new field of study, and I am confident that this approach is pointing us towards some of the core hurdles in this young field of AI alignment.

Most of the AI field focuses on problems of the first class. Deep learning, for example, is a very powerful and exciting tool for solving problems that we know how to brute-force, but which were, up until a few years ago, wildly intractable. Class 1 problems tend to be important problems for building more capable AI systems, but lower-priority for ensuring that highly capable systems are aligned with our interests.In reality, of course, there aren't clean separations between these steps. The "prediction" step must be more of a ranking-dependent planning step, to avoid wasting computation predicting outcomes that will obviously be poorly-ranked. The modeling step depends on the prediction step, because which parts of the world-model are refined depends on what the world-model is going to be used for. A realistic agent would need to make use of meta-planning to figure out how to allocate resources between these activities, etc. This diagram is a fine first approximation, though: if a system doesn't do something like modeling the world, predicting outcomes, and ranking them somewhere along the way, then it will have a hard time steering the future.In reinforcement learning problems, this issue is avoided via a special "reward channel" intended to stand in indirectly for something the supervisor wants. (For example, the supervisor may push a reward button every time the learner takes an action that seems, to the supervisor, to be useful for making diamonds.) Then the programmers can, by hand, single out the reward channel inside the world-model and program the system to execute actions that it predicts lead to high reward. This is much easier than designing world-models in such a way that the system can reliably identify representations of carbon atoms and covalent bonds within it (especially if the world is modeled in terms of Newtonian mechanics one day and quantum mechanics the next), but doesn't provide a framework for agents that must autonomously learn how to achieve some goal. Correct behavior in highly intelligent systems will not always be reducible to maximizing a reward signal controlled by a significantly less intelligent system (e.g., a human supervisor).The idea of a search algorithm that optimizes according to modeled facts about the world rather than just expected percepts may sound basic, but we haven't found any deep insights (or clever hacks) that allow us to formalize this idea (e.g., as a brute-force algorithm). If we could formalize it, we would likely get a better understanding of the kind of abstract modeling of objects and facts that is required for self-referential, logically uncertain, programmer-inspectable reasoning.We also suspect that a brute-force algorithm for building multi-level world models would be much more amenable to being "scaled down" than Solomonoff induction, and would therefore lend some insight into how to build multi-level world models in a practical setting.For example, instead of asking what problems remain when given lots of computing power, you could instead ask whether we can reduce the problem of building an aligned AI to the problem of making reliable predictions about human behavior: an approach advocated by others.The post MIRI's Approach appeared first on Machine Intelligence Research Institute.

MIRI’s Approach


Four Background Claims

MIRI's mission is to ensure that the creation of smarter-than-human artificial intelligence has a positive impact. Why is this mission important, and why do we think that there's work we can do today to help ensure any such thing?
In this post and my next one, I'll try to answer those questions. This post will lay out what I see as the four most important premises underlying our mission. Related posts include Eliezer Yudkowsky's "Five Theses" and Luke Muehlhauser's "Why MIRI?"; this is my attempt to make explicit the claims that are in the background whenever I assert that our mission is of critical importance.

Claim #1: Humans have a very general ability to solve problems and achieve goals across diverse domains.
We call this ability "intelligence," or "general intelligence." This isn't a formal definition — if we knew exactly what general intelligence was, we'd be better able to program it into a computer — but we do think that there's a real phenomenon of general intelligence that we cannot yet replicate in code.
Alternative view: There is no such thing as general intelligence. Instead, humans have a collection of disparate special-purpose modules. Computers will keep getting better at narrowly defined tasks such as chess or driving, but at no point will they acquire "generality" and become significantly more useful, because there is no generality to acquire. (Robin Hanson has argued for versions of this position.)
Short response: I find the "disparate modules" hypothesis implausible in light of how readily humans can gain mastery in domains that are utterly foreign to our ancestors. That's not to say that general intelligence is some irreducible occult property; it presumably comprises a number of different cognitive faculties and the interactions between them. The whole, however, has the effect of making humans much more cognitively versatile and adaptable than (say) chimpanzees.
Why this claim matters: Humans have achieved a dominant position over other species not by being stronger or more agile, but by being more intelligent. If some key part of this general intelligence was able to evolve in the few million years since our common ancestor with chimpanzees lived, this suggests there may exist a relatively short list of key insights that would allow human engineers to build powerful generally intelligent AI systems.
Further reading: Salamon et al., "How Intelligible is Intelligence?"

Claim #2: AI systems could become much more intelligent than humans.
Researchers at MIRI tend to lack strong beliefs about when smarter-than-human machine intelligence will be developed. We do, however, expect that (a) human-equivalent machine intelligence will eventually be developed (likely within a century, barring catastrophe); and (b) machines can become significantly more intelligent than any human.
Alternative view #1: Brains do something special that cannot be replicated on a computer.
Short response: Brains are physical systems, and if certain versions of the Church-Turing thesis hold then computers can in principle replicate the functional input/output behavior of any physical system. Also, note that "intelligence" (as I'm using the term) is about problem-solving capabilities: even if there were some special human feature (such as qualia) that computers couldn't replicate, this would be irrelevant unless it prevented us from designing problem-solving machines.
Alternative view #2: The algorithms at the root of general intelligence are so complex and indecipherable that human beings will not be able to program any such thing for many centuries.
Short response: This seems implausible in light of evolutionary evidence. The genus Homo diverged from other genera only 2.8 million years ago, and the intervening time — a blink in the eye of natural selection — was sufficient for generating the cognitive advantages seen in humans. This strongly implies that whatever sets humans apart from less intelligent species is not extremely complicated: the building blocks of general intelligence must have been present in chimpanzees.
In fact, the relatively intelligent behavior of dolphins suggests that the building blocks were probably there even as far back as the mouse-sized common ancestor of humans and dolphins. One could argue that mouse-level intelligence will take many centuries to replicate, but this is a more difficult claim to swallow, given rapid advances in the field of AI. In light of evolutionary evidence and the last few decades of AI research, it looks to me like intelligence is something we will be able to comprehend and program into machines.
Alternative view #3: Humans are already at or near peak physically possible intelligence. Thus, although we may be able to build human-equivalent intelligent machines, we won't be able to build superintelligent machines.
Short response: It would be surprising if humans were perfectly designed reasoners, for the same reason it would be surprising if airplanes couldn't fly faster than birds. Simple physical calculations bear this intuition out: for example, it seems well possible, within the boundaries of physics, to run a computer simulation of a human brain at thousands of times the normal speed.
Some expect that speed wouldn't matter, because the real bottleneck is waiting for data to come in from physical experiments. This seems unlikely to me. There are many interesting physical experiments that can be sped up, and I have a hard time believing that a team of humans running at a 1000x speedup would fail to outperform their normal-speed counterparts (not least because they could rapidly develop new tools and technology to assist them).
I furthermore expect it's possible to build better reasoners (rather than just faster reasoners) that use computing resources more effectively than humans do, even running at the same speed.
Why this claim matters: Human-designed machines often knock the socks off of biological creatures when it comes to performing tasks we care about: automobiles cannot heal or reproduce, but they sure can carry humans a lot farther and faster than a horse. If we can build intelligent machines specifically designed to solve the world's largest problems through scientific and technological innovation, then they could improve the world at an unprecedented pace. In other words, AI matters.
Further reading: Chalmers, "The Singularity: A Philosophical Analysis"

Claim #3: If we create highly intelligent AI systems, their decisions will shape the future.
Humans use their intelligence to create tools and plans and technology that allow them to shape their environments to their will (and fill them with refrigerators, and cars, and cities). We expect that systems which are even more intelligent would have even more ability to shape their surroundings, and thus, smarter-than-human AI systems could wind up with significantly more control over the future than humans have.
Alternative view: An AI system would never be able to out-compete humanity as a whole, no matter how intelligent it became. Our environment is simply too competitive; machines would have to work with us and integrate into our economy.
Short response: I have no doubt that an autonomous AI system attempting to accomplish simple tasks would initially have strong incentives to integrate with our economy: if you build an AI system that collects stamps for you, it will likely start by acquiring money to purchase stamps. But what if the system accrues a strong technological or strategic advantage?
As an extreme example, we can imagine the system developing nanomachines and using them to convert as much matter as it can into stamps; it wouldn't necessarily care whether that matter came from "dirt" or "money" or "people." Selfish actors only have an incentive to participate in the economy when their gains from trade are greater than the net gains they would get by ignoring the economy and just taking the resources for their own.
So the question is whether it will be possible for an AI system to gain a decisive technological or strategic advantage. I see this as the most uncertain claim out of the ones I've listed here. However, I expect that the answer is still a clear "yes."
Historically, conflicts between humans have often ended with the technologically superior group dominating its rival. At present, there are a number of technological and social innovations that seem possible but have not yet been developed. Humans coordinate slowly and poorly, compared to what distributed software systems could achieve. All of this suggests that if we build a machine that does science faster or better than we can, it could quickly gain a technological and/or strategic advantage over humanity for itself or for its operators. This is particularly true if its intellectual advantage allows it to socially manipulate humans, acquire new hardware (legally or otherwise), produce better hardware, create copies of itself, or improve its own software. For good or ill, much of the future is likely to be determined by superintelligent decision-making machines.
Why this claim matters: Because the future matters. If we want things to be better in the future (or at least not get worse), then it is prudent to prioritize research into the processes that will have high leverage over the future.
Further reading: Armstrong, Smarter Than Us

Claim #4: Highly intelligent AI systems won't be beneficial by default.
We'd like to see the smarter-than-human AI systems of the future working together with humanity to build a better future; but that won't happen by default. In order to build AI systems that have a beneficial impact, we have to solve a number of technical challenges over and above building more powerful and general AI systems.
Alternative view: As humans have become smarter, we've also become more peaceful and tolerant. As AI becomes smarter, it will likewise be able to better figure out our values, and will better execute on them.
Short response: Sufficiently intelligent artificial reasoners would be able to figure out our intentions and preferences; but this does not imply that they would execute plans that are in accordance with them.
A self-modifying AI system could inspect its code and decide whether to continue pursuing the goals it was given or whether it would rather change them. But how is the program deciding which modification to execute?
The AI system is a physical system, and somewhere inside it, it's constructing predictions about how the universe would look if it did various things. Some other part of the system is comparing those outcomes and then executing actions that lead towards outcomes that the current system ranks highly. If the agent is initially programmed to execute plans that lead towards a universe in which it predicts that cancer is cured, then it will only modify its goal if it predicts that this will lead to a cure for cancer.
Regardless of their intelligence level, and regardless of your intentions, computers do exactly what you programmed them to do. If you program an extremely intelligent machine to execute plans that it predicts lead to futures where cancer is cured, then it may be that the shortest path it can find to a cancer-free future entails kidnapping humans for experimentation (and resisting your attempts to alter it, as those would slow it down).
There isn't any spark of compassion that automatically imbues computers with respect for other sentients once they crosses a certain capability threshold. If you want compassion, you have to program it in.
Why this claim matters: A lot of the world's largest problems would be much easier to solve with superintelligent assistance — but attaining those benefits requires that we do more than just improve the capabilities of AI systems. You only get a system that does what you intended if you know how to program it to take your intentions into account, and execute plans that fulfill them.
Further reading: Bostrom, "The Superintelligent Will"

These four claims form the core of the argument that artificial intelligence is important: there is such a thing as general reasoning ability; if we build general reasoners, they could be far smarter than humans; if they are far smarter than humans, they could have an immense impact; and that impact will not be beneficial by default.
At present, billions of dollars and thousands of person-years are pouring into AI capabilities research, with comparatively little effort going into AI safety research. Artificial superintelligence may arise sometime in the next few decades, and will almost surely be created in one form or another over the next century or two, barring catastrophe. Superintelligent systems will either have an extremely positive impact on humanity, or an extremely negative one; it is up to us to decide which.

The post Four Background Claims appeared first on Machine Intelligence Research Institute.

Four Background Claims


Why Now Matters

I'm often asked whether donations now are more important than donations later. Allow me to deliver an emphatic yes: I currently expect that donations to MIRI today are worth much more than donations to MIRI in five years. As things stand, I would very likely take $10M today over $20M in five years.
That's a bold statement, and there are a few different reasons for this. First and foremost, there is a decent chance that some very big funders will start entering the AI alignment field over the course of the next five years. It looks like the NSF may start to fund AI safety research, and Stuart Russell has already received some money from DARPA to work on value alignment. It's quite possible that in a few years' time significant public funding will be flowing into this field.
(It's also quite possible that it won't, or that the funding will go to all the wrong places, as was the case with funding for nanotechnology. But if I had to bet, I would bet that it's going to be much easier to find funding for AI alignment research in five years' time).
In other words, the funding bottleneck is loosening — but it isn't loose yet.
We don't presently have the funding to grow as fast as we could over the coming months, or to run all the important research programs we have planned. At our current funding level, the research team can grow at a steady pace — but we could get much more done over the course of the next few years if we had the money to grow as fast as is healthy.
Which brings me to the second reason why funding now is probably much more important than funding later: because growth now is much more valuable than growth later.
There's an idea picking up traction in the field of AI: instead of focusing only on increasing the capabilities of intelligent systems, it is important to also ensure that we know how to build beneficial intelligent systems. Support is growing for a new paradigm within AI that seriously considers the long-term effects of research programs, rather than just the immediate effects. Years down the line, these ideas may seem obvious, and the AI community's response to these challenges may be in full swing. Right now, however, there is relatively little consensus on how to approach these issues — which leaves room for researchers today to help determine the field's future direction.
People at MIRI have been thinking about these problems for a long time, and that puts us in an unusually good position to influence the field of AI and ensure that some of the growing concern is directed towards long-term issues in addition to shorter-term ones. We can, for example, help avert a scenario where all the attention and interest generated by Musk, Bostrom, and others gets channeled into short-term projects (e.g., making drones and driverless cars safer) without any consideration for long-term risks that are less well-understood.
It's likely that MIRI will scale up substantially at some point; but if that process begins in 2018 rather than 2015, it is plausible that we will have already missed out on a number of big opportunities.
The alignment research program within AI is just now getting started in earnest, and it may even be funding-saturated in a few years' time. But it's nowhere near funding-saturated today, and waiting five or ten years to begin seriously ramping up our growth would likely give us far fewer opportunities to shape the methodology and research agenda within this new AI paradigm. The projects MIRI takes on today can make a big difference years down the line, and supporting us today will drastically affect how much we can do quickly. Now matters.

The post Why Now Matters appeared first on Machine Intelligence Research Institute.

Why Now Matters


Targets 1 and 2: Growing MIRI

Momentum is picking up in the domain of AI safety engineering. MIRI needs to grow fast if it's going to remain at the forefront of this new paradigm in AI research. To that end, we're kicking off our 2015 Summer Fundraiser!
Rather than naming a single funding target, we've decided to lay out the activities we could pursue at different funding levels and let you, our donors, decide how quickly we can grow. In this post, I'll describe what happens if we hit our first two fundraising targets: $250,000 ("continued growth") and $500,000 ("accelerated growth").

Continued growth
Over the past twelve months, MIRI's research team has had a busy schedule — running workshops, attending conferences, visiting industry AI teams, collaborating with outside researchers, and recruiting. We've presented papers at two AAAI conferences and an AGI conference, attended FLI's Puerto Rico conference, written four papers that have been accepted for publication later this year, produced around ten technical reports, and posted a number of new preliminary results to the Intelligent Agent Foundations Forum.
That's what we've been able to do with a three-person research team. What could MIRI's researchers accomplish with a team two or three times as large?

MIRI research fellows Eliezer Yudkowsky and Benja Fallenstein discuss
reflective reasoning at one of our July workshops.
Thanks to a few game-changing donations and grants from the last few years, we have the funds to hire several new researchers: Jessica Taylor will be joining our full-time research team on August 1, and we'll be taking on another new research fellow on September 1. That will bring our research team up to five, and leave us with about one year of runway to continue all of our current activities.
To grow even further — especially if we grow as quickly as we'd like to — we'll need more funding.
Our activities this summer have been aimed at further expanding our research team. The MIRI Summer Fellows program, designed in collaboration with the Center for Applied Rationality to teach mathematicians and AI researchers some AI alignment research skills, is currently underway; and in three weeks, we'll be running the fifth of six summer workshops aimed at bringing our research agenda to a wider audience.
Hitting our first funding target, $250,000, would allow us to hire the very best candidates from our summer workshops and fellows program. That money would allow us to hire 1–3 additional researchers without shortening our runway, bringing the total size of the research team up to 6–8.
Accelerated growth
Growing to a team of eight, though, still isn't nearing the limit of how fast we could be growing this year, if we really hit the accelerator. This is a critical juncture for the field of AI alignment, and we have a rare opportunity to help set the agenda for the coming decades of AI safety work. Our donors tend to be pretty ambitious, so it's worth asking:
How fast could we grow if MIRI had its largest fundraiser ever?
If we hit our $250,000 funding target before the end of August, it will be time to start thinking big. Our second funding target, $500,000, would allow us to recruit even more aggressively, and expand to a roughly ten-person core research team. This is about as large as we think we can grow in the near term without a major increase in our recruitment efforts.
Ten researchers would massively enhance our ability to polish our new results into peer-reviewed publications, attend conferences, and run workshops. But, most crucially of all, it would give us vastly more time to devote to basic research. We have identified dozens of open problems that appear relevant to the task of aligning smarter-than-human AI with the right values; with a larger team, our hope is to start solving those problems quickly.
But growing the core research team as fast as is sustainable in the short term is not all we could be doing: in fact, there are a number of other projects waiting on the sidelines.
Because researchers at MIRI have been thinking about the AI alignment problem for a long time, we have quite a few projects which we intend to work on as soon as we have the necessary time and funding. At the $500,000 level, we can start executing on several of the highest-priority projects without diverting attention away from our core technical agenda.
Project: Type theory in type theory. There are a number of tools missing in modern-day theorem-provers that would be necessary in order to study certain types of self-referential reasoning.1 Given the funding, we would hire one or two type theorists to work on developing relevant tools full-time.
Our type theory project would enable us to study programs reasoning about similar programs in a concrete setting — lines of code, as opposed to equations on whiteboards. The tools developed could also be quite useful to the type theory community, as they would have a number of short-term applications. (E.g., for building modular and highly secure operating systems where a verifier component can verify that it is OK to swap out the verifier for a more powerful verifier.) We would benefit from the input from specialists in type theory, we would benefit from the tools, and we would also benefit from increased engagement with the larger type theory community, which is an important community when it comes to writing highly reliable computer programs.
Project: Visiting scholar program. Another project we can quickly implement given sufficient funding is a visiting scholar program. One surefire way to increase our engagement with the academic community would be to have interested professors drop by for the summer, while we pay their summer salaries and work with them on projects where our interests overlap. These sorts of visits, combined with a few visiting graduate students or postdocs each summer, would likely go a long way towards getting academic communities more involved with the open problems that we think are most urgent, while also giving us the opportunity to get input from some of the top minds in the modern field of AI.
A program such as this could be extended into one where MIRI directly supports graduate students and/or professors who are willing to spend some portion of their time on AI alignment research. This program could also prove useful if MIRI reaches the point of forming new research groups specializing in approaches to the AI alignment challenge other than the one described in our technical agenda. Those extensions of the program would require significantly more time and funding, but at the $500,000 level we could start sowing the seeds.
MIRI's future shape
That's only a brief look at the new activities we could take on with appropriate funding. More detailed exposition is forthcoming, both about the state of our existing projects and about the types of projects we are prepared to execute on given the opportunity. Stay tuned for future posts!
If these plans have already gotten you excited, then you can help make them a reality by contributing to our summer fundraiser. There is much to be done in the field of AI alignment, and both MIRI and the wider field of AI will be better off if we can make real progress as early as possible. Future posts will discuss not only the work we want to branch out into, but also reasons why we think that this is an especially critical period in the history of the field, and why we expect that giving now will be much more valuable than waiting to give later.
Until then, thank you again for all of your support — our passionate donor base is what has enabled us to grow as much as we have over the last few years, and we feel privileged to have this chance to realize the potential our supporters have seen in MIRI.

Donate Now

Roughly speaking, in order to study self-reference in a modern dependently typed programming language, we need to be able to write programs that can manipulate and typecheck quoted programs written in the same language. This would be a ripe initial setting in which to study programs that reason about themselves with high confidence. However, to our knowledge, nobody has yet written (e.g.) the type of Agda programs in Agda — though there have been some promising first steps.The post Targets 1 and 2: Growing MIRI appeared first on Machine Intelligence Research Institute.

Targets 1 and 2: Growing MIRI


MIRI's 2015 Summer Fundraiser!

This last year has been pretty astounding. Since its release twelve months ago, Nick Bostrom's book Superintelligence has raised awareness about the challenge that MIRI exists to address: long-term risks posed by smarter-than-human artificially intelligent systems. Academic and industry leaders echoed these concerns in an open letter advocating "research aimed at ensuring that increasingly capable AI systems are robust and beneficial." To jump-start this new safety-focused paradigm in AI, the Future of Life Institute has begun distributing $10M as grants to dozens of research groups, Bostrom and MIRI among them.
MIRI comes to this budding conversation with a host of relevant open problems already in hand. Indeed, a significant portion of the research priorities document accompanying the open letter is drawn from our work on this topic. Having already investigated these issues at some length, MIRI is well-positioned to shape this field as it enters a new phase in its development.
This is a big opportunity. MIRI is already growing and scaling its research activities, but the speed at which we scale in the coming months and years can be increased by more funding. For that reason, MIRI is starting a six-week fundraiser aimed at increasing our rate of growth.
And here it is!

—  Progress Bar  —

Donate Now

Rather than running a matching fundraiser with a single fixed donation target, we'll be letting you help choose MIRI's course, based on the details of our funding situation and how we would make use of marginal dollars. In particular, we'll be blogging over the coming weeks about how our plans would scale up at different funding levels:

Target 1 — $250k: Continued growth. At this level, we would have enough funds to maintain a twelve-month runway while continuing all current operations, including running workshops, writing papers, and attending conferences. We will also be able to scale the research team up by one to three additional researchers, on top of our three current researchers and two new researchers who are starting this summer. This would ensure that we have the funding to hire the most promising researchers who come out of the MIRI Summer Fellows Program and our summer workshop series.

Target 2 — $500k: Accelerated growth. At this funding level, we could grow our team more aggressively, while maintaining a twelve-month runway. We would have the funds to expand the research team to about ten core researchers, while also taking on a number of exciting side-projects, such as hiring one or two type theorists. Recruiting specialists in type theory, a field at the intersection of computer science and mathematics, would enable us to develop tools and code that we think are important for studying verification and reflection in artificial reasoners.

Target 3 — $1.5M: Taking MIRI to the next level. At this funding level, we would start reaching beyond the small but dedicated community of mathematicians and computer scientists who are already interested in MIRI's work. We'd hire a research steward to spend significant time recruiting top mathematicians from around the world, we'd make our job offerings more competitive, and we'd focus on hiring highly qualified specialists in relevant areas of mathematics. This would allow us to grow the research team as fast as is sustainable, while maintaining a twelve-month runway.

Target 4 — $3M: Bolstering our fundamentals. At this level of funding, we'd start shoring up our basic operations. We'd spend resources and experiment to figure out how to build the most effective research team we can. We'd branch out into additional high-value projects outside the scope of our core research program, such as hosting specialized conferences and retreats, upgrading our equipment and online resources, and running programming tournaments to spread interest about certain open problems. At this level of funding we'd also start extending our runway, and prepare for sustained aggressive growth over the coming years.

Target 5 — $6M: A new MIRI. At this point, MIRI would become a qualitatively different organization. With this level of funding, we would start forking the research team into multiple groups attacking the AI alignment problem from very different angles. Our current technical agenda is not the only way to approach the challenges that lie ahead — indeed, there are a number of research teams that we would be thrilled to start up inside MIRI given the opportunity.

We also have plans that extend beyond the $6M level: for more information, shoot me an email at <EMAIL>. I also invite you to email me with general questions or to set up a time to chat.
If you intend to make use of corporate matching (check here to see whether your employer will match your donation), email <EMAIL> and we'll include the matching contributions in the fundraiser total.
Some of these targets are quite ambitious, and I'm excited to see what happens when we lay out the available possibilities and let our donors collectively decide how quickly we develop as an organization.
We'll be using this fundraiser as an opportunity to explain our research and our plans for the future. If you have any questions about what MIRI does and why, email them to <EMAIL>. Answers will be posted to this blog every Monday and Friday.
Below is a list of explanatory posts written for this fundraiser, which we'll be updating regularly:

July 1 — Grants and Fundraisers. Why we've decided to experiment with a multi-target fundraiser.
July 16 — An Astounding Year. Recent successes for MIRI, and for the larger field of AI safety.
July 18 — Targets 1 and 2: Growing MIRI. MIRI's plans if we hit the $250k or $500k funding target.
July 20 — Why Now Matters. Two reasons to give now, rather than wait to give later.
July 24 — Four Background Claims. Basic assumptions behind MIRI's focus on smarter-than-human AI.
July 27 — MIRI's Approach. How we identify technical problems to work on.
July 31 — MIRI FAQ. Summarizing common sources of misunderstanding.
August 3 — When AI Accelerates AI. Some reasons to get started on safety work early.
August 7 — Target 3: Taking It To The Next Level. Our plans if we hit the $1.5M funding target.
August 10 — Assessing Our Past And Potential Impact. Why expect MIRI in particular to make a difference?
August 14 — What Sets MIRI Apart? Distinguishing MIRI from groups in academia and industry.
August 18 — Powerful Planners, Not Sentient Software. Why advanced AI isn't "evil robots."
August 28 — AI and Effective Altruism. On MIRI's role in the EA community.

Our hope is that these new resources will help you, our donors, make more informed decisions during our fundraiser, and also that our fundraiser will serve as an opportunity for people to learn a lot more about our activities and strategic outlook.
As scientists, engineers, and policymakers begin to take notice of the AI alignment problem, MIRI is in a unique position to direct this energy and attention in a useful direction. Donating today will help us rise to this challenge and secure a place at the forefront of this critical field.

Donate Now

The post MIRI's 2015 Summer Fundraiser! appeared first on Machine Intelligence Research Institute.

MIRI’s 2015 Summer Fundraiser!


An Astounding Year

It's safe to say that this past year exceeded a lot of people's expectations.
Twelve months ago, Nick Bostrom's Superintelligence had just been published. Long-term questions about smarter-than-human AI systems were simply not a part of mainstream discussions about the social impact of AI, and fewer than five people were working on the AI alignment challenge full-time.
Twelve months later, we live in a world where Elon Musk, Bill Gates, and Sam Altman readily cite Superintelligence as a guide to the questions we should be asking about AI's future as a field. For Gates, the researchers who aren't concerned about advanced AI systems are the ones who now need to explain their views:
I am in the camp that is concerned about super intelligence. First the machines will do a lot of jobs for us and not be super intelligent. That should be positive if we manage it well. A few decades after that though the intelligence is strong enough to be a concern. I agree with Elon Musk and some others on this and don't understand why some people are not concerned.
As far as I can tell, the turning point occurred in January 2015, when Max Tegmark and the newly-formed Future of Life Institute organized a "Future of AI" conference in San Juan, Puerto Rico to bring together top AI academics, top research groups from industry, and representatives of the organizations studying long-term AI risk.
The atmosphere at the Puerto Rico conference was electric. I stepped off the plane expecting to field objections to the notion that superintelligent machines pose a serious risk. Instead, I was met with a rapidly-formed consensus that many challenges lie ahead, and a shared desire to work together to develop a response.

Attendees of the Puerto Rico conference included, among others, Stuart Russell (co-author of the
leading textbook in AI), Thomas Dietterich (President of AAAI), Francesca Rossi (President of IJCAI),
Bart Selman, Tom Mitchell, Murray Shanahan, Vernor Vinge, Elon Musk, and representatives from
Google DeepMind, Vicarious, FHI, CSER, and MIRI.
This consensus resulted in a widely endorsed open letter, and an accompanying research priorities document that cites MIRI's past work extensively. Impressed by the speed with which AI researchers were pivoting toward investigating the alignment problem, Elon Musk donated $10M to a grants program aimed at jump-starting this new paradigm in AI research.
Since then, the pace has been picking up. Nick Bostrom received $1.5M of the Elon Musk donation to start a new Strategic Research Center for Artificial Intelligence, which will focus on the geopolitical challenges posed by powerful AI. MIRI has received $300,000 in FLI grants directly to continue its technical and strategic research programs, and participated in a few other collaborative grants. The Cambridge Centre for the Study of Existential Risk has received a number of large grants that have allowed it to begin hiring. Stuart Russell and I recently visited Washington, D.C. to participate in a panel at a leading public policy think tank. We are currently in talks with the NSF about possibilities for extending their funding program to cover some of the concerns raised by the open letter.
The field of AI, too, is taking notice. AAAI, the leading scientific society in AI, hosted its first workshop on safety and ethics (I gave a presentation there), and the two major machine learning conferences — IJCAI and NIPS — will, for the first time, have sessions or workshops dedicated to the discussion of AI safety research.
Years down the line, I expect that some will look back on the Puerto Rico conference as the birthplace of the field of AI alignment. From the outside, 2015 will likely look like the year that AI researchers started seriously considering the massive hurdles that stand between us and the benefits that artificially intelligent systems could bring.
Our long-time backers, however, have seen the work that went into making these last few months possible. It's thanks to your longstanding support that existential risk mitigation efforts have reached this tipping point. A sizable amount of our current momentum can plausibly be traced back, by one path or another, to exchanges at early summits or on blogs, and to a number of early research and outreach efforts. Thank you for beginning a conversation about these issues long before they began to filter into the mainstream, and thank you for helping us get to where we are now.
Progress at MIRI
Meanwhile at MIRI, the year has been a busy one.
In the wake of the Puerto Rico conference, we've been building relationships and continuing our conversations with many different industry groups, including DeepMind, Vicarious, and the newly formed Good AI team. We've been thrilled to engage more with the academic community, via a number of collaborative papers that are in the works, two collaborative grants through the FLI grant program, and conversations with various academics about the content of our research program. During the last few weeks, Stuart Russell and Bart Selman have both come on as official MIRI research advisors.
We've also been hard at work on the research side. In March, we hired Patrick LaVictoire as a research fellow. We've attended a number of conferences, including AAAI's safety and ethics workshop. We had a great time co-organizing a productive decision theory conference at Cambridge University, where I had the pleasure of introducing our unique take on decision theory (inspired by our need for runnable programs) to a number of academic decision theorists who I both respect and admire — and I'm happy to say that our ideas were very well received.
We've produced a number of new resources and results in recent months, including:

a series of overview papers describing our technical agenda written in preparation for the Puerto Rico conference;
a number of tools that are useful for studying many of these open problems, available at our github repository;
a theory of reflective oracle machines (in collaboration with Paul Christiano at U.C. Berkeley), which are a promising step towards both better models of logical uncertainty and better models of agents that reason about other agents that are as powerful (or more powerful) than they are; and
a technique for implementing reflection in the HOL theorem-prover (in collaboration with Ramana Kumar at Cambridge University): code here.

We have also launched the Intelligent Agent Foundations Forum to provide a location for publishing and discussing partial results with the broader community working on these problems.
That's not all, though. After the Puerto Rico conference, we anticipated the momentum that it would create, and we started gearing up for growth. We set up a series of six summer workshops to introduce interested researchers to open problems in AI alignment, and we worked with the Center for Applied Rationality to create a MIRI summer fellows program aimed at helping computer scientists and mathematicians effectively contribute to AI alignment research. We're now one week into the summer fellows program, and we've run four of our six summer workshops.
Our goal with these projects is to loosen our talent bottleneck and find more people who can do MIRI-style AI alignment research, and that has been paying off. Two new researchers have already signed on to start at MIRI in the late summer, and it is likely that we will get a few new hires out of the summer fellows program and the summer workshops as well.
Next steps
We now find ourselves in a wonderful position. The projects listed above have been a lot for a small research team of three, and there's much more that we hope to take on as we grow the research team further. Where many other groups are just starting to think about how to approach the challenges of AI alignment, MIRI already has a host of ideas that we're ready to execute on, as soon as we get the personpower and the funding.
The question now is: how quickly can we grow? We already have the funding to sign on an additional researcher (or possibly two) while retaining a twelve-month runway, and it looks like we could grow much faster than that given sufficient funding.
Tomorrow, we're officially kicking off our summer fundraiser (though you're welcome to give now at our Donation page). Upcoming posts will describe in more detail what we could do with more funding, but for now I wanted to make it clear why we're so excited about the state of AI alignment research, and why we think this is a critical moment in the history of the field of AI.
Here's hoping our next year is half as exciting as this last year was! Thank you again — and stay tuned for our announcement tomorrow.
The post An Astounding Year appeared first on Machine Intelligence Research Institute.

An Astounding Year


July 2015 Newsletter

Hello, all! I'm Rob Bensinger, MIRI's Outreach Coordinator. I'll be keeping you updated on MIRI's activities and on relevant news items. If you have feedback or questions, you can get in touch with me by email.

Research updates

A new paper: "The Asilomar Conference: A Case Study in Risk Mitigation."
New at AI Impacts: Update on All the AI Predictions; Predictions of Human-Level AI Timelines; Accuracy of AI Predictions
New at IAFF: A Simple Model of the Löbstacle; Structural Risk Minimization
We ran two introductory workshops this month, on decision theory and Vingean reflection.

General updates

Our team is growing! If you are interested in joining us, click through to see our Office Manager job posting.
This was Nate Soares' first month as our Executive Director. We have big plans in store for the next two months, which Nate has begun laying out here: fundraising thoughts.
MIRI has been awarded a $250,000 grant from the Future of Life Institute spanning three years to make headway on our research agenda. This will fund three workshops and several researcher-years of work on a number of open technical problems. MIRI has also been awarded a $49,310 FLI grant to fund strategy research at AI Impacts.
Owain Evans of the Future of Humanity Institute, in collaboration with new MIRI hire Jessica Taylor, has been awarded a $227,212 FLI grant to develop algorithms that learn human preferences from behavioral data in the presence of irrational and otherwise suboptimal behavior.
Cambridge computational logician Ramana Kumar and MIRI research fellow Benja Fallenstein have been awarded a $36,750 FLI grant to study self-referential reasoning in HOL (higher-order logic) proof assistants.
Stuart Russell, co-author of the standard textbook on artificial intelligence, has become a MIRI research advisor.
Nate and Stuart Russell participated in a panel discussion about AI risk at the Information Technology and Innovation Foundation, one of the world's leading public policy think tanks: video.
On the Effective Altruism Forum, Nate answered a large number of questions about MIRI's strategy and priorities. Excerpts here.

News and links

Nick Bostrom has won the $1,500,000 FLI center grant to set up a new Cambridge-Oxford institute, the Strategic Research Center for Artificial Intelligence. The center will "develop policies to be enacted by governments, industry leaders, and others in order to minimize risks and maximize benefit from artificial intelligence development in the longer term."
In a Cambridge speech, Stuart Russell discusses accelerating progress in AI and calls for a revised conception of the field's goal.
MIRI research advisor Roman Yampolskiy's new book Artificial Superintelligence: A Futuristic Approach comes out next week.
Luke Muehlhauser reviews claims made in Wait But Why's popular blog series on superintelligence.

The post July 2015 Newsletter appeared first on Machine Intelligence Research Institute.

July 2015 Newsletter


Grants and fundraisers

Two big announcements today:
1. MIRI has won $299,310 from the Future of Life Institute's grant program to jumpstart the field of long-term AI safety research.

$250,000 will go to our research program over the course of three years. This will go towards running workshops and funding a few person-years of research on the open problems discussed in our technical agenda.
$49,310 will go towards AI Impacts, a project which aims to shed light on the implications of advanced artificial intelligence using empirical data and rigorous analysis.

MIRI will also collaborate with the primary investigators on two other large FLI grants:

$227,212 has been awarded to Owain Evans at the Future of Humanity Institute to develop algorithms that learn human preferences from data despite human irrationalities. This will be carried out in collaboration with Jessica Taylor, who will become a MIRI research fellow at the end of this summer.
$36,750 has been awarded to Ramana Kumar at Cambridge University to study self-reference in the HOL theorem prover. This will be done in collaboration with MIRI research fellow Benja Fallenstein.

The money comes from Elon Musk's extraordinary donation of $10M to fund FLI's first-of-its-kind grant competition for research aimed at keeping AI technologies beneficial as capabilities improve.
This funding, coming on the heels of the payments from our sale of the Singularity Summit (which recently concluded) and an extremely generous surprise donation from Jed McCaleb at the end of 2013, means we can continue to ramp up our research efforts. That doesn't mean our job is done, of course. In January, shortly after the FLI conference, we came to the conclusion that the funding situation for our field was set to improve, and decided to start gearing up for growth. That prediction has turned out to be correct, which puts us in an excellent position.
We're now, indeed, set to grow—the only question is, "How quickly?" Which brings me to announcement number two.
2. Our summer fundraiser is starting in mid-July, and we're going to try something new.
Every summer for the past few years, MIRI has run a matching fundraiser, where we get some of our biggest donors to pledge their donations conditional upon your support. Conventional wisdom states that matching fundraisers make it easier to raise funds, and MIRI has had a lot of success with them in the past. They seem to be an excellent way to get donors excited, and the deadline helps create a sense of urgency.
However, a few different people, including the folks over at GiveWell and effective altruism writer Ben Kuhn, have voiced skepticism about the effectiveness of matching fundraisers. Most of our large donors are happy to donate regardless of whether we raise matching funds, and matching fundraisers tend to put the focus on interactions between small and large donors, rather than on the exciting projects that we could be running with sufficient funding.
Our experience with our donors has been that they are exceptionally thoughtful, and that they have thought themselves about how (and how quickly) they want MIRI to grow. So this fundraiser we'd like to give you more resources to make an informed decision about where to send your money, with better knowledge about how different levels of funding will affect our operations.
Details are forthcoming mid-July, along with a whole lot more information about what we've been up to and what we have planned.
As always, thanks for everything: it's exciting to receive one of the very first grants in this burgeoning field, and we haven't forgotten that it's only thanks to your support that the field has made it this far in the first place.
The post Grants and fundraisers appeared first on Machine Intelligence Research Institute.

Grants and fundraisers


Wanted: Office Manager (aka Force Multiplier)

We're looking for a full-time office manager to support our growing team. It's a big job that requires organization, initiative, technical chops, and superlative communication skills. You'll develop, improve, and manage the processes and systems that make us a super-effective organization. You'll obsess over our processes (faster! easier!) and our systems (simplify! simplify!). Essentially, it's your job to ensure that everyone at MIRI, including you, is able to focus on their work and Get Sh*t Done.
That's a super-brief intro to what you'll be working on. But first, you need to know if you'll even like working here.

A Bit About Us
We're a research nonprofit working on the critically important problem of superintelligence alignment: how to bring smarter-than-human artificial intelligence into alignment with human values.1 Superintelligence alignment is a burgeoning field, and arguably the most important and under-funded research problem in the world. Experts largely agree that AI is likely to exceed human levels of capability on most cognitive tasks in this century—but it's not clear when, and we aren't doing a very good job of preparing for the possibility. Given how disruptive smarter-than-human AI would be, we need to start thinking now about AI's global impact. Over the past year, a number of leaders in science and industry have voiced their support for prioritizing this endeavor:

Stuart Russell, co-author of the leading textbook on artificial intelligence and a MIRI advisor, gives a compelling argument for doing this work sooner rather than later.
Nick Bostrom of Oxford University, another MIRI research advisor, published Superintelligence: Paths, Dangers, Strategies, which details the potential value of smarter-than-human AI systems as well as the potential hazards.
Elon Musk (Paypal, SpaceX, Tesla), Bill Gates (Microsoft co-founder), Stephen Hawking (world-renowned theoretical physicist), and others have publicly stated their concerns about long-term AI risk.
Hundreds of AI researchers and engineers recently signed an open letter advocating for more research into robust and beneficial artificial intelligence. A number of MIRI publications are cited in the corresponding Research Priorities document.

People are starting to discuss these issues in a more serious way, and MIRI is well-positioned to be a thought leader in this important space. As interest in AI safety grows, we're growing too—we've gone from a single full-time researcher in 2013 to what will likely be a half-dozen research fellows by the end of 2015, and intend to continue growing in 2016.
All of which is to say: we really need an office manager who will support our efforts to hack away at the problem of superintelligence alignment!
If our overall mission seems important to you, and you love running well-oiled machines, you'll probably fit right in. If that's the case, we can't wait to hear from you.
What It's Like to Work at MIRI
We try really hard to make working at MIRI an amazing experience. We have a team full of truly exceptional people—the kind you'll be excited to work with. Here's how we operate:
Flexible Hours
We do not have strict office hours. Simply ensure you're here enough to be available to the team when needed, and to fulfill all of your duties and responsibilities.
Modern Work Spaces
Many of us have adjustable standing desks with multiple large external monitors. We consider workspace ergonomics important, and try to rig up work stations to be as comfortable as possible.
Living in the Bay Area
We're located in downtown Berkeley, California. Berkeley's monthly average temperature ranges from 60°F in the winter to 75°F in the summer. From our office you're:

A 10-second walk to the roof of our building, from which you can view the Berkeley Hills, the Golden Gate Bridge, and San Francisco.
A 30-second walk to the BART (Bay Area Rapid Transit), which can get you around the Bay Area.
A 3-minute walk to UC Berkeley Campus.
A 5-minute walk to dozens of restaurants (including ones in Berkeley's well-known Gourmet Ghetto).
A 30-minute BART ride to downtown San Francisco.
A 30-minute drive to the beautiful west coast.
A 3-hour drive to Yosemite National Park.

Vacation Policy
Our vacation policy is that we don't have a vacation policy. That is, take the vacations you need to be a happy, healthy, productive human. There are checks in place to ensure this policy isn't abused, but we haven't actually run into any problems since initiating the policy.
We consider our work important, and we care about whether it gets done well, not about how many total hours you log each week. We'd much rather you take a day off than extend work tasks just to fill that extra day.
Regular Team Dinners and Hangouts
We get the whole team together every few months, order a bunch of food, and have a great time.
Top-Notch Benefits
We provide top-notch health and dental benefits. We care about our team's health, and we want you to be able to get health care with as little effort and annoyance as possible.
Agile Methodologies
Our ops team follows standard Agile best practices, meeting regularly to plan, as a team, the tasks and priorities over the coming weeks. If the thought of being part of an effective, well-functioning operation gets you really excited, that's a promising sign!
Other Tidbits

Moving to the Bay Area? We'll cover up to $3,500 in moving expenses.
Use public transit to get to work? You get a transit pass with a large monthly allowance.
All the snacks and drinks you could want at the office.
You'll get a smartphone and full plan.
This is a salaried position. (That is, your job is not to sit at a desk for 40 hours a week. Your job is to get your important work done, even if this occasionally means working on a weekend or after hours.)

It can also be surprisingly motivating to realize that your day job is helping people explore the frontiers of human understanding, mitigate global catastrophic risk, etc., etc. At MIRI, we try to tackle the very largest problems facing humanity, and that can be a pretty satisfying feeling.
If this sounds like your ideal work environment, read on! It's time to talk about your role.
What an Office Manager Does and Why it Matters
Our ops team and researchers (and collection of remote contractors) are swamped making progress on the huge task we've taken on as an organization.
That's where you come in. An office manager is the oil that keeps the engine running. They're indispensable. Office managers are force multipliers: a good one doesn't merely improve their own effectiveness—they make the entire organization better.
We need you to build, oversee, and improve all the "behind-the-scenes" things that ensure MIRI runs smoothly and effortlessly. You will devote your full attention to looking at the big picture and the small details and making sense of it all. You'll turn all of that into actionable information and tools that make the whole team better. That's the job.
Sometimes this looks like researching and testing out new and exciting services. Other times this looks like stocking the fridge with drinks, sorting through piles of mail, lugging bags of groceries, or spending time on the phone on hold with our internet provider. But don't think that the more tedious tasks are low-value. If the hard tasks don't get done, none of MIRI's work is possible. Moreover, you're actively encouraged to find creative ways to make the boring stuff more efficient—making an awesome spreadsheet, writing a script, training a contractor to take on the task—so that you can spend more time on what you find most exciting.
We're small, but we're growing, and this is an opportunity for you to grow too. There's room for advancement at MIRI (if that interests you), based on your interests and performance.
Sample Tasks
You'll have a wide variety of responsibilities, including, but not necessarily limited to, the following:

Orienting and training new staff.
Onboarding and offboarding staff and contractors.
Managing employee benefits and services, like transit passes and health care.
Payroll management; handling staff questions.
Championing our internal policies and procedures wiki—keeping everything up to date, keeping everything accessible, and keeping staff aware of relevant information.
Managing various services and accounts (ex. internet, phone, insurance).
Championing our work space, with the goal of making the MIRI office a fantastic place to work.
Running onsite logistics for introductory workshops.
Processing all incoming mail packages.
Researching and implementing better systems and procedures.

Your "value-add" is by taking responsibility for making all of these things happen. Having a competent individual in charge of this diverse set of tasks at MIRI is extremely valuable!
A Day in the Life
A typical day in the life of MIRI's office manager may look something like this:

Come in.
Process email inbox.
Process any incoming mail, scanning/shredding/dealing-with as needed.
Stock the fridge, review any low-stocked items, and place an order online for whatever's missing.
Onboard a new contractor.
Spend some time thinking of a faster/easier way to onboard contractors. Implement any hacks you come up with.
Follow up with Employee X about their benefits question.
Outsource some small tasks to TaskRabbit or Upwork. Follow up with previously outsourced tasks.
Notice that you've spent a few hours per week the last few weeks doing xyz. Spend some time figuring out whether you can eliminate the task completely, automate it in some way, outsource it to a service, or otherwise simplify the process.
Review the latest post drafts on the wiki. Polish drafts as needed and move them to the appropriate location.
Process email.
Go home.

You're the One We're Looking For If:

You are authorized to work in the US. (Prospects for obtaining an employment-based visa for this type of position are slim; sorry!)
You can solve problems for yourself in new domains; you find that you don't generally need to be told what to do.
You love organizing information. (There's a lot of it, and it needs to be super-accessible.)
Your life is organized and structured.
You enjoy trying things you haven't done before. (How else will you learn which things work?)
You're way more excited at the thought of being the jack-of-all-trades than at the thought of being the specialist.
You are good with people—good at talking about things that are going great, as well as things that aren't.
People thank you when you deliver difficult news. You're that good.
You can notice all the subtle and wondrous ways processes can be automated, simplified, streamlined… while still keeping the fridge stocked in the meantime.
You know your way around a computer really well.
You enjoy eliminating unnecessary work, automating automatable work, outsourcing outsourcable work, and executing on everything else.
You want to do what it takes to help all other MIRI employees focus on their jobs.
You're the sort of person who sees the world, organizations, and teams as systems that can be observed, understood, and optimized.
You think Sam is the real hero in Lord of the Rings.
You have the strong ability to take real responsibility for an issue or task, and ensure it gets done. (This doesn't mean it has to get done by you; but it has to get done somehow.)
You celebrate excellence and relentlessly pursue improvement.
You lead by example.

Bonus Points:

Your technical chops are really strong. (Dabbled in scripting? HTML/CSS? Automator?)
Involvement in the Effective Altruism space.
Involvement in the broader AI-risk space.
Previous experience as an office manager.

Experience & Education Requirements

Let us know about anything that's evidence that you'll fit the bill.

How to Apply
Apply by July 31, 2015. The application deadline has passed. Thanks for your consideration.
P.S. Share the love! If you know someone who might be a perfect fit, we'd really appreciate it if you pass this along!
More details on our About page.The post Wanted: Office Manager (aka Force Multiplier) appeared first on Machine Intelligence Research Institute.


New report: "The Asilomar Conference: A Case Study in Risk Mitigation"

Today we release a new report by Katja Grace, "The Asilomar Conference: A Case Study in Risk Mitigation" (PDF, 67pp).
The 1975 Asilomar Conference on Recombinant DNA is sometimes cited as an example of successful action by scientists who preemptively identified an emerging technology's potential dangers and intervened to mitigate the risk. We conducted this investigation to check whether that basic story is true, and what lessons those events might carry for AI and other unprecedented technological risks.
To prepare this report, Grace consulted several primary and secondary sources, and also conducted four interviews that are cited in the report. The interviews are published here:

David Baltimore on Asilomar
Paul Berg on Asilomar
George Church on Asilomar
Alexander Berger on early responses to future risks

The basic conclusions of this report, which have not been separately vetted, are:

The specific dangers that motivated the Asilomar conference were relatively immediate, rather than long-term. These dangers turned out to be effectively nonexistent. Experts disagree as to whether scientists should have known better with the information they had at the time.
The conference appears to have caused improvements in general lab safety practices.
The conference plausibly averted regulation and helped scientists to be on better terms with the public. Whether these effects are positive for society depends on (e.g.) whether it is better for this category of scientific activities to go unregulated, a question not addressed by this report.

The post New report: "The Asilomar Conference: A Case Study in Risk Mitigation" appeared first on Machine Intelligence Research Institute.

New report: “The Asilomar Conference: A Case Study in Risk Mitigation”


June 2015 Newsletter

Dear friends of MIRI,
As we announced on May 6th, I've decided to take a research position at GiveWell. With unanimous support from the Board, MIRI research fellow Nate Soares will be taking my place as Executive Director starting June 1st. Nate has introduced himself here.
I'm proud of what the MIRI team has accomplished during my tenure as Executive Director, and I'm excited to watch Nate take MIRI to the next level. My enthusiasm for MIRI's work remains as strong as ever, and I look forward to supporting MIRI going forward, both financially and as a close advisor. (See here for further details on my transition to GiveWell.)
Thank you all for your support!
– Luke Muehlhauser

Research updates

Two MIRI papers were accepted to AGI-15.
Slides and videos from MIRI-related talks at a Cambridge University decision theory conference we co-organized.
New at AI Impacts: Similarity Between Historical and Contemporary AI Predictions; Publication biases toward shorter predictions; Selection bias from optimistic experts; Why do AGI researchers expect AI so soon?; A new approach to predicting brain-computer parity; Group Differences in AI Predictions; The Maes-Garreau Law; AI Timeline predictions in surveys and statements; MIRI AI Predictions Dataset; Brain performance in TEPS.

News updates

New MIRIx groups in Berkeley and New York City.

Other updates

Nick Bostrom's TED talk on machine superintelligence.
Effective Altruism Global is this August, in the San Francisco Bay Area (USA), Oxford (UK), and Melbourne (Australia). Keynote speaker is Elon Musk. Apply by June 10th!

The post June 2015 Newsletter appeared first on Machine Intelligence Research Institute.

June 2015 Newsletter


Introductions

Hello, I'm Nate Soares, and I'm pleased to be taking the reins at MIRI on Monday morning.
For those who don't know me, I've been a research fellow at MIRI for a little over a year now. I attended my first MIRI workshop in December of 2013 while I was still working at Google, and was offered a job soon after. Over the last year, I wrote a dozen papers, half as primary author. Six of those papers were written for the MIRI technical agenda, which we compiled in preparation for the Puerto Rico conference put on by FLI in January 2015. Our technical agenda is cited extensively in the research priorities document referenced by the open letter that came out of that conference. In addition to the Puerto Rico conference, I attended five other conferences over the course of the year, and gave a talk at three of them. I also put together the MIRI research guide (a resource for students interested in getting involved with AI alignment research), and of course I spent a fair bit of time doing the actual research at workshops, at researcher retreats, and on my own. It's been a jam-packed year, and it's been loads of fun.
I've always had a natural inclination towards leadership: in the past, I've led a F.I.R.S.T. Robotics team, managed two volunteer theaters, served as president of an Entrepreneur's Club, and co-founded a startup or two. However, this is the first time I've taken a professional leadership role, and I'm grateful that I'll be able to call upon the experience and expertise of the board, of our advisors, and of outgoing executive director Luke Muehlhauser.
MIRI has improved greatly under Luke's guidance these last few years, and I'm honored to have the opportunity to continue that trend. I've spent a lot of time in conversation with Luke over the past few weeks, and he'll remain a close advisor going forward. He and the management team have spent the last year or so really tightening up the day-to-day operations at MIRI, and I'm excited about all the opportunities we have open to us now.
The last year has been pretty incredible. Discussion of long-term AI risks and benefits has finally hit the mainstream, thanks to the success of Bostrom's Superintelligence and FLI's Puerto Rico conference, and due in no small part to years of movement-building and effort made possible by MIRI's supporters. Over the last year, I've forged close connections with our friends at the Future of Humanity Institute, the Future of Life Institute, and the Centre for the Study of Existential Risk, as well as with a number of industry teams and academic groups who are focused on long-term AI research. I'm looking forward to our continued participation in the global conversation about the future of AI. These are exciting times in our field, and MIRI is well-poised to grow and expand. Indeed, one of my top priorities as executive director is to grow the research team.
That project is already well under way. I'm pleased to announce that Jessica Taylor has accepted a full-time position as a MIRI researcher starting in August 2015. We are also hosting a series of summer workshops focused on various technical AI alignment problems, the second of which is just now concluding. Additionally, we are working with the Center for Applied Rationality to put on a summer fellows program designed for people interested in gaining the skills needed for research in the field of AI alignment.
I want to take a moment to extend my heartfelt thanks to all those supporters of MIRI who have brought us to where we are today: We have a slew of opportunities before us, and it's all thanks to your effort and support these past years. MIRI couldn't have made it as far as it has without you. Exciting times are ahead, and your continued support will allow us to grow quickly and pursue all the opportunities that the last year opened up.
Finally, in case you want to get to know me a little better, I'll be answering questions on the effective altruism forum at 3PM Pacific time on Thursday June 11th.
Onwards,
Nate
The post Introductions appeared first on Machine Intelligence Research Institute.

Introductions


Two papers accepted to AGI-15

MIRI has two papers forthcoming in the conference proceedings of AGI-15. The first paper, previously released as a MIRI technical report, is "Reflective variants of Solomonoff induction and AIXI," by Benja Fallenstein, Nate Soares, and Jessica Taylor.
The second paper, "Two Attempts to Formalize Counterpossible Reasoning in Deterministic Settings," by Nate Soares and Benja Fallenstein, is a compressed version of some material from an earlier technical report. This new paper's abstract is:
This paper motivates the study of counterpossibles (logically impossible counterfactuals) as necessary for developing a decision theory suitable for generally intelligent agents embedded within their environments. We discuss two attempts to formalize a decision theory using counterpossibles, one based on graphical models and another based on proof search.
Fallenstein will be attending AGI-15.
The post Two papers accepted to AGI-15 appeared first on Machine Intelligence Research Institute.

Two papers accepted to AGI-15


MIRI-related talks from the decision theory conference at Cambridge University

Recently, MIRI co-organized a conference at Cambridge University titled Self-prediction in decision theory and artificial intelligence. At least six of the conference's talks directly discussed issues raised in MIRI's technical agenda:

MIRI research fellow (and soon, Executive Director) Nate Soares gave a talk titled "What is a what if?" (.pdf w/o notes, .pptx w/ notes), on theories of counterfactuals in the context of AI.
MIRI research fellow Patrick LaVictoire gave a talk titled "Decision theory and the logic of provability" (.pdf), on the modal agents framework.
MIRI research fellow Benja Fallenstein gave a talk titled "Vingean reflection" (.pdf).
Googler Vladimir Slepnev, a past MIRI workshop attendee, gave a talk titled "Models of decision-making based on logical counterfactuals" (.pdf).
MIRI research associate Stuart Armstrong (Oxford) gave a talk titled "Anthropic decision theory" (.pdf, video).
The conference also coincided with a public lecture by Stuart Russell titled "The long-term future of artificial intelligence" (video).

Our thanks to everyone who attended, and especially to our co-organizers: Arif Ahmed, Huw Price, and Seán Ó hÉigeartaigh!
The post MIRI-related talks from the decision theory conference at Cambridge University appeared first on Machine Intelligence Research Institute.