MIRI's July Newsletter: Fundraiser and New Papers

Greetings from the Executive Director
Dear friends,
Another busy month! Since our last newsletter, we've published 3 new papers and 2 new "analysis" blog posts, we've significantly improved our website (especially the Research page), we've relocated to downtown Berkeley, and we've launched our summer 2013 matching fundraiser!
MIRI also recently presented at the Effective Altruism Summit, a gathering of 60+ effective altruists in Oakland, CA. As philosopher Peter Singer explained in his TED talk, effective altruism "combines both the heart and the head." The heart motivates us to be empathic and altruistic toward others, while the head can "make sure that what [we] do is effective and well-directed," so that altruists can do not just some good but as much good as possible.
As I explain in Friendly AI Research as Effective Altruism, MIRI was founded in 2000 on the premise that creating Friendly AI might be a particularly efficient way to do as much good as possible. Effective altruists focus on a variety of other causes, too, such as poverty reduction. As I say in Four Focus Areas of Effective Altruism, I think it's important for effective altruists to cooperate and collaborate, despite their differences of opinion about which focus areas are optimal. The world needs more effective altruists, of all kinds.
MIRI engages in direct efforts — e.g. Friendly AI research — to improve the odds that machine superintelligence has a positive rather than a negative impact. But indirect efforts — such as spreading rationality and effective altruism — are also likely to play a role, for they will influence the context in which powerful AIs are built. That's part of why we created CFAR.
If you think this work is important, I hope you'll donate now to support our work. MIRI is entirely supported by private funders like you. And if you donate before August 15th, your contribution will be matched by one of the generous backers of our current fundraising drive.
Thank you,
Luke Muehlhauser
Executive Director

Our Summer 2013 Matching Fundraiser
Thanks to the generosity of several major donors, every donation to MIRI made from now until August 15th, 2013 will be matched dollar-for-dollar, up to a total of $200,000!
Now is your chance to double your impact while helping us raise up to $400,000 (with matching) to fund our research program.
Early this year we made a transition from movement-building to research, and we've hit the ground running with six major new research papers, six new strategic analyses on our blog, and much more. Give now to support our ongoing work on the future's most important problem.
Accomplishments in 2013 so far

Changed our name to MIRI and launched our new website at intelligence.org.
Released six new research papers: Definability of Truth in Probabilistic Logic, Intelligence Explosion Microeconomics, Tiling Agents for Self-Modifying AI, Robust Cooperation in the Prisoner's Dilemma, A Comparison of Decision Algorithms on Newcomblike Problems, and Responses to Catastrophic AGI Risk: A Survey.
Held our 2nd research workshop. (Our 3rd workshop is currently ongoing.)
Published six new analyses to our blog: The Lean Nonprofit, AGI Impact Experts and Friendly AI Experts, Five Theses…, When Will AI Be Created?, Friendly AI Research as Effective Altruism, and What is Intelligence?
Published the Facing the Intelligence Explosion ebook.
Published several other substantial articles: Recommended Courses for MIRI Researchers, Decision Theory FAQ, A brief history of ethically concerned scientists, Bayesian Adjustment Does Not Defeat Existential Risk Charity, and others.
And of course much more.

Future Plans You Can Help Support

We will host many more research workshops, including one in September, and one in December (with John Baez attending, among others).
Eliezer will continue to publish about open problems in Friendly AI. (Here is #1 and #2.)
We will continue to publish strategic analyses, mostly via our blog.
We will publish nicely-edited ebooks (Kindle, iBooks, and PDF) for more of our materials, to make them more accessible: The Sequences, 2006-2009 and The Hanson-Yudkowsky AI Foom Debate.
We will continue to set up the infrastructure (e.g. new offices, researcher endowments) required to host a productive Friendly AI research team, and (over several years) recruit enough top-level math talent to launch it.

(Other projects are still being surveyed for likely cost and strategic impact.)
We appreciate your support for our high-impact work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at <PHONE> or <EMAIL>.

New Research Page, Three New Publications
 Our new Research page has launched!
Our previous research page was a simple list of articles, but the new page describes the purpose of our research, explains four categories of research to which we contribute, and highlights the papers we think are most important to read.
We've also released three new research articles.
Tiling Agents for Self-Modifying AI, and the Löbian Obstacle (discuss it here), by Yudkowsky and Herreshoff, explains one of the key open problems in MIRI's research agenda:
We model self-modification in AI by introducing "tiling" agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring's goals). Constructing a formalism in the most straightforward way produces a Gödelian difficulty, the "Löbian obstacle." By technical methods we demonstrates the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed. We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.
Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic (discuss it here), by LaVictoire et al., explains some progress in program equilibrium made by MIRI research associate Patrick LaVictoire and several others during MIRI's April 2013 workshop:
Rational agents defect on the one-shot prisoner's dilemma even though mutual cooperation would yield higher utility for both agents. Moshe Tennenholtz showed that if each program is allowed to pass its playing strategy to all other players, some programs can then cooperate on the one-shot prisoner's dilemma. Program equilibria is Tennenholtz's term for Nash equilibria in a context where programs can pass their playing strategies to the other players. One weakness of this approach so far has been that any two programs which make different choices cannot "recognize" each other for mutual cooperation, even if they are functionally identical. In this paper, provability logic is used to enable a more flexible and secure form of mutual cooperation.
Responses to Catastrophic AGI Risk: A Survey (discuss it here), by Sotala and Yampolskiy, is a summary of the extant literature (250+ references) on AGI risk, and can serve either as a guide for researchers or as an introduction for the uninitiated.
Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field's proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.

Two New Analyses
MIRI publishes some of its most substantive research to its blog, under the Analysis category. For example, When Will AI Be Created? is the product of 20+ hours of work, and has 14 footnotes and 40+ scholarly references (all of them linked to PDFs).
Last month, we published two new analyses.
Friendly AI Research as Effective Altruism presents a bare-bones version of an argument that Friendly AI research is a particularly efficient way to purchase expected value, so that the argument can be elaborated and critiqued by MIRI and others.
What is Intelligence? argues that imprecise working definitions can be useful, an explains the particular imprecise working definition for intelligence that we tend to use at MIRI: efficient cross-domain optimization. A future post will discuss some potentially useful working definitions for "artificial general intelligence."

Grant Writer Needed
MIRI would like to hire someone to write grant applications, both for our research efforts and for STEM education. If you have experience with either, please apply here.
The pay will depend on skill and experience, and is negotiable.

Featured Volunteer
Oliver Habryka helps out by proofreading MIRI's paper, and would be able to contribute to our research at some point, perhaps on the subject of "lessons for ethics from machine ethics." Independent of his direct contributions to MIRI's work, Oliver has also lectured on topics related to MIRI's work at his high school, and has also taught a class on rationality, where he inspired participation by using a "leveling up" reward system. Oliver is currently studying the foundations of mathematics and hopes one day to direct his career goals in such a way that his contributions to our mission increase over time.

The post MIRI's July Newsletter: Fundraiser and New Papers appeared first on Machine Intelligence Research Institute.

MIRI’s July Newsletter: Fundraiser and New Papers


2013 Summer Matching Challenge!

Thanks to the generosity of several major donors,† every donation to the Machine Intelligence Research Institute made from now until August 15th, 2013 will be matched dollar-for-dollar, up to a total of $200,000!

Donate Now!

$0

$50,000

$100,000

$150,000

$200,000

We have reached our goal of $200,000!
Now is your chance to double your impact while helping us raise up to $400,000 (with matching) to fund our research program.

Early this year we made a transition from movement-building to research, and we've hit the ground running with six major new research papers, six new strategic analyses on our blog, and much more. Give now to support our ongoing work on the future's most important problem.

Accomplishments in 2013 so far

Changed our name to MIRI and launched our new website at intelligence.org.
Released six new research papers: Definability of Truth in Probabilistic Logic, Intelligence Explosion Microeconomics, Tiling Agents for Self-Modifying AI, Robust Cooperation in the Prisoner's Dilemma, A Comparison of Decision Algorithms on Newcomblike Problems, and Responses to Catastrophic AGI Risk: A Survey.
Held our 2nd research workshop. (Our 3rd workshop is currently ongoing.)
Published six new analyses to our blog: The Lean Nonprofit, AGI Impact Experts and Friendly AI Experts, Five Theses…, When Will AI Be Created?, Friendly AI Research as Effective Altruism, and What is Intelligence?
Published the Facing the Intelligence Explosion ebook.
Published several other substantial articles: Recommended Courses for MIRI Researchers, Decision Theory FAQ, A brief history of ethically concerned scientists, Bayesian Adjustment Does Not Defeat Existential Risk Charity, and others.
And of course much more.

Future Plans You Can Help Support

We will host many more research workshops, including one in September, and one in December (with John Baez attending, among others).
Eliezer will continue to publish about open problems in Friendly AI. (Here is #1 and #2.)
We will continue to publish strategic analyses, mostly via our blog.
We will publish nicely-edited ebooks (Kindle, iBooks, and PDF) for more of our materials, to make them more accessible: The Sequences, 2006-2009 and The Hanson-Yudkowsky AI Foom Debate.
We will continue to set up the infrastructure (e.g. new offices, researcher endowments) required to host a productive Friendly AI research team, and (over several years) recruit enough top-level math talent to launch it.

(Other projects are still being surveyed for likely cost and strategic impact.)
We appreciate your support for our high-impact work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at <PHONE> or <EMAIL>.
† $200,000 of total matching funds has been provided by Jaan Tallinn, Loren Merritt, Rick Schwall, and Alexei Andreev.
The post 2013 Summer Matching Challenge! appeared first on Machine Intelligence Research Institute.

2013 Summer Matching Challenge!


MIRI Has Moved!

For the past several months, MIRI and its child organization CFAR have been working from a much-too-small office on the outskirts of Berkeley. At the end of June, MIRI and CFAR took over the 3rd floor of 2030 Addison St. in downtown Berkeley, which has sufficient space for both organizations.
Our new office is 0.5 blocks from the Downtown Berkeley BART exit at Shattuck & Addison, and 2 blocks from the UC Berkeley campus. Here's a photo of the campus from our roof:

The proximity to UC Berkeley will make it easier for MIRI to network with Berkeley's professors and students. Conveniently, UC Berkeley is ranked 5th in the world in mathematics, and 1st in the world in mathematical logic.
Sharing an office with CFAR carries many benefits for both organizations:

CFAR and MIRI can "flex" into each other's space for short periods as needed, for example when MIRI is holding a week-long research workshop.
We can share resources (printers, etc.).
Both organizations can benefit from interaction between our two communities.

Getting the new office was a team effort, but the person most responsible for this success was MIRI Deputy Director Louie Helm.
Note that MIRI isn't yet able to accommodate "drop in" visitors, as we keep irregular hours throughout the week. So if you'd like to visit, please contact us first.
We retain 2721 Shattuck Ave. #1023 as an alternate mailing address.
The post MIRI Has Moved! appeared first on Machine Intelligence Research Institute.

MIRI Has Moved!


MIRI's September 2013 Workshop

From September 7-13, MIRI will host its 4th Workshop on Logic, Probability, and Reflection. The focus of this workshop will be the foundations of decision theory.
Participants confirmed so far include:

Paul Christiano (UC Berkeley)
Wei Dai (independent)
Gary Drescher (independent)
Kenny Easwaran (USC)
Cameron Freer (MIT)
Patrick LaVictoire (Quixey)
Ilya Shpitser (U Southampton)
Vladimir Slepnev (Google)
Nisan Stiennon (Stanford)
Andreas Stuhlmüller (MIT & Stanford)
Eliezer Yudkowsky (MIRI)

If you have a strong mathematics background and might like to attend this workshop, it's not too late to apply! And even if this workshop doesn't fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).
The post MIRI's September 2013 Workshop appeared first on Machine Intelligence Research Institute.

MIRI’s September 2013 Workshop


Responses to Catastrophic AGI Risk: A Survey

MIRI is self-publishing another technical report that was too lengthy (60 pages) for publication in a journal: Responses to Catastrophic AGI Risk: A Survey.
The report, co-authored by past MIRI researcher Kaj Sotala and University of Louisville's Roman Yampolskiy, is a summary of the extant literature (250+ references) on AGI risk, and can serve either as a guide for researchers or as an introduction for the uninitiated.
Here is the abstract:
Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field's proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.
The preferred discussion page for the paper is here.
Update: This report has now been published in Physica Scripta, available here.
The post Responses to Catastrophic AGI Risk: A Survey appeared first on Machine Intelligence Research Institute.


What is Intelligence?

When asked their opinions about "human-level artificial intelligence" — aka "artificial general intelligence" (AGI)1 — many experts understandably reply that these terms haven't yet been precisely defined, and it's hard to talk about something that hasn't been defined.2 In this post, I want to briefly outline an imprecise but useful "working definition" for intelligence we tend to use at MIRI. In a future post I will write about some useful working definitions for artificial general intelligence.

Imprecise definitions can be useful
Precise definitions are important, but I concur with Bertrand Russell that
[You cannot] start with anything precise. You have to achieve such precision… as you go along.
Physicist Milan Ćirković agrees, and gives an example:
The formalization of knowledge — which includes giving precise definitions — usually comes at the end of the original research in a given field, not at the very beginning. A particularly illuminating example is the concept of number, which was properly defined in the modern sense only after the development of axiomatic set theory in the… twentieth century.3
For a more AI-relevant example, consider the concept of a "self-driving car," which has been given a variety of vague definitions since the 1930s. Would a car guided by a buried cable qualify? What about a modified 1955 Studebaker that could use sound waves to detect obstacles and automatically engage the brakes if necessary, but could only steer "on its own" if each turn was preprogrammed? Does that count as a "self-driving car"?
What about the "VaMoRs" of the 1980s that could avoid obstacles and steer around turns using computer vision, but weren't advanced enough to be ready for public roads? How about the 1995 Navlab car that drove across the USA and was fully autonomous for 98.2% of the trip, or the robotic cars which finished the 132-mile off-road course of the 2005 DARPA Grand Challenge, supplied only with the GPS coordinates of the route? What about the winning cars of the 2007 DARPA Grand Challenge, which finished an urban race while obeying all traffic laws and avoiding collisions with other cars? Does Google's driverless car qualify, given that it has logged more than 500,000 autonomous miles without a single accident under computer control, but still struggles with difficult merges and snow-covered roads?4
Our lack of a precise definition for "self-driving car" doesn't seem to have hindered progress on self-driving cars very much.5 And I'm glad we didn't wait to seriously discuss self-driving cars until we had a precise definition for the term.
Similarly, I don't think we should wait for a precise definition of AGI before discussing the topic seriously. On the other hand, the term is useless if it carries no information. So let's work our way toward a stipulative, operational definition for AGI. We'll start by developing an operational definition for intelligence.

A definition for "intelligence"
Legg and Hutter (2007) found that definitions of intelligence converge toward the idea that "Intelligence measures an agent's ability to achieve goals in a wide range of environments." Let's call this the "optimization power" concept of intelligence, because it measures an agent's power to optimize the world according to its preferences.
I think this is a productive approach to the issue, since it identifies intelligence with externally measurable performance rather than with the details of how that performance might be achieved (e.g. via consciousness, brute force calculation, "complexity," or something else). Moreover, it's usually performance we care about: we tend to care most about whether an AI will perform well enough to replace human workers, or whether it will perform well enough improve its own abilities without human assistance, not whether it has some particular internal feature.6
Furthermore, the concept of optimization power allows us to compare the intelligence of different kinds of agents. As Albus (1991) said, "A useful definition of intelligence… should include both biological and machine embodiments, and these should span an intellectual range from that of an insect to that of an Einstein, from that of a thermostat to that of the most sophisticated computer system that could ever be built."
I'd like to add one more consideration, though. What if two agents have roughly equal ability to optimize the world according to their preferences, but the second agent requires far more resources to do so? These agents have the same optimization power, but the first one seems to be optimizing more intelligently. So perhaps we could use "intelligence" to mean "optimization power divided by resources used" — what Yudkowsky called efficient cross-domain optimization.7
Other definitions8 have their merits, too. But at MIRI we find the concept of "efficient cross-domain optimization" sufficiently useful that it serves as our (still imprecise!) working definition for intelligence.
In a future post (Edit: here), I'll discuss some useful working definitions for artificial general intelligence.

I use the HLAI and AGI interchangeably, but lately I've been using AGI almost exclusively, because I've learned that many people in the AI community react negatively to any mention of "human-level" AI but have no objection to the concept of narrow vs. general intelligence. See also Ben Goertzel's comments here.Asked when he thought HLAI would be created, Pat Hayes (a past president of AAAI) replied: "I do not consider this question to be answerable, as I do not accept this (common) notion of 'human-level intelligence' as meaningful." Asked the same question, AI scientist William Uther replied: "You ask a lot about 'human level AGI'. I do not think this term is well defined," while AI scientist Alan Bundy replied: "I don't think the concept of 'human-level machine intelligence' is well formed."Sawyer (1943) gives another example: "Mathematicians first used the sign √-1, without in the least knowing what it could mean, because it shortened work and led to correct results. People naturally tried to find out why this happened and what √-1 really meant. After two hundreds years they succeeded." Dennett (2013) makes a related comment: "Define your terms, sir! No, I won't. That would be premature… My [approach] is an instance of nibbling on a tough problem instead of trying to eat (and digest) the whole thing from the outset… In Elbow Room, I compared my method to the sculptor's method of roughing out the form in a block of marble, approaching the final surfaces cautiously, modestly, working by successive approximation."With self-driving cars, researchers did use many precise external performance measures (e.g. accident rates, speed, portion of the time they could run unassisted, frequency of getting stuck) to evaluate progress, as well as internal performance metrics (speed of search, bounded loss guarantees, etc.). Researchers could see that these bits of progress were in the right direction, even if their relative contribution long-term was unclear. And so it is with AI in general. AI researchers use many precise external and internal performance measures to evaluate progress, but it is difficult to know the relative contribution of these bits of progress toward the final goal of AGI.Heck, we've had pornography for millennia and still haven't been able to define it precisely. Encyclopedia entries for "pornography" often simply quote Justice Potter Stewart: "I shall not today attempt further to define the kinds of material I understand to be [pornography]… but I know it when I see it."We might care about whether machines are conscious in addition to being intelligent, but we already have a convenient term for that: consciousness. In particular, we might care about machine consciousness because the slow, plodding invention of AI may involve the creation and destruction of millions of partially-conscious near-AIs that are switched on, suffer for a while, and are then switched off — all while being unable to signal to us that they are suffering. This is especially likely if we remain unclear about the nature of consciousness for several more decades, and thus have no principled way (e.g. via nonperson predicates) to create intelligent machines that we know are not conscious (and are thus incapable of suffering). One of the first people to make this point clearly was Metzinger (2003), p. 621: "What would you say if someone came along and said, 'Hey, we want to genetically engineer mentally retarded human infants! For reasons of scientific progress we need infants with certain cognitive and emotional deficits in order to study their postnatal psychological development—we urgently need some funding for this important and innovative kind of research!' You would certainly think this was not only an absurd and appalling but also a dangerous idea. It would hopefully not pass any ethics committee in the democratic world. However, what today's ethics committees don't see is how the first machines satisfying a minimally sufficient set of constraints for conscious experience could be just like such mentally retarded infants. They would suffer from all kinds of functional and representational deficits too. But they would now also subjectively experience those deficits. In addition, they would have no political lobby—no representatives in any ethics committee." Metzinger repeats the point in Metzinger (2010), starting on page 194.Admittedly, this is still pretty vague. One step toward precision would be to propose a definition of intelligence as optimization power for some canonical distribution of possible preferences, over some canonical distribution of environments, with a penalty for resource use. The canonical preferences and canonical environments could be weighted toward preferences and environments relevant to our concerns: we care more about whether AIs can do science than whether they can paint abstract art, and we care more about whether they can achieve their goals in our solar system than whether they can achieve their goals inside a black hole. Also see Goertzel (2010)'s "efficient pragmatic general intelligence."Hibbard (2011); Legg & Veness (2011); Wang (2008); Schaul et al. (2011); Dowe & Hernandez-Orallo (2012); Goertzel (2010); Adams et al. (2011).The post What is Intelligence? appeared first on Machine Intelligence Research Institute.

What is Intelligence?


MIRI's July 2013 Workshop

From July 8-14, MIRI will host its 3rd Workshop on Logic, Probability, and Reflection. The focus of this workshop will be the Löbian obstacle to self-modifying systems.
Participants confirmed so far include:

Andrew Critch (just finished his math PhD at UC Berkeley, now working at CFAR)
Abram Demski (USC)
Benja Fallenstein (Bristol U)
Marcello Herreshoff (Google)
Jonathan Lee (Cambridge)
Will Sawin (Princeton)
Qioachu Yuan (UC Berkeley)
Eliezer Yudkowsky (MIRI)

If you have a strong mathematics background and might like to attend this workshop, it's not too late to apply! And even if this workshop doesn't fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).
Information on past workshops:

Our 1st Workshop (Nov. 11-18, 2012; 4 participants) resulted in Christiano's probabilistic logic, an attack on the Löbian obstacle for self-modifying systems.

Our 2nd Workshop (Apr. 3-24, 2013; 12 participants coming in and out) resulted in (1) some as-yet unpublished progress on Christiano's probabilistic logic, (2) some progress on program equilibrium recorded in LaVictoire et al. (2013), and some progress on the Löbian obstacle resulting in Yudkowsky & Herreschoff (2013).

The post MIRI's July 2013 Workshop appeared first on Machine Intelligence Research Institute.

MIRI’s July 2013 Workshop


New Research Page and Two New Articles

Our new Research page has launched!
Our previous research page was a simple list of articles, but the new page describes the purpose of our research, explains four categories of research to which we contribute, and highlights the papers we think are most important to read.
We've also released drafts of two new research articles.
Tiling Agents for Self-Modifying AI, and the Löbian Obstacle (discuss it here), by Yudkowsky and Herreshoff, explains one of the key open problems in MIRI's research agenda:
We model self-modification in AI by introducing "tiling" agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring's goals). Constructing a formalism in the most straightforward way produces a Gödelian difficulty, the "Löbian obstacle." By technical methods we demonstrates the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed. We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.
Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic (discuss it here), by LaVictoire et al., explains some progress in program equilibrium made by MIRI research associate Patrick LaVictoire and several others during MIRI's April 2013 workshop:
Rational agents defect on the one-shot prisoner's dilemma even though mutual cooperation would yield higher utility for both agents. Moshe Tennenholtz showed that if each program is allowed to pass its playing strategy to all other players, some programs can then cooperate on the one-shot prisoner's dilemma. Program equilibria is Tennenholtz's term for Nash equilibria in a context where programs can pass their playing strategies to the other players.
One weakness of this approach so far has been that any two programs which make different choices cannot "recognize" each other for mutual cooperation, even if they are functionally identical. In this paper, provability logic is used to enable a more flexible and secure form of mutual cooperation.
Participants of MIRI's April workshop also made progress on Christiano's probabilistic logic (an attack on the Löbian obstacle), but that work is not yet ready to be released.
We've also revamped the Get Involved page, which now includes an application form for forthcoming workshops. If you might like to work with MIRI on some of its open research problems sometime in the next 18 months, please apply! Likewise, if you know someone who might enjoy attending such a workshop, please encourage them to apply.
The post New Research Page and Two New Articles appeared first on Machine Intelligence Research Institute.

New Research Page and Two New Articles


Friendly AI Research as Effective Altruism

MIRI was founded in 2000 on the premise that creating1 Friendly AI might be a particularly efficient way to do as much good as possible.
Some developments since then include:

The field of "effective altruism" — trying not just to do good but to do as much good as possible2 — has seen more publicity and better research than ever before, in particular through the work of GiveWell, the Center for Effective Altruism, the philosopher Peter Singer, and the community at Less Wrong.3
In his recent PhD dissertation, Nick Beckstead has clarified the assumptions behind the claim that shaping the far future (e.g. via Friendly AI) is overwhelmingly important.
Due to research performed by MIRI, the Future of Humanity Institute (FHI), and others, our strategic situation with regard to machine superintelligence is more clearly understood, and FHI's Nick Bostrom has organized much of this work in a forthcoming book.4
MIRI's Eliezer Yudkowsky has begun to describe in more detail which open research problems constitute "Friendly AI research," in his view.

Given these developments, we are in a better position than ever before to assess the value of Friendly AI research as effective altruism.
Still, this is a difficult question. It is challenging enough to evaluate the cost-effectiveness of anti-malaria nets or direct cash transfers. Evaluating the cost-effectiveness of attempts to shape the far future (e.g. via Friendly AI) is even more difficult than that. Hence, this short post sketches an argument that can be given in favor of Friendly AI research as effective altruism, to enable future discussion, and is not intended as a thorough analysis.

An argument for Friendly AI research as effective altruism
Beckstead (2013) argues5 for the following thesis:
From a global perspective, what matters most (in expectation) is that we do what is best (in expectation) for the general trajectory along which our descendants develop over the coming millions, billions, and trillions of years.
Why think this? Astronomical facts suggest that humanity (including "post-humanity") could survive for billions or trillions of years (Adams 2008), and could thus produce enormous amounts of good.6 But the value produced by our future depends on our development trajectory. If humanity destroys itself with powerful technologies in the 21st century, then nearly all that future value is lost. And if we survive but develop along a trajectory dominated by conflict and poor decisions, then the future could be much less good than if our trajectory is dominated by altruism and wisdom. Moreover, some of our actions today can have "ripple effects"7 which determine the trajectory of human development, because many outcomes are path-dependent. Hence, actions which directly or indirectly precipitate particular trajectory changes (e.g. mitigating existential risks) can have vastly more value (in expectation) than actions with merely proximate benefits (e.g. saving the lives of 20 wild animals). Beckstead calls this the "rough future-shaping argument."
If we accept the normative assumptions lurking behind this argument (e.g. risk neutrality; see Beckstead's dissertation), then the far future is enormously valuable (if it goes at least as well on average as the past century), and existential risk reduction is much more important than producing proximate benefits (e.g. global health, poverty reduction) or speeding up development (which could in fact increase existential risks, and even if it doesn't, has lower expected value than existential risk reduction).
However, Beckstead's conclusion is not necessarily that existential risk reduction should be our global priority, because
there may be other ways to have a large, persistent effect on the far future without reducing existential risk… Some persistent changes in values and social norms could make the future [some fraction] better or worse… Sure, succeeding in preventing an existential catastrophe would be better than making a smaller trajectory change, but creating a small positive trajectory change may be significantly easier.
Instead, Beckstead's arguments suggest that "what matters most for shaping the far future is producing positive trajectory changes and avoiding negative ones." Existential risk reduction is one important kind of positive trajectory change that could turn out to be the intervention with the highest expected value.
One important clarification is in order. It could turn out to be that working toward proximate benefits or development acceleration does more good than "direct" efforts for trajectory change, if working toward proximate benefits or development acceleration turns out to have major ripple effects which produce important trajectory change. For example, perhaps an "ordinary altruistic effort" like solving India's iodine deficiency problem would cause there to be thousands of "extra" world-class elite thinkers two generations from now, which could increase humanity's chances of intelligently navigating the crucial 21st century and spreading to the stars. (I don't think this is likely; I suggest it merely for illustration.)
For the sake of argument, suppose you agree with Beckstead's core thesis that "what matters most (in expectation) is that we do what is best (in expectation) for the general trajectory along which our descendants develop." Suppose you also think, as I do, that machine superintelligence is probably inevitable.8
In that case, you might think that Friendly AI research is a uniquely foreseeable and impactful way to shape the far future in an enormously positive way, because "our effects on the far future must almost entirely pass through our effects on the development of machine superintelligence." All other developing trends might be overridden by the overwhelming effectiveness of machine superintelligence — and specifically, by the values that were (explicitly or implicitly, directly or indirectly) written into the machine superintelligence(s).
If that's right, our situation is a bit like sending an interstellar probe to colonize distant solar systems before they recede beyond the cosmological horizon and can thus never be reached from Earth again due to the expansion of the universe. Anything on Earth that doesn't affect the content of the probe will have no impact on those solar systems. (See also this comment.)

Potential defeaters
The rough argument above — in favor of Friendly AI research as an efficient form of effective altruism — deserves to be "fleshed out" in more detail.9
Potential defeaters should also be examined:

Perhaps we ought to reject one or more of the normative assumptions behind Beckstead's rough future-shaping argument.
Perhaps it's not true that "our effects on the far future must almost entirely pass through our effects on the development of machine superintelligence."
Perhaps Friendly AI research is not (today) a particularly efficient way to positively affect the development of machine superintelligence. Competing interventions may include: (1) AI risk strategy research, (2) improving technological forecasting, (3) improving science in general, (4) improving and spreading effective altruism and rationality, and (5) many others.

In future blog posts, members of the effective altruist community (including myself) will expand on the original argument and examine potential defeaters.

Acknowledgements
My thanks to those who provided feedback on this post: Carl Shulman, Nick Beckstead, Jonah Sinick, and Eliezer Yudkowsky.
In this post, I talk about the value of humanity in general creating Friendly AI, though MIRI co-founder Eliezer Yudkowsky usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI. This is because I am not as confident as Yudkowsky that it is best for MIRI to attempt to build Friendly AI. When updating MIRI's bylaws in early 2013, Yudkowsky and I came to a compromise on the language of MIRI's mission statement, which now reads: "[MIRI] exists to ensure that the creation of smarter-than-human intelligence has a positive impact. Thus, the charitable purpose of [MIRI] is to: (a) perform research relevant to ensuring that smarter-than-human intelligence has a positive impact; (b) raise awareness of this important issue; (c) advise researchers, leaders and laypeople around the world; and (d) as necessary, implement a smarter-than-human intelligence with humane, stable goals" (emphasis added). My own hope is that it will not be necessary for MIRI (or a functional equivalent) to attempt to build Friendly AI itself. But of course I must remain open to the possibility that this will be the wisest course of action as the first creation of AI draws nearer. There is also the question of capability: few people think that a non-profit research organization has much chance of being the first to build AI. I worry, however, that the world's elites will not find it fashionable to take this problem seriously until the creation of AI is only a few decades away, at which time it will be especially difficult to develop the mathematics of Friendly AI in time, and humanity will be forced to take a gamble on its very survival with powerful AIs we have little reason to trust.One might think of effective altruism as a straightforward application of decision theory to the subject of philanthropy. Philanthropic agents of all kinds (individuals, groups, foundations, etc.) ask themselves: "How can we choose philanthropic acts (e.g. donations) which (in expectation) will do as much good as possible, given what we care about?" The consensus recommendation for all kinds of choices under uncertainty, including philanthropic choices, is to maximize expected utility (Chater & Oaksford 2012; Peterson 2004; Stein 1996; Schmidt 1998:19). Different philanthropic agents value different things, but decision theory suggests that each of them can get the most of what they want if they each maximize their expected utility. Choices which maximize expected utility are in this sense "optimal," and thus another term for effective altruism is "optimal philanthropy." Note that effective altruism in this sense is not too dissimilar from earlier approaches to philanthropy, including high-impact philanthropy (making "the biggest difference possible, given the amount of capital invested"), strategic philanthropy, effective philanthropy, and wise philanthropy. Note also that effective altruism does not say that a philanthropic agent should specify complete utility and probability functions over outcomes and then compute the philanthropic act with the highest expected utility — that is impractical for bounded agents. We must keep in mind the distinction between normative, descriptive, and prescriptive models of decision-making (Baron 2007): "normative models tell us how to evaluate… decisions in terms of their departure from an ideal standard. Descriptive models specify what people in a particular culture actually do and how they deviate from the normative models. Prescriptive models are designs or inventions, whose purpose is to bring the results of actual thinking into closer conformity to the normative model." The prescriptive question — about what bounded philanthropic agents should do to maximize expected utility with their philanthropic choices — tends to be extremely complicated, and is the subject of most of the research performed by the effective altruism community.See, for example: Efficient Charity, Efficient Charity: Do Unto Others, Politics as Charity, Heuristics and Biases in Charity, Public Choice and the Altruist's Burden, On Charities and Linear Utility, Optimal Philanthropy for Human Beings, Purchase Fuzzies and Utilons Separately, Money: The Unit of Caring, Optimizing Fuzzies and Utilons: The Altruism Chip Jar, Efficient Philanthropy: Local vs. Global Approaches, The Effectiveness of Developing World Aid, Against Cryonics & For Cost-Effective Charity, Bayesian Adjustment Does Not Defeat Existential Risk Charity, How to Save the World, and What is Optimal Philanthropy?I believe Beckstead and Bostrom have done the research community an enormous service in creating a framework, a shared language, for discussing trajectory changes, existential risks, and machine superintelligence. When discussing these topics with my colleagues, it has often been the case that the first hour of conversation is spent merely trying to understand what the other person is saying — how they are using the terms and concepts they employ. Beckstead's and Bostrom's recent work should enable clearer and more efficient communication between researchers, and therefore greater research productivity. Though I am not aware of any controlled, experimental studies on the effect of shared language on research productivity, a shared language is widely considered to be of great benefit for any field of research, and I shall provide a few examples of this claim which appear in print. Fuzzi et al. (2006): "The use of inconsistent terms can easily lead to misunderstandings and confusion in the communication between specialists from different [disciplines] of atmospheric and climate research, and may thus potentially inhibit scientific progress." Hinkel (2008): "Technical languages enable their users, e.g. members of a scientific discipline, to communicate efficiently about a domain of interest." Madin et al. (2007): "terminological ambiguity slows scientific progress, leads to redundant research efforts, and ultimately impedes advances towards a unified foundation for ecological science."In addition to Beckstead's thesis, see also A Proposed Adjustment to the Astronomical Waste Argument.Beckstead doesn't mention this, but I would like to point out that moral realism is not required for Beckstead's arguments to go through. In fact, I generally accept Beckstead's arguments even though most philosophers would not consider me a moral realist, though to some degree that is a semantic debate (Muehlhauser 2011; Joyce 2012). If you're a moral realist and you believe your intuitive moral judgments are data about what is morally true, then Beckstead's arguments (if successful) have something to say about what is morally true, and about what you should do if you want to act in morally good ways. If you're a moral anti-realist but you think your intuitive judgments are data about what you value — or about what you would value if you had more time to think about your values and how to resolve the contradictions among them — then Beckstead's arguments (if successful) have something to say about what you value, and about what you should do if you want to help achieve what you value.Karnofsky calls these "flow-through effects."See Bostrom (forthcoming) for an extended argument. Perhaps the most likely defeater for machine superintelligence is that global catastrophe may halt scientific progress before human-level AI is created.Beckstead, in personal communication, suggested (but didn't necessarily endorse) the following formalization of the rough argument sketched in the main text of the blog post: "(1) To a first approximation, the future of humanity is all that matters. (2) To a much greater extent than anything else, the future of humanity is highly sensitive to how machine intelligence unfolds. (3) Therefore, there is a very strong presumption in favor of working on any project which makes machine intelligence unfold in a better way. (4) FAI research is the most promising route to making machine intelligence unfold in a better way. (5) Therefore, there is a very strong presumption in favor of doing FAI research." Beckstead (2013) examines the case for (1). Bostrom (forthcoming), in large part, examines the case for (2). Premise (3) informally follows from (1) and (2), and the conclusion (5) informally follows from (3) and (4). Premise (4) appears to me to be the most dubious part of the argument, and the least explored in the extant literature.The post Friendly AI Research as Effective Altruism appeared first on Machine Intelligence Research Institute.


MIRI May Newsletter: Intelligence Explosion Microeconomics and Other Publications

Greetings From the Executive Director
Dear friends,
It's been a busy month!
Mostly, we've been busy publishing things. As you'll see below, Singularity Hypotheses has now been published, and it includes four chapters by MIRI researchers or research associates. We've also published two new technical reports — one on decision theory and another on intelligence explosion microeconomics — and several new blog posts analyzing various issues relating to the future of AI. Finally, we added four older articles to the research page, including Ideal Advisor Theories and Personal CEV (2012).
In our April newsletter we spoke about our April 11th party in San Francisco, celebrating our relaunch as the Machine Intelligence Research Institute and our transition to mathematical research. Additional photos from that event are now available as a Facebook photo album. We've also uploaded a video from the event, in which I spend 2 minutes explaining MIRI's relaunch and some tentative results from the April workshop. After that, visiting researcher Qiaochu Yuan spends 4 minutes explaining one of MIRI's core research questions: the Löbian obstacle to self-modifying systems.
Some of the research from our April workshop will be published in June, so if you'd like to read about those results right away, you might like to subscribe to our blog.
Cheers!
Luke Muehlhauser
Executive Director

Intelligence Explosion Microeconomics
Our largest new publication this month is Yudkowsky's 91-page Intelligence Explosion Microeconomics (discuss here). In this article, Yudkowsky takes some initial steps toward tackling the key quantitative issue in the intelligence explosion, "reinvestable returns on cognitive investments": what kind of returns can you get from an investment in cognition, can you reinvest it to make yourself even smarter, and does this process die out or blow up? The article can be thought of as a compact and hopefully more coherent successor to the AI Foom Debate of 2009, featuring Yudkowsky and GMU economist Robin Hanson.
Here is the abstract:
I. J. Good's thesis of the 'intelligence explosion' is that a sufficiently advanced machine intelligence could build a smarter version of itself, which could in turn build an even smarter version of itself, and that this process could continue enough to vastly exceed human intelligence. As Sandberg (2010) correctly notes, there are several attempts to lay down return-on-investment formulas intended to represent sharp speedups in economic or technological growth, but very little attempt has been made to deal formally with I. J. Good's intelligence explosion thesis as such.
I identify the key issue as returns on cognitive reinvestment — the ability to invest more computing power, faster computers, or improved cognitive algorithms to yield cognitive labor which produces larger brains, faster brains, or better mind designs. There are many phenomena in the world which have been argued as evidentially relevant to this question, from the observed course of hominid evolution, to Moore's Law, to the competence over time of machine chess-playing systems, and many more. I go into some depth on the sort of debates which then arise on how to interpret such evidence. I propose that the next step forward in analyzing positions on the intelligence explosion would be to formalize return-on-investment curves, so that each stance can say formally which possible microfoundations they hold to be falsified by historical observations already made. More generally, I pose multiple open questions of 'returns on cognitive reinvestment' or 'intelligence explosion microeconomics'. Although such questions have received little attention thus far, they seem highly relevant to policy choices affecting the outcomes for Earth-originating intelligent life.
The dedicated mailing list will be small and restricted to technical discussants: apply for it here.

When Will AI Be Created?
In part, intelligence explosion microeconomics seeks to answer the question "How quickly will human-level AI self-improve to become superintelligent?" Another major question in AI forecasting is, of course: "When will we create human-level AI?"
This is another difficult question, and Luke Muehlhauser surveyed those difficulties in a recent (and quite detailed) blog post: When Will AI Be Created? He concludes:
Given these considerations, I think the most appropriate stance on the question "When will AI be created?" is something like this:
"We can't be confident AI will come in the next 30 years, and we can't be confident it'll take more than 100 years, and anyone who is confident of either claim is pretending to know too much."
How confident is "confident"? Let's say 70%. That is, I think it is unreasonable to be 70% confident that AI is fewer than 30 years away, and I also think it's unreasonable to be 70% confident that AI is more than 100 years away.
This statement admits my inability to predict AI, but it also constrains my probability distribution over "years of AI creation" quite a lot.
I think the considerations above justify these constraints on my probability distribution, but I haven't spelled out my reasoning in great detail. That would require more analysis than I can present here. But I hope I've at least summarized the basic considerations on this topic, and those with different probability distributions than mine can now build on my work here to try to justify them.
Muehlhauser also explains four methods for reducing our uncertainty about AI timelines: explicit quantification, leveraging aggregation, signposting the future, and decomposing the phenomena.
As it turns out, you can participate in the first two methods for improving our AI forecasts by signing up for GMU's DAGGRE program. Muehlhauser himself has signed up.
Muehlhauser also wrote a 400-word piece on the difficulty of AI forecasting for Quartz magazine: Robots will take our jobs, but it's hard to say when. Here's a choice quote:
We've had the computing power of a honeybee's brain for quite a while now, but that doesn't mean we know how to build tiny robots that fend for themselves outside the lab, find their own sources of energy, and communicate with others to build their homes in the wild.

Singularity Hypothesis Published

Singularity Hypotheses: A Scientific and Philosophical Assessment has now been published by Springer, in hardcover and ebook forms.
The book contains 20 chapters about the prospect of machine superintelligence, including 4 chapters by MIRI researchers and research associates:

Intelligence Explosion: Evidence and Import by Luke Muehlhauser and Anna Salamon
Intelligence Explosion and Machine Ethics by Luke Muehlhauser and Louie Helm
Friendly Artificial Intelligence by Eliezer Yudkowsky, a shortened version of Yudkowsky (2008)
Artificial General Intelligence and the Human Mental Model by Roman Yampolskiy and (MIRI research associate) Joshua Fox

For more details, see the blog post.

Timeless Decision Theory Paper Published
 During his time as a research fellow for MIRI, Alex Altair wrote an article on Timeless Decision Theory (TDT) that has now been published: "A Comparison of Decision Algorithms on Newcomblike Problems."
Altair's article is both more succinct and also more precise in its formulation of TDT than Yudkowsky's earlier paper "Timeless Decision Theory." Thus, Altair's paper should serve as a handy introduction to TDT for philosophers, computer scientists, and mathematicians, while Yudkowsky's paper remains required reading for anyone interested to develop TDT further, for it covers more ground than Altair's paper.
For a gentle introduction to the entire field of normative decision theory (including TDT), see Muehlhauser and Williamson's Decision Theory FAQ.

AGI Impacts Experts and Friendly AI Experts
In AGI Impacts Experts and Friendly AI Experts, Luke Muehlhauser explains the two types of experts MIRI hopes to cultivate.
AGI impacts experts develop skills related to predicting technological development (e.g. building computational models of AI development or reasoning about intelligence explosion microeconomics), predicting AGI's likely impacts on society, and identifying which interventions are most likely to increase humanity's chances of safely navigating the creation of AGI. For overviews, see Bostrom & Yudkowsky (2013); Muehlhauser & Salamon (2013).
Friendly AI experts develop skills useful for the development of mathematical architectures that can enable AGIs to be trustworthy (or "human-friendly"). This work is carried out at MIRI research workshops and in various publications, e.g. Christiano et al. (2013); Hibbard (2013). Note that the term "Friendly AI" was selected (in part) to avoid the suggestion that we understand the subject very well — a phrase like "Ethical AI" might sound like the kind of thing one can learn a lot about by looking it up in an encyclopedia, but our present understanding of trustworthy AI is too impoverished for that.
For more details on which skills these kinds of experts should develop, read the blog post.

MIRI's Mission in Five Theses and Two Lemmas
Yudkowsky sums up MIRI's research mission in the blog post Five theses, two lemmas, and a couple of strategic implications. The five theses are:

Intelligence explosion
Orthogonality of intelligence and goals
Convergent instrumental goals
Complexity of value
Fragility of value

According to Yudkowsky, these theses imply two important lemmas:

Indirect normativity
Large bounded extra difficulty of Friendliness

In turn, these two lemmas have two important strategic implications:

We have a lot of work to do on things like indirect normativity and stable self-improvement. At this stage a lot of this work looks really foundational — that is, we can't describe how to do these things using infinite computing power, let alone finite computing power.  We should get started on this work as early as possible, since basic research often takes a lot of time.
There needs to be a Friendly AI project that has some sort of boost over competing projects which don't live up to a (very) high standard of Friendly AI work — a project which can successfully build a stable-goal-system self-improving AI, before a less-well-funded project hacks together a much sloppier self-improving AI. Giant supercomputers may be less important to this than being able to bring together the smartest researchers… but the required advantage cannot be left up to chance. Leaving things to default means that projects less careful about self-modification would have an advantage greater than casual altruism is likely to overcome.

For more details on the theses and lemmas, read the blog post and its linked articles.

Our Final Invention available for preorder
James Barrat, a documentary filmmaker for National Geographic, Discovery, PBS, and other broadcasters, has written a wonderful new book about the intelligence explosion called Our Final Invention: Artificial Intelligence and the End of the Human Era. It will be released October 1st, and is available for pre-order on Amazon.
Here are some blurbs from people who have read an advance copy:
"A hard-hitting book about the most important topic of this century and possibly beyond — the issue of whether our species can survive. I wish it was science fiction but I know it's not."
—Jaan Tallinn, co-founder of Skype
"The compelling story of humanity's most critical challenge. A Silent Spring for the 21st Century."
—Michael Vassar, former MIRI president
"Our Final Invention is a thrilling detective story, and also the best book yet written on the most important problem of the 21st century."
—Luke Muehlhauser, MIRI executive director
"An important and disturbing book."
—Huw Price, co-founder, Cambridge University Center for the Study of Existential Risk

MIRI Needs Advisors
MIRI currently has a few dozen volunteer advisors on a wide range of subjects, but we need more! We're especially hoping for additional advisors in mathematical logic, theoretical computer science, artificial intelligence, economics, and game theory.
If you sign up, we will occasionally ask you questions, or send you early drafts of upcoming writings for feedback.
We don't always want technical advice ("Well, you can do that with a relativized arithmetical hierarchy…"); often, we just want to understand how different groups of experts respond to our writing ("The tone of this paragraph rubs me the wrong way because…").
Even if you don't have much time to help, please sign up! We will of course respect your own limits on availability.

Featured Volunteer – Florian Blumm
Florian Bumm has been one of our most active translators. Florian translates materials from English into his native tongue, German. Historically a software engineer, he is now on a traveling vacation in Bolivia progressively extended by remote contract labor, which he has found conducive to his volunteering for MIRI. After leaving a position as a Java engineer for a financial services company, he has decided that he would rather contribute directly to a cause of some sort, and has determined that there is nothing more important than mitigating existential risks from artificial intelligence.
Thanks, Florian!
To join Florian and dozens of other volunteers, visit MIRIvolunteers.org.

The post MIRI May Newsletter: Intelligence Explosion Microeconomics and Other Publications appeared first on Machine Intelligence Research Institute.


New Transcript: Yudkowsky and Aaronson

In When Will AI Be Created?, I referred to a bloggingheads.tv conversation between Eliezer Yudkowsky and Scott Aaronson. A transcript of that dialogue is now available, thanks to MIRI volunteers Ethan Dickinson, Daniel Kokotajlo, and Rick Schwall.
See also the transcript for a bloggingheads.tv conversation between Eliezer Yudkowsky and Massimo Pigliucci.
To join these volunteers in assisting our cause, visit MIRIvolunteers.org!
The post New Transcript: Yudkowsky and Aaronson appeared first on Machine Intelligence Research Institute.

New Transcript: Yudkowsky and Aaronson


Sign up for DAGGRE to improve science & technology forecasting

In When Will AI Be Created?, I named four methods that might improve our forecasts of AI and other important technologies. Two of these methods were explicit quantification and leveraging aggregation, as exemplified by IARPA's ACE program, which aims to "dramatically enhance the accuracy, precision, and timeliness of… forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many… analysts."
GMU's DAGGRE program, one of five teams participating in ACE, recently announced a transition from geopolitical forecasting to science & technology forecasting:
DAGGRE will continue, but it will transition from geo-political forecasting to science and technology (S&T) forecasting to better use its combinatorial capabilities. We will have a brand new shiny, friendly and informative interface co-designed by Inkling Markets, opportunities for you to provide your own forecasting questions and more!
Another exciting development is that our S&T forecasting prediction market will be open to everyone in the world who is at least eighteen years of age. We're going global!
If you want to help improve humanity's ability to forecast important technological developments like AI, please register for DAGGRE's new S&T prediction website here.
I did.
The post Sign up for DAGGRE to improve science & technology forecasting appeared first on Machine Intelligence Research Institute.


Four Articles Added to Research Page

Four older articles have been added to our research page.
The first is the early draft of Christiano et al.'s "Definability of 'Truth' in Probabilistic Logic" previously discussed here and here. The draft was last updated on April 2, 2013.
The second paper is a cleaned-up version of an article originally published in December 2012 by Luke Muehlhauser and Chris Williamson to Less Wrong: "in December 2012 by Luke Muehlhauser and Chris Williamson to Less Wrong: "Ideal Advisor Theories and Personal CEV."
The third and fourth papers were originally published by Bill Hibbard in the AGI 2012 Conference Proceedings: "AGI 2012 Conference Proceedings: "Avoiding Unintended AI Behaviors" and "Decision Support for Safe AI Design." Hibbard wrote these articles before he became a MIRI research associate, but he gave us permission to include them on our research page because (1) he became a MIRI research associate during the AGI-12 conference at which the articles were published, (2) the articles were partly inspired by a public dialogue with Luke Muehlhauser, and (3) the articles build on MIRI's paper "public dialogue with Luke Muehlhauser, and (3) the articles build on MIRI's paper "Intelligence Explosion and Machine Ethics."
As mentioned in our December 2012 newsletter, "Avoiding Unintended AI Behaviors" was awarded MIRI's $1000 Turing Prize for Best AGI Safety Paper. The prize was awarded in honor of Alan Turing, who not only discovered some of the key ideas of machine intelligence, but also grasped its importance, writing that "…it seems probable that once [human-level machine thinking] has started, it would not take long to outstrip our feeble powers… At some stage therefore we should have to expect the machines to take control…"
The post Four Articles Added to Research Page appeared first on Machine Intelligence Research Institute.

Four Articles Added to Research Page


When Will AI Be Created?

Strong AI appears to be the topic of the week. Kevin Drum at Mother Jones thinks AIs will be as smart as humans by 2040. Karl Smith at Forbes and "M.S." at The Economist seem to roughly concur with Drum on this timeline. Moshe Vardi, the editor-in-chief of the world's most-read computer science magazine, predicts that "by 2045 machines will be able to do if not any work that humans can do, then a very significant fraction of the work that humans can do."
But predicting AI is more difficult than many people think.
To explore these difficulties, let's start with a 2009 bloggingheads.tv conversation between MIRI researcher Eliezer Yudkowsky and MIT computer scientist Scott Aaronson, author of the excellent Quantum Computing Since Democritus. Early in that dialogue, Yudkowsky asked:
It seems pretty obvious to me that at some point in [one to ten decades] we're going to build an AI smart enough to improve itself, and [it will] "foom" upward in intelligence, and by the time it exhausts available avenues for improvement it will be a "superintelligence" [relative] to us. Do you feel this is obvious?
Aaronson replied:
The idea that we could build computers that are smarter than us… and that those computers could build still smarter computers… until we reach the physical limits of what kind of intelligence is possible… that we could build things that are to us as we are to ants — all of this is compatible with the laws of physics… and I can't find a reason of principle that it couldn't eventually come to pass…
The main thing we disagree about is the time scale… a few thousand years [before AI] seems more reasonable to me.
Those two estimates — several decades vs. "a few thousand years" — have wildly different policy implications.
If there's a good chance that AI will replace humans at the steering wheel of history in the next several decades, then we'd better put our gloves on and get to work making sure that this event has a positive rather than negative impact. But if we can be pretty confident that AI is thousands of years away, then we needn't worry about AI for now, and we should focus on other global priorities. Thus it appears that "When will AI be created?" is a question with high value of information for our species.
Let's take a moment to review the forecasting work that has been done, and see what conclusions we might draw about when AI will likely be created.

The challenge of forecasting AI
Expert elicitation
Maybe we can ask the experts? Astronomers are pretty good at predicting eclipses, even decades or centuries in advance. Technological development tends to be messier than astronomy, but maybe the experts can still give us a range of years during which we can expect AI to be built? This method is called expert elicitation.
Several people have surveyed experts working in AI or computer science about their AI timelines. Unfortunately, most of these surveys suffer from rather strong sampling bias, and thus aren't very helpful for our purposes.1
Should we expect experts to be good at predicting AI, anyway? As Armstrong & Sotala (2012) point out, decades of research on expert performance2 suggest that predicting the first creation of AI is precisely the kind of task on which we should expect experts to show poor performance — e.g. because feedback is unavailable and the input stimuli are dynamic rather than static. Muehlhauser & Salamon (2013) add, "If you have a gut feeling about when AI will be created, it is probably wrong."
That said, the experts surveyed in Michie (1973) — a more representative sample than in other surveys3 — did pretty well. When asked to estimate a timeline for "[computers] exhibiting intelligence at adult human level," the most common response was "More than 50 years." Assuming (as most people do) that AI will not arrive by 2023, these experts will have been correct.
Unfortunately, "more than 50 years" is a broad time frame that includes both "several decades from now" and "thousands of years from now." So we don't yet have any evidence that a representative survey of experts can predict AI within a few decades, and we have general reasons to suspect experts may not be capable of doing this kind of forecasting very well — although various aids (e.g. computational models; see below) may help them to improve their performance.
How else might we forecast when AI will be created?
Trend extrapolation
Many have tried to forecast the first creation of AI by extrapolating various trends. Like Kevin Drum, Vinge (1993) based his own predictions about AI on hardware trends (e.g. Moore's Law). But in a 2003 reprint of his article, Vinge noted the insufficiency of this reasoning: even if we acquire hardware sufficient for AI, we may not have the software problem solved.4 As Robin Hanson reminds us, "AI takes software, not just hardware."
Perhaps instead we could extrapolate trends in software progress?5 Some people estimate the time until AI by asking what proportion of human abilities today's software can match, and how quickly machines are "catching up."6 Unfortunately, it's not clear how to divide up the space of "human abilities," nor how much each ability matters. Moreover, software progress seems to come in fits and starts.7 With the possible exception of computer chess progress, I'm not aware of any trend in software progress as robust across multiple decades as Moore's Law is in computing hardware.
On the other hand, Tetlock (2005) points out that, at least in his large longitudinal database of pundit's predictions about politics, simple trend extrapolation is tough to beat. Consider one example from the field of AI: when David Levy asked 1989 World Computer Chess Championship participants when a chess program would defeat the human World Champion, their estimates tended to be inaccurately pessimistic,8 despite the fact that computer chess had shown regular and predictable progress for two decades by that time. Those who forecasted this event with naive trend extrapolation (e.g. Kurzweil 1990) got almost precisely the correct answer (1997).
Hence, it may be worth searching for a measure for which (a) progress is predictable enough to extrapolate, and for which (b) a given level of performance on that measure robustly implies the arrival of Strong AI. But to my knowledge, this has not yet been done, and it's not clear that trend extrapolation can tell us much about AI timelines until such an argument is made, and made well.
Disruptions
Worse, several events could significantly accelerate or decelerate our progress toward AI, and we don't know which of these events will occur, nor in what order. For example:

An end to Moore's Law. The "serial speed" version of Moore's Law broke down in 2004, requiring a leap to parallel processors, which raises substantial new difficulties for software developers (Fuller & Millett 2011). The most economically relevant formulation of Moore's law, computations per dollar, has been maintained thus far,9 but it remains unclear whether this will continue much longer (Mack 2011; Esmaeilzadeh et al. 2012).
Depletion of low-hanging fruit. Progress is not only a function of effort but also of the difficulty of the progress. Some fields see a pattern of increasing difficulty with each successive discovery (Arbesman 2011). AI may prove to be a field in which new progress requires far more effort than earlier progress. That is clearly the case for many parts of AI already, for example natural language processing (Davis 2012).
Societal collapse. Political, economic, technological, or natural disasters may cause a societal collapse during which progress in AI would be essentially stalled (Posner 2004; Bostrom and Ćirković 2008).
Disinclination. Chalmers (2010) and Hutter (2012a) think the most likely "speed bump" in our progress toward AI will be disinclination. As AI technologies become more powerful, humans may question whether it is wise to create machines more powerful than themselves.
A breakthrough in cognitive neuroscience. It is difficult, with today's tools, to infer the cognitive algorithms behind human intelligence (Trappenberg 2009). New tools and methods, however, might enable cognitive neuroscientists to decode how the human brain achieves its own intelligence, which might allow AI scientists to replicate that approach in silicon.
Human enhancement. Human enhancement technologies may make scientists more effective via cognitive enhancement pharmaceuticals (Bostrom and Sandberg 2009), brain-computer interfaces (Groß 2009), and genetic selection or engineering for cognitive enhancement.10
Quantum computing. Quantum computing has overcome some of its early hurdles (Rieffel and Polak 2011), but it remains difficult to predict whether quantum computing will contribute significantly to the development of machine intelligence. Progress in quantum computing depends on particularly unpredictable breakthroughs. Furthermore, it seems likely that even if built, a quantum computer would provide dramatic speedups only for specific applications (e.g. searching unsorted databases).
A tipping point in development incentives. The launch of Sputnik in 1957 demonstrated the possibility of space flight to the public. This event triggered a space race between the United States and the Soviet Union, and led to long-term funding for space projects from both governments. If there is a "Sputnik moment" for AI that makes it clear to the public and to governments that smarter-than-human AI is inevitable, a race to Strong AI may ensue, especially since the winner of the AI race might reap extraordinary economic, technological and geopolitical advantage.11

Great uncertainty
Given these considerations, I think the most appropriate stance on the question "When will AI be created?" is something like this:
We can't be confident AI will come in the next 30 years, and we can't be confident it'll take more than 100 years, and anyone who is confident of either claim is pretending to know too much.
How confident is "confident"? Let's say 70%. That is, I think it is unreasonable to be 70% confident that AI is fewer than 30 years away, and I also think it's unreasonable to be 70% confident that AI is more than 100 years away.
This statement admits my inability to predict AI, but it also constrains my probability distribution over "years of AI creation" quite a lot.
I think the considerations above justify these constraints on my probability distribution, but I haven't spelled out my reasoning in great detail. That would require more analysis than I can present here. But I hope I've at least summarized the basic considerations on this topic, and those with different probability distributions than mine can now build on my work here to try to justify them.
How to reduce our ignorance
But let us not be satisfied with a declaration of ignorance. Admitting our ignorance is an important step, but it is only the first step. Our next step should be to reduce our ignorance if we can, especially for high-value questions that have large strategic implications concerning the fate of our entire species.
How can we improve our long-term forecasting performance? Horowitz & Tetlock (2012), based on their own empirical research and prediction training, offer some advice on the subject:12

Explicit quantification: "The best way to become a better-calibrated appraiser of long-term futures is to get in the habit of making quantitative probability estimates that can be objectively scored for accuracy over long stretches of time. Explicit quantification enables explicit accuracy feedback, which enables learning."
Signposting the future: Thinking through specific scenarios can be useful if those scenarios "come with clear diagnostic signposts that policymakers can use to gauge whether they are moving toward or away from one scenario or another… Falsifiable hypotheses bring high-flying scenario abstractions back to Earth."13
Leveraging aggregation: "the average forecast is often more accurate than the vast majority of the individual forecasts that went into computing the average…. [Forecasters] should also get into the habit that some of the better forecasters in [an IARPA forecasting tournament called ACE] have gotten into: comparing their predictions to group averages, weighted-averaging algorithms, prediction markets, and financial markets." See Ungar et al. (2012) for some aggregation-leveraging results from the ACE tournament.

Many forecasting experts add that when making highly uncertain predictions, it usually helps to decompose the phenomena into many parts and make predictions about each of the parts.14 As Raiffa (1968) succinctly put it, our strategy should be to "decompose a complex problem into simpler problems, get one's thinking straight [on] these simpler problems, paste these analyses together with a logical glue, and come out with a program for action for the complex problem" (p. 271). MIRI's The Uncertain Future is a simple toy model of this kind, but more sophisticated computational models — like those successfully used in climate change modeling (Allen et al. 2013) — could be produced, and integrated with other prediction techniques.
We should expect AI forecasting to be difficult, but we need not be as ignorant about AI timelines as we are today.
Acknowledgements
My thanks to Carl Shulman, Ernest Davis, Louie Helm, Scott Aaronson, and Jonah Sinick for their helpful feedback on this post.
First, Sandberg & Bostrom (2011) gathered the AI timeline predictions of 35 participants at a 2011 academic conference on human-level machine intelligence. Participants were asked by what year they thought there is a 10%, 50%, and 90% chance that AI will have been built, assuming that "no global catastrophe halts progress." Five of the 35 respondents expressed varying degrees of confidence that human-level AI would never be achieved. The median figures, calculated from the views of the other 30 respondents, were: 2028 for "10% chance," 2050 for "50% chance," and 2150 for "90% chance." Second, Baum et al. (2011) surveyed 21 participants at a 2009 academic conference on machine intelligence, and found estimates similar to those in Sandberg & Bostrom (2011). Third, Kruel (2012) has, as of May 7th, 2013, interviewed 34 people about AI timelines and risks via email, 33 of whom could be considered "experts" of one kind or another in AI or computer science (Richard Carrier is a historian). Of those 33 experts, 19 provided full, quantitative answers to Kruel's question about AI timelines: "Assuming beneficial political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans (or better, perhaps unevenly) at science, mathematics, engineering and programming?" For those 19 experts, the median estimates for 10%, 50%, and 90% were 2025, 2035, and 2070, respectively (spreadsheet here). Fourth, Bainbridge (2005), surveying participants of 3 conferences on "Nano-Bio-Info-Cogno" technological convergence, found a median estimate of 2085 for "the computing power and scientific knowledge will exist to build machines that are functionally equivalent to the human brain." However, the participants in these four surveys were disproportionately HLAI enthusiasts, and this introduces a significant sampling bias. The database of AI forecasts discussed in Armstrong & Sotala (2012) probably suffers from a similar problem: individuals who thought AI was imminent rather than distant were more likely to make public predictions of AI.Shanteau (1992); Kahneman and Klein (2009).Another survey was taken at the AI@50 conference in 2006. When participants were asked "When will computers be able to simulate every aspect of human intelligence?", 41% said "More than 50 years" and another 41% said "Never." Unfortunately, many of the survey participants were not AI experts but instead college students who were attending the conference. Moreover, the phrasing of the question may have introduced a bias. The "Never" answer may have been given as often as it was because some participants took "every aspect of human intelligence" to include consciousness, and many people have philosophical objections to the idea that machines could be conscious. Had they instead been asked "When will AIs replace humans in almost all jobs?", I suspect the "Never" answer would have been far less common. As for myself, I don't accept any of the in-principle objections to the possibility of AI. For replies to the most common of these objections, see Chalmers (1996), ch. 9, and Chalmers (2012).Though, Muehlhauser & Salamon (2013) point out that "Hardware extrapolation may be a more useful method in a context where the intelligence software is already written: whole brain emulation [WBE]. Because WBE seems to rely mostly on scaling up existing technologies like microscopy and large-scale cortical simulation, WBE may be largely an "engineering" problem, and thus the time of its arrival may be more predictable than is the case for other kinds of AI." However, it is especially difficult to forecast WBE while we do not even have a proof of concept via a simple organism like C. elegans (David Dalrymple is working on this). Moreover, much progress in neuroscience will be required (Sandberg & Bostrom 2011), and such progress is probably less predictable than hardware extrapolation.I'm not sure what a general measure of software progress would look like, though we can certainly identify local examples of software progress. For example, Holdren et al. (2010) notes: "in many areas, performance gains due to improvements in algorithms have vastly exceeded even the dramatic performance gains due to increased processor speed… [For example] Martin Grötschel…, an expert in optimization, observes that a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later – in 2003 – this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms! Grötschel also cites an algorithmic improvement of roughly 30,000 for mixed integer programming between 1991 and 2008." Muehlhauser & Salamon (2013) give another example: "For example, IBM's Deep Blue played chess at the level of world champion Garry Kasparov in 1997 using about 1.5 trillion instructions per second (TIPS), but a program called Deep Junior did it in 2003 using only 0.015 TIPS. Thus, the computational efficiency of the chess algorithms increased by a factor of 100 in only six years (Richards and Shaw 2004)." A third example is Setty et al. (2012), which improved the efficiency of a probabilistically checkable proof method by 20 orders of magnitude with a single breakthrough. On the other hand, one can easily find examples of very slow progress, too (Davis 2012).For example, see Good (1970).As I wrote earlier: "Increases in computing power are pretty predictable, but for AI you probably need fundamental mathematical insights, and it's damn hard to predict those. In 1900, David Hilbert posed 23 unsolved problems in mathematics. Imagine trying to predict when those would be solved." Some of these problems were solved quickly, some of them required several decades to solve, and many of them remain unsolved. Even the order in which Hilbert's problems would be solved was hard to predict. According to Erdős & Graham (1980), p. 7, "Hilbert lectured in the early 1920's on problems in mathematics and said something like this: probably all of us will see the proof of the Riemann hypothesis, some of us… will see the proof of Fermat's last theorem, but none of us will see the proof that √2√2 is transcendental." In fact, these results came in the reverse order: the last was proved by Kusmin a few years later, Fermat's last theorem was proved by Wiles in 1994, and the Riemann hypothesis still has not been proved or disproved.According to Levy & Newborn (1991), one participant guessed the correct year (1997), thirteen participants guessed years from <PHONE>, twenty-eight participants guessed years from <PHONE>, and one participant guessed "Never." Of the twenty-eight who guessed years from <PHONE>, eleven guessed year 2010 or later.As Fuller & Millett (2011, p. 81) note, "When we talk about scaling computing performance, we implicitly mean to increase the computing performance that we can buy for each dollar we spend." Most of us don't really care whether our new computer has more transistors or some other structure; we just want it to do more stuff, more cheaply. Kurzweil (2012), ch. 10, footnote 10 shows "calculations per second per $1,000" growing exponentially from 1900 through 2010, including several data points after the serial speed version of Moore's Law broke down in 2004. The continuation of this trend is confirmed by "instructions per second per dollar" data for 2006-2011, gathered from Intel and other sources by Chris Hallquist (spreadsheet here). Thus it seems that the computations per dollar form of Moore's Law has continued unabated, at least for now.One possible breakthrough here may be iterated embryo selection. See Miller (2012, ch. 9) for more details.It is interesting, however, that the United States did not pursue extraordinary economic, technological and geopolitical advantage in the period during which it was the sole possessor of nuclear weapons. Also, it is worth noting that violence and aggression have steadily declined throughout human history (Pinker 2012).Tetlock (2010) adds another recommendation: "adversarial collaboration" (Mellers et al. 2001). Tetlock explains: "The core idea is simple: rival epistemic and political camps would nominate experts to come together to reach agreements on how they disagree on North Korea or deficit reduction or global warming — and then would figure out how to resolve at least a subset of their factual disputes. The disputants would need to specify, ex ante, how much belief change each side would 'owe' the other if various agreed-upon empirical tests were to work out one way or the other. When adversarial collaboration works as intended, it shifts the epistemic incentives from favoring cognitive hubris (generating as many reasons why one's own side is right and the other is wrong) and toward modesty (taking seriously the possibility that some of the other side's objections might have some validity). This is so because there is nothing like the prospect of imminent falsification to motivate pundits to start scaling back their more grandiose generalizations: 'I am not predicting that North Korea will become conciliatory in this time frame if we did x — I merely meant that they might become less confrontational in this wider time frame if we did x, y and z — and if there are no unexpected endogenous developments and no nasty exogenous shocks.'" Tetlock (2012), inspired by Gawande (2009), also tentatively recommends the use of checklists in forecasting: "The intelligence community has begun developing performance-appraisal checklists for analysts that nudge them in the direction of thinking more systematically about how they think. But it has yet — to our knowledge — taken the critical next step of checking the usefulness of the checklists against independent real-world performance criteria, such as the accuracy of current assessments and future projections. Our experience in the [ACE] IARPA forecasting tournament makes us cautiously optimistic that this next step is both feasible and desirable."But, let us not fool ourselves concerning the difficulty of this task. Good (1976) asserted that human-level performance in computer chess was a good signpost for human-level AI, writing that "a computer program of Grandmaster strength would bring us within an ace of [machine ultra-intelligence]." But of course this was not so. We may chuckle at this prediction today, but how obviously wrong was Good's prediction in 1976?E.g. Armstrong & Sotala (2012); MacGregor (2001); Lawrence et al. (2006).The post When Will AI Be Created? appeared first on Machine Intelligence Research Institute.

When Will AI Be Created?


Advise MIRI with Your Domain-Specific Expertise

MIRI currently has a few dozen volunteer advisors on a wide range of subjects, but we need more! If you'd like to help MIRI pursue its mission more efficiently, please sign up to be a MIRI advisor.
If you sign up, we will occasionally ask you questions, or send you early drafts of upcoming writings for feedback.
We don't always want technical advice ("Well, you can do that with a relativized arithmetical hierarchy…"); often, we just want to understand how different groups of experts respond to our writing ("The tone of this paragraph rubs me the wrong way because…").
At the moment, we are most in need of advisors on the following subjects:

Mathematical logic (especially computability theory and proof theory, more-especially provability logic, and most-especially the combination of logic and probabilism)
Theoretical computer science (especially computability theory, computational complexity theory, and AIXI)
Artificial intelligence (especially machine learning, agent architectures, and probabilistic graphical models)
Economics (especially endogenous growth theory and the economics of innovation)
Game theory (especially program equilibrium and normative decision theory)

Even if you don't have much time to help, please sign up! We will of course respect your own limits on availability.
The post Advise MIRI with Your Domain-Specific Expertise appeared first on Machine Intelligence Research Institute.


Five theses, two lemmas, and a couple of strategic implications

MIRI's primary concern about self-improving AI isn't so much that it might be created by 'bad' actors rather than 'good' actors in the global sphere; rather most of our concern is in remedying the situation in which no one knows at all how to create a self-modifying AI with known, stable preferences.  (This is why we see the main problem in terms of doing research and encouraging others to perform relevant research, rather than trying to stop 'bad' actors from creating AI.)
This, and a number of other basic strategic views, can be summed up as a consequence of 5 theses about purely factual questions about AI, and 2 lemmas we think are implied by them, as follows:
Intelligence explosion thesis. A sufficiently smart AI will be able to realize large, reinvestable cognitive returns from things it can do on a short timescale, like improving its own cognitive algorithms or purchasing/stealing lots of server time. The intelligence explosion will hit very high levels of intelligence before it runs out of things it can do on a short timescale. See: Chalmers (2010); Muehlhauser & Salamon (2013); Yudkowsky (2013).
Orthogonality thesis. Mind design space is huge enough to contain agents with almost any set of preferences, and such agents can be instrumentally rational about achieving those preferences, and have great computational power. For example, mind design space theoretically contains powerful, instrumentally rational agents which act as expected paperclip maximizers and always consequentialistically choose the option which leads to the greatest number of expected paperclips. See: Bostrom (2012); Armstrong (2013).
Convergent instrumental goals thesis. Most utility functions will generate a subset of instrumental goals which follow from most possible final goals. For example, if you want to build a galaxy full of happy sentient beings, you will need matter and energy, and the same is also true if you want to make paperclips. This thesis is why we're worried about very powerful entities even if they have no explicit dislike of us: "The AI does not love you, nor does it hate you, but you are made of atoms it can use for something else." Note though that by the Orthogonality Thesis you can always have an agent which explicitly, terminally prefers not to do any particular thing — an AI which does love you will not want to break you apart for spare atoms. See: Omohundro (2008); Bostrom (2012).
Complexity of value thesis. It takes a large chunk of Kolmogorov complexity to describe even idealized human preferences. That is, what we 'should' do  is a computationally complex mathematical object even after we take the limit of reflective equilibrium (judging your own thought processes) and other standard normative theories. A superintelligence with a randomly generated utility function would not do anything we see as worthwhile with the galaxy, because it is unlikely to accidentally hit on final preferences for having a diverse civilization of sentient beings leading interesting lives. See: Yudkowsky (2011); Muehlhauser & Helm (2013).
Fragility of value thesis. Getting a goal system 90% right does not give you 90% of the value, any more than correctly dialing 9 out of 10 digits of my phone number will connect you to somebody who's 90% similar to Eliezer Yudkowsky. There are multiple dimensions for which eliminating that dimension of value would eliminate almost all value from the future. For example an alien species which shared almost all of human value except that their parameter setting for "boredom" was much lower, might devote most of their computational power to replaying a single peak, optimal experience over and over again with slightly different pixel colors (or the equivalent thereof). Friendly AI is more like a satisficing threshold than something where we're trying to eke out successive 10% improvements. See: Yudkowsky (2009, 2011).
These five theses seem to imply two important lemmas:
Indirect normativity. Programming a self-improving machine intelligence to implement a grab-bag of things-that-seem-like-good-ideas will lead to a bad outcome, regardless of how good the apple pie and motherhood sounded. E.g., if you give the AI a final goal to "make people happy" it'll just turn people's pleasure centers up to maximum. "Indirectly normative" is Bostrom's term for an AI that calculates the 'right' thing to do via, e.g., looking at human beings and modeling their decision processes and idealizing those decision processes (e.g. what you would-want if you knew everything the AI knew and understood your own decision processes, reflective equilibria, ideal advisior theories, and so on), rather than being told a direct set of 'good ideas' by the programmers. Indirect normativity is how you deal with Complexity and Fragility. If you can succeed at indirect normativity, then small variances in essentially good intentions may not matter much — that is, if two different projects do indirect normativity correctly, but one project has 20% nicer and kinder researchers, we could still hope that the end results would be of around equal expected value. See: Muehlhauser & Helm (2013).
Large bounded extra difficulty of Friendliness. You can build a Friendly AI (by the Orthogonality Thesis), but you need a lot of work and cleverness to get the goal system right. Probably more importantly, the rest of the AI needs to meet a higher standard of cleanness in order for the goal system to remain invariant through a billion sequential self-modifications. Any sufficiently smart AI to do clean self-modification will tend to do so regardless, but the problem is that intelligence explosion might get started with AIs substantially less smart than that — for example, with AIs that rewrite themselves using genetic algorithms or other such means that don't preserve a set of consequentialist preferences. In this case, building a Friendly AI could mean that our AI has to be smarter about self-modification than the minimal AI that could undergo an intelligence explosion. See: Yudkowsky (2008) and Yudkowsky (2013).
These lemmas in turn have two major strategic implications:

We have a lot of work to do on things like indirect normativity and stable self-improvement. At this stage a lot of this work looks really foundational — that is, we can't describe how to do these things using infinite computing power, let alone finite computing power.  We should get started on this work as early as possible, since basic research often takes a lot of time.
There needs to be a Friendly AI project that has some sort of boost over competing projects which don't live up to a (very) high standard of Friendly AI work — a project which can successfully build a stable-goal-system self-improving AI, before a less-well-funded project hacks together a much sloppier self-improving AI.  Giant supercomputers may be less important to this than being able to bring together the smartest researchers (see the open question posed in Yudkowsky 2013) but the required advantage cannot be left up to chance.  Leaving things to default means that projects less careful about self-modification would have an advantage greater than casual altruism is likely to overcome.

The post Five theses, two lemmas, and a couple of strategic implications appeared first on Machine Intelligence Research Institute.