A fond farewell and a new Executive Director

Dear friends and supporters of MIRI,
I have some important news to share with you about the future of MIRI.
Given my passion for doing research, I’m excited to have accepted a research position at GiveWell. Like MIRI, GiveWell is an excellent cultural fit for me, and I believe they’re doing important work. I look forward to joining their team on June 1st. I’m also happy to report that I will be leaving MIRI in capable leadership hands.
Back in 2011, when MIRI’s Board of Directors asked me to take the Executive Director role, I was reluctant to leave the research position I held at the time. But I also wanted to do what best served MIRI’s mission. Looking back at the past three years, I’m proud of what the MIRI team has accomplished during my tenure as Executive Director. We’ve built a solid foundation, and our research program has picked up significant momentum. MIRI will continue to thrive as I transition out of my leadership role.
My enthusiasm for MIRI’s work remains as strong as ever, and I look forward to supporting MIRI going forward, both financially and as a close advisor. I’ll also continue to write about the future of AI on my personal blog.
Nate Soares will be stepping into the Executive Director role upon my departure, with unanimous support from myself and the rest of the Board.
Nate was our top choice for many reasons. During the past year at MIRI, Nate has demonstrated his commitment to the mission, his technical abilities, his strong work ethic, his ability to rapidly acquire new skills, his ability to work well with others, his ability to communicate clearly, his ability to think through big-picture strategic issues, and other aspects of executive capability.
During the transition, I’ll be sharing with Nate everything I think I’ve learned in the past three years about running an effective research institute, and I look forward to seeing where he leads MIRI next.
MIRI continues to seek additional research and executive capacity, and our need for both will only grow as I depart and as Nate transitions from a research role to the Executive Director role. If you are a math or computer science researcher, or if you have significant executive experience, and you are interested in participating in MIRI’s vital and significant research effort, please apply here.
The post A fond farewell and a new Executive Director appeared first on Machine Intelligence Research Institute.


May 2015 Newsletter

Research updates

Two new papers on reflective oracles and agents.
New articles on AI Impacts (last 2 months): Preliminary prices for human-level hardware, Trends in the cost of computing, Glial signaling, Scale of the human brain, Neuron firing rates in humans, Metabolic estimates of rate of cortical firing, Current FLOPS prices, The cost of TEPS, Kurzweil and Allen on the singularity, Wikipedia history of GFLOPS costs.

News updates

You can now earmark donations for AI Impacts, if you care more about AI forecasting research than superintelligence alignment research. But if you trust MIRI to allocate funds, please make normal, unrestricted donations.
CFAR will (conditionally) be running a three week summer program this July for MIRI, designed to increase participants' ability to do technical research into the superintelligence alignment problem. Details and application form are here.

Other updates

Musk and Gates on machine superintelligence. Gates has seconded Musk's recommendation of Superintelligence.

As always, please don't hesitate to let us know if you have any questions or comments.

Best,
Luke Muehlhauser
Executive Director

The post May 2015 Newsletter appeared first on Machine Intelligence Research Institute.

May 2015 Newsletter


New papers on reflective oracles and agents

We recently released two new papers on reflective oracles and agents.
The first is “Reflective oracles: A foundation for classical game theory,” by Benja Fallenstein, Jessica Taylor, and Paul Christiano.
Abstract:
Classical game theory treats players as special—a description of a game contains a full, explicit enumeration of all players—even though in the real world, “players” are no more fundamentally special than rocks or clouds. It isn’t trivial to find a decision-theoretic foundation for game theory in which an agent’s co-players are a non-distinguished part of the agent’s environment. Attempts to model both players and the environment as Turing machines, for example, fail for standard diagonalization reasons.
In this paper, we introduce a “reflective” type of oracle, which is able to answer questions about the outputs of oracle machines with access to the same oracle. These oracles avoid diagonalization by answering some queries randomly. We show that machines with access to a reflective oracle can be used to define rational agents using causal decision theory. These agents model their environment as a probabilistic oracle machine, which may contain other agents as a non-distinguished part.
We show that if such agents interact, they will play a Nash equilibrium, with the randomization in mixed strategies coming from the randomization in the oracle’s answers. This can be seen as providing a foundation for classical game theory in which players aren’t special.
The second paper develops these ideas in the context of Solomonoff induction and Marcus Hutter’s AIXI. It is “Reflective variants of Solomonoff induction and AIXI,” by Benja Fallenstein, Nate Soares, and Jessica Taylor.
Abstract:
Solomonoff induction and AIXI model their environment as an arbitrary Turing machine, but are themselves uncomputable. This fails to capture an essential property of real-world agents, which cannot be more powerful than the environment they are embedded in; for example, AIXI cannot accurately model game-theoretic scenarios in which its opponent is another instance of AIXI.
In this paper, we define reflective variants of Solomonoff induction and AIXI, which are able to reason about environments containing other, equally powerful reasoners. To do so, we replace Turing machines by probabilistic oracle machines (stochastic Turing machines with access to an oracle). We then use reflective oracles, which answer questions of the form, “is the probability that oracle machine M outputs 1 greater than p, when run on this same oracle?” Diagonalization can be avoided by allowing the oracle to answer randomly if this probability is equal to p; given this provision, reflective oracles can be shown to exist. We show that reflective Solomonoff induction and AIXI can themselves be implemented as oracle machines with access to a reflective oracle, making it possible for them to model environments that contain reasoners as powerful as themselves.
The post New papers on reflective oracles and agents appeared first on Machine Intelligence Research Institute.


April 2015 newsletter

Research updates

We've launched a new research forum, the Intelligent Agent Foundations Forum (IAFF), devoted solely to technical research on the superintelligence alignment challenge.
On IAFF, three "forum digest" posts summarize much of the work conducted on that forum prior to its public launch: on UDT, on reflective oracles, and on corrigibility-related work.
New report: “An Introduction to Löb’s Theorem in MIRI Research"
New interview: Bill Hibbard.
Recent AI control brainstorming by Stuart Armstrong.
Slides from Benja's talk: "Beneficial Smarter-than-human Intelligence: the Challenges and the Path Forward"

News updates

We've released Yudkowsky's Less Wrong Sequences in ebook form, as Rationality: From AI to Zombies. Paper versions should be available later this year.
MIRI's 2014 annual review.

Other news

The Center for the Study of Existential Risk at the University of Cambridge is hiring four new research associates to work on their research project, "Towards a Science of Extreme Technological Risk."
The Future of Humanity Institute at the University of Oxford is hiring one researcher to work on the long-term AI control challenge.
The Future of Life Institute now has a News page.
Smarter Than Us and related books were recently reviewed in Financial Times.

As always, please don't hesitate to let us know if you have any questions or comments.

Best,
Luke Muehlhauser
Executive Director

The post April 2015 newsletter appeared first on Machine Intelligence Research Institute.

April 2015 newsletter


Recent AI control brainstorming by Stuart Armstrong

MIRI recently sponsored Oxford researcher Stuart Armstrong to take a solitary retreat and brainstorm new ideas for AI control. This brainstorming generated 16 new control ideas, of varying usefulness and polish. During the past month, he has described each new idea, and linked those descriptions from his index post: New(ish) AI control ideas.
He also named each AI control idea, and then drew a picture to represent (very roughly) how the new ideas related to each other. In the picture below, an arrow Y→X can mean “X depends on Y”, “Y is useful for X”, “X complements Y on this problem” or “Y inspires X.” The underlined ideas are the ones Stuart currently judges to be most important or developed.

Previously, Stuart developed the AI control idea of utility indifference, which plays a role in MIRI’s paper Corrigibility (Stuart is a co-author). He also developed anthropic decision theory and some ideas for reduced impact AI and oracle AI. He has contributed to the strategy and forecasting challenges of ensuring good outcomes from advanced AI, e.g. in Racing to the Precipice and How We’re Predicting AI — or Failing To. MIRI previously contracted him to write a short book introducing the superintelligence control challenge to a popular audience, Smarter Than Us.
The post Recent AI control brainstorming by Stuart Armstrong appeared first on Machine Intelligence Research Institute.


2014 in review

It’s time for my review of MIRI in 2014.1 A post about our next strategic plan will follow in the next couple months, and I’ve included some details about ongoing projects at the end of this review.

2014 Summary
Since early 2013, MIRI’s core goal has been to help create a new field of research devoted to the technical challenges of getting good outcomes from future AI agents with highly general capabilities, including the capability to recursively self-improve.2
Launching a new field has been a team effort. In 2013, MIRI decided to focus on its comparative advantage in defining open problems and making technical progress on them. We’ve been fortunate to coordinate with other actors in this space — FHI, CSER, FLI, and others — who have leveraged their comparative advantages in conducting public outreach, building coalitions, pitching the field to grantmakers, interfacing with policymakers, and more.3
MIRI began 2014 with several open problems identified, and with some progress made toward solving them, but with very few people available to do the work. Hence, most of our research program effort in 2014 was aimed at attracting new researchers to the field and making it easier for them to learn the material and contribute. This was the primary motivation for our new technical agenda overview, the MIRIx program, our new research guide, and more (see below). Nick Bostrom’s Superintelligence was also quite helpful for explaining why this field of research should exist in the first place.
Today the field is much larger and healthier than it was at the beginning of 2014. MIRI now has four full-time technical researchers instead of just one. Around 85 people have attended one or more MIRIx workshops. There are so many promising researchers who have expressed interest in our technical research that ~25 of them have already confirmed interest and availability to attend a MIRI introductory workshop this summer, and this mostly doesn’t include people who have attended past MIRI workshops, nor have we sent out all the invites yet. Moreover, there are now several researchers we know who are plausible MIRI hires in the next 1-2 years.
I am extremely grateful to MIRI’s donors, without whom this progress would have been impossible.
The rest of this post provides a more detailed summary of our activities in 2014.

Overview of 2014 activities

Technical research: We hired 2 new research fellows, launched the MIRIx program, hosted many visiting researchers, and released 14 technical papers/reports, including our new technical agenda overview.
Strategic research: We published 15 analyses, 5 papers, and 54 expert interviews.
Outreach: We organized some talks and media stories, and released Smarter Than Us. Yudkowsky also continued writing Harry Potter and the Methods of Rationality, which has proven to be a surprisingly effective outreach tool for MIRI’s work.
Fundraising: We raised $1,237,557 in contributions in 2014. This is slightly less than we raised in 2013, but only because 2013’s numbers include a one-time, outlier donation of ~$525,000.
Operations: We made many improvements to MIRI’s organizational efficiency and robustness.

2014 Technical Research
Two of the top three goals in our mid-2014 strategic plan were to (1) increase our technical research output and (2) invest heavily in recruiting additional technical researchers (via a prospecting -> prospect development -> hiring funnel). The third goal concerned fundraising; see below.
As for (1): this past year we released 14 technical papers/reports4 and gave a few technical talks for academic audiences. To give our staff researchers more time to write up existing results, and to focus more on recruiting, we held only one research workshop in 2014.
As for (2), in 2014 we:

Published our new research agenda overview, which makes it much easier for newcomers to understand what we’re doing and why.
Hired two new full-time research fellows. Benja Fallenstein and Nate Soares joined in April 2014. (Patrick LaVictoire joined MIRI in March 2015.)
Launched our MIRIx program, now with 14 active groups around the world. This program allows mathematicians and computer scientists to spend time studying and discussing MIRI’s research agenda, and is a key component of our prospecting and development pipeline.
Published A Guide to MIRI’s Research, which guides students through the topics and papers they should study to become familiar with each area of MIRI’s research agenda.
Hosted several visiting researchers to work with us on MIRI’s research problems in Berkeley for a few days or weeks at a time.5
Co-sponsored the SPARC 2014 camp, which trains mathematically talented youth to apply their quantitative thinking skills to their lives and the world. SPARC didn’t teach participants about MIRI’s research, but it did teach participants about effective altruism, and brought them into contact with our social circles more generally. At least one SPARC graduate has subsequently expressed serious interest in working for MIRI in the future (but mostly, they are still too early in their studies to be considered).

In addition, some of our outreach activities (described below) double as researcher prospecting activities.
In my estimation, the growth of our technical research program in 2014 fell short of my earlier goals, mostly due to insufficient staff capacity to launch new recruitment initiatives. Thankfully, this situation improved in early 2015. We are still seeking to hire additional operations staff to help us grow our technical research program,6 but in the meantime we have met some of our immediate capacity needs by (for example) contracting MIRIx participant James Cook to help us better steward and grow the MIRIx program, and contracting Jesse Galef to help us organize our summer 2015 workshops.

2014 Strategic Research
As planned, in 2014 we decreased our output of strategic research.7 Even still we published a sizable amount of strategic work in 2014:

15 strategic analyses posted on MIRI’s blog and elsewhere.8
54 expert interviews on a wide range of topics.
5 papers/chapters on AI strategy topics.9

Nearly all of the interviews were begun in 2013 or early 2014, even if they were not finished and published until much later. Mid-way through 2014, we decided to de-prioritize expert interviews, due to apparent diminishing returns.
This level of strategic research output aligns closely with our earlier goals.

2014 Outreach
Our outreach efforts declined this year in favor of increased focus on our technical research. Our outreach efforts in 2014 included:

We released Smarter Than Us: The Rise of Machine Intelligence.
We gave four talks at the 2014 Effective Altruism Retreat and Effective Altruism Summit.
We hosted Nick Bostrom at UC Berkeley as part of his book tour for Superintelligence.
Eliezer Yudkowsky continued writing Harry Potter and the Methods of Rationality, which has proven to be a surprisingly effective outreach tool for MIRI’s work.10
We gave interviews for various media outlets.

This level of outreach output aligns closely with our earlier goals, except that we had planned to release the ebook version of The Sequences in 2014, and this release was delayed until March 2015.
Despite MIRI’s declining outreach efforts, public outreach about MIRI’s core concerns massively increased in 2014 due mostly to the efforts of others. FHI‘s Nick Bostrom released Superintelligence, which is now the best available summary of the problem MIRI exists to solve. Several prominent figures — in particular Stephen Hawking and Elon Musk — promoted long-term AI safety concerns to the media’s attention. In addition, two new organizations, CSER and FLI, did substantial outreach on this issue. Largely as a result of these efforts, Edge.org decided to make its widely-read 2015 Annual Question about long-term AI risks. Several of the respondents expressed views basically aligned with MIRI’s thinking on the issue, including Sam Harris, Stuart Russell, Jaan Tallinn, Max Tegmark, Steve Omohundro, Nick Bostrom, and of course MIRI’s own Eliezer Yudkowsky.

2014 Fundraising
Originally, we set a very ambitious fundraising goal for 2014. Shortly thereafter, we decided to focus on recruiting-related efforts rather than fundraising. So while our 2014 fundraising fell far short of our original (very ambitious) goal, I think the decision to focus on recruiting rather than fundraising in 2014 was the right choice.
In 2014 we raised $1,237,557 in contributions.11 Our largest sources of funding were:

~$400,000 from our summer matching challenge (this includes the $200,000 in matching funds from Jaan Tallinn, Edwin Evans, and Rick Schwall)
~$200,000 from our winter matching challenge (this includes $100,000 in matching funds from the Thiel Foundation)
~$175,000 from the one-day SV Gives fundraiser, ~$63,000 of which was from prizes and matching funds from donors who wouldn’t normally contribute to MIRI
~$150,000 from the Thiel Foundation (in addition to the $100,000 in matching funds for the winter matching challenge)
~$105,000 from Jed McCaleb.12

It is difficult to meaningfully compare MIRI’s 2013 and 2014 fundraising income to MIRI’s fundraising income in earlier years, because MIRI was a fairly different organization in 2012 and earlier.13 But I’ll show the comparison anyway:

Total donations were lower in 2014 than in 2013, but this is due to a one-time outlier donation in 2013 from Jed McCaleb, who was then a new donor. (By the way, this one donation allowed our research program to jump forward more quickly than I had originally been planning at the time.) If we set aside McCaleb’s large 2013 and 2014 gifts, MIRI’s fundraising grew slightly from 2013 to 2014.

New donor growth was strong in 2014, though this mostly came from small donations made during the SV Gives fundraiser. A significant portion of growth in returning donors can also be attributed to lapsed donors making small contributions during the SV Gives fundraiser.

This graph shows how much of our support during the past few years came from small, mid-sized, large, and very large donors. My understanding is that the distributions shown for 2012, 2013, and 2014 are fairly typical of non-profits our size. (Again, the green bar is taller in 2013 than in 2014 due to Jed McCaleb’s outlier 2013 donation.)

2014 Operations
Building on our 2013 organizational improvements, our operational efficiency and robustness improved substantially throughout 2014. Operations-related tasks, including the operational processes specific to our research program, now take up a smaller fraction of staff time than before, which has allowed us to divert more capacity to growing our research program. We also implemented many new policies and services that make MIRI more robust in the face of staff turnover, cyberattack, fluctuations in income, etc. I won’t go into much detail on operations in this post, but we’re typically happy to share what we’ve learned about running an efficient and robust organization when someone asks us to.

Coming Soon
Our next strategic plan post won’t be ready for another month or two, but of course we already have many projects in motion. Here’s what you can expect from MIRI over the next few months:

We have several technical reports and conference papers nearing completion.
We are running a series of workshops this summer. Many of the most promising people who applied to come to future workshops, or who are showing promise in MIRIx groups around the world, are being invited.
We are beginning to pay particularly productive MIRIx participants for part-time remote research on problems in MIRI’s technical agenda.
We are co-organizing a decision theory conference at Cambridge University in May.
Once again we are sponsoring SPARC‘s summer camp for mathematically talented high-schoolers.
We are “putting the finishing touches” on two large pieces of strategy research conducted in 2014. We will also finish running the Superintelligence reading group and then assemble the resulting Superintelligence reading guide. We will also contribute additional articles to AI Impacts, until our earmarked funding for that work runs out.14

Stay tuned for our next strategic plan, which will contain more detail about our planned programs.
This year’s annual review is shorter than last year’s 5-part review of 2013, in part because 2013 was an unusually complicated focus-shifting year, and in part because, in retrospect, last year’s 5-part review simply took more effort to produce than it was worth. Also, because we recently finished switching to accrual accounting, I can now more easily provide annual reviews of each calendar year rather than of a March-through-February period. As such, this review of calendar year 2014 will overlap a bit with what was reported in the previous annual review (of March 2013 through February 2014).Clearly there are forecasting and political challenges as well, and there are technical challenges related to ensuring good outcomes from nearer-term AI systems, but MIRI has chosen to specialize in the technical challenges of aligning superintelligence with human interests. See also: Friendly AI research as effective altruism and Why MIRI?Obviously, the division of labor was more complex than I’ve described here. For example, FHI produced some technical research progress in 2014, and MIRI did some public outreach.These were, roughly in chronological order: BotWorld, Program Equilibrium in the Prisoner’s Dilemma via Löb’s Theorem, Problems of self-reference in self-improving space-time embedded intelligence, Loudness, Distributions allowing tiling of staged subjective EU maximizers, Non-omniscience, probabilistic inference, and metamathematics, Corrigibility, UDT with known search order, Aligning superintelligence with human interests, Toward idealized decision theory, Computable probability distributions which converge…, Tiling agents in causal graphs, Concept learning for safe autonomous AI. I’m also counting Rob Bensinger’s blog post sequence on naturalized induction as one semi-technical “report.” A few of the technical agenda overview’s supporting papers weren’t announced on our blog until 2015. These aren’t counted here. For comparison’s sake, we released 10 technical papers/reports in 2013, but 7 of these were uncommonly short technical reports immediately following our December 2013 workshop. The 10 technical papers/reports from 2013 are: Scientific induction in probabilistic metamathematics, Fallenstein’s monster, Recursively-defined logical theories are well-defined, Tiling agents for self-modifying AI, and the Löbian obstacle, The procrastination paradox, A comparison of decision algorithms on Newcomblike problems, Definability of truth in probabilistic logic, The 5-and-10 problem and the tiling agents formalism, Decreasing mathematical strength in one formalization of parametric polymorphism, and An infinitely descending sequence of sound theories each proving the next consistent.Visiting researchers in 2014 included Abram Demski, Scott Garrabrant, Nik Weaver, Nisan Stiennon, Vladimir Slepnev, Tsvi Benson-Tilsen, Danny Hintze, and Ilya Shpitser.Hiring additional development staff will help grow our research program by freeing up more of my own time for research program work, and hiring one or more additional executives will directly add new staff capacity directed toward growing our research program.This year, I have collapsed my previous categories of “strategic” and “expository” research into one category I simply call “strategic research.”These were, roughly in chronological order: How Big is the Field of Artificial Intelligence?, Robust Cooperation: A Case Study in Friendly AI Research, The world’s distribution of computation, Is my view contrarian?, Exponential and non-exponential trends in information technology, How to study superintelligence strategy, Tentative tips for people engaged in an exercise that involves some form of prediction or forecasting, Groundwork for AGI safety engineering, Loosemore on AI safety and attractors, AGI outcomes and civilizational competence, The Financial Times story on MIRI, Three misconceptions in Edge.org’s conversation on “The Myth of AI”, Two mistakes about the threat from artificial intelligence, Brooks and Searle on AI volition and timelines, Davis on AI capability and motivation.These were, roughly in chronological order: Embryo selection for cognitive enhancement, Why we need Friendly AI, The errors, insights, and lessons of famous AI predictions, The ethics of artificial intelligence, and Exploratory engineering in artificial intelligence.HPMoR has now finished, but it wasn’t finished yet in 2014.MIRI has some sources of funding besides contributions. For example, in 2014 our realized and unrealized gains, plus interest and dividends — but not including realized and unrealized gains for our cryptocurrency holdings, which are highly volatile — amounted to ~$97,000. We also made ~$7,000 from ebook sales, and ~$5,000 from Give for Free programs.This donation was made in Ripple, which we eventually sold.See “Comparison to past years” here. Also, during 2014 we switched to accrual accounting, which confuses the comparison to past years even further. Furthermore, the numbers in this section might not exactly match past published estimates, because every now and then we still find and correct old errors in our donor database. Finally, note that in-kind donations are not included in the numbers or graphs on this page.We are happy to support such strategic research given earmarked funding and low management overhead, but otherwise we are focusing on our technical research program.The post 2014 in review appeared first on Machine Intelligence Research Institute.

2014 in review


New report: “An Introduction to Löb’s Theorem in MIRI Research”

Today we publicly release a new technical report by Patrick LaVictoire, titled “An Introduction to Löb’s Theorem in MIRI Research.” The report’s introduction begins:
This expository note is devoted to answering the following question: why do many MIRI research papers cite a 1955 theorem of Martin Löb, and indeed, why does MIRI focus so heavily on mathematical logic? The short answer is that this theorem illustrates the basic kind of self-reference involved when an algorithm considers its own output as part of the universe, and it is thus germane to many kinds of research involving self-modifying agents, especially when formal verification is involved or when we want to cleanly prove things in model problems. For a longer answer, well, welcome!
I’ll assume you have some background doing mathematical proofs and writing computer programs, but I won’t assume any background in mathematical logic beyond knowing the usual logical operators, nor that you’ve even heard of Löb’s Theorem before.
If you’d like to discuss the article, please do so here.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New report: “An Introduction to Löb’s Theorem in MIRI Research” appeared first on Machine Intelligence Research Institute.


Introducing the Intelligent Agent Foundations Forum

Today we are proud to publicly launch the Intelligent Agent Foundations Forum (RSS), a forum devoted to technical discussion of the research problems outlined in MIRI’s technical agenda overview, along with similar research problems.
Patrick’s welcome post explains:
Broadly speaking, the topics of this forum concern the difficulties of value alignment- the problem of how to ensure that machine intelligences of various levels adequately understand and pursue the goals that their developers actually intended, rather than getting stuck on some proxy for the real goal or failing in other unexpected (and possibly dangerous) ways. As these failure modes are more devastating the farther we advance in building machine intelligences, MIRI’s goal is to work today on the foundations of goal systems and architectures that would work even when the machine intelligence has general creative problem-solving ability beyond that of its developers, and has the ability to modify itself or build successors.
The forum has been privately active for several months, so many interesting articles have already been posted, including:

Slepnev, Using modal fixed points to formalize logical causality
Fallenstein, Utility indifference and infinite improbability drives
Benson-Tilsen, Uniqueness of UDT for transparent universes
Christiano, Stable self-improvement as a research problem
Fallenstein, Predictors that don’t try to manipulate you(?)
Soares, Why conditioning on “the agent takes action a” isn’t enough
Fallenstein, An implementation of modal UDT
LaVictoire, Modeling goal stability in machine learning

Also see How to contribute.

The post Introducing the Intelligent Agent Foundations Forum appeared first on Machine Intelligence Research Institute.


Rationality: From AI to Zombies

Between 2006 and 2009, senior MIRI researcher Eliezer Yudkowsky wrote several hundred essays for the blogs Overcoming Bias and Less Wrong, collectively called “the Sequences.” With two days remaining until Yudkowsky concludes his other well-known rationality book, Harry Potter and the Methods of Rationality, we are releasing around 340 of his original blog posts as a series of six books, collected in one ebook volume under the title Rationality: From AI to Zombies.
Yudkowsky’s writings on rationality, which were previously scattered in a constellation of blog posts, have been cleaned up, organized, and collected together for the first time. This new version of the Sequences should serve as a more accessible long-form introduction to formative ideas behind MIRI, CFAR, and substantial parts of the rationalist and effective altruist communities.
While the books’ central focus is on applying probability theory and the sciences of mind to personal dilemmas and philosophical controversies, a considerable range of topics is covered. The six books explore rationality theory and applications from multiple angles:
I. Map and Territory. A lively introduction to the Bayesian conception of rational belief in cognitive science, and how it differs from other kinds of belief.
II. How to Actually Change Your Mind. A guide to overcoming confirmation bias and motivated cognition.
III. The Machine in the Ghost. A collection of essays on the general topic of minds, goals, and concepts.
IV. Mere Reality. Essays on science and the physical world, as they relate to rational inference.
V. Mere Goodness. A wide-ranging discussion of human values and ethics.
VI. Becoming Stronger. An autobiographical account of Yudkowsky’s philosophical mistakes, followed by a discussion of self-improvement and group rationality.
These essays are packaged together as a single electronic text, making it easier to investigate links between essays and search for keywords. The ebook is available on a pay-what-you-want basis (link), and on Amazon.com for $4.99 (link). In the coming months, we will also be releasing print versions of these six books, and Castify will be releasing the official audiobook version.
The post Rationality: From AI to Zombies appeared first on Machine Intelligence Research Institute.

Rationality: From AI to Zombies


Bill Hibbard on Ethical Artificial Intelligence

Bill Hibbard is an Emeritus Senior Scientist at the University of Wisconsin-Madison Space Science and Engineering Center, currently working on issues of AI safety and unintended behaviors. He has a BA in Mathematics and MS and PhD in Computer Sciences, all from the University of Wisconsin-Madison. He is the author of Super-Intelligent Machines, “Avoiding Unintended AI Behaviors,” “Decision Support for Safe AI Design,” and “Ethical Artificial Intelligence.” He is also principal author of the Vis5D, Cave5D, and VisAD open source visualization systems.

Luke Muehlhauser: You recently released a self-published book, Ethical Artificial Intelligence, which “combines several peer reviewed papers and new material to analyze the issues of ethical artificial intelligence.” Most of the book is devoted to the kind of exploratory engineering in AI that you and I described in a recent CACM article, such that you mathematically analyze the behavioral properties of classes of future AI agents, e.g. utility-maximizing agents.
Many AI scientists have the intuition that such early, exploratory work is very unlikely to pay off when we are so far from building an AGI, and don’t what an AGI will look like. For example, Michael Littman wrote:
…proposing specific mechanisms for combatting this amorphous threat [of AGI] is a bit like trying to engineer airbags before we’ve thought of the idea of cars. Safety has to be addressed in context and the context we’re talking about is still absurdly speculative.
How would you defend the value of the kind of work you do in Ethical Artificial Intelligence to Littman and others who share his skepticism?

Bill Hibbard: This is a good question, Luke. The analogy with cars is useful. Unlike engineering airbags before cars are even thought of, we are already working hard to develop AI and can anticipate various types of dangers.
When cars were first imagined, engineers probably knew that they would propel human bodies at speed and that they would need to carry some concentrated energy source. They knew from accidents with horse carriages that human bodies travelling at speed are liable to injury, and they knew that concentrated energy sources are liable to fire and explosion which may injure humans. This is analogous with what we know about future AI: that to serve humans well AI will have to know a lot about individual humans and that humans will not be able to monitor every individual action by AI. These properties of future AI pose dangers just as the basic properties of cars (propelling humans and carrying energy) pose dangers.
Early car designers could have anticipated that no individual car would carry all of humanity and thus car accidents would not pose existential threats to humanity. To the extent that cars threaten human safety and health via pollution, we have time to notice these threats and address them. With AI we can anticipate possible scenarios that do threaten humanity and that may be difficult to address once the AI system is operational. For example, as described in the first chapter of my book, the Omniscience AI, with a detailed model of human society and a goal of maximizing profits, threatens to control human society. However, AI poses much greater potential benefits than cars but also much greater dangers. This justifies greater effort to anticipate the dangers of AI.
It’s also worth noting that the abstract frameworks for exploratory engineering apply to any reasonable future AI design. As the second chapter of my book describes, any set of complete and transitive preferences among outcomes can be expressed by a utility function. If preferences are incomplete then there are outcomes A and B with no preference between them, so the AI agent cannot decide. If preferences are not transitive then there are outcomes A, B, and C such that A is preferred to B, B is preferred to C, and C is preferred to A. Again, the AI agent cannot decide. Thus our exploratory engineering can assume utility maximizing agents and cover all cases in which the AI agent can decide among outcomes.
Similarly, the dangers discussed in the book are generally applicable. Any design for powerful AI should explain how it will avoid the self-delusion problem described by Ring and Orseau, the problem of corrupting the reward generator as described by Hutter, and the problem of unintended instrumental actions as described by Omohundro (he called them basic AI drives).
The threat level from AI justifies addressing AI dangers now and with significant resources. And we are developing tools that enable us to analyze dangers of AI systems before we know the specifics of their designs.

Luke: Your book mostly discusses AGIs rather than contemporary narrow AI systems. Roughly when do you expect humanity will develop something resembling the kind of AGIs you have in mind? Or, what does your probability distribution over “Years to AGI” look like?

Bill: In my 2002 book, Super-Intelligent Machines, I wrote that “machines as intelligent as humans are possible and will exist within the next century or so.” (The publisher owns the copyright for my 2002 book, preventing me from giving electronic copies to people, and charges more than $100 per print copy. This largely explains my decision to put my current book on arxiv.org.) I like to say that we will get to human-level AI during the lives of children already born and in fact I can’t help looking at children with amazement, contemplating the events they will see.
In his 2005 book, The Singularity is Near, Ray Kurzweil predicted human-level AI by 2029. He has a good track record at technology prediction and I hope he is right: I was born in 1948 so have a good chance of living until 2029. He also predicted the singularity by 2045, which must include the kind of very powerful AI systems discussed in my recent book.
Although it has nowhere near human-level intelligence, the DeepMind Atari player is a general AI system in the sense that it has no foreknowledge of Atari games other than knowing that the goal is to get a high score. The remarkable success of this system increases my confidence that we will create true AGI systems. DeepMind was purchased by Google, and all the big IT companies are energetically developing AI. It is the combination of AGI techniques and access to hundreds of millions of human users that can create the scenario of the Omniscience AI described in Chapter 1 of my book. Similarly for government surveillance agencies, which have hundreds of millions of unwitting users.
In 1983 I made a wager that a computer would beat the world Go champion by 2013, and lost. In fact, most predictions about AI have been wrong. Thus we must bring some humility to our predictions about the dates of AI milestones.
Because Ray Kurzweil’s predictions are based on quantitative extrapolation from historical trends and because of his good track record, I generally defer to his predictions. If human- level AI will exist by 2029 and very capable and dangerous AGI systems will exist by 2045, it is urgent that we understand the social effects and dangers of AI as soon as possible.

Luke: Which section(s) of your book do you think are most likely to be intriguing to computer scientists, because they’ll learn something that seems novel (to them) and plausibly significant?

Bill: Thanks Luke. There are several sections of the book that may be interesting or useful.
At the Workshop on AI and Ethics at AAAI-15 there was some confusion about the generality of utilitarian ethics, based on the connotation that a utility function is defined as a linear sum of features or similar simple expression. However, as explained in Chapter 2 and in my first answer in this interview, more complex utility functions can express any set of complete and transitive preferences among outcomes. That is, if an agent always has a most preferred outcome among any finite set of outcomes, then that agent can be expressed as a utility-maximizing agent.
Chapter 4 goes into detail on the issues of agents whose environment models are finite stochastic programs. Most of the papers in the AGI community assume that environments are modeled by programs for universal Turing machines, with no limit on their memory use. I think that much can be added to what I wrote in Chapter 4, and hope that someone will do that.
The self-modeling agents of Chapter 8 are the formal framework analog of value learners such as the DeepMind Atari player, and their use as a formal framework is novel. Self-modeling agents have useful properties, such as the capability to value agent resource increases and a way to avoid the problem of the agent utility function being inconsistent with the agent’s definition. An example of this problem is what Armstrong refers to as “motivated value selection.” More generally, it is the problem of adding any “special” actions to a utility maximizing agent, where those special actions do not maximize the utility function. In motivated value selection, the special action is the agent evolving its utility function. A utility maximizing agent may choose an action of removing the special actions from its definition, as counter-productive to maximizing its utility function. Self-modeling agents include such evolutionary special actions in the definition of their value functions, and they learn a model of their value function which they use to choose their next action. Thus there is no
inconsistency. I think these ideas should be interesting to other computer scientists.
At the FLI conference in San Juan in January 2015 there was concern about the kind of technical AI risks described in Chapters 5 – 9 of my book, and concern about technological unemployment. However, there was not much concern about the dangers associated with:

Large AI servers connected to the electronic companions that will be carried by large numbers of people and the
ability of the human owners of those AI servers to manipulate society, and
A future world in which great wealth can buy increased intelligence and superior intelligence can generate increased wealth. This positive feedback loop will result in a power law distribution of intelligence as opposed to the current normal distribution of IQs with mean = 100 and standard deviation = 15.

These issues are discussed in Chapters 1 and 10 of my book. The Global Brain researchers study the way intelligence is exhibited by the network of humans; the change in distribution of intelligence of humans and machines who are nodes of the network will have profound effects on the nature of the Global Brain. Beyond computer scientists, I think the public needs to be aware of these issues.
Finally, I’d like to expand on my previous answer, specifically that the DeepMind Atari player is an example of general AI. In Chapter 1 of my book I describe how current AI systems have environment models that are designed by human engineers, whereas future AI systems will need to learn environment models that are too complex to be designed by human engineers. The DeepMind system does not use an environment model designed by engineers. It is “model-free” but the value function that it learns is just as complex as an environment model and in fact encodes an implicit environment model. Thus the DeepMind system is the first example of a future AI system with significant functionality.

Luke: Can elaborate what you mean by saying that “the self-modeling agents of Chapter 8 are the formal framework analog of value learners such as the DeepMind Atari player”? Are you saying that the formal work you do in chapter 8 has implications even for an extant system like the DeepMind Atari player, because they are sufficiently analogous?

Bill: To elaborate on what I mean by “the self-modeling agents of Chapter 8 are the formal framework analog of value learners such as the DeepMind Atari player,” self-modeling agents and value learners both learn a function v(ha) that produces the expected value of proposed action a after interaction history h (that is, h is a sequence of observations and actions; see my book for details). For the DeepMind Atari player, v(ha) is the expected game score after action a and h is restricted to the most recent observation (i.e., a game screen snapshot). Whereas the DeepMind system must be practically computable, the self-modeling agent framework is a purely mathematical definition. This framework is finitely computable but any practical implementation would have to use approximations. The book offers a few suggestions about computing techniques, but the discussion is not very deep.
Because extant systems such as the DeepMind Atari player are not yet close to human-level intelligence, there is no implication that this system should be subject to safety constraints. It is encouraging that the folks at DeepMind and at Vicarious are concerned about AI ethics, for two reasons: 1) They are likely to apply ethical requirements to their systems as they approach human-level, and 2) They are very smart and can probably add a lot to AI safety research.
Generally, research on safe and ethical AI complicates the task of creating AI by adding requirements. My book develops a three-argument utility function expressing human values which will be very complex to compute. Similarly for other components of the definition of self-modeling agents in the book.
I think there are implications the other way around. The self-modeling framework is based on statistical learning and the success of the DeepMind Atari player, the Vicarious captcha solver, IBM’s Watson, and other practical systems that use statistical learning techniques increases our confidence that these techniques can actually work for AI capability and safety.
Some researchers suggest that safe AI should rely on logical deduction rather than statistical learning. This idea offers greater possibility of proving safety properties of AI, but so far there are no compelling demonstrations of AI systems based on logical deduction (at least, none that I am aware of). Such demonstrations would add a lot of confidence in our ability to prove safety properties of AI systems.

Luke: Your 10th chapter considers the political aspects of advanced AI. What do you think can be done now to improve our chances of solving the political challenges of AI in the future? Sam Altman of YC has proposed various kinds of regulation — do you agree with his general thinking? What other ideas do you have?

Bill: The central point of my 2002 book was the need for public education about and control over above-human-level AI. The current public discussion by Stephen Hawking, Bill Gates, Elon Musk, Ray Kurzweil, and others about the dangers of AI is very healthy, as it educates the public. Similarly for the Singularity Summits organized by the Singularity Institute (MIRI’s predecessor), which I thought were the best thing the Singularity Institute did.
In the US people cannot own automatic weapons, guns of greater than .50 caliber, or explosives without a license. It would be absurd to license such things but to allow unregulated development of above-human-level AI. As the public is educated about AI, I think some form of regulation will be inevitable.
However, as they say, the devil will be in the details and humans will be unable to compete with future AI on details. Complex details will be AI’s forte. So formulating effective regulation will be a political challenge. The Glass-Steagal Act of 1933, regulating banking, was 37 pages long. The Dodd-Frank bill of 2010, also to regulate banking 77 years later, was 848 pages long. An army of lawyers drafted the bill, many employed to protect the interests of groups affected by the bill. The increasing complexity of laws reflects efforts by regulated entities to lighten the burden of regulation. The stakes in regulating AI will be huge and we can expect armies of lawyers, with the aid of the AI systems being regulated, to create very complex laws.
In the second chapter of my book, I conclude that ethical rules are inevitably ambiguous and base my proposed safe AI design on human values expressed in a utility function rather than rules. Consider the current case before the US Supreme Court to interpret the meaning of the words “established by the state” in the context of the 363,086 words of the Affordable Care Act. This is a good example of the ambiguity of rules. Once AI regulations become law, armies of lawyers, aided by AI, will be engaged in debates over their interpretation and application.
The best counterbalance to armies of lawyers creating complexity on any legal issue is a public educated about the issue and engaged in protecting their own interests. Automobile safety is a good example. This will also be the case with AI regulation. And, as discussed in the introductory section of Chapter 10, there is precedent for the compassionate intentions of some wealthy and powerful people and this may serve to counterbalance their interest in creating complexity.
Privacy regulations, which affect existing large IT systems employing AI, already exist in the US and even more so in Europe. However, many IT services depend on accurate models of users’ preferences. At the recent FLI conference in San Juan, I tried to make the point that a danger from AI will be that people will want the kind of close, personal relationship with AI systems that will enable intrusion and manipulation by AI. The Omniscience AI described in Chapter 1 of my book is an example. As an astute IT lawyer said at the FLI conference, the question of whether an IT innovation will be legal depends on whether it will be popular.
This brings us back to the need for public education about AI. For people to resist being seduced by the short term benefits of close relationships with AI, they need to understand the long term consequences. I think it is not realistic to prohibit close relationships between people and AI, but perhaps the public, if it understands the issues, can demand some regulation over the goals for which those relationships are exploited.
The final section of my Chapter 10 says that AI developers and testers should recognize that they are acting as agents for the future of humanity and that their designs and test results should be transparent to the public. The FLI open letter and Google’s panel on AI ethics are encouraging signs that AI developers do recognize their role as agents for future humanity. Also, DeepMind has been transparent about the technology of their Atari player, even making source code available for non-commercial purposes.
AI developers deserve to be rewarded for their success. On the other hand, people have a right to avoid losing control over their own lives to an all-powerful AI and its wealthy human owners. The problem is to find a way to achieve both of these goals.
Among current humans, with naturally evolved brains, IQ has a normal distribution. When brains are artifacts, their intelligence is likely to have a power law distribution. This is the pattern of distributions of sizes of other artifacts such as trucks, ships, buildings, and computers. The average human will not be able to understand or ever learn the languages used by the most intelligent minds. This may mean the end of any direct voice in public policy decisions for average humans – effectively the end of democracy. But if large AI systems are maximizing utility functions that account for the values of individual humans, that may take the place of direct democracy.
Chapters 6 – 8 of my book propose mathematical definitions for an AI design that does balance the values of individual humans. Chapter 10 suggests that this design may be modified to provide different weights to the values of different people, for example to reward those who develop AI systems. I must admit that the connection between the technical chapters of my book and Chapter 10, on politics, is weak. Political issues are just difficult. For example, the future will probably have multiple AI systems with conflicting utility functions and a power law distribution of intelligence. It is difficult to predict how such a society would function and how it would affect humans, and this unpredictability is a risk. Creating a world with a single powerful AI system also poses risks, and may be difficult to achieve.
Since my first paper about future AI in 2001, I have thought that the largest risks from AI are political rather than technical. We have an ethical obligation to educate the public about the future of AI, and an educated public is an essential element of finding a good outcome from AI.

Luke: Thanks, Bill!
The post Bill Hibbard on <em>Ethical Artificial Intelligence</em> appeared first on Machine Intelligence Research Institute.


Fallenstein talk for APS March Meeting 2015

MIRI researcher Benja Fallenstein recently delivered an invited talk at the March 2015 meeting of the American Physical Society in San Antonio, Texas. Her talk was one of four in a special session on artificial intelligence.
Fallenstein’s title was “Beneficial Smarter-than-human Intelligence: the Challenges and the Path Forward.” The slides are available here. Abstract:
Today, human-level machine intelligence is still in the domain of futurism, but there is every reason to expect that it will be developed eventually. A generally intelligent agent as smart or smarter than a human, and capable of improving itself further, would be a system we’d need to design for safety from the ground up: There is no reason to think that such an agent would be driven by human motivations like a lust for power; but almost any goals will be easier to meet with access to more resources, suggesting that most goals an agent might pursue, if they don’t explicitly include human welfare, would likely put its interests at odds with ours, by incentivizing it to try to acquire the physical resources currently being used by humanity. Moreover, since we might try to prevent this, such an agent would have an incentive to deceive its human operators about its true intentions, and to resist interventions to modify it to make it more aligned with humanity’s interests, making it difficult to test and debug its behavior. This suggests that in order to create a beneficial smarter-than-human agent, we will need to face three formidable challenges: How can we formally specify goals that are in fact beneficial? How can we create an agent that will reliably pursue the goals that we give it? And how can we ensure that this agent will not try to prevent us from modifying it if we find mistakes in its initial version? In order to become confident that such an agent behaves as intended, we will not only want to have a practical implementation that seems to meet these challenges, but to have a solid theoretical understanding of why it does so. In this talk, I will argue that even though human-level machine intelligence does not exist yet, there are foundational technical research questions in this area which we can and should begin to work on today. For example, probability theory provides a principled framework for representing uncertainty about the physical environment, which seems certain to be helpful to future work on beneficial smarter-than-human agents, but standard probability theory assumes omniscience about logical facts; no similar principled framework for representing uncertainty about the outputs of deterministic computations exists as yet, even though any smarter-than-human agent will certainly need to deal with uncertainty of this type. I will discuss this and other examples of ongoing foundational work.
Stuart Russell of UC Berkeley also gave a talk at this session, about the long-term future of AI.
The post Fallenstein talk for APS March Meeting 2015 appeared first on Machine Intelligence Research Institute.


March 2015 newsletter

Research updates

We’ve made significant updates to three pages: Research, All Publications, and A Guide to MIRI’s Research.
We’ve published an annotated bibliography and dedicated page for our new technical research agenda overview.
This new mailing list sends one email per new technical paper, and nothing else.
New analyses: Davis on AI capability and motivation, How AI timelines are estimated, The slow traversal of ‘human-level’, Multipolar research questions, At-least-human-level-at-human-cost AI, and three analyses of discontinuous technological progress (to investigate the plausibility of AI fast takeoff): Penicillin and syphilis, Discontinuous progress investigation, and What’s up with nuclear weapons?

News updates

MIRI’s paper on value learning was covered in some detail at Nautilus.
The Superintelligence reading group is in week 24, on morality models and “do what I mean.”

Other news

The Future of Life Institute has released a new survey of research questions for robust and beneficial AI.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post March 2015 newsletter appeared first on Machine Intelligence Research Institute.

March 2015 newsletter


Davis on AI capability and motivation

In a review of Superintelligence, NYU computer scientist Ernest Davis voices disagreement with a number of claims he attributes to Nick Bostrom: that “intelligence is a potentially infinite quantity with a well-defined, one-dimensional value,” that a superintelligent AI could “easily resist and outsmart the united efforts of eight billion people” and achieve “virtual omnipotence,” and that “though achieving intelligence is more or less easy, giving a computer an ethical point of view is really hard.”
These are all stronger than Bostrom’s actual claims. For example, Bostrom never characterizes building a generally intelligent machine as “easy.” Nor does he say that intelligence can be infinite or that it can produce “omnipotence.” Humans’ intelligence and accumulated knowledge gives us a decisive advantage over chimpanzees, even though our power is limited in important ways. An AI need not be magical or all-powerful in order to have the same kind of decisive advantage over humanity.
Still, Davis’ article is one of the more substantive critiques of MIRI’s core assumptions that I have seen, and he addresses several deep issues that directly bear on AI forecasting and strategy. I’ll sketch out a response to his points here.

Measuring an intelligence explosion
Davis writes that Bostrom assumes “that a large gain in intelligence would necessarily entail a correspondingly large increase in power.” This is again too strong. (Or it’s trivial, if we’re using the word “intelligence” to pick out a specific kind of power.)
Bostrom is interested in intelligence for its potential to solve practical problems and shape the future. If there are other kinds of intelligence, they’re presumably of less economic and strategic importance than the “cognitive superpowers” Bostrom describes in chapter 6. It is the potential power autonomous machines could exhibit that should primarily concern us from a safety standpoint, and “intelligence” seems as good a term as any for the kind of power that doesn’t depend on an agent’s physical strength or the particularities of its environment.
When it comes to Bostrom’s intelligence explosion thesis, I don’t think ‘does an increase in intelligence always yield a corresponding increase in power?’ gets at the heart of the issue. Consider David Chalmers’ version of the argument:
[I]t is not unreasonable to hold that we can create systems with greater programming ability than our own, and that systems with greater programming ability will be able to create systems with greater programming ability in turn. It is also not unreasonable to hold that programming ability will correlate with increases in various specific reasoning abilities. If so, we should expect that absent defeaters, the reasoning abilities in question will explode.
Here, there’s no explicit appeal to “intelligence,” which is replaced with programming ability plus an arbitrarily large number of “specific reasoning abilities.” Yet if anything I find this argument more plausible than Bostrom’s formulation. For that reason, I agree with Bostrom that the one-dimensional representation of intelligence is inessential and “one could, for example, instead represent a cognitive ability profile as a hypersurface in a multidimensional space” (p. 273).1
Moreover, the relevant question isn’t whether an increase in a self-improving AI’s general programming ability always yields a corresponding increase in its ability to improve its own programming ability. Nor is the question whether either of those abilities always correlates with the other cognitive capabilities Bostrom is interested in (“strategizing,” “social manipulation,” “hacking,” “technology research,” “economic productivity”). I’d instead say that the five core questions from Bostrom’s point of view are:

Is the first superintelligent AI likely to result from self-improving AI systems?
If so, how much of the AI’s self-improvement is likely to be driven by improvements to some cognitive capability (e.g., programming ability) that facilitates further enhancement of the capability in question?
Are improvements to this self-improving capability likely to accelerate, as early advances result in cascades of more rapid advances? Or will the self-improving capability repeatedly stall out, advancing in small fits and starts?
If self-improvement cascades are likely, are they also likely to result in improvements to other cognitive capabilities that we more directly care about?2 Or will those other cognitive capabilities lag far behind the capabilities that ‘explode’?
If an ability like ‘programming’ is likely to be self-reinforcing in an accelerating way, and is likely to foster accelerating improvements in other cognitive abilities, exactly how fast will those accelerations be? Are we talking about a gap of decades between the AI’s first self-reinforcing self-improvements and its attainment of superintelligence? Months? Hours?

Bostrom’s position on those questions — that a fast or moderate intelligence explosion is likely — at no point presupposes that “intelligence” is a well-defined scalar value one could do calculus with, except as a toy model for articulating various qualitative possibilities. When he writes out differential equations, Bostrom is careful to note that intelligence cannot be infinite, that one-dimensionality is a simplifying assumption, and that his equations are “intended for illustration only.”3
We should expect artificial general intelligence (AGI) to specialize in some domains and neglect others. Bostrom’s own analysis assumes that a recursively self-improving AI would tend to prioritize acquiring skills like electrical engineering over skills like impressionist painting, all else being equal.4 For that matter, present-day AI is already superhuman in some cognitive tasks (e.g., chess and mental arithmetic), yet subhuman in many others. Any attempt to quantify the ‘overall intelligence’ of Deep Blue or Google Maps will obscure some important skills and deficits of these systems.5 Still, using concepts like ‘intelligence’ or ‘power’ to express imprecise hypotheses is better than substituting precise metrics that overstate how much we currently know about general intelligence and the future of AI.6

Superintelligence superiority
In Superintelligence (pp. 59-61), Bostrom lists a variety of ways AGI may surpass humans in intelligence, owing to differences in hardware (speed and number of computational elements, internal communication speed, storage capacity, reliability, lifespan, and sensors) and software (editability, duplicability, goal coordination, memory sharing, and new modules, modalities, and algorithms). Davis grants that these may allow AGI to outperform humans, but expresses skepticism that this could give AGI a decisive advantage over humans. To paraphrase his argument:
Elephants’ larger brains don’t make them superintelligent relative to mice; squirrels’ speed doesn’t give them a decisive strategic advantage over turtles; and we don’t know enough about what makes Einstein smarter than the village idiot to make confident predictions about how easy it is to scale up from village idiot to Einstein, or from Einstein to super-Einstein. So there’s no particular reason to expect a self-improving AGI to be able to overpower humanity.
My response is that the capabilities Bostrom describes, like “speed superintelligence,” predict much larger gaps than the gaps we see between elephants and mice. Bostrom writes (in a footnote on p. 270):
At least a millionfold speedup compared to human brains is physically possible, as can be seen by considering the difference in speed and energy of relevant brain processes in comparison to more efficient information processing. The speed of light is more than a million times greater than that of neural transmission, synaptic spikes dissipate more than a million times more heat than is thermodynamically necessary, and current transistor frequencies are more than a million times faster than neuron spiking frequencies.
Davis objects that “all that running faster does is to save you time,” noting that a slower system could eventually perform all the feats of a faster one. But the ability to save time is exactly the kind of ability Bostrom is worried about. Even if one doubts that large improvements in collective or quality superintelligence are possible, a ‘mere’ speed advantage makes an enormous practical difference.
Imagine a small community of scientist AIs whose only advantages over human scientists stem from their hardware speed — they can interpret sensory information and assess hypotheses and policies a million times faster than a human. At that speed, an artificial agent could make ~115 years of intellectual progress in an hour, ~2700 years of progress in a day, and 250,000 years of progress in three months.
The effect of this speedup would be to telescope human history. Scientific and technological advances that would have taken us tens of thousands of years to reach can be ours by Tuesday. If we sent a human a thousand years into the past, equipped with all the 21st-century knowledge and technologies they wanted, they could conceivably achieve dominant levels of wealth and power in that time period. This gives us cause to worry about building machines that can rapidly accumulate millennia of experience over humans, even before we begin considering any potential advantages in memory, rationality, editability, etc.7
At least some of the routes to superintelligence described by Bostrom look orders of magnitude larger than the cognitive advantages Einstein has over a village idiot (or than an elephant has over a mouse). We can’t rule out the possibility that we’ll run into larger-than-expected obstacles when we attempt to build AGI, but we shouldn’t gamble on that possibility. Black swans happen, but superintelligent AI is a white swan, and white swans happen too.
Since Bostrom’s pathways to superintelligence don’t have a lot in common with the neurological differences we can observe in mammals, there is no special reason to expect the gap between smarter-than-human AI and humans to resemble the gap between elephants and mice. Bostrom’s pathways also look difficult to biologically evolve, which means that their absence in the natural world tells us little about their feasibility.
We have even less cause to expect, then, that the gap between advanced AI and humans will resemble the gap between Einstein and a median human. If a single generation of random genetic recombination can produce an Einstein, the planning and design abilities of human (and artificial) engineers should make much greater feats a possibility.

Delegating AI problems to the AI
Separately, Davis makes the claim that “developing an understanding of ethics as contemporary humans understand it is actually one of the easier problems facing AI”. Again, I’ll attempt to summarize his argument:
Morality doesn’t look particularly difficult, especially compared to, e.g., computer vision. Moreover, if we’re going to build AGI, we’re going to solve computer vision. Even if morality is as tough as vision, why assume one will be solved and not the other?
Here my response is that you probably don’t need to solve every AI problem to build an AGI. We may be able to cheat at vision, for example, by tasking a blind AGI with solving the problem for us. But it is much easier to observe whether an algorithm is making progress on solving visual puzzles than to observe whether one is making progress on ethical questions, where there is much more theoretical and object-level disagreement among humans, there is less incentive for artificial agents to move from partial solutions to complete solutions, and failures don’t necessarily reduce the system’s power.
Davis notes that a socially fluent superintelligence would need to be an expert moralist:
Bostrom refers to the AI’s ‘social manipulation superpowers’. But if an AI is to be a master manipulator, it will need a good understanding of what people consider moral; if it comes across as completely amoral, it will be at a very great disadvantage in manipulating people. […] If the AI can understand human morality, it is hard to see what is the technical difficulty in getting it to follow that morality.
This is similar to Richard Loosemore’s argument against AI safety research, which I’ve responded to in a blog post. My objection is that an AI could come to understand human morality without thereby becoming moral, just as a human can come to understand the motivations of a stranger without thereby acquiring those motivations.
Since we don’t understand our preferences in enough generality or detail to translate them into code, it would be nice to be able to delegate the bulk of this task to a superintelligence. But if we program it to hand us an answer to the morality problem, how will we know whether it is being honest with us? To trust an AGI’s advice about how to make it trustworthy, we’d need to already have solved enough of the problem ourselves to make the AGI a reliable advisor.
Taking Davis’ proposal as an example, we can imagine instilling behavioral prescriptions into the AI by programming it to model Gandhi’s preferences and do what Gandhi would want it to. But if we try to implement this idea in code before building seed AI, we’re stuck with our own fallible attempts to operationalize concepts like ‘Gandhi’ and ‘preference;’ and if we try to implement it after, recruiting the AI to solve the problem, we’ll need to have already instilled some easier-to-program safeguards into it. What makes AGI safety research novel and difficult is our lack of understanding of how to initiate this bootstrapping process with any confidence.
The idea of using the deceased for value learning is interesting. Bostrom endorses the generalized version of this approach when he says that it is critical for value learning that the locus of value be “an object at a particular time” (p. 193). However, this approach may still admit of non-obvious loopholes, and it may still be too complicated for us to directly implement without recourse to an AGI. If so, it will need to be paired with solutions to the problems of corrigibility and stability in self-modifying AI, just as “shutdown buttons” and other tripwire solutions will.
From Bostrom’s perspective, what makes advanced AI a game-changer is first and foremost its capacity to meaningfully contribute to AI research. The vision problem may be one of many areas where we can outsource sophisticated AGI problems to AGI, or to especially advanced narrow-AI algorithms. This is the idea underlying the intelligence explosion thesis, and it also underlies Bostrom’s worry that capabilities research will continue to pull ahead of safety research.
In self-improving AI scenarios, the key question is which AI breakthroughs are prerequisites for automating high-leverage computer science tasks. This holds for capabilities research, and it also holds for safety research. Even if AI safety turned out to be easier than computer vision in an absolute sense, it would still stand out as a problem that is neither a prerequisite for building a self-improving AI, nor one we can safely delegate to such an AI.

 Still more fine-grained versions of the same argument may be possible. E.g., “programming ability” might decompose into multiple abilities, such as the ability to efficiently explore search spaces for code that meets constraints and the ability to efficiently test candidate code.  For example: If an AI approaching superintelligence stumbles upon a cascade of improvements to its programming ability, will its capabilities and decision criteria also result in repeated improvements its physics modules, or its psychology modules?  On page 76, for example, Bostrom writes: “This particular growth trajectory has a positive singularity at t = 18 months. In reality, the assumption that recalcitrance is constant would cease to hold as the system began to approach the physical limits of information processing, if not sooner.” On page 77, Bostrom says that the point he intends to illustrate is only that if AI progress is primarily AI-driven, resultant feedback loops that do arise will have a larger accelerating effect.  This is because a wide variety of final goals are best served through the acquisition or resources and the building of infrastructure, a set of objectives that are more likely to be furthered by electrical engineering skills than by painting skills. This argument is an instance of Bostrom’s instrumental convergence thesis in chapter 7. While it turns out that many intelligence-related characteristics correlate with a single easily-measured number in humans (g), this still doesn’t allow us to make fine-grained predictions about individual competency levels. I also can’t think of an obvious reason to expect a number even as informative as g to arise for AGI, especially early-stage AGI. Bostrom writes (p. 93):
[S]uppose we could somehow establish that a certain future AI will have an IQ of 6,455: then what? We would have no idea of what such an AI could actually do. We would not even know that such an AI had as much general intelligence as a normal human adult–perhaps the AI would instead have a bundle of special-purpose algorithms enabling it to solve typical intelligence test questions with superhuman efficiency but not much else.
Some recent efforts have been made to develop measurements of cognitive capacity that could be applied to a wider range of information-processing systems, including artificial intelligences. Work in this direction, if it can overcome various technical difficulties, may turn out to be quite useful for some scientific purposes including AI development. For purposes of the present investigation, however, its usefulness would be limited since we would remain unenlightened about what a given superhuman performance score entails for actual ability to achieve practically important outcomes in the world.  Imagine a Sumerian merchant living 5,000 years ago, shortly after the invention of writing, who has noticed the value of writing for storing good ideas over time, and not just market transactions. Writing could even allow one to transmit good ideas to someone you’ve never met, such as a future descendant. The merchant notices that his own successes have often involved collecting others’ good ideas, and that good ideas often open up pathways to coming up with other, even better ideas. From his armchair, he concludes that if writing becomes sufficiently popular, it will allow a quantity called society’s knowledge level to increase in an accelerating fashion; which, if the knowledge is used wisely, could result in unprecedented improvements to human life.
In retrospect we can say ‘knowledge’ would have been too coarse-grained a category to enable any precise predictions, and that there have really been multiple important breakthroughs that can be considered ‘knowledge explosions’ in different senses. Yet the extremely imprecise prediction can still give us a better sense of what to expect than we previously had. It’s a step up from the (historically common) view that civilizational knowledge only diminishes over time, the view that things will always stay the same, the view that human welfare will radically improve for reasons unrelated to knowledge build-up, etc.
The point of this analogy is not that people are good at making predictions about the distant future. Rather, my point is that hand-wavey quantities like ‘society’s knowledge level’ can be useful for making predictions, and can be based on good evidence, even if the correspondence between the quantity and the phenomenon it refers to is inexact.  Most obviously, a speed advantage can give the AI the time to design an even better AI.
None of this means that we can make specific highly confident predictions about when and how AI will achieve superintelligence. An AGI that isn’t very human-like may be slower than a human at specific tasks, or faster, in hard-to-anticipate ways. If a certain scientific breakthrough requires that one first build a massive particle accelerator, then the resources needed to build that accelerator may be a more important limiting factor than the AGI’s thinking speed. In that case, humans would have an easier time monitoring and regulating an AGI’s progress. We can’t rule out the possibility that speed superintelligence will face large unexpected obstacles, but we also shouldn’t gamble on that possibility or take it for granted. The post Davis on AI capability and motivation appeared first on Machine Intelligence Research Institute.

Davis on AI capability and motivation


New annotated bibliography for MIRI’s technical agenda

Today we release a new annotated bibliography accompanying our new technical agenda, written by Nate Soares. If you’d like to discuss the paper, please do so here.
Abstract:
How could superintelligent systems be aligned with the interests of humanity? This annotated bibliography compiles some recent research relevant to that question, and categorizes it into six topics: (1) realistic world models; (2) idealized decision theory; (3) logical uncertainty; (4) Vingean reflection; (5) corrigibility; and (6) value learning. Within each subject area, references are organized in an order amenable to learning the topic. These are by no means the only six topics relevant to the study of alignment, but this annotated bibliography could be used by anyone who wants to understand the state of the art in one of these six particular areas of active research.
Today we’ve also released a page that collects the technical agenda and supporting reports. See our Technical Agenda page.
The post New annotated bibliography for MIRI’s technical agenda appeared first on Machine Intelligence Research Institute.


New mailing list for MIRI math/CS papers only

As requested, we now offer email notification of new technical (math or computer science) papers and reports from MIRI. Simply subscribe to the mailing list below.
This list sends one email per new technical paper, and contains only the paper’s title, author(s), and abstract, plus a link to the paper.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New mailing list for MIRI math/CS papers only appeared first on Machine Intelligence Research Institute.


February 2015 Newsletter

Research Updates

Four new reports in support of our new technical agenda overview, on logical uncertainty, Vingean reflection, realistic world models, and value learning.
AI Impacts site relaunched with new content.
Reply to Rodney Brooks and John Searle on AI volition and timelines.
Interview: Matthias Troyer on quantum computers.

News Updates

Our Superintelligence online reading group is in its 20th week, discussing the value loading problem.

Other Updates

Top AI scientists and many others have signed an open letter advocating more research into robust and beneficial AI. The letter cites several MIRI papers.
Elon Musk has provided $10 million in funding for the types of research described in the open letter. The funding will be distributed in grants by the Future of Life Institute. Apply here.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

Facebook  |  Twitter  |  Google+  |  Forward to a friend

You’re receiving this because you subscribed to the MIRI newsletter.
unsubscribe from this list | update subscription preferences

The post February 2015 Newsletter appeared first on Machine Intelligence Research Institute.

February 2015 Newsletter


New report: “The value learning problem”

Today we release a new technical report by Nate Soares, “The value learning problem.” If you’d like to discuss the paper, please do so here.
Abstract:
A superintelligent machine would not automatically act as intended: it will act as programmed, but the fit between human intentions and formal specification could be poor. We discuss methods by which a system could be constructed to learn what to value. We highlight open problems specific to inductive value learning (from labeled training data), and raise a number of questions about the construction of systems which model the preferences of their operators and act accordingly.
This is the last of six new major reports which describe and motivate MIRI’s current research agenda at a high level.
Update May 29, 2016: A revised version of “The Value Learning Problem” (available at the original link) has been accepted to the IJCAI-16 Ethics for Artificial Intelligence workshop. The original version of the paper can be found here.
The post New report: “The value learning problem” appeared first on Machine Intelligence Research Institute.


New report: “Formalizing Two Problems of Realistic World Models”

Today we release a new technical report by Nate Soares, “Formalizing two problems of realistic world models.” If you’d like to discuss the paper, please do so here.
Abstract:
An intelligent agent embedded within the real world must reason about an environment which is larger than the agent, and learn how to achieve goals in that environment. We discuss attempts to formalize two problems: one of induction, where an agent must use sensory data to infer a universe which embeds (and computes) the agent, and one of interaction, where an agent must learn to achieve complex goals in the universe. We review related problems formalized by Solomonoff and Hutter, and explore challenges that arise when attempting to formalize analogous problems in a setting where the agent is embedded within the environment.
This is the 5th of six new major reports which describe and motivate MIRI’s current research agenda at a high level.
The post New report: “Formalizing Two Problems of Realistic World Models” appeared first on Machine Intelligence Research Institute.


New report: “Vingean Reflection: Reliable Reasoning for Self-Improving Agents”

Today we release a new technical report by Benja Fallenstein and Nate Soares, “Vingean Reflection: Reliable Reasoning for Self-Improving Agents.” If you’d like to discuss the paper, please do so here.
Abstract:
Today, human-level machine intelligence is in the domain of futurism, but there is every reason to expect that it will be developed eventually. Once artificial agents become able to improve themselves further, they may far surpass human intelligence, making it vitally important to ensure that the result of an “intelligence explosion” is aligned with human interests. In this paper, we discuss one aspect of this challenge: ensuring that the initial agent’s reasoning about its future versions is reliable, even if these future versions are far more intelligent than the current reasoner. We refer to reasoning of this sort as Vingean reflection.
A self-improving agent must reason about the behavior of its smarter successors in abstract terms, since if it could predict their actions in detail, it would already be as smart as them. This is called the Vingean principle, and we argue that theoretical work on Vingean reflection should focus on formal models that reflect this principle. However, the framework of expected utility maximization, commonly used to model rational agents, fails to do so. We review a body of work which instead investigates agents that use formal proofs to reason about their successors. While it is unlikely that real-world agents would base their behavior entirely on formal proofs, this appears to be the best currently available formal model of abstract reasoning, and work in this setting may lead to insights applicable to more realistic approaches to Vingean reflection.
This is the 4th of six new major reports which describe and motivate MIRI’s current research agenda at a high level.
The post New report: “Vingean Reflection: Reliable Reasoning for Self-Improving Agents” appeared first on Machine Intelligence Research Institute.


An improved “AI Impacts” website

Recently, MIRI received a targeted donation to improve the AI Impacts website initially created by frequent MIRI collaborator Paul Christiano and part-time MIRI researcher Katja Grace. Collaborating with Paul and Katja, we ported the old content to a more robust and navigable platform, and made some improvements to the content. You can see the result at AIImpacts.org.
As explained in the site’s introductory blog post,
AI Impacts is premised on two ideas (at least!):

The details of the arrival of human-level artificial intelligence matter
Seven years to prepare is very different from seventy years to prepare. A weeklong transition is very different from a decade-long transition. Brain emulations require different preparations than do synthetic AI minds. Etc.
Available data and reasoning can substantially educate our guesses about these details
We can track progress in AI subfields. We can estimate the hardware represented by the human brain. We can detect the effect of additional labor on software progress. Etc.

Our goal is to assemble relevant evidence and considerations, and to synthesize reasonable views on questions such as when AI will surpass human-level capabilities, how rapid development will be at that point, what advance notice we might expect, and what kinds of AI are likely to reach human-level capabilities first.
The meat of the website is in its articles. Here are two examples to start with:

A summary of AI surveys
Cases of discontinuous technological progress

The post An improved “AI Impacts” website appeared first on Machine Intelligence Research Institute.

An improved “AI Impacts” website


New report: “Questions of reasoning under logical uncertainty”

Today we release a new technical report by Nate Soares and Benja Fallenstein, “Questions of reasoning under logical uncertainty.” If you’d like to discuss the paper, please do so here.
Abstract:
A logically uncertain reasoner would be able to reason as if they know both a programming language and a program, without knowing what the program outputs. Most practical reasoning involves some logical uncertainty, but no satisfactory theory of reasoning under logical uncertainty yet exists. A better theory of reasoning under logical uncertainty is needed in order to develop the tools necessary to construct highly reliable artificial reasoners. This paper introduces the topic, discusses a number of historical results, and describes a number of open problems.
This is the 3rd of six new major reports which describe and motivate MIRI’s current research agenda at a high level.
The post New report: “Questions of reasoning under logical uncertainty” appeared first on Machine Intelligence Research Institute.