MIRI strategy update: 2016

This post is a follow-up to Malo's 2015 review, sketching out our new 2016-2017 plans. Briefly, our top priorities (in decreasing order of importance) are to (1) make technical progress on the research problems we've identified, (2) expand our team, and (3) build stronger ties to the wider research community.
As discussed in a previous blog post, the biggest update to our research plans is that we'll be splitting our time going forward between our 2014 research agenda (the "agent foundations" agenda) and a new research agenda oriented toward machine learning work led by Jessica Taylor: "Alignment for Advanced Machine Learning Systems."
Three additional news items:
1. I'm happy to announce that MIRI has received support from a major new donor: entrepreneur and computational biologist Blake Borgeson, who has made a $300,000 donation to MIRI. This is the second-largest donation MIRI has received in its history, beaten only by Jed McCaleb's 2013 cryptocurrency donation. As a result, we've been able to execute on our growth plans with more speed, confidence, and flexibility.
2. This year, instead of running separate summer and winter fundraisers, we're merging them into one more ambitious fundraiser, which will take place in September.
3. I'm also pleased to announce that Abram Demski has accepted a position as a MIRI research fellow. Additionally, Ryan Carey has accepted a position as an assistant research fellow, and we've hired some new administrative staff.
I'll provide more details on these and other new developments below.

Priority 1: Make progress on open technical problems
Since 2013, MIRI's primary goal has been to make technical progress on AI alignment. Nearly all of our other activities are either directly or indirectly aimed at producing more high-quality alignment research, either at MIRI or at other institutions.
As mentioned above, Jessica Taylor is now leading an "Alignment for Advanced Machine Learning Systems" program, which will occupy about half of our research efforts going forward. Our goal with this work will be to develop formal models and theoretical tools that we predict would aid in the alignment of highly capable AI systems, under the assumption that such systems will be qualitatively similar to present-day machine learning systems. Our research communications manager, Rob Bensinger, has summarized themes in our new work and its relationship to other AI safety research proposals.
Earlier in the year, I jotted down a summary of how much technical progress I thought we'd made on our research agenda in 2015 (noted by Malo in our 2015 review), relative to my expectations. In short, I expected modest progress in all of our research areas except value specification (which was low-priority for us in 2015). We made progress more quickly than expected on some problems, and more slowly than expected on others.
In naturalized induction and logical uncertainty, we exceeded my expectations, making sizable progress. In error tolerance, we undershot my expectations and made only limited progress. In our other research areas, we made about as much progress as I expected: modest progress in decision theory and Vingean reflection, and limited progress in value specification.
I also made personal predictions earlier in the year about how much progress we'd make through the end of 2016: modest progress in decision theory, error tolerance, and value specification; limited progress in Vingean reflection; and sizable progress in logical uncertainty and naturalized induction. (Starting in 2017, I'll be making my predictions publicly early in the year.)
Breaking these down:

Vingean reflection is a lower priority for us this year. This is in part because we're less confident that there's additional low-hanging fruit to be plucked here, absent additional progress in logical uncertainty or decision theory. Although we've been learning a lot about implementation snags through Benya Fallenstein, Ramana Kumar, and Jack Gallagher's ongoing HOL-in-HOL project, we haven't seen any major theoretical breakthroughs in this area since Benya developed model polymorphism in late 2012. Benya and Kaya Fallenstein are still studying this topic occasionally.

In contrast, we've continued to make steady gains in the basic theory of logical uncertainty, naturalized induction, and decision theory over the years. Benya, Kaya, Abram, Scott Garrabrant, Vanessa Kosoy, and Tsvi Benson-Tilsen will be focusing on these areas over the coming months, and I expect to see advances in 2016 of similar importance to what we saw in 2015.

Our machine learning agenda is primarily focused on error tolerance and value specification, making these much higher priorities for us this year. I expect to see modest progress from Jessica Taylor, Patrick LaVictoire, Andrew Critch, Stuart Armstrong, and Ryan Carey's work on these problems. It's harder to say whether there will be any big breakthroughs here, given how new the program is.

Eliezer Yudkowsky and I will be splitting our time between working on these problems and doing expository writing. Eliezer is writing about alignment theory, while I'll be writing about MIRI strategy and forecasting questions.
We spent large portions of the first half of 2016 writing up existing results and research proposals and coordinating with other researchers (such as through our visit to FHI and our Colloquium Series on Robust and Beneficial AI), and we have a bit more writing ahead of us in the coming weeks. We managed to get a fair bit of research done — we'll be announcing a sizable new logical uncertainty result once the aforementioned writing is finished — but we're looking forward to a few months of uninterrupted research time at the end of the year, and I'm excited to see what comes of it.

Priority 2: Expand our team
Growing MIRI's research team is a high priority. We're also expanding our admin team, with a goal of freeing up more of my time and better positioning MIRI to positively influence the booming AI risk conversation.
After making significant contributions to our research over the past year as a research associate (e.g., "Inductive Coherence" and Structural Risk Mitigation) and participating in our CSRBAI and MIRI Summer Fellows programs, Abram Demski has signed on to join our core research team. Abram is planning to join in late 2016 or early 2017, after completing his computer science PhD at the University of Southern California. Mihály Bárász is also slated to join our core research team at a future date, and we are considering several other promising candidates for research fellowships.
In the nearer term, data scientist Ryan Carey has been collaborating with us on our machine learning agenda and will be joining us as an assistant research fellow in September.
We've also recently hired a new office manager, Aaron Silverbook, and a communications and development admin, Colm Ó Riain.
We have an open type theorist job ad, and are more generally seeking research fellows with strong mathematical intuitions and a talent for formalizing and solving difficult problems, or for fleshing out and writing up results for publication.
We're also seeking communications and outreach specialists (e.g., computer programmers with very strong writing skills) to help us keep pace with the lively public and academic AI risk conversation. If you're interested, send a résumé and nonfiction writing samples to Rob.
Priority 3: Collaborate and communicate with other researchers
There have been a number of new signs in 2016 that AI alignment is going (relatively) mainstream:

Stuart Russell and his students' recent work on value learning and corrigibility (including a joint grant project with MIRI);
positive reactions from Eric Schmidt and the press to a Google DeepMind / Future of Humanity Institute collaboration on corrigibility (partly supported by MIRI);
the new "Concrete Problems in AI Safety" research proposal announced by Google Research and OpenAI (along with the Open Philanthropy Project's declaration of interest in funding such research);
and other, smaller developments.

MIRI's goal is to ensure that the AI alignment problem gets solved, whether it's MIRI solving it or some other group. As such, we're excited by the new influx of attention directed at the alignment problem, and view this as an important time to nurture the field.
As AI safety research goes more mainstream, the pool of researchers we can dialogue with is becoming larger. At the same time, our own approach to the problem — specifically focused on the most long-term, high-stakes, and poorly-understood parts of the problem, and the parts that are least concordant with academic and industry incentives — remains unusual. Absent MIRI, I think that this part of the conversation would be almost entirely neglected.
To help promote our approach and grow the field, we intend to host more workshops aimed at diverse academic audiences. We'll be hosting a machine learning workshop in the near future, and might run more events like CSRBAI going forward. We also have a backlog of past technical results to write up, which we expect to be valuable for engaging more researchers in computer science, economics, mathematical logic, decision theory, and other areas.
We're especially interested in finding ways to hit priorities 1 and 3 simultaneously, pursuing important research directions that also help us build stronger ties to the wider academic world. One of several reasons for our new research agenda is its potential to encourage more alignment work by the ML community.

Short version: in the medium term, our research program will have a larger focus on error-tolerance and value specification research, with more emphasis on ML-inspired AI approaches, and we're increasing the size of our research team in pursuit of that goal.
Rob, Malo, and I will be saying more about our funding situation and organizational strategy in September, when we kick off our 2016 fundraising drive. As part of that series of posts, I'll also be writing more about how our current strategy fits into our long-term goals and priorities.
Finally, if you're attending Effective Altruism Global this weekend, note that we'll be running two workshops (one on Jessica's new project, another on the aforementioned new logical uncertainty results), as well as some office hours (both with the research team and with the admin team). If you're there, feel free to drop by, say hello, and ask more about what we've been up to.
The post MIRI strategy update: 2016 appeared first on Machine Intelligence Research Institute.

MIRI strategy update: 2016


August 2016 Newsletter

Research updates

A new paper: "Alignment for Advanced Machine Learning Systems." Half of our research team will be focusing on this research agenda going forward, while the other half continues to focus on the agent foundations agenda.
New at AI Impacts: Returns to Scale in Research
Evan Lloyd represented MIRIxLosAngeles at AGI-16 this month, presenting "Asymptotic Logical Uncertainty and the Benford Test" (slides).
We'll be announcing a breakthrough in logical uncertainty this month, related to Scott Garrabrant's previous results.

General updates

Our 2015 in review, with a focus on the technical problems we made progress on.
Another recap: how our summer colloquium series and fellows program went.
We've uploaded our first CSRBAI talks: Stuart Russell on "AI: The Story So Far" (video), Alan Fern on "Toward Recognizing and Explaining Uncertainty" (video), and Francesca Rossi on "Moral Preferences" (video).
We submitted our recommendations to the White House Office of Science and Technology Policy, cross-posted to our blog.
We attended IJCAI and the White House's AI and economics event. Furman on technological unemployment (video) and other talks are available online.
Talks from June's safety and control in AI event are also online. Speakers included Microsoft's Eric Horvitz (video), FLI's Richard Mallah (video), Google Brain's Dario Amodei (video), and IARPA's Jason Matheny (video).

News and links

Complexity No Bar to AI: Gwern Branwen argues that computational complexity theory provides little reason to doubt that AI can surpass human intelligence.
Bill Nordhaus, the world's leading climate change economist, writes a paper on the economics of singularity scenarios.
The Open Philanthropy Project has awarded Robin Hanson a three-year $265,000 grant to study multipolar AI scenarios. See also Hanson's new argument for expecting a long era of whole-brain emulations prior to the development of AI with superhuman reasoning abilities.
"Superintelligence Cannot Be Contained" discusses computability-theoretic limits to AI verification.
The Financial Times runs a good profile of Nick Bostrom.
DeepMind software reduces Google's data center cooling bill by 40%.
In a promising development, US federal regulators argue for the swift development and deployment of self-driving cars to reduce automobile accidents: "We cannot wait for perfect. We lose too many lives waiting for perfect."

The post August 2016 Newsletter appeared first on Machine Intelligence Research Institute.

August 2016 Newsletter


2016 summer program recap

As previously announced, we recently ran a 22-day Colloquium Series on Robust and Beneficial AI (CSRBAI) at the MIRI office, co-hosted with the Oxford Future of Humanity Institute. The colloquium was aimed at bringing together safety-conscious AI scientists from academia and industry to share their recent work. The event served that purpose well, initiating some new collaborations and a number of new conversations between researchers who hadn't interacted before or had only talked remotely.
Over 50 people attended from 25 different institutions, with an average of 15 people present on any given talk or workshop day. In all, there were 17 talks and four weekend workshops on the topics of transparency, robustness and error-tolerance, preference specification, and agent models and multi-agent dilemmas. The full schedule and talk slides are available on the event page. Videos from the first day of the event are now available, and we'll be posting the rest of the talks online soon:

Stuart Russell, professor of computer science at UC Berkeley and co-author of Artificial Intelligence: A Modern Approach, gave the opening keynote. Russell spoke on "AI: The Story So Far" (slides). Abstract:
I will discuss the need for a fundamental reorientation of the field of AI towards provably beneficial systems. This need has been disputed by some, and I will consider their arguments. I will also discuss the technical challenges involved and some promising initial results.
Russell discusses his recent work on cooperative inverse reinforcement learning 36 minutes in. This paper and Dylan Hadfield-Menell's related talk on corrigibility (slides) inspired lots of interest and discussion at CSRBAI.

Alan Fern, associate professor of computer science at Oregon State University, discussed his work with AAAI president and OSU distinguished professor of computer science Tom Dietterich in "Toward Recognizing and Explaining Uncertainty" (slides 1, slides 2). Fern and Dietterich's work is described in a Future of Life Institute grant proposal:
The development of AI technology has progressed from working with "known knowns"—AI planning and problem solving in deterministic, closed worlds—to working with "known unknowns"—planning and learning in uncertain environments based on probabilistic models of those environments. A critical challenge for future AI systems is to behave safely and conservatively in open worlds, where most aspects of the environment are not modeled by the AI agent—the "unknown unknowns".
Our team, with deep experience in machine learning, probabilistic modeling, and planning, will develop principles, evaluation methodologies, and algorithms for learning and acting safely in the presence of the unknown unknowns. For supervised learning, we will develop UU-conformal prediction algorithms that extend conformal prediction to incorporate nonconformity scores based on robust anomaly detection algorithms. This will enable supervised learners to behave safely in the presence of novel classes and arbitrary changes in the input distribution. For reinforcement learning, we will develop UU-sensitive algorithms that act to minimize risk due to unknown unknowns. A key principle is that AI systems must broaden the set of variables that they consider to include as many variables as possible in order to detect anomalous data points and unknown side-effects of actions.

Francesca Rossi, professor of computer science at Padova University in Italy, research scientist at IBM, and president of IJCAI, spoke on "Moral Preferences" (slides). Abstract:
Intelligent systems are going to be more and more pervasive in our everyday lives. They will take care of elderly people and kids, they will drive for us, and they will suggest doctors how to cure a disease. However, we cannot let them do all this very useful and beneficial tasks if we don't trust them. To build trust, we need to be sure that they act in a morally acceptable way. So it is important to understand how to embed moral values into intelligent machines.
Existing preference modeling and reasoning framework can be a starting point, since they define priorities over actions, just like an ethical theory does. However, many more issues are involved when we mix preferences (that are at the core of decision making) and morality, both at the individual level and in a social context. I will discuss some of these issues as well as some possible solutions.
Other speakers at the event included Tom Dietterich (OSU), Bart Selman (Cornell), Paul Christiano (UC Berkeley), and MIRI researchers Jessica Taylor and Andrew Critch.

The preference specification workshop attracted the most excitement and activity at CSRBAI. Other activities and discussion topics at CSRBAI included:

Discussions about potential applications of complexity theory to transparency: using interactive polynomial-time proof protocols or probabilistically checkable proofs to communicate complicated beliefs and reasons from powerful AI systems to humans.
Some progress clarifying different methods of training explanation systems for informed oversight.
Investigations into the theory of cooperative inverse reinforcement learning and other unobserved-reward games, led by Jan Leike and Tom Everitt of Australian National University.
Discussions about the hazards associated with reinforcement learning agents that manipulate the source of their reward function (which is the human or a learned representation of the human).
Interesting discussions about corrigibility viewed as a value-of-information problem.
Development of AI safety environments by Rafael Cosman and other attendees for the OpenAI Reinforcement Learning Gym, illustrating topics like interruptibility and semi-supervised learning. Ideas and conversation from Chris Olah, Dario Amodei, Paul Christiano, and Jessica Taylor helped seed these gyms, and CSRBAI participants who helped develop them included Owain Evans, Sune Jakobsen, Stuart Armstrong, Tom Everitt, Rafael Cosman, and David Krueger.
Discussions of ideas for an OpenGym environment asking for low-impact agents, using an adversarial distinguisher.
Discussions of Jessica Taylor's memoryless Cartesian environments aimed at extending the idea to non-Cartesian worlds / logical counterfactuals using reference-class decision-making. Discussions of using "logically past" experience to learn about counterfactuals and do exploration without having a high chance of exploring in the real world.
New insights into the problem of logical counterfactuals, with new associated formalisms. Applications of MIRI's recent logical uncertainty advances to decision theory.
A lot of advance discussion of MIRI's "Alignment for Advanced Machine Learning Systems" technical agenda.

The colloquium series ran quite smoothly, and we received positive feedback from attendees. Attendees noted that the event would have likely benefited from more structure. When we run events like this in the future, our main adjustment will be to compress the schedule and run more focused events similar to our past workshops.

We also co-ran a 16-day MIRI Summer Fellows program with the Center for Applied Rationality in June. The program's 14 attendees came from a variety of technical backgrounds and ranged from startup founders to undergraduates to assistant professors.
Our MIRISF programs have proven useful in the past for identifying future MIRI hires (one full-time and two part-time MIRI researchers from the 2015 MIRISF program). The primary focus, however, is on developing new problem-solving skills and mathematical intuitions for CS researchers and providing an immersive crash course on MIRI's active research projects.
The program had four distinct phases: a four-day CFAR retreat (followed by a rest day), a two-day course in MIRI's research agenda, three days of working together on research topics (similar to a MIRI research workshop, and followed by another off day), and three days of miscellaneous activities: Tetlock-style forecasting practice, one-on-ones with MIRI researchers, security mindset discussions, planning ahead for future research and collaborations, etc.
To receive notifications from us about future programs like MIRISF, use this form. To get in touch with us about collaborating at future MIRI workshops like the ones at CSRBAI, send us your info via our general application form.
The post 2016 summer program recap appeared first on Machine Intelligence Research Institute.

2016 summer program recap


2015 in review

As Luke had done in years past (see 2013 in review and 2014 in review), I (Malo) wanted to take some time to review our activities from last year. In the coming weeks Nate will provide a big-picture strategy update. Here, I'll take a look back at 2015, focusing on our research progress, academic and general outreach, fundraising, and other activities.
After seeing signs in 2014 that interest in AI safety issues was on the rise, we made plans to grow our research team. Fueled by the response to Bostrom's Superintelligence and the Future of Life Institute's "Future of AI" conference, interest continued to grow in 2015. This suggested that we could afford to accelerate our plans, but it wasn't clear how quickly.
In 2015 we did not release a mid-year strategic plan, as Luke did in 2014. Instead, we laid out various conditional strategies dependent on how much funding we raised during our 2015 Summer Fundraiser. The response was great; we had our most successful fundraiser to date. We hit our first two funding targets (and then some), and set out on an accelerated 2015/2016 growth plan.
As a result, 2015 was a big year for MIRI. After publishing our technical agenda at the start of the year, we made progress on many of the open problems it outlined, doubled the size of our core research team, strengthened our connections with industry groups and academics, and raised enough funds to maintain our growth trajectory. We're very grateful to all our supporters, without whom this progress wouldn't have been possible.

2015 Research Progress
Our "Agent Foundations for Aligning Machine Intelligence with Human Interests" research agenda divides open problems into three categories: high reliability (which includes logical uncertainty, naturalized induction, decision theory, and Vingean reflection), error tolerance, and value specification.1 MIRI's top goal in 2015 was to make progress on these problems.
We met our expectations for research progress in each category, with the exception of logical uncertainty and naturalized induction (where we made more progress than expected) and error tolerance (where we made less progress than expected).
Below I've provided a brief summary of our progress in each area, with additional details and a full publication list in collapsed "Read More" sections. Some of the papers we published in 2015 were based on research from 2014 or earlier, and some of our 2015 results weren't published until 2016 (or remain unpublished). In this review I'll focus on 2015's new technical developments, rather than on pre-2015 material that happened to be published in that year.
Logical Uncertainty and Naturalized Induction
We expected to make modest progress on these two problems in 2015. I'm pleased to report we made sizable progress.
2015 saw the tail end of our development of reflective oracles, and early work on "optimal estimators." Our most important research advance of the year, however, was likely our success dividing logical uncertainty into two subproblems, which happened in late 2015 and the very beginning of 2016.
One intuitive constraint on correct logically uncertain reasoning is that one's probabilities reflect known logical relationships between claims. For example, if you know that two claims are mutually exclusive (such as "this computation outputs a 3" and "this computation outputs a 7"), then even if you can't evaluate the claims, you should assign probabilities to the two claims that sum to at most 1.
A second intuitive constraint is that one's probabilities reflect empirical regularities. Once you observe enough digits of π, you should eventually guess that the numbers 8 and 3 occur equally often in π's decimal expansion, even if you have not yet proven that π is normal.
In 2015, we developed two different algorithms to solve these two subproblems in isolation.

 Read More

In collaboration with Benya Fallenstein and other MIRI researchers, Scott Garrabrant solved the problem of respecting logical relationships in a series of Intelligent Agent Foundations Forum (IAFF) posts, resulting in the "Inductive Coherence" paper. The problem of respecting observational patterns in logical sentences was solved by Scott and the MIRIxLosAngeles group in "Asymptotic Logical Uncertainty and the Benford Test," which was further developed into the "Asymptotic Convergence in Online Learning with Unbounded Delays" paper in 2016.
These two approaches to logical uncertainty were not only nonequivalent, but seemed to preclude each other. The obvious next step is to investigate whether there is a way to solve both subproblems at once with a single procedure—a task we have since made some (soon-to-be-announced) progress on in 2016.
MIRI research associate Vanessa Kosoy's work on his "optimal estimators" framework represents a large separate corpus of work on logical uncertainty, which may also have applications for decision theory. Vanessa's work has not yet been officially published, but much of it is available on IAFF.
Our other significant result in logical uncertainty was Benya Fallenstein, Jessica Taylor, and Paul Christiano's reflective oracles, building on work that began before 2015 (IAFF digest). Reflective oracles avoid a number of paradoxes that normally arise when agents attempt to answer questions about equivalently powerful agents, allowing us to study multi-agent dilemmas and reflective reasoning with greater precision.
Reflective oracles are interesting in their own right, and have proven applicable to a number of distinct open problems. The fact that reflective oracles require no privileged agent/environment distinction suggests that they're a step in the right direction for naturalized induction. Jan Leike has recently demonstrated that reflective oracles also solve a longstanding open problem in game theory, the grain of truth problem. Reflective oracles provide the first complete decision-theoretic foundation for game theory, showing that general-purpose methods for maximizing expected utility can achieve approximate Nash equilibria in repeated games.
In summary, our 2015 logical uncertainty and naturalized induction papers based on pre-2015 work were:

B Fallenstein, J Taylor, P Christiano. "Reflective Oracles: A Foundation for Classical Game Theory." arXiv:1508.04145 [cs.AI]. Published in abridged form in Proceedings of LORI 2015.
N Soares. "Formalizing Two Problems of Realistic World-Models." MIRI tech report 2015-3.
N Soares, B Fallenstein. "Questions of Reasoning under Logical Uncertainty." MIRI tech report 2015-1.

2015 research published the same year:

B Fallenstein, N Soares, J Taylor. "Reflective Variants of Solomonoff Induction and AIXI." Published in Proceedings of AGI 2015.

2015 research published in 2016 or forthcoming:

S Garrabrant, S Bhaskhar, A Demski, J Garrabrant, G Koleszarik, E Lloyd. "Asymptotic Logical Uncertainty and The Benford Test." arXiv:1510.03370 [cs.LG]. Forthcoming at AGI 2016.
S Garrabrant, B Fallenstein, A Demski, N Soares. "Inductive Coherence." arXiv:1604.05288 [cs:AI].
S Garrabrant, N Soares, J Taylor. "Asymptotic Convergence in Online Learning with Unbounded Delays." arXiv:1604.05280 [cs:LG].
V Kosoy. Formally unpublished results on the optimal estimators framework.
J Leike, J Taylor, B Fallenstein. "A Formal Solution to the Grain of Truth Problem." Presented at the 32nd Conference on Uncertainty in Artificial Intelligence.

For other logical uncertainty work on IAFF, see The Two-Update Problem, Subsequence Induction, and Strict Dominance for the Modified Demski Prior.

Decision Theory
In 2015 we produced a number of new incremental advances in decision theory, constituting modest progress, in line with our expectations.
Of these advances, we have published Andrew Critch's proof of a version of Löb's theorem and Gödel's second incompleteness that holds for bounded reasoners.
Critch applies this parametric bounded version of Löb's theorem to prove that a wide range of resource-limited software agents, given access to each other's source code, can achieve unexploitable mutual cooperation in the one-shot prisoner's dilemma. Although we considered our past robust cooperation results strong reason to believe that bounded cooperation was possible, the confirmation is useful and gives us new formal tools for studying bounded reasoners.
Over this period, Eliezer Yudkowsky, Benya Fallenstein, and Nate Soares also improved our technical (and philosophical) understanding of the decision theory we currently favor, "functional decision theory"—a slightly modified version of updateless decision theory.

 Read More

The biggest obstacle to formalizing decision theory currently seems to be that we lack a suitable formal account of logical counterfactuals. Logical counterfactuals are questions of the form "If X (which I know to be false) were true, what (if anything) would that imply about Y?" These are important in decision theory, one special case being off-policy predictions. (Even if I can predict that I'm definitely not taking action X, I want to be able to ask what would ensue if I did; a wrong answer to this can lead to me accepting substandard self-fulfilling prophecies like two-boxing in the transparent Newcomb problem.)
In 2015, we examined a decision theory related to functional decision theory, proof-based decision theory, that has proven easier to formalize. We found that proof-based decision theory's lack of logical counterfactuals is a serious weakness for the theory.
We explored some proof-length-based approaches to logical counterfactuals, and ultimately rejected them, though we have continued to devote some thought to this approach. During our first 2015 workshop, Scott Garrabrant proposed an informal conjecture on proof length and counterfactuals, which was subsequently revised; but both versions of the conjecture were shown to be false by Sam Eisenstat (1, 2). (See also Scott's Optimal and Causal Counterfactual Worlds.)
In a separate line of research, Patrick LaVictoire and others applied the proof-based decision theory framework to questions of bargaining and division of trade gains. For other decision theory work on IAFF, see Vanessa and Scott's Superrationality in Arbitrary Games and Armstrong's Reflective Oracles and Superrationality: Prisoner's Dilemma.
Our github repository contains lots of new code from our work on modal agents, representing our most novel work on decision theory in the past year. We have one or two papers in progress that will explain the advances we've made in decision theory via this work. See "Evil" Decision Problems in Provability Logic and other posts in the decision theory IAFF digest for background on modal universes.
Pre-2015 work published in 2015:

N Soares, B Fallenstein. "Toward Idealized Decision Theory." 2014 tech report published in abridged form in Proceedings of AGI 2015.

2015 research published in 2016 or forthcoming:

A Critch. "Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents." arXiv:1602.04184 [cs:GT].
B Fallenstein. Formally unpublished results on modal universes.
S Garrabrant, S Eisenstat, P LaVictoire, J Lee, H Dell. Formally unpublished results on logical counterfactuals.
E Yudkowsky, N Soares. Unpublished results on functional decision theory.

Vingean Reflection
We were expecting modest progress on these problems in 2015, and we made modest progress.
Benya Fallenstein and Ramana Kumar's "Proof-Producing Reflection for HOL" demonstrates a practical form of self-reference (and a partial solution to both the Löbian obstacle and the procrastination paradox) in the HOL theorem prover. This result provides some evidence that it is possible for a reasoning system to trust another reasoning system that reasons the same way, so long as the systems have different internal states.

 Read More

More specifically, this paper establishes that it is possible to formally specify an infinite chain of reasoning systems such that each system trusts the next system in the chain, as long as the reasoners are unable to delegate any individual task indefinitely.
There is some internal debate within MIRI about what more is required for real-world Vingean reflection, aside from satisfactory accounts of logical uncertainty and logical counterfactuals. There's also debate about whether any better results than this are likely to be possible in the absence of a full theory of logical uncertainty. Regardless, "Proof-Producing Reflection for HOL" demonstrates, via machine-checked proof, that it is possible to implement a form of reflective reasoning that is remarkably strong.
Benya and Ramana's work also provides us with an environment in which to build better toy models of reflective reasoners. Jack Gallagher, a MIRI research intern, is currently implementing a cellular automaton in HOL that will let us implement reflective agents.
By applying results from the reflective oracles framework mentioned above, we also improved our theoretical understanding of Vingean reflection. In the IAFF post A Limit-Computable, Self-Reflective Distribution, research associate Tsvi Benson-Tilsen helped solidify our understanding of what kinds of reflection are and aren't possible. Jessica, working with Benya and Paul, further showed that reflective oracles can't readily be used to define reflective probabilistic logics.
Pre-2015 work published in 2015:

B Fallenstein, N Soares. "Vingean Reflection: Reliable Reasoning for Self-Improving Agents." MIRI tech report 2015-2.

2015 research published the same year:

B Fallenstein, R Kumar. "Proof-Producing Reflection for HOL: With an Application to Model Polymorphism." Published in Interactive Theorem Proving: 6th International Conference, ITP 2015, Nanjing, China, August 24-27, 2015, Proceedings.

Other relevant IAFF posts include A Simple Model of the Löbstacle, Waterfall Truth Predicates, and Existence of Distributions that are Expectation-Reflective and Know It.

Error Tolerance
We were expecting modest progress on these problems in 2015, but we made only limited progress.
Corrigibility was a mid-level priority for us in 2015, and we spent some effort trying to build better models of corrigible agents. In spite of this, we didn't achieve any big breakthroughs. We made some progress on fixing minor defects in our understanding of corrigibility, reflected, e.g., in our error-tolerance IAFF digest, Stuart Armstrong's AI control ideas, and Jessica Taylor's overview post; but these results are relatively small.

 Read More

In 2015 our main novelties were Google DeepMind researcher Laurent Orseau and FHI researcher / MIRI research associate Stuart Armstrong's work on corrigibility ("Safely Interruptible Agents"), along with work on two other error tolerance subproblems: mild optimization (Jessica's Quantilizers and Abram Demski's Structural Risk Minimization) and conservative concepts (Jessica's Learning a Concept Using Only Positive Examples).
Pre-2015 work published in 2015:

N Soares, B Fallenstein, E Yudkowsky, S Armstrong. "Corrigibility." 2014 tech report presented at the AAAI 2015 Ethics and Artificial Intelligence Workshop.

2015 research published in 2016 or forthcoming:

L Orseau, S Armstrong. "Safely Interruptible Agents." Presented at the 32nd Conference on Uncertainty in Artificial Intelligence.
J Taylor. "Quantilizers: A Safer Alternative to Maximizers for Limited Optimization." Presented at the AAAI 2016 AI, Ethics and Society Workshop.

Our failure to make much progress on corrigibility may be a sign that corrigibility is not as tractable a problem as we thought, or that more progress is needed in areas like logical uncertainty (so that we can build better models of AI systems that model their operators as uncertain about the implications of their preferences) before we can properly formalize corrigibility.
We are more optimistic about corrigibility research, however, in light of recent advances in logical uncertainty and some promising discussions of related topics at our recent colloquium series: "Cooperative Inverse Reinforcement Learning" (via Stuart Russell's group), "Avoiding Wireheading with Value Reinforcement Learning" (via Tom Everitt), and some items in Stuart Armstrong's bag of tricks.

Value Specification
We were expecting limited progress on these problems in 2015, and we made limited progress.
Value learning and related problems were low-priority for us last year, so we didn't see any big advances.

 Read More

MIRI research associate Kaj Sotala made value specification his focus, examining several interesting questions outside our core research agenda. Jessica Taylor also began investigating the problem on the research forum.
Pre-2015 work published in 2015:

K Sotala. "Concept Learning for Safe Autonomous AI." Presented at the AAAI 2015 Ethics and Artificial Intelligence Workshop.

2015 research published in 2016 or forthcoming:

K Sotala. "Defining Human Values for Value Learners." Presented at the AAAI 2016 AI, Ethics and Society Workshop.

Error-tolerant agent designs and value specification will be larger focus areas for us going forward, under the alignment for advanced machine learning systems research program.

Miscellaneous
We released our technical agenda in late 2014 and early 2015. The overview paper, "Agent Foundations for Aligning Machine Intelligence with Human Interests," is slated for external publication in The Technological Singularity in 2017.
In 2015 we also produced some research unrelated to our agent foundations agenda. This research generally focused on forecasting and strategy questions.

 Read More

Pre-2015 work published in 2015:

S Armstrong, N Bostrom, C Shulman. "Racing to the Precipice: A Model of Artificial Intelligence Development." 2013 tech report published in AI & Society.
K Grace. "The Asilomar Conference: A Case Study in Risk Mitigation." MIRI tech report 2015-9.
K Grace. "Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation." MIRI tech report 2015-10.
P LaVictoire. "An Introduction to Löb's Theorem in MIRI Research." MIRI tech report 2015-6.

2015 research published in 2016 or forthcoming:

T Benson-Tilsen, N Soares. "Formalizing Convergent Instrumental Goals." Presented at the AAAI 2016 AI, Ethics and Society Workshop.

Beginning in 2015, new AI strategy/forecasting research supported by MIRI has been hosted on Katja Grace's independent AI Impacts project. AI Impacts featured 31 new articles and 27 new blog posts in 2015, on topics from the range of human intelligence to computing cost trends.

On the whole, we're happy about our 2015 research output and expect our team growth to further accelerate technical progress.

2015 Research Support Activities
Focusing on activities that directly grew the technical research community or facilitated technical research and collaborations, in 2015 we:

Launched the Intelligent Agent Foundations Forum, a public discussion forum for AI alignment researchers. MIRI researchers and collaborators made 139 top-level posts to IAFF in 2015.
Hired four new full-time research fellows. Patrick LaVictoire joined in March, Jessica Taylor in August, Andrew Critch in September, and Scott Garrabrant in December. With Nate transitioning to a non-research role, overall we grew from a three-person research team (Eliezer, Benya, and Nate) to a six-person team.
Overhauled our research associates program. Before 2015, our research associates were mostly unpaid collaborators with varying levels of involvement in our active research. Following our successful summer fundraiser, we made "research associate" a paid position in which researchers based at other institutions spend significant amounts of time on research projects for us. Under this program, Stuart Armstrong, Tsvi Benson-Tilsen, Abram Demski, Vanessa Kosoy, Ramana Kumar, Kaj Sotala, and (prior to joining MIRI full-time) Scott Garrabrant all made significant contributions in associate roles.
Hired three research interns. Kaya Stechly and Rafael Cosman worked on polishing and consolidating old MIRI results (example on IAFF), while Jack Gallagher worked on our type theory in type theory project (github repo).
Acquired two new research advisors, Stuart Russell and Bart Selman.
Hosted six summer workshops and sponsored the three-week MIRI Summer Fellows program. These events helped forge a number of new academic connections and directly resulted in us making job offers to two extremely promising attendees: Mihály Bárász (who has plans to join at a future date) and Scott Garrabrant.
Helped organize two other academic events, a Cambridge decision theory conference and a ten-week AI alignment seminar series at UC Berkeley. We also ran 6 research retreats, sponsored 36 MIRIx events, and spoke at an Oxford Big Picture Thinking seminar series.
Spoke at five other academic events. We participated in the Future of Life Institute's "Future of AI" conference, AAAI-15, AGI-15, LORI 2015, and APS 2015. We also attended NIPS.

I'm excited about our 2015 progress in growing our team and collaborating with the larger academic community. Over the course of the year, we built closer relationships with people at Google DeepMind, Google Brain, OpenAI, Vicarious, Good AI, the Future of Humanity Institute, and other research groups. All of this has put us in a better position to share our research results, methodology, and goals with other researchers, and to attract new talent to AI alignment work.

2015 General Activities
Beyond direct research support, in 2015 we:

Transitioned to new leadership under Nate Soares. The transition went smoothly, attesting to our organizational robustness.2
Moved to larger offices and hired an office manager to support our growing team.
Gave talks and participated in panel discussions at EA Global and at ITIF.
Published 20 new strategic and expository pieces: 11 strategic analyses, four MIRI strategy overviews, two interviews, a new MIRI FAQ, a new About MIRI page, and an annotated research agenda bibliography. Some of our best introductory posts have been collected on intelligence.org/info. Eliezer also contributed to Edge.org and Nate participated in an Ask Me Anything on the EA Forum.
Released Rationality: From AI to Zombies on the eve of Eliezer's completion of Harry Potter and the Methods of Rationality. We also concluded a reading group for Nick Bostrom's Superintelligence.
Revamped many of our online tools and webpages: AI Impacts, A Guide to MIRI's Research, and our Get Involved page and general application form. We also launched a MIRI technical results mailing list.
Were prominently cited in the "Research Priorities for Robust and Beneficial Artificial Intelligence" report, and were initial signatories of the attached AI safety open letter. Some of our basic AI forecasting arguments were subsequently echoed by Open Philanthropy Project analysts.
Received press coverage from The Atlantic, Nautilus, MIT Technology Review, Financial Times, Slate, PC World, National Post, CBC News, Tech Times, and the Discover blog.

Although we have deemphasized outreach efforts, we continue to expect these activities to be useful for spreading general awareness about MIRI, our research program, and AI safety research more generally. Ultimately, we expect this to help build our donor base, as well as attract potential future researchers (to MIRI and the field more generally), as with our past outreach and capacity-building efforts.

2015 Fundraising
I am very pleased with our fundraising performance. In 2015 we:

Continued our strong fundraising growth, with a total of $1,584,109 in contributions.3
Received $166,943 in grants from the Future of Life Institute (FLI), with another ~$80,000 annually for the next two years.4
Experimented with a new kind of fundraiser (non-matching, with multiple targets). I consider these experiments to have been successful. Our summer fundraiser was our biggest fundraiser of to date, raising $632,011, and our winter fundraiser also went well, raising $328,148.

Total contributions grew 28% in 2015. This was driven by an increase in contributions from new funders, including a one-time $219,000 contribution from an anonymous funder, $166,943 in FLI grants, and at least $137,023 from Raising for Effective Giving (REG) and regranting from the Effective Altruism Foundation.5 The decrease in contributions from returning funders is due to Peter Thiel's discontinuation of support in 2015, plus a large one-time outlier donation from Jed McCaleb in the years prior ($526,316 arriving in 2013, $104,822 in 2014).
Drawing conclusions from these year-by-year comparisons is a little tricky. MIRI underwent significant organizational changes over this time span, particularly in 2013. We switched to accrual-based accounting in 2014, which also complicates comparisons with previous years.6 In general, though, we're continuing to see solid fundraising growth.

The number of new funders decreased from 2014 to 2015. In our 2014 review, Luke explains the large increase in funders in 2014:
New donor growth was strong in 2014, though this mostly came from small donations made during the SV Gives fundraiser. A significant portion of growth in returning donors can also be attributed to lapsed donors making small contributions during the SV Gives fundraiser.
Comparing our numbers in 2015 and 2013, we see healthy growth in the number of returning funders and total number of funders.

The above chart shows contributions in past years from small, mid-sized, large, and very large funder segments. Contributions from the three largest segments increased (approximately) proportionally from last year, with the notable exception of contributions from large funders, which increased from 26% to 31% of total contributions. We had a small year-over-year decrease in contributions in the small funder segment, which is again due having received an unusually large amount of small contributions during SV Gives in 2014.
As in past years, a full report on our finances (in the form of an independent accountant's review report) will be made available on our transparency and financials page. The report will most likely be up in late August or early September.

2016 and Beyond
What's next? Beyond our research goal of making significant progress in five of our six focus areas, we set the following operational goals for ourselves in July/August 2015:

Accelerated growth: "expand to a roughly ten-person core research team." (source)
Type theory in type theory project: "hire one or two type theorists to work on developing relevant tools full-time." (source)
Visiting scholar program: "have interested professors drop by for the summer, while we pay their summer salaries and work with them on projects where our interests overlap." (source)
Independent review: "We're also looking into options for directly soliciting public feedback from independent researchers regarding our research agenda and early results." (source)
Higher-visibility publications: "Our current plan this year is to focus on producing a few high-quality publications in elite venues." (source)

In 2015 we doubled the size of our research team from three to six. With the restructuring of our research associates program and the addition of two research interns, I'm pleased with the growth we achieved in 2015. We deemphasized growth in the first half of 2016 in order to focus on onboarding, but plan to expand again by the end of the year.
We have a job ad out for our type theorist position, which will likely be filled after we make our next few core researcher hires. In the interim, we've been having our research intern Jack Gallagher work on the type theory in type theory project, and we also ran an April 2016 type theory workshop.
With help from our research advisors, our visiting scholars program morphed into a three-week-long colloquium series. Rather than hosting a handful of researchers for longer periods of time, we hosted over fifty researchers for shorter stretches of time, comparing notes on a wide variety of active AI safety research projects. Speakers at the event included Stuart Russell, Francesca Rossi, Tom Dietterich, and Bart Selman. We're also collaborating with Stuart Russell on a corrigibility grant.
Work is underway on conducting an external review of our research program; the results should be available in the next few months.
With regards to our fifth goal, in addition to "Proof-Producing Reflection for HOL" (which was presented at ITP 2015 in late August), we've since published papers at LORI-V ("Reflective Oracles"), at UAI 2016 ("Safely Interruptible Agents" and "A Formal Solution to the Grain of Truth Problem"), and at an IJCAI 2016 workshop ("The Value Learning Problem"). Of those venues, UAI is generally considered more prestigious than most venues that we have published in in the past. I'd count this as moderate (but not great) progress towards the goal of publishing in more elite venues. Nate will have more to say about our future publication plans.
Elaborating further on our plans would take me beyond the scope of this review. In the coming weeks, Nate will be providing more details on our 2016 activities and our goals going forward in a big-picture MIRI strategy post.7
This paper was originally titled "Aligning Superintelligence with Human Interests." We've renamed it in order to emphasize that this research agenda takes a specific approach to the alignment problem, and other approaches are possible too—including, relevantly, Jessica Taylor's new "Alignment for Advanced Machine Learning Systems" agenda.I (Malo Bourgon) more recently took on a leadership role as MIRI's new COO and second-in-command.$80,480 of this was earmarked funding for the AI Impacts project.MIRI is administering three FLI grants (and participated in a fourth). We are to receive $250,000 over three years to fund work on our agent foundations technical agenda, $49,310 towards AI Impacts, and we are administering Ramana's $36,750 to study self-reference in the HOL theorem prover in collaboration with Benya.This only counts direct contributions through REG to MIRI. REG's support for MIRI is likely closer to $200,000 when accounting for contributions made directly to MIRI as a result of REG's advice to funders.Also note that numbers in this section might not exactly match previously published estimates, since small corrections are often made to contributions data. Finally, note that these number do not include in-kind donations.My thanks to Rob Bensinger for his substantial contributions to this review.The post 2015 in review appeared first on Machine Intelligence Research Institute.

2015 in review


New paper: "Alignment for advanced machine learning systems"

MIRI's research to date has focused on the problems that we laid out in our late 2014 research agenda, and in particular on formalizing optimal reasoning for bounded, reflective decision-theoretic agents embedded in their environment. Our research team has since grown considerably, and we have made substantial progress on this agenda, including a major breakthrough in logical uncertainty that we will be announcing in the coming weeks.
Today we are announcing a new research agenda, "Alignment for advanced machine learning systems." Going forward, about half of our time will be spent on this new agenda, while the other half is spent on our previous agenda. The abstract reads:
We survey eight research areas organized around one question: As learning systems become increasingly intelligent and autonomous, what design principles can best ensure that their behavior is aligned with the interests of the operators? We focus on two major technical obstacles to AI alignment: the challenge of specifying the right kind of objective functions, and the challenge of designing AI systems that avoid unintended consequences and undesirable behavior even in cases where the objective function does not line up perfectly with the intentions of the designers.
Open problems surveyed in this research proposal include: How can we train reinforcement learners to take actions that are more amenable to meaningful assessment by intelligent overseers? What kinds of objective functions incentivize a system to "not have an overly large impact" or "not have many side effects"? We discuss these questions, related work, and potential directions for future research, with the goal of highlighting relevant research topics in machine learning that appear tractable today.
Co-authored by Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch, our new report discusses eight new lines of research (previously summarized here). Below, I'll explain the rationale behind these problems, as well as how they tie in to our old research agenda and to the new "Concrete problems in AI safety" agenda spearheaded by Dario Amodei and Chris Olah of Google Brain.

Increasing safety by reducing autonomy
The first three research areas focus on issues related to act-based agents, notional systems that base their behavior on their users' short-term instrumental preferences:
1. Inductive ambiguity identification: How can we train ML systems to detect and notify us of cases where the classification of test data is highly under-determined from the training data?
2. Robust human imitation: How can we design and train ML systems to effectively imitate humans who are engaged in complex and difficult tasks?
3. Informed oversight: How can we train a reinforcement learning system to take actions that aid an intelligent overseer, such as a human, in accurately assessing the system's performance?
These three problems touch on different ways we can make tradeoffs between capability/autonomy and safety. At one extreme, a fully autonomous, superhumanly capable system would make it uniquely difficult to establish any strong safety guarantees. We could reduce risk somewhat by building systems that are still reasonably smart and autonomous, but will pause to consult operators in cases where their actions are especially high-risk. Ambiguity identification is one approach to fleshing out which scenarios are "high-risk": ones where a system's experiences to date are uninformative about some fact or human value it's trying to learn.
At the opposite extreme, we can consider ML systems that are no smarter than their users, and take no actions other than what their users would do, or what their users would tell them to do. If we can correctly design a system to do what it thinks a trusted, informed human would do, we can trade away some of the potential benefits of advanced ML systems in exchange for milder failure modes.
These two extremes, human imitation and (mostly) autonomous goal pursuit, are useful objects of study because they help simplify and factorize out key parts of the problem. In practice, however, ambiguity identification is probably too mild a restriction on its own, and strict human imitation probably isn't efficiently implementable. Informed oversight considers more moderate approaches to keeping humans in the loop: designing more transparent ML systems that help operators understand the reasons behind selected actions.
Increasing safety without reducing autonomy
Whatever guarantees we buy by looping humans into AI systems' decisions, we will also want to improve systems' reliability in cases where oversight is unfeasible. Our other five problems focus on improving the reliability and error-tolerance of systems autonomously pursuing real-world goals, beginning with the problem of specifying such goals in a robust and reliable way:
4. Generalizable environmental goals: How can we create systems that robustly pursue goals defined in terms of the state of the environment, rather than defined directly in terms of their sensory data?
5. Conservative concepts: How can a classifier be trained to develop useful concepts that exclude highly atypical examples and edge cases?
6. Impact measures: What sorts of regularizers incentivize a system to pursue its goals with minimal side effects?
7. Mild optimization: How can we design systems that pursue their goals "without trying too hard"—stopping when the goal has been pretty well achieved, as opposed to expending further resources searching for ways to achieve the absolute optimum expected score?
8. Averting instrumental incentives: How can we design and train systems such that they robustly lack default incentives to manipulate and deceive their operators, compete for scarce resources, etc.?
Whereas ambiguity-identifying learners are designed to predict potential ways they might run into edge cases and defer to human operators in those cases, conservative learners are designed to err in a safe direction in edge cases. If a cooking robot notices the fridge is understocked, should it try to cook the cat? The ambiguity identification approach says to notice when the answer to "Are cats food?" is unclear, and pause to consult a human operator; the conservative concepts approach says to just assume cats aren't food in uncertain cases, since it's safer for cooking robots to underestimate how many things are food than to overestimate it. It remains unclear, however, how one might formalize this kind of reasoning.
Impact measures provide another avenues for limiting the potential scope of AI mishaps. If we can define some measure of "impact," we could design systems that can distinguish intuitively high-impact actions from low-impact ones and generally choose lower-impact options.
Alternatively, instead of designing systems to try as hard as possible to have a low impact, we might design "mild" systems that simply don't try very hard to do anything. Limiting the resources a system will put into its decision (via mild optimization) is distinct from limiting how much change a system will decide to cause (via impact measures); both are under-explored risk reduction approaches.
Lastly, we will explore a variety of different approaches to preventing default system incentives to treat operators adversarially under the "averting instrumental incentives" umbrella category. Our hope in pursuing all of these research directions simultaneously is that systems combining these features will permit much higher confidence than systems implementing any one of them. This approach also serves as a hedge in case some of these problems turn out to be unsolvable in practice, and allows for ideas that worked well on one problem to be re-applied on others.
Connections to other research agendas
Our new technical agenda, our 2014 agenda, and "Concrete problems in AI safety" take different approaches to the problem of aligning AI systems with human interests, though there is a fair bit of overlap between the research directions they propose.
We've changed the name of our 2014 agenda to "Agent foundations for aligning machine intelligence with human interests" (from "Aligning superintelligence with human interests") to help highlight the ways it is and isn't similar to our newer agenda. For reasons discussed in our advance announcement of "Alignment for advanced machine learning systems," our new agenda is intended to help more in scenarios where advanced AI is relatively near and relatively directly descended from contemporary ML techniques, while our agent foundations agenda is more agnostic about when and how advanced AI will be developed.
As we recently wrote, we believe that developing a basic formal theory of highly reliable reasoning and decision-making "could make it possible to get very strong guarantees about the behavior of advanced AI systems — stronger than many currently think is possible, in a time when the most successful machine learning techniques are often poorly understood." Without such a theory, AI alignment will be a much more difficult task.
The authors of "Concrete problems in AI safety" write that their own focus "is on the empirical study of practical safety problems in modern machine learning systems, which we believe is likely to be robustly useful across a broad variety of potential risks, both short- and long-term." Their paper discusses a number of the same problems as the alignment for ML agenda (or closely related ones), but directed more toward building on existing work and finding applications in present-day systems.
Where the agent foundations agenda can be said to follow the principle "start with the least well-understood long-term AI safety problems, since those seem likely to require the most work and are the likeliest to seriously alter our understanding of the overall problem space," the concrete problems agenda follows the principle "start with the long-term AI safety problems that are most applicable to systems today, since those problems are the easiest to connect to existing work by the AI research community."
Taylor et al.'s new agenda is less focused on present-day and near-future systems than "Concrete problems in AI safety," but is more ML-oriented than the agent foundations agenda. This chart helps map some of the correspondences between the topics the agent foundations agenda (plain text), the concrete problems agenda (italics), and the alignment for ML agenda (bold) discuss:
Work related to high reliability

realistic world-models ~ generalizable environmental goals ~ avoiding reward hacking

naturalized induction
ontology identification

decision theory
logical uncertainty
Vingean reflection

Work related to error tolerance

inductive ambiguity identification = ambiguity identification ~ robustness to distributional change
robust human imitation
informed oversight ~ scalable oversight
conservative concepts
impact measures = domesticity ~ avoiding negative side effects
mild optimization
averting instrumental incentives

corrigibility

safe exploration

"~" notes (sometimes very rough) similarities and correspondences, while "=" notes different names for the same concept.
As an example, "realistic world-models" and "generalizable environmental goals" are both aimed at making the environment and goal representations of reinforcement learning formalisms like AIXI more robust, and both can be viewed as particular strategies for avoiding reward hacking. Our work under the agent foundations agenda has mainly focused on formal models of AI systems in settings without clear agent/environment boundaries (naturalized induction), while our work under the new agenda will focus more on the construction of world-models that admit of the specification of goals that are environmental rather than simply perceptual (ontology identification).
For a fuller discussion of the relationship between these research topics, see Taylor et al.'s paper.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New paper: "Alignment for advanced machine learning systems" appeared first on Machine Intelligence Research Institute.

New paper: “Alignment for advanced machine learning systems”


Submission to the OSTP on AI outcomes

The White House Office of Science and Technology Policy recently put out a request for information on "(1) The legal and governance implications of AI; (2) the use of AI for public good; (3) the safety and control issues for AI; (4) the social and economic implications of AI;" and a variety of related topics. I've reproduced MIRI's submission to the RfI below:

I. Review of safety and control concerns
AI experts largely agree that AI research will eventually lead to the development of AI systems that surpass humans in general reasoning and decision-making ability. This is, after all, the goal of the field. However, there is widespread disagreement about how long it will take to cross that threshold, and what the relevant AI systems are likely to look like (autonomous agents, widely distributed decision support systems, human/AI teams, etc.).
Despite the uncertainty, a growing subset of the research community expects that advanced AI systems will give rise to a number of foreseeable safety and control difficulties, and that those difficulties can be preemptively addressed by technical research today. Stuart Russell, co-author of the leading undergraduate textbook in AI and professor at U.C. Berkeley, writes:
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what you ask for, not what you want.
Researchers' worries about the impact of AI in the long term bear little relation to the doomsday scenarios most often depicted in Hollywood movies, in which "emergent consciousness" allows machines to throw off the shackles of their programmed goals and rebel. The concern is rather that such systems may pursue their programmed goals all too well, and that the programmed goals may not match the intended goals, or that the intended goals may have unintended negative consequences.
These challenges are not entirely novel. We can compare them to other principal-agent problems where incentive structures are designed with the hope that blind pursuit of those incentives promotes good outcomes. Historically, principal-agent problems have been difficult to solve even in domains where the people designing the incentive structures can rely on some amount of human goodwill and common sense. Consider the problem of designing tax codes to have reliably beneficial consequences, or the problem of designing regulations that reliably reduce corporate externalities. Advanced AI systems naively designed to optimize some objective function could result in unintended consequences that occur on digital timescales, but without goodwill and common sense to blunt the impact.
Given that researchers don't know when breakthroughs will occur, and given that there are multiple lines of open technical research that can be pursued today to address these concerns, we believe it is prudent to begin serious work on those technical obstacles to improve the community's preparedness.

II. Technical research directions for safety and control
There are several promising lines of technical research that may help ensure that the AI systems of the future have a positive social impact. We divide this research into three broad categories:

Value specification (VS): research that aids in the design of objective functions that capture the intentions of the operators, and/or that describe socially beneficial goals. Example: cooperative inverse reinforcement learning, a formal model of AI agents that inductively learn the goals of other agents (e.g., human operators).

High reliability (HR): research that aids in the design of AI systems that robustly, reliably, and verifiably pursue the given objectives. Example: the PAC learning framework, which gives statistical guarantees about the correctness of solutions to certain types of classification problems. This framework is a nice example of research done far in advance of the development of advanced AI systems that is nevertheless likely to aid in the design of systems that are robust and reliable.

Error tolerance (ET): research that aids in the design of AI systems that are fail-safe and robust to design errors. Example: research into the design of objective functions that allow an agent to be shut down, but do not give that agent incentives to cause or prevent shutdown.

Our "Agent foundations for aligning machine intelligence with human interests" report discusses these three targets in depth, and outlines some neglected technical research topics that are likely to be relevant to the future design of robustly beneficial AI systems regardless of their specific architecture. Our "Alignment for advanced machine learning systems" report discusses technical research topics relevant to these questions under the stronger assumption that the advanced systems of the future will be qualitatively similar to modern-day machine learning (ML) systems. We also recommend a research proposal led by Dario Amodei and Chris Olah of Google Brain, "Concrete problems in AI safety," for technical research problems that are applicable to near-future AI systems and are likely to also be applicable to more advanced systems down the road. Actionable research directions discussed in these agendas include (among many other topics):
– robust inverse reinforcement learning: designing reward-based agents to learn human values in contexts where observed behavior may reveal biases or ignorance in place of genuine preferences. (VS)
– safe exploration: designing reinforcement learning agents to efficiently learn about their environments without performing high-risk experiments. (ET)
– low-impact agents: specifying decision-making systems that deliberately avoid having a large impact, good or bad, on their environment. (ET)
There are also a number of research areas that would likely aid in the development of safe AI systems, but which are not well-integrated into the existing AI community. As an example, many of the techniques in use by the program verification and high-assurance software communities cannot be applied to modern ML algorithms. Fostering more collaboration between these communities is likely to make it easier for us to design AI systems suitable for use in safety-critical situations. Actionable research directions for ML analysis and verification include:
– algorithmic transparency: developing more formal tools for analyzing how and why ML algorithms perform as they do. (HR)
– type theory for program verification: developing high-assurance techniques for the re-use of verified code in new contexts. (HR)
– incremental re-verification: confirming the persistence of safety properties for adaptive systems. (HR)
Another category of important research for AI reliability is the development of basic theoretical tools for formally modeling intelligent agents. As an example, consider the interaction of probability theory (a theoretical tool for modeling uncertain reasoners) with modern machine learning algorithms. While modern ML systems do not strictly follow the axioms of probability theory, many of the theoretical guarantees that can be applied to them are probability-theoretic, taking the form "this agent will converge on a policy that is very close to the optimal policy, with very high probability." Probability theory is an example of basic research that was developed far in advance of present-day ML techniques, but has proven important for attaining strong (statistical) guarantees about the behavior of ML systems. We believe that more basic research of this kind can be done, and that it could prove to be similarly valuable.
There are a number of other aspects of good reasoning where analogous foundations are lacking, such as situations where AI systems have to allocate attention given limited computational resources, or predict the behavior of computations that are too expensive to run, or analyze the effects of potential alterations to their hardware or software. Further research into basic theoretical models of ideal reasoning (including research into bounded rationality) could yield tools that would help attain stronger theoretical guarantees about AI systems' behavior. Actionable research directions include:
– decision theory: giving a formal account of reasoning in settings where an agent must engage in metacognition, reflection, self-modification, or reasoning about violations of the agent/environment boundary. (HR)
– logical uncertainty: generalizing Bayesian probability theory to settings where agents are uncertain about mathematical (e.g., computational) facts. (HR)
We believe that there are numerous promising avenues of foundational research which, if successful, could make it possible to get very strong guarantees about the behavior of advanced AI systems — stronger than many currently think is possible, in a time when the most successful machine learning techniques are often poorly understood. We believe that bringing together researchers in machine learning, program verification, and the mathematical study of formal agents would be a large step towards ensuring that highly advanced AI systems will have a robustly beneficial impact on society.

III. Coordination prospects
It is difficult to say much with confidence about the long-term impact of AI. For now, we believe that the lines of technical research outlined above are the best available tool for addressing concerns about advanced AI systems, and for learning more about what needs to be done.
Looking ahead, we expect the risks associated with transformative AI systems in the long term to be exacerbated if the designers of such systems (be they private-sector, public-sector, or part of some international collaboration) act under excessive time pressure. It is our belief that any policy designed to ensure that the social impact of AI is beneficial should first and foremost ensure that transformative AI systems are deployed with careful consideration, rather than in fear or haste. If scientists and engineers are worried about losing a race to the finish, they will have more incentives to cut corners on safety and control, obviating the benefits of safety-conscious work.
In the long term, we recommend that policymakers make use of incentives to encourage designers of AI systems to work together cooperatively, perhaps through multinational and multicorporate collaborations, in order to discourage the development of race dynamics. In light of high levels of uncertainty about the future of AI among experts, and in light of the large potential of AI research to save lives, solve social problems, and serve the common good in the near future, we recommend against broad regulatory interventions in this space. We recommend that effort instead be put towards encouraging interdisciplinary technical research into the AI safety and control challenges that we have outlined above.

The post Submission to the OSTP on AI outcomes appeared first on Machine Intelligence Research Institute.

Submission to the OSTP on AI outcomes


July 2016 Newsletter

Research updates

A new paper: "A Formal Solution to the Grain of Truth Problem." The paper was presented at UAI-16, and describes the first general reduction of game-theoretic reasoning to expected utility maximization.
Participants in MIRI's recently-concluded Colloquium Series on Robust and Beneficial AI (CSRBAI) have put together AI safety environments for the OpenAI Reinforcement Learning Gym.1 Help is welcome creating more safety environments and conducting experiments on the current set. Questions can be directed to <EMAIL>.

General updates

We attended the White House's Workshop on Safety and Control in AI.
Our 2016 MIRI Summer Fellows Program recently drew to a close. The program, run by the Center for Applied Rationality, aims to train AI scientists' and mathematicians' research and decision-making skills.
"Why Ain't You Rich?": Nate Soares discusses decision theory in Dawn or Doom. See "Toward Idealized Decision Theory" for context.
Numerai, an anonymized distributed hedge fund for machine learning researchers, has added an option for donating earnings to MIRI "as a hedge against things going horribly right" in the field of AI.

News and links

The White House is requesting information on "safety and control issues for AI," among other questions. Public submissions will be accepted through July 22.
"Concrete Problems in AI Safety": Researchers from Google Brain, OpenAI, and academia propose a very promising new AI safety research agenda. The proposal is showcased on the Google Research Blog and the OpenAI Blog, as well as the Open Philanthropy Blog, and has received press coverage from Bloomberg, The Verge, and MIT Technology Review.
After criticizing the thinking behind OpenAI earlier in the month, Alphabet executive chairman Eric Schmidt comes out in favor of AI safety research:

Do we worry about the doomsday scenarios? We believe it's worth thoughtful consideration. Today's AI only thrives in narrow, repetitive tasks where it is trained on many examples. But no researchers or technologists want to be part of some Hollywood science-fiction dystopia. The right course is not to panic—it's to get to work. Google, alongside many other companies, is doing rigorous research on AI safety, such as how to ensure people can interrupt an AI system whenever needed, and how to make such systems robust to cyberattacks.

Dylan Hadfield-Mennell, Anca Dragan, Pieter Abbeel, and Stuart Russell propose a formal definition of the value alignment problem as "Cooperative Inverse Reinforcement Learning," a two-player game where a human and robot are both "rewarded according to the human's reward function, but the robot does not initially know what this is." In a CSRBAI talk (slides), Hadfield-Mennell discusses applications for AI corrigibility.
Jaan Tallinn brings his AI risk focus to the Bulletin of Atomic Scientists.
Stephen Hawking weighs in on intelligence explosion (video). Sam Harris and Neil DeGrasse Tyson debate the idea at greater length (audio, at 1:22:37).
Ethereum developer Vitalik Buterin discusses the implications of value complexity and fragility and other AI safety concepts for cryptoeconomics.
Wired covers a "demonically clever" backdoor based on chips' analog properties.
CNET interviews MIRI and a who's who of AI scientists for a pair of articles: "AI, Frankenstein? Not So Fast, Experts Say" and "When Hollywood Does AI, It's Fun But Farfetched."
Next month's Effective Altruism Global conference is accepting applicants.

Inspiration for these gyms came in part from Chris Olah and Dario Amodei in a conversation with Rafael.The post July 2016 Newsletter appeared first on Machine Intelligence Research Institute.

July 2016 Newsletter


New paper: "A formal solution to the grain of truth problem"

Future of Humanity Institute Research Fellow Jan Leike and MIRI Research Fellows Jessica Taylor and Benya Fallenstein have just presented new results at UAI 2016 that resolve a longstanding open problem in game theory: "A formal solution to the grain of truth problem."
Game theorists have techniques for specifying agents that eventually do well on iterated games against other agents, so long as their beliefs contain a "grain of truth" — nonzero prior probability assigned to the actual game they're playing. Getting that grain of truth was previously an unsolved problem in multiplayer games, because agents can run into infinite regresses when they try to model agents that are modeling them in turn. This result shows how to break that loop: by means of reflective oracles.
In the process, Leike, Taylor, and Fallenstein provide a rigorous and general foundation for the study of multi-agent dilemmas. This work provides a surprising and somewhat satisfying basis for approximate Nash equilibria in repeated games, folding a variety of problems in decision and game theory into a common framework.
The paper's abstract reads:
A Bayesian agent acting in a multi-agent environment learns to predict the other agents' policies if its prior assigns positive probability to them (in other words, its prior contains a grain of truth). Finding a reasonably large class of policies that contains the Bayes-optimal policies with respect to this class is known as the grain of truth problem. Only small classes are known to have a grain of truth and the literature contains several related impossibility results.
In this paper we present a formal and general solution to the full grain of truth problem: we construct a class of policies that contains all computable policies as well as Bayes-optimal policies for every lower semicomputable prior over the class. When the environment is unknown, Bayes-optimal agents may fail to act optimally even asymptotically. However, agents based on Thompson sampling converge to play ε-Nash equilibria in arbitrary unknown computable multi-agent environments. While these results are purely theoretical, we show that they can be computationally approximated arbitrarily closely.
Traditionally, when modeling computer programs that model the properties of other programs (such as when modeling an agent reasoning about a game), the first program is assumed to have access to an oracle (such as a halting oracle) that can answer arbitrary questions about the second program. This works, but it doesn't help with modeling agents that can reason about each other.
While a halting oracle can predict the behavior of any isolated Turing machine, it cannot predict the behavior of another Turing machine that has access to a halting oracle. If this were possible, the second machine could use its oracle to figure out what the first machine-oracle pair thinks it will do, at which point it can do the opposite, setting up a liar paradox scenario. For analogous reasons, two agents with similar resources, operating in real-world environments without any halting oracles, cannot perfectly predict each other in full generality.
Game theorists know how to build formal models of asymmetric games between a weaker player and a stronger player, where the stronger player understands the weaker player's strategy but not vice versa. For the reasons above, however, games between agents of similar strength have resisted full formalization. As a consequence of this, game theory has until now provided no method for designing agents that perform well on complex iterated games containing other agents of similar strength.

Usually, the way to build an ideal agent is to have the agent consider a large list of possible policies, predict how the world would respond to each policy, and then choose the best policy by some metric. However, in multi-player games, if your agent considers a big list of policies that both it and the opponent might play, then the best policy for the opponent is usually some alternative policy that was not in your list. (And if you add that policy to your list, then the new best policy for the opponent to play is now a new alternative that wasn't in the list, and so on.)
This is the grain of truth problem, first posed by Kalai and Lehrer in 1993: define a class of policies that is large enough to be interesting and realistic, and for which the best response to an agent that considers that policy class is inside the class.1
Taylor and Fallenstein have developed a formalism that enables a solution: reflective oracles capable of answering questions about agents with access to equivalently powerful oracles. Leike has led work on demonstrating that this formalism can solve the grain of truth problem, and in the process shows that the Bayes-optimal policy generally does not converge to a Nash equilibrium. Thompson sampling, however, does converge to a Nash equilibrium — a result that comes out of another paper presented at UAI 2016, Leike, Lattimore, Orseau, and Hutter's "Thompson sampling is asymptotically optimal in general environments."
The key feature of reflective oracles is that they avoid diagonalization and paradoxes by randomizing in the relevant cases.2 This allows agents with access to a reflective oracle to consistently reason about the behavior of arbitrary agents that also have access to a reflective oracle, which in turn makes it possible to model agents that converge to Nash equilibria by their own faculties (rather than by fiat or assumption).
This framework can be used to, e.g., define games between multiple copies of AIXI. As originally formulated, AIXI cannot entertain hypotheses about its own existence, or about the existence of similarly powerful agents; classical Bayes-optimal agents must be larger and more intelligent than their environments. With access to a reflective oracle, however, Fallenstein, Soares, and Taylor have shown that AIXI can meaningfully entertain hypotheses about itself and copies of itself while avoiding diagonalization.
The other main novelty of this paper is that reflective oracles turn out to be limit-computable, and so allow for approximation by anytime algorithms. The reflective oracles paradigm is therefore likely to be quite valuable for investigating game-theoretic questions involving generally intelligent agents that can understand and model each other.3

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

It isn't hard to solve the grain of truth problem for policy classes that are very small. Consider a prisoner's dilemma where the only strategies the other player can select take the form "cooperate until the opponent defects, then defect forever" or "cooperate n times in a row (or until the opponent defects, whichever happens first) and then defect forever." Leike, Taylor, and Fallenstein note:
The Bayes-optimal behavior is to cooperate until the posterior belief that the other agent defects in the time step after the next is greater than some constant (depending on the discount function) and then defect afterwards.
But this is itself a strategy in the class under consideration. If both players are Bayes-optimal, then both will have a grain of truth (i.e., their actual strategy is assigned nonzero probability by the other player) "and therefore they converge to a Nash equilibrium: either they both cooperate forever or after some finite time they both defect forever."
Slightly expanding the list of policies an agent might deploy, however, can make it hard to find a policy class that contains a grain of truth. For example, if "tit for tat" is added to the policy class, then, depending on the prior, the grain of truth may be lost. In this case, if the first agent thinks that the second agent is very likely "always defect" but maybe "tit for tat," then the best policy might be something like "defect until they cooperate, then play tit for tat," but this policy is not in the policy class. The question resolved by this paper is how to find priors that contain a grain of truth for much richer policy classes.Specifically, reflective oracles output 1 if a specified machine returns 1 with probability greater than a specified probability p, and they output 0 if the probability that the machine outputs 0 is greater than 1-p. When the probability is exactly p, however—or the machine has some probability of not halting, and p hits this probability mass—the oracle can output 0, 1, or randomize between the two. This allows reflective oracles to avoid probabilistic versions of the liar paradox: any attempt to ask the reflective oracle an unanswerable question will yield a meaningless placeholder answer.Thanks to Tsvi Benson-Tilsen, Chana Messinger, Nate Soares, and Jan Leike for helping draft this announcement.The post New paper: "A formal solution to the grain of truth problem" appeared first on Machine Intelligence Research Institute.

New paper: “A formal solution to the grain of truth problem”


June 2016 Newsletter

Research updates

New paper: "Safely Interruptible Agents." The paper will be presented at UAI-16, and is a collaboration between Laurent Orseau of Google DeepMind and Stuart Armstrong of the Future of Humanity Institute (FHI) and MIRI; see FHI's press release. The paper has received (often hyperbolic) coverage from a number of press outlets, including Business Insider, Motherboard, Newsweek, Gizmodo, BBC News, eWeek, and Computerworld.
New at IAFF: All Mathematicians are Trollable: Divergence of Naturalistic Logical Updates; Two Problems with Causal-Counterfactual Utility Indifference
New at AI Impacts: Metasurvey: Predict the Predictors; Error in Armstrong and Sotala 2012
Marcus Hutter's research group has released a new paper based on results from a MIRIx workshop: "Self-Modification of Policy and Utility Function in Rational Agents." Hutter's team is presenting several other AI alignment papers at AGI-16 next month: "Death and Suicide in Universal Artificial Intelligence" and "Avoiding Wireheading with Value Reinforcement Learning."
"Asymptotic Logical Uncertainty and The Benford Test" has been accepted to AGI-16.

General updates

MIRI and FHI's Colloquium Series on Robust and Beneficial AI (talk abstracts and slides now up) has kicked off with opening talks by Stuart Russell, Francesca Rossi, Tom Dietterich, and Alan Fern.
We visited FHI to discuss new results in logical uncertainty, our new machine-learning-oriented research program, and a range of other topics.

News and links

Following an increase in US spending on autonomous weapons, The New York Times reports that the Pentagon is turning to Silicon Valley for an edge.
IARPA director Jason Matheny, a former researcher at FHI, discusses forecasting and risk from emerging technologies (video).
FHI Research Fellow Owen Cotton-Barratt gives oral evidence to the UK Parliament on the need for robust and transparent AI systems.
Google reveals a hidden reason for AlphaGo's exceptional performance against Lee Se-dol: a new integrated circuit design that can speed up machine learning applications by an order of magnitude.
Elon Musk answers questions about SpaceX, Tesla, OpenAI, and more (video).
Why worry about advanced AI? Stuart Russell (in Scientific American), George Dvorsky (in Gizmodo), and SETI director Seth Shostak (in Tech Times) explain.
Olle Häggeström's new book, Here Be Dragons, serves as an unusually thoughtful and thorough introduction to existential risk and future technological development, including a lucid discussion of artificial superintelligence.
Robin Hanson examines the implications of widespread whole-brain emulation in his new book, The Age of Em: Work, Love, and Life when Robots Rule the Earth.
Bill Gates highly recommends Nick Bostrom's Superintelligence. The paperback edition is now out, with a newly added afterword.
FHI Research Associate Paul Christiano has joined OpenAI as an intern. Christiano has also written new posts on AI alignment: Efficient and Safely Scalable, Learning with Catastrophes, Red Teams, and The Reward Engineering Problem.

The post June 2016 Newsletter appeared first on Machine Intelligence Research Institute.

June 2016 Newsletter


New paper: "Safely interruptible agents"

Google DeepMind Research Scientist Laurent Orseau and MIRI Research Associate Stuart Armstrong have written a new paper on error-tolerant agent designs, "Safely interruptible agents." The paper is forthcoming at the 32nd Conference on Uncertainty in Artificial Intelligence.
Abstract:
Reinforcement learning agents interacting with a complex environment like the real world are unlikely to behave optimally all the time. If such an agent is operating in real-time under human supervision, now and then it may be necessary for a human operator to press the big red button to prevent the agent from continuing a harmful sequence of actions—harmful either for the agent or for the environment—and lead the agent into a safer situation. However, if the learning agent expects to receive rewards from this sequence, it may learn in the long run to avoid such interruptions, for example by disabling the red button — which is an undesirable outcome.
This paper explores a way to make sure a learning agent will not learn to prevent (or seek!) being interrupted by the environment or a human operator. We provide a formal definition of safe interruptibility and exploit the off-policy learning property to prove that either some agents are already safely interruptible, like Q-learning, or can easily be made so, like Sarsa. We show that even ideal, uncomputable reinforcement learning agents for (deterministic) general computable environments can be made safely interruptible.
Orseau and Armstrong's paper constitutes a new angle of attack on the problem of corrigibility. A corrigible agent is one that recognizes it is flawed or under development and assists its operators in maintaining, improving, or replacing itself, rather than resisting such attempts.

In the case of superintelligent AI systems, corrigibility is primarily aimed at averting unsafe convergent instrumental policies (e.g., the policy of defending its current goal system from future modifications) when such systems have incorrect terminal goals. This leaves us more room for approximate, trial-and-error, and learning-based solutions to AI value specification.
Interruptibility is an attempt to formalize one piece of the intuitive idea of corrigibility. Utility indifference (in Soares, Fallenstein, Yudkowsky, and Armstrong's "Corrigibility") is an example of a past attempt to define a different piece of corrigibility: systems that are indifferent to programmers' interventions to modify their terminal goals, and will therefore avoid trying to force their programmers either to make such modifications or to avoid such modifications. "Safely interruptible agents" instead attempts to define systems that are indifferent to programmers' interventions to modify their policies, and will not try to stop programmers from intervening on their everyday activities (nor try to force them to intervene).
Here the goal is to make the agent's policy converge to whichever policy is optimal if the agent believed there would be no future interruptions. Even if the agent has experienced interruptions in the past, it should act just as though it will never experience any further interruptions. Orseau and Armstrong show that several classes of agent are safely interruptible, or can be easily made safely interruptible.
Further reading:

Stuart Armstrong's Smarter Than Us, an informal introduction to the AI alignment problem.
Laurent Orseau on Artificial General Intelligence.
Orseau and Ring's "Space-time embedded intelligence."
Soares and Fallenstein's "Problems of self-reference in self-improving space-time embedded intelligence."

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New paper: "Safely interruptible agents" appeared first on Machine Intelligence Research Institute.

New paper: “Safely interruptible agents”


May 2016 Newsletter

Research updates

Two new papers split logical uncertainty into two distinct subproblems: "Uniform Coherence" and "Asymptotic Convergence in Online Learning with Unbounded Delays."
New at IAFF: An Approach to the Agent Simulates Predictor Problem; Games for Factoring Out Variables; Time Hierarchy Theorems for Distributional Estimation Problems
We will be presenting "The Value Learning Problem" at the IJCAI-16 Ethics for Artificial Intelligence workshop instead of the AAAI Spring Symposium where it was previously accepted.

General updates

We're launching a new research program with a machine learning focus. Half of MIRI's team will be investigating potential ways to specify goals and guard against errors in advanced neural-network-inspired systems.
We ran a type theory and formal verification workshop this past month.

News and links

The Open Philanthropy Project explains its strategy of high-risk, high-reward hits-based giving and its decision to make AI risk its top focus area this year.
Also from OpenPhil: Is it true that past researchers over-hyped AI? Is there a realistic chance of AI fundamentally changing civilization in the next 20 years?
From Wired: Inside OpenAI, and Facebook is Building AI That Builds AI.
The White House announces a public workshop series on the future of AI.
The Wilberforce Society suggests policies for narrow and general AI development.
Two new AI safety papers: "A Model of Pathways to Artificial Superintelligence Catastrophe for Risk and Decision Analysis" and "The AGI Containment Problem."
Peter Singer weighs in on catastrophic AI risk.
Digital Genies: Stuart Russell discusses the problems of value learning and corrigibility in AI.
Nick Bostrom is interviewed at CeBIT (video) and also gives a presentation on intelligence amplification and the status quo bias (video).
Jeff MacMahan critiques philosophical critiques of effective altruism.
Yale political scientist Allan Dafoe is seeking research assistants for a project on political and strategic concerns related to existential AI risk.
The Center for Applied Rationality is accepting applicants to a free workshop for machine learning researchers and students.

The post May 2016 Newsletter appeared first on Machine Intelligence Research Institute.

May 2016 Newsletter