June 2020 Newsletter

MIRI researcher Evan Hubinger reviews "11 different proposals for building safe advanced AI under the current machine learning paradigm", comparing them on outer alignment, inner alignment, training competitiveness, and performance competitiveness.
Other updates

We keep being amazed by new shows of support — following our last two announcements, MIRI has received a donation from another anonymous donor totaling ~$265,000 in euros, facilitated by Effective Giving UK and the Effective Altruism Foundation. Massive thanks to the donor for their generosity, and to both organizations for their stellar support for MIRI and other longtermist organizations!
Hacker News discusses Eliezer Yudkowsky's There's No Fire Alarm for AGI.
MIRI researcher Buck Shlegeris talks about deference and inside-view models on the EA Forum.
OpenAI unveils GPT-3, a massive 175-billion parameter language model that can figure out how to solve a variety of problems without task-specific training or fine-tuning. Gwern Branwen's pithy summary:
GPT-3 is terrifying because it's a tiny model compared to what's possible, trained in the dumbest way possible on a single impoverished modality on tiny data, yet the first version already manifests crazy runtime meta-learning—and the scaling curves still are not bending!
Further discussion by Branwen and by Rohin Shah.

Stuart Russell gives this year's Turing Lecture online, discussing "provably beneficial AI".

The post June 2020 Newsletter appeared first on Machine Intelligence Research Institute.

June 2020 Newsletter


May 2020 Newsletter

MIRI has received an anonymous donation of ~$275,000 in euros, facilitated by Effective Giving UK. Additionally, the Survival and Flourishing Fund, working with funders Jaan Tallinn and Jed McCaleb, has announced $340,000 in grants to MIRI. SFF is a new fund that is taking over much of BERI's grantmaking work.
To everyone involved in both decisions to support our research, thank you!
Other updates

An Orthodox Case Against Utility Functions: MIRI researcher Abram Demski makes the case against utility functions that rely on a microphysical "view from nowhere".
Stuart Armstrong's "If I were a well-intentioned AI…" sequence looks at alignment problems from the perspective of a well-intentioned but ignorant agent.
AI Impacts' Asya Bergal summarizes takeaways from "safety-by-default" researchers.
Agarwal and Norouzi report improvements in offline RL.
Daniel Kokotajlo's Three Kinds of Competitiveness distinguishes performance-competitive, cost-competitive, and date-competitive AI systems.
The Stanford Encyclopedia of Philosophy's new Ethics of AI and Robotics article includes a discussion of existential risk from AGI.

The post May 2020 Newsletter appeared first on Machine Intelligence Research Institute.

May 2020 Newsletter


April 2020 Newsletter

MIRI has been awarded its largest grant to date — $7,703,750 split over two years from Open Philanthropy, in partnership with Ben Delo, co-founder of the cryptocurrency trading platform BitMEX!
We have also been awarded generous grants by the Berkeley Existential Risk Initiative ($300,000) and the Long-Term Future Fund ($100,000). Our thanks to everybody involved!
Other updates

Buck Shlegeris of MIRI and Rohin Shah of CHAI discuss Rohin's 2018–2019 overview of technical AI alignment research on the AI Alignment Podcast.
From MIRI's Abram Demski: Thinking About Filtered Evidence Is (Very!) Hard and Bayesian Evolving-to-Extinction. And from Evan Hubinger: Synthesizing Amplification and Debate.
From OpenAI's Beth Barnes, Paul Christiano, Long Ouyang, and Geoffrey Irving: Progress on AI Safety via Debate.
Zoom In: An Introduction to Circuits: OpenAI's Olah, Cammarata, Schubert, Goh, Petrov, and Carter argue, "Features are the fundamental unit of neural networks. They correspond to directions [in the space of neuron activations]. […] Features are connected by weights, forming circuits. […] Analogous features and circuits form across models and tasks."
DeepMind's Agent57 appears to meet one of the AI benchmarks in AI Impacts' 2016 survey, "outperform professional game testers on all Atari games using no game specific knowledge", earlier than NeurIPS/ICML authors predicted.
From DeepMind Safety Research: Specification gaming: the flip side of AI ingenuity.

The post April 2020 Newsletter appeared first on Machine Intelligence Research Institute.

April 2020 Newsletter


MIRI's largest grant to date!

A big announcement today: MIRI has been awarded a two-year $7,703,750 grant by Open Philanthropy — our largest grant to date. In combination with the ~$1.06M Open Philanthropy is also disbursing to MIRI this year (the second half of their 2019 grant), this amounts to $4.38M per year over two years, or roughly 60% of our predicted 2020–2021 budgets.
While ~$6.24M of Open Philanthropy's new grant comes from their main funders, $1.46M was made possible by Open Philanthropy's new partnership with Ben Delo, co-founder of the cryptocurrency trading platform BitMEX. Ben Delo has teamed up with Open Philanthropy to support their long-termist grantmaking, which includes (quoting Open Philanthropy):
reducing potential risks from advanced artificial intelligence, furthering biosecurity and pandemic preparedness, and other initiatives to combat global catastrophic risks, as well as much of the work we fund on effective altruism.
We're additionally happy to announce a $300,000 grant from the Berkeley Existential Risk Initiative. I'll note that at the time of our 2019 fundraiser, we expected to receive a grant from BERI in early 2020, and incorporated this into our reserves estimates. However, we predicted the grant size would be $600k; now that we know the final grant amount, that estimate should be $300k lower.
Finally, MIRI has been awarded a $100,000 grant by the Effective Altruism Funds Long-Term Future Fund, managed by the Centre for Effective Altruism. The fund plans to release a write-up describing the reasoning for their new round of grants in a couple of weeks.
Our thanks to Open Phil, Ben Delo and Longview Philanthropy (Ben Delo's philanthropic advisor, formerly known as Effective Giving UK), BERI, and the Long-Term Future Fund for this amazing support! Going into 2020–2021, these new grants put us in an unexpectedly good position to grow and support our research team. To learn more about our growth plans, see our 2019 fundraiser post and our 2018 strategy update.
The post MIRI's largest grant to date! appeared first on Machine Intelligence Research Institute.

MIRI’s largest grant to date!


March 2020 Newsletter

As the COVID-19 pandemic progresses, the LessWrong team has put together a database of resources for learning about the disease and staying updated, and 80,000 Hours has a new write-up on ways to help in the fight against COVID-19. In my non-MIRI time, I've been keeping my own quick and informal notes on various sources' COVID-19 recommendations in this Google Doc. Stay safe out there!

Updates

My personal cruxes for working on AI safety: a talk transcript from MIRI researcher Buck Shlegeris.
Daniel Kokotajlo of AI Impacts discusses Cortés, Pizarro, and Afonso as Precedents for Takeover.
O'Keefe, Cihon, Garfinkel, Flynn, Leung, and Dafoe's "The Windfall Clause" proposes "an ex ante commitment by AI firms to donate a significant amount of any eventual extremely large profits" that result from "fundamental, economically transformative breakthroughs" like AGI.
Microsoft announces the 17-billion-parameter language model Turing-NLG.
Oren Etzioni thinks AGI is too far off to deserve much thought, and cites Andrew Ng's "overpopulation on Mars" metaphor approvingly — but he's also moving the debate in a very positive direction by listing specific observations that would make him change his mind.

The post March 2020 Newsletter appeared first on Machine Intelligence Research Institute.

March 2020 Newsletter


February 2020 Newsletter

Updates

Colm reviews our 2019 fundraiser: taking into account matching, we received a total of $601,120 from 250+ donors. Our thanks again for all the support!
Evan Hubinger's Exploring Safe Exploration clarifies points he raised in Safe Exploration and Corrigibility. The issues raised here are somewhat subtler than may be immediately apparent, since we tend to discuss things in ways that collapse the distinctions Evan is making.
Logician Arthur Milchior reviews the AIRCS workshop and MIRI's application process based on his first-hand experience with both. See also follow-up discussion with MIRI and CFAR staff.
Rohin Shah posts an in-depth 2018–19 review of the field of AI alignment.
From Shevlane and Dafoe's new "The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?":

[T]he existing conversation around AI has heavily borrowed concepts and conclusions from one particular field: vulnerability disclosure in computer security. We caution against AI researchers treating these lessons as immediately applicable. There are important differences between vulnerabilities in software and the types of vulnerabilities exploited by AI. […]
	Patches to software are often easy to create, and can often be made in a matter of weeks. These patches fully resolve the vulnerability. The patch can be easily propagated: for downloaded software, the software is often automatically updated over the internet; for websites, the fix can take effect immediately.
	[… F]or certain technologies, there is no low-cost, straightforward, effective defence. [… C]onsider biological research that provides insight into the manufacture of pathogens, such as a novel virus. A subset of viruses are very difficult to vaccinate for (there is still no vaccination for HIV) or otherwise prepare against. This lowers the defensive benefit of publication, by blocking a main causal pathway by which publication leads to greater protection. This contrasts with the case where an effective treatment can be developed within a reasonable time period[.]

Yann LeCun and Eliezer Yudkowsky discuss the concept "AGI".
CFAR's Anna Salamon contrasts "reality-revealing" and "reality-masking" puzzles.
Scott Alexander reviews Stuart Russell's Human Compatible.

Links from the research team

MIRI researchers anonymously summarize and comment on recent posts and papers:

Re ACDT: a hack-y acausal decision theory — "Stuart Armstrong calls this decision theory a hack. I think it might be more elegant than he's letting on (i.e., a different formulation could look less hack-y), and is getting at something."
Re Predictors exist: CDT going bonkers… forever — "I don't think Stuart Armstrong's example really adds much over some variants of Death in Damascus, but there's some good discussion of CDT vs. EDT stuff in the comments."
Re Is the term mesa optimizer too narrow? — "Matthew Barnett poses the important question, '[I]f even humans are not mesa optimizers, why should we expect mesa optimizers to be the primary real world examples of [malign generalization]?'"
Re Malign generalization without internal search — "I think Matthew Barnett's question here is an important one. I lean toward the 'yes, this is a problem' camp—I don't think we can entirely eliminate malign generalization by eliminating internal search. But it is possible that this falls into other categories of misalignment (which we don't want to term 'inner alignment')."
Re (A -> B) -> A in Causal DAGs and Formulating Reductive Agency in Causal Models — "I've wanted something like this for a while. Bayesian influence diagrams model agents non-reductively, by boldly asserting that some nodes are agentic. Can we make models which represent agents, without declaring a basic 'agent' type like that? John Wentworth offers an approach, representing agents via 'strange loops' across a use-mention boundary; and discusses how this might break down even further, with fully reductive agency. I'm not yet convinced that Wentworth has gotten it right, but it's exciting to see an attempt."

The post February 2020 Newsletter appeared first on Machine Intelligence Research Institute.

February 2020 Newsletter


Our 2019 Fundraiser Review

Our 2019 fundraiser ended on December 31 with the month-long campaign raising $601,1201 from over2 259 donors. While we didn't reach our $1M fundraising goal, the amount raised will be a very substantial help to us going into 2020. We're grateful to all who supported us during the fundraiser.

20% of our fundraiser total was raised in a single minute on December 3 during Facebook's Giving Tuesday matching event, deserving special mention. Facebook's matching pool of $7M was exhausted within 14 seconds of their event's 5am PST start but in spite of the shortness of the event, a cohort of lightning-fast early risers secured a significant amount of Facebook matching for MIRI this year; of the $77,325 donated to MIRI on Giving Tuesday, $45,915 was donated early enough to be matched by Facebook. Thank you to everybody who set their clocks early to support us3 and a shout out to the EA Giving Tuesday/Rethink Charity collaboration which helped direct $563k in matching funds to EA nonprofits on Giving Tuesday.

Beyond Giving Tuesday, we're very grateful to the Effective Altruism Foundation for providing a channel for residents of a number of EU countries including Germany, Switzerland, and the Netherlands to donate to MIRI in a tax-advantaged way and also to donors who leveraged their companies' matching generosity to maximum effect during the fundraiser.
Our fundraiser fell well short of our $1M target this year, and also short of our in-fundraiser support in 2018 ($947k) and 2017 ($2.5M). It's plausible that some of the following (non-mutually-exclusive) factors may have contributed to this, though we don't know the relative strength of these factors:

The value of cryptocurrency, especially ETH, was significantly lower during the fundraiser than in 2017. Some donors have explicitly told us they're waiting for more advantageous ETH prices before supporting us again.

MIRI's current nondisclosed-by-default research approach makes it difficult for some donors to financially support us at the moment, either because they disagree with the policy itself or because (absent more published research from MIRI) they feel like they personally lack the data they need to evaluate us against other giving opportunities. Several donors have voiced one or both of these sentiments to me, and they were cited prominently in this 2019 alignment research review.

The changes to US tax law in 2018 relating to individual deductions have caused some supporters to adjust the scale and timing of their donations, a trend noticed across US charitable giving in general. It's plausible that having future MIRI fundraisers straddle the new year (e.g. starting in 2020 and ending in 2021) might provide supporters with more flexibility in their giving; if you would personally find this helpful (or unhelpful), I'm interested to hear about it at <EMAIL>.

This fundraiser saw MIRI involved in less counterfactual matching opportunities than in previous years — one compared to three in 2018 — which may have reduced the attraction for some of our more leverage-sensitive supporters this time around.

Since MIRI's budget and size have grown a great deal over the past few years, some donors may think that we're hitting diminishing returns on marginal donations, at a rate that makes them want to look for other giving opportunities.

The scale of the funds received during MIRI fundraisers tends to be strongly affected by 1-4 large donors each year, a fair number of whom are one-time or sporadic donors. Since this has certainly given us higher-than-expected results on a number of past occasions, it's perhaps not surprising that such a randomness-heavy phenomenon would sometimes yield lower-than-expected support by chance. Specifically, we received no donations over $100,000 during this fundraiser, and the two donations over $50,000 were especially welcome.

Some MIRI supporters who had been previously following an earning-to-give strategy have moved to direct work as 80,000 Hours' developing thoughts on the subject continue to influence the EA community.

In past years, when answering supporters' questions about the discount rate on their potential donations to MIRI, we've leaned towards a "now > later" approach. This plausibly resulted in a front-loading of some donations in 2017 and 2018.

As always, I'm interested in hearing individual supporters' thoughts about how they personally are thinking about their giving strategy; you're welcome to shoot me an email at <EMAIL>
Overall, we're extremely grateful for all the support we received during this fundraiser. Although we didn't hit our target, the support we received allows us to continue pursuing the majority of our growth plans, with cash reserves of 1.2–1.4 years at the start of 2020. To everyone who contributed, thank you.

The exact total is still subject to change as we continue to process a small number of donations.

There were significantly more anonymous donors than in previous years — plausibly a result of new data protection legislation like the European Union's GDPR and California's CCPA — which we aggregate as a single donor.

Including Richard Schwall, Luke Stebbing, Simon Sáfár, Laura Soares, John Davis, Cliff Hyra, Noah Topper, Gwen Murray, and Daniel Kokotajlo. Thanks, all!

The post Our 2019 Fundraiser Review appeared first on Machine Intelligence Research Institute.

Our 2019 Fundraiser Review


January 2020 Newsletter

Updates

Our 2019 fundraiser ended Dec. 31. We'll have more to say in a few weeks in our fundraiser retrospective, but for now, a big thank you to the ~240 donors who together donated more than $526,000, including $67,484 in the first 20 seconds of Giving Tuesday (not counting matching dollars, which have yet to be announced).
Jan. 15 is the final day of CFAR's annual fundraiser. CFAR also recently ran an AMA and has posted their workshop participant handbook online.
Understanding "Deep Double Descent": MIRI researcher Evan Hubinger describes a fascinating phenomenon in ML, and an interesting case study in ML research aimed at deepening our understanding, and not just advancing capabilities. In a follow-up post, Evan also considers possible implications for alignment research.
Safe Exploration and Corrigibility: Evan notes an important (and alignment-relevant) way that notions of exploration in deep RL have shifted.
"Learning Human Objectives by Evaluating Hypothetical Behavior": UC Berkeley and DeepMind researchers "present a method for training reinforcement learning agents from human feedback in the presence of unknown unsafe states".

Links from the research team
This continues my experiment from last month: having MIRI researchers anonymously pick out AI Alignment Forum posts to highlight and comment on.

Re (When) is Truth-telling Favored in AI debate? — "A paper by Vojtěch Kovařík and Ryan Carey; it's good to see some progress on the debate model!"
Re Recent Progress in the Theory of Neural Networks — "Noah MacAulay provides another interesting example of research attempting to explain what's going on with NNs."
Re When Goodharting is optimal — "I like Stuart Armstrong's post for the systematic examination of why we might be afraid of Goodharting. The example at the beginning is an interesting one, because it seems (to me at least) like the robot really should go back and forth (staying a long time at each side to minimize lost utility). But Stuart is right that this answer is, at least, quite difficult to justify."
Re Seeking Power is Instrumentally Convergent in MDPs and Clarifying Power-Seeking and Instrumental Convergence — "It's nice to finally have a formal model of this, thanks to Alex Turner and Logan Smith. Instrumental convergence has been an informal part of the discussion for a long time."
Re Critiquing "What failure looks like" — "I thought that Grue Slinky's post was a good critical analysis of Paul Christiano's 'going out with a whimper' scenario, highlighting some of the problems it seems to have as a concrete AI risk scenario. In particular, I found the analogy given to the simplex algorithm persuasive in terms of showcasing how, despite the fact that many of our current most powerful tools already have massive differentials in how well they work on different problems, those values which are not served well by those tools don't seem to have lost out massively as a result. I still feel like there may be a real risk along the lines of 'going out with a whimper', but I think this post presents a real challenge to that scenario as it has been described so far."
Re Counterfactual Induction — "A proposal for logical counterfactuals by Alex Appel. This could use some more careful thought and critique; it's not yet clear exactly how much or little it accomplishes."
Re A dilemma for prosaic AI alignment — "Daniel Kokotajlo outlines key challenges for prosaic alignment: '[…] Now I think the problem is substantially harder than that: To be competitive prosaic AI safety schemes must deliberately create misaligned mesa-optimizers and then (hopefully) figure out how to align them so that they can be used in the scheme.'"

The post January 2020 Newsletter appeared first on Machine Intelligence Research Institute.

January 2020 Newsletter


December 2019 Newsletter

From now through the end of December, MIRI's 2019 Fundraiser is live! See our fundraiser post for updates on our past year and future plans.
One of our biggest updates, I'm happy to announce, is that we've hired five new research staff, with a sixth to join us in February. For details, see Workshops and Scaling Up in the fundraiser post.
Also: Facebook's Giving Tuesday matching opportunity is tomorrow at 5:00am PT! See Colm's post for details on how to get your donation matched.

Other updates

Our most recent hire, "Risks from Learned Optimization" co-author Evan Hubinger, describes what he'll be doing at MIRI. See also Nate Soares' comment on how MIRI does nondisclosure-by-default.
Buck Shlegeris discusses EA residencies as an outreach opportunity.
OpenAI releases Safety Gym, a set of tools and environments for incorporating safety constraints into RL tasks.
CHAI is seeking interns; application deadline Dec. 15.

Thoughts from the research team

This month, I'm trying something new: quoting MIRI researchers' summaries and thoughts on recent AI safety write-ups.
I've left out names so that these can be read as a snapshot of people's impressions, rather than a definitive "Ah, researcher X believes Y!" Just keep in mind that these will be a small slice of thoughts from staff I've recently spoken to, not anything remotely like a consensus take.

Re Will transparency help catch deception? — "A good discussion of an important topic. Matthew Barnett suggests that any weaknesses in a transparency tool may turn it into a detrimental middle-man, and directly training supervisors to catch deception may be preferable."
Re Chris Olah's views on AGI safety — "I very much agree with Evan Hubinger's idea that collecting different perspectives — different 'hats' — is a useful thing to do. Chris Olah's take on transparency is good to see. The concept of microscope AI seems like a useful one, and Olah's vision of how the ML field could be usefully shifted is quite interesting."
Re Defining AI Wireheading — "Stuart Armstrong takes a shot at making a principled distinction between wireheading and the rest of Goodhart."
Re How common is it to have a 3+ year lead? — "This seems like a pretty interesting question for AI progress models. The expected lead time and questions of expected takeoff speed greatly influence the extent to which winner-take-all dynamics are plausible."
Re Thoughts on Implementing Corrigible Robust Alignment — "Steve Byrnes provides a decent overview of some issues around getting 'pointer' type values."

The post December 2019 Newsletter appeared first on Machine Intelligence Research Institute.

December 2019 Newsletter


MIRI's 2019 Fundraiser

MIRI's 2019 fundraiser is concluded.
Over the past two years, huge donor support has helped us double the size of our AI alignment research team. Hitting our $1M fundraising goal this month will put us in a great position to continue our growth in 2020 and beyond, recruiting as many brilliant minds as possible to take on what appear to us to be the central technical obstacles to alignment.
Our fundraiser progress, updated in real time (including donations and matches made during the Facebook Giving Tuesday event):

MIRI is a CS/math research group with a goal of understanding how to reliably "aim" future general-purpose AI systems at known goals. For an introduction to this research area, see Ensuring Smarter-Than-Human Intelligence Has A Positive Outcome and Risks from Learned Optimization in Advanced Machine Learning Systems. For background on how we approach the problem, see 2018 Update: Our New Research Directions and Embedded Agency.
At the end of 2017, we announced plans to substantially grow our research team, with a goal of hiring "around ten new research staff over the next two years." Two years later, I'm happy to report that we're up eight research staff, and we have a ninth starting in February of next year, which will bring our total research team size to 20.1
We remain excited about our current research directions, and continue to feel that we could make progress on them more quickly by adding additional researchers and engineers to the team. As such, our main organizational priorities remain the same: push forward on our research directions, and grow the research team to accelerate our progress.
While we're quite uncertain about how large we'll ultimately want to grow, we plan to continue growing the research team at a similar rate over the next two years, and so expect to add around ten more research staff by the end of 2021.
Our projected budget for 2020 is $6.4M–$7.4M, with a point estimate of $6.8M,2 up from around $6M this year.3 In the mainline-growth scenario, we expect our budget to look something like this:

Looking further ahead, since staff salaries account for the vast majority of our expenses, I expect our spending to increase proportionately year-over-year while research team growth continues to be a priority.
Given our $6.8M budget for 2020, and the cash we currently have on hand, raising $1M in this fundraiser will put us in a great position for 2020. Hitting $1M positions us with cash reserves of 1.25–1.5 years going into 2020, which is exactly where we want to be to support ongoing hiring efforts and to provide the confidence we need to make and stand behind our salary and other financial commitments.
For more details on what we've been up to this year, and our plans for 2020, read on!

1. Workshops and scaling up
If you lived in a world that didn't know calculus, but you knew something was missing, what general practices would have maximized your probability of coming up with it?
What if you didn't start off knowing something was missing? Could you and some friends have gotten together and done research in a way that put you in a good position to notice it, to ask the right questions?
MIRI thinks that humanity is currently missing some of the core concepts and methods that AGI developers will need in order to align their systems down the road. We think we've found research paths that may help solve that problem, and good ways to rapidly improve our understanding via experiments; and we're eager to add more researchers and engineers' eyes and brains to the effort.
A significant portion of MIRI's current work is in Haskell, and benefits from experience with functional programming and dependent type systems. More generally, if you're a programmer who loves hunting for the most appropriate abstractions to fit some use case, developing clean concepts, making and then deploying elegant combinators, or audaciously trying to answer the deepest questions in computer science—then we think you should apply to work here, get to know us at a workshop, or reach out with questions.
As noted above, our research team is growing fast. The latest additions to the MIRI team include:
Evan Hubinger, a co-author on "Risks from Learned Optimization in Advanced Machine Learning Systems". Evan previously designed the functional programming language Coconut, was an intern at OpenAI, and has done software engineering work at Google, Yelp, and Ripple.
Jeremy Schlatter, a software engineer who previously worked at Google and OpenAI. Some of the public projects Jeremy has contributed to include OpenAI's Dota 2 bot and a debugger for the Go programming language.
Seraphina Nix, joining MIRI in February 2020. Seraphina graduates this month from Oberlin College with a major in mathematics and minors in computer science and physics. She has previously done research on ultra-lightweight dark matter candidates, deep reinforcement learning, and teaching neural networks to do high school mathematics.
Rafe Kennedy, who joins MIRI after working as an independent existential risk researcher at the Effective Altruism Hotel. Rafe previously worked at the data science startup NStack, and he holds an MPhysPhil from the University of Oxford in Physics & Philosophy.
MIRI's hires and job trials are typically drawn from our 4.5-day, all-expense-paid AI Risk for Computer Scientists (AIRCS) workshop series.
Our workshop program is the best way we know of to bring promising talented individuals into what we think are useful trajectories towards being highly-contributing AI researchers and engineers. Having established an experience that participants love and that we believe to be highly valuable, we plan to continue experimenting with new versions of the workshop, and expect to run ten workshops over the course of 2020, up from eight this year.
These programs have led to a good number of new hires at MIRI as well as other AI safety organizations, and we find them valuable for everything from introducing talented outsiders to AI safety, to leveling up people who have been thinking about these issues for years.
If you're interested in attending, apply here. If you have any questions, we highly encourage you to shoot Buck Shlegeris an email.
Our MIRI Summer Fellows Program plays a similar role for us, but is more targeted at mathematicians. We're considering running MSFP in a shorter format in 2020. For any questions about MSFP, email Colm Ó Riain.

2. Research and write-ups
Our 2018 strategy update continues to be a great overview of where MIRI stands today, describing how we think about our research, laying out our case for working here, and explaining why most of our work currently isn't public-facing.
Given the latter point, I'll focus in this section on spotlighting what we've written up this past year, providing snapshots of some of the work individuals at MIRI are currently doing (without any intended implication that this is representative of the whole), and conveying some of our current broad impressions about how our research progress is going.
Some of our major write-ups and publications this year were:

"Risks from Learned Optimization in Advanced Machine Learning Systems," by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. The process of generating this paper significantly clarified our own thinking, and informed Scott and Abram's discussion of subsystem alignment in "Embedded Agency."Scott views "Risks from Learned Optimization" as being of comparable importance to "Embedded Agency" as exposition of key alignment difficulties, and we've been extremely happy about the new conversations and research that the field at large has produced to date in dialogue with the ideas in "Risks from Learned Optimization".
Thoughts on Human Models, by Scott Garrabrant and DeepMind-based MIRI Research Associate Ramana Kumar, argues that the AI alignment research community should begin prioritizing "approaches that work well in the absence of human models."The role of human models in alignment plans strikes us as one of the most important issues for MIRI and other research groups to wrestle with, and we're generally interested in seeing what new approaches groups outside MIRI might come up with for leveraging AI for the common good in the absence of human models.
"Cheating Death in Damascus," by Nate Soares and Ben Levinstein. We presented this decision theory paper at the Formal Epistemology Workshop in 2017, but a lightly edited version has now been accepted to The Journal of Philosophy, previously voted the second highest-quality academic journal in philosophy.
The Alignment Research Field Guide, a very accessible and broadly applicable resource both for individual researchers and for groups getting off the ground.

Our other recent public writing includes an Effective Altruism Forum AMA with Buck Shlegeris, Abram Demski's The Parable of Predict-O-Matic, and the many interesting outputs of the AI Alignment Writing Day we hosted toward the end of this year's MIRI Summer Fellows Program.
Turning to our research team, last year we announced that prolific Haskell programmer Edward Kmett joined the MIRI team, freeing him up to do the thing he's passionate about—improving the state of highly reliable (and simultaneously highly efficient) programming languages. MIRI Executive Director Nate Soares views this goal as very ambitious, though would feel better about the world if there existed programming languages that were both efficient and amenable to strong formal guarantees about their properties.
This year Edward moved to Berkeley to work more closely with the rest of the MIRI team. We've found it very helpful to have him around to provide ideas and contributions to our more engineering-oriented projects, helping give some amount of practical grounding to our work. Edward has also continued to be a huge help with recruiting through his connections in the functional programming and type theory world.
Meanwhile, our newest addition, Evan Hubinger, plans to continue working on solving inner alignment for amplification. Evan has outlined his research plans on the AI Alignment Forum, noting that relaxed adversarial training is a fairly up-to-date statement of his research agenda. Scott and other researchers at MIRI consider Evan's work quite exciting, both in the context of amplification and in the context of other alignment approaches it might prove useful for.
Abram Demski is another MIRI researcher who has written up a large number of his research thoughts over the last year. Abram reports (fuller thoughts here) that he has moved away from a traditional decision-theoretic approach this year, and is now spending more time on learning-theoretic approaches, similar to MIRI Research Associate Vanessa Kosoy. Quoting Abram:
Around December 2018, I had a big update against the "classical decision-theory" mindset (in which learning and decision-making are viewed as separate problems), and towards taking a learning-theoretic approach. [… I have] made some attempts to communicate my update against UDT and toward learning-theoretic approaches, including this write-up. I talked to Daniel Kokotajlo about it, and he wrote The Commitment Races Problem, which I think captures a good chunk of it.
For her part, Vanessa's recent work includes the paper "Delegative Reinforcement Learning: Learning to Avoid Traps with a Little Help," which she presented at the ICLR 2019 SafeML workshop.
I'll note again that the above are all snapshots of particular research directions various researchers at MIRI are pursuing, and don't necessarily represent other researchers' views or focus. As Buck recently noted, MIRI has a pretty flat management structure. We pride ourselves on minimizing bureaucracy, and on respecting the ability of our research staff to form their own inside-view models of the alignment problem and of what's needed next to make progress. Nate recently expressed similar thoughts about how we do nondisclosure-by-default.
As a consequence, MIRI's more math-oriented research especially tends to be dictated by individual models and research taste, without the expectation that everyone will share the same view of the problem.
Regarding his overall (very high-level) sense of how MIRI's new research directions are progressing, Nate Soares reports:
Progress in 2019 has been slower than expected, but I have a sense of steady progress. In particular, my experience is one of steadily feeling less confused each week than the week before—of me and other researchers having difficulties that were preventing us from doing a thing we wanted to do, staring at them for hours, and then realizing that we'd been thinking wrongly about this or that, and coming away feeling markedly more like we know what's going on.
An example of the kind of thing that causes us to feel like we're making progress is that we'll notice, "Aha, the right tool for thinking about all three of these apparently-dissimilar problems was order theory," or something along those lines; and disparate pieces of frameworks will all turn out to be the same, and the relevant frameworks will become simpler, and we'll be a little better able to think about a problem that I care about. This description is extremely abstract, but represents the flavor of what I mean by "steady progress" here, in the same vein as my writing last year about "deconfusion."
Our hope is that enough of this kind of progress gives us a platform from which we can generate particular exciting results on core AI alignment obstacles, and I expect to see such results reasonably soon. To date, however, I have been disappointed by the amount of time that's instead been spent on deconfusing myself and shoring up my frameworks; I previously expected to have more exciting results sooner.
In research of the kind we're working on, it's not uncommon for there to be years between sizeable results, though we should also expect to sometimes see cascades of surprisingly rapid progress, if we are indeed pushing in the right directions. My inside view of our ongoing work currently predicts that we're on a productive track and should expect to see results we are more excited about before too long.
Our research progress, then, is slower than we had hoped, but the rate and quality of progress continues to be such that we consider this work very worthwhile, and we remain optimistic about our ability to convert further research staff hours into faster progress. At the same time, we are also (of course) looking for where our research bottlenecks are and how we can make our work more efficient, and we're continuing to look for tweaks we can make that might boost our output further.
If things go well over the next few years—which seems likely but far from guaranteed—we'll continue to find new ways of making progress on research threads we care a lot about, and continue finding ways to hire people to help make that happen.
Research staff expansion is our biggest source of expense growth, and by encouraging us to move faster on exciting hiring opportunities, donor support plays a key role in how we execute on our research agenda. Though the huge support we've received to date has put us in a solid position even at our new size, further donor support is a big help for us in continuing to grow. If you want to play a part in that, thank you.

Donate Now

This number includes a new staff member who is currently doing a 6-month trial with us.These estimates were generated using a model similar to the one I used last year. For more details see our 2018 fundraiser post.This falls outside the $4.4M–$5.5M range I estimated in our 2018 fundraiser post, but is in line with the higher end of revised estimates we made internally in Q1 2019.The post MIRI's 2019 Fundraiser appeared first on Machine Intelligence Research Institute.

MIRI’s 2019 Fundraiser


Giving Tuesday 2019

Update January 25, 2020: $77,325 was donated to MIRI through Facebook on Giving Tuesday. $45,915 of this was donated within 13.5 seconds of the Facebook matching event starting at 5:00AM PT and was matched by Facebook. Thank you to everybody who set their clocks early to support us so generously! Shout out too to the EA Giving Tuesday and Rethink Charity team for their amazing efforts on behalf of the EA Community.

Update December 2, 2019: This page has been updated to reflect (a) observed changes in Facebook's flow for donations of $500 or larger (b) additional information on securing matching for donations of $2,500 or larger during Facebook's matching event and (c) a pointer to Paypal's newly announced, though significantly smaller, matching event(s). Please check in here for more updates before the Facebook Matching event begins at 5am PT on December 3.

MIRI's annual fundraiser begins this Monday, December 2, 2019 and Giving Tuesday takes place the next day; starting at 5:00:00am PT (8:00:00am ET) on December 3, Facebook will match donations made on fundraiser pages on their platform up to a total of $7,000,000. This post focuses on this Facebook matching event. (You can find information on Paypal's significantly smaller matching events in the footnotes.1)

Donations during Facebook's Giving Tuesday event will be matched dollar for dollar on a first-come, first-served basis until the $7,000,000 in matching funds are used up. Based on trends in previous years, this will probably occur within 10 seconds.
Any US-based 501(c)(3) nonprofit eligible to receive donations on Facebook, e.g. MIRI, can be matched.
Facebook will match up to a total of $100,000 per nonprofit organization.
Each donor can have up to $20,000 in eligible donations matched on Giving Tuesday. There is a default limit of $2,499 per donation. Donors who wish to donate more than $2,499 have multiple strategies to choose from (below) to increase the chances of their donations being matched.

 In 2018, Facebook's matching pool of $7M was exhausted within 16 seconds of the event starting and  in that time, 66% of our lightning-fast donors got their donations to MIRI matched, securing a total of $40,072 in matching funds. This year, we're aiming for the per-organization $100,000 maximum and since it's highly plausible that this year's matching event will end within 4-10 seconds, here are some tips to improve the chances of your donation to MIRI's Fundraiser Page on Facebook being matched.
Pre-Event Preparation (before Giving Tuesday)

Confirm your FB account is operational.
Add your preferred credit card(s) as payment method(s) in your FB settings page. Credit cards are plausibly mildly preferable to Paypal as a payment option in terms of donation speed.
Test your payment method(s) ahead of time by donating small amount(s) to MIRI's Fundraiser page.
If your credit card limit is lower than the amount you're considering donating, it may be possible to (a) overpay the balance ahead of time and/or (b) call your credit card asking them to (even temporarily) increase your limit.
If you plan to donate more than $2,499, see below for instructions.
Sync whatever clock you'll be using with time.is.
Consider pledging your donation to MIRI at EA Giving Tuesday.2

Donating on Giving Tuesday
On Tuesday, December 3, BEFORE 5:00:00am PT — it's advisable to be alert and ready 10-20 minutes before the event — prepare your donation, so you can make your donation with a single click when the event begins at 5:00:00am PT.

Open an accurate clock at time.is.
In a different browser window alongside, open MIRI's Fundraiser Page on Facebook in your browser.
Click on the Donate button.
In the "Donate" popup that surfaces:

Enter your donation amount — between $5 and $2,499. See below for larger donations.
Choose whichever card you're using for your donation.
Optionally enter a note and/or adjust the donation visibility.

At 05:00:00 PST, click on the Green Donate button. If your donation amount is $500 or larger, you may be presented with an additional "Confirm Your Donation" dialog. If so, click on its Donate button as quickly as possible.

Larger Donations
By default, Facebook places a limit of $2,499 per donation (in the US3), and will match up to $20,000 per donor. If you're in a position to donate $2,500 or more to MIRI, you can:

Use multiple browser windows/tabs for each individual donation: open up MIRI's Fundraiser Page on Facebook in as many tabs as needed in your browser and follow the instructions above in each window/tab so you have multiple Donate buttons ready to click, one in each tab. Then at 5:00:00 PT, channel your lightning and click as fast as you can — one of our star supporters last year made 8 donations within 21 seconds, 5 of which were matched.

and/or
Before the event — ideally not the morning of — follow EA Giving Tuesday's instructions on how to increase your per-donation limit on Facebook above $2,499. Our friends at EA Giving Tuesday estimate that "you are likely to be able to successfully donate up to $9,999 per donation" after following these instructions. Their analysis also suggests that going higher than $10,000 for an individual donation plausibly significantly increases the probability of being declined and therefore advise not going beyond $9,999 per donation. It is possible that Facebook may put a cap of $2,499 on individual donations closer to the event.

Using a combination of the above, a generous supporter could, for example, make 2 donations of $9,999 each — in separate browser windows — within seconds of the event starting.

Paypal has 3 separate matching events on Giving Tuesday — all of which add 10% to eligible donations — for the USA, Canada, and the UK.
Thanks to Ari, William, Rethink Charity and all at EA Giving Tuesday for their work to help EA organizations maximize their share of Facebook's matching funds.
For up-to-date information on Facebook's donation limits outside the US, check out EA Giving Tuesday's doc.

The post Giving Tuesday 2019 appeared first on Machine Intelligence Research Institute.

Giving Tuesday 2019


November 2019 Newsletter

I'm happy to announce that Nate Soares and Ben Levinstein's "Cheating Death in Damascus" has been accepted for publication in The Journal of Philosophy (previously voted the second-highest-quality journal in philosophy).
In other news, MIRI researcher Buck Shlegeris has written over 12,000 words on a variety of MIRI-relevant topics in an EA Forum AMA. (Example topics: advice for software engineers; what alignment plans tend to look like; and decision theory.)
Other updates

Abram Demski's The Parable of Predict-O-Matic is a great read: the predictor/optimizer issues it covers are deep, but I expect a fairly wide range of readers to enjoy it and get something out of it.
Evan Hubinger's Gradient Hacking describes an important failure mode that hadn't previously been articulated.
Vanessa Kosoy's LessWrong shortform has recently discussed some especially interesting topics related to her learning-theoretic agenda.
Stuart Armstrong's All I Know Is Goodhart constitutes nice conceptual progress on expected value maximizers that are aware of Goodhart's law and trying to avoid it.
Reddy, Dragan, and Levine's paper on modeling human intent cites (of all things) Harry Potter and the Methods of Rationality as inspiration.

News and links

Artificial Intelligence Research Needs Responsible Publication Norms: Crootof provides a good review of the issue on Lawfare.
Stuart Russell's new book is out: Human Compatible: Artificial Intelligence and the Problem of Control (excerpt). Rohin Shah's review does an excellent job of contextualizing Russell's views within the larger AI safety ecosystem, and Rohin highlights the quote:
The task is, fortunately, not the following: given a machine that possesses a high degree of intelligence, work out how to control it. If that were the task, we would be toast. A machine viewed as a black box, a fait accompli, might as well have arrived from outer space. And our chances of controlling a superintelligent entity from outer space are roughly zero. Similar arguments apply to methods of creating AI systems that guarantee we won't understand how they work; these methods include whole-brain emulation — creating souped-up electronic copies of human brains — as well as methods based on simulated evolution of programs. I won't say more about these proposals because they are so obviously a bad idea.

Jacob Steinhardt releases an AI Alignment Research Overview.
Patrick LaVictoire's AlphaStar: Impressive for RL Progress, Not for AGI Progress raises some important questions about how capable today's state-of-the-art systems are.

The post November 2019 Newsletter appeared first on Machine Intelligence Research Institute.

November 2019 Newsletter


October 2019 Newsletter

Updates

Ben Pace summarizes a second round of AI Alignment Writing Day posts.
The Zettelkasten Method: MIRI researcher Abram Demski describes a note-taking system that's had a large positive effect on his research productivity.
Will MacAskill writes a detailed critique of functional decision theory; Abram Demski (1, 2) and Matthew Graves respond in the comments.

News and links

Recent AI alignment posts: Evan Hubinger asks "Are minimal circuits deceptive?", Paul Christiano describes the strategy-stealing assumption, and Wei Dai lists his resolved confusions about Iterated Distillation and Amplification. See also Rohin Shah's comparison of recursive approaches to AI alignment.
Also on LessWrong: A Debate on Instrumental Convergence Between LeCun, Russell, Bengio, Zador, and More.
FHI's Ben Garfinkel and Allan Dafoe argue that conflicts between nations tend to exhibit "offensive-then-defensive scaling".
OpenAI releases a follow-up report on GPT-2, noting that several groups "have explicitly adopted similar staged release approaches" to OpenAI.
NVIDIA Applied Deep Learning Research has trained a model that appears to essentially replicate GPT-2, with 5.6x as many parameters, slightly better WikiText perplexity, and slightly worse LAMBADA accuracy. The group has elected to share their training and evaluation code, but not the model weights.
OpenAI fine-tunes GPT-2 for text continuation and summarization tasks that incorporate human feedback, noting, "Our motivation is to move safety techniques closer to the general task of 'machines talking to humans,' which we believe is key to extracting information about human values."

The post October 2019 Newsletter appeared first on Machine Intelligence Research Institute.

October 2019 Newsletter


September 2019 Newsletter

Updates

We ran a very successful MIRI Summer Fellows Program, which included a day where participants publicly wrote up their thoughts on various AI safety topics. See Ben Pace's first post in a series of roundups.
A few highlights from the writing day: Adele Lopez's Optimization Provenance; Daniel Kokotajlo's Soft Takeoff Can Still Lead to Decisive Strategic Advantage and The "Commitment Races" Problem; Evan Hubinger's Towards a Mechanistic Understanding of Corrigibility; and John Wentworth's Markets are Universal for Logical Induction and Embedded Agency via Abstraction.
New posts from MIRI staff and interns: Abram Demski's Troll Bridge; Matthew Graves' View on Factored Cognition; Daniel Filan's Verification and Transparency; and Scott Garrabrant's Intentional Bucket Errors and Does Agent-like Behavior Imply Agent-like Architecture?
See also a forum discussion on "proof-level guarantees" in AI safety.

News and links

From Ben Cottier and Rohin Shah: Clarifying Some Key Hypotheses in AI Alignment
Classifying Specification Problems as Variants of Goodhart's Law: Victoria Krakovna and Ramana Kumar relate DeepMind's SRA taxonomy to mesa-optimizers, selection and control, and Scott Garrabrant's Goodhart taxonomy. Also new from DeepMind: Ramana, Tom Everitt, and Marcus Hutter's Designing Agent Incentives to Avoid Reward Tampering.
From OpenAI: Testing Robustness Against Unforeseen Adversaries. 80,000 Hours also recently interviewed OpenAI's Paul Christiano, with some additional material on decision theory.
From AI Impacts: Evidence Against Current Methods Leading to Human-Level AI and Ernie Davis on the Landscape of AI Risks
From Wei Dai: Problems in AI Alignment That Philosophers Could Potentially Contribute To
Richard Möhn has put together a calendar of upcoming AI alignment events.
The Berkeley Existential Risk Initiative is seeking an Operations Manager.

The post September 2019 Newsletter appeared first on Machine Intelligence Research Institute.

September 2019 Newsletter


August 2019 Newsletter

Updates

MIRI research associate Stuart Armstrong is offering $1000 for good questions to ask an Oracle AI.
Recent AI safety posts from Stuart: Indifference: Multiple Changes, Multiple Agents; Intertheoretic Utility Comparison: Examples; Normalising Utility as Willingness to Pay; and Partial Preferences Revisited.
MIRI researcher Buck Shlegeris has put together a quick and informal AI safety reading list.
There's No Fire Alarm for AGI reports on a researcher's January 2017 prediction that "in the next two years, we will not get 80, 90%" on Winograd schemas, an NLP test. Although this prediction was correct, researchers at Microsoft, Carnegie Mellon and Google Brain, and Facebook have now (2.5 years later) achieved Winograd scores of 89.0 and 90.4.
Ortega et al.'s "Meta-Learning of Sequential Strategies" includes a discussion of mesa-optimization, independent of Hubinger et al.'s "Risks from Learned Optimization in Advanced Machine Learning Systems," under the heading of "spontaneous meta-learning."

News and links

Wei Dai outlines forum participation as a research strategy.
On a related note, the posts on the AI Alignment Forum this month were very good — I'll spotlight them all this time around. Dai wrote on the purposes of decision theory research; Shah on learning biases and rewards simultaneously; Kovarik on AI safety debate and its applications; Steiner on the Armstrong agenda and the intentional stance; Trazzi on manipulative AI; Cohen on IRL and imitation; and Manheim on optimizing and Goodhart effects (1, 2, 3).
Jade Leung discusses AI governance on the AI Alignment Podcast.
CMU and Facebook researchers' Pluribus program beats human poker professionals — using only $144 in compute. The developers also choose not to release the code: "Because poker is played commercially, the risk associated with releasing the code outweighs the benefits."
Microsoft invests $1 billion in OpenAI. From Microsoft's press release: "Through this partnership, the companies will accelerate breakthroughs in AI and power OpenAI's efforts to create artificial general intelligence (AGI)." OpenAI has also released a paper on "The Role of Cooperation in Responsible AI Development."
Ought has a new preferred introduction to their work. See also Paul Christiano's Ought: Why it Matters and Ways to Help.
FHI has 11 open research positions; applications are due by Aug. 16. You can also apply to CSER's AGI risk research associate position through Aug. 26.

The post August 2019 Newsletter appeared first on Machine Intelligence Research Institute.

August 2019 Newsletter


July 2019 Newsletter

Hubinger et al.'s "Risks from Learned Optimization in Advanced Machine Learning Systems", one of our new core resources on the alignment problem, is now available on arXiv, the AI Alignment Forum, and LessWrong.
In other news, we received an Ethereum donation worth $230,910 from Vitalik Buterin — the inventor and co-founder of Ethereum, and now our third-largest all-time supporter!
Also worth highlighting, from the Open Philanthropy Project's Claire Zabel and Luke Muehlhauser: there's a pressing need for security professionals in AI safety and biosecurity.

It's more likely than not that within 10 years, there will be dozens of GCR-focused roles in information security, and some organizations are already looking for candidates that fit their needs (and would hire them now, if they found them).
It's plausible that some people focused on high-impact careers (as many effective altruists are) would be well-suited to helping meet this need by gaining infosec expertise and experience and then moving into work at the relevant organizations.

Other updates

Mesa Optimization: What It Is, And Why We Should Care — Rohin Shah's consistently excellent Alignment Newsletter discusses "Risks from Learned Optimization…" and other recent AI safety work.
MIRI Research Associate Stuart Armstrong releases his Research Agenda v0.9: Synthesising a Human's Preferences into a Utility Function.
OpenAI and MIRI staff help talk Munich student Connor Leahy out of releasing an attempted replication of OpenAI's GPT-2 model. (LessWrong discussion.) Although Leahy's replication attempt wasn't successful, write-ups like his suggest that OpenAI's careful discussion surrounding GPT-2 is continuing to prompt good reassessments of publishing norms within ML. Quoting from Leahy's postmortem:

Sometime in the future we will have reached a point where the consequences of our research are beyond what we can discover in a one-week evaluation cycle. And given my recent experiences with GPT2, we might already be there. The more complex and powerful our technology becomes, the more time we should be willing to spend in evaluating its consequences. And if we have doubts about safety, we should default to caution.
	We tend to live in an ever accelerating world. Both the industrial and academic R&D cycles have grown only faster over the decades. Everyone wants "the next big thing" as fast as possible. And with the way our culture is now, it can be hard to resist the pressures to adapt to this accelerating pace. Your career can depend on being the first to publish a result, as can your market share.
	We as a community and society need to combat this trend, and create a healthy cultural environment that allows researchers to take their time. They shouldn't have to fear repercussions or ridicule for delaying release. Postponing a release because of added evaluation should be the norm rather than the exception. We need to make it commonly accepted that we as a community respect others' safety concerns and don't penalize them for having such concerns, even if they ultimately turn out to be wrong. If we don't do this, it will be a race to the bottom in terms of safety precautions.

From Abram Demski: Selection vs. Control; Does Bayes Beat Goodhart?; and Conceptual Problems with Updateless Decision Theory and Policy Selection
Vox's Future Perfect Podcast interviews Jaan Tallinn and discusses MIRI's role in originating and propagating AI safety memes.
The AI Does Not Hate You, journalist Tom Chivers' well-researched book about the rationality community and AI risk, is out in the UK.

News and links

Other recent AI safety write-ups: David Krueger's Let's Talk About "Convergent Rationality"; Paul Christiano's Aligning a Toy Model of Optimization; and Owain Evans, William Saunders, and Andreas Stuhlmüller's Machine Learning Projects on Iterated Distillation and Amplification
From DeepMind: Vishal Maini puts together an AI reading list, Victoria Krakovna recaps the ICLR Safe ML workshop, and Pushmeet Kohli discusses AI safety on the 80,000 Hours Podcast.
The EA Foundation is awarding grants for "efforts to reduce risks of astronomical suffering (s-risks) from advanced artificial intelligence"; apply by Aug. 11.
Additionally, if you're a young AI safety researcher (with a PhD) based at a European university or nonprofit, you may want to apply for ~$60,000 in funding from the Bosch Center for AI.

The post July 2019 Newsletter appeared first on Machine Intelligence Research Institute.

July 2019 Newsletter


New paper: "Risks from learned optimization"

Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant have a new paper out: "Risks from learned optimization in advanced machine learning systems."
The paper's abstract:
We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as mesa-optimization, a neologism we introduce in this paper.
We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned? In this paper, we provide an in-depth analysis of these two primary questions and provide an overview of topics for future research.
The critical distinction presented in the paper is between what an AI system is optimized to do (its base objective) and what it actually ends up optimizing for (its mesa-objective), if it optimizes for anything at all. The authors are interested in when ML models will end up optimizing for something, as well as how the objective an ML model ends up optimizing for compares to the objective it was selected to achieve.
The distinction between the objective a system is selected to achieve and the objective it actually optimizes for isn't new. Eliezer Yudkowsky has previously raised similar concerns in his discussion of optimization daemons, and Paul Christiano has discussed such concerns in "What failure looks like."
The paper's contents have also been released this week as a sequence on the AI Alignment Forum, cross-posted to LessWrong. As the authors note there:
We believe that this sequence presents the most thorough analysis of these questions that has been conducted to date. In particular, we plan to present not only an introduction to the basic concerns surrounding mesa-optimizers, but also an analysis of the particular aspects of an AI system that we believe are likely to make the problems related to mesa-optimization relatively easier or harder to solve. By providing a framework for understanding the degree to which different AI systems are likely to be robust to misaligned mesa-optimization, we hope to start a discussion about the best ways of structuring machine learning systems to solve these problems.
Furthermore, in the fourth post we will provide what we think is the most detailed analysis yet of a problem we refer as deceptive alignment which we posit may present one of the largest—though not necessarily insurmountable—current obstacles to producing safe advanced machine learning systems using techniques similar to modern machine learning.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New paper: "Risks from learned optimization" appeared first on Machine Intelligence Research Institute.

New paper: “Risks from learned optimization”


June 2019 Newsletter

	Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant have released the first two (of five) posts on "mesa-optimization":
The goal of this sequence is to analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as mesa-optimization.
We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned?
The sequence begins with Risks from Learned Optimization: Introduction and continues with Conditions for Mesa-Optimization. (LessWrong mirror.)
Other updates

New research posts: Nash Equilibria Can Be Arbitrarily Bad; Self-Confirming Predictions Can Be Arbitrarily Bad; And the AI Would Have Got Away With It Too, If…; Uncertainty Versus Fuzziness Versus Extrapolation Desiderata
We've released our annual review for 2018.
Applications are open for two AI safety events at the EA Hotel in Blackpool, England: the Learning-By-Doing AI Safety Workshop (Aug. 16-19), and the Technical AI Safety Unconference (Aug. 22-25).
A discussion of takeoff speed, including some very incomplete and high-level MIRI comments.

News and links

Other recent AI safety posts: Tom Sittler's A Shift in Arguments for AI Risk and Wei Dai's "UDT2" and "against UD+ASSA".
Talks from the SafeML ICLR workshop are now available online.
From OpenAI: "We're implementing two mechanisms to responsibly publish GPT-2 and hopefully future releases: staged release and partnership-based sharing."
FHI's Jade Leung argues that "states are ill-equipped to lead at the formative stages of an AI governance regime," and that "private AI labs are best-placed to lead on AI governance".

The post June 2019 Newsletter appeared first on Machine Intelligence Research Institute.

June 2019 Newsletter


2018 in review

Our primary focus at MIRI in 2018 was twofold: research—as always!—and growth.
Thanks to the incredible support we received from donors the previous year, in 2018 we were able to aggressively pursue the plans detailed in our 2017 fundraiser post. The most notable goal we set was to "grow big and grow fast," as our new research directions benefit a lot more from a larger team, and require skills that are a lot easier to hire for. To that end, we set a target of adding 10 new research staff by the end of 2019.
2018 therefore saw us accelerate the work we started in 2017, investing more in recruitment and shoring up the foundations needed for our ongoing growth. Since our 2017 fundraiser post, we're up 3 new research staff, including noted Haskell developer Edward Kmett. I now think that we're most likely to hit 6–8 hires by the end of 2019, though hitting 9–10 still seems quite possible to me, as we are still engaging with many promising candidates, and continue to meet more.
Overall, 2018 was a great year for MIRI. Our research continued apace, and our recruitment efforts increasingly paid out dividends.

Below I'll elaborate on our:

research progress and outputs,
research program support activities, including more details on our recruitment efforts,
outreach related activities, and
fundraising and spending.

2018 Research
Our 2018 update discussed the new research directions we're pursuing, and the nondisclosure-by-default policy we've adopted for our research overall. As described in the post, these new directions aim at deconfusion (similar to our traditional research programs, which we continue to pursue), and include the themes of "seeking entirely new low-level foundations for optimization," "endeavoring to figure out parts of cognition that can be very transparent as cognition," and "experimenting with some [relatively deep] alignment problems," and require building software systems and infrastructure.
In 2018, our progress on these new directions and the supporting infrastructure was steady and significant, in line with our high expectations, albeit proceeding significantly slower than we'd like, due in part to the usual difficulties associated with software development. On the whole, our excitement about these new directions is high, and we remain very eager to expand the team to accelerate our progress.
In parallel, Agent Foundations work continued to be a priority at MIRI. Our biggest publication on this front was "Embedded Agency," co-written by MIRI researchers Scott Garabrant and Abram Demski. "Embedded Agency" reframes our Agent Foundations research agenda as different angles of attack on a single central difficulty: we don't know how to characterize good reasoning and decision-making for agents embedded in their environment.
Below are notable technical results and analyses we released in each research category last year.1 These are accompanied by predictions made last year by Scott Garrabrant, the research lead for MIRI's Agent Foundations work, and Scott's assessment of the progress our published work represents against those predictions. The research categories below are explained in detail in "Embedded Agency."
The actual share of MIRI's research that was non-public in 2018 ended up being larger than Scott expected when he registered his predictions. The list below is best thought of as a collection of interesting (though not groundbreaking) results and analyses that demonstrate the flavor of some of the directions we explored in our research last year. As such, these assessments don't represent our model of our overall progress, and aren't intended to be a good proxy for that question. Given the difficulty of predicting what we'll disclose for our 2019 public-facing results, we won't register new predictions this year.
Decision theory

Predicted progress: 3 (modest)
Actual progress: 2 (weak-to-modest)

Scott sees our largest public decision theory result of 2018 as Prisoners' Dilemma with Costs to Modeling, a modified version of open-source prisoners' dilemmas in which agents must pay resources in order to model each other.
Other significant write-ups include:

Logical Inductors Converge to Correlated Equilibria (Kinda): A game-theoretic analysis of logical inductors.
New results in Asymptotic Decision Theory and When EDT=CDT, ADT Does Well represent incremental progress on understanding what's possible with respect to learning the right counterfactuals.

Additional decision theory research posts from 2018:

From Alex Appel, a MIRI contractor and summer intern: (a) Distributed Cooperation; (b) Cooperative Oracles; (c) When EDT=CDT, ADT Does Well; (d) Conditional Oracle EDT Equilibria in Games
From Abram Demski: (a) In Logical Time, All Games are Iterated Games; (b) A Rationality Condition for CDT Is That It Equal EDT (Part 1); (c) A Rationality Condition for CDT Is That It Equal EDT (Part 2)
From Scott Garrabrant: (a) Knowledge is Freedom; (b) Counterfactual Mugging Poker Game; (c) (A → B) → A
From Alex Mennen, a MIRI summer intern: When Wishful Thinking Works

Embedded world-models

Predicted progress: 3 (modest)
Actual progress: 1 (limited)

Some of our relatively significant results related to embedded world-models included:

Sam Eisenstat's untrollable prior, explained in illustrated form by Abram Demski, shows that there is a Bayesian solution to one of the basic problems which motivated the development of non-Bayesian logical uncertainty tools (culminating in logical induction). This informs our picture of what's possible, and may lead to further progress in the direction of Bayesian logical uncertainty.
Sam Eisenstat and Tsvi Benson-Tilsen's formulation of Bayesian logical induction. This framework, which has yet to be written up, forces logical induction into a Bayesian framework by constructing a Bayesian prior which trusts the beliefs of a logical inductor (which must supply those beliefs to the Bayesian regularly).

Sam and Tsvi's work can be viewed as evidence that "true" Bayesian logical induction is possible. However, it can also be viewed as a demonstration that we have to be careful what we mean by "Bayesian"—the solution is arguably cheating, and it isn't clear that you get any new desirable properties by doing things this way.
Scott assigns the untrollable prior result a 2 (weak-to-modest progress) rather than a 1 (limited progress), but is counting this among our 2017 results, since it was written up in 2018 but produced in 2017.
Other recent work in this category includes:

From Alex Appel: (a) Resource-Limited Reflective Oracles; (b) Bounded Oracle Induction
From Abram Demski: (a) Toward a New Technical Explanation of Technical Explanation; (b) Probability is Real, and Value is Complex

Robust delegation

Predicted progress: 2 (weak-to-modest)
Actual progress: 1 (limited)

Our most significant 2018 public result in this category is perhaps Sam Eisenstat's logical inductor tiling result, which solves a version of the tiling problem for logically uncertain agents.2
Other posts on robust delegation:

From Stuart Armstrong (MIRI Research Associate): (a) Standard ML Oracles vs. Counterfactual Ones; (b) "Occam's Razor is Insufficient to Infer the Preferences of Irrational Agents"
From Abram Demski: Stable Pointers to Value II: Environmental Goals
From Scott Garrabrant: Optimization Amplifies
From Vanessa Kosoy (MIRI Research Associate): (a) Quantilal Control for Finite Markov Decision Processes; (b) Computing An Exact Quantilal Policy
From Alex Mennen: Safely and Usefully Spectating on AIs Optimizing Over Toy Worlds

Subsystem alignment

Predicted progress: 2 (weak-to-modest)
Actual progress: 2

We achieved greater clarity on subsystem alignment in 2018, largely reflected in Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse,3 and Scott Garrabrant's forthcoming paper, "Risks from Learned Optimization in Advanced Machine Learning Systems."4 This paper is currently being rolled out on the AI Alignment Forum, as a sequence on "Mesa-Optimization."5
Scott Garrabrant's Robustness to Scale also discusses issues in subsystem alignment ("robustness to relative scale"), alongside other issues in AI alignment.
Other

Predicted progress: 2 (weak-to-modest)
Actual progress: 2

Some of the 2018 publications we expect to be most useful cut across all of the above categories:

"Embedded Agency," Scott and Abram's new introduction to all of the above research directions.
Fixed Point Exercises, a set of exercises created by Scott to introduce people to the core ideas and tools in agent foundations research.

Here, other noteworthy posts include:

From Scott Garrabrant: (a) Sources of Intuitions and Data on AGI; (b) History of the Development of Logical Induction

2018 Research Program Support
We added three new research staff to the team in 2018: Ben Weinstein-Raun, James Payor, and Edward Kmett.
We invested a large share of our capacity into growing the research team in 2018, and generally into activities aimed at increasing the amount of alignment research in the world, including:

Running eight AI Risk for Computer Scientist (AIRCS) workshops. This is an ongoing all-expenses-paid workshop series for computer scientists and programmers who want to get started thinking about or working on AI alignment. At these workshops, we introduce AI risk and related concepts, share some CFAR-style rationality content, and introduce participants to the work done by MIRI and other safety research teams. Our overall aim is to cause good discussions to happen, improve participants' ability to make progress on whether and how to contribute, and in the process work out whether they may be interested in joining MIRI or other alignment groups. Of 2018 workshop participants, we saw one join MIRI full-time, four take on internships with us, and on the order of ten with good prospects of joining MIRI within a year, in addition to several who have since joined other safety-related organizations.
Running a 2.5-week AI Summer Fellows Program (AISFP) with CFAR.6 Additionally, MIRI researcher Tsvi Benson-Tilsen and MIRI summer intern Alex Zhu ran a mid-year AI safety retreat for MIT students and alumni.
Running a 10-week research internship program over the summer, reviewed in our summer updates. Interns also participated in AISFP and in a joint research workshop with interns from the Center for Human-Compatible AI. Additionally, we hosted three more research interns later in the year. We are hopeful that at least one of them will join the team in 2019.
Making grants to two individuals as part of our AI Safety Retraining Program. In 2018 we received $150k in restricted funding from the Open Philanthropy Project, "to provide stipends and guidance to a few highly technically skilled individuals. The goal of the program is to free up 3–6 months of time for strong candidates to spend on retraining, so that they can potentially transition to full-time work on AI alignment." We issued grants to two people in 2018, including Carroll Wainwright who went on to become a Research Scientist at Partnership on AI.

In addition to the above, in 2018 we:

Hired additional operations staff to ensure we have the required operational capacity to support our continued growth.
Moved into new larger office space.

2018 Outreach and Exposition
On the outreach, coordination, and exposition front, we:

Released a new edition of Rationality: From AI to Zombies, beginning with volumes one and two, featuring a number of updates to the text and an official print edition. We also made Stuart Armstrong's 2014 book on AI risk, Smarter Than Us: The Rise of Machine Intelligence, available on the web for free at smarterthan.us.
Released 2018 Update: Our New Research Directions, a lengthy discussion of our research, our nondisclosure-by-default policies, and the case for computer scientists and software engineers to apply to join our team.
Produced other expository writing: Two Clarifications About "Strategic Background"; Challenges to Paul Christiano's Capability Amplification Proposal (discussion on LessWrong, including follow-up conversations); Comment on Decision Theory; The Rocket Alignment Problem (LessWrong link).
Received press coverage in Axios, Forbes, Gizmodo, and Vox (1, 2), and were interviewed in Nautilus and on Sam Harris' podcast.
Spoke at Effective Altruism Global in San Francisco and at the Human-Aligned AI Summer School in Prague.
Presented on logical induction at the joint Applied Theory Workshop / Workshop in Economic Theory.
Released a paper, "Categorizing Variants of Goodhart's Law," based on Scott Garrabrant's 2017 "Goodhart Taxonomy." We also reprinted Nate Soares' "The Value Learning Problem" and Nick Bostrom and Eliezer Yudkowsky's "The Ethics of Artificial Intelligence" in Artificial Intelligence Safety and Security.
Several MIRI researchers also received recognition from the AI Alignment Prize, including Scott Garrabrant receiving first place and second place in the first round and second round, respectively, MIRI Research Associate Vanessa Kosoy winning first prize in the third round, and Scott and Abram Demski tying Alex Turner for first place in the fourth round.
MIRI senior staff also participated in AI research and strategy events and conversations throughout the year.

2018 Finances
Fundraising
2018 was another strong year for MIRI's fundraising. While the total raised of just over $5.1M was a 12% drop from the amount raised in 2017, the graph below shows that our strong growth trend continued—with 2017, as I surmised in last year's review, looking like an outlier year driven by the large influx of cryptocurrency contributions during a market high in December 2017.7

(In this chart and those that follow, "Unlapsed" indicates contributions from past supporters who did not donate in the previous year.)
Highlights include:

$1.02M, our largest ever single donation by an individual, from "Anonymous Ethereum Investor #2," based in Canada, made through Rethink Charity Forward's recently established tax-advantaged fund for Canadian MIRI supporters.8
$1.4M in grants from the Open Philanthropy Project, $1.25M in general support and $150k for our AI Safety Retraining Program.
$951k during our annual fundraiser, driven in large part by MIRI supporters' participation in multiple matching campaigns during the fundraiser, including WeTrust Spring's Ethereum-matching campaign, Facebook's Giving Tuesday event, and in partnership with Raising for Effective Giving (REG), professional poker players' Double Up Drive.
$529k from 2 grants recommended by the EA Funds Long-Term Future Fund.
$115K from Poker Stars, also through REG.

In 2018, we received contributions from 637 unique contributors, 16% less than in 2017. This drop was largely driven by a 27% reduction in the number of new donors, partly offset by the continuing trend of steady growth in the number of returning donors9:

Donations of cryptocurrency were down in 2018 both in absolute terms (-$1.2M in value) and as a percentage of total contributions (23% compared to 42% in 2017). It's plausible that if cryptocurrency values continue to rebound in 2019, we may see this trend reversed.
In 2017, donations received from matching initiatives dramatically increased with almost a five-fold increase over the previous year. In 2018, our inclusion in two different REG-administered matching challenges, a significantly increased engagement among MIRI supporters with Facebook's Giving Tuesday, and MIRI's winning success in WeTrust's Spring campaign, offset a small decrease in corporate match dollars to improve slightly on 2017's matching total. The following graph represents the matching amounts received over the last 5 years:

Spending
In our 2017 fundraiser post, I projected that we'd spend ~$2.8M in 2018. Towards the end of last year, I revised our estimate:
Following the amazing show of support we received from donors last year (and continuing into 2018), we had significantly more funds than we anticipated, and we found more ways to usefully spend it than we expected. In particular, we've been able to translate the "bonus" support we received in 2017 into broadening the scope of our recruiting efforts. As a consequence, our 2018 spending, which will come in at around $3.5M, actually matches the point estimate I gave in 2017 for our 2019 budget, rather than my prediction for 2018—a large step up from what I predicted, and an even larger step from last year's [2017] budget of $2.1M.
The post goes on to give an overview of the ways in which we put this "bonus" support to good use. These included, in descending order by cost:

Investing significantly more in recruiting-related activities, including our AIRCS workshop series; and scaling up the number of interns we hosted, with an increased willingness to pay higher wages to attract promising candidates to come intern/trial with us.
Filtering less on price relative to fit when choosing new office space to accommodate our growth, and spending more on renovations, than we otherwise would have been able to, in order to create a more focused working environment for research staff.
Raising salaries for some existing staff, who were being paid well below market rates.

With concrete numbers now in hand, I'll go into more detail below on how we put those additional funds to work.
Total spending came in just over $3.75M. The chart below compares our actual spending in 2018 with our projections, and with our spending in 2017.10

At a high level, as expected, personnel costs in 2018 continued to account for the majority of our spending—though represented a smaller share of total spending than in 2017, due to increased spending on recruitment-related activities along with one-time costs related to securing and renovating our new office space.
Our spending on recruitment-related activities is captured in the program activities category. The major ways we put additional funds to use, which account for the increase over my projections, break down as follows:

~$170k on internships: We hosted nine research interns for an average of ~2.5 months each. We were able to offer more competitive wages for internships, allowing us to recruit interns (especially those with an engineering focus) that we otherwise would have had a much harder time attracting, given the other opportunities they had available to them. We are actively interested in hiring three of these interns, and have made formal offers to two of them. I'm hopeful that we'll have added at least one of them to the team by the end of this year.
$54k on AI Safety Retraining Program grants, described above.
The bulk of the rest of the additional funds we spent in this category went towards funding our ongoing series of AI Risk for Computer Scientists workshops, described above.

Expenses related to our new office space are accounted for in the cost of doing business category. The surplus spending in this category resulted from:

~$300k for securing, renovating, and filling out our new office space. Finding a suitable new space to accommodate our growth in Berkeley ended up being much more challenging and time-consuming than we expected.11 We made use of additional funds to secure our preferred space ahead of when we were prepared to move, and to renovate the space to meet our needs, whereas if we'd been operating with the budget I originally projected, we would have almost certainly ended up in a much worse space.
The remainder of the spending beyond my projection in this category comes from higher-than-expected legal costs to secure visas for staff, and slightly higher-than-projected spending across many other subcategories.

Our summaries of our more significant results below largely come from our 2018 fundraiser post.Not to be confused with Nate Soares' forthcoming tiling agents paper.Evan was a MIRI research intern, while Chris, Vladimir, and Joar are external collaborators.This paper was previously cited in "Embedded Agency" under the working title "The Inner Alignment Problem."The full PDF version of the paper will be released in conjuction with the last post of the sequence.As noted in our summer updates:
We had a large and extremely strong pool of applicants, with over 170 applications for 30 slots (versus 50 applications for 20 slots in 2017). The program this year was more mathematically flavored than in 2017, and concluded with a flurry of new analyses by participants. On the whole, the program seems to have been more successful at digging into AI alignment problems than in previous years, as well as more successful at seeding ongoing collaborations between participants, and between participants and MIRI staff.
The program ended with a very active blogathon, with write-ups including: Dependent Type Theory and Zero-Shot Reasoning; Conceptual Problems with Utility Functions (and follow-up); Complete Class: Consequentialist Foundations; and Agents That Learn From Human Behavior Can't Learn Human Values That Humans Haven't Learned Yet.Note that amounts in this section may vary slightly from our audited financial statements, due to small differences between how we tracked donations internally, and how we are required to report them in our financial statements.A big thanks to Colm for all the work he's put into setting this up; have a look at our Tax-Advantaged Donations page for more information.2014 is anomalously high on this graph due to the community's active participation in our memorable SVGives campaign.Note that these numbers will differ slightly compared to our forthcoming audited financial statements for 2018, due to subtleties of how certain types of expenses are tracked. For example, in the financial statements, renovation costs are considered to be a fixed asset that depreciates over time, and as such, won't show up as an expense.The number of options available in the relevant time frame were very limited, and most did not meet many of our requirements. Of the available spaces, the option that offered the best combination of size, layout, and location, was looking for a tenant starting November 1st 2018, while we weren't able to move until early January 2019. Additionally, the space was configured with a very open layout that wouldn't have met our needs, but that many other prospective tenants found desirable, such that we'd have to cover renovation costs.The post 2018 in review appeared first on Machine Intelligence Research Institute.

2018 in review


May 2019 Newsletter

Updates

A new paper from MIRI researcher Vanessa Kosoy, presented at the ICLR SafeML workshop this week: "Delegative Reinforcement Learning: Learning to Avoid Traps with a Little Help."
New research posts: Learning "Known" Information When the Information is Not Actually Known; Defeating Goodhart and the "Closest Unblocked Strategy" Problem; Reinforcement Learning with Imperceptible Rewards
The Long-Term Future Fund has announced twenty-three new grant recommendations, and provided in-depth explanations of the grants. These include a $50,000 grant to MIRI, and grants to CFAR and Ought. LTFF is also recommending grants to several individuals with AI alignment research proposals whose work MIRI staff will be helping assess.
We attended the Global Governance of AI Roundtable at the World Government Summit in Dubai.

News and links

Rohin Shah reflects on the first year of the Alignment Newsletter.
Some good recent AI alignment discussion: Alex Turner asks for the best reasons for pessimism about impact measures; Henrik Åslund and Ryan Carey discuss corrigibility as constrained optimization; Wei Dai asks about low-cost AGI coordination; and Chris Leong asks, "Would solving counterfactuals solve anthropics?"
From DeepMind: Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification.
Ilya Sutskever and Greg Brockman discuss OpenAI's new status as a "hybrid of a for-profit and nonprofit".
Misconceptions about China and AI: Julia Galef interviews Helen Toner. (Excerpts.)

The post May 2019 Newsletter appeared first on Machine Intelligence Research Institute.

May 2019 Newsletter


New paper: "Delegative reinforcement learning"

MIRI Research Associate Vanessa Kosoy has written a new paper, "Delegative reinforcement learning: Learning to avoid traps with a little help." Kosoy will be presenting the paper at the ICLR 2019 SafeML workshop in two weeks. The abstract reads:
Most known regret bounds for reinforcement learning are either episodic or assume an environment without traps. We derive a regret bound without making either assumption, by allowing the algorithm to occasionally delegate an action to an external advisor. We thus arrive at a setting of active one-shot model-based reinforcement learning that we call DRL (delegative reinforcement learning.)
The algorithm we construct in order to demonstrate the regret bound is a variant of Posterior Sampling Reinforcement Learning supplemented by a subroutine that decides which actions should be delegated. The algorithm is not anytime, since the parameters must be adjusted according to the target time discount. Currently, our analysis is limited to Markov decision processes with finite numbers of hypotheses, states and actions.
The goal of Kosoy's work on DRL is to put us on a path toward having a deep understanding of learning systems with human-in-the-loop and formal performance guarantees, including safety guarantees. DRL tries to move us in this direction by providing models in which such performance guarantees can be derived.
While these models still make many unrealistic simplifying assumptions, Kosoy views DRL as already capturing some of the most essential features of the problem—and she has a fairly ambitious vision of how this framework might be further developed.
Kosoy previously described DRL in the post Delegative Reinforcement Learning with a Merely Sane Advisor. One feature of DRL Kosoy described here but omitted from the paper (for space reasons) is DRL's application to corruption. Given certain assumptions, DRL ensures that a formal agent will never have its reward or advice channel tampered with (corrupted). As a special case, the agent's own advisor cannot cause the agent to enter a corrupt state. Similarly, the general protection from traps described in "Delegative reinforcement learning" also protects the agent from harmful self-modifications.
Another set of DRL results that didn't make it into the paper is Catastrophe Mitigation Using DRL. In this variant, a DRL agent can mitigate catastrophes that the advisor would not be able to mitigate on its own—something that isn't supported by the more strict assumptions about the advisor in standard DRL.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New paper: "Delegative reinforcement learning" appeared first on Machine Intelligence Research Institute.

New paper: “Delegative reinforcement learning”


April 2019 Newsletter

Updates

New research posts: Simplified Preferences Needed, Simplified Preferences Sufficient; Smoothmin and Personal Identity; Example Population Ethics: Ordered Discounted Utility; A Theory of Human Values; A Concrete Proposal for Adversarial IDA
MIRI has received a set of new grants from the Open Philanthropy Project and the Berkeley Existential Risk Initiative.

News and links

From the DeepMind safety team and Alex Turner: Designing Agent Incentives to Avoid Side Effects.
From Wei Dai: Three Ways That "Sufficiently Optimized Agents Appear Coherent" Can Be False; What's Wrong With These Analogies for Understanding Informed Oversight and IDA?; and The Main Sources of AI Risk?
Other recent write-ups: Issa Rice's Comparison of Decision Theories; Paul Christiano's More Realistic Tales of Doom; and Linda Linsefors' The Game Theory of Blackmail.
OpenAI's Geoffrey Irving describes AI safety via debate on FLI's AI Alignment Podcast.
A webcomic's take on AI x-risk concepts: Seed.

The post April 2019 Newsletter appeared first on Machine Intelligence Research Institute.

April 2019 Newsletter


New grants from the Open Philanthropy Project and BERI

I'm happy to announce that MIRI has received two major new grants:

A two-year grant totaling $2,112,500 from the Open Philanthropy Project.
A $600,000 grant from the Berkeley Existential Risk Initiative.

The Open Philanthropy Project's grant was awarded as part of the first round of grants recommended by their new committee for effective altruism support:
We are experimenting with a new approach to setting grant sizes for a number of our largest grantees in the effective altruism community, including those who work on long-termist causes. Rather than have a single Program Officer make a recommendation, we have created a small committee, comprised of Open Philanthropy staff and trusted outside advisors who are knowledgeable about the relevant organizations. […] We average the committee members' votes to arrive at final numbers for our grants.
The Open Philanthropy Project's grant is separate from the three-year $3.75 million grant they awarded us in 2017, the third $1.25 million disbursement of which is still scheduled for later this year. This new grant increases the Open Philanthropy Project's total support for MIRI from $1.4 million1 in 2018 to ~$2.31 million in 2019, but doesn't reflect any decision about how much total funding MIRI might receive from Open Phil in 2020 (beyond the fact that it will be at least ~$1.06 million).
Going forward, the Open Philanthropy Project currently plans to determine the size of any potential future grants to MIRI using the above committee structure.
We're very grateful for this increase in support from BERI and the Open Philanthropy Project—both organizations that already numbered among our largest funders of the past few years. We expect these grants to play an important role in our decision-making as we continue to grow our research team in the ways described in our 2018 strategy update and fundraiser posts.
The $1.4 million counts the Open Philanthropy Project's $1.25 million disbursement in 2018, as well as a $150,000 AI Safety Retraining Program grant to MIRI.The post New grants from the Open Philanthropy Project and BERI appeared first on Machine Intelligence Research Institute.


March 2019 Newsletter

  Want to be in the reference class "people who solve the AI alignment problem"?
  We now have a guide on how to get started, based on our experience of what tends to make research groups successful. (Also on the AI Alignment Forum.)
Other updates

Demski and Garrabrant's introduction to MIRI's agent foundations research, "Embedded Agency," is now available (in lightly edited form) as an arXiv paper.
New research posts: How Does Gradient Descent Interact with Goodhart?; "Normative Assumptions" Need Not Be Complex; How the MtG Color Wheel Explains AI Safety; Pavlov Generalizes
Several MIRIx groups are expanding and are looking for new members to join.
Our summer fellows program is accepting applications through March 31.
LessWrong's web edition of Rationality: From AI to Zombies at lesswrong.com/rationality is now fully updated to reflect the print edition of Map and Territory and How to Actually Change Your Mind, the first two books. (Announcement here.)

News and links

OpenAI's GPT-2 model shows meaningful progress on a wide variety of language tasks. OpenAI adds:
Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights. […] We believe our release strategy limits the initial set of organizations who may choose to [open source our results], and gives the AI community more time to have a discussion about the implications of such systems.

The Verge discusses OpenAI's language model concerns along with MIRI's disclosure policies for our own research. See other discussion by Jeremy Howard, John Seymour, and Ryan Lowe.
AI Impacts summarizes evidence on good forecasting practices from the Good Judgment Project.
Recent AI alignment ideas and discussion: Carey on quantilization; Filan on impact regularization methods; Saunders' HCH Is Not Just Mechanical Turk and RL in the Iterated Amplification Framework; Dai on philosophical difficulty (1, 2); Hubinger on ascription universality; and Everitt's Understanding Agent Incentives with Causal Influence Diagrams.
The Open Philanthropy Project announces their largest grant to date: $55 million to launch the Center for Security and Emerging Technology, a Washington, D.C. think tank with an early focus on "the intersection of security and artificial intelligence". See also CSET's many jobpostings.

The post March 2019 Newsletter appeared first on Machine Intelligence Research Institute.

March 2019 Newsletter