Applications are open for the MIRI Summer Fellows Program!

CFAR and MIRI are running our fifth annual MIRI Summer Fellows Program (MSFP) in the San Francisco Bay Area from August 9 to August 24, 2019.
MSFP is an extended retreat for mathematicians and programmers with a serious interest in making technical progress on the problem of AI alignment.  It includes an overview of CFAR’s applied rationality content, a breadth-first grounding in the MIRI perspective on AI safety, and multiple days of actual hands-on research with participants and MIRI staff attempting to make inroads on open questions.

Program Description
The intent of the program is to boost participants, as far as possible, in four overlapping areas:
Doing rationality inside a human brain: understanding, with as much fidelity as possible, what phenomena and processes drive and influence human thinking and reasoning, so that we can account for our own biases and blindspots, better recruit and use the various functions of our brains, and, in general, be less likely to trick ourselves, gloss over our confusions, or fail to act in alignment with our endorsed values.
Epistemic rationality, especially the subset of skills around deconfusion. Building the skill of noticing where the dots don’t actually connect; answering the question “why do we think we know what we think we know?”, particularly when it comes to predictions and assertions around the future development of artificial intelligence.
Grounding in the current research landscape surrounding AI: being aware of the primary disagreements among leaders in the field, and the arguments for various perspectives and claims. Understanding the current open questions, and why different ones seem more pressing or real under different assumptions. Being able to follow the reasoning behind various alignment schemes/theories/proposed interventions, and being able to evaluate those interventions with careful reasoning and mature (or at least more-mature-than-before) intuitions.
Generative research skill: the ability to make real and relevant progress on questions related to the field of AI alignment without losing track of one’s own metacognition.  The parallel processes of using one’s mental tools, critiquing and improving one’s mental tools, and making one’s own progress or deconfusion available to others through talks, papers, and models.  Anything and everything involved in being the sort of thinker who can locate a good question, sniff out promising threads, and collaborate effectively with others and with the broader research ecosystem.
Food and lodging are provided free of charge at CFAR’s workshop venue in Bodega Bay, California.  Participants must be able to remain onsite, largely undistracted for the duration of the program (e.g. no major appointments in other cities, no large looming academic or professional deadlines just after the program).
If you have any questions or comments, please send an email to colm@intelligence.org, or, if you suspect others would also benefit from hearing the answer, post them here.
Update April 23, 2019: Applications closed on March 31, 2019 and finalists are being contacted by a MIRI staff member for 1–2 Skype interviews. Admissions decisions — yes, no, waitlist — will go out no later than April 30th.

The post Applications are open for the MIRI Summer Fellows Program! appeared first on Machine Intelligence Research Institute.


A new field guide for MIRIx

We’ve just released a field guide for MIRIx groups, and for other people who want to get involved in AI alignment research.
MIRIx is a program where MIRI helps cover basic expenses for outside groups that want to work on open problems in AI safety. You can start your own group or find information on existing meet-ups at intelligence.org/mirix.
Several MIRIx groups have recently been ramping up their activity, including:

UC Irvine: Daniel Hermann is starting a MIRIx group in Irvine, California. Contact him if you’d like to be involved.
Seattle: MIRIxSeattle is a small group that’s in the process of restarting and increasing its activities. Contact Pasha Kamyshev if you’re interested.
Vancouver: Andrew McKnight and Evan Gaensbauer are looking for more people who’d like to join MIRIxVancouver events.

The new alignment field guide is intended to provide tips and background models to MIRIx groups, based on our experience of what tends to make a research group succeed or fail.
The guide begins:

Preamble I: Decision Theory
Hello!  You may notice that you are reading a document.
This fact comes with certain implications.  For instance, why are you reading this?  Will you finish it?  What decisions will you come to as a result?  What will you do next?
Notice that, whatever you end up doing, it’s likely that there are dozens or even hundreds of other people, quite similar to you and in quite similar positions, who will follow reasoning which strongly resembles yours, and make choices which correspondingly match.
Given that, it’s our recommendation that you make your next few decisions by asking the question “What policy, if followed by all agents similar to me, would result in the most good, and what does that policy suggest in my particular case?”  It’s less of a question of trying to decide for all agents sufficiently-similar-to-you (which might cause you to make the wrong choice out of guilt or pressure) and more something like “if I were in charge of all agents in my reference class, how would I treat instances of that class with my specific characteristics?”
If that kind of thinking leads you to read further, great.  If it leads you to set up a MIRIx chapter, even better.  In the meantime, we will proceed as if the only people reading this document are those who justifiably expect to find it reasonably useful.
Preamble II: Surface Area
Imagine that you have been tasked with moving a cube of solid iron that is one meter on a side.  Given that such a cube weighs ~16000 pounds, and that an average human can lift ~100 pounds, a naïve estimation tells you that you can solve this problem with ~150 willing friends.
But of course, a meter cube can fit at most something like 10 people around it.  It doesn’t matter if you have the theoretical power to move the cube if you can’t bring that power to bear in an effective manner.  The problem is constrained by its surface area.
MIRIx chapters are one of the best ways to increase the surface area of people thinking about and working on the technical problem of AI alignment.  And just as it would be a bad idea to decree “the 10 people who happen to currently be closest to the metal cube are the only ones allowed to think about how to think about this problem”, we don’t want MIRI to become the bottleneck or authority on what kinds of thinking can and should be done in the realm of embedded agency and other relevant fields of research.
The hope is that you and others like you will help actually solve the problem, not just follow directions or read what’s already been written.  This document is designed to support people who are interested in doing real groundbreaking research themselves.
(Read more)

The post A new field guide for MIRIx appeared first on Machine Intelligence Research Institute.

A new field guide for MIRIx


February 2019 Newsletter

Updates

Ramana Kumar and Scott Garrabrant argue that the AGI safety community should begin prioritizing “approaches that work well in the absence of human models”:
[T]o the extent that human modelling is a good idea, it is important to do it very well; to the extent that it is a bad idea, it is best to not do it at all. Thus, whether or not to do human modelling at all is a configuration bit that should probably be set early when conceiving of an approach to building safe AGI.

New research forum posts: Conditional Oracle EDT Equilibria in Games; Non-Consequentialist Cooperation?; When is CDT Dutch-Bookable?; CDT=EDT=UDT
The MIRI Summer Fellows Program is accepting applications through the end of March! MSFP is a free two-week August retreat co-run by MIRI and CFAR, intended to bring people up to speed on problems related to embedded agency and AI alignment, train research-relevant skills and habits, and investigate open problems in the field.
MIRI’s Head of Growth, Colm Ó Riain, reviews how our 2018 fundraiser went.
From Eliezer Yudkowsky: “Along with adversarial resistance and transparency, what I’d term ‘conservatism’, or trying to keep everything as interpolation rather than extrapolation, is one of the few areas modern ML can explore that I see as having potential to carry over directly to serious AGI safety.”

News and links

Eric Drexler has released his book-length AI safety proposal: Reframing Superintelligence: Comprehensive AI Services as General Intelligence. See discussion by Peter McCluskey, Richard Ngo, and Rohin Shah.
Other recent AI alignment posts include Andreas Stuhlmüller’s Factored Cognition and Alex Turner’s Penalizing Impact via Attainable Utility Preservation, and a host of new write-ups by Stuart Armstrong.

The post February 2019 Newsletter appeared first on Machine Intelligence Research Institute.

February 2019 Newsletter


Thoughts on Human Models

This is a joint post by MIRI Research Associate and DeepMind Research Scientist Ramana Kumar and MIRI Research Fellow Scott Garrabrant, cross-posted from the AI Alignment Forum and LessWrong.

Human values and preferences are hard to specify, especially in complex domains. Accordingly, much AGI safety research has focused on approaches to AGI design that refer to human values and preferences indirectly, by learning a model that is grounded in expressions of human values (via stated preferences, observed behaviour, approval, etc.) and/or real-world processes that generate expressions of those values. There are additionally approaches aimed at modelling or imitating other aspects of human cognition or behaviour without an explicit aim of capturing human preferences (but usually in service of ultimately satisfying them). Let us refer to all these models as human models.
In this post, we discuss several reasons to be cautious about AGI designs that use human models. We suggest that the AGI safety research community put more effort into developing approaches that work well in the absence of human models, alongside the approaches that rely on human models. This would be a significant addition to the current safety research landscape, especially if we focus on working out and trying concrete approaches as opposed to developing theory. We also acknowledge various reasons why avoiding human models seems difficult.

Problems with Human Models
To be clear about human models, we draw a rough distinction between our actual preferences (which may not be fully accessible to us) and procedures for evaluating our preferences. The first thing, actual preferences, is what humans actually want upon reflection. Satisfying our actual preferences is a win. The second thing, procedures for evaluating preferences, refers to various proxies for our actual preferences such as our approval, or what looks good to us (with necessarily limited information or time for thinking). Human models are in the second category; consider, as an example, a highly accurate ML model of human yes/no approval on the set of descriptions of outcomes. Our first concern, described below, is about overfitting to human approval and thereby breaking its connection to our actual preferences. (This is a case of Goodhart’s law.)

Less Independent Audits
Imagine we have built an AGI system and we want to use it to design the mass transit system for a new city. The safety problems associated with such a project are well recognised; suppose we are not completely sure we have solved them, but are confident enough to try anyway. We run the system in a sandbox on some fake city input data and examine its outputs. Then we run it on some more outlandish fake city data to assess robustness to distributional shift. The AGI’s outputs look like reasonable transit system designs and considerations, and include arguments, metrics, and other supporting evidence that they are good. Should we be satisfied and ready to run the system on the real city’s data, and to implement the resulting proposed design?
We suggest that an important factor in the answer to this question is whether the AGI system was built using human modelling or not. If it produced a solution to the transit design problem (that humans approve of) without human modelling, then we would more readily trust its outputs. If it produced a solution we approve of with human modelling, then although we expect the outputs to be in many ways about good transit system design (our actual preferences) and in many ways suited to being approved by humans, to the extent that these two targets come apart we must worry about having overfit to the human model at the expense of the good design. (Why not the other way around? Because our assessment of the sandboxed results uses human judgement, not an independent metric for satisfaction of our actual preferences.)
Humans have a preference for not being wrong about the quality of a design, let alone being fooled about it. How much do we want to rely on having correctly captured these preferences in our system? If the system is modelling humans, we strongly rely on the system learning and satisfying these preferences, or else we expect to be fooled to the extent that a good-looking but actually bad transit system design is easier to compose than an actually-good design. On the other hand, if the system is not modelling humans, then the fact that its output looks like a good design is better evidence that it is in fact a good design. Intuitively, if we consider sampling possible outputs and condition on the output looking good (via knowledge of humans), the probability of it being good (via knowledge of the domain) is higher when the system’s knowledge is more about what is good than what looks good.
Here is a handle for this problem: a desire for an independent audit of the system’s outputs. When a system uses human modelling, the mutual information between its outputs and the auditing process (human judgement) is higher. Thus, using human models reduces our ability to do independent audits.
Avoiding human models does not avoid this problem altogether. There is still an “outer-loop optimisation” version of the problem. If the system produces a weird or flawed design in sandbox, and we identify this during an audit, we will probably reject the solution and attempt to debug the system that produced it. This introduces a bias on the overall process (involving multiple versions of the system over phases of auditing and debugging) towards outputs that fool our auditing procedure.
However, outer-loop optimisation pressures are weaker, and therefore less worrying, than in-loop optimisation pressures. We would argue that the problem is much worse, i.e., the bias towards fooling is stronger, when one uses human modelling. This is because the relevant optimisation is in-loop instead and is encountered more often.
As one more analogy to illustrate this point, consider a classic Goodhart’s law example of teaching to the test. If you study the material, then take a test, your test score reveals your knowledge of the material fairly well. If you instead study past tests, your test score reveals your ability to pass tests, which may be correlated with your knowledge of the material but is increasingly less likely to be so correlated as your score goes up. Here human modelling is analogous to past tests and actual preferences are analogous to the material. Taking the test is analogous to an audit, which we want to be independent from the study regimen.
Risk from Bugs
We might implement our first AGI system incorrectly in a mundane sense. Specifically, even if we fully develop a theory of safe or aligned AGI, we might fail to implement that theory due to bugs or problems with our implementation techniques. In this case, we would be relatively better off if the mutual information between the AGI’s knowledge and human preferences is low. We expect the system’s behaviour to be dependent on its knowledge in some way, and we expect implementation errors to shift the nature of that dependence away from our intentions and expectations. Incorrect behaviour that depends on human preferences seems more dangerous than incorrect behaviour that does not.
Consider the space of AGI system implementations, under a metric like similarity to an intended design (equivalently: severity of deviation from the design due to bugs). We want all the points near the first AGI system we build to be safe, because we may end up with a slightly different design than intended for reasons such as being confused about what we are doing or making implementation errors.
There are at least three ways in which the risk from bugs can manifest.
Incorrectly Encoded Values: Supposing we intend the first use of AGI to be solving some bounded and well-specified task, but we misunderstand or badly implement it so much that what we end up with is actually unboundedly optimising some objective function. Then it seems better if that objective is something abstract like puzzle solving rather than something more directly connected to human preferences: consider, as a toy example, if the sign (positive/negative) around the objective were wrong.
Manipulation: The earlier arguments for independent audits do not just apply to the specific tasks we would plan to audit, but also to any activities an AGI system might carry out that humans might disapprove of. Examples include finding ways to hack into our supposedly secure systems, hiding its intentions and activity from us, or outright manipulating us. These tasks are much easier with access to a good psychological model of humans, which can be used to infer what mistakes we might make, or what loopholes we might overlook, or how we might respond to different behaviour from the system.
Human modelling is very close to human manipulation in design space. A system with accurate models of humans is close to a system which successfully uses those models to manipulate humans.
Threats: Another risk from bugs comes not from the AGI system caring incorrectly about our values, but from having inadequate security. If our values are accurately encoded in an AGI system that cares about satisfying them, they become a target for threats from other actors who can gain from manipulating the first system. More examples and perspectives on this problem have been described here.
The increased risk from bugs of human modelling can be summarised as follows: whatever the risk that AGI systems produce catastrophic outcomes due to bugs, the very worst outcomes seem more likely if the system was trained using human modelling because these worst outcomes depend on the information in human models.
Less independent audits and the risk from bugs can both be mitigated by preserving independence of the system from human model information, so the system cannot overfit to that information or use it perversely. The remaining two problems we consider, mind crime and unexpected agents, depend more heavily on the claim that modelling human preferences increases the chances of simulating something human-like.
Mind Crime
Many computations may produce entities that are morally relevant because, for example, they constitute sentient beings that experience pain or pleasure. Bostrom calls improper treatment of such entities “mind crime”. Modelling humans in some form seems more likely to result in such a computation than not modelling them, since humans are morally relevant and the system’s models of humans may end up sharing whatever properties make humans morally relevant.
Unexpected Agents
Similar to the mind crime point above, we expect AGI designs that use human modelling to be more at risk of producing subsystems that are agent-like, because humans are agent-like. For example, we note that trying to predict the output of consequentialist reasoners can reduce to an optimisation problem over a space of things that contains consequentialist reasoners. A system engineered to predict human preferences well seems strictly more likely to run into problems associated with misaligned sub-agents. (Nevertheless, we think the amount by which it is more likely is small.)

Safe AGI Without Human Models is Neglected
Given the independent auditing concern, plus the additional points mentioned above, we would like to see more work done on practical approaches to developing safe AGI systems that do not depend on human modelling. At present, this is a neglected area in the AGI safety research landscape. Specifically, work of the form “Here’s a proposed approach, here are the next steps to try it out or investigate further”, which we might term engineering-focused research, is almost entirely done in a human-modelling context. Where we do see some safety work that eschews human modelling, it tends to be theory-focused research, for example, MIRI’s work on agent foundations. This does not fill the gap of engineering-focused work on safety without human models.
To flesh out the claim of a gap, consider the usual formulations of each of the following efforts within safety research: iterated distillation and amplification, debate, recursive reward modelling, cooperative inverse reinforcement learning, and value learning. In each case, there is human modelling built into the basic setup for the approach. However, we note that the technical results in these areas may in some cases be transportable to a setup without human modelling, if the source of human feedback (etc.) is replaced with a purely algorithmic, independent system.
Some existing work that does not rely on human modelling includes the formulation of safely interruptible agents, the formulation of impact measures (or side effects), approaches involving building AI systems with clear formal specifications (e.g., some versions of tool AIs), some versions of oracle AIs, and boxing/containment. Although they do not rely on human modelling, some of these approaches nevertheless make most sense in a context where human modelling is happening: for example, impact measures seem to make most sense for agents that will be operating directly in the real world, and such agents are likely to require human modelling. Nevertheless, we would like to see more work of all these kinds, as well as new techniques for building safe AGI that does not rely on human modelling.

Difficulties in Avoiding Human Models
A plausible reason why we do not yet see much research on how to build safe AGI without human modelling is that it is difficult. In this section, we describe some distinct ways in which it is difficult.
Usefulness
It is not obvious how to put a system that does not do human modelling to good use. At least, it is not as obvious as for the systems that do human modelling, since they draw directly on sources (e.g., human preferences) of information about useful behaviour. In other words, it is unclear how to solve the specification problem—how to correctly specify desired (and only desired) behaviour in complex domains—without human modelling. The “against human modelling” stance calls for a solution to the specification problem wherein useful tasks are transformed into well-specified, human-independent tasks either solely by humans or by systems that do not model humans.
To illustrate, suppose we have solved some well-specified, complex but human-independent task like theorem proving or atomically precise manufacturing. Then how do we leverage this solution to produce a good (or better) future? Empowering everyone, or even a few people, with access to a superintelligent system that does not directly encode their values in some way does not obviously produce a future where those values are realised. (This seems related to Wei Dai’s human-safety problem.)
Implicit Human Models
Even seemingly “independent” tasks leak at least a little information about their origins in human motivations. Consider again the mass transit system design problem. Since the problem itself concerns the design of a system for use by humans, it seems difficult to avoid modelling humans at all in specifying the task. More subtly, even highly abstract or generic tasks like puzzle solving contain information about the sources/designers of the puzzles, especially if they are tuned for encoding more obviously human-centred problems. (Work by Shah et al. looks at using the information about human preferences that is latent in the world.)
Specification Competitiveness / Do What I Mean
Explicit specification of a task in the form of, say, an optimisation objective (of which a reinforcement learning problem would be a specific case) is known to be fragile: there are usually things we care about that get left out of explicit specifications. This is one of the motivations for seeking more and more high level and indirect specifications, leaving more of the work of figuring out what exactly is to be done to the machine. However, it is currently hard to see how to automate the process of turning tasks (vaguely defined) into correct specifications without modelling humans.
Performance Competitiveness of Human Models
It could be that modelling humans is the best way to achieve good performance on various tasks we want to apply AGI systems to for reasons that are not simply to do with understanding the problem specification well. For example, there may be aspects of human cognition that we want to more or less replicate in an AGI system, for competitiveness at automating those cognitive functions, and those aspects may carry a lot of information about human preferences with them in a hard to separate way.

What to Do Without Human Models?
We have seen arguments for and against aspiring to solve AGI safety using human modelling. Looking back on these arguments, we note that to the extent that human modelling is a good idea, it is important to do it very well; to the extent that it is a bad idea, it is best to not do it at all. Thus, whether or not to do human modelling at all is a configuration bit that should probably be set early when conceiving of an approach to building safe AGI.
It should be noted that the arguments above are not intended to be decisive, and there may be countervailing considerations which mean we should promote the use of human models despite the risks outlined in this post. However, to the extent that AGI systems with human models are more dangerous than those without, there are two broad lines of intervention we might attempt. Firstly, it may be worthwhile to try to decrease the probability that advanced AI develops human models “by default”, by promoting some lines of research over others. For example, an AI trained in a procedurally-generated virtual environment seems significantly less likely to develop human models than an AI trained on human-generated text and video data.
Secondly, we can focus on safety research that does not require human models, so that if we eventually build AGI systems that are highly capable without using human models, we can make them safer without needing to teach them to model humans. Examples of such research, some of which we mentioned earlier, include developing human-independent methods to measure negative side effects, to prevent specification gaming, to build secure approaches to containment, and to extend the usefulness of task-focused systems.

Acknowledgements: thanks to Daniel Kokotajlo, Rob Bensinger, Richard Ngo, Jan Leike, and Tim Genewein for helpful comments on drafts of this post.

The post Thoughts on Human Models appeared first on Machine Intelligence Research Institute.

Thoughts on Human Models


Our 2018 Fundraiser Review

Our 2018 Fundraiser ended on December 31 with the five week campaign raising $951,8171 from 348 donors to help advance MIRI’s mission. We surpassed our Mainline Target ($500k) and made it more than halfway again to our Accelerated Growth Target ($1.2M). We’re grateful to all of you who supported us. Thank you!

Target 1$500,000Completed

      Target 2
$1,200,000
In Progress

Fundraiser Concluded
348 donors contributed

×
Target Descriptions

Target 1
Target 2

$500k: Mainline target
This target represents the difference between what we’ve raised so far this year, and our point estimate for business-as-usual spending next year.

$1.2M: Accelerated growth target
This target represents what’s needed for our funding streams to keep pace with our growth toward the upper end of our projections.

With cryptocurrency prices significantly lower than during our 2017 fundraiser, we received less of our funding (~6%) from holders of cryptocurrency this time around. Despite this, our fundraiser was a success, in significant part thanks to the leverage gained by MIRI supporters’ participation in multiple matching campaigns during the fundraiser, including WeTrust Spring’s Ethereum-matching campaign, Facebook’s Giving Tuesday event and professional poker player Dan Smith’s Double Up Drive, expertly administered by Raising for Effective Giving.

Together with significant matching funds generated through donors’ employer matching programs, matching donations accounted for ~37% of the total funds raised during the fundraiser.
1. WeTrust Spring
MIRI participated, along with 17 other non-profit organizations, in WeTrust’s Spring innovative ETH-matching event, which ran thru Giving Tuesday, November 27. The event was the first-ever implementation of Glen Weyl, Zoë Hitzig, and Vitalik Buterin’s Liberal Radicalism (LR) model for non-profit funding matching. Unlike most matching campaigns, which match exclusively based on total amount donated, this campaign matched in a way that heavily factored in the number of unique donors when divvying out the matching pool, a feature WeTrust highlighted as “Democratic Donation Matching”.
During MIRI’s week-long campaign leading up to Giving Tuesday, some supporters went deep into trying to determine exactly what instantiation of the model WeTrust had created — how exactly DO the large donations provide leverage of 450% match rate for minimum donations of .1 ETH? Our supporters’ excitement about this new matching model was also evident in the many donations that were made — as WeTrust reported in their blog post, “MIRI, the Machine Intelligence Research Institute was the winner, clocking in 64 qualified donations totaling 147.751 ETH, then Lupus Foundation in second with 22 qualified donations and 23.851 total ETH.” Thanks to our supporters’ donations, MIRI received over 91% of the matching funds allotted by WeTrust and, all told, we received ETH worth more than $31,000 from the campaign. Thank you!
2. Facebook Giving Tuesday Event
Some of our hardiest supporters set their alarms clocks extra early to support us in Facebook’s Giving Tuesday matching event, which kicked off at 5:00am EST on Giving Tuesday. Donations made before the $7M matching pool was exhausted were matched 1:1 by Facebook/PayPal, up to a maximum of $250,000 per organization, and a limit of $20,000 per donor, and $2500 per donation.
MIRI supporters, some with our tipsheet in hand, pointed their browsers — and credit cards — at MIRI’s fundraiser Facebook Page (and another page set up by the folks behind the EA Giving Tuesday Donation Matching Initiative — thank you Avi and William!), and clicked early and often. During the previous year’s event, it took only 86 seconds for the $2M matching pool to be exhausted. This year saw a significantly larger $7M pool being exhausted dramatically quicker, sometime in the 16th second. Fortunately, before it ended, 11 MIRI donors had already made 20 donations totalling $40,072.

Overall, 66% of the $61,023 donated to MIRI on Facebook on Giving Tuesday was matched by Facebook/PayPal, resulting in a total of $101,095. Thank you to everyone who participated, especially the early risers who so effectively leveraged matching funds on MIRI’s behalf including Quinn Maurmann, Richard Schwall, Alan Chang, William Ehlhardt, Daniel Kokotajlo, John Davis, Herleik Holtan and others. You guys rock!
You can read more about the general EA community’s fundraising performance on Giving Tuesday in Ari Norowitz’s retrospective on the EA Forum.
3. Double Up Drive Challenge

Poker player Dan Smith and a number of his fellow professional players came together for another end-of-year Matching Challenge — once again administered by Raising for Effective Giving (REG), who have facilitated similar matching opportunities in years past.
Starting on Giving Tuesday, November 27, $940,000 in matching funds was made available for eight charities focused on near-term causes (Malaria Consortium, GiveDirectly, Helen Keller International, GiveWell, Animal Charity Evaluators, Good Food Institute, StrongMinds and the Massachusetts Bail Fund); and, with the specific support of poker pro Aaron Merchak, $200,000 in matching funds was made available for two charities focused on improving the long-term future of our civilization, MIRI and REG.
With the addition of an anonymous sponsor to Dan’s roster in early December, an extra $150,000 was added to the near-term causes pool and, then, a week later, after his win at the DraftKings World Championship, Tom Crowley followed through on his pledge to donate half of his total event winnings to the drive, adding significantly increased funding, $1.127M, to the drive’s overall matching pool as well as 2 more organizations — Against Malaria Foundation and EA Funds’ Long-Term Future Fund.
The last few days of the Drive saw a whirlwind of donations being made to all organizations, causing the pool of $2.417M to be exhausted 24 hours before the declared end of the drive (December 29) at which point Martin Crowley came in to match all donations made in the last day, thus increasing the matched donations to over $2.7M.
In total, MIRI donors had $229,000 matched during the event. We’re very grateful to all these donors, to Dan Smith for instigating this phenomenally successful event, and to his fellow sponsors and especially Aaron Merchak and Martin Crowley for matching donations made to MIRI. Finally, a big shout-out to REG for facilitating and administering so effectively – thank you Stefan and Alfredo!
4. Corporate Matching
A number of MIRI supporters work at corporations that match contributions made by their employees to 501(c)(3) organizations like MIRI. During the duration of MIRI’s fundraiser, over $62,000 in matching funds from various Employee Matching Programs was leveraged by our supporters, adding to the significant matching corporate funds already leveraged during 2018 by these and other MIRI supporters.

We’re extremely grateful for all the support we received during this fundraiser, especially the effective leveraging of the numerous matching opportunities, and are excited about the opportunity it creates for us to continue to grow our research team.
If you know of — or want to discuss — any giving/matching/support MIRI opportunities in 2019, please get in touch with me2 at colm@intelligence.org. Thank you!

The exact total is still subject to change as we continue to process a small number of donations.

Colm Ó Riain is MIRI’s Head of Growth. Colm coordinates MIRI’s philanthropic and recruitment strategy to support MIRI’s growth plans.

The post Our 2018 Fundraiser Review appeared first on Machine Intelligence Research Institute.

Our 2018 Fundraiser Review


January 2019 Newsletter

Our December fundraiser was a success, with 348 donors contributing just over $950,000. Supporters leveraged a variety of matching opportunities, including employer matching programs, WeTrust Spring’s Ethereum-matching campaign, Facebook’s Giving Tuesday event, and professional poker players Dan Smith, Aaron Merchak, and Martin Crowley’s Double Up Drive, expertly facilitated by Raising for Effective Giving.
In all, matching donations accounted for just over one third of the funds raised. Thank you to everyone who contributed!

News and links

From NVIDIA’s Tero Karras, Samuli Laine, and Timo Aila: “A Style-Based Generator Architecture for Generative Adversarial Networks.”
How AI Training Scales: OpenAI’s Sam McCandlish, Jared Kaplan, and Dario Amodei introduce a method that “predicts the parallelizability of neural network training on a wide range of tasks”.
From Vox: The case for taking AI seriously as a threat to humanity. See also Gizmodo‘s article on AI risk.

The post January 2019 Newsletter appeared first on Machine Intelligence Research Institute.

January 2019 Newsletter


December 2018 Newsletter

Edward Kmett has joined the MIRI team! Edward is a prominent Haskell developer who popularized the use of lenses for functional programming, and currently maintains many of the libraries around the Haskell core libraries.
I’m also happy to announce another new recruit: James Payor. James joins the MIRI research team after three years at Draftable, a software startup. He previously studied math and CS at MIT, and he holds a silver medal from the International Olympiad in Informatics, one of the most prestigious CS competitions in the world.
In other news, we’ve today released a new edition of Rationality: From AI to Zombies with a fair amount of textual revisions and (for the first time) a print edition!
Finally, our 2018 fundraiser has passed the halfway mark on our first target! (And there’s currently $136,000 available in dollar-for-dollar donor matching through the Double Up Drive!)
Other updates

A new paper from Stuart Armstrong and Sören Mindermann: “Occam’s Razor is Insufficient to Infer the Preferences of Irrational Agents.”
New AI Alignment Forum posts: Kelly Bettors; Bounded Oracle Induction
OpenAI’s Jack Clark and Axios discuss research-sharing in AI, following up on our 2018 Update post.
A throwback post from Eliezer Yudkowsky: Should Ethicists Be Inside or Outside a Profession?

News and links

New from the DeepMind safety team: Jan Leike’s Scalable Agent Alignment via Reward Modeling (arXiv) and Viktoriya Krakovna’s Discussion on the Machine Learning Approach to AI Safety.
Two recently released core Alignment Forum sequences: Rohin Shah’s Value Learning and Paul Christiano’s Iterated Amplification.
On the 80,000 Hours Podcast, Catherine Olsson and Daniel Ziegler discuss paths for ML engineers to get involved in AI safety.
Nick Bostrom has a new paper out: “The Vulnerable World Hypothesis.”

The post December 2018 Newsletter appeared first on Machine Intelligence Research Institute.

December 2018 Newsletter


Announcing a new edition of “Rationality: From AI to Zombies”

MIRI is putting out a new edition of Rationality: From AI to Zombies, including the first set of R:AZ print books! Map and Territory (volume 1) and How to Actually Change Your Mind (volume 2) are out today!

Map and Territory is:

$6.50 on Amazon, for the print version.
Pay-what-you-on Gumroad, for PDF, EPUB, and MOBI versions.

How to Actually Change Your Mind is:

$8 on Amazon, for the print version.
Pay-what-you-on Gumroad, for PDF, EPUB, and MOBI versions (available in the next day).

The Rationality: From AI to Zombies compiles Eliezer Yudkowsky’s original Overcoming Bias and Less Wrong sequences, modified to form a more cohesive whole as books.
Map and Territory is the canonical starting point, though we’ve tried to make How to Actually Change Your Mind a good jumping-on point too, since we expect different people to take interest in one book or the other.
The previous edition of Rationality: From AI to Zombies was digital-only, and took the form of a single sprawling ebook. The new version has been revised a fair amount, with larger changes including:

The first sequence in Map and Territory, “Predictably Wrong,” has been substantially reorganized and rewritten, with a goal of making it a much better experience for new readers.

More generally, the books are now more optimized for new readers and less focused on extreme fidelity to Eliezer’s original blog posts, as this was one of the largest requests we got in response to the previous edition of Rationality: From AI to Zombies. Although the book as a whole is mostly unchanged, this represented an update about which option to pick in quite a few textual tradeoffs.

A fair number of essays have been added, removed, or rearranged. The “Against Doublethink” sequence in How to Actually Change Your Mind has been removed entirely, except for one essay (“Singlethink”).

Important links and references are now written out rather than hidden behind Easter egg hyperlinks, so that they’ll show up in print editions too.

Easter egg links are kept around if they’re interesting enough to be worth retaining, but not important enough to deserve a footnote; so there will still be some digital-only content, but the goal is for this to be pretty minor.

A glossary has been added to the back of each book.

Over the coming months, We’ll be rolling out the other four volumes of Rationality: From AI to Zombies. To learn more, see the R:AZ landing page.

The post Announcing a new edition of “Rationality: From AI to Zombies” appeared first on Machine Intelligence Research Institute.


2017 in review

This post reviews MIRI’s activities in 2017, including research, recruiting, exposition, and fundraising activities.
2017 was a big transitional year for MIRI, as we took on new research projects that have a much greater reliance on hands-on programming work and experimentation. We’ve continued these projects in 2018, and they’re described more in our 2018 update. This meant a major focus on laying groundwork for much faster growth than we’ve had in the past, including setting up infrastructure and changing how we recruit to reach out to more people with engineering backgrounds.

At the same time, 2017 was our best year to date on fundraising, as we saw a significant increase in support both from the Open Philanthropy Project and from the cryptocurrency community, which responded to the crypto boom with a great deal of generosity toward us. This put us in an excellent position to move ahead with our plans with confidence, and to focus more of our effort on technical research and growth.
The review this year is coming out far later than usual, for which I apologize. One of the main reasons for this is that I felt that a catalogue of our 2017 activities would be much less informative if I couldn’t cite our 2018 update, which explains a lot of the reasoning behind our new work and how the things we’re doing relate to each other. I apologize for any inconvenience this might have caused people trying to track what MIRI’s been up to. I plan to have our next annual review out much earlier, in the first quarter of 2019.

2017 Research Progress
As described in our 2017 organizational update and elaborated on in much more detail in our recent 2018 update, 2017 saw a significant shift in where we’re putting our research efforts. Although an expanded version of the Agent Foundations agenda continues to be a major focus at MIRI, we’re also now tackling a new set of alignment research directions that lend themselves more to code experiments.
Since early 2017, we’ve been increasingly adopting a policy of not disclosing many of our research results, which has meant that less of our new output is publicly available. Some of our work in 2017 (and 2018) has continued to be made public, however, including research on the AI Alignment Forum.
In 2017, Scott Garrabrant refactored our Agent Foundations agenda into four new categories: decision theory, embedded world-models, robust delegation, and subsystem alignment. Abram Demski and Scott have now co-written an introduction to these four problems, considered as different aspects of the larger problem of “Embedded Agency.”
Comparing our predictions (from March 20171) to our progress over 2017, and using a 1-5 scale where 1 means “limited” progress, 3 means “modest” progress, and 5 means “sizable” progress, we get the following retrospective take on our public-facing research progress:
Decision theory

2015 progress: 3. (Predicted: 3.)
2016 progress: 3. (Predicted: 3.)
2017 progress: 3. (Predicted: 3.)

Our most significant 2017 results include posing and solving a version of the converse Lawvere problem;2 developing cooperative oracles; and improving our understanding of how causal decision theory relates to evidential decision theory (e.g., in Smoking Lesion Steelman).
We also released a number of introductory resources on decision theory, including “Functional Decision Theory” and Decisions Are For Making Bad Outcomes Inconsistent.
Embedded world-models

2015 progress: 5. (Predicted: 3.)
2016 progress: 5. (Predicted: 3.)
2017 progress: 2. (Predicted: 2.)

Key 2017 results in this area include the finding that logical inductors that can see each other dominate each other; and, as a corollary, logical inductor limits dominate each other.
Beyond that, Scott Garrabrant reports that Hyperreal Brouwer shifted his thinking significantly with respect to probabilistic truth predicates, reflective oracles, and logical inductors. Additionally, Vanessa Kosoy’s “Forecasting Using Incomplete Models” built on our previous work on logical inductors to create a cleaner (purely learning-theoretic) formalism for modeling complex environments, showing that the methods developed in “Logical Induction” are useful for applications in classical sequence prediction unrelated to logic.
Robust delegation

2015 progress: 3. (Predicted: 3.)
2016 progress: 4. (Predicted: 3.)
2017 progress: 4. (Predicted: 1.)

We made significant progress on the tiling problem, and also clarified our thinking about Goodhart’s Law (see “Goodhart Taxonomy“). Other noteworthy work in this area includes Vanessa Kosoy’s Delegative Inverse Reinforcement Learning framework, Abram Demski’s articulation of “stable pointers to value” as a central desideratum for value loading, and Ryan Carey’s “Incorrigibility in the CIRL Framework.”
Subsystem alignment (new category)
One of the more significant research shifts at MIRI in 2017 was orienting toward the subsystem alignment problem at all, following discussion such as Eliezer Yudkowsky’s Optimization Daemons, Paul Christiano’s What Does the Universal Prior Actually Look Like?, and Jessica Taylor’s Some Problems with Making Induction Benign. Our high-level thoughts about this problem can be found in Scott Garrabrant and Abram Demski’s recent write-up.

2017 also saw a reduction in our focus on the Alignment for Advanced Machine Learning Systems (AAMLS) agenda. Although we view these problems as highly important, and continue to revisit them regularly, we’ve found AAMLS work to be less obviously tractable than our other research agendas thus far.
On the whole, we continue (at year’s end in 2018) to be very excited by the alignment avenues of attack that we started exploring in earnest in 2017, both with respect to embedded agency and with respect to our newer lines of research.

2017 Research Support Activities
As discussed in our 2018 Update, the new lines of research we’re tackling are much easier to hire for than has been the case for our Agent Foundations research:

This work seems to “give out its own guideposts” more than the Agent Foundations agenda does. While we used to require extremely close fit of our hires on research taste, we now think we have enough sense of the terrain that we can relax those requirements somewhat. We’re still looking for hires who are scientifically innovative and who are fairly close on research taste, but our work is now much more scalable with the number of good mathematicians and engineers working at MIRI.

For that reason, one of our top priorities in 2017 (continuing into 2018) was to set MIRI up to be able to undergo major, sustained growth. We’ve been helped substantially in ramping up our recruitment by Blake Borgeson, a Nature-published computational biologist (and now a MIRI board member) who previously co-founded Recursion Pharmaceuticals and led its machine learning work as CTO.
Concretely, in 2017 we:

Hired research staff including Sam Eisenstat, Abram Demski, Tsvi Benson-Tilsen, Jesse Liptrap, and Nick Tarleton.
Ran the AI Summer Fellows Program with CFAR.
Ran 3 research workshops on the Agent Foundations agenda, the AAMLS agenda, and Paul Christiano’s research agenda. We also ran a large number of internal research retreats and other events.
Ran software engineer trials where participants spent the summer training up to become research engineers, resulting in a hire.

2017 Conversations and Exposition
One of our 2017 priorities was to sync up and compare models more on the strategic landscape with other existential risk and AI safety groups. For snapshots of some of the discussions over the years, see Daniel Dewey’s thoughts on HRAD and Nate’s response; and more recently, Eliezer Yudkowsky and Paul Christiano’s conversations about Paul’s research proposals.
We also did a fair amount of public dialoguing, exposition, and outreach in 2017. On that front we:

Released Inadequate Equilibria, a book by Eliezer Yudkowsky on group- and system-level inefficiencies, and when individuals can hope to do better than the status quo.
Produced research exposition: On Motivations for MIRI’s Highly Reliable Agent Design Research; Ensuring Smarter-Than-Human Intelligence Has a Positive Outcome; Security Mindset and Ordinary Paranoia; Security Mindset and the Logistic Success Curve
Produced strategy and forecasting exposition: Response to Cegłowski on Superintelligence; There’s No Fire Alarm for Artificial General Intelligence; AlphaGo Zero and the Foom Debate; Why We Should Be Concerned About Artificial Superintelligence; A Reply to Francois Chollet on Intelligence Explosion
Received press coverage in The Huffington Post, Vanity Fair, Nautilus, and Wired. We were also interviewed for Mark O’Connell’s To Be A Machine and Richard Clarke and R.P. Eddy’s Warnings: Finding Cassandras to Stop Catastrophes.
Spoke at the O’Reilly AI Conference, and on panels at the Beneficial AI conference and Effective Altruism Global (1, 2).
Presented papers at TARK 2017, FEW 2017, and the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society, and published the Agent Foundations agenda in The Technological Singularity: Managing the Journey.
Participated in other events, including the “Envisioning and Addressing Adverse AI Outcomes” workshop at Arizona State and the UCLA Colloquium on Catastrophic and Existential Risk.

2017 Finances
Fundraising
2017 was by far our best fundraising year to date. We raised a total of $5,849,500, more than 2.5× what we raised in 2016.3 During our annual fundraiser, we also raised double our highest target. We are very grateful for this incredible show of support. This unexpected fundraising success enabled us to move forward with our growth plans with a lot more confidence, and boosted our recruiting efforts in a variety of ways.4
The large increase in funding we saw in 2017 was significantly driven by:

A large influx of cryptocurrency contributions, which made up ~42% of our total contributions in 2017. The largest of these were:

$1.01M in ETH from an anonymous donor.
$764,970 in ETH from Vitalik Buterin, the inventor and co-founder of Ethereum.
$367,575 in BTC from Christian Calderon.
$295,899 in BTC from professional poker players Dan Smith, Tom Crowley and Martin Crowley as part of their Matching Challenge in partnership with Raising for Effective Giving.

Other contributions including:

A $1.25M grant disbursement from the Open Philanthropy Project, significantly increased from the $500k grant they awarded MIRI in 2016.
$200k in grants from the Berkeley Existential Risk Initiative.

As the graph below shows, although our fundraising has increased year over year since 2014, 2017 looks very much like an outlier year relative to our previous growth rate. This was largely driven by the large influx of cryptocurrency contributions, but even excluding those contributions, we raised ~$3.4M, which is 1.5× what we raised in 2016.5

(In this chart and those that follow, “Unlapsed” indicates contributions from past supporters who did not donate in the previous year.)
While the largest contributions drove the overall trend, we saw growth in both the number of contributors and amount contributed across all contributor sizes.

In 2017 we received contributions from 745 unique contributors, 38% more than in 2016 and nearly as many as in 2014 when we participated in SVGives.
Support from international contributors increased from 20% in 2016 to 42% in 2017. This increase was largely driven by the $1.01M ETH donation, but support still increased from 20% to 25% if we ignore this donation. Starting in late 2016, we’ve been working hard to find ways for our international supporters to be able to contribute in a tax-advantaged manner. I expect this percentage to substantially increase in 2018 due to those efforts.6

Spending
In our 2016 fundraiser post, we projected that we’d spend $2–2.2M in 2017. Later in 2017, we revised our estimates to $2.1–2.5M (with a point estimate of $2.25M) along with a breakdown of our estimate across our major budget categories.
Overall, our projections were fairly accurate. Total spending came in at just below $2.1M. The graph below compares our actual spending with our projections.7

The largest deviation from our projected spending came as a result of the researchers who had been working on our AAMLS agenda moving on (on good terms) to other projects.

For past annual reviews, see: 2016, 2015, 2014, and 2013; and for more recent information on what we’ve been up to following 2017, see our 2018 update and fundraiser posts.
These predictions have been edited below to match Scott’s terminology changes as described in 2018 research plans and predictions.Scott coined the name for this problem in his post: The Ubiquitous Converse Lawvere Problem.Note that amounts in this section may vary slightly from our audited financial statements, due to small differences between how we tracked donations internally, and how we are required to report them in our financial statements.See our 2018 fundraiser post for more information.This is similar to 2013, when 33% of our contributions that year came from a single Ripple donation from Jed McCaleb.A big thanks to Colm for all the work he’s put into this; have a look at our Tax-Advantaged Donations page for more information.Our subsequent budget projections have used a simpler set of major budget categories. I’ve translated our 2017 budget projections into this categorization scheme, for our comparison with actual spending, in order to remain consistent with this new scheme.The post 2017 in review appeared first on Machine Intelligence Research Institute.

2017 in review


MIRI’s newest recruit: Edward Kmett!

Prolific Haskell developer Edward Kmett has joined the MIRI team!
Edward is perhaps best known for popularizing the use of lenses for functional programming. Lenses are a tool that provides a compositional vocabulary for accessing parts of larger structures and describing what you want to do with those parts.
Beyond the lens library, Edward maintains a significant chunk of all libraries around the Haskell core libraries, covering everything from automatic differentiation (used heavily in deep learning, computer vision, and financial risk) to category theory (biased heavily towards organizing software) to graphics, SAT bindings, RCU schemes, tools for writing compilers, and more.
Initial support for Edward joining MIRI is coming in the form of funding from long-time MIRI donor Jaan Tallinn. Increased donor enthusiasm has put MIRI in a great position to take on more engineers in general, and to consider highly competitive salaries for top-of-their-field engineers like Edward who are interested in working with us.
At MIRI, Edward is splitting his time between helping us grow our research team and diving in on a line of research he’s been independently developing in the background for some time: building a new language and infrastructure to make it easier for people to write highly complex computer programs with known desirable properties. While we are big fans of his work, Edward’s research is independent of the directions we described in our 2018 Update, and we don’t consider it part of our core research focus.
We’re hugely excited to have Edward at MIRI. We expect to learn and gain a lot from our interactions, and we also hope that having Edward on the team will let him and other MIRI staff steal each other’s best problem-solving heuristics and converge on research directions over time.

As described in our recent update, our new lines of research are heavy on the mix of theoretical rigor and hands-on engineering that Edward and the functional programming community are well-known for:
In common between all our new approaches is a focus on using high-level theoretical abstractions to enable coherent reasoning about the systems we build. A concrete implication of this is that we write lots of our code in Haskell, and are often thinking about our code through the lens of type theory.
MIRI’s nonprofit mission is to ensure that smarter-than-human AI systems, once developed, have a positive impact on the world. And we want to actually succeed in that goal, not just go through the motions of working on the problem.
Our current model of the challenges involved says that the central sticking point for future engineers will likely be that the building blocks of AI just aren’t sufficiently transparent. We think that someone, somewhere, needs to develop some new foundations and deep theory/insights, above and beyond what’s likely to arise from refining or scaling up currently standard techniques.
We think that the skillset of functional programmers tends to be particularly well-suited to this kind of work, and we believe that our new research areas can absorb a large number of programmers and computer scientists. So we want this hiring announcement to double as a hiring pitch: consider joining our research effort!
To learn more about what it’s like to work at MIRI and what kinds of candidates we’re looking for, see our last big post, or shoot MIRI researcher Buck Shlegeris an email.
The post MIRI’s newest recruit: Edward Kmett! appeared first on Machine Intelligence Research Institute.

MIRI’s newest recruit: Edward Kmett!


November 2018 Newsletter

In 2018 Update: Our New Research Directions, Nate Soares discusses MIRI’s new research; our focus on “deconfusion”; some of the thinking behind our decision to default to nondisclosure on new results; and why more people than you might think should come join the MIRI team!
Additionally, our 2018 fundraiser begins today! To kick things off, we’ll be participating in three separate matching campaigns, all focused around Giving Tuesday, Nov. 27; details in our fundraiser post.
Other updates

New alignment posts: A Rationality Condition for CDT Is That It Equal EDT (1, 2); Standard ML Oracles vs. Counterfactual Ones; Addressing Three Problems with Counterfactual Corrigibility; When EDT=CDT, ADT Does Well. See also Paul Christiano’s EDT vs. CDT (1, 2).
Embedded Agency: Scott Garrabrant and Abram Demski’s full sequence is up! The posts serve as our new core introductory resource to MIRI’s Agent Foundations research.
“Sometimes people ask me what math they should study in order to get into agent foundations. My first answer is that I have found the introductory class in every subfield to be helpful, but I have found the later classes to be much less helpful. My second answer is to learn enough math to understand all fixed point theorems….” In Fixed Point Exercises, Scott provides exercises for getting into MIRI’s Agent Foundations research.
MIRI is seeking applicants for a new series of AI Risk for Computer Scientists workshops, aimed at technical people who want to think harder about AI alignment.

News and links

Vox unveils Future Perfect, a new section of their site focused on effective altruism.
80,000 Hours interviews Paul Christiano, including discussion of MIRI/Paul disagreements and Paul’s approach to AI alignment research.
80,000 Hours surveys effective altruism orgs on their hiring needs.
From Christiano, Buck Shlegeris, and Dario Amodei: Learning Complex Goals with Iterated Amplification (arXiv paper).

The post November 2018 Newsletter appeared first on Machine Intelligence Research Institute.

November 2018 Newsletter


MIRI’s 2018 Fundraiser

Update January 2019: MIRI’s 2018 fundraiser is now concluded.

Target 1
$500,000
CompletedTarget 2
$1,200,000In Progress

$946,981
|

|
$0

|
$300,000

|
$600,000

|
$900,000

|
$1,200,000

Fundraiser concluded
345 donors contributed

×
Target Descriptions

Target 1
Target 2

$500k: Mainline target
This target represents the difference between what we’ve raised so far this year, and our point estimate for business-as-usual spending next year.

$1.2M: Accelerated growth target
This target represents what’s needed for our funding streams to keep pace with our growth toward the upper end of our projections.

Donate Now

MIRI is a math/CS research nonprofit with a mission of maximizing the potential humanitarian benefit of smarter-than-human artificial intelligence. You can learn more about the kind of work we do in “Ensuring Smarter-Than-Human Intelligence Has A Positive Outcome” and “Embedded Agency.”
Our funding targets this year are based on a goal of raising enough in 2018 to match our “business-as-usual” budget next year. We view “make enough each year to pay for the next year” as a good heuristic for MIRI, given that we’re a quickly growing nonprofit with a healthy level of reserves and a budget dominated by researcher salaries.
We focus on business-as-usual spending in order to factor out the (likely very large) cost of moving to new spaces in the next couple of years as we continue to grow, which introduces a high amount of variance to the model.1
My current model for our (business-as-usual, outlier-free) 2019 spending ranges from $4.4M to $5.5M, with a point estimate of $4.8M—up from $3.5M this year and $2.1M in 2017. The model takes as input estimated ranges for all our major spending categories, but the overwhelming majority of the variance comes from the number of staff we’ll add to the team.2
In the mainline scenario, our 2019 budget breakdown looks roughly like this:

In this scenario, we currently have ~1.5 years of reserves on hand. Since we’ve raised ~$4.3M between Jan. 1 and Nov. 25,3 our two targets are:

Target 1 ($500k), representing the difference between what we’ve raised so far this year, and our point estimate for business-as-usual spending next year.
Target 2 ($1.2M), what’s needed for our funding streams to keep pace with our growth toward the upper end of our projections.4

Below, we’ll summarize what’s new at MIRI and talk more about our room for more funding.

1. Recent updates
We’ve released a string of new posts on our recent activities and strategy:

2018 Update: Our New Research Directions discusses the new set of research directions we’ve been ramping up over the last two years, how they relate to our Agent Foundations research and our goal of “deconfusion,” and why we’ve adopted a “nondisclosed-by-default” policy for this research.

Embedded Agency describes our Agent Foundations research agenda as different angles of attack on a single central difficulty: we don’t know how to characterize good reasoning and decision-making for agents embedded in their environment.

Summer MIRI Updates discusses new hires, new donations and grants, and new programs we’ve been running to recruit researcher staff and grow the total pool of AI alignment researchers.

And, added Nov. 28:

MIRI’s Newest Recruit: Edward Kmett announces our latest hire: noted Haskell developer Edward Kmett, who popularized the use of lenses for functional programming and maintains a large number of the libraries around the Haskell core libraries.

2017 in Review recaps our activities and donors’ support from last year.

Our 2018 Update also discusses the much wider pool of engineers and computer scientists we’re now trying to recruit, and the much larger total number of people we’re trying to add to the team in the near future:
We’re seeking anyone who can cause our “become less confused about AI alignment” work to go faster.
In practice, this means: people who natively think in math or code, who take seriously the problem of becoming less confused about AI alignment (quickly!), and who are generally capable. In particular, we’re looking for high-end Google programmer levels of capability; you don’t need a 1-in-a-million test score or a halo of destiny. You also don’t need a PhD, explicit ML background, or even prior research experience.
If the above might be you, and the idea of working at MIRI appeals to you, I suggest sending in a job application or shooting your questions at Buck Shlegeris, a researcher at MIRI who’s been helping with our recruiting.
We’re also hiring for Agent Foundations roles, though at a much slower pace. For those roles, we recommend interacting with us and other people hammering on AI alignment problems on Less Wrong and the AI Alignment Forum, or at local MIRIx groups. We then generally hire Agent Foundations researchers from people we’ve gotten to know through the above channels, visits, and events like the AI Summer Fellows program.
A great place to start developing intuitions for these problems is Scott Garrabrant’s recently released fixed point exercises, or various resources on the MIRI Research Guide. Some examples of recent public work on Agent Foundations / embedded agency include:

Sam Eisenstat’s untrollable prior, explained in illustrated form by Abram Demski, shows that there is a Bayesian solution to one of the basic problems which motivated the development of non-Bayesian logical uncertainty tools (culminating in logical induction). This informs our picture of what’s possible, and may lead to further progress in the direction of Bayesian logical uncertainty.
Scott Garrabrant outlines a taxonomy of ways that Goodhart’s law can manifest.
Sam’s logical inductor tiling result solves a version of the tiling problem for logically uncertain agents.
Prisoners’ Dilemma with Costs to Modeling: A modified version of open-source prisoners’ dilemmas in which agents must pay resources in order to model each other.
Logical Inductors Converge to Correlated Equilibria (Kinda): A game-theoretic analysis of logical inductors.
New results in Asymptotic Decision Theory and When EDT=CDT, ADT Does Well represent incremental progress on understanding what’s possible with respect to learning the right counterfactuals.

These results are relatively small, compared to Nate’s forthcoming tiling agents paper or Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant’s forthcoming “The Inner Alignment Problem.” However, they should give a good sense of some of the recent directions we’ve been pushing in with our Agent Foundations work.

2. Room for more funding
As noted above, the biggest sources of uncertainty in our 2018 budget estimates are about how many research staff we hire, and how much we spend on moving to new offices.
In our 2017 fundraiser, we set a goal of hiring 10 new research staff in 2018–2019. So far, we’re up two research staff, with enough promising candidates in the pipeline that I still consider 10 a doable (though ambitious) goal.
Following the amazing show of support we received from donors last year (and continuing into 2018), we had significantly more funds than we anticipated, and we found more ways to usefully spend it than we expected. In particular, we’ve been able to translate the “bonus” support we received in 2017 into broadening the scope of our recruiting efforts. As a consequence, our 2018 spending, which will come in at around $3.5M, actually matches the point estimate I gave in 2017 for our 2019 budget, rather than my prediction for 2018—a large step up from what I predicted, and an even larger step from last year’s budget of $2.1M.5
Our two fundraiser goals, Target 1 ($500k) and Target 2 ($1.2M), correspond to the budgetary needs we can easily predict and account for. Our 2018 went much better as a result of donors’ greatly increased support in 2017, and it’s possible that we’re in a similar situation today, though I’m not confident that this is the case.
Concretely, some ways that our decision-making changed as a result of the amazing support we saw were:

We spent more on running all-expenses-paid AI Risk for Computer Scientists workshops. We ran the first such workshop in February, and saw a lot of value in it as a venue for people with relevant technical experience to start reasoning more about AI risk. Since then, we’ve run another seven events in 2018, with more planned for 2019.
As hoped, these workshops have also generated interest in joining MIRI and other AI safety research teams. This has resulted in one full-time MIRI research staff hire, and on the order of ten candidates with good prospects of joining full-time in 2019, including two recent interns.

We’ve been more consistently willing and able to pay competitive salaries for top technical talent. A special case of this is hiring relatively senior research staff like Edward Kmett.

We raised salaries for some existing staff members. We have a very committed staff, and some staff at MIRI had previously asked for fairly low salaries in order to help keep MIRI’s organizational costs lower. The inconvenience this caused staff members doesn’t make much sense at our current organizational size, both in terms of our team’s productivity and in terms of their well-being.

We ran a summer research internship program, on a larger scale than we otherwise would have.

As we’ve considered options for new office space that can accommodate our expansion, we’ve been able to filter less on price relative to fit. We’ve also been able to spend more on renovations that we expect to produce a working environment where our researchers can do their work with fewer distractions or inconveniences.

2018 brought a positive update about our ability to cost-effectively convert surprise funding increases into (what look from our perspective like) very high-value actions, and the above list hopefully helps clarify what that can look like in practice. We can’t promise to be able to repeat that in 2019, given an overshoot in this fundraiser, but we have reason for optimism.

Donate Now

That is, our business-as-usual model tries to remove one-time outlier costs so that it’s easier to see what “the new normal” is in MIRI’s spending and think about our long-term growth curve.This estimation is fairly rough and uncertain. One weakness of this model is that it treats the inputs as though they were independent, which is not always the case. I also didn’t try to account for the fact that we’re likely to spend more in worlds where we see more fundraising success.
However, a sensitivity analysis on the final output showed that the overwhelming majority of the uncertainty in this estimate comes from how many research staff we hire, which matches my expectations and suggests that the model is doing a decent job of tracking the intuitively important variables. I also ended up with similar targets when I ran the numbers on our funding status in other ways and when I considered different funding scenarios.This excludes earmarked funding for AI Impacts, an independent research group that’s institutionally housed at MIRI.We could also think in terms of a “Target 0.5” of $100k in order to hit the bottom of the range, $4.4M. However, I worried that a $100k target would be misleading given that we’re thinking in terms of a $4.4–5.5M budget.Quoting our 2017 fundraiser post: “If we succeed, our point estimate is that our 2018 budget will be $2.8M and our 2019 budget will be $3.5M, up from roughly $1.9M in 2017.” The $1.9M figure was an estimate from before 2017 had ended. We’ve now revised this figure to $2.1M, which happens to bring it in line with our 2016 point estimate for how much we’d spend in 2017.The post MIRI’s 2018 Fundraiser appeared first on Machine Intelligence Research Institute.

MIRI’s 2018 Fundraiser