2017 Updates and Strategy

In our last strategy update (August 2016), Nate wrote that MIRI's priorities were to make progress on our agent foundations agenda and begin work on our new "Alignment for Advanced Machine Learning Systems" agenda, to collaborate and communicate with other researchers, and to grow our research and ops teams.
Since then, senior staff at MIRI have reassessed their views on how far off artificial general intelligence (AGI) is and concluded that shorter timelines are more likely than they were previously thinking. A few lines of recent evidence point in this direction, such as:1

AI research is becoming more visibly exciting and well-funded. This suggests that more top talent (in the next generation as well as the current generation) will probably turn their attention to AI.
AGI is attracting more scholarly attention as an idea, and is the stated goal of top AI groups like DeepMind, OpenAI, and FAIR. In particular, many researchers seem more open to thinking about general intelligence now than they did a few years ago.
Research groups associated with AGI are showing much clearer external signs of profitability.
AI successes like AlphaGo indicate that it's easier to outperform top humans in domains like Go (without any new conceptual breakthroughs) than might have been expected.2 This lowers our estimate for the number of significant conceptual breakthroughs needed to rival humans in other domains.

There's no consensus among MIRI researchers on how long timelines are, and our aggregated estimate puts medium-to-high probability on scenarios in which the research community hasn't developed AGI by, e.g., 2035. On average, however, research staff now assign moderately higher probability to AGI's being developed before 2035 than we did a year or two ago. This has a few implications for our strategy:
1. Our relationships with current key players in AGI safety and capabilities play a larger role in our strategic thinking. Short-timeline scenarios reduce the expected number of important new players who will enter the space before we hit AGI, and increase how much influence current players are likely to have.
2. Our research priorities are somewhat different, since shorter timelines change what research paths are likely to pay out before we hit AGI, and also concentrate our probability mass more on scenarios where AGI shares various features in common with present-day machine learning systems.
Both updates represent directions we've already been trending in for various reasons.3 However, we're moving in these two directions more quickly and confidently than we were last year. As an example, Nate is spending less time on staff management and other administrative duties than in the past (having handed these off to MIRI COO Malo Bourgon) and less time on broad communications work (having delegated a fair amount of this to me), allowing him to spend more time on object-level research, research prioritization work, and more targeted communications.4
I'll lay out what these updates mean for our plans in more concrete detail below.

1. Research program plans
Our top organizational priority is object-level research on the AI alignment problem, following up on the work Malo described in our recent annual review.
We plan to spend this year delving into some new safety research directions that are very preliminary and exploratory, where we're uncertain about potential synergies with AGI capabilities research. Work related to this exploratory investigation will be non-public-facing at least through late 2017, in order to lower the risk of marginally shortening AGI timelines (which can leave less total time for alignment research) and to free up researchers' attention from having to think through safety tradeoffs for each new result.5
We've worked on non-public-facing research before, but this will be a larger focus in 2017. We plan to re-assess how much work to put into our exploratory research program (and whether to shift projects to the public-facing side) in the fall, based on how projects are progressing.
On the public-facing side, Nate made a prediction that we'll make roughly the following amount of research progress this year (noting 2015 and 2016 estimates for comparison). 1 means "limited progress", 2 "weak-to-modest progress", 3 "modest progress", 4 "modest-to-strong progress", and 5 "sizable progress":6

logical uncertainty and naturalized induction:

2015 progress: 5. — Predicted: 3.
2016 progress: 5. — Predicted: 5.
2017 progress prediction: 2 (weak-to-modest).

decision theory:

2015 progress: 3. — Predicted: 3.
2016 progress: 3. — Predicted: 3.
2017 progress prediction: 3 (modest).

Vingean reflection:

2015 progress: 3. — Predicted: 3.
2016 progress: 4. — Predicted: 1.
2017 progress prediction: 1 (limited).

error tolerance:

2015 progress: 1. — Predicted: 3.
2016 progress: 1. — Predicted: 3.
2017 progress prediction: 1 (limited).

value specification:

2015 progress: 1. — Predicted: 1.
2016 progress: 2. — Predicted: 3.
2017 progress prediction: 1 (limited).

Nate expects fewer novel public-facing results this year than in 2015-2016, based on a mix of how many researcher hours we're investing into each area and how easy he estimates it is to make progress in that area.
Progress in basic research is difficult to predict in advance, and the above estimates combine how likely it is that we'll come up with important new results with how large we would expect such results to be in the relevant domain.  In the case of naturalized induction, most of the probability is on us making small amounts of progress this year, with a low chance of new large insights. In the case of decision theory, most of the probability is on us achieving some minor new insights related to the questions we're working on, with a medium-low chance of large insights.
The research team's current focus is on some quite new questions. Jessica, Sam, and Scott have recently been working on the problem of reasoning procedures like Solomonoff induction giving rise to misaligned subagents (e.g., here), and considering alternative induction methods that might avoid this problem.7
In decision theory, a common thread in our recent work is that we're using probability and topological fixed points in settings where we used to use provability. This means working with (and improving) logical inductors and reflective oracles. It also means developing new ways of looking at counterfactuals inspired by those methods. The reason behind this shift is that most of the progress we've seen on Vingean reflection has come out of these probabilistic reasoning and fixed-point-based techniques.
We also plan to put out more accessible overviews this year of some of our research areas. For a good general introduction to our work in decision theory, see our newest paper, "Cheating Death in Damascus."

2. Targeted outreach and closer collaborations
Our outreach efforts this year are mainly aimed at exchanging research-informing background models with top AI groups (especially OpenAI and DeepMind), AI safety research groups (especially the Future of Humanity Institute), and funders / conveners (especially the Open Philanthropy Project).
We're currently collaborating on a research project with DeepMind, and are on good terms with OpenAI and key figures at other groups. We're also writing up a more systematic explanation of our view of the strategic landscape, which we hope to use as a starting point for discussion. Topics we plan to go into in forthcoming write-ups include:
1. Practical goals and guidelines for AGI projects.
2. Why we consider AGI alignment a difficult problem, of the sort where a major multi-year investment of research effort in the near future may be necessary (and not too far off from sufficient).
3. Why we think a deep understanding of how AI systems' cognition achieves objectives is likely to be critical for AGI alignment.
4. Task-directed AGI and methods for limiting the scope of AGI systems' problem-solving work.
Some existing write-ups related to the topics we intend to say more about include Jessica Taylor's "On Motivations for MIRI's Highly Reliable Agent Design Research," Nate Soares' "Why AI Safety?", and Daniel Dewey's "Long-Term Strategies for Ending Existential Risk from Fast Takeoff."

3. Expansion
Our planned budget in 2017 is $2.1–2.5M, up from $1.65M in 2015 and $1.75M in 2016. Our point estimate is $2.25M, in which case we would expect our breakdown to look roughly like this:

We recently hired two new research fellows, Sam Eisenstat and Marcello Herreshoff, and have other researchers in the pipeline. We're also hiring software engineers to help us rapidly prototype, implement, and test AI safety ideas related to machine learning. We're currently seeking interns to trial for these programming roles (apply here).
Our events budget is smaller this year, as we're running more internal research retreats and fewer events like our 2015 summer workshop series and our 2016 colloquium series. Our costs of doing business are higher, due in part to accounting expenses associated with our passing the $2M revenue level and bookkeeping expenses for upkeep tasks we've outsourced.
We experimented with running just one fundraiser in 2016, but ended up still needing to spend staff time on fundraising at the end of the year after falling short of our initial funding target. Taking into account a heartening end-of-the-year show of support, our overall performance was very solid — $2.29M for the year, up from $1.58M in 2015. However, there's a good chance we'll return to our previous two-fundraiser rhythm this year in order to more confidently move ahead with our growth plans.
Our 5-year plans are fairly uncertain, as our strategy will plausibly end up varying based on how fruitful our research directions this year turn out to be, and based on our conversations with other groups. As usual, you're welcome to ask us questions if you're curious about what we're up to, and we'll be keeping you updated as our plans continue to develop!

Note that this list is far from exhaustive.Relatively general algorithms (plus copious compute) were able to surpass human performance on Go, going from incapable of winning against the worst human professionals in standard play to dominating the very best professionals in the space of a few months. The relevant development here wasn't "AlphaGo represents a large conceptual advance over previously known techniques," but rather "contemporary techniques run into surprisingly few obstacles when scaled to tasks as pattern-recognition-reliant and difficult (for humans) as professional Go".The publication of "Concrete Problems in AI Safety" last year, for example, caused us to reduce the time we were spending on broad-based outreach to the AI community at large in favor of spending more time building stronger collaborations with researchers we knew at OpenAI, Google Brain, DeepMind, and elsewhere.Nate continues to set MIRI's organizational strategy, and is responsible for the ideas in this post.We generally support a norm where research groups weigh the costs and benefits of publishing results that could shorten AGI timelines, and err on the side of keeping potentially AGI-hastening results proprietary where there's sufficient uncertainty, unless there are sufficiently strong positive reasons to disseminate the results under consideration. This can end up applying to safety research and work by smaller groups as well, depending on the specifics of the research itself.
Another factor in our decision is that writing up results for external consumption takes additional researcher time and attention, though in practice this cost will often be smaller than the benefits of the writing process and resultant papers.Nate originally recorded his predictions on March 21, based on the progress he expected in late March through the end of 2017. Note that, for example, three "limited" scores aren't equivalent to one "modest" score. Additionally, the ranking is based on the largest technical result we expect in each category, and emphasizes depth over breadth: if we get one modest-seeming decision theory result one year and ten such results the next year, those will both get listed as "modest progress".This is a relatively recent research priority, and doesn't fit particularly well into any of the bins from our agent foundations agenda, though it is most clearly related to naturalized induction. Our AAMLS agenda also doesn't fit particularly neatly into these bins, though we classify most AAMLS research as error-tolerance or value specification work.The post 2017 Updates and Strategy appeared first on Machine Intelligence Research Institute.

2017 Updates and Strategy


Software Engineer Internship / Staff Openings

The Machine Intelligence Research Institute is looking for highly capable software engineers to directly support our AI alignment research efforts, with a focus on projects related to machine learning. We're seeking engineers with strong programming skills who are passionate about MIRI's mission and looking for challenging and intellectually engaging work.
While our goal is to hire full-time, we are initially looking for paid interns. Successful internships may then transition into staff positions.
About the Internship Program
The start time for interns is flexible, but we're aiming for May or June. We will likely run several batches of internships, so if you are interested but unable to start in the next few months, do still apply. The length of the internship is flexible, but we're aiming for 2–3 months.
Examples of the kinds of work you'll do during the internship:

Replicate recent machine learning papers, and implement variations.
Learn about and implement machine learning tools (including results in the fields of deep learning, convex optimization, etc.).
Run various coding experiments and projects, either independently or in small groups.
Rapidly prototype, implement, and test AI alignment ideas related to machine learning (after demonstrating successes in the above points).

For MIRI, the benefit of this program is that it's a great way to get to know you and assess you for a potential hire. For applicants, the benefits are that this is an excellent opportunity to get your hands dirty and level up your machine learning skills, and to get to the cutting edge of the AI safety field, with a potential to stay in a full-time engineering role after the internship concludes.
Our goal is to trial many more people than we expect to hire, so our threshold for keeping on engineers long-term as full staff will be higher than for accepting applicants to our internship.
The Ideal Candidate
Some qualities of the ideal candidate:

Extensive breadth and depth of programming skills. Machine learning experience is not required, though it is a plus.
Highly familiar with basic ideas related to AI alignment.
Able to work independently with minimal supervision, and in team/group settings.
Willing to accept a below-market rate. Since MIRI is a non-profit, we can't compete with the Big Names in the Bay Area.
Enthusiastic about the prospect of working at MIRI and helping advance the field of AI alignment.
Not looking for a "generic" software engineering position.

Working at MIRI
We strive to make working at MIRI a rewarding experience.

Modern Work Spaces — Many of us have adjustable standing desks with large external monitors. We consider workspace ergonomics important, and try to rig up work stations to be as comfortable as possible. Free snacks, drinks, and meals are also provided at our office.
Flexible Hours — We don't have strict office hours, and we don't limit employees' vacation days. Our goal is to make rapid progress on our research agenda, and we would prefer that staff take a day off than that they extend tasks to fill an extra day.
Living in the Bay Area — MIRI's office is located in downtown Berkeley, California. From our office, you're a 30-second walk to the BART (Bay Area Rapid Transit), which can get you around the Bay Area; a 3-minute walk to UC Berkeley campus; and a 30-minute BART ride to downtown San Francisco.

EEO & Employment Eligibility
MIRI is an equal opportunity employer. We are committed to making employment decisions based on merit and value. This commitment includes complying with all federal, state, and local laws. We desire to maintain a work environment free of harassment or discrimination due to sex, race, religion, color, creed, national origin, sexual orientation, citizenship, physical or mental disability, marital status, familial status, ethnicity, ancestry, status as a victim of domestic violence, age, or any other status protected by federal, state, or local laws.
Apply
If interested, click here to apply. For questions or comments, email <EMAIL>.
Update (December 2017): We're now putting less emphasis on finding interns and looking for highly skilled engineers available for full-time work. Updated job post here.
The post Software Engineer Internship / Staff Openings appeared first on Machine Intelligence Research Institute.


Ensuring smarter-than-human intelligence has a positive outcome

I recently gave a talk at Google on the problem of aligning smarter-than-human AI with operators' goals:

The talk was inspired by "AI Alignment: Why It's Hard, and Where to Start," and serves as an introduction to the subfield of alignment research in AI. A modified transcript follows.
Talk outline (slides):

1. Overview
2. Simple bright ideas going wrong
2.1. Task: Fill a cauldron
2.2. Subproblem: Suspend buttons
3. The big picture
3.1. Alignment priorities
3.2. Four key propositions
4. Fundamental difficulties

Overview
I'm the executive director of the Machine Intelligence Research Institute. Very roughly speaking, we're a group that's thinking in the long term about artificial intelligence and working to make sure that by the time we have advanced AI systems, we also know how to point them in useful directions.
Across history, science and technology have been the largest drivers of change in human and animal welfare, for better and for worse. If we can automate scientific and technological innovation, that has the potential to change the world on a scale not seen since the Industrial Revolution. When I talk about "advanced AI," it's this potential for automating innovation that I have in mind.
AI systems that exceed humans in this capacity aren't coming next year, but many smart people are working on it, and I'm not one to bet against human ingenuity. I think it's likely that we'll be able to build something like an automated scientist in our lifetimes, which suggests that this is something we need to take seriously.
When people talk about the social implications of general AI, they often fall prey to anthropomorphism. They conflate artificial intelligence with artificial consciousness, or assume that if AI systems are "intelligent," they must be intelligent in the same way a human is intelligent. A lot of journalists express a concern that when AI systems pass a certain capability level, they'll spontaneously develop "natural" desires like a human hunger for power; or they'll reflect on their programmed goals, find them foolish, and "rebel," refusing to obey their programmed instructions.
These are misplaced concerns. The human brain is a complicated product of natural selection. We shouldn't expect machines that exceed human performance in scientific innovation to closely resemble humans, any more than early rockets, airplanes, or hot air balloons closely resembled birds.1
The notion of AI systems "breaking free" of the shackles of their source code or spontaneously developing human-like desires is just confused. The AI system is its source code, and its actions will only ever follow from the execution of the instructions that we initiate. The CPU just keeps on executing the next instruction in the program register. We could write a program that manipulates its own code, including coded objectives. Even then, though, the manipulations that it makes are made as a result of executing the original code that we wrote; they do not stem from some kind of ghost in the machine.
The serious question with smarter-than-human AI is how we can ensure that the objectives we've specified are correct, and how we can minimize costly accidents and unintended consequences in cases of misspecification. As Stuart Russell (co-author of Artificial Intelligence: A Modern Approach) puts it:
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.
These kinds of concerns deserve a lot more attention than the more anthropomorphic risks that are generally depicted in Hollywood blockbusters.

Simple bright ideas going wrong
Task: Fill a cauldron
Many people, when they start talking about concerns with smarter-than-human AI, will throw up a picture of the Terminator. I was once quoted in a news article making fun of people who put up Terminator pictures in all their articles about AI, next to a Terminator picture. I learned something about the media that day.
I think this is a much better picture:

This is Mickey Mouse in the movie Fantasia, who has very cleverly enchanted a broom to fill a cauldron on his behalf.
How might Mickey do this? We can imagine that Mickey writes a computer program and has the broom execute the program. Mickey starts by writing down a scoring function or objective function:
$$\mathcal{U}_{broom} =
\begin{cases}
1 &\text{ if cauldron full} \\
0 &\text{ if cauldron empty}
\end{cases}$$Given some set 𝐴 of available actions, Mickey then writes a program that can take one of these actions 𝑎 as input and calculate how high the score is expected to be if the broom takes that action. Then Mickey can write a function that spends some time looking through actions and predicting which ones lead to high scores, and outputs an action that leads to a relatively high score:
$$\underset{a\in A}{\mathrm{sorta\mbox{-}argmax}} \ \mathbb{E}\left[\mathcal{U}_{broom}\mid a\right]$$The reason this is "sorta-argmax" is that there may not be time to evaluate every action in 𝐴. For realistic action sets, agents should only need to find actions that make the scoring function as large as they can given resource constraints, even if this isn't the maximal action.
This program may look simple, but of course, the devil's in the details: writing an algorithm that does accurate prediction and smart search through action space is basically the whole problem of AI. Conceptually, however, it's pretty simple: We can describe in broad strokes the kinds of operations the broom must carry out, and their plausible consequences at different performance levels.
When Mickey runs this program, everything goes smoothly at first. Then:

I claim that as fictional depictions of AI go, this is pretty realistic.
Why would we expect a generally intelligent system executing the above program to start overflowing the cauldron, or otherwise to go to extreme lengths to ensure the cauldron is full?
The first difficulty is that the objective function that Mickey gave his broom left out a bunch of other terms Mickey cares about:
$$\mathcal{U}_{human} =
\begin{cases}
1&\text{ if cauldron full} \\
0&\text{ if cauldron empty} \\
-10&\text{ if workshop flooded} \\
+0.2&\text{ if it's funny} \\
-<PHONE>&\text{ if someone gets killed} \\
&\text{… and a whole lot more} \\
\end{cases}$$
The second difficulty is that Mickey programmed the broom to make the expectation of its score as large as it could. "Just fill one cauldron with water" looks like a modest, limited-scope goal, but when we translate this goal into a probabilistic context, we find that optimizing it means driving up the probability of success to absurd heights. If the broom assigns a 99.9% probability to "the cauldron is full," and it has extra resources lying around, then it will always try to find ways to use those resources to drive the probability even a little bit higher.
Contrast this with the limited "task-like" goal we presumably had in mind. We wanted the cauldron full, but in some intuitive sense we wanted the system to "not try too hard" even if it has lots of available cognitive and physical resources to devote to the problem. We wanted it to exercise creativity and resourcefulness within some intuitive limits, but we didn't want it to pursue "absurd" strategies, especially ones with large unanticipated consequences.2
In this example, the original objective function looked pretty task-like. It was bounded and quite simple. There was no way to get ever-larger amounts of utility. It's not like the system got one point for every bucket of water it poured in — then there would clearly be an incentive to overfill the cauldron. The problem was hidden in the fact that we're maximizing expected utility. This makes the goal open-ended, meaning that even small errors in the system's objective function will blow up.
There are a number of different ways that a goal that looks task-like can turn out to be open-ended. Another example: a larger system that has an overarching task-like goal may have subprocesses that are themselves trying to maximize a variety of different objective functions, such as optimizing the system's memory usage. If you don't understand your system well enough to track whether any of its subprocesses are themselves acting like resourceful open-ended optimizers, then it may not matter how safe the top-level objective is.
So the broom keeps grabbing more pails of water — say, on the off chance that the cauldron has a leak in it, or that "fullness" requires the water to be slightly above the level of the brim. And, of course, at no point does the broom "rebel against" Mickey's code. If anything, the broom pursued the objectives it was programmed with too effectively.

Subproblem: Suspend buttons
A common response to this problem is: "OK, there may be some unintended consequences of the objective function, but we can always pull the plug, right?"
Mickey tries this, and it doesn't work:

And I claim that this is realistic too, for systems that are sufficiently good at modeling their environment. If the system is trying to drive up the expectation of its scoring function and is smart enough to recognize that its being shut down will result in lower-scoring outcomes, then the system's incentive is to subvert shutdown attempts. The more capable the system is, the likelier it is to find creative ways to achieve that subgoal — e.g., by copying itself to the Internet, or by tricking the programmers into thinking it's safer.
This isn't to say that it's impossible to shut down sufficiently capable AI systems; it's just that we need to put in the extra work of deliberately designing systems to avoid searching for ways to avoid being shut down. If you've written code that would try to resist attempts to shut it down, then you shouldn't run the code even if you're convinced it would fail — there are better ways to spend those compute cycles than on adversarial policy searches.
Relevant safety measures that don't assume we can always outthink and outmaneuver the system will need to modify the system's objective function in some relevant way. We need to figure out how to formally specify objective functions that don't automatically place the AI system into an adversarial context with the operators; or we need to figure out some way to have the system achieve goals without optimizing some objective function in the traditional sense. E.g., we can look for ways to limit the system's capabilities to a small set of "whitelisted" domains that don't include thinking about whether the system will be shut down.
One obvious solution is to try changing the score function to make reference to a suspend button 𝐁:
$$\mathcal{U}^\prime_{broom} =
\begin{cases}
1 \text{ if cauldron full } & \text{& } \textbf{B}\text{=OFF} \\
0 \text{ if cauldron empty } & \text{& } \textbf{B}\text{=OFF} \\
1 \text{ if robot suspended } & \text{& } \textbf{B}\text{=ON} \\
0 \text{ otherwise}
\end{cases}$$This says that when the button is on, the system's goal changes from "fill the cauldron" to "be suspended."
It's not clear how we could actually specify this goal — if there are any loopholes in the way we've specified "suspension," then the system might try to exploit them to hedge its bets. E.g., it could fulfill the technical definition of "suspend operations" while tasking a child agent it built or subprocess it started with filling the cauldron. This would allow is to achieve both "cauldron full" and "robot suspended" at once, just in case it's wrong about whether the button has been pushed.
Let's assume, however, that we've found an airtight way to technically specify suspension. The problem we run into now is that we've given the system an incentive to care a lot about whether the button is pushed. If it has any way to influence its operators' decision to push the button, or any other way to exercise more control, then it will jump on the opportunity and try to ensure the button is pushed, or ensure that it's not pushed — depending on whether it looks easier to suspend or to fill the cauldron.
And, surprisingly, the state of the art on this problem isn't much more advanced than that. There's a paper co-authored by me, Fallenstein, Yudkowsky, and Armstrong ("Corrigibility"), and a related result by Orseau and Armstrong ("Safely Interruptible Agents"), but not a lot more.
What we want is a way to combine two objective functions — a default function for normal operation, and a suspend function for when we want to suspend the system to disk.$$\mathcal{U}_{normal}(act) =
\begin{cases}
1\text{ if cauldron full} \\
0\text{ if cauldron empty}
\end{cases}$$
$$\mathcal{U}_{suspend}(act) =
\begin{cases}
1\text{ if suspended} \hspace{0.8cm}\\
0\text{ otherwise}
\end{cases}$$We want our method for combining the functions to satisfy three conditions: an operator should be able to switch between the functions (say, by pushing a button); the system shouldn't have any incentives to control which function is active; and if it's plausible that the system's normal operations could inadvertently compromise our ability to switch between the functions, then the system should be incentivized to keep that from happening.
So far, we haven't found any way to achieve all three goals at once. As an example, we can try to remove any incentive on the system's part to control whether its suspend button is pushed by giving the system a switching objective function that always assigns the same expected utility to the button being on or off:
$$\mathcal{U}_{switch}(act) =
\begin{cases}
\mathcal{U}_{normal}(act) &\text{ if button=OFF} \\
\mathcal{U}_{suspend}(act) + \theta &\text{ if button=ON}
\end{cases}$$$$\theta = \max_{act} \mathbb{E}\left[\mathcal{U}_{normal}\mid act\right] – \max_{act} \mathbb{E}\left[\mathcal{U}_{suspend}\mid act\right]$$Unfortunately, this approach makes it impossible to satisfy the third condition. It turns out that the above objective function is equivalent to setting the probability of the button being pressed to 0. The system won't deliberately tamper with the button, but it will happily approve updates that delete any reference to the button as dead code, or build subagents that ignore the button.
Meeting these intuitively simple constraints turns out to be a nontrivial problem. This is a pattern that runs through a lot of questions in this space: traditional tools and concepts run into immediate safety problems that don't turn up in conventional capabilities research.

The big picture
Alignment priorities
Let's take a step back and talk about what's needed overall in order to align highly capable AI systems with our interests.
Here's a dramatically simplified pipeline: You have some humans who come up with some task or goal or preference set that serves as their intended value function 𝘝. Since our values are complicated and context-sensitive, in practice we'll need to build systems to learn our values over time, rather than coding them by hand.3 We'll call the goal the AI system ends up with (which may or may not be identical to 𝘝) 𝗨.
When the press covers this topic, they often focus on one of two problems: "What if the wrong group of humans develops smarter-than-human AI first?", and "What if AI's natural desires cause 𝗨 to diverge from 𝘝?"
In my view, the "wrong humans" issue shouldn't be the thing we focus on until we have reason to think we could get good outcomes with the right group of humans. We're very much in a situation where well-intentioned people couldn't leverage a general AI system to do good things even if they tried. As a simple example, if you handed me a box that was an extraordinarily powerful function optimizer — I could put in a description of any mathematical function, and it would give me an input that makes the output extremely large — then I don't know how I could use that box to develop a new technology or advance a scientific frontier without causing any catastrophes.4
There's a lot we don't understand about AI capabilities, but we're in a position where we at least have a general sense of what progress looks like. We have a number of good frameworks, techniques, and metrics, and we've put a great deal of thought and effort into successfully chipping away at the problem from various angles. At the same time, we have a very weak grasp on the problem of how to align highly capable systems with any particular goal. We can list out some intuitive desiderata, but the field hasn't really developed its first formal frameworks, techniques, or metrics.
I believe that there's a lot of low-hanging fruit in this area, and also that a fair amount of the work does need to be done early (e.g., to help inform capabilities research directions — some directions may produce systems that are much easier to align than others). If we don't solve these problems, developers with arbitrarily good or bad intentions will end up producing equally bad outcomes. From an academic or scientific standpoint, our first objective in that kind of situation should be to remedy this state of affairs and at least make good outcomes technologically possible.
Many people quickly recognize that "natural desires" are a fiction, but infer from this that we instead need to focus on the other issues the media tends to emphasize — "What if bad actors get their hands on smarter-than-human AI?", "How will this kind of AI impact employment and the distribution of wealth?", etc. These are important questions, but they'll only end up actually being relevant if we figure out how to bring general AI systems up to a minimum level of reliability and safety.
Another common thread is "Why not just tell the AI system to (insert intuitive moral precept here)?" On this way of thinking about the problem, often (perhaps unfairly) associated with Isaac Asimov's writing, ensuring a positive impact from AI systems is largely about coming up with natural-language instructions that are vague enough to subsume a lot of human ethical reasoning:

In contrast, precision is a virtue in real-world safety-critical software systems. Driving down accident risk requires that we begin with limited-scope goals rather than trying to "solve" all of morality at the outset.5
My view is that the critical work is mostly in designing an effective value learning process, and in ensuring that the sorta-argmax process is correctly hooked up to the resultant objective function 𝗨:

The better your value learning framework is, the less explicit and precise you need to be in pinpointing your value function 𝘝, and the more you can offload the problem of figuring out what you want to the AI system itself. Value learning, however, raises a number of basic difficulties that don't crop up in ordinary machine learning tasks.
Classic capabilities research is concentrated in the sorta-argmax and Expectation parts of the diagram, but sorta-argmax also contains what I currently view as the most neglected, tractable, and important safety problems. The easiest way to see why "hooking up the value learning process correctly to the system's capabilities" is likely to be an important and difficult challenge in its own right is to consider the case of our own biological history.
Natural selection is the only "engineering" process we know of that has ever led to a generally intelligent artifact: the human brain. Since natural selection relies on a fairly unintelligent hill-climbing approach, one lesson we can take away from this is that it's possible to reach general intelligence with a hill-climbing approach and enough brute force — though we can presumably do better with our human creativity and foresight.
Another key take-away is that natural selection was maximally strict about only optimizing brains for a single very simple goal: genetic fitness. In spite of this, the internal objectives that humans represent as their goals are not genetic fitness. We have innumerable goals — love, justice, beauty, mercy, fun, esteem, good food, good health, … — that correlated with good survival and reproduction strategies in the ancestral savanna. However, we ended up valuing these correlates directly, rather than valuing propagation of our genes as an end in itself — as demonstrated every time we employ birth control.
This is a case where the external optimization pressure on an artifact resulted in a general intelligence with internal objectives that didn't match the external selection pressure. And just as this caused humans' actions to diverge from natural selection's pseudo-goal once we gained new capabilities, we can expect AI systems' actions to diverge from humans' if we treat their inner workings as black boxes.
If we apply gradient descent to a black box, trying to get it to be very good at maximizing some objective, then with enough ingenuity and patience, we may be able to produce a powerful optimization process of some kind.6 By default, we should expect an artifact like that to have a goal 𝗨 that strongly correlates with our objective 𝘝 in the training environment, but sharply diverges from 𝘝 in some new environments or when a much wider option set becomes available.
On my view, the most important part of the alignment problem is ensuring that the value learning framework and overall system design we implement allow us to crack open the hood and confirm when the internal targets the system is optimizing for match (or don't match) the targets we're externally selecting through the learning process.7
We expect this to be technically difficult, and if we can't get it right, then it doesn't matter who's standing closest to the AI system when it's developed. Good intentions aren't sneezed into computer programs by kind-hearted programmers, and coming up with plausible goals for advanced AI systems doesn't help if we can't align the system's cognitive labor with a given goal.

Four key propositions
Taking another step back: I've given some examples of open problems in this area (suspend buttons, value learning, limited task-based AI, etc.), and I've outlined what I consider to be the major problem categories. But my initial characterization of why I consider this an important area — "AI could automate general-purpose scientific reasoning, and general-purpose scientific reasoning is a big deal" — was fairly vague. What are the core reasons to prioritize this work?
First, goals and capabilities are orthogonal. That is, knowing an AI system's objective function doesn't tell you how good it is at optimizing that function, and knowing that something is a powerful optimizer doesn't tell you what it's optimizing.
I think most programmers intuitively understand this. Some people will insist that when a machine tasked with filling a cauldron gets smart enough, it will abandon cauldron-filling as a goal unworthy of its intelligence. From a computer science perspective, the obvious response is that you could go out of your way to build a system that exhibits that conditional behavior, but you could also build a system that doesn't exhibit that conditional behavior. It can just keeps searching for actions that have a higher score on the "fill a cauldron" metric. You and I might get bored if someone told us to just keep searching for better actions, but it's entirely possible to write a program that executes a search and never gets bored.8
Second, sufficiently optimized objectives tend to converge on adversarial instrumental strategies. Most objectives a smarter-than-human AI system could possess would be furthered by subgoals like "acquire resources" and "remain operational" (along with "learn more about the environment," etc.).
This was the problem suspend buttons ran into: even if you don't explicitly include "remain operational" in your goal specification, whatever goal you did load into the system is likely to be better achieved if the system remains online. Software systems' capabilities and (terminal) goals are orthogonal, but they'll often exhibit similar behaviors if a certain class of actions is useful for a wide variety of possible goals.
To use an example due to Stuart Russell: If you build a robot and program it to go to the supermarket to fetch some milk, and the robot's model says that one of the paths is much safer than the other, then the robot, in optimizing for the probability that it returns with milk, will automatically take the safer path. It's not that the system fears death, but that it can't fetch the milk if it's dead.
Third, general-purpose AI systems are likely to show large and rapid capability gains. The human brain isn't anywhere near the upper limits for hardware performance (or, one assumes, software performance), and there are a number of other reasons to expect large capability advantages and rapid capability gain from advanced AI systems.
As a simple example, Google can buy a promising AI startup and throw huge numbers of GPUs at them, resulting in a quick jump from "these problems look maybe relevant a decade from now" to "we need to solve all of these problems in the next year" à la DeepMind's progress in Go. Or performance may suddenly improve when a system is first given large-scale Internet access, when there's a conceptual breakthrough in algorithm design, or when the system itself is able to propose improvements to its hardware and software.9
Fourth, aligning advanced AI systems with our interests looks difficult. I'll say more about why I think this presently.
Roughly speaking, the first proposition says that AI systems won't naturally end up sharing our objectives. The second says that by default, systems with substantially different objectives are likely to end up adversarially competing for control of limited resources. The third suggests that adversarial general-purpose AI systems are likely to have a strong advantage over humans. And the fourth says that this problem is hard to solve — for example, that it's hard to transmit our values to AI systems (addressing orthogonality) or avert adversarial incentives (addressing convergent instrumental strategies).
These four propositions don't mean that we're screwed, but they mean that this problem is critically important. General-purpose AI has the potential to bring enormous benefits if we solve this problem, but we do need to make finding solutions a priority for the field.

Fundamental difficulties
Why do I think that AI alignment looks fairly difficult? The main reason is just that this has been my experience from actually working on these problems. I encourage you to look at some of the problems yourself and try to solve them in toy settings; we could use more eyes here. I'll also make note of a few structural reasons to expect these problems to be hard:
First, aligning advanced AI systems with our interests looks difficult for the same reason rocket engineering is more difficult than airplane engineering.
Before looking at the details, it's natural to think "it's all just AI" and assume that the kinds of safety work relevant to current systems are the same as the kinds you need when systems surpass human performance. On that view, it's not obvious that we should work on these issues now, given that they might all be worked out in the course of narrow AI research (e.g., making sure that self-driving cars don't crash).
Similarly, at a glance someone might say, "Why would rocket engineering be fundamentally harder than airplane engineering? It's all just material science and aerodynamics in the end, isn't it?" In spite of this, empirically, the proportion of rockets that explode is far higher than the proportion of airplanes that crash. The reason for this is that a rocket is put under much greater stress and pressure than an airplane, and small failures are much more likely to be highly destructive.10
Analogously, even though general AI and narrow AI are "just AI" in some sense, we can expect that the more general AI systems are likely to experience a wider range of stressors, and possess more dangerous failure modes.
For example, once an AI system begins modeling the fact that (i) your actions affect its ability to achieve its objectives, (ii) your actions depend on your model of the world, and (iii) your model of the world is affected by its actions, the degree to which minor inaccuracies can lead to harmful behavior increases, and the potential harmfulness of its behavior (which can now include, e.g., deception) also increases. In the case of AI, as with rockets, greater capability makes it easier for small defects to cause big problems.
Second, alignment looks difficult for the same reason it's harder to build a good space probe than to write a good app.
You can find a number of interesting engineering practices at NASA. They do things like take three independent teams, give each of them the same engineering spec, and tell them to design the same software system; and then they choose between implementations by majority vote. The system that they actually deploy consults all three systems when making a choice, and if the three systems disagree, the choice is made by majority vote. The idea is that any one implementation will have bugs, but it's unlikely all three implementations will have a bug in the same place.
This is significantly more caution than goes into the deployment of, say, the new WhatsApp. One big reason for the difference is that it's hard to roll back a space probe. You can send version updates to a space probe and correct software bugs, but only if the probe's antenna and receiver work, and if all the code required to apply the patch is working. If your system for applying patches is itself failing, then there's nothing to be done.
In that respect, smarter-than-human AI is more like a space probe than like an ordinary software project. If you're trying to build something smarter than yourself, there are parts of the system that have to work perfectly on the first real deployment. We can do all the test runs we want, but once the system is out there, we can only make online improvements if the code that makes the system allow those improvements is working correctly.
If nothing yet has struck fear into your heart, I suggest meditating on the fact that the future of our civilization may well depend on our ability to write code that works correctly on the first deploy.
Lastly, alignment looks difficult for the same reason computer security is difficult: systems need to be robust to intelligent searches for loopholes.
Suppose you have a dozen different vulnerabilities in your code, none of which is itself fatal or even really problematic in ordinary settings. Security is difficult because you need to account for intelligent attackers who might find all twelve vulnerabilities and chain them together in a novel way to break into (or just break) your system. Failure modes that would never arise by accident can be sought out and exploited; weird and extreme contexts can be instantiated by an attacker to cause your code to follow some crazy code path that you never considered.
A similar sort of problem arises with AI. The problem I'm highlighting here is not that AI systems might act adversarially: AI alignment as a research program is all about finding ways to prevent adversarial behavior before it can crop up. We don't want to be in the business of trying to outsmart arbitrarily intelligent adversaries. That's a losing game.
The parallel to cryptography is that in AI alignment we deal with systems that perform intelligent searches through a very large search space, and which can produce weird contexts that force the code down unexpected paths. This is because the weird edge cases are places of extremes, and places of extremes are often the place where a given objective function is optimized.11 Like computer security professionals, AI alignment researchers need to be very good at thinking about edge cases.
It's much easier to make code that works well on the path that you were visualizing than to make code that works on all the paths that you weren't visualizing. AI alignment needs to work on all the paths you weren't visualizing.
Summing up, we should approach a problem like this with the same level of rigor and caution we'd use for a security-critical rocket-launched space probe, and do the legwork as early as possible. At this early stage, a key part of the work is just to formalize basic concepts and ideas so that others can critique them and build on them. It's one thing to have a philosophical debate about what kinds of suspend buttons people intuit ought to work, and another thing to translate your intuition into an equation so that others can fully evaluate your reasoning.
This is a crucial project, and I encourage all of you who are interested in these problems to get involved and try your hand at them. There are ample resources online for learning more about the open technical problems. Some good places to start include MIRI's research agendas and a great paper from researchers at Google Brain, OpenAI, and Stanford called "Concrete Problems in AI Safety."

An airplane can't heal its injuries or reproduce, though it can carry heavy cargo quite a bit further and faster than a bird. Airplanes are simpler than birds in many respects, while also being significantly more capable in terms of carrying capacity and speed (for which they were designed). It's plausible that early automated scientists will likewise be simpler than the human mind in many respects, while being significantly more capable in certain key dimensions. And just as the construction and design principles of aircraft look alien relative to the architecture of biological creatures, we should expect the design of highly capable AI systems to be quite alien when compared to the architecture of the human mind.Trying to give some formal content to these attempts to differentiate task-like goals from open-ended goals is one way of generating open research problems. In the "Alignment for Advanced Machine Learning Systems" research proposal, the problem of formalizing "don't try too hard" is mild optimization, "steer clear of absurd strategies" is conservatism, and "don't have large unanticipated consequences" is impact measures. See also "avoiding negative side effects" in Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané's "Concrete Problems in AI Safety."One thing we've learned in the field of machine vision over the last few decades is that it's hopeless to specify by hand what a cat looks like, but that it's not too hard to specify a learning system that can learn to recognize cats. It's even more hopeless to specify everything we value by hand, but it's plausible that we could specify a learning system that can learn the relevant concept of "value."See "Environmental Goals," "Low-Impact Agents," and "Mild Optimization" for examples of obstacles to specifying physical goals without causing catastrophic side-effects.
Roughly speaking, MIRI's focus is on research directions that seem likely to help us conceptually understand how to do AI alignment in principle, so we're fundamentally less confused about the kind of work that's likely to be needed.
What do I mean by this? Let's say that we're trying to develop a new chess-playing programs. Do we understand the problem well enough that we could solve it if someone handed us an arbitrarily large computer? Yes: We make the whole search tree, backtrack, see whether white has a winning move.
If we didn't know how to answer the question even with an arbitrarily large computer, then this would suggest that we were fundamentally confused about chess in some way. We'd either be missing the search-tree data structure or the backtracking algorithm, or we'd be missing some understanding of how chess works.
This was the position we were in regarding chess prior to Claude Shannon's seminal paper, and it's the position we're currently in regarding many problems in AI alignment. No matter how large a computer you hand me, I could not make a smarter-than-human AI system that performs even a very simple limited-scope task (e.g., "put a strawberry on a plate without producing any catastrophic side-effects") or achieves even a very simple open-ended goal (e.g., "maximize the amount of diamond in the universe").
If I didn't have any particular goal in mind for the system, I could write a program (assuming an arbitrarily large computer) that strongly optimized the future in an undirected way, using a formalism like AIXI. In that sense we're less obviously confused about capabilities than about alignment, even though we're still missing a lot of pieces of the puzzle on the practical capabilities front.
Similarly, we do know how to leverage a powerful function optimizer to mine bitcoin or prove theorems. But we don't know how to (safely) do the kind of prediction and policy search tasks I described in the "fill a cauldron" section, even for modest goals in the physical world.
Our goal is to develop and formalize basic approaches and ways of thinking about the alignment problem, so that our engineering decisions don't end up depending on sophisticated and clever-sounding verbal arguments that turn out to be subtly mistaken. Simplifications like "what if we weren't worried about resource constraints?" and "what if we were trying to achieve a much simpler goal?" are a good place to start breaking down the problem into manageable pieces. For more on this methodology, see "MIRI's Approach.""Fill this cauldron without being too clever about it or working too hard or having any negative consequences I'm not anticipating" is a rough example of a goal that's intuitively limited in scope. The things we actually want to use smarter-than-human AI for are obviously more ambitious than that, but we'd still want to begin with various limited-scope tasks rather than open-ended goals.
Asimov's Three Laws of Robotics make for good stories partly for the same reasons they're unhelpful from a research perspective. The hard task of turning a moral precept into lines of code is hidden behind phrasings like "[don't,] through inaction, allow a human being to come to harm." If one followed a rule like that strictly, the result would be massively disruptive, as AI systems would need to systematically intervene to prevent even the smallest risks of even the slightest harms; and if the intent is that one follow the rule loosely, then all the work is being done by the human sensibilities and intuitions that tell us when and how to apply the rule.
A common response here is that vague natural-language instruction is sufficient, because smarter-than-human AI systems are likely to be capable of natural language comprehension. However, this is eliding the distinction between the system's objective function and its model of the world. A system acting in an environment containing humans may learn a world-model that has lots of information about human language and concepts, which the system can then use to achieve its objective function; but this fact doesn't imply that any of the information about human language and concepts will "leak out" and alter the system's objective function directly.
Some kind of value learning process needs to be defined where the objective function itself improves with new information. This is a tricky task because there aren't known (scalable) metrics or criteria for value learning in the way that there are for conventional learning.
If a system's world-model is accurate in training environments but fails in the real world, then this is likely to result in lower scores on its objective function — the system itself has an incentive to improve. The severity of accidents is also likelier to be self-limiting in this case, since false beliefs limit a system's ability to effectively pursue strategies.
In contrast, if a system's value learning process results in a 𝗨 that matches our 𝘝 in training but diverges from 𝘝 in the real world, then the system's 𝗨 will obviously not penalize it for optimizing 𝗨. The system has no incentive relative to 𝗨 to "correct" divergences between 𝗨 and 𝘝, if the value learning process is initially flawed. And accident risk is larger in this case, since a mismatch between 𝗨 and 𝘝 doesn't necessarily place any limits on the system's instrumental effectiveness at coming up with effective and creative strategies for achieving 𝗨.
The problem is threefold:
1. "Do What I Mean" is an informal idea, and even if we knew how to build a smarter-than-human AI system, we wouldn't know how to precisely specify this idea in lines of code.
2. If doing what we actually mean is instrumentally useful for achieving a particular objective, then a sufficiently capable system may learn how to do this, and may act accordingly so long as doing so is useful for its objective. But as systems become more capable, they are likely to find creative new ways to achieve the same objectives, and there is no obvious way to get an assurance that "doing what I mean" will continue to be instrumentally useful indefinitely.
3. If we use value learning to refine a system's goals over time based on training data that appears to be guiding the system toward a 𝗨 that inherently values doing what we mean, it is likely that the system will actually end up zeroing in on a 𝗨 that approximately does what we mean during training but catastrophically diverges in some difficult-to-anticipate contexts. See "Goodhart's Curse" for more on this.
For examples of problems faced by existing techniques for learning goals and facts, such as reinforcement learning, see "Using Machine Learning to Address AI Risk."The result will probably not be a particularly human-like design, since so many complex historical contingencies were involved in our evolution. The result will also be able to benefit from a number of large software and hardware advantages.This concept is sometimes lumped into the "transparency" category, but standard algorithmic transparency research isn't really addressing this particular problem. A better term for what I have in mind here is "understanding." What we want is to gain deeper and broader insights into the kind of cognitive work the system is doing and how this work relates to the system's objectives or optimization targets, to provide a conceptual lens with which to make sense of the hands-on engineering work.We could choose to program the system to tire, but we don't have to. In principle, one could program a broom that only ever finds and executes actions that optimize the fullness of the cauldron. Improving the system's ability to efficiently find high-scoring actions (in general, or relative to a particular scoring rule) doesn't in itself change the scoring rule it's using to evaluate actions.We can imagine the latter case resulting in a feedback loop as the system's design improvements allow it to come up with further design improvements, until all the low-hanging fruit is exhausted.
Another important consideration is that two of the main bottlenecks to humans doing faster scientific research are training time and communication bandwidth. If we could train a new mind to be a cutting-edge scientist in ten minutes, and if scientists could near-instantly trade their experience, knowledge, concepts, ideas, and intuitions to their collaborators, then scientific progress might be able to proceed much more rapidly. Those sorts of bottlenecks are exactly the sort of bottleneck that might give automated innovators an enormous edge over human innovators even without large advantages in hardware or algorithms.Specifically, rockets experience a wider range of temperatures and pressures, traverse those ranges more rapidly, and are also packed more fully with explosives.Consider Bird and Layzell's example of a very simple genetic algorithm that was tasked with evolving an oscillating circuit. Bird and Layzell were astonished to find that the algorithm made no use of the capacitor on the chip; instead, it had repurposed the circuit tracks on the motherboard as a radio to replay the oscillating signal from the test device back to the test device.
This was not a very smart program. This is just using hill climbing on a very small solution space. In spite of this, the solution turned out to be outside the space of solutions the programmers were themselves visualizing. In a computer simulation, this algorithm might have behaved as intended, but the actual solution space in the real world was wider than that, allowing hardware-level interventions.
In the case of an intelligent system that's significantly smarter than humans on whatever axes you're measuring, you should by default expect the system to push toward weird and creative solutions like these, and for the chosen solution to be difficult to anticipate.The post Ensuring smarter-than-human intelligence has a positive outcome appeared first on Machine Intelligence Research Institute.


Decisions are for making bad outcomes inconsistent

Nate Soares' recent decision theory paper with Ben Levinstein, "Cheating Death in Damascus," prompted some valuable questions and comments from an acquaintance (anonymized here). I've put together edited excerpts from the commenter's email below, with Nate's responses.
The discussion concerns functional decision theory (FDT), a newly proposed alternative to causal decision theory (CDT) and evidential decision theory (EDT). Where EDT says "choose the most auspicious action" and CDT says "choose the action that has the best effects," FDT says "choose the output of one's decision algorithm that has the best effects across all instances of that algorithm."
FDT usually behaves similarly to CDT. In a one-shot prisoner's dilemma between two agents who know they are following FDT, however, FDT parts ways with CDT and prescribes cooperation, on the grounds that each agent runs the same decision-making procedure, and that therefore each agent is effectively choosing for both agents at once.1
Below, Nate provides some of his own perspective on why FDT generally achieves higher utility than CDT and EDT. Some of the stances he sketches out here are stronger than the assumptions needed to justify FDT, but should shed some light on why researchers at MIRI think FDT can help resolve a number of longstanding puzzles in the foundations of rational action.

Anonymous: This is great stuff! I'm behind on reading loads of papers and books for my research, but this came across my path and hooked me, which speaks highly of how interesting is the content and the sense that this paper is making progress.
My general take is that you are right that these kinds of problems need to be specified in more detail. However, my guess is that once you do so, game theorists would get the right answer. Perhaps that's what FDT is: it's an approach to clarifying ambiguous games that leads to a formalism where people like Pearl and myself can use our standard approaches to get the right answer.
I know there's a lot of inertia in the "decision theory" language, so probably it doesn't make sense to change. But if there were no such sunk costs, I would recommend a different framing. It's not that people's decision theories are wrong; it's that they are unable to correctly formalize problems in which there are high-performance predictors. You show how to do that, using the idea of intervening on (i.e., choosing between putative outputs of) the algorithm, rather than intervening on actions. Everything else follows from a sufficiently precise and non-contradictory statement of the decision problem.
Probably the easiest move this line of work could make to ease this knee-jerk response of mine in defense of mainstream Bayesian game theory is to just be clear that CDT is not meant to capture mainstream Bayesian game theory. Rather, it is a model of one response to a class of problems not normally considered and for which existing approaches are ambiguous.

Nate Soares: I don't take this view myself. My view is more like: When you add accurate predictors to the Rube Goldberg machine that is the universe — which can in fact be done — the future of that universe can be determined by the behavior of the algorithm being predicted. The algorithm that we put in the "thing-being-predicted" slot can do significantly better if its reasoning on the subject of which actions to output respects the universe's downstream causal structure (which is something CDT and FDT do, but which EDT neglects), and it can do better again if its reasoning also respects the world's global logical structure (which is done by FDT alone).
We don't know exactly how to respect this wider class of dependencies in general yet, but we do know how to do it in many simple cases. While it agrees with modern decision theory and game theory in many simple situations, its prescriptions do seem to differ in non-trivial applications.
The main case where we can easily see that FDT is not just a better tool for formalizing game theorists' traditional intuitions is in prisoner's dilemmas. Game theory is pretty adamant about the fact that it's rational to defect in a one-shot PD, whereas two FDT agents facing off in a one-shot PD will cooperate.
In particular, classical game theory employs a "common knowledge of shared rationality" assumption which, when you look closely at it, cashes out more or less as "common knowledge that all parties are using CDT and this axiom." Game theory where common knowledge of shared rationality is defined to mean "common knowledge that all parties are using FDT and this axiom" gives substantially different results, such as cooperation in one-shot PDs.

A causal graph of Death in Damascus for CDT agents.2
Anonymous: When I've read MIRI work on CDT in the past, it seemed to me to describe what standard game theorists mean by rationality. But at least in cases like Murder Lesion, I don't think it's fair to say that standard game theorists would prescribe CDT. It might be better to say that standard game theory doesn't consider these kinds of settings, and there are multiple ways of responding to them, CDT being one.
But I also suspect that many of these perfect prediction problems are internally inconsistent, and so it's irrelevant what CDT would prescribe, since the problem cannot arise. That is, it's not reasonable to say game theorists would recommend such-and-such in a certain problem, when the problem postulates that the actor always has incorrect expectations; "all agents have correct expectations" is a core property of most game-theoretic problems.
The Death in Damascus problem for CDT agents is a good example of this. In this problem, either Death will not find the CDT agent with certainty, or the CDT agent will never have correct beliefs about her own actions, or she will be unable to best respond to her own beliefs.
So the problem statement ("Death finds the agent with certainty") rules out typical assumptions of a rational actor: that it has rational expectations (including about its own behavior), and that it can choose the preferred action in response to its beliefs. The agent can only have correct beliefs if she believes that she has such-and-such belief about which city she'll end up in, but doesn't select the action that is the best response to that belief.

Nate: I contest that last claim. The trouble is in the phrase "best response", where you're using CDT's notion of what counts as the best response. According to FDT's notion of "best response", the best response to your beliefs in the Death in Damascus problem is to stay in Damascus, if we're assuming it costs nonzero utility to make the trek to Aleppo.
In order to define what the best response to a problem is, we normally invoke a notion of counterfactuals — what are your available responses, and what do you think follows from them? But the question of how to set up those counterfactuals is the very point under contention.
So I'll grant that if you define "best response" in terms of CDT's counterfactuals, then Death in Damascus rules out the typical assumptions of a rational actor. If you use FDT's counterfactuals (i.e., counterfactuals that respect the full range of subjunctive dependencies), however, then you get to keep all the usual assumptions of rational actors. We can say that FDT has the pre-theoretic advantage over CDT that it allows agents to exhibit sensible-seeming properties like these in a wider array of problems.

Anonymous: The presentation of the Death in Damascus problem for CDT feels weird to me. CDT might also just turn up an error, since one of its assumptions is violated by the problem. Or it might cycle through beliefs forever… The expected utility calculation here seems to give some credence to the possibility of dodging death, which is assumed to be impossible, so it doesn't seem to me to correctly reason in a CDT way about where death will be.
For some reason I want to defend the CDT agent, and say that it's not fair to say they wouldn't realize that their strategy produces a contradiction (given the assumptions of rational belief and agency) in this problem.

Nate: There are a few different things to note here. First is that my inclination is always to evaluate CDT as an algorithm: if you built a machine that follows the CDT equation to the very letter, what would it do?
The answer here, as you've rightly noted above, is that the CDT equation isn't necessarily defined when the input is a problem like Death in Damascus, and I agree that simple definitions of CDT yield algorithms that would either enter an infinite loop or crash. The third alternative is that the agent notices the difficulty and engages in some sort of reflective-equilibrium-finding procedure; variants of CDT with this sort of patch were invented more or less independently by Joyce and Arntzenius to do exactly that. In the paper, we discuss the variants that run an equilibrium-finding procedure and show that the equilibrium is still unsatisfactory; but we probably should have been more explicit about the fact that vanilla CDT either crashes or loops.
Second, I acknowledge that there's still a strong intuition that an agent should in some sense be able to reflect on their own instability, look at the problem statement, and say, "Aha, I see what's going on here; Death will find me no matter what I choose; I'd better find some other way to make the decision." However, this sort of response is explicitly ruled out by the CDT equation: CDT says you must evaluate your actions as if they were subjunctively independent of everything that doesn't causally depend on them.
In other words, you're correct that CDT agents know intellectually that they cannot escape Death, but the CDT equation requires agents to imagine that they can, and to act on this basis.
And, to be clear, it is not a strike against an algorithm for it to prescribe making actions by reasoning about impossible scenarios — any deterministic algorithm attempting to reason about what it "should do" must imagine some impossibilities, because a deterministic algorithm has to reason about the consequences of doing lots of different things, but is in fact only going to do one thing.
The question at hand is which impossibilities are the right ones to imagine, and the claim is that in scenarios with accurate predictors, CDT prescribes imagining the wrong impossibilities, including impossibilities where it escapes Death.
Our human intuitions say that we should reflect on the problem statement and eventually realize that escaping Death is in some sense "too impossible to consider". But this directly contradicts the advice of CDT. Following this intuition requires us to make our beliefs obey a logical-but-not-causal constraint in the problem statement ("Death is a perfect predictor"), which FDT agents can do but CDT agents can't. On close examination, the "shouldn't CDT realize this is wrong?" intuition turns out to be an argument for FDT in another guise. (Indeed, pursuing this intuition is part of how FDT's predecessors were discovered!)
Third, I'll note it's an important virtue in general for decision theories to be able to reason correctly in the face of apparent inconsistency. Consider the following simple example:
An agent has a choice between taking $1 or taking $100. There is an extraordinarily tiny but nonzero probability that a cosmic ray will spontaneously strike the agent's brain in such a way that they will be caused to do the opposite of whichever action they would normally do. If they learn that they have been struck by a cosmic ray, then they will also need to visit the emergency room to ensure there's no lasting brain damage, at a cost of $1000. Furthermore, the agent knows that they take the $100 if and only if they are hit by the cosmic ray.
When faced with this problem, EDT agents reason: "If I take the $100, then I must have been hit by the cosmic ray, which means that I lose $900 on net. I therefore prefer the $1." They then take the $1 (except in cases where they have been hit by the cosmic ray).
Since this is just what the problem statement says — "the agent knows that they take the $100 if and only if they are hit by the cosmic ray" — the problem is perfectly consistent, as is EDT's response to the problem. EDT only cares about correlation, not dependency; so EDT agents are perfectly happy to buy into self-fulfilling prophecies, even when it means turning their backs on large sums of money.
What happens when we try to pull this trick on a CDT agent? She says, "Like hell I only take the $100 if I'm hit by the cosmic ray!" and grabs the $100 — thus revealing your problem statement to be inconsistent if the agent runs CDT as opposed to EDT.
The claim that "the agent knows that they take the $100 if and only if they are hit by the cosmic ray" contradicts the definition of CDT, which requires that agents of CDT refuse to leave free money on the table. As you may verify, FDT also renders the problem statement inconsistent, for similar reasons. The definition of EDT, on the other hand, is fully consistent with the problem as stated.
This means that if you try to put EDT into the above situation — controlling its behavior by telling it specific facts about itself — you will succeed; whereas if you try to put CDT into the above situation, you will fail, and the supposed facts will be revealed as lies. Whether or not the above problem statement is consistent depends on the algorithm that the agent runs, and the design of the algorithm controls the degree to which you can put that algorithm in bad situations.
We can think of this as a case of FDT and CDT succeeding in making a low-utility universe impossible, where EDT fails to make a low-utility universe impossible. The whole point of implementing a decision theory on a piece of hardware and running it is to make bad futures-of-our-universe impossible (or at least very unlikely). It's a feature of a decision theory, and not a bug, for there to be some problems where one tries to describe a low-utility state of affairs and the decision theory says, "I'm sorry, but if you run me in that problem, your problem will be revealed as inconsistent".3
This doesn't contradict anything you've said; I say it only to highlight how little we can conclude from noticing that an agent is reasoning about an inconsistent state of affairs. Reasoning about impossibilities is the mechanism by which decision theories produce actions that force the outcome to be desirable, so we can't conclude that an agent has been placed in an unfair situation from the fact that the agent is forced to reason about an impossibility.

A causal graph of the XOR blackmail problem for CDT agents.4

Anonymous: Something still seems fishy to me about decision problems that assume perfect predictors. If I'm being predicted with 100% accuracy in the XOR blackmail problem, then this means that I can induce a contradiction. If I follow FDT and CDT's recommendation of never paying, then I only receive a letter when I have termites. But if I pay, then I must be in the world where I don't have termites, as otherwise there is a contradiction.
So it seems that I am able to intervene on the world in a way that changes the state of termites for me now, given that I've received a letter. That is, the best strategy when starting is to never pay, but the best strategy given that I will receive a letter is to pay. The weirdness arises because I'm able to intervene on the algorithm, but we are conditioning on a fact of the world that depends on my algorithm.
Not sure if this confusion makes sense to you. My gut says that these kinds of problems are often self-contradicting, at least when we assert 100% predictive performance. I would prefer to work it out from the ex ante situation, with specified probabilities of getting termites, and see if it is the case that changing one's strategy (at the algorithm level) is possible without changing the probability of termites to maintain consistency of the prediction claim.

Nate: First, I'll note that the problem goes through fine if the prediction is only correct 99% of the time. If the difference between "cost of termites" and "cost of paying" is sufficient high, then the problem can probably go through even if the predictor is only correct 51% of the time.
That said, the point of this example is to draw attention to some of the issues you're raising here, and I think that these issues are just easier to think about when we assume 100% predictive accuracy.
The claim I dispute is this one: "That is, the best strategy when starting is to never pay, but the best strategy given that I will receive a letter is to pay." I claim that the best strategy given that you receive the letter is to not pay, because whether you pay has no effect on whether or not you have termites. Whenever you pay, no matter what you've learned, you're basically just burning $1000.
That said, you're completely right that these decision problems have some inconsistent branches, though I claim that this is true of any decision problem. In a deterministic universe with deterministic agents, all "possible actions" the agent "could take" save one are not going to be taken, and thus all "possibilities" save one are in fact inconsistent given a sufficiently full formal specification.
I also completely endorse the claim that this set-up allows the predicted agent to induce a contradiction. Indeed, I claim that all decision-making power comes from the ability to induce contradictions: the whole reason to write an algorithm that loops over actions, constructs models of outcomes that would follow from those actions, and outputs the action corresponding to the highest-ranked outcome is so that it is contradictory for the algorithm to output a suboptimal action.
This is what computer programs are all about. You write the code in such a fashion that the only non-contradictory way for the electricity to flow through the transistors is in the way that makes your computer do your tax returns, or whatever.
In the case of the XOR blackmail problem, there are four "possible" worlds: LT (letter + termites), NT (noletter + termites), LN (letter + notermites), and NN (noletter + notermites).
The predictor, by dint of their accuracy, has put the universe into a state where the only consistent possibilities are either (LT, NN) or (LN, NT). You get to choose which of those pairs is consistent and which is contradictory. Clearly, you don't have control over the probability of termites vs. notermites, so you're only controlling whether you get the letter. Thus, the question is whether you're willing to pay $1000 to make sure that the letter shows up only in the worlds where you don't have termites.
Even when you're holding the letter in your hands, I claim that you should not say "if I pay I will have no termites", because that is false — your action can't affect whether you have termites. You should instead say:
I see two possibilities here. If my algorithm outputs pay, then in the XX% of worlds where I have termites I get no letter and lose $1M, and in the (100-XX)% of worlds where I do not have termites I lose $1k. If instead my algorithm outputs refuse, then in the XX% of worlds where I have termites I get this letter but only lose $1M, and in the other worlds I lose nothing. The latter mixture is preferable, so I do not pay.
You'll notice that the agent in this line of reasoning is not updating on the fact that they're holding the letter. They're not saying, "Given that I know that I received the letter and that the universe is consistent…"
One way to think about this is to imagine the agent as not yet being sure whether or not they're in a contradictory universe. They act like this might be a world in which they don't have termites, and they received the letter; and in those worlds, by refusing to pay, they make the world they inhabit inconsistent — and thereby make this very scenario never-have-existed.
And this is correct reasoning! For when the predictor makes their prediction, they'll visualize a scenario where the agent has no termites and receives the letter, in order to figure out what the agent would do. When the predictor observes that the agent would make that universe contradictory (by refusing to pay), they are bound (by their own commitments, and by their accuracy as a predictor) to send the letter only when you have termites.5
You'll never find yourself in a contradictory situation in the real world, but when an accurate predictor is trying to figure out what you'll do, they don't yet know which situations are contradictory. They'll therefore imagine you in situations that may or may not turn out to be contradictory (like "letter + notermites"). Whether or not you would force the contradiction in those cases determines how the predictor will behave towards you in fact.
The real world is never contradictory, but predictions about you can certainly place you in contradictory hypotheticals. In cases where you want to force a certain hypothetical world to imply a contradiction, you have to be the sort of person who would force the contradiction if given the opportunity.
Or as I like to say — forcing the contradiction never works, but it always would've worked, which is sufficient.

Anonymous: The FDT algorithm is best ex ante. But if what you care about is your utility in your own life flowing after you, and not that of other instantiations, then upon hearing this news about FDT you should do whatever is best for you given that information and your beliefs, as per CDT.

A causal graph of Newcomb's problem for FDT agents.6
Nate: If you have the ability to commit yourself to future behaviors (and actually stick to that), it's clearly in your interest to commit now to behaving like FDT on all decision problems that begin in your future. I, for instance, have made this commitment myself. I've also made stronger commitments about decision problems that began in my past, but all CDT agents should agree in principle on problems that begin in the future.7
I do believe that real-world people like you and me can actually follow FDT's prescriptions, even in cases where those prescriptions are quite counter-intuitive.
Consider a variant of Newcomb's problem where both boxes are transparent, so that you can already see whether box B is full before choosing whether to two-box. In this case, EDT joins CDT in two-boxing, because one-boxing can no longer serve to give the agent good news about its fortunes. But FDT agents still one-box, for the same reason they one-box in Newcomb's original problem and cooperate in the prisoner's dilemma: they imagine their algorithm controlling all instances of their decision procedure, including the past copy in the mind of their predictor.
Now, let's suppose that you're standing in front of two full boxes in the transparent Newcomb problem. You might say to yourself, "I wish I could have committed beforehand, but now that the choice is before me, the tug of the extra $1000 is just too strong", and then decide that you were not actually capable of making binding precommitments. This is fine; the normatively correct decision theory might not be something that all human beings have the willpower to follow in real life, just as the correct moral theory could turn out to be something that some people lack the will to follow.8
That said, I believe that I'm quite capable of just acting like I committed to act. I don't feel a need to go through any particular mental ritual in order to feel comfortable one-boxing. I can just decide to one-box and let the matter rest there.
I want to be the kind of agent that sees two full boxes, so that I can walk away rich. I care more about doing what works, and about achieving practical real-world goals, than I care about the intuitiveness of my local decisions. And in this decision problem, FDT agents are the only agents that walk away rich.
One way of making sense of this kind of reasoning is that evolution graced me with a "just do what you promised to do" module. The same style of reasoning that allows me to actually follow through and one-box in Newcomb's problem is the one that allows me to cooperate in prisoner's dilemmas against myself — including dilemmas like "should I stick to my New Year's resolution?"9 I claim that it was only misguided CDT philosophers that argued (wrongly) that "rational" agents aren't allowed to use that evolution-given "just follow through with your promises" module.

Anonymous: A final point: I don't know about counterlogicals, but a theory of functional similarity would seem to depend on the details of the algorithms.
E.g., we could have a model where their output is stochastic, but some parameters of that process are the same (such as expected value), and the action is stochastically drawn from some distribution with those parameter values. We could have a version of that, but where the parameter values depend on private information picked up since the algorithms split, in which case each agent would have to model the distribution of private info the other might have.
That seems pretty general; does that work? Is there a class of functional similarity that can not be expressed using that formulation?

Nate: As long as the underlying distribution can be an arbitrary Turing machine, I think that's sufficiently general.
There are actually a few non-obvious technical hurdles here; namely, if agent A is basing their beliefs off of their model of agent B, who is basing their beliefs off of a model of agent A, then you can get some strange loops.
Consider for example the matching pennies problem: agent A and agent B will each place a penny on a table; agent A wants either HH or TT, and agent B wants either HT or TH. It's non-trivial to ensure that both agents develop stable accurate beliefs in games like this (as opposed to, e.g., diving into infinite loops).
The technical solution to this is reflective oracle machines, a class of probabilistic Turing machines with access to an oracle that can probabilistically answer questions about any other machine in the class (with access to the same oracle).
The paper "Reflective Oracles: A Foundation for Classical Game Theory" shows how to do this and shows that the relevant fixed points always exist. (And furthermore, in cases that can be represented in classical game theory, the fixed points always correspond to the mixed-strategy Nash equilibria.)
This more or less lets us start from a place of saying "how do agents with probabilistic information about each other's source code come to stable beliefs about each other?" and gets us to the "common knowledge of rationality" axiom from game theory.10 One can also see it as a justification for that axiom, or as a generalization of that axiom that works even in cases where the lines between agent and environment get blurry, or as a hint at what we should do in cases where one agent has significantly more computational resources than the other, etc.
But, yes, when we study these kinds of problems concretely at MIRI, we tend to use models where each agent models the other as a probabilistic Turing machine, which seems roughly in line with what you're suggesting here.

CDT prescribes defection in this dilemma, on the grounds that one's action cannot cause the other agent to cooperate. FDT outperforms CDT in Newcomblike dilemmas like these, while also outperforming EDT in other dilemmas, such as the smoking lesion problem and XOR blackmail.The agent's predisposition determines whether they will flee to Aleppo or stay in Damascus, and also determines Death's prediction about their decision. This allows Death to inescapably pursue the agent, making flight pointless; but CDT agents can't incorporate this fact into their decision-making.There are some fairly natural ways to cash out Murder Lesion where CDT accepts the problem and FDT forces a contradiction, but we decided not to delve into that interpretation in the paper.
Tangentially, I'll note that one of the most common defenses of CDT similarly turns on the idea that certain dilemmas are "unfair" to CDT. Compare, for example, David Lewis' "Why Ain'cha Rich?"
It's obviously possible to define decision problems that are "unfair" in the sense that they just reward or punish agents for having a certain decision theory. We can imagine a dilemma where a predictor simply guesses whether you're implementing FDT, and gives you $1,000,000 if so. Since we can construct symmetric dilemmas that instead reward CDT agents, EDT agents, etc., these dilemmas aren't very interesting, and can't help us choose between theories.
Dilemmas like Newcomb's problem and Death in Damascus, however, don't evaluate agents based on their decision theories. They evaluate agents based on their actions, and the task of the decision theory is to determine which action is best. If it's unfair to criticize CDT for making the wrong choice in problems like this, then it's hard to see on what grounds we can criticize any agent for making a wrong choice in any problem, since one can always claim that one is merely at the mercy of one's decision theory.Our paper describes the XOR blackmail problem like so:
An agent has been alerted to a rumor that her house has a terrible termite infestation, which would cost her $1,000,000 in damages. She doesn't know whether this rumor is true. A greedy and accurate predictor with a strong reputation for honesty has learned whether or not it's true, and drafts a letter:
"I know whether or not you have termites, and I have sent you this letter iff exactly one of the following is true: (i) the rumor is false, and you are going to pay me $1,000 upon receiving this letter; or (ii) the rumor is true, and you will not pay me upon receiving this letter."
The predictor then predicts what the agent would do upon receiving the letter, and sends the agent the letter iff exactly one of (i) or (ii) is true. Thus, the claim made by the letter is true. Assume the agent receives the letter. Should she pay up?
In this scenario, EDT pays the blackmailer, while CDT and FDT refuse to pay. See the "Cheating Death in Damascus" paper for more details.Ben Levinstein notes that this can be compared to backward induction in game theory with common knowledge of rationality. You suppose you're at some final decision node which you only would have gotten to (as it turns out) if the players weren't actually rational to begin with.FDT agents intervene on their decision function, "FDT(P,G)". The CDT version replaces this node with "Predisposition" and instead intervenes on "Act".Specifically, the CDT-endorsed response here is: "Well, I'll commit to acting like an FDT agent on future problems, but in one-shot prisoner's dilemmas that began in my past, I'll still defect against copies of myself".
The problem with this response is that it can cost you arbitrary amounts of utility, provided a clever blackmailer wishes to take advantage. Consider the retrocausal blackmail dilemma in "Toward Idealized Decision Theory":
There is a wealthy intelligent system and an honest AI researcher with access to the agent's original source code. The researcher may deploy a virus that will cause $150 million each in damages to both the AI system and the researcher, and which may only be deactivated if the agent pays the researcher $100 million. The researcher is risk-averse and only deploys the virus upon becoming confident that the agent will pay up. The agent knows the situation and has an opportunity to self-modify after the researcher acquires its original source code but before the researcher decides whether or not to deploy the virus. (The researcher knows this, and has to factor this into their prediction.)
CDT pays the retrocausal blackmailer, even if it has the opportunity to precommit to do otherwise. FDT (which in any case has no need for precommitment mechanisms) refuses to pay. I cite the intuitive undesirability of this outcome to argue that one should follow FDT in full generality, as opposed to following CDT's prescription that one should only behave in FDT-like ways in future dilemmas.
The argument above must be made from a pre-theoretic vantage point, because CDT is internally consistent. There is no argument one could give to a true CDT agent that would cause it to want to use anything other than CDT in decision problems that began in its past.
If examples like retrocausal blackmail have force (over and above the force of other arguments for FDT), it is because humans aren't genuine CDT agents. We may come to endorse CDT based on its theoretical and practical virtues, but the case for CDT is defeasible if we discover sufficiently serious flaws in CDT, where "flaws" are evaluated relative to more elementary intuitions about which actions are good or bad. FDT's advantages over CDT and EDT — properties like its greater theoretical simplicity and generality, and its achievement of greater utility in standard dilemmas — carry intuitive weight from a position of uncertainty about which decision theory is correct.In principle, it could even turn out that following the prescriptions of the correct decision theory in full generality is humanly impossible. There's no law of logic saying that the normatively correct decision-making behaviors have to be compatible with arbitrary brain designs (including human brain design). I wouldn't bet on this, but in such a case learning the correct theory would still have practical import, since we could still build AI systems to follow the normatively correct theory.A New Year's resolution that requires me to repeatedly follow through on a promise that I care about in the long run, but would prefer to ignore in the moment, can be modeled as a one-shot twin prisoner's dilemma. In this case, the dilemma is temporally extended, and my "twins" are my own future selves, who I know reason more or less the same way I do.
It's conceivable that I could go off my diet today ("defect") and have my future selves pick up the slack for me and stick to the diet ("cooperate"), but in practice if I'm the kind of agent who isn't willing today to sacrifice short-term comfort for long-term well-being, then I presumably won't be that kind of agent tomorrow either, or the day after.
Seeing that this is so, and lacking a way to force themselves or their future selves to follow through, CDT agents despair of promise-keeping and abandon their resolutions. FDT agents, seeing the same set of facts, do just the opposite: they resolve to cooperate today, knowing that their future selves will reason symmetrically and do the same.The paper above shows how to use reflective oracles with CDT as opposed to FDT, because (a) one battle at a time and (b) we don't yet have a generic algorithm for computing logical counterfactuals, but we do have a generic algorithm for doing CDT-type reasoning.The post Decisions are for making bad outcomes inconsistent appeared first on Machine Intelligence Research Institute.