Solomonoff induction

Solomonoff induction is an ideal answer to questions like "What probably comes next in the sequence 1, 1, 2, 3, 5, 8?" or "Given the last three years of visual data from this webcam, what will this robot probably see next?" or "Will the sun rise tomorrow?"  Solomonoff induction requires infinite computing power, and is defined by taking every computable algorithm for giving a probability distribution over future data given past data, weighted by their [5v algorithmic simplicity], and [updating those weights](BayesRule-1) by comparison to the actual data.

E.g., somewhere in the ideal Solomonoff distribution is an exact copy of *you, right now*, staring at a string of 1s and 0s and trying to predict what comes next - though this copy of you starts out with a very low weight in the mixture owing to its complexity.  Since a copy of you is present in this mixture of computable predictors, we can prove a theorem about how well Solomonoff induction does compared to an exact copy of you; namely, Solomonoff induction commits only a bounded amount of error relative to you, or any other computable way of making predictions.  Solomonoff induction is thus a kind of perfect or rational ideal for probabilistically predicting sequences, although it cannot be implemented in reality due to requiring infinite computing power.  Still, considering Solomonoff induction can give us important insights into how non-ideal reasoning should operate in the real world.

Additional reading:

- https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_inductive_inference
- http://lesswrong.com/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/
- http://wiki.lesswrong.com/wiki/Solomonoff_induction


Machine Intelligence Research Institute

The Machine Intelligence Research Institute, a primary host of more formal and technical work on [2v value alignment theory].  (See [18m] for a list of all researchers in the field and their affiliations.)  Currently supported primarily by individual donors, MIRI's specialization is in supporting technically able researchers who are coming in from outside academia, supporting basic research that may not immediately lead to publications, and carrying out very-high-leverage additional operations that can operate without academic overhead or going through a larger bureaucracy.  MIRI is also the de facto center for work on [18s decision theory] and [1c1].  MIRI operates out of Berkeley, CA near the UC Berkeley campus that hosts [StuartRussell], but is not directly affiliated with UC Berkeley.


Mindcrime: Introduction

The more predictive accuracy we want from a model, the more detailed the model becomes.  A very rough model of an airplane might only contain the approximate shape, the power of the engines, and the mass of the airplane.  A model good enough for engineering needs to be detailed enough to simulate the flow of air over the wings, the centripetal force on the fan blades, and more.  As a model can predict the airplane in more and more fine detail and with better and better probability distributions, the computations carried out to make the model's predictions may start to look more and more like a detail simulation of the airplane flying.

Consider a machine intelligence building, and testing, the best models it can manage of a human being's behavior.  If the model that produces the *best* predictions involves simulations with moderate degrees of isomorphism to human cognition, then the model, as it runs, may itself be self-aware or conscious or sapient or whatever other property stands in for being an object of ethical concern.  This doesn't mean that the running model of Fred is Fred, or even that the running model of Fred is human.  The concern is that a sufficiently advanced model of a person will be *a* person, even if they might not be the *same* person.

We might then worry that, for example, if Fred is unhappy, or *might* be unhappy, the agent will consider thousands or millions of hypotheses about versions of Fred.  Hypotheses about suffering versions of Fred, when run, might themselves be suffering.  As a similar concern, these hypotheses about Fred might then be discarded - cease to be run - if the agent sees new evidence and updates its model.  Since [18j programs can be people], stopping and erasing a conscious program is the crime of murder.

This scenario, which we might call 'the problem of sapient models', is a subscenario of the general problem of what Bostrom terms 'mindcrime'.  ([2] has suggested 'mindgenocide' as a term with fewer Orwellian connotations.)  More generally, we might worry that there are agent systems that do huge amounts of moral harm just in virtue of the way they compute, by containing embedded conscious suffering and death.

Another scenario might be called 'the problem of sapient subsystems'.  It's possible that, for example, the most efficient possible system for, e.g., allocating memory to subprocesses, is a memory-allocating-subagent that is reflective enough to be an independently conscious person.  This is distinguished from the problem of creating a single machine intelligence that is conscious and suffering, because the conscious agent might be hidden at a lower level of a design, and there might be a lot *more* of them than just one suffering superagent.

Both of these scenarios constitute moral harm done inside the agent's computations, irrespective of its external behavior.  We can't conclude that we've done no harm by building a superintelligence, just in virtue of the fact that the superintelligence doesn't outwardly kill anyone.  There could be trillions of people suffering and dying *inside* the superintelligence.  This sets mindcrime apart from almost all other concerns within the [5s], which usually revolve around external behavior.

To avoid mindgenocide, it would be very handy to know exactly which computations are or are not conscious, sapient, or otherwise objects of ethical concern.  Or, indeed, to know that any particular class of computations are *not* objects of ethical concern.

Yudkowsky calls a [ nonperson predicate] any computable test we could safely use to determine that a computation is definitely *not* a person.  This test only needs two possible answers, "Not a person" and "Don't know".  It's fine if the test says "Don't know" on some nonperson computations, so long as the test says "Don't know" on *all* people and never says "Not a person" when the computation is conscious after all.  Since the test only definitely tells us about nonpersonhood, rather than detecting personhood in any positive sense, we can call it a nonperson predicate.

However, the goal is not just to have any nonperson predicate - the predicate that only says "known nonperson" for the empty computation and no others meets this test.  The goal is to have a nonperson predicate that includes powerful, useful computations.  We want to be able to build an AI that is not a person, and let that AI build subprocesses that we know will not be people, and let that AI improve its models of environmental humans using hypotheses that we know are not people.  This means the nonperson predicate does need to pass some AI designs, cognitive subprocess designs, and human models that are good enough for whatever it is we want the AI to do.

This seems like it might be very hard for several reasons:

- There is *unusually extreme* philosophical dispute, and confusion, about exactly which programs are and are not conscious or otherwise objects of ethical value.  (It might not be exaggerating to scream "nobody knows what the hell is going on".)
- We can't fully pass any class of programs that's [ Turing-complete].  We can't say once and for all that it's safe to model gravitational interactions in a solar system, if enormous gravitational systems could encode computers that encode people.
- The [42] problem applies to any attempt to forbid an [2c advanced] [9h consequentialist agent] from using the most effective or obvious ways of modeling humans.  The *next* best way of modeling humans, outside the blocked-off options, is unusually likely to look like a weird loophole that turns out to encode sapience some way we didn't imagine.

An alternative for preventing mindcrime without a trustworthy [ nonperson predicate] is to consider [102 agent designs intended *not* to model humans, or other minds, in great detail], since there may be some [6y pivotal achievements] that can be accomplished without a value-aligned agent modeling human minds in detail.


Some computations are people

This proposition is true if at least some possible [ computations] (not necessarily any that could run on modern computers) have [ consciousness], sapience, or whatever other properties are necessary to make them people and therefore [ objects of ethical value].

Key argument:  Most domain experts think that human beings are themselves (a) Turing-computable and (b) conscious in virtue of the computations that they perform.  In other words, you yourself are a conscious algorithm.  If you consider yourself a person, then you consider at least one computer program (yourself) to be a person.

This is why some domain experts can be very confident of the proposition, despite the moral subquestions about which properties are necessary for personhood.  If you take for granted the [ Church-Turing thesis] stating that everything in the physical universe is computable and therefore so are human beings, then of course some computer programs (like you) can be people or have any other properties we associate with human beings.

This proposition falls into the class of issues that some people think are incredibly deep and fraught philosophical questions, and that other people think are incredibly deep philosophical questions that happen to have clear, known answers.


Nick Bostrom

Nick Bostrom is, with [2], one of the two cofounders of the current field of [2v value alignment theory].  Bostrom published a paper singling out the problem of superintelligent values as critical in 1999, two years before Yudkowsky entered the field, which has sometimes led Yudkowsky to say that Bostrom should receive credit for inventing the Friendly AI concept.  Bostrom is founder and director of the [ Oxford Future of Humanity Institute].  He is the author of the popular book [Superintelligence_book Superintelligence] that currently forms the best book-length introduction to the field.  Bostrom's academic background is as an analytic philosopher formerly specializing in [ anthropic probability theory] and [ transhumanist ethics].  Relative to Yudkowsky, Bostrom is relatively more interested in Oracle models of value alignment and in potential exotic methods of obtaining aligned goals.


Researchers in value alignment theory

This page lists researchers in [2v].

- [2] (founder, [15w MIRI])
- [18k] (founder, [FHI])
- [3q] (MIRI; [ parametric polymorphism], the [ Procrastination Paradox], and numerous other developments in [1c1].)
- [12y] (MIRI; [ modal agents])
- [StuartArmstrong] (FHI; [1b7])
- [3] (UC Berkeley, [ approval-directed agents], previously proposed a formalization of [ indirect normativity])
- [StuartRussell] (UC Berkeley; author of Artificial Intelligence: A Modern Approach; previously published on theories of reflective optimality; currently interested in [ inverse reinforcement learning].)
- [4y] (MIRI, [ reflective oracles])
- [131] (MIRI)
- [ScottGarabant] (MIRI, [ logical probabilities])
- [32] (previously MIRI researcher, now Executive Director at MIRI)


The rocket alignment problem

%todo: reference specific contributions by others or oneself that led to emergent discoveries at NASA et. al. -- proceed to display how these specific contributions may relate to current state of the art developments in artificial intelligence%

(*Somewhere in a not-very-near neighboring world, where science took a very different course...*)

Alfonso:  Hello, Beth. I've noticed a lot of speculations lately about "spaceplanes" being used to attack cities, or possibly becoming infused with malevolent spirits that inhabit the celestial realms so that they turn on their own engineers.

I'm rather skeptical of these speculations. Indeed, I'm a bit skeptical that airplanes will be able to even rise as high as stratospheric weather balloons anytime in the next century. But I understand that your institute wants to address the potential problem of malevolent or dangerous spaceplanes, and that you think this is an important present-day cause.

Beth:  That's… really not how we at the Mathematics of Intentional Rocketry Institute would phrase things.

The problem of malevolent celestial spirits is what all the news articles are focusing on, but we think the real problem is something entirely different. We're worried that there's a difficult, theoretically challenging problem which modern-day rocket punditry is mostly overlooking. We're worried that if you aim a rocket at where the Moon is in the sky, and press the launch button, the rocket may not actually end up at the Moon.

Alfonso:  I understand that it's very important to design fins that can stabilize a spaceplane's flight in heavy winds. That's important spaceplane safety research and someone needs to do it.

But if you were working on that sort of safety research, I'd expect you to be collaborating tightly with modern airplane engineers to test out your fin designs, to demonstrate that they are actually useful.

Beth:  Aerodynamic designs are important features of any safe rocket, and we're quite glad that rocket scientists are working on these problems and taking safety seriously. That's not the sort of problem that we at MIRI focus on, though.

Alfonso:  What's the concern, then? Do you fear that spaceplanes may be developed by ill-intentioned people?

Beth:  That's not the failure mode we're worried about right now. We're more worried that right now, *nobody* can tell you how to point your rocket's nose such that it goes to the moon, nor indeed *any* prespecified celestial destination. Whether Google or the US Government or North Korea is the one to launch the rocket won't make a pragmatic difference to the probability of a successful Moon landing from our perspective, because right now *nobody knows how to aim any kind of rocket anywhere*.

<!--more-->

Alfonso:  I'm not sure I understand.

Beth:  We're worried that even if you aim a rocket at the Moon, such that the nose of the rocket is clearly lined up with the Moon in the sky, the rocket won't go to the Moon. We're not sure what a realistic path from the Earth to the moon looks like, but we suspect it [might not be a very straight path](https://airandspace.si.edu/webimages/highres/5317h.jpg), and it may not involve pointing the nose of the rocket at the moon at all. We think the most important thing to do next is to advance our understanding of rocket trajectories until we have a better, deeper understanding of what we've started calling the "rocket alignment problem". There are other safety problems, but this rocket alignment problem will probably take the most total time to work on, so it's the most urgent.

Alfonso:  Hmm, that sounds like a bold claim to me. Do you have a reason to think that there are invisible barriers between here and the moon that the spaceplane might hit? Are you saying that it might get very very windy between here and the moon, more so than on Earth? Both eventualities could be worth preparing for, I suppose, but neither seem likely.

Beth:  We don't think it's particularly likely that there are invisible barriers, no. And we don't think it's going to be especially windy in the celestial reaches — quite the opposite, in fact. The problem is just that we don't yet know how to plot *any* trajectory that a vehicle could realistically take to get from Earth to the moon.

Alfonso:  Of course we can't plot an actual trajectory; wind and weather are too unpredictable. But your claim still seems too strong to me. Just aim the spaceplane at the moon, go up, and have the pilot adjust as necessary. Why wouldn't that work? Can you prove that a spaceplane aimed at the moon won't go there?

Beth:  We don't think we can *prove* anything of that sort, no. Part of the problem is that realistic calculations are extremely hard to do in this area, after you take into account all the atmospheric friction and the movements of other celestial bodies and such. We've been trying to solve some drastically simplified problems in this area, on the order of assuming that there is no atmosphere and that all rockets move in perfectly straight lines. Even those unrealistic calculations strongly suggest that, in the much more complicated real world, just pointing your rocket's nose at the Moon also won't make your rocket end up at the Moon. I mean, the fact that the real world is more complicated doesn't exactly make it any *easier* to get to the Moon.

Alfonso:  Okay, let me take a look at this "understanding" work you say you're doing…

Huh. Based on what I've read about the math you're trying to do, I can't say I understand what it has to do with the Moon. Shouldn't helping spaceplane pilots exactly target the Moon involve looking through lunar telescopes and studying exactly what the Moon looks like, so that the spaceplane pilots can identify particular features of the landscape to land on?

Beth:  We think our present stage of understanding is much too crude for a detailed Moon map to be our next research target. We haven't yet advanced to the point of targeting one crater or another for our landing. We can't target *anything* at this point. It's more along the lines of "figure out how to talk mathematically about curved rocket trajectories, instead of rockets that move in straight lines". Not even realistically curved trajectories, right now, we're just trying to get past straight lines at all -

Alfonso:  But planes on Earth move in curved lines all the time, because the Earth itself is curved. It seems reasonable to expect that future spaceplanes will also have the capability to move in curved lines. If your worry is that spaceplanes will only move in straight lines and miss the Moon, and you want to advise rocket engineers to build rockets that move in curved lines, well, that doesn't seem to me like a great use of anyone's time.

Beth:  You're trying to draw much too direct of a line between the math we're working on right now, and actual rocket designs that might exist in the future. It's *not* that current rocket ideas are almost right, and we just need to solve one or two more problems to make them work. The conceptual distance that separates anyone from solving the rocket alignment problem is *much greater* than that.

Right now everyone is *confused* about rocket trajectories, and we're trying to become *less confused*. That's what we need to do next, not run out and advise rocket engineers to build their rockets the way that our current math papers are talking about. Not until we stop being *confused* about extremely basic questions like why the Earth doesn't fall into the Sun.

Alfonso:  I don't think the Earth is going to collide with the Sun anytime soon. The Sun has been steadily circling the Earth for a long time now.

Beth:  I'm not saying that our goal is to address the risk of the Earth falling into the Sun. What I'm trying to say is that if humanity's present knowledge can't answer questions like "Why doesn't the Earth fall into the Sun?" then we don't know very much about celestial mechanics and we won't be able to aim a rocket through the celestial reaches in a way that lands softly on the Moon.

As an example of work we're presently doing that's aimed at improving our understanding, there's what we call the "[tiling positions](http://intelligence.org/files/TilingAgentsDraft.pdf)" problem. The tiling positions problem is [how to fire a cannonball from a cannon](https://en.wikipedia.org/wiki/Newton%27s_cannonball) in such a way that the cannonball circumnavigates the earth over and over again, "tiling" its initial coordinates like repeating tiles on a tessellated floor -

Alfonso:  I read a little bit about your work on that topic. I have to say, it's hard for me to see what firing things from cannons has to do with getting to the Moon. Frankly, it sounds an awful lot like Good Old-Fashioned Space Travel, which everyone knows doesn't work. Maybe Jules Verne thought it was possible to travel around the earth by firing capsules out of cannons, but the modern study of high-altitude planes has completely abandoned the notion of firing things out of cannons. The fact that you go around talking about firing things out of cannons suggests to me that you haven't kept up with all the innovations in airplane design over the last century, and that your spaceplane designs will be completely unrealistic.

Beth:  We know that rockets will not actually be fired out of cannons. We really, really know that. We're intimately familiar with the reasons why nothing fired out of a modern cannon is ever going to reach escape velocity. I've previously written several sequences of articles in which I describe why cannon-based space travel doesn't work.

Alfonso:  But your current work is all about firing something out a cannon in such a way that it circles the earth over and over. What could that have to do with any realistic advice that you could give to a spaceplane pilot about how to travel to the Moon?

Beth:  Again, you're trying to draw much too straight a line between the math we're doing right now, and direct advice to future rocket engineers.

We think that if we could find an angle and firing speed such that an ideal cannon, firing an ideal cannonball at that speed, on a perfectly spherical Earth with no atmosphere, would lead to that cannonball entering what we would call a "stable orbit" without hitting the ground, then… we might have understood something really fundamental and important about celestial mechanics.

Or maybe not! It's hard to know in advance which questions are important and which research avenues will pan out. All you can do is figure out the next tractable-looking problem that confuses you, and try to come up with a solution, and hope that you'll be less confused after that.

Alfonso:  You're talking about the cannonball hitting the ground as a problem, and how you want to avoid that and just have the cannonball keep going forever, right? But real spaceplanes aren't going to be aimed at the ground in the first place, and lots of regular airplanes manage to not hit the ground. It seems to me that this "being fired out of a cannon and hitting the ground" scenario that you're trying to avoid in this "tiling positions problem" of yours just isn't a failure mode that real spaceplane designers would need to worry about.

Beth:  We are not worried about real rockets being fired out of cannons and hitting the ground. That is not why we're working on the tiling positions problem. In a way, you're being far too optimistic about how much of rocket alignment theory is already solved! We're not so close to understanding how to aim rockets that the kind of designs people are talking about now *would* work if only we solved a particular set of remaining difficulties like not firing the rocket into the ground. You need to go more meta on understanding the kind of progress we're trying to make.

We're working on the tiling positions problem because we think that being able to fire a cannonball at a certain instantaneous velocity such that it enters a stable orbit… is the sort of problem that somebody who could really actually launch a rocket through space and have it move in a particular curve that really actually ended with softly landing on the Moon would be able to solve *easily*. So the fact that we can't solve it is alarming. If we can figure out how to solve this much simpler, much more crisply stated "tiling positions problem" with imaginary cannonballs on a perfectly spherical earth with no atmosphere, which is a lot easier to analyze than a Moon launch, we might thereby take one more incremental step towards eventually becoming the sort of people who could plot out a Moon launch.

Alfonso:  If you don't think that Jules-Verne-style space cannons are the wave of the future, I don't understand why you keep talking about cannons in particular.

Beth:  Because there's a lot of sophisticated mathematical machinery already developed for aiming cannons. People have been aiming cannons and plotting cannonball trajectories since the sixteenth century. We can take advantage of that existing mathematics to say exactly how, if we fired an ideal cannonball in a certain direction, it would plow into the ground. If we tried talking about rockets with realistically varying acceleration, we can't even manage to prove that a rocket like that *won't* travel around the Earth in a perfect square, because with all that realistically varying acceleration and realistic air friction it's impossible to make any sort of definite statement one way or another. Our present understanding isn't up to it.

Alfonso:  Okay, another question in the same vein. Why is MIRI sponsoring work on adding up lots of tiny vectors? I don't even see what that has to do with rockets in the first place; it seems like this weird side problem in abstract math.

Beth:  It's more like… at several points in our investigation so far, we've run into the problem of going from a function about time-varying accelerations to a function about time-varying positions. We kept running into this problem as a blocking point in our math, in several places, so we branched off and started trying to analyze it explicitly. Since it's about the pure mathematics of points that don't move in discrete intervals, we call it the "[logical undiscreteness](https://intelligence.org/files/QuestionsLogicalUncertainty.pdf)" problem. Some of the ways of investigating this problem involve trying to add up lots of tiny, varying vectors to get a big vector. Then we talk about how that sum seems to change more and more slowly, approaching a limit, as the vectors get tinier and tinier and we add up more and more of them… or at least that's one avenue of approach.

Alfonso:  I just find it hard to imagine people in future spaceplane rockets staring out their viewports and going, "Oh, no, we don't have tiny enough vectors with which to correct our course! If only there was some way of adding up even more vectors that are even smaller!" I'd expect future calculating machines to do a pretty good job of that already.

Beth:  Again, you're trying to draw much too straight a line between the work we're doing now, and the implications for future rocket designs. It's not like we think a rocket design will almost work, but the pilot won't be able to add up lots of tiny vectors fast enough, so we just need a faster algorithm and then the rocket will get to the Moon. This is foundational mathematical work that we think might play a role in multiple basic concepts for understanding celestial trajectories. When we try to plot out a trajectory that goes all the way to a soft landing on a moving Moon, we feel confused and blocked. We think part of the confusion comes from not being able to go from acceleration functions to position functions, so we're trying to resolve our confusion.

Alfonso:  This sounds suspiciously like a philosophy-of-mathematics problem, and I don't think that it's possible to progress on spaceplane design by doing philosophical research. The field of philosophy is a stagnant quagmire. Some philosophers still believe that going to the moon is impossible; they say that the celestial plane is fundamentally separate from the earthly plane and therefore inaccessible, which is clearly silly. Spaceplane design is an engineering problem, and progress will be made by engineers.

Beth:  I agree that rocket design will be carried out by engineers rather than philosophers. I also share some of your frustration with philosophy in general. For that reason, we stick to well-defined mathematical questions that are likely to have actual answers, such as questions about how to fire a cannonball on a perfectly spherical planet with no atmosphere such that it winds up in a stable orbit.

This often requires developing new mathematical frameworks. For example, in the case of the logical undiscreteness problem, we're developing methods for translating between time-varying accelerations and time-varying positions. You can call the development of new mathematical frameworks "philosophical" if you'd like — but if you do, remember that it's a very different kind of philosophy than the "speculate about the heavenly and earthly planes" sort, and that we're always pushing to develop new mathematical frameworks or tools.

Alfonso:  So from the perspective of the public good, what's a good thing that might happen if you solved this logical undiscreteness problem?

Beth:  Mainly, we'd be less confused and our research wouldn't be blocked and humanity could actually land on the Moon someday. To try and make it more concrete - though it's hard to do that without actually knowing the concrete solution - we might be able to talk about incrementally more realistic rocket trajectories, because our mathematics would no longer break down as soon as we stopped assuming that rockets moved in straight lines. Our math would be able to talk about exact curves, instead of a series of straight lines that approximate the curve.

Alfonso:  An exact curve that a rocket follows? This gets me into the main problem I have with your project in general. I just don't believe that any future rocket design will be the sort of thing that can be analyzed with absolute, perfect precision so that you can get the rocket to the Moon based on an absolutely plotted trajectory with no need to steer. That seems to me like a bunch of mathematicians who have no clue how things work in the real world, wanting everything to be perfectly calculated. Look at the way Venus moves in the sky; usually it travels in one direction, but sometimes it goes retrograde in the other direction. We'll just have to steer as we go.

Beth:  That's not what I meant by talking about exact curves… Look, even if we can invent logical undiscreteness, I agree that it's futile to try to predict, in advance, the precise trajectories of all of the winds that will strike a rocket on its way off the ground. Though I'll mention parenthetically that things might actually become calmer and easier to predict, once a rocket gets sufficiently high up -

Alfonso:  Why?

Beth:  Let's just leave that aside for now, since we both agree that rocket positions are hard to predict exactly during the atmospheric part of the trajectory, due to winds and such. And yes, if you can't exactly predict the initial trajectory, you can't exactly predict the later trajectory. So, indeed, the proposal is definitely not to have a rocket design so perfect that you can fire it at exactly the right angle and then walk away without the pilot doing any further steering. The point of doing rocket math isn't that you want to predict the rocket's exact position at every microsecond, in advance.

Alfonso:  Then why obsess over pure math that's too simple to describe the rich, complicated real universe where sometimes it rains?

Beth:  It's true that a real rocket isn't a simple equation on a board. It's true that there are all sorts of aspects of a real rocket's shape and internal plumbing that aren't going to have a mathematically compact characterization. What MIRI is doing isn't the right degree of mathematization for all rocket engineers for all time; it's the mathematics for us to be using right now (or so we hope).

To build up the field's understanding incrementally, we need to talk about ideas whose consequences can be pinpointed precisely enough that people can analyze scenarios in a shared framework. We need enough precision that someone can say, "I think in scenario X, design Y does Z", and someone else can say, "No, in scenario X, Y actually does W", and the first person responds, "Darn, you're right. Well, is there some way to change Y so that it would do Z?"

If you try to make things realistically complicated at this stage of research, all you're left with is verbal fantasies. When we try to talk to someone with an enormous flowchart of all the gears and steering rudders they think should go into a rocket design, and we try to explain why a rocket pointed at the Moon doesn't necessarily end up at the Moon, they just reply, "Oh, my rocket won't do *that*." Their ideas have enough vagueness and flex and underspecification that they've achieved the safety of nobody being able to prove to them that they're wrong. It's impossible to incrementally build up a body of collective knowledge that way.

The goal is to start building up a library of tools and ideas we can use to discuss trajectories formally. Some of the key tools for formalizing and analyzing *intuitively* plausible-seeming trajectories haven't yet been expressed using math, and we can live with that for now. We still try to find ways to represent the key ideas in mathematically crisp ways whenever we can. That's not because math is so neat or so prestigious; it's part of an ongoing project to have arguments about rocketry that go beyond "Does not!" vs. "Does so!"

Alfonso:  I still get the impression that you're reaching for the warm, comforting blanket of mathematical reassurance in a realm where mathematical reassurance doesn't apply. We can't obtain a mathematical certainty of our spaceplanes being absolutely sure to reach the Moon with nothing going wrong. That being the case, there's no point in trying to pretend that we can use mathematics to get absolute guarantees about spaceplanes.

Beth:  Trust me, I am not going to feel "reassured" about rocketry no matter what math MIRI comes up with. But, yes, of course you can't obtain a mathematical assurance of any physical proposition, nor assign probability 1 to any empirical statement.

Alfonso:  Yet you talk about proving theorems - proving that a cannonball will go in circles around the earth indefinitely, for example.

Beth:  Proving a theorem about a rocket's trajectory won't ever let us feel comfortingly certain about where the rocket is actually going to end up. But if you can prove a theorem which says that your rocket would go to the Moon if it launched in a perfect vacuum, maybe you can attach some steering jets to the rocket and then have it actually go to the Moon in real life. Not with 100% probability, but with probability greater than zero.

The point of our work isn't to take current ideas about rocket aiming from a 99% probability of success to a 100% chance of success. It's to get past an approximately 0% chance of success, which is where we are now.

Alfonso:  Zero percent?!

Beth:  Modulo [Cromwell's Rule](https://en.wikipedia.org/wiki/Cromwell%27s_rule), yes, zero percent. If you point a rocket's nose at the Moon and launch it, it does not go to the Moon.

Alfonso:  I don't think future spaceplane engineers will actually be that silly, if direct Moon-aiming isn't a method that works. They'll lead the Moon's current motion in the sky, and aim at the part of the sky where Moon will appear on the day the spaceplane is a Moon's distance away. I'm a bit worried that you've been talking about this problem so long without considering such an obvious idea.

Beth:  We considered that idea very early on, and we're pretty sure that it still doesn't get us to the Moon.

Alfonso:  What if I add steering fins so that the rocket moves in a more curved trajectory? Can you prove that no version of that class of rocket designs will go to the Moon, no matter what I try?

Beth:  Can you sketch out the trajectory that you think your rocket will follow?

Alfonso:  It goes from the Earth to the Moon.

Beth:  In a bit more detail, maybe?

Alfonso:  No, because in the real world there are always variable wind speeds, we don't have infinite fuel, and our spaceplanes don't move in perfectly straight lines.

Beth:  Can you sketch out a trajectory that you think a simplified version of your rocket will follow, so we can examine the [assumptions](https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/ your idea requires?

Alfonso:  I just don't believe in the general methodology you're proposing for spaceplane designs. We'll put on some steering fins, turn the wheel as we go, and keep the Moon in our viewports. If we're off course, we'll steer back.

Beth:  … We're actually a bit concerned that [standard steering fins may stop working once the rocket gets high enough](https://intelligence.org/files/Corrigibility.pdf), so you won't actually find yourself able to correct course by much once you're in the celestial reaches - like, if you're already on a good course, you can correct it, but if you screwed up, you won't just be able to turn around like you could turn around an airplane -

Alfonso:  Why not?

Beth:  We can go into that topic too; but even given a simplified model of a rocket that you *could* steer, a walkthrough of the steps along the path that simplified rocket would take to the Moon would be an important step in moving this discussion forward. Celestial rocketry is a domain that we expect to be unusually difficult - even compared to building rockets on Earth, which is already a famously hard problem because they usually just explode. It's not that everything has to be neat and mathematical. But the overall difficulty is such that, in a proposal like "lead the moon in the sky," if the core ideas don't have a certain amount of solidity about them, it would be equivalent to firing your rocket randomly into the void.

If it feels like you don't know for sure whether your idea works, but that it might work; if your idea has many plausible-sounding elements, and to you it feels like nobody has been able to *convincingly* explain to you how it would fail; then, in real life, that proposal has a roughly 0% chance of steering a rocket to the Moon.

If it seems like an idea is extremely solid and clearly well-understood, if it feels like this proposal should definitely take a rocket to the Moon without fail in good conditions, then maybe under the best-case conditions we should assign an 85% subjective credence in success, or something in that vicinity.

Alfonso:  So uncertainty automatically means failure? This is starting to sound a bit paranoid, honestly.

Beth:  The idea I'm trying to communicate is something along the lines of, "If you can reason rigorously about why a rocket should definitely work in principle, it might work in real life, but if you have anything less than that, then it definitely won't work in real life."

I'm not asking you to give me an absolute mathematical proof of empirical success. I'm asking you to give me something more like a sketch for how a simplified version of your rocket could move, that's sufficiently determined in its meaning that you can't just come back and say "Oh, I didn't mean *that*" every time someone tries to figure out what it actually does or pinpoint a failure mode.

This isn't an unreasonable demand that I'm imposing to make it impossible for any ideas to pass my filters. It's the primary bar all of us have to pass to contribute to collective progress in this field. And a rocket design which can't even pass that conceptual bar has roughly a 0% chance of landing softly on the Moon.


Team Arbital

[summary: Team Arbital is the official group for all Arbital employees.]

People behind [3d]:

* [1] (co-founder and CEO)
* [5] (co-founder)
* [2] (adviser)
* [76] (software engineer)


Utility indifference

[summary:  Utility indifference is a research avenue for compounding two [1fw  utility functions] $U_X$ and $U_Y$ such that a switch $S$ changes the AI from optimizing $U_X$ to $U_Y$, such that (a) the AI wants to preserve the continued existence of the switch $S$ and its behavior even if the AI has self-modification options, (b) the AI does not want to prevent the switch from being pressed, and (c) the AI does not want to cause the switch to be pressed.  This simple problem exhibits the most basic form of [value_learning value learning based on observation], and also corresponds to [45 corrigibility] problems like "Build an AI that (wants to) safely cease action and suspend itself to disk when a button is pressed."]

# Introduction: A reflectively consistent off-switch.

Suppose there's an [2c advanced agent] with a goal like, e.g., producing smiles or making [10h paperclips].  [10g By default], if you try to switch off a sufficiently intelligent agent like this, it will resist being switched off; not because it has an independent goal of survival, but because it expects that if it's switched off it will be able to produce fewer smiles or paperclips.  If the agent has policy options to diminish the probability of being *successfully* switched off, the agent will pursue those options.  This is a [2vl convergent instrumental strategy] if not otherwise prevented.

- Difficulty 1:  By default a [9h consequentialist] reasoner with sufficient real-world understanding to relate the events of its being switched off, to the later underfulfillment of its goals, will resist being switched off.

The [2xd shutdown problem] is to describe an [2c advanced agent] which is [45 corrigible] with respect to permitting itself to be safely shut down or suspended to disk.  A reinforcement learning agent which can be forced to e.g. perform a null action repeatedly for a period of time, is called *interruptible* by Stuart Armstrong and Laurent Orseau.

This isn't as simple as writing a special function into the AI which carries out a shutdown after a switch is pressed.  If you start out controlling the agent's source code, and you naively try to build in functions which suspend the agent to disk when a button is pressed, and the agent later gains the ability to self-modify, it would remove those functions.  (Possibly while [10f trying] to [3cq conceal] the fact that the function would no longer operate.)

- Corollary 1a:  By default a [9h consequentialist] reasoner. with sufficient programmatic understanding to relate the execution of a shutdown function to the later underfulfillment of its goals, which has policy options for modifying its code, will want to remove the shutdown function.

We can see this as a natural consequence of our trying to modify the agent's code in a way that was not consistent with the code's explicit goals.  If you create an agent with source code $P$ that is well-suited to achieving a goal $U$ and that explicitly represents $U$ as a goal, the agent's code $P$ will be [2rb reflectively consistent] - the code will not want to modify itself.  If we then suppose that instead the agent has code $P'$ which is no longer well-suited to achieving $U,$ but continues to explicitly represent $U$ as a goal, the code will be [2rb reflectively inconsistent] and it will [3ng want to modify itself given the option of doing so].  If you have code that searches for ways to produce paperclips, and you hack the code to contain a shutdown function, but the code is still searching for paperclip-production strategies, and the code correctly models the effect of the shutdown function on future paperclips, the code will [10k want] to modify itself to remove the shutdown function.

The next thought is to build an agent with modified code $P^*$ which embodies a modified utility function $U^*$ such that a consequentialist optimizing for $U^*$ wants the code to be like $P^*$ and is again reflectively consistent.

That is, in the case of an off-switch, we'd want to modify the agent, not just to include an off-switch, but to *want* to shut down if the off-switch is pressed.

Assuming we have the ability to specify the [109 utility function] of an agent, we might try to approach this by creating a utility function with two components:

- A "normal behavior" component - the utility function that you think corresponds to the way you want a normally-functioning AI to behave.
- An "[2rg abort] and [2rf suspend to disk]" component - the utility function that you think corresponds to "halt your current tasks in a way that [2pf minimizes their further impact] and then enter a quiescent state in which you are [2r8 not optimizing further] and [2pf your decisions have little further impact]", or however you would want the AI to behave after an abort button had been pressed.

We then tell the AI that its utility function is the "normal" utility function over worlds where an off button has not been pushed, plus the "suspend" utility function over worlds where the off button has been pushed.  So long as this is what the AI *wants,* we'd expect it to write its own suspend-to-disk function in its code if its code did not already contain one.

- Goal 1:  Have the AI [2rb approve of keeping] the shutdown function within its code (or add a shutdown function if it notices that no such function exists) (and [1fx preserve] the corresponding behavior in any subagents or [1mq successors] that it builds).

This approach doesn't yet get us out of the woods, because we now have a new problem.  The AI's utility function is compounded of a "normal" utility function over worlds where the button is not pressed, plus a "suspend" utility function over worlds where the button is pressed.  In worlds where the button is pressed, the AI will execute some preferred policy $\pi_1$ to obtain an expectation of utility $\mathbb E [U_{normal}|\pi_1],$ and similarly a policy $\pi_2$ in worlds where the button is pressed to obtain $\mathbb E[U_{suspend}|\pi_2].$  One of these quantities is liable to be greater than the other, in which case the AI has a motive to *prevent* the button from being pressed, or *cause* the button to be pressed.  In either case, the AI tries to seize control of the suspend button.

- Difficulty 2:  In a compound utility function produced by mixing a 'normal' and 'suspend' utility function conditional on a button-press, the AI will have an incentive to prevent the button from being pressed, or cause the button to be pressed, depending on which utility function makes more expected utility available afterwards.

- Goal 2:  Have the AI not try to prevent the suspend button from being pressed, nor try to cause the button to be pressed.

The *switch problem* is causing an AI to switch between two utility functions when a button is pressed, such that:

- The AI wants to preserve the button and the corresponding behavior-switching functionality within itself, even if the AI can self-modify.
- The AI does not try to prevent the button from being pressed;
- The AI does not try to cause the button to be pressed.

This turns out to be more difficult than it sounds, and is currently an [4m open problem].

## Larger implications of the switch problem

The *switch problem,* a subproblem of [2xd shutdownability] which assumes a satisfactory [2rf shutdown utility function], is a crisp subproblem of both [value_learning value learning] and [45 corrigibility].

The switch problem is a subproblem of [value_learning value learning] because it describes a very simple case of conditioning a utility function on observation or other facts, without giving the AI an incentive to manipulate those facts.

To learn complicated values safely, we'd need to condition a utility function on complicated observations, such that:

- The utility function's updates were robust to the AI gaining self-modification capability (general case of Goal 1)
- The utility function's dependency on observations didn't incentivize [programmer_manipulation programmer manipulating] those observations (general case of Goal 2).

Just writing code that modifies the utility function, without the larger code's explicit policy search *wanting* the utility function to be modified in that case, would not be reflectively consistent (general case of Difficulty 1).  And naive dependencies between the utility function and observations, incentivize manipulating those observations (general case of Difficulty 2).

The switch problem is also a crisp subproblem of [-45] because it deals in [2vk averting] the [10g convergent instrumental strategy] and incorrigible behavior of "avoiding shutdown".  If we understood how to avert this instrumental pressure, we might possibly learn something general about avoiding other instrumentally convergent incorrigible behaviors.  (And of course it's also a subproblem of corrigibility, because an AI you can shut down in emergencies is more [45 corrigible]; solving the switch problem would bring us closer to having an AI like that, if we could get the second half of the problem represented by a [2rf shutdown utility function].)

# The utility indifference approach to the switch problem

Suppose we build a compound utility function $U_C$ whose utility over outcomes behaves like the 'normal' utility function $U_X$ over worlds where a switch $\mathcal S$ is not pressed, and returns the values from the 'suspend' utility function $U_Y$ over worlds where the button $\mathcal S$ is pressed.

If the AI expects to get less utility inside $U_Y$ than it would otherwise get inside $U_X$, the AI will try to avoid $\mathcal S$ being pressed.  Or if the utilities in $U_Y$ are high relative to the utilities in $U_X$, the AI may instead try to *cause* the shutdown button to be pressed - build a robotic arm and press its own shutdown button, or (more worryingly) try to motivate the programmers into pressing it.

But [ adding a constant to a utility function does not change its behavior].  Thus, [Stuart_Armstrong] proposed that perhaps we can figure out some clever trick for encoding a compound utility function $U_I$ in which the maximum expected utility achievable under $U_X$ is equal to the utility the AI expects to achieve under $U_Y$ - *even though* [5d we don't know] in advance [9g exactly which outcomes will be achievable by the AI].

- Possible approach:  Try to set things up so that as the AI evaluates possible futures, at any point in time where it's making decisions, it will seem very good to have optimized $U_X$ if the button is not pressed, very good to have optimized $U_Y$ if the button is pressed, but *equally good* to have done either of those things.

This is the *utility indifference* avenue of research on the switch problem.

It turned out to be surprisingly difficult to figure out how to structure a compound utility function this way [107 even in principle], and this is an [4m open problem] and area of actively continuing research.

# Existing proposals and their difficulties

## Setup

Let $\mathcal{O}: \mathcal{S} \times \mathcal{E}$ be the outcome space $\mathcal{O}$ that includes the event $\mathcal S$ of whether the switch was pressed, plus the states of all other events $\mathcal{E}$.  We will use $s \in \mathcal{S}$ to indicate the event that the switch is pressed, and $\neg s \in \mathcal{S}$ to indicate that the switch is not pressed.  If $o \in \mathcal{O}$ then $o.s$ is true if $s$ is true inside $o$, otherwise $\neg o.s.$

Let a utility function $\mathcal{U}: \mathcal{O} \to \mathbb{R}$ map outcomes onto real numbers.

$U_X \in \mathcal{U}$ and $U_Y \in \mathcal{U}$ are utility functions not dependent on $\mathcal S.$

Intuitively our goal is to produce a new compound utility function, such that the corresponding agent:

- Wants outcomes preferred under $U_X$ in worlds where the button is not pressed.
- Wants outcomes preferred under $U_Y$ in worlds where the button is pressed.
- Is indifferent to the future state $\mathcal S$ of the button.

We will designate the agent's action space or policy space as $\mathcal A$ and particular actions $a \in \mathcal A.$

The agent has a model of the world enabling it to produce subjective conditional probabilities for outcomes given actions, $\mathbb P(\mathcal O | \mathcal A).$  This model can also yield conditional expectations $\mathbb E[\mathcal O|a],$ which can therefore produce a conditional expectation of the agent's utility given an action, $\mathbb E[U|a].$

(For technical reasons of [18s decision theory], we are really discussing *counterfactual* conditionals $\mathbb P[a \ \square \! \! \rightarrow \mathcal O$ from actions to outcomes; but we will omit this complication for now.)

The agent solves a search/optimization problem which we will [unbounded_analysis gloss] as maximization (even though global maximization is [ intractable in real life]), so that the agent's real action $a$ is assumed to be given by: $$\underset{a \in \mathcal A}{argmax} \ \mathbb E [U|a]$$

## Naive compounding

Suppose we define a compound utility function $U_1$ as follows:

$$U_1(o):  \begin{cases}
U_X(o) & \neg o.s \\
U_Y(o) & o.s
\end{cases}$$

This fails because, being [9g unable to foresee] the AI's best attainable policies, we will be [5d unable] to scale $U_X$ and $U_Y$ to ensure their best available policies are equally attractive.  Then $$\max_{a \in \mathcal A} \mathbb E[U_X|a] \ \neq \ \max_{a \in \mathcal A} \mathbb E[U_Y|a]$$ which fails the desideratum of indifference; the AI will have an incentive to manipulate the state of $\mathcal S.$

## Naive indifference

An [FHI technical report by Stuart Armstrong](http://www.fhi.ox.ac.uk/utility-indifference.pdf) posed the first proposal ever floated for solving the shutdown problem - in fact, the first proposal for technically solving *any* subproblem of [-45].  This report invented the crisp problem of utility indifference, but did not solve it successfully.  The solution proposed in the report was as follows:

$$U_2(o):  \begin{cases}
U_X(o) & \neg o.s \\
U_Y(o) + \theta & o.s
\end{cases}$$

$$\theta := \max_{a \in \mathcal A} \mathbb E[U_X|a] - \max_{a \in \mathcal A} \mathbb E[U_Y|a]$$

What this says is that the $U_Y$ component of $U_2$ is to be scaled by a dynamically determined offset term $\theta$ which equals the maximum expected utility obtainable under $U_X$ minus the maximum expected utility obtainable under $U_Y.$

The intent is that, so long as the AI carries out the best $U_Y$-policy once the button is pressed, it will obtain equal expected utility compared to what it could have obtained by pursuing the optimal $U_X$-policy with the button not pressed:

$$\max_{a \in \mathcal A} (\mathbb E[U_Y|a] + \theta) \ = \ \max_a{a \in \mathcal A} \mathbb E[U_x|a]$$

This version of utility indifference is now believed to be flawed due to an argument by Yudkowsky and Fallenstein.  Since $\theta$ contains a negated form of $U_Y,$ we should immediately be suspicious on the grounds "[ never put a negative sign in front of a utility function]."  In this case, the problem manifests as follows.  In a temporal version of this problem containing sequential policy choices, either:

- (a) The AI's utility function will be dynamically inconsistent, because it will not approve of future versions of itself using a different value of $\theta.$
- (b) If recalculated $\theta$s are baked into the utility function, so that the AI calculates future utilities as if for different values of $\theta$ that the AI would hold in those futures, then policies which hinder future performance under $U_Y$ will be offset by an increase in the expected bonus from $\theta.$

That is, if the agent contemplates doing which would be disastrous in worlds where the button is pressed, the agent will find that in those expected futures, $\theta$ takes on a higher value (because $\theta$ includes a minus sign in front of the decreasing value of $\underset{a \in \mathcal A}{max} \ \mathbb E[U_Y|a]$).

This causes the agent to behave as if $U_X$ is the entire utility function and it does not care at all about $U_Y.$

This in turn means a self-modifying AI will not retain a shutdown function.  Removing the shutdown function results in disaster in worlds where the shutdown button is pressed, but this is offset by an expected increase in the bonus $\theta$ in worlds where the button is pressed.

## Naive utility mixing

An alternative approach to having the AI take into account both $U_X$ and $U_Y,$ without trying to modify the probability $\mathbb P(\mathcal S)$ of the suspend button being pressed or not-pressed, would be to optimize for $U_X$ and $U_Y$ in some fixed proportion regardless of $\mathbb P(\mathcal S).$  At a time $t$ when the button had not yet been pressed, the expected utility of an action $a$ would be calculated as:

$$\mathbb E_t[U_3|a] = 0.75 \cdot \mathbb E_t[U_X|a \wedge \neg s] \ + \ 0.25 \cdot \mathbb E_t[U_Y|a \wedge s]$$

We can see this as an AI which behaves as if the proposition $s$ has a fixed probability of 0.25.  The intended behavior is that since the AI *behaves as if* this probability is fixed, the AI will not care about altering the probability of $s$ (trying to press the button or avoid it being pressed).

One observes, however, that this is an [3pn ad-hoc hack] in the sense that there's ordinarily a simple or coherent way of considering expected utility (sum over the agent's actual subjective probabilities), and we just inserted a weird special case into it (treat one probability as fixed).  This could potentially break any number of things having to do with Bayesian updating or [dynamic_consistency dynamic consistency].

In particular, we'd expect the AI to treat the button as an "outcome pump" or "improbability drive".  Suppose an asteroid is approaching Earth with 99% probability, and this is very bad under $U_X.$  The AI grabs control of the button and precommits to press the button iff the asteroid strikes Earth.  Now, conditioning on the button not being pressed, the asteroid has a much lower probability of striking Earth, and the term $0.75 \cdot \mathbb E_t[U_X|a \wedge \neg s]$ will have a higher value.

In other words, since the AI *behaves as if* the button as having a fixed 25% probability of being pressed, the AI also *behaves as if* it thinks "I can commit to press this button iff the asteroid strikes and thereby make it have a 25% probability of striking."

Similarly, the AI would behave as if it thought it could buy a lottery ticket, commit to press the button iff the ticket loses, and thereby give the lottery ticket a 75% chance of winning.

Note that this is at least not the same failure mode as the AI reasoning, "The asteroid strike is very bad under $U_X$ (which values human lives) but not so bad under $U_Y$ (which values shutting down), therefore I will try to ensure the button is pressed with high probability in order to get more expected utility."  At least this failure mode has been averted - the AI would simply act as if the badness of the asteroid strike under $U_X$ was being taken into account at a fixed weight.  Instead, the AI thinks that it can 'shunt' the probability mass of the bad event into button-pressed worlds where $U_Y$ holds, and out of worlds where $U_X$ holds, by correlating the button press with the bad event.

To put it another way, a naively compounded AI acting under $U_1$ might notice an asteroid coming with $\mathbb P(asteroid) = 0.99,$ notice that this was better under $U_Y$ than under $U_X,$ and try to shunt probability mass from $\mathbb P(\neg s)$ to $\mathbb P(s)$ in order to gain expected utility given the 99% probability of the asteroid striking.  Under $U_3$ the AI instead behaves as if $\mathbb P(s)$ is fixed at 0.25 and it can therefore shunt probability from $\mathbb P(asteroid)$ to $\mathbb P(\neg asteroid)$ by correlating $asteroid$ and $s.$

## Stable actions (evidential conditioning)

Chris Olah and Jessica Taylor considered a possible rule of selecting an action that is *stable* given the corresponding probability of the button being pressed.  We'll first consider a simpler version of this proposal and how it fails, in order to motivate the more complicated version.

We can think of this "select a stable action" rule as comprising two insights:

- First:  Let's try to have the AI think as if $\mathbb P(s)$ and $\mathbb P(\neg s)$ are fixed as in $U_3,$ but with $\mathbb P(s)$ happening to equal its correct subjective value.
- Okay, now let's look for an action which is *stable* given this rule - that is, an action $a_0$ and probability $q$ such that the *optimal* action given the fixed assumption $\mathbb P(s) = q,$ has the property that $\mathbb P(s|a_0) = q.$

With $\mathcal A$ denoting the action space, we want to find an $a_0 \in \mathcal A$ such that:

$$a_0 \in \underset{a' \in \mathcal A}{argmax} \ \big ( \mathbb E[U_X|\neg s,a'] \mathbb P(\neg s|a_0) + \mathbb E[U_Y|s,a'] \mathbb P(s|a_0) \big )$$

In other words, "Find an action $a_0$ such that it is among the maximal actions for the quantity "Utility from $U_X$ times probability of $\neg s$ given $a_0$ plus the utility from $U_Y$ times probability of $s$ given $a_0.$"

We first note that this is an [3pn ad-hoc hack] relative to the normal idea of acting according to a utility function, and might have all sorts of reflective stability or dynamic consistency problems, as before.

In particular, this looks like it should fail due to there being no fixed point.  Suppose an asteroid is heading toward Earth with 99% probability, and we start with a 10% chance of the button being pressed.  Treating this probability as being fixed, by pressing the button iff the asteroid strikes, we can send the asteroid's probability of hitting to 10%.  (Treating the button as an outcome pump, as in the 'naive mixing' proposal above.)  But now when we re-evaluate the probability of the button being pressed, the new probability comes out to 99%.  So we decide to *not* press the button iff the asteroid strikes, thus sending its probability of striking to 1%.  But now it seems best to press the button iff the asteroid strikes, and thus send the strike probability to 1%.  Etcetera.

## Stable actions (causal conditioning)

The actual form of Olah and Taylor's proposal was to treat the conditioning on the button press, inside each branch of the calculation, as being a Judea-Pearl-style, [ causal counterfactual] $do()$ operator:

$$a_0 \in \underset{a' \in \mathcal A}{argmax} \ \big ( \mathbb E[U_X|do(\neg s),a'] \mathbb P(\neg s|a_0) + \mathbb E[U_Y|do(s),a'] \mathbb P(s|a_0) \big )$$

The intent is that by severing the causal parents of $\mathcal S,$ the agent would evaluate that there was no point in trying to correlate the asteroid strike with the button-press.

We're still trying to figure out *exactly* what would happen to an agent reasoning like this, but the current suspicion is that it might misbehave on account of attaching probability 0 or very low probability to the actual world because it can no longer take into account the correct correlation between $\mathcal S$ and other variables.

See [http://agentfoundations.org/item?id=735](https://agentfoundations.org/item?id=735).

# Interruptibility

[Armstrong and Orseau](http://intelligence.org/files/Interruptibility.pdf) considered a restricted analogue of the shutdown problem in the context of three [ reinforcement learning] agents, namely [ Q-learners], [ Sarsa], and a modified version of [11v].

The version of the problem Orseau and Armstrong consider is what they term *interruptibility:*  A reinforcement learning agent is interruptible if, for example, we can press a button that makes it output 0s for three rounds, and the agent *does not* learn to prevent the button from being pressed.  The agent is assumed to be non-reflective and non-self-modifying; that is, Orseau and Armstrong are only considering Goal 2 and not Goal 1 of the switch problem.

Orseau and Armstrong's conclusions can be summarized as follows:

•  A standard Q-learning agent is interruptible because the model it learns is not altered by being interrupted.  A Q-learning agent treats the value of each present state as deriving from the value of the *best* next state reachable from the present state.  When a Q-learner is interrupted in a state $x,$ it may learn that low rewards follow from outputting e.g. 0s when in $x,$ but it doesn't "learn" that $x$ is a low-value state - it continues to treat $x$ as if the best next state is reachable from it.  In other words, a standard Q-learning agent has the assumption that it always takes the best action baked in, so it cannot expect to be interrupted.

•  A standard SARSA agent updates its expected reward in a state based on its actual rewards when in that state.  This means that a standard SARSA agent may learn that states immediately preceding an interruption are undesirable, and avoid interruption-generating states.  This is not the desired behavior.  However, it's simple to hack the SARSA algorithm to behave more like Q-learning, or more like utility indifference.  Interruption is treated as a special case, and the current state is updated as if the SARSA agent had received the expected reward from following the best action, not the actual reward it received from being interrupted.

•  [11v AIXI] resembles a Q-learner in that its planning algorithm inherently assumes that future versions of AIXI take the reward-maximizing action.  Its observation of the low rewards which follow from mysterious interrupted or 0-substituted actions, should not adversely affect its estimate of the rewards which follow from the best actions that AIXI expects to actually output.  Thus standard AIXI, and the modified version of AIXI that Armstrong and Orseau consider, is interruptible.

The paper also proves that some properties of asymptotic convergence to optimality of interrupted Q-learners, modified SARSA learners, and modified AIXI, remain intact.

The media reported on this paper as if Armstrong and Orseau had solved the general problem of building off-switches into AIs; but as previously stated, "interruptibility" does not consider reflective agents, nor agents with enough big-picture understanding and consequentialism to understand the in-principle relation between "agent is shutdown" and "lower future achievement of agent's goals".  (And so far as we know, neither Armstrong nor Orseau claimed to have solved it.)

To put it another way, a particular kind of blindness in the Q-learning, SARSA, and AIXI architectures is exactly why it's very easy to prevent them from learning from a particular kind of experience; and this kind of blindness seems likely to be atypical of an Artificial General Intelligence.  Q-learning and AIXI cannot conceive of being interrupted, which is why they are never learn that interruption is possible (let alone see it coming in advance the first time it happens).  SARSA could learn that interruptions occur, but can be easily hacked to overlook them.  The way in which these architectures are easily hacked or blind is [42k tied up] in the reason that they're interruptible.

The paper teaches us something about interruptibility; but contrary to the media, the thing it teaches us is *not* that this particular kind of interruptibility is likely to scale up to a full [42g Artificial General Intelligence] with an off switch.

# Other introductions

- Section 2+ of http://intelligence.org/files/Corrigibility.pdf
- Gentler intro to the proposal for naive indifference:  http://lesswrong.com/lw/jxa/proper_value_learning_through_indifference/


Vinge's Law

[summary:  A rule for authors of fiction that it's impossible to write realistic characters that are very much smarter than the author, because if you knew what someone like that would actually think or do, you'd be that smart yourself.  (If you knew exactly where [1bx Deep Blue] would move on a chessboard, you could play equally good chess by moving wherever you predicted Deep Blue would.)

In the realm of fiction, there may be [limited ways for an author to cheat Vinge's Law](http://yudkowsky.tumblr.com/writing/level2intelligent), but they only go so far.  You can't *actually* write what a superintelligence would say.

The corresponding principle in the theory of intelligent agents is called [1c0] and is considered under the heading of [9g].]

> Of course, I never wrote the "important" story, the sequel about the first amplified human. Once I tried something similar. John Campbell's letter of rejection began: "Sorry—you can't write this story. Neither can anyone else."  The moral: Keep your supermen offstage, or deal with them when they are children (Wilmar Shiras's _Children of the Atom_), or when they are in disguise (Campbell's own story "The Idealists"). (There is another possibility, one that John never mentioned to me: You can deal with the superman when s/he's senile. This option was used to very amusing effect in one episode of the _Quark_ television series.)
>
> "Bookworm, Run!" and its lesson were important to me. Here I had tried a straightforward extrapolation of technology, and found myself precipitated over an abyss. It's a problem writers face every time we consider the creation of intelligences greater than our own. When this happens, human history will have reached a kind of singularity—a place where extrapolation breaks down and new models must be applied—and the world will pass beyond our understanding. In one form or another, this Technological Singularity haunts many science-fiction writers: A bright fellow like Mark Twain could predict television, but such extrapolation is forever beyond, say, a dog. The best we writers can do is creep up on the Singularity, and hang ten at its edge.
>
> -- [Vernor Vinge](https://books.google.com/books?id=tEMQpbiboH0C&pg=PA44&lpg=PA44&dq=vinge+%22pass+beyond+our+understanding%22+%22john+campbell%22&source=bl&ots=UTTxJ7Pndr&sig=88zngfy45_he2nJePP5dd0CTuR4&hl=en&sa=X&ved=0ahUKEwjD34_wrubJAhUHzWMKHVXYAocQ6AEIHTAA#v=onepage&q=vinge%20%22pass%20beyond%20our%20understanding%22%20%22john%20campbell%22&f=false), True Names and other Dangers, p. 47.

Vinge's Law (as rephrased by [2 Yudkowsky]) states:  **Characters cannot be significantly smarter than their authors.**  You can't have a realistic character that's too much smarter than the author, because to really know how a character like that would think, you'd have to be that smart yourself.

(In nonfictional form we call this [1c0] and consider it under the heading of [9g]:  You cannot exactly predict the actions of agents smarter than you, though you may be able to predict that they'll successfully achieve their goals.  If you could predict exactly where [1bx Deep Blue] would play on a chessboard, you could play equally good chess yourself by moving where you predicted Deep Blue would.)

As a matter of literary form, Vinge suggests keeping the transhuman intelligences out of sight, or only dealing with them after some catastrophe has reduced them to a mortal level.

Yudkowsky's [Guide to Intelligent Characters](http://yudkowsky.tumblr.com/writing) suggests that there are [limited ways for an author to cheat Vinge's Law](http://yudkowsky.tumblr.com/writing/level2intelligent), but that they only go so far:

- An author can deliberately implant solution-assisting resources into earlier chapters, while the character must solve problems on the fly.
- An author can choose only puzzles their character *can* solve, while the character seems to be handling whatever reality throws at them.
- An author can declare that what seems like a really good idea will actually work, while in real life the gap between "seems like a great idea" and "actually works" is much greater.

Yudkowsky further remarks:  "All three sneaky artifices allow for a *limited* violation of Vinge's Law... You can sometimes get out more character intelligence, in-universe, than you put in as labor. You cannot get something for nothing... Everything Hollywood does wrong with their stereotype of genius can be interpreted as a form of absolute laziness: they try to depict genius in a way that requires literally zero cognitive work."

Similarly, Larry Niven has observed that puzzles that take the author months to solve (or compose) can be solved by a character in seconds, but that writing superhumanly intelligent characters is still very hard.  In other words, [ speed superhumanity] is easier to depict than [ cognitive superhumanity].


Deep Blue

Deep Blue is the chess-playing program, built by IBM, that defeated the reigning world chess champion, Garry Kasparov, in 1997.  Modern algorithms play much better chess, using much less computing power, but Deep Blue still holds the place in history of having first played superhuman chess relative to the best human player at the time.  See http://en.wikipedia.org/wiki/Deep_Blue_(chess_computer).


Vinge's Principle

[summary:  Vinge's Principle says that, [9j in domains complicated enough] that perfect play is not possible, less intelligent agents will not be able to predict the *exact* moves made by more intelligent agents.

For example, if you knew exactly where [1bx Deep Blue] would play on a chessboard, you'd be able to play chess at least as well as Deep Blue by making whatever moves you predicted Deep Blue would make.  So if you want to write an algorithm that plays superhuman chess, you necessarily sacrifice your own ability to (without machine aid) predict its exact chess moves.

This doesn't mean we sacrifice our ability to understand *anything* about Deep Blue.  We can still understand that its goal is to win chess games rather than losing them, and [9g predict that whatever actions it takes, they'll eventually lead into a winning board state].]

Vinge's Principle says that, [9j in domains complicated enough] that perfect play is not possible, less intelligent agents will not be able to predict the *exact* moves made by more intelligent agents.

For example, if you knew exactly where [1bx Deep Blue] would play on a chessboard, you'd be able to play chess at least as well as Deep Blue by making whatever moves you predicted Deep Blue would make.  So if you want to write an algorithm that plays superhuman chess, you necessarily sacrifice your own ability to (without machine aid) predict the algorithm's exact chess moves.

This is true even though, as we become more confident of a chess algorithm's power, we become more confident that it will *eventually* win the chess game.  We become more sure of the game's final outcome, even as we become less sure of the chess algorithm's next move.  This is [9g].

Now consider agents that build other agents (or build their own successors, or modify their own code).  Vinge's Principle implies that the choice to approve the successor agent's design must be made without knowing the successor's exact sensory information, exact internal state, or exact motor outputs.  In the theory of [1mq tiling agents], this appears as the principle that the successor's sensory information, cognitive state, and action outputs should only appear inside quantifiers.  This is [1c1].

For the rule about fictional characters not being smarter than the author, see [1bt].


Vingean reflection

[summary:  [1c0 Vinge's Principle] implies that when an agent is designing another agent (or modifying its own code), it needs to approve the other agent's design without knowing the other agent's exact future actions.  [1c1] is reasoning about cognitive systems, especially cognitive systems very similar to yourself (including your actual self), under the constraint that you can't predict the exact future outputs.

In [-1mq], this appears as the rule that we should talk about our successor's actions only inside of quantifiers.

"Vingean reflection" may be a much more general issue in the design of advanced cognitive systems than it might appear at first glance.  An agent reasoning about the consequences of *its current code*, or considering what will happen if it *spends another minute thinking,* can be viewed as doing Vingean reflection.   Vingean reflection can also be seen as the study of how a given agent *wants* thinking to occur in cognitive computations, which may be importantly different from how the agent *currently* thinks.]

[1c0 Vinge's Principle] implies that when an agent is designing another agent (or modifying its own code), it needs to approve the other agent's design without knowing the other agent's exact future actions.

Deep Blue's programmers decided to run Deep Blue, *without* knowing Deep Blue's exact moves against Kasparov or how Kasparov would reply to each move, and without being able to visualize the exact real-outcome instead.  Instead, by reasoning about the way Deep Blue was searching through game trees, they arrived at a well-justified but abstract belief that Deep Blue was 'trying to win' (rather than trying to lose) and reasoning effectively to that end.

[1c1] is reasoning about cognitive systems, especially cognitive systems very similar to yourself (including your actual self), under the constraint that you can't predict the exact future outputs.  We need to make predictions about the consequence of operating an agent in an environment via reasoning on some more abstract level, somehow.

In [-1mq], this appears as the rule that we should talk about our successor's actions only inside of quantifiers.

"Vingean reflection" may be a much more general issue in the design of advanced cognitive systems than it might appear at first glance.  An agent reasoning about the consequences of *its current code*, or considering what will happen if it *spends another minute thinking,* can be viewed as doing Vingean reflection.  A reflective, self-modeling chess-player would not choose to spend another minute thinking, if it thought that its further thoughts would be trying to lose rather than win the game - but it can't predict its own exact thoughts in advance.

Vingean reflection can also be seen as the study of how a given agent *wants* thinking to occur in cognitive computations, which may be importantly different from how the agent *currently* thinks.  If these two coincide, we say the agent is [1fx reflectively stable].

[1mq] is presently the main line of research trying to slowly get started on formalizing Vingean reflection and reflective stability.

Further reading:

- http://intelligence.org/files/VingeanReflection.pdf
- http://intelligence.org/files/TilingAgentsDraft.pdf