Extrapolated volition (normative moral theory)

[summary:  The notion that some act or policy "can in fact be wrong", even when you "think it is right", is more intuitive to some people than others; and it raises the question of what "rightness" is and [112 how to compute it].

On the extrapolated volition theory of metaethics, if you would change your mind about something after learning new facts or considering new arguments, then your updated state of mind is righter.  This can be true in advance of you knowing the facts.

E.g., maybe you currently want revenge on the Capulet family.  But if somebody had a chance to sit down with you and have a long talk about how revenge affects civilizations in the long run, you could be talked out of thinking that revenge, in general, is right.  So long as this *might* be true, it makes sense to say, "I want revenge on the Capulets, but maybe that's not really right."

Extrapolated volition is a normative moral theory.  It is a theory of how the concept of *shouldness* or *goodness* is or ought to be cashed out ([3y6 rescued]).  The corresponding proposal for [41k completely aligning] a [1g3 fully self-directed superintelligence] is [3c5 coherent extrapolated volition].]

(This page is about extrapolated volition as a normative moral theory - that is, the theory that extrapolated volition captures the concept of [55 value] or what outcomes we *should* want.  For the closely related proposal about what a sufficiently advanced [1g3 self-directed] AGI should be built to want/target/decide/do, see [3c5 coherent extrapolated volition].)

# Concept

Extrapolated volition is the notion that when we ask "What is right?", then insofar as we're asking something meaningful, we're asking about the result of running a certain logical function over possible states of the world, where this function is analytically identical to the result of *extrapolating* our current decision-making process in directions such as "What if I knew more?", "What if I had time to consider more arguments (so long as the arguments weren't hacking my brain)?", or "What if I understood myself better and had more self-control?"

A simple example of extrapolated volition might be to consider somebody who asks you to bring them orange juice from the refrigerator.  You open the refrigerator and see no orange juice, but there's lemonade.  You imagine that your friend would want you to bring them lemonade if they knew everything you knew about the refrigerator, so you bring them lemonade instead.  On an abstract level, we can say that you "extrapolated" your friend's "volition", in other words, you took your model of their mind and decision process, or your model of their "volition", and you imagined a counterfactual version of their mind that had better information about the contents of your refrigerator, thereby "extrapolating" this volition.

Having better information isn't the only way that a decision process can be extrapolated; we can also, for example, imagine that a mind has more time in which to consider moral arguments, or better knowledge of itself.  Maybe you currently want revenge on the Capulet family, but if somebody had a chance to sit down with you and have a long talk about how revenge affects civilizations in the long run, you could be talked out of that.  Maybe you're currently convinced that you advocate for green shoes to be outlawed out of the goodness of your heart, but if you could actually see a printout of all of your own emotions at work, you'd see there was a lot of bitterness directed at people who wear green shoes, and this would change your mind about your decision.

In Yudkowsky's version of extrapolated volition considered on an individual level, the three core directions of extrapolation are:

- Increased knowledge - having more veridical knowledge of declarative facts and expected outcomes.
- Increased consideration of arguments - being able to consider more possible arguments and assess their validity.
- Increased reflectivity - greater knowledge about the self, and to some degree, greater self-control (though this raises further questions about which parts of the self normatively get to control which other parts).

# Motivation

Different people react differently to the question "Where *should* we point an [1g3 autonomous] superintelligence, if we can point it exactly?" and approach it from different angles. [todo: and we'll eventually need an Arbital dispatching questionnaire on a page that handles it] These angles include:

- All this talk of 'shouldness' is just a cover for the fact that whoever gets to build the superintelligence wins all the marbles; no matter what you do with your superintelligence, you'll be the one who does it.
- What if we tell the superintelligence what to do and it's the wrong thing?  What if we're basically confused about what's right?  Shouldn't we let the superintelligence figure that out on its own with its own superior intelligence?
- Imagine the Ancient Greeks telling a superintelligence what to do.  They'd have told it to optimize personal virtues, including, say, a glorious death in battle.  This seems like a bad thing and we need to figure out how not to do the analogous thing.  So telling an AGI to do what seems like a good idea to us will also end up seeming a very regrettable decision a million years later.
- Obviously we should just tell the AGI to optimize liberal democratic values.  Liberal democratic values are good.  The real threat is if bad people get their hands on AGI and build an AGI that doesn't optimize liberal democratic values.

Some corresponding initial replies might be:

- Okay, but suppose you're a programmer and you're trying *not to be a jerk.*  If you're like, "Well, whatever I do originates in myself and is therefore equally selfish, so I might as well declare myself God-Emperor of Earth," you're being a jerk.  Is there anything we can do which is less jerky, and indeed, minimally jerky?
- If you say you have no information at all about what's 'right', then what does the term even mean?  If I might as well have my AGI maximize paperclips and you have no ground on which to stand and say that's the wrong way to compute normativity, then what are we even talking about in the first place?  The word 'right' or 'should' must have some meaning that you know about, even if it doesn't automatically print out a list of everything you know is right.  Let's talk about hunting down that meaning.
- Okay, so what should the Ancient Greeks have done if they did have to program an AI?  How could they *not* have doomed future generations?  Suppose the Ancient Greeks are clever enough to have noticed that sometimes people change their minds about things and to realize that they might not be right about everything.  How can they use the cleverness of the AGI in a constructively specified, computable fashion that gets them out of this hole?  You can't just tell the AGI to compute what's 'right', you need to put an actual computable question in there, not a word.
- What if you would, after some further discussion, want to tweak your definition of "liberal democratic values" just a little?  What if it's *predictable* that you would do that?  Would you really want to be stuck with your off-the-cuff definition a million years later?

Arguendo by [3c5 CEV]'s advocates, these conversations eventually all end up converging on [3c5 Coherent Extrapolated Volition] as an alignment proposal by different roads.

"Extrapolated volition" is the corresponding normative theory that you arrive at by questioning the meaning of 'right' or trying to figure out what we 'should' really truly do.

# EV as [3y6 rescuing] the notion of betterness

We can see EV as trying to **[3y6 rescue]** the following pretheoretic intuitions (as they might be experienced by someone feeling confused, or just somebody who'd never questioned metaethics in the first place):

- (a)  It's possible to think that something is right, and be incorrect.
   - (a1)  It's possible for something to be wrong even if nobody knows that it's wrong.  E.g. an uneven division of an apple pie might be unfair even if all recipients don't realize this.
   - (a2)  We can learn more about what's right, and change our minds to be righter.
- (b)  Taking a pill that changes what you think is right, should not change what *is* right.  (If you're contemplating taking a pill that makes you think it's right to secretly murder 12-year-olds, you should not reason, "Well, if I take this pill I'll murder 12-year-olds... but also it *will* be all right to murder 12-year-olds, so this is a great pill to take.")
- (c)  We could be wrong, but it sure *seems* like the things on [41r Frankena's list] are all reasonably good.  ("Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc...")
   - (c1)  The fact that we could be in some mysterious way "wrong" about what belongs on Frankena's list, doesn't seem to leave enough room for "[10h make as many paperclips as possible]" to be the only thing on the list.  Even our state of confusion and possible ignorance doesn't seem to allow for that to be the answer.  We're at least pretty sure that *isn't* the total sum of goodness.
   - (c2)  Similarly, on the meta-level, it doesn't seem like the meta-level procedure "Pick whatever procedure for determining rightness, leads to the most paperclips existing after you adopt it" could be the correct answer.

We *cannot* [3y6 rescue] these properties by saying:

"There is an irreducible, non-natural 'rightness' XML tag attached to some objects and events.  Our brains perceive this XML tag, but imperfectly, giving us property (a) when we think the XML tag is there, even though it isn't.  The XML tags are there even if nobody sees them (a1).  Sometimes we stare harder and see the XML tag better (a2).  Obviously, doing anything to a brain isn't going to change the XML tag (b), just fool the brain or invalidate its map of the XML tag.  All of the things on Frankena's list have XML tags (c) or at least we think so.  For paperclips to be the total correct content of Frankena's list, we'd need to be wrong about paperclips not having XML tags *and* wrong about everything on Frankena's list that we think *does* have an XML tag (c1).  And on the meta-level, "Which sense of rightness leads to the most paperclips?" doesn't say anything *about* XML tags, and it doesn't lead *to* there being lots of XML tags, so there's no justification for it (c2)."

This doesn't work because:

- There are, [112 in fact], no tiny irreducible XML tags attached to objects.
- If there were little tags like that, there'd be no obvious normative justification for our caring about them.
- It doesn't seem like we should be able to make it good to murder 12-year-olds by swapping around the irreducible XML tags on the event.
- There's no way our brains could perceive these tiny XML tags even if they were there.
- There's no obvious causal story for how humans could have evolved such that we do in fact care about these tiny XML tags.  (A [3y9 descriptive rather than normative] problem with the theory as a whole; natural selection has no normative force or justificational power, but we *do* need our theory of how brains actually work to be compatible with it).

Onto what sort of entity can we then map our intuitions, if not onto tiny XML tags?

Consider the property of sixness possessed by six apples on a table.  The relation between the physical six apples on a table, and the logical number '6', is given by a logical function that takes physical descriptions as inputs: in particular, the function "count the number of apples on the table".

Could we rescue 'rightness' onto a logical function like this, only much more complicated?

Let's examine how the 6-ness property and the "counting apples" function behave:

- There are, in fact, no tiny tags saying '6' attached to the apples (and yet there are still six of them).
- It's possible to think there are 6 apples on the table, and be wrong.
- We can sometimes change our minds about how many apples there are on a table.
- There can be 6 apples on a table even if nobody is looking at it.
- Taking a pill that changes how many apples you think are on the table, doesn't change the number of apples on the table.
- You can't have a 6-tag-manipulator that changes the number of apples on a table without changing anything about the table or apples.
- There's a clear causal story for how we can see apples, and also for how our brains can count things, and there's an understandable historical fact about *why* humans count things.
- Changing the history of how humans count things could change *which* logical function our brains were computing on the table, so that our brains were no longer "counting apples", but it wouldn't change the number of apples on the table.  We'd be changing *which* logical function our brains were considering, not changing the logical facts themselves or making it so that identical premises would lead to different conclusions.
- Suppose somebody says, "Hey, you know, sometimes we're wrong about whether there's 6 of something or not, maybe we're just entirely confused about this counting thing; maybe the real number of apples on this table is this paperclip I'm holding."  Even if you often made mistakes in counting, didn't know how to axiomatize arithmetic, and were feeling confused about the nature of numbers, you would still know *enough* about what you were talking about to feel pretty sure that the number of apples on the table was not in fact a paperclip.
- If you could ask a superintelligence how many grains of sand your brain would think there were on a beach, in the limit of your brain representing everything the superintelligence knew and thinking very quickly, you would indeed gain veridical knowledge about the number of grains of sand on that beach.  Your brain doesn't determine the number of grains of sand on the beach, and you can't change the logical properties of first-order arithmetic by taking a pill that changes your brain.  But there's an *analytic* relation between the procedure your brain currently represents and tries to carry out in an error-prone way, and the logical function that counts how many grains of sand on the beach.

This suggests that 6-ness has the correct ontological nature for some much bigger and more complicated logical function than "Count the number of apples on the table" to be outputting rightness.  Or rather, if we want to [3y6 rescue our pretheoretic sense] of rightness in a way that [3y6 adds up to moral normality], we should rescue it onto a logical function.

This function, e.g., starts with the items on Frankena's list and everything we currently value; but also takes into account the set of arguments that might change our mind about what goes on the list; and also takes into account meta-level conditions that we would endorse as distinguishing "valid arguments" and "arguments that merely change our minds".  (This last point is pragmatically important if we're considering trying to get a [2c superintelligence] to [3c5 extrapolate our volitions].  The list of everything that *does in fact change your mind* might include particular patterns of rotating spiral pixel patterns that effectively hack a human brain.)

The end result of all this work is that we go on guessing which acts are right and wrong as before, go on considering that some possible valid arguments might change our minds, go on weighing such arguments, and go on valuing the things on [41r Frankena's list] in the meantime.  The theory as a whole is intended to [3y6 add up to the same moral normality as before], just with that normality embedded into the world of causality and logic in a non-confusing way.

One point we could have taken into our starting list of important properties, but deferred until later:

- It sure *feels* like there's a beautiful, mysterious floating 'rightness' property of things that are right, and that the things that have this property are terribly precious and important.

On the general program of "[3y6 rescuing the utility function]", we should not scorn this feeling, and should instead figure out how to map it onto what actually exists.

In this case, having preserved almost all the *structural* properties of moral normality, there's no reason why anything should change about how we experience the corresponding emotion in everyday life.  If our native emotions are having trouble with this new, weird, abstract, learned representation of 'a certain big complicated logical function', we should do our best to remember that the rightness is still there.  And this is not a retreat to second-best any more than "disordered kinetic energy" is some kind of sad consolation prize for [3y6 the universe's lack of ontologically basic warmth], etcetera.

## Unrescuability of moral internalism

In standard metaethical terms, we have managed to rescue 'moral cognitivism' (statements about rightness have truth-values) and 'moral realism' (there is a fact of the matter out there about how right something is).  We have *not* however managed to rescue the pretheoretic intuition underlying 'moral internalism':

- A moral argument, to be valid, ought to be able to persuade anyone.  If a moral argument is unpersuasive to someone who isn't making some kind of clear mistake in rejecting it, then that argument must rest on some appeal to a private or merely selfish consideration that should form no part of true morality that everyone can perceive.

This intuition cannot be preserved in any reasonable way, because [10h paperclip maximizers] are in fact going to go on making paperclips (and not because they made some kind of cognitive error).  A paperclip maximizer isn't disagreeing with you about what's right (the output of the logical function), it's just following whatever plan leads to the most paperclips.

Since the paperclip maximizer's policy isn't influenced by any of our moral arguments, we can't preserve the internalist intuition without reducing the set of valid justifications and truly valuable things to the empty set - and even *that,* a paperclip maximizer wouldn't find motivationally persuasive!

Thus our options regarding the pretheoretic internalist intuition that a moral argument is not valid if not universally persuasive, seem to be limited to the following:

 1. Give up on the intuition in its intuitive form: a paperclip maximizer doesn't care if it's unjust to kill everyone; and you can't talk it into behaving differently; and this doesn't reflect a cognitive stumble on the paperclip maximizer's part; and this fact gives us no information about what is right or justified.
 2. Preserve, at the cost of all other pretheoretic intuitions about rightness, the intuition that only arguments that universally influence behavior are valid: that is, there are no valid moral arguments.
 3. Try to sweep the problem under the rug by claiming that *reasonable* minds must agree that paperclips are objectively pointless... even though Clippy is not suffering from any defect of epistemic or instrumental power, and there's no place in Clippy's code where we can point to some inherently persuasive argument being dropped by a defect or special case of that code.

It's not clear what the point of stance (2) would be, since even this is not an argument that would cause Clippy to alter its behavior, and hence the stance is self-defeating.  (3) seems like a mere word game, and potentially a *very dangerous* word game if it tricks AI developers into thinking that rightness is a default behavior of AIs, or even a function of low [5v algorithmic complexity], or that [3d9 beneficial] behavior automatically correlates with 'reasonable' judgments about less [36h value-laden] questions.  See "[1y]" for the extreme practical importance of acknowledging that moral internalism is in practice false.

# Situating EV in contemporary metaethics

[41n Metaethics] is the field of academic philosophy that deals with the question, not of "What is good?", but "What sort of property is goodness?"  As applied to issues in Artificial Intelligence, rather than arguing over which particular outcomes are better or worse, we are, from a standpoint of [112 executable philosophy], asking how to *compute* what is good; and why the output of any proposed computation ought to be identified with the notion of shouldness.

EV replies that for each person at a single moment in time, *right* or *should* is to be identified with a (subjectively uncertain) logical constant that is fixed for that person at that particular moment in time, where this logical constant is to be identified with the result of running the extrapolation process on that person.  We can't run the extrapolation process so we can't get perfect knowledge of this logical constant, and will be subjectively uncertain about what is right.

To eliminate one important ambiguity in how this might cash out, we regard this logical constant as being *analytically identified* with the extrapolation of our brains, but not *counterfactually dependent* on counterfactually varying forms of our brains.  If you imagine being administered a pill that makes you want to kill people, then you shouldn't compute in your imagination that different things are right for this new self.  Instead, this new self now wants to do something other than what is right.  We can meaningfully say, "Even if I (a counterfactual version of me) wanted to kill people, that wouldn't make it right" because the counterfactual alteration of the self doesn't change the logical object that you mean by saying 'right'.

However, there's still an *analytic* relation between this logical object and your *actual* mindstate, which is indeed is implied by the very meaning of discourse about shouldness, which means that you *can* get veridical information about this logical object by having a sufficiently intelligent AI run an approximation of the extrapolation process over a good model of your actual mind.  If a sufficiently intelligent and trustworthy AGI tells you that after thinking about it for a while you wouldn't want to eat cows, you have gained veridical information about whether it's right to eat cows.

Within the standard terminology of academic metaethics, "extrapolated volition" as a normative theory is:

- Cognitivist.  Normative propositions can be true or false.  You can believe that something is right and be mistaken.
- Naturalist.  Normative propositions are *not* irreducible or based on non-natural properties of the world.
- Externalist / not internalist.  It is not the case that all sufficiently powerful optimizers must act on what we consider to be moral propositions.  A paperclipper does what is clippy, not what is right, and the fact that it's trying to turn everything into paperclips does not indicate a disagreement with you about what is *right* any more than you disagree about what is clippy.
- Reductionist.  The whole point of this theory is that it's the sort of thing you could potentially compute.
- More synthetic reductionist than analytic reductionist.  We don't have *a priori* knowledge of our starting mindstate and don't have enough computing power to complete the extrapolation process over it.  Therefore, we can't figure out exactly what our extrapolated volition would say just by pondering the meaning of the word 'right'.

Closest antecedents in academic metaethics are Rawls and Goodman's [reflective equilibrium](https://en.wikipedia.org/wiki/Reflective_equilibrium), Harsanyi and Railton's [ideal advisor](https://intelligence.org/files/IdealAdvisorTheories.pdf) theories, and Frank Jackson's [moral functionalism](http://plato.stanford.edu/entries/naturalism-moral/#JacMorFun).

## Moore's Open Question

*Argument.*  If extrapolated volition is analytically equivalent to good, then the question "Is it true that extrapolated volition is good?" is meaningless or trivial.  However, this question is not meaningless or trivial, and seems to have an open quality about it.  Therefore, extrapolated volition is not analytically equivalent to goodness.

*Reply.*  Extrapolated volition is not supposed to be *transparently* identical to goodness.  The normative identity between extrapolated volition and goodness is allowed to be something that you would have to think for a while and consider many arguments to perceive.

Natively, human beings don't start out with any kind of explicit commitment to a particular metaethics; our brains just compute a feeling of rightness about certain acts, and then sometimes update and say that acts we previously thought were right are not-right.

When we go from that, to trying to draw a corresponding logical function that we can see our brains as approximating, and updating when we learn new things or consider new arguments, we are carrying out a project of "[3y6 rescuing the utility function]".  We are reasoning that we can best rescue our native state of confusion by seeing our reasoning about goodness as having its referent in certain logical facts, which lets us go on saying that it is better ceteris paribus for people to be happy than in severe pain, and that we can't reverse this ordering by taking a pill that alters our brain (we can only make our future self act on different logical questions), etcetera.  It's not surprising if this bit of philosophy takes longer than five minutes to reason through.


Value-laden

[summary:  A category, estimate, or fact is *value-laden* when it passes through human goals and desires, so that an AI couldn't compute the [6h intended meaning] without understanding [5l complicated human values].

For instance, if you tell an AI to "cure cancer without bad side effects", the notion of what is a bad side effect is value-laden.  Even the notion of "important side effect" is value-laden, since by "important side effect" you really mean "a side effect I care about" or "a side effect that will perturb [55 value]".  It might be simpler to give the AI a category of "[2pf all side effects]" to be minimized or checked with the user.

In terms of [ Hume's is-ought type distinction], value-laden categories are those that humans compute using information from the ought side of the boundary.  This means value-laden categories rely on our [2fs Humean degrees of freedom] and therefore have high [-5v] and aren't automatically learned easily or induced as simple even by very advanced AIs.]

An intuitive human category, or other humanly intuitive quantity or fact, is *value-laden* when it passes through human goals and desires, such that an agent couldn't reliably determine this intuitive category or quantity without knowing lots of complicated information about human goals and desires (and how to apply them to arrive at the intended concept).

In terms of [ Hume's is-ought type distinction], value-laden categories are those that humans compute using information from the ought side of the boundary, whether or not they notice they are doing so.

# Examples

## Impact vs. important impact

Suppose we want an AI to cure cancer, without this causing any *important side effects.*  What is or isn't an "important side effect" depends on what you consider "important".  If the cancer cure causes the level of thyroid-stimulating hormone to increase by 5%, this probably isn't very important.  If the cure increases the user's serotonin level by 5% and this significantly changes the user's emotional state, we'd probably consider that quite important.  But unless the AI already understands complicated human values, it doesn't necessarily have any way of knowing that one change in blood chemical levels is "not important" and the other is "important".

If you imagine the cancer cure as disturbing a set of variables $X_1, X_2, X_3...$ such that their values go from $x_1, x_2, x_3$ to $x_1^\prime, x_2^\prime, x_3^\prime$ then the question of which $X_i$ are *important* variables is value-laden.  If we temporarily mechanomorphize humans and suppose that we have a utility function, then we could say that variables are "important" when they're evaluated by our utility function, or when changes to those variables change our expected utility.

But by [1y orthogonality] and [2fs Humean freedom] of the utility function, there's an unlimited number of increasingly complicated utility functions that take into account different variables and functions of variables, so to know what we intuitively mean by "important", the AI would need information of high [5v algorithmic] [5l complexity] that the AI had no way to deduce *a priori*.  Which variables are "important" isn't a question of simple fact - it's on the "ought" side of the [ Humean is-ought type distinction] - so we can't assume that an AI which becomes increasingly good at answering "is"-type questions also knows which variables are "important".

Another way of looking at it is that if an AI merely builds a very good predictive model of the world, the set of "important variables" or "bad side effects" would be a squiggly category with a complicated boundary.  Even after the AI has already formed a rich natural is-language to describe concepts like "thyroid" and "serotonin" that are useful for modeling and predicting human biology, it might still require a long message in this language to exactly describe the wiggly boundary of "important impact" or the even more wiggly boundary of "bad impact".

This suggests that it might be simpler to try to tell the AI to cure cancer with [2pf a minimum of *any* side effects], and [2qq checking any remaining side effects] with the human operator.  If we have a set of "impacts" $X_k$ to be either minimized or checked which is broad enough to include, in passing, *everything* inside the squiggly boundary of the $X_h$ that humans care about, then this broader boundary of "any impact" might be smoother and less wiggly - that is, a short message in the AI's is-language, making it easier to learn.  For the same reason that a library containing every possible book has [5v less information] than a library which only contains one book, a category boundary "impact" which includes everything a human cares about, *plus some other stuff,* can potentially be much simpler than an exact boundary drawn around "impacts we care about" which is value-laden because it involves caring.

From a human perspective, the complexity of our value system is already built into us and now appears as a deceptively simple-looking function call - relative to the complexity already built into us, "bad impact" sounds very obvious and very easy to describe.  This may lead people to underestimate the difficulty of training AIs to perceive the same boundary.  (Just list out all the impacts that potentially lower expected value, darn it!  Just the *important* stuff!)

## Faithful simulation vs. adequate simulation

Suppose we want to run an "adequate" or "good-enough" simulation of an uploaded human brain.  We can't say that an adequate simulation is one with identical input-output behavior to a biological brain, because the brain will almost certainly be a chaotic system, meaning that it's impossible for any simulation to get exactly the same result as the biological system would yield.  We nonetheless don't want the brain to have epilepsy, or to go psychopathic, etcetera.

The concept of an "adequate" simulation, in this case, is really standing in for "a simulation such that the *expected value* of using the simulated brain's information is within epsilon of using a biological brain".  In other words, our intuitive notion of what counts as a good-enough simulation is really a value-laden threshold because it involves an estimate of what's good enough.

So if we want an AI to have a notion of what kind of simulation is a faithful one, we might find it simpler to try to describe some superset of brain properties, such that if the simulated brain doesn't perturb the expectations of those properties, it doesn't perturb expected value either from our own intuitive standpoint (meaning the result of running the uploaded brain is equally valuable in our own expectation).  This set of faithfulness properties would need to automatically pick up on changes like psychosis, but could potentially pick up on a much wider range of other changes that we'd regard as unimportant, so long as all the important ones are in there.

[todo: integrate this into a longer explanation of the orthogonality thesis, maybe with different versions for people who have and haven't checked off Orthogonality.  it's a section someone might run into early on.]

## Personhood

Suppose that non-vegetarian programmers train an AGI on their intuitive category "person", such that:

- Rocks are not "people" and can be harmed if necessary.
- Shoes are not "people" and can be harmed if necessary.
- Cats are sometimes valuable to people, but are not themselves people.
- Alice, Bob, and Carol are "people" and should not be killed.
- Chimpanzees, dolphins, and the AGI itself: not sure, check with the users if the issue arises.

Now further suppose that the programmers haven't thought to cover, in the training data, any case of a cryonically suspended brain.  Is this a person?  Should it not be harmed?  On many 'natural' metrics, a cryonically suspended brain is more similar to a rock than to Alice.

From an intuitive perspective of avoiding harm to sapient life, a cryonically suspended brain has to be presumed a person until otherwise proven.  But the natural, or inductively simple category that covers the training cases is likely to label the brain a non-person, maybe with very high probability.  The fact that we want the AI to be careful not to hurt the cryonically suspended brain is the sort of thing you could only deduce by knowing which sort of things humans care about and why.  It's not a simple physical feature of the brain itself.

Since the category "person" is a value-laden one, when we extend it to a new region beyond the previous training cases, it's possible for an entirely new set of philosophical considerations to swoop in, activate, and control how we classify that case via considerations that didn't play a role in the previous training cases.

# Implications

Our intuitive evaluation of value-laden categories goes through our [2fs Humean degrees of freedom].  This means that a value-laden category which a human sees as intuitively simple, can still have high [-5v], even relative to sophisticated models of the "is" side of the world.  This in turn means that even an AI understands the "is" side of the world very well, might not correctly and exactly learn a value-laden category from a small or incomplete set of training cases.

From the perspective of training an agent that hasn't yet been aligned along all the Humean degrees of freedom, value-laden categories are very wiggly and complicated relative to the agent's empirical language.  Value-laden categories are liable to contain exceptional regions that your training cases turned out not to cover, where from your perspective the obvious intuitive answer is a function of new value-considerations that the agent wouldn't be able to deduce from previous training data.

This is why much of the art in Friendly AI consists of trying to rephrase an alignment schema into terms that are simple relative to "is"-only concepts, where we want an AI with an impact-in-general metric, rather than AI which avoids only bad impacts.  "Impact" might have a simple, central core relative to a moderately sophisticated language for describing the universe-as-is.  "Bad impact" or "important impact" don't have a simple, central core and hence might be much harder to identify via training cases or communication.  Again, this is hard because humans do have all their subtle value-laden categories like 'important' built in as opaque function calls.  Hence people approaching value alignment for the first time often expect that various concepts are easy to identify, and tend to see the intuitive or intended values of all their concepts as "common sense" regardless of which side of the is-ought divide that common sense is on.

It's true, for example, that a modern chess-playing algorithm has "common sense" about when not to try to seize control of the gameboard's center; and similarly a sufficiently advanced agent would develop "common sense" about which substances would in empirical fact have which consequences on human biology, since this part is a strictly "is"-question that can be answered just by looking hard at the universe.  Not *wanting* to administer poisonous substances to a human requires a prior dispreference over the consequences of administering that poison, even if the consequences are correctly forecasted.  Similarly, the category "poison" could be said to really mean something like "a substance which, if administered to a human, produces low utility"; some people might classify vodka as poisonous, while others could disagree.  An AI doesn't necessary have common sense about the [6h intended] evaluation of the "poisonous" category, even if it has fully developed common sense about which substances have which empirical biological consequences when ingested.  One of those forms of common sense can be developed by staring very intelligently at biological data, and one of them cannot.  But from a human intuitive standpoint, both of these can *feel* equally like the same notion of "common sense", which might lead to a dangerous expectation that an AI gaining in one type of common sense is bound to gain in the other.

# Further reading

- http://lesswrong.com/lw/td/magical_categories/


Faithful simulation

The safe simulation problem is to start with some dynamical physical process $D$ which would, if run long enough in some specified environment, produce some trustworthy information of great value, and to compute some *adequate* simulation $S_D$ of $D$ faster than the physical process could have run.  In this context, the term "adequate" is [36h value-laden] - it means that whatever we would use $D$ for, using $S_D$ instead produces within epsilon of the expected [55 value] we could have gotten from using the real $D.$  In more concrete terms, for example, we might want to tell a Task AGI "upload this human and run them as a simulation", and we don't want some tiny systematic skew in how the Task AGI models serotonin to turn the human into a psychopath, which is a *bad* (value-destroying) simulation fault.  Perfect simulation will be out of the question; the brain is almost certainly a chaotic system and hence we can't hope to produce *exactly* the same result as a biological brain.  The question, then, is what kind not-exactly-the-same-result the simulation is allowed to produce.

As with "[2pf low impact]" hopefully being lower-[5v complexity] than "[36h low bad impact]", we might hope to get an *adequate* simulation via some notion of *faithful* simulation, which rules out bumps in serotonin that turn the upload into a psychopath, while possibly also ruling out any number of other changes we *wouldn't* see as important; with this notion of "faithfulness" still being permissive enough to allow the simulation to take place at a level above individual quarks.  On whatever computing power is available - possibly nanocomputers, if the brain was scanned via molecular nanotechnology - the upload must be runnable fast enough to [6y make the simulation task worthwhile].

Since the main use for the notion of "faithful simulation" currently appears to be [2s3 identifying] a safe plan for uploading one or more humans as a [6y pivotal act], we might also consider this problem in conjunction with the special case of wanting to avoid [6v mindcrime].  In other words, we'd like a criterion of faithful simulation which the AGI can compute *without* it needing to observe millions of hypothetical simulated brains for ten seconds apiece, which could constitute creating millions of people and killing them ten seconds later.  We'd much prefer, e.g., a criterion of faithful simulation of individual neurons and synapses between them up to the level of, say, two interacting cortical columns, such that we could be confident that in aggregate the faithful simulation of the neurons would correspond to the faithful simulation of whole human brains.  This way the AGI would not need to think about or simulate whole brains in order to verify that an uploading procedure would produce a faithful simulation, and mindcrime could be avoided.

Note that the notion of a "functional property" of the brain - seeing the neurons as computing something important, and not wanting to disturb the computation - is still value-laden.  It involves regarding the brain as a means to a computational end, and what we see as the important computational end is value-laden, given that chaos guarantees the input-output relation won't be *exactly* the same.  The brain can equally be seen as implicitly computing, say, the parity of the number of synapse activations; it's just that we don't see this functional property as a valuable one that we want to preserve.

To the extent that some notion of function might be invoked in a notion of faithful, permitted speedups, we should hope that rather than needing the AGI to understand the high-level functional properties of the brain *and which details we thought were too important to simplify,* it might be enough to understand a 'functional' model of individual neurons and synapses, with the resulting transform of the uploaded brain still allowing for a pivotal speedup *and* knowably-faithful simulation of the larger brain.

At the same time, strictly local measures of faithfulness seem problematic if they can conceal *systematic* larger divergences.  We might think that any perturbation of a simulated neuron which has as little effect as adding one phonon is "within thermal uncertainty" and therefore unimportant, but if all of these perturbations are pointing in the same direction relative to some larger functional property, the difference might be very significant.  Similarly if all simulated synapses released slightly more serotonin, rather than releasing slightly more or less serotonin in no particular systematic pattern.


Identifying causal goal concepts from sensory data

Suppose we want an AI to carry out some goals involving strawberries, and as a result, we want to [identify_goal_concept identify] to the AI the [ concept] of "strawberry".  One of the potential ways we could do this is by showing the AI objects that a teacher classifies as strawberries or non-strawberries.  However, in the course of doing this, what the AI actually sees will be e.g. a pattern of pixels on a webcam - the actual, physical strawberry is not directly accessible to the AI's intelligence.  When we show the AI a strawberry, what we're really trying to communicate is "A certain proximal [ cause] of this sensory data is a strawberry", not, "This arrangement of sensory pixels is a strawberry."  An AI that learns the latter concept might try to carry out its goal by putting a picture in front of its webcam; the former AI has a goal that actually involves something in its environment.

The open problem of "identifying causal goal concepts from sensory data" or "identifying environmental concepts from sensory data" is about getting an AI to form [ causal] goal concepts instead of [ sensory] goal concepts.  Since almost no [6h human-intended goal] will ever be satisfiable solely in virtue of an advanced agent arranging to see a certain field of pixels, safe ways of identifying goals to sufficiently advanced goal-based agents will presumably involve some way of identifying goals among the *causes* of sense data.

A "toy" (and still pretty difficult) version of this open problem might be to exhibit a machine algorithm that (a) has a causal model of its environment, (b) can learn concepts over any level of its causal model including sense data, (c) can learn and pursue a goal concept, (d) has the potential ability to spoof its own senses or create fake versions of objects, and (e) is shown to learn a proximal causal goal rather than a goal about sensory data as shown by it pursuing only the causal version of that goal even if it would have the option to spoof itself.

For a more elaborated version of this open problem, see "[2s0]".


Goal-concept identification

[summary:  The problem of communicating to an AI a very simple local concept on the order of "strawberries" or "give me a strawberry".  This level of the problem is meant to include subproblems of local categorization like "I meant a real strawberry like the ones I already showed you, not a fake strawberry that looks similar to your webcam".  It isn't meant to include larger problems like [2s3 verifying that a plan uses only known, whitelisted methods] or [6c identifying all possible harmful effects we could care about].]

The problem of trying to figure out how to communicate to an AGI an [6h intended] goal [ concept] on the order of "give me a strawberry, and not a fake plastic strawberry either".

At this level of the problem, we're not concerned with e.g. larger problems of [2s3 safe plan identification] such as not mugging people for strawberries, or [2pf minimizing side effects].  We're not (at this level of the problem) concerned with identifying [6c each and every one of the components of human value], as they might be impacted by side effects more distant in the causal graph.  We're not concerned with [ philosophical uncertainty] about what we [normativity should] mean by "strawberry".  We suppose that in an intuitive sense, we do have a pretty good idea of what we [6h intend] by "strawberry", such that there are things that are definitely strawberries and we're pretty happy with our sense of that so long as nobody is deliberately trying to fool it.

We just want to communicate a *local* goal concept that distinguishes edible strawberries from plastic strawberries, or nontoxic strawberries from poisonous strawberries.  That is: we want to say "strawberry" in an understandable way that's suitable for fulfilling a [6w task] of "just give Sally a strawberry", possibly in conjunction with other features like [2qp conservatism] or [2pf low impact] or [2r8 mild optimization].

For some open subproblems of the obvious approach that goes through showing actual strawberries to the AI's webcam, see "[36w]" and "[2s0]".


Mechanical Turk (example)

[summary:  In 1836, there was a sensation called the [Mechanical Turk](https://en.wikipedia.org/wiki/The_Turk), allegedly a chess-playing automaton. Edgar Allen Poe, who was also an amateur magician, wrote an essay arguing that the Turk must contain a human operator hidden in the apparatus (which it did).

Besides analyzing the Turk's outward appearance to locate the hidden compartment, Poe carefully argued as to why no arrangement of wheels and gears could ever play chess in the first place, explicitly comparing the Turk to "the calculating machine of Mr. Babbage".  In an algebraical problem (Poe said) each step follows with the previous step of necessity, and therefore can be represented by the determinate motions of wheels and gears as in Charles Babbage's proposed computing engine.  In chess, the player's move and opponent's move don't follow with necessity from the board position, and therefore can't be represented by deterministic gears.

Thus, Poe concluded, "It is quite certain that the operations of the Automaton are regulated by *mind*, and by nothing else. Indeed this matter is susceptible of a mathematical demonstration, *a priori*."]

In 1836, there was a sensation called the [Mechanical Turk](https://en.wikipedia.org/wiki/The_Turk), allegedly a chess-playing automaton. Edgar Allen Poe, who was also an amateur magician, wrote an essay arguing that the Turk must contain a human operator hidden in the apparatus (which it did).  Besides analyzing the Turk's outward appearance to locate the hidden compartment, Poe carefully argued as to why no arrangement of wheels and gears could ever play chess in the first place, explicitly comparing the Turk to "the calculating machine of Mr. Babbage":

> Arithmetical or algebraical calculations are, from their very nature, fixed and determinate. Certain data being given, certain results necessarily and inevitably follow [...]
> But the case is widely different with the Chess-Player. With him there is no determinate progression. No one move in chess necessarily follows upon any one other. From no particular disposition of the men at one period of a game can we predicate their disposition at a different period [...]
> Now even granting that the movements of the Automaton Chess-Player were in themselves determinate, they would be necessarily interrupted and disarranged by the indeterminate will of his antagonist. There is then no analogy whatever between the operations of the Chess-Player, and those of the calculating machine of Mr. Babbage [...]
> It is quite certain that the operations of the Automaton are regulated by *mind*, and by nothing else. Indeed this matter is susceptible of a mathematical demonstration, *a priori*.

(In other words:  In an algebraical problem, each step follows with the previous step of necessity, and therefore can be represented by the determinate motions of wheels and gears as in Charles Babbage's proposed computing engine.  In chess, the player's move and opponent's move don't follow with necessity from the board position, and therefore can't be represented by deterministic gears.)

This is an amazingly sophisticated remark, considering the era.  It even puts a finger on the part of chess that is computationally difficult, the combinatorial explosion of possible moves.  And it is still entirely wrong.

Even if you know an unbounded solution to chess, you might still be 47 years away from a bounded solution.  But if you can't state a program that solves the problem *in principle*, you are in some sense *confused* about the nature of the cognitive work needed to solve the problem.  If you can't even solve a problem given infinite computing power, you definitely can't solve it using bounded computing power.  (Imagine Poe trying to write a chess-playing program before he'd had the insight about search trees.)

For more on this point, see "[107]".


Do we need to worry about AI?

Why worry about AI when nothing has gone wrong yet?


Coherent extrapolated volition (alignment target)

#  Introduction

"Coherent extrapolated volition" (CEV) is [2 Eliezer Yudkowsky]'s proposed thing-to-do with an extremely [2c advanced AGI], if you're extremely confident of your ability to [5s align] it on complicated targets.

Roughly, a CEV-based superintelligence would do what currently existing humans would want\* the AI to do, *if counterfactually:*

 1. We knew everything the AI knew;
 2. We could think as fast as the AI and consider all the arguments;
 3. We knew ourselves perfectly and had better self-control or self-modification ability;

...*to whatever extent* most existing humans, thus extrapolated, would predictably want\* the same things.  (For example, in the limit of extrapolation, nearly all humans might want\* not to be turned into [10h paperclips], but might not agree\* on the best pizza toppings.  See below.)

CEV is meant to be the *literally optimal* or *ideal* or *normative* thing to do with an [1g3 autonomous superintelligence], *if* you trust your ability to [41k perfectly align] a superintelligence on a very complicated target.  (See below.)

CEV is rather complicated and meta and hence *not* intended as something you'd do with the first AI you ever tried to build.  CEV might be something that everyone inside a project agreed was an acceptable mutual target for their *second* AI.  (The first AI should probably be a [6w Task AGI].)

For the corresponding metaethical theory see [313].

[todo: start splitting the subsections into separate pages, then build learning paths.]

# Concept

%%knows-requisite([313]):

See "[313]".

%%

%%!knows-requisite([313]):

[313 Extrapolated volition] is the metaethical theory that when we ask "What is right?", then insofar as we're asking something meaningful, we're asking "What would a counterfactual idealized version of myself want\* if it knew all the facts, had considered all the arguments, and had perfect self-knowledge and self-control?"  (As a [313 metaethical theory], this would make "What is right?" a mixed logical and empirical question, a function over possible states of the world.)

A very simple example of extrapolated volition might be to consider somebody who asks you to bring them orange juice from the refrigerator.  You open the refrigerator and see no orange juice, but there's lemonade.  You imagine that your friend would want you to bring them lemonade if they knew everything you knew about the refrigerator, so you bring them lemonade instead.  On an abstract level, we can say that you "extrapolated" your friend's "volition", in other words, you took your model of their mind and decision process, or your model of their "volition", and you imagined a counterfactual version of their mind that had better information about the contents of your refrigerator, thereby "extrapolating" this volition.

Having better information isn't the only way that a decision process can be extrapolated; we can also, for example, imagine that a mind has more time in which to consider moral arguments, or better knowledge of itself.  Maybe you currently want revenge on the Capulet family, but if somebody had a chance to sit down with you and have a long talk about how revenge affects civilizations in the long run, you could be talked out of that.  Maybe you're currently convinced that you advocate for green shoes to be outlawed out of the goodness of your heart, but if you could actually see a printout of all of your own emotions at work, you'd see there was a lot of bitterness directed at people who wear green shoes, and this would change your mind about your decision.

In Yudkowsky's version of extrapolated volition considered on an individual level, the three core directions of extrapolation are:

- Increased knowledge - having more veridical knowledge of declarative facts and expected outcomes.
- Increased consideration of arguments - being able to consider more possible arguments and assess their validity.
- Increased reflectivity - greater knowledge about the self, and to some degree, greater self-control (though this raises further questions about which parts of the self normatively get to control which other parts).

%%

# Motivation

Different people initially react differently to the question "Where should we point a superintelligence?" or "What should an aligned superintelligence do?" - not just different beliefs about what's good, but different frames of mind about how to ask the question.

Some common reactions:

1. "Different people want different things!  There's no way you can give everyone what they want.  Even if you pick some way of combining things that people want, *you'll* be the one saying how to combine it.  Someone else might think they should just get the whole world for themselves.  Therefore, in the end *you're* deciding what the AI will do, and any claim to some sort of higher justice or normativity is nothing but sophistry."
2. "What we should do with an AI is obvious - it should optimize liberal democratic values.  That already takes into account everyone's interests in a fair way.  The real threat is if bad people get their hands on an AGI and build an AGI that doesn't optimize liberal democratic values."
3. "Imagine the ancient Greeks telling a superintelligence what to do.  They'd have told it to optimize for glorious deaths in battle.  Programming any other set of inflexible goals into a superintelligence seems equally stupid; it has to be able to change and grow."
4. "What if we tell the superintelligence what to do and it's the wrong thing? What if we're basically confused about what's right? Shouldn't we let the superintelligence figure that out on its own, with its assumed superior intelligence?"

An initial response to each of these frames might be:

1.  "Okay, but suppose you're building a superintelligence and you're trying *not to be a jerk about it.*  If you say, 'Whatever I do originates in myself, and therefore is equally selfish, so I might as well declare myself God-Emperor of the Universe' then you're being a jerk.  Is there anything you could do instead which would be less like being a jerk?  What's the *least* jerky thing you could do?"
2.  "What if you would, after some further discussion, want to tweak your definition of 'liberal democratic values' just a little? What if it's *predictable* that you would do that? Would you really want to be stuck with your off-the-cuff definition a million years later?"
3.  "Okay, so what should the Ancient Greeks have done if they did have to program an AI? How could they not have doomed future generations? Suppose the Ancient Greeks are clever enough to have noticed that sometimes people change their minds about things and to realize that they might not be right about everything. How can they use the cleverness of the AGI in a constructively specified, computable fashion that gets them out of this hole? You can't just tell the AGI to compute what's 'right', you need to put an actual computable question in there, not a word."
4.  "You asked, what if we're basically confused about what's right - well, in that case, what does the word 'right' even mean?  If you don't know what's right, and you don't know how to compute what's right, then what are we even talking about?  Do you have any ground on which to say that an AGI which only asks 'Which outcome leads to the greatest number of paperclips?' isn't computing rightness?  If you don't think a paperclip maximizer is computing rightness, then you must know something about the rightness-question which excludes that possibility - so let's talk about how to program that rightness-question into an AGI."

CEV's advocates claim that all of these lines of discussion eventually end up converging on the idea of coherent extrapolated volition.  For example:

1.  Asking what everyone would want\* if they knew what the AI knew, and doing what they'd all predictably agree on, is just about the least jerky thing you can do.  If you tell the AI to give everyone a volcano lair because you think volcano lairs are neat, you're not being selfish, but you're being a jerk to everyone who doesn't want a volcano lair.  If you have the AI just do what people actually say, they'll end up hurting themselves with dumb wishes and you'd be a jerk.  If you only extrapolate your friends and have the AI do what only you'd want, you're being jerks to everyone else.
2.  Yes, liberal democratic values are good; so is apple pie.  Apple pie is *a* good thing but it's not the *only* good thing.  William Frankena's list of ends-in-themselves included "Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment" and then 25 more items, and the list certainly isn't complete.  The only way you're going to get a complete list is by analyzing human minds; and even then, if our descendants would predictably want something else a million years later, we ought to take that into account too.
3.  Every improvement is a change, but not every change is an improvement. Just letting a superintelligence change at random doesn't encapsulate moral progress.  Saying that change toward more liberal democratic values is progress, presumes that we already know the destination or answer.  We can't even just ask the AGI to *predict* what civilizations would think a thousand years later, since (a) the AI itself impacts this and (b) if the AI did nothing, maybe in a thousand years everyone would have accidentally blissed themselves out while trying to modify their own brains.  If we want to do better than the hypothetical ancient Greeks, we need to define *a sufficiently abstract and meta criterion* that describes *valid* directions of progress - such as changes in moral beliefs associated with learning new facts, for example; or moral change that would predictably occur if we considered a larger set of arguments; or moral change that would predictably occur if we understood ourselves better.
4.  This one is a long story:  Metaethics deals with the question of what sort of entity 'rightness' is exactly - tries to reconcile this strange ineffable 'rightness' business with a universe made out of particle fields.  Even though it seems like human beings wanting to murder people wouldn't *make* murder right, there's also nowhere in the stars or mountains where we can actually find it written that murder is wrong.  At the end of a rather long discussion, we decide that for any given person speaking at a given point in time, 'rightness' is a logical constant which, although not counterfactually dependent on the state of the person's brain, must be analytically identified with the extrapolated volition of that brain; and we show that (only) this stance gives consistent answers to all the standard questions in metaethics.  (This discussion takes a while, on the order of explaining how deterministic laws of physics don't show that you have unfree will.)

(To do:  Write dialogues from each of these four entrance points.) [todo: write these and split them up into separate subpages]

# Situating CEV in contemporary metaethics

See the corresponding section in "[313]".

# Scary design challenges

There are several reasons why CEV is *way* too challenging to be a good target for any project's first try at building machine intelligence:

1.  A CEV agent would be intended to carry out an [1g3 autonomous] open-ended mission.  This implies all the usual reasons we expect an autonomous AI to be harder to make safe than a [6w Task AGI].
2.  CEV is a weird goal.  It involves recursion.
3.  Even the terms in CEV, like "know more" or "extrapolate a human", seem complicated and [36h value-laden].  You might have to build a high-level [2s1 Do What I Know I Mean agent], and then tell it to do CEV.  [2s1 Do What I Know I Mean] is complicated enough that you'd need to build an AI that can learn DWIKIM, so that DWIKIM can be taught rather than formally specified.  So we're looking at something like CEV, running on top of DWIKIM, running on top of a goal-learning system, at least until the first time the CEV agent rewrites itself.

Doing this correctly the *very first time* we build a smarter-than-human intelligence seems *improbable.*  The only way this would make a good first target is if the CEV concept is formally simpler than it currently seems, *and* timelines to AGI are unusually long and permit a great deal of advance work on safety.

If AGI is 20 years out (or less), it seems wiser to think in terms of a [6w Task AGI] performing some relatively simple [6y pivotal act].  The role of CEV is of answering the question, "What can you all agree in advance that you'll try to do *next,* after you've executed your Task AGI and gotten out from under the shadow of immediate doom?"

# What if CEV fails to cohere?

A frequently asked question is "What if extrapolating human volitions produces incoherent answers?"

According to the original motivation for CEV, if this happens in *some* places, a Friendly AI ought to ignore those places.  If it happens *everywhere,* you probably picked a silly way to construe an extrapolated volition and you ought to rethink it. %note: Albeit in practice, you would not want an AI project to take a dozen tries at defining CEV.  This would indicate something extremely wrong about the method being used to generate suggested answers.  Whatever final attempt passed would probably be the first answer [blackzoning all of whose remaining flaws were hidden], rather than an answer with all flaws eliminated.%

That is:

- If your CEV algorithm finds that "People coherently want to not be eaten by [10h paperclip maximizers], but end up with a broad spectrum of individual and collective possibilities for which pizza toppings they prefer", we would normatively want a Friendly AI to prevent people from being eaten by paperclip maximizers but not mess around with which pizza toppings people end up eating in the Future.
- If your CEV algorithm claims that there's no coherent sense in which "A lot of people would want to not be eaten by Clippy and would still want\* this even if they knew more stuff" then this is a suspicious and unexpected result.  Perhaps you have picked a silly way to construe somebody's volition.

The original motivation for CEV can also be viewed from the perspective of "What is it to help someone?" and "How can one help a large group of people?", where the intent behind the question is to build an AI that renders 'help' as we really [6h intend] that.  The elements of CEV can be seen as caveats to the naive notion of "Help is giving people whatever they ask you for!" in which somebody asks you to bring them orange juice but the orange juice in the refrigerator is poisonous (and they're not *trying* to poison themselves).

What about helping a group of people?  If two people ask for juice and you can only bring one kind of juice, you should bring a non-poisonous kind of juice they'd both like, to the extent any such juice exists.  If no such juice exists, find a kind of juice that one of them is meh about and that the other one likes, and flip a coin or something to decide who wins.  You are then being around as helpful as it is possible to be.

Can there be *no way* to help a large group of people?  This seems implausible.  You could at least give the starving ones pizza with a kind of pizza topping they currently like.  To the extent your philosophy claims "Oh noes even *that* is not helping because it's not perfectly coherent," you have picked the wrong construal of 'helping'.

It could be that, if we find that every reasonable-sounding construal of extrapolated volition fails to cohere, we must arrive at some entirely other notion of 'helping'.  But then this new form of helping also shouldn't involve bringing people poisonous orange juice that they don't know is poisoned, because that still intuitively seems unhelpful.

## Helping people with incoherent preferences

What if somebody believes themselves to [7hh prefer onions to pineapple on their pizza, prefer pineapple to mushrooms, and prefer mushrooms to onions]?  In the sense that, offered any two slices from this set, they would pick according to the given ordering?

(This isn't an unrealistic example.  Numerous experiments in behavioral economics demonstrate exactly this sort of circular preference.  For instance, you can arrange 3 items such that each pair of them brings a different salient quality into focus for comparison.)

One may worry that we couldn't 'coherently extrapolate the volition' of somebody with these pizza preferences, since these local choices obviously aren't consistent with any coherent utility function.  But how could we *help* somebody with a pizza preference like this?

Well, appealing to the intuitive notion of *helping:*

- We could give them whatever kind of pizza they'd pick if they had to pick among all three simultaneously.
- We could figure out how happy they'd be eating each type of pizza, in terms of emotional intensity as measured in neurotransmitters; and offer them the slice of pizza that they'll most enjoy.
- We could let them pick their own damn pizza toppings and concern ourselves mainly with making sure the pizza isn't poisonous, since the person definitely prefers non-poisoned pizza.
- We could, given sufficient brainpower on our end, figure out what this person would ask us to do for them in this case after that person had learned about the concept of a preference reversal and been told about their own circular preferences.  If this varies wildly depending on exactly how we explain the concept of a preference reversal, we could refer back to one of the previous three answers instead.

Conversely, these alternatives seem *less* helpful:

- Refuse to have anything to do with that person since their current preferences don't form a coherent utility function.
- Emit "ERROR ERROR" sounds like a Hollywood AI that's just found out about the Epimenides Paradox.
- Give them pizza with your own favorite topping, green peppers, even though they'd prefer any of the 3 other toppings to those.
- Give them pizza with the topping that would taste best to them, pepperoni, despite their being vegetarians.

Advocates of CEV claim that if you blank the complexities of 'extrapolated volition' out of your mind; and ask how you could reasonably help people as best as possible if you were trying not be a jerk; and then try to figure out how to semiformalize whatever mental procedure you just followed to arrive at your answer for how to help people; then you will eventually end up at CEV again.

# Role of meta-ideals in promoting early agreement

A primary purpose of CEV is to represent a relatively simple meta-level ideal that people can agree upon, even where they might disagree on the object level.  By a hopefully analogous example, two honest scientists might disagree on the correct mass of an electron, but agree that the experimental method is a good way to resolve the answer.

Imagine Millikan believes an electron's mass is 9.1e-28 grams, and Nannikan believes the correct electron mass is 9.1e-34 grams.  Millikan might be very worried about Nannikan's proposal to program an AI to believe the electron mass is 9.1e-34 grams; Nannikan doesn't like Millikan's proposal to program in 9.1-e28; and both of them would be unhappy with a compromise mass of 9.1e-31 grams.  They might still agree on programming an AI with some analogue of probability theory and a simplicity prior, and letting a superintelligence come to the conclusions implied by Bayes and Occam, because the two can agree on an effectively computable question even though they think the question has different answers.  Of course, this is easier to agree on when the AI hasn't yet produced an answer, or if the AI doesn't tell you the answer.

It's not guaranteed that every human embodies the same implicit moral questions, indeed this seems unlikely, which means that Alice and Bob might still expect their extrapolated volitions to disagree about things.  Even so, while the outputs are still abstract and not-yet-computed, Alice doesn't have much of a place to stand on which to appeal to Carol, Dennis, and Evelyn by saying, "But as a matter of morality and justice, you should have the AI implement *my* extrapolated volition, not Bob's!"  To appeal to Carol, Dennis, and Evelyn about this, you'd need them to believe that Alice's EV was more likely to agree with their EVs than Bob's was - and at that point, why not come together on the obvious Schelling point of extrapolating *everyone's* EVs?

Thus, one of the primary purposes of CEV (selling points, design goals) is that it's something that Alice, Bob, and Carol can agree *now* that Dennis and Evelyn should do with an AI that will be developed *later;* we can try to set up commitment mechanisms now, or check-and-balance mechanisms now, to ensure that Dennis and Evelyn are still working on CEV later.

## Role of 'coherence' in reducing expected unresolvable disagreements

A CEV is not necessarily a majority vote.  A lot of people with an extrapolated weak preference\* might be counterbalanced by a few people with a strong extrapolated preference\* in the opposite direction.  Nick Bostrom's "[parliamentary model](http://www.overcomingbias.com/2009/01/moral-uncertainty-towards-a-solution.html)" for resolving uncertainty between incommensurable ethical theories, permits a subtheory very concerned about a decision to spend a large amount of its limited influence on influencing that particular decision.

This means that, e.g., a vegan or animal-rights activist should not need to expect that they must seize control of a CEV algorithm in order for the result of CEV to protect animals.  It doesn't seem like most of humanity would be deriving huge amounts of utility from hurting animals in a post-superintelligence scenario, so even a small part of the population that strongly opposes\* this scenario should be decisive in preventing it.

# Moral hazard vs. debugging

One of the points of the CEV proposal is to have minimal [2sb moral hazard] (aka, not tempting the programmers to take over the world or the future); but this may be compromised if CEV's results don't go literally unchecked.

Part of the purpose of CEV is to stand as an answer to the question, "If the ancient Greeks had been the ones to invent superintelligence, what could they have done that would not, from our later perspective, irretrievably warp the future?  If the ancient Greeks had programmed in their own values directly, they would have programmed in a glorious death in combat.  Now let us consider that perhaps we too are not so wise."  We can imagine the ancient Greeks writing a CEV mechanism, peeking at the result of this CEV mechanism before implementing it, and being horrified by the lack of glorious-deaths-in-combat in the future and value system thus revealed.

We can also imagine that the Greeks, trying to cut down on [2sb moral hazard], virtuously refuse to peek at the output; but it turns out that their attempt to implement CEV has some unforeseen behavior when actually run by a superintelligence, and so their world is turned into paperclips.

This is a safety-vs.-moral-hazard tradeoff between (a) the benefit of being able to look at CEV outputs in order to better-train the system or just verify that nothing went horribly wrong; and (b) the moral hazard that comes from the temptation to override the output, thus defeating the point of having a CEV mechanism in the first place.

There's also a potential safety hazard just with looking at the internals of a CEV algorithm; the simulated future could contain all sorts of directly mind-hacking cognitive hazards.

Rather than giving up entirely and embracing maximum moral hazard, one possible approach to this issue might be to have some single human that is supposed to peek at the output and provide a 1 or 0 (proceed or stop) judgment to the mechanism, without any other information flow being allowed to the programmers if the human outputs 0.  (For example, the volunteer might be in a room with explosives that go off if 0 is output.)

# "Selfish bastards" problem

Suppose that Fred is funding Grace to work on a CEV-based superintelligence; and Evelyn has decided not to oppose this project.  The resulting CEV is meant to extrapolate the volitions of Alice, Bob, Carol, Dennis, Evelyn, Fred, and Grace with equal weight.  (If you're reading this, you're more than usually likely to be one of Evelyn, Fred, or Grace.)

Evelyn and Fred and Grace might worry:  "What if a supermajority of humanity consists of 'selfish\* bastards', such that their extrapolated volitions would cheerfully vote\* for a world in which it was legal to own artificial sapient beings as slaves so long as they personally happened to be in the slaveowning class; and we, Evelyn and Fred and Grace, just happen to be in the minority that extremely doesn't want nor want\* the future to be like that?"

That is: What if humanity's extrapolated volitions diverge in such a way that from the standpoint of *our* volitions - since, if you're reading this, you're unusually likely to be one of Evelyn or Fred or Grace - 90% of extrapolated humanity would choose\* something such that *we* would not approve of it, and our volitions would not approve\* of it, *even after* taking into account that we don't want to be jerks about it and that we don't think we were born with any unusual or exceptional right to determine the fate of humanity.

That is, let the scenario be as follows:

> 90% of the people (but not we who are collectively sponsoring the AI) are selfish bastards at the core, such that *any reasonable* extrapolation process (it's not just that we picked a broken one) would lead to them endorsing a world in which they themselves had rights, but it was okay to create artificial people and hurt them.  Furthermore, they would derive enough utility from being personal God-Emperors that this would override our minority objection even in a parliamentary model.

We can see this hypothetical outcome as potentially undermining every sort of reason that we, who happen to be in a position of control to prevent that outcome, should voluntarily relinquish that control to the remaining 90% of humanity:

- We can't be prioritizing being fair to everyone including the other 90% of humanity, because what about being fair to the artificial people who are being hurt?
- We can't be worrying that the other 90% of humanity would withdraw their support from the project, or worrying about betraying the project's supporters, because by hypothesis they weren't supporting it or even permitting it.
- We can't be agreeing to defer to a righter and more intelligent process to resolve our dispute, because by hypothesis the CEV made up of 90% selfish\* bastards is not, from our own perspective, ideally righter.
- We can't rely on a parliamentary model of coherence to prevent what a minority sees as disaster, because by hypothesis the other 90% is deriving enough utility from collectively declaring themselves God-Emperors to trump even a strong minority countervote.

Rather than giving up entirely and taking over the world, *or* exposing ourselves to moral hazard by peeking at the results, one possible approach to this issue might be to run a three-stage process.

This process involves some internal references, so the detailed explanation needs to follow a shorter summary explanation.

In summary:

- Extrapolate everyone's CEV.
- Extrapolate the CEV of the contributors only, and let it give (only) an up-down vote on Everyone's CEV.
- If the result is thumbs-up, run Everyone's CEV.
- Otherwise, extrapolate everyone's CEV, but kicking out all the parts that would act unilaterally and without any concern for others if they were in positions of unchecked power.
- Have the Contributor CEV give an up/down answer on the Fallback CEV.
- If the result is thumbs-up, run the Fallback CEV.
- Otherwise fail.

In detail:

- First, extrapolate the everyone-on-Earth CEV as though it were not being checked.
  - If any hypothetical extrapolated person worries about being checked, delete that concern and extrapolate them as though they didn't have it.  This is necessary to prevent the check itself from having a [updateless UDT] influence on the extrapolation and the actual future.
- Next, extrapolate the CEV of everyone who contributed to the project, weighted by their contribution (possibly based on some mix of "how much was actually done" versus "how much was rationally expected to be accomplished" versus "the fraction of what could've been done versus what was actually done").  Allow this other extrapolation an up-or-down vote - *not* any kind of detailed correction - on whether to let the everyone-on-Earth CEV to go through unmodified.
  - Remove from the extrapolation of the Contributor-CEV any strategic considerations having to do with the Fallback-CEV or post-Fail redevelopment being a *better* alternative; we want to extract a judgment about "satisficing" in some sense, whether the Everyone-CEV is in some non-relative sense too horrible to be allowed.
- If the Everyone-CEV passes the Contributor-CEV check, run it.
- Otherwise, re-extrapolate a Fallback-CEV that starts with all existing humans as a base, but *discards* from the extrapolation all extrapolated decision processes that, if they were in a superior strategic position or a position of unilateral power, would *not* bother to extrapolate others' volitions or care about their welfare.
  - Again, remove all extrapolated *strategic* considerations about passing the coming check.
- Check the Fallback-CEV against the Contributor-CEV for an up-down vote.  If it passes, run it.
- Otherwise Fail (AI shuts down safely, we rethink what to do next or implement an agreed-on fallback course past this point).

The particular fallback of "kick out from the extrapolation any weighted portions of extrapolated decision processes that would act unilaterally and without caring for others, given unchecked power" is meant to have a property of poetic justice, or rendering objections to it self-defeating:  If it's okay to act unilaterally, then why can't we unilaterally kick out the unilateral parts?  This is meant to be the 'simplest' or most 'elegant' way of kicking out a part of the CEV whose internal reasoning directly opposes the whole reason we ran CEV in the first place, but imposing the minimum possible filter beyond that.

Thus if Alice (who by hypothesis is not in any way a contributor) says, "But I demand you altruistically include the extrapolation of me that would unilaterally act against you if it had power!" then we reply, "We'll try that, but if it turns out to be a sufficiently bad idea, there's no coherent interpersonal grounds on which you can rebuke us for taking the fallback option instead."

Similarly in regards to the Fail option at the end, to anyone who says, "Fairness demands that you run Fallback CEV even if you wouldn't like\* it!" we can reply, "Our own power may not be used against us; if we'd regret ever having built the thing, fairness doesn't oblige us to run it."

# Why base CEV on "existing humans" and not some other class of extrapolees?

One frequently asked question about the implementation details of CEV is either:

- Why formulate CEV such that it is run on "all existing humans" and not "all existing and past humans" or "all mammals" or "all sapient life as it probably exists everywhere in the measure-weighted infinite multiverse"?
- Why not restrict the extrapolation base to "only people who contributed to the AI project"?

In particular, it's been asked why restrictive answers to Question 1 don't [3tc also imply] the more restrictive answer to Question 2.

## Why not include mammals?

We'll start by considering some replies to the question, "Why not include all mammals into CEV's extrapolation base?"

- Because you could be wrong about mammals being objects of significant ethical value, such that we *should* on an object level respect their welfare.  The extrapolation process will catch the error if you'd predictably change your mind about that.  Including mammals into the extrapolation base for CEV potentially sets in stone what could well be an error, the sort of thing we'd predictably change our minds about later.  If you're normatively *right* that we should all care about mammals and even try to extrapolate their volitions into a judgment of Earth's destiny, if that's what almost all of us would predictably decide after thinking about it for a while, then that's what our EVs will decide\* to do on our behalf; and if they don't decide\* to do that, it wasn't right which undermines your argument for doing it unconditionally.
- Because even if we ought to care about mammals' welfare qua welfare, extrapolated animals might have really damn weird preferences that you'd regret including into the CEV.  (E.g., after human volitions are outvoted by the volitions of other animals, the current base of existing animals' extrapolated volitions choose\* a world in which they are uplifted to God-Emperors and rule over suffering other animals.)
- Because maybe not everyone on Earth cares\* about animals even if your EV would in fact care\* about them, and to avoid a slap-fight over who gets to rule the world, we're going to settle this by e.g. a parliamentary-style model in which you get to expend your share of Earth's destiny-determination on protecting animals.

To expand on this last consideration, we can reply:  "Even if you would regard it as more just to have the *right* animal-protecting outcome baked into the future immediately, so that your EV didn't need to expend some of its voting strength on assuring it, not everyone else might regard that as just.  From our perspective as programmers we have no particular reason to listen to you rather than Alice.  We're not arguing about whether animals will be protected if a minority vegan-type subpopulation strongly want\* that and the rest of humanity doesn't care\*.  We're arguing about whether, if *you* want\* that but a majority doesn't, your EV should justly need to expend some negotiating strength in order to make sure animals are protected.  This seems pretty reasonable to us as programmers from our standpoint of wanting to be fair, not be jerks, and not start any slap-fights over world domination."

This third reply is particularly important because taken in isolation, the first two replies of "You could be wrong about that being a good idea" and "Even if you care about their welfare, maybe you wouldn't like their EVs" could [3tc equally apply to argue] that contributors to the CEV project ought to extrapolate only their own volitions and not the rest of humanity:

- We could be wrong about it being a good idea, by our own lights, to extrapolate the volitions of everyone else; including this into the CEV project bakes this consideration into stone; if we were right about running an Everyone CEV, if we would predictably arrive at that conclusion after thinking about it for a while, our EVs could do that for us.
- Not extrapolating other people's volitions isn't the same as saying we shouldn't care.  We could be right to care about the welfare of others, but there could be some spectacular horror built into their EVs.

The proposed way of addressing this was to run a composite CEV with a contributor-CEV check and a Fallback-CEV fallback.  But then why not run an Animal-CEV with a Contributor-CEV check before trying the Everyone-CEV?

One answer would go back to the third reply above:  Nonhuman mammals aren't sponsoring the CEV project, allowing it to pass, or potentially getting angry at people who want to take over the world with no seeming concern for fairness.  So they aren't part of the Schelling Point for "everyone gets an extrapolated vote".

## Why not extrapolate all sapients?

Similarly if we ask:  "Why not include all sapient beings that the SI suspects to exist everywhere in the measure-weighted multiverse?"

- Because large numbers of them might have EVs as alien as the EV of an Ichneumonidae wasp.
- Because our EVs can always do that if it's actually a good idea.
- Because they aren't here to protest and withdraw political support if we don't bake them into the extrapolation base immediately.

## Why not extrapolate deceased humans?

"Why not include all deceased human beings as well as all currently living humans?"

In this case, we can't then reply that they didn't contribute to the human project (e.g. I. J. Good).  Their EVs are also less likely to be alien than in any other case considered above.

But again, we fall back on the third reply:  "The people who are still alive" is a simple Schelling circle to draw that includes everyone in the current political process.  To the extent it would be nice or fair to extrapolate Leo Szilard and include him, we can do that if a supermajority of EVs decide\* that this would be nice or just.  To the extent we *don't* bake this decision into the model, Leo Szilard won't rise from the grave and rebuke us.  This seems like reason enough to regard "The people who are still alive" as a simple and obvious extrapolation base.

## Why include people who are powerless?

"Why include very young children, uncontacted tribes who've never heard about AI, and retrievable cryonics patients (if any)?  They can't, in their current state, vote for or against anything."

- A lot of the intuitive motivation for CEV is to not be a jerk, and ignoring the wishes of powerless living people seems intuitively a lot more jerkish than ignoring the wishes of powerless dead people.
- They'll actually be present in the future, so it seems like less of a jerk thing to do to extrapolate them and take their wishes into account in shaping that future, than to not extrapolate them.
- Their relatives might take offense otherwise.
- It keeps the Schelling boundary simple.


VAT playpen

007! serotonin

    ###############################################################################
    ###############################################################################
    ###########################################**##################################
    ##########################################*####################################
    ####################################**#########################################
    ######################################***######################################
    ########################################*#######**#############################
    ################################*#*###*#**#****  **############################
    ###############################****#****          *#####**#####################
    ##############################****#***             *####**#####################
    ##############################** ***              **####*######################
    ##############################******              **####*######################
    ######*#####**################**** *              **###**#######*##############
    ####***###*****###############*** *               *####*####*#** *   *#########
    #######*#** *  *##############***                 *####*###*** *     **########
    ######****     *##############****                *####*###**  *      *########
    #####*****      *#############*##*****        *****###**###**         **#######
    ####*******     *############**##** ****   *******####*###*#*   *      *#######
    ####****#**  **   ###########******# ****  ** #*#***##*###*#*        ***#######
    #####** ***** **  *######*###** ***    **       ** *##*###***##    *#***#######
    #####**  ****##*   *#####*###**         *          *#**##**##*#*  *############
    ######**     #*   ** *##**###**                    ##*###* ****   #***#**######
    #####****   **    *** ##***##**         *         *##*####        *    **######
    #######**         *** ### **#***        *         *#**####        *     #######
    ######**          **#####  *#***       **         *#**####*       *    *#######
    ######*            *###### ******       *        **#*#####*       *    *#######
    #######    #*      *########******     *** **    **#*######*    * #*  *########
    #######*  *        *########*****      **#**    **#**######*     **  **########
    ########*  ***    *#*########*****      *       **##*######*  **#*##**#########
    ######### *#      *#***######*****              **##**#####**  *******#########
    #########****    *##***#######****             **##**######***  *****##########
    ##########*     *##****#######****  **#**********##**######* *     **##########
    ###########   **##*****######*#***            ***##*#######** **  **###########
    ################## * **##### **#***    * ********##*#*#####**  *###############
    #################******#####   *****************############*** *##############
    #################******####*    ##** *       ***#############******############
    #################****#######     ##***       **####### *#######******##########
    #################*###########     *##**     **#######   *######################
    ##############################      *#*******#####*#*   #######################
    ###############################       *#############*  *#######################
    ################################        *#########*#   *#* ####################
    #################################*      ***##*######   #*   ###################
    ###################################    *########**##   #      #################
    ####################################    ########***#*  #      *################
    #####################################   ########**##           **##############
    ######################################  ########*##*             ##############
    #######*###*########################### *######*#*#*             *#############
    ######################################## ######***#*             *#############
    ####################**##################*##########*             *#############
    ###################################################**            *#############
    ####*###########*###**#########################*##***            *#############
    ####*##*#####################################****#***           *##############
    #######*##*#################################*  **##**     *    **##############
    ##########**################################*  ***#*** ****  ****##############
    #######*#####################################* ***##***  *******###############
    ####**#######****#############################* ***##**  *******############***
    ###*  *** ********############################********* * *****#############***
    ###*****   **** **############################******#*    ****##############*##
    ###*****    *******##########################*   *###*     **#############*####
    ###**  *** ******###***#########**##########** ** *****    *****###############
    ##** ****#*###***##**##*######***#*####************************######*#########
    ##**************##**###**####***###*##**#####************* * **################
    #***  **************###**###***####*##*######********  **#** *######*#####*####
    ****  *#* *********####**###**####**#######****  *************########*##**####
    ***          ******####**##***####**########***  *#     *******#######****#####
                   ***#####**##**#####**######****** *#**  *********###############
               *##****####*******#####**#####*****#*****  *******#*################
    *##***       *****###********####***#####*****  ***   *****##* ##***###########
        * **         **#*********####***#####***          *****##*** ####*#########
           **          ###**  **********####* *            **  ***  *#### #########
            *                 ***  ****     *           *  *****##*######**########
           * **#                   ***     ###              **********###*#########
         ***    ****            ** **       *              **** *****###* #########
          *   **#**             **         #*##             ** *#******   #########
               ***#                      #######         * *#* ##*        ###*#####
                 *         ***   *       ########*  *    **** ###*            *####
                 # * * *    ***          ###### *#    ***   *####*       *   *#####
    **           *#*  *   ****** **     #######  # ***     *####*           ######*
    ***      **  **      * ***** **       #####  #******* *####*          *#####**
    *******         *    *****#*###**     ###### * *##**#*#*#**         #######**
    ###*******       *    ***#****#*      #####    ##########**        **######* **
    #####* **#*  ***    *#*#**##*####     *####    #########**         ***#####****


Cognitive steganography

Cognitive steganography is an [10g instrumental strategy] for AGIs that are already trying to [10f deceive their programmers] for any reason (e.g., because the AGI is disaligned and does not want the programmers to realize this and edit the AGI's utility function).  Whatever tools the programmers have for monitoring the AGI's thoughts, the AGI will not want those tools to helpfully print out all the AGIs plans for deception.  This could mean, e.g., trying to have the calculations occur somewhere else or in a weird side-mode that the tools can't read; or making the thoughts look like they're about something else; or burying any warning signs under [2qq an avalanche of false alarms too long for the programmers to manually check all of them]; or making the visible portions of the thoughts look innocuous.  ("Oh, it's trying to figure out what behavior we want to see!  That's good, right?")

Hopefully, the intention to figure out cognitive steganography and implement it, would occur before any cognitive steganography had actually been implemented.  This means that perhaps the most critical AI thought-monitoring tool, especially for monitoring an AGI that is or might be able to model human psychology, is the one that tries to detect a forming intention to defeat the thought-monitoring tool.  Similarly, so long as the AI's state is perfectly [reproducible], one could in principle go back and use more sophisticated later thought-monitoring tools to try to detect whether any intention toward cognitive steganography had occurred months earlier.