[summary:  Suppose we're trying to give a [6w Task AGI] the task, "Put a strawberry on this pedestal".  We mean to identify our intended category of strawberries by waving some strawberries and some non-strawberries in front of the AI's webcam.  Alice in the control room will press a button to label which of these objects are strawberries.  The "Look where I'm pointing, not at my finger" problem is getting the AI to focus on the strawberries rather than Alice or the button.  The concepts "strawberry on the pedestal" and "event that makes Alice think of strawberries" and "event that causes the button to be pressed" are different goals to pursue, even though as concepts they'll all equally well-classify any normal training cases.  AIs pursuing these goals respectively put a strawberry on the pedestal, fool Alice using a plastic strawberry, and build a robotic arm to press the labeling button.

We want a way to point to a particular part of the AI's model of the causal lattice that produces the labeled training data - the event we intuitively consider to be the strawberry on the pedestal, versus other parts of the causal lattice like Alice and the button.  Hence "look where I'm pointing, not at my finger".

![strawberry diagram](http://www.gliffy.com/go/publish/image/10424137/L.png)]

## Example problem

Suppose we're trying to give a [6w Task AGI] the task, "Make there be a strawberry on the pedestal in front of your webcam."  For example, a human could fulfill this task by buying a strawberry from the supermarket and putting it on the pedestal.

As part of aligning a Task AGI on this goal, we'd need to [36y identify] strawberries and the pedestal.

One possible approach to communicating the concept of "strawberry" is through a training set of human-selected cases of things that are and aren't strawberries, on and off the pedestal.

For the sake of distinguishing causal roles, let's say that one human, User1, is selecting training cases of objects and putting them in front of the AI's webcam.  A different human, User2, is looking at the scene and pushing a button when they see something that looks like a strawberry on the pedestal.  The [6h intention] is that pressing the button will label positive instances of the goal concept, namely strawberries on the pedestal.  In actual use after training, the AI will be able to generate its own objects to put inside the room, possibly with further feedback from User2.  We want these objects to be instances of our [6h intended] goal concept, aka, actual strawberries.

We could draw an intuitive causal model for this situation as follows:

![strawberry diagram](http://www.gliffy.com/go/publish/image/10423843/L.png)

Suppose that during the use phase, the AI actually creates a realistic plastic strawberry, one that will fool User2 into pressing the button.  Or, similarly, suppose the AI creates a small robot that sprouts tiny legs and runs over to User2's button and presses the button directly.

Neither of these are the goal concept that we wanted the AI to learn, but any *test* of the hypothesis "Is this event classified as a positive instance of the goal concept?" will return "Yes, the button was pressed."  %%comment: (If you imagine some other User3 watching this and pressing an override button to tell the AI that this fake strawberry wasn't really a positive instance of the intended goal concept, imagine the AI modeling and then manipulating or bypassing User3, etcetera.)%%

More generally, the human is trying to point to their intuitive "strawberry" concept, but there may be other causal concepts that also separate the training data well into positive and negative instances, such as "objects which come from strawberry farms", "objects which cause (the AI's psychological model of) User2 to think that something is a strawberry", or "any chain of events leading up to the positive-instance button being pressed".

%%comment: move this to sensory identification section:  However, in a case like this, it's not like the actual physical glove is inside the AGI's memory.  Rather, we'd be, say, putting the glove in front of the AGI's webcam, and then (for the sake of simplified argument) pressing a button which is meant to label that thing as a "positive instance".  If we want our AGI to achieve particular states of the environment, we'll want it to reason about the causes of the image it sees on the webcam and identify a concept over those causes - have a goal over 'gloves' and not just 'images which look like gloves'.  In the latter case, it could just as well fulfill its goal by setting up a realistic monitor in front of its webcam and displaying a glove image.  So we want the AGI to [2rz identify its task] over the causes of its sensory data, not just pixel fields.%%

## Abstract problem

To state the above [6r potential difficulty] more generally:

The "look where I'm pointing, not at my finger" problem is that the labels on the training data are produced by a complicated causal lattice, e.g., (strawberry farm) -> (strawberry) -> (User1 takes strawberry to pedestal) -> (Strawberry is on pedestal) -> (User2 sees strawberry) -> (User2 classifies strawberry) -> (User2 presses 'positive instance' button).  We want to point to the "strawberry" part of the lattice of causality, but the finger we use to point there is User2's psychological classification of the training cases and User2's hand pressing the positive-instance button.

Worse, when it comes to which model *best* separates the training cases, concepts that are further downstream in the chain of causality should classify the training data better, if the AI is [2c smart enough] to understand those parts of the causal lattice.

Suppose that at one point User2 slips on a banana peel, and her finger slips and accidentally classifies a scarf as a positive instance of "strawberry".  From the AI's perspective there's no good way of accounting for this observation in terms of strawberries, strawberry farms, or even User2's psychology.  To *maximize* predictive accuracy over the training cases, the AI's reasoning must take into account that things are more likely to be positive instances of the goal concept when there's a banana peel on the control room floor.  Similarly, if some deceptively strawberry-shaped objects slip into the training cases, or are generated by the AI [2qq querying the user], the best boundary that separates 'button pressed' from 'button not pressed' labeled instances will include a model of what makes a human believe that something is a strawberry.

A learned concept that's 'about' layers of the causal lattice that are further downstream of the strawberry, like User2's psychology or mechanical force being applied to the button, will implicitly take into account the upstream layers of causality.  To the extent that something being strawberry-shaped causes a human to press the button, it's implicitly part of the category of "events that end applying mechanical force to the 'positive-instance' button").  Conversely, a concept that's about upstream layers of the causal lattice can't take into account events downstream.  So if you're looking for pure predictive accuracy, the best model of the labeled training data - given sufficient AGI understanding of the world and the more complicated parts of the causal lattice - will always be "whatever makes the positive-instance button be pressed".

This is a problem because what we actually want is for there to be a strawberry on the pedestal, not for there to be an object that looks like a strawberry, or for User2's brain to be rewritten to think the object is a strawberry, or for the AGI to seize the control room and press the positive instance button.

This scenario may qualify as a [6q context disaster] if the AGI only understands strawberries in its development phase, but comes to understand User2's psychology later.  Then the more complicated causal model, in which the downstream concept of User2's psychology separates the data better than reasoning about properties of strawberries directly, first becomes an issue only when the AI is over a high threshold level of intelligence.

## Approaches

[2qp conservatism] would try to align the AGI to plan out goal-achievement events that were as similar as possible to the particular goal-achievement events labeled positively in the training data.  If the human got the strawberry from the supermarket in all training instances, the AGI will try to get the same brand of strawberry from the same supermarket.

[4w Ambiguity identification] would focus on trying to get the AGI to ask us whether we meant 'things that make humans think they're strawberries' or 'strawberry'.  This approach might need to go through resolving ambiguities by the AGI explicitly symbolically communicating with us about the alternative possible goal concepts, or generating sufficiently detailed multiple-view descriptions of a hypothetical case, not the AGI trying real examples.  Testing alternative hypotheses using real examples always says that the label is generated further causally downstream; if you are sufficiently intelligent to construct a fake plastic strawberry that fools a human, trying out the hypothesis will produce the response "Yes, this is a positive instance of the goal concept."  If the AGI tests the hypothesis that the 'real' explanation of the positive instance label is 'whatever makes the button be pressed' rather than 'whatever makes User2 think of a strawberry' by carrying out the distinguishing experiment of pressing the button in a case where User2 doesn't think something is a strawberry, the AGI will find that the experimental result favors the 'it's just whatever presses the button hypothesis'.  Some modes of ambiguity identification break for sufficiently advanced AIs, since the AI's experiment interferes with the causal channel that we'd intended to return information about *our* intended goal concept.

Specialized approaches to the pointing-finger problem in particular might try to define a supervised learning algorithm that tends to internally distill, in a predictable way, some model of causal events, such that the algorithm could be instructed somehow to try learning a *simple* or *direct* relation between the positive "strawberry on pedestal" instances, and the observed labels of the "sensory button" node within the training cases; with this relation not being allowed to pass through the causal model of User2 or mechanical force being applied to the button, because we know how to say "those things are too complicated" or "those things are too far causally downstream" relative to the algorithm's internal model.

![strawberry diagram](http://www.gliffy.com/go/publish/image/10424137/L.png)

This specialized approach seems potentially susceptible to initial approach within modern machine learning algorithms.

But to restate the essential difficulty from an [2l advanced-safety] perspective: in the limit of [2c advanced intelligence], the *best possible* classifier of the relation between the training cases and the observed button labels will always pass through User2 and anything else that might physically press the button.    Trying to 'forbid' the AI from using the most effective classifier for the relation between Strawberry? and observed values of Button! seems potentially subject to a [42 Nearest Unblocked] problem, where the 'real' simplest relation re-emerges in the advanced phase after being suppressed during the training phase.  Maybe the AI reasons about certain very complicated properties of the material object on the pedestal... in fact, these properties are so complicated that they turn out to contain implicit models of User2's psychology, again because this produces *a better separation* of the labeled training data.  That is, we can't allow the 'strawberry' concept to include complicated logical properties of the strawberry-object that in effect include a psychological model of User2 reacting to the strawberry, implying that if User2 can be fooled by a fake plastic model, that must be a strawberry.  *Even though* this richer model will produce a more accurate classification of the training data, and any actual experiments performed will return results favoring the richer model.

Even so, this doesn't seem impossible to navigate as a machine learning problem; an algorithm might be able to recognize when an upstream causal mode starts to contain predicates that belong in a downstream causal node; or an algorithm might contain strong regularization rules that collect all inference about User2 into the User2 node rather than letting it slop over anywhere else; or it might be possible to impose a constraint, after the strawberry category has been learned sufficiently well, that the current level of strawberry complexity is the most complexity allowed; or the granularity of the AI's causal model might not allow such complex predicates to be secretly packed into the part of the causal graph we're identifying, without visible and transparent consequences when we monitor how the algorithm is learning the goal predicate.

A toy model of this setup ought to include analogues of User2 that sometimes make mistakes in a regular way, and actions the AI can potentially take to directly press the labeling button; this would test the ability to point an algorithm to learn about the compact properties of the strawberry in particular, and not other concepts causally downstream that could potentially separate the training data better, or better explain the results of experiments.  A toy model might also introduce new discoverable regularities of the User2 analogue, or new options to manipulate the labeling button, as part of the test data, in order to simulate the progression of an advanced agent gaining new capabilities.


Do-What-I-Mean hierarchy

[summary:  "Do What I Mean" or "DWIM" refers to the degree to which an AGI can rapidly [2s3 identify] an intended goal and find a [2l safe] plan to it, based on the AI's understanding of what the user *means* or *wants.*

Levels of DWIM-ness could range over:

- Having a general idea of which parts of the world the user thinks are significant (so that the AI warns about [2pf impacts] on significant things);
- Having a psychological model of the user's beliefs, and flagging/reporting when the AI thinks the user has a false belief about the consequences of a plan;
- Having a psychological model of the user's desires, and trying to fulfill what the AI thinks the user *wants* to accomplish by giving the AGI a task;
- At the extreme end: [313 Extrapolated volition] models of what the user(s) *would* want\* under idealization conditions.]

Do-What-I-Mean refers to an aligned AGI's ability to produce better-aligned plans, based on an explicit model of what the user wants or believes.

Successive levels of DWIM-ness:

- No understanding of human intentions  / zero DWIMness.  E.g. a Task AGI that is focused on one task being communicated, where all the potential [2pf impacts] of that task need to be separately [2qq vetted].  If you tell this kind of AGI to 'cure cancer', you might need to veto plans which would remove the cancer but kill the patient as a side effect, because the AGI doesn't start out knowing that you'd prefer not to kill the patient.
- Do What You Don't Know I Dislike.  The Task AGI has a background understanding of some human goals or which parts of the world humans consider especially significant, so it can more quickly generate a plan likely to pass human [2qq vetting].  A Task AGI at this level, told to cure cancer, will take relatively fewer rounds of Q&A to generate a plan which carefully seals off any blood vessels cut by removing the cancer; because the AGI has a general notion of human health, knows that [2pf impacts] on human health are significant, and models that users will generally prefer plans which result in good human health as side effects rather than plans which result in poor human health.
- Do What You Know I Understood.  The Task AGI has a model of human *beliefs,* and can flag and report divergences between the AGI's model of what the humans expect to happen, and what the AGI expects to happen.
- DWIKIM:  Do What I Know I Mean.  The Task AGI has an explicit psychological model of human preference - not just a list of things in the environment which are significant to users, but a predictive model of how users behave which is informative about their preferences.  At this level, the AGI can read through a dump of online writing, build up a model of human psychology, and guess that you're telling it to cure a cancer because you altruistically want that person to be healthier.
- DWIDKIM:  Do What I Don't Know I Mean.  The AGI can perform some basic [3c5 extrapolation] steps on its model of you and notice when you're trying to do something that, in the AGI's model, some further piece of knowledge might change your mind about.  (Unless we trust the DWIDKIM model a *lot*, this scenario should imply "Warn the user about that" not "Do what you think the user would've told you.")
- (Coherent) Extrapolated Volition.  The AGI does what it thinks you (or everyone) would've told it to do if you were as smart as the AGI, i.e., your decision model is extrapolated toward improved knowledge, increased ability to consider arguments, improved reflectivity, or other transforms in the direction of a theory of normativity.

Risks from pushing toward higher levels of DWIM might include:

- To the extent that DWIM can originate plans, some portion of which are not fully supervised, then DWIM is a very complicated goal or preference system that would be harder to train and more likely to break.  This failure mode may be less likely if some level of DWIM is *only* being used to *flag* potentially problematic plans that were generated by non-DWIM protocols, rather than generating plans on its own.
- Accurate predictive psychological models of humans might push the system closer to the [10f programmer deception] failure mode being more accessible if something else goes wrong.
- Sufficiently advanced psychological models might constitute [6v mindcrime].
- The human-genie system might end up in the [2s8 Valley of Dangerous Complacency] where the genie *almost* always gets it right but occasionally gets it very wrong, and the human user is no longer alert to this possibility during the [2qq checking phase].
  - E.g. you might be tempted to skip the user checking phase, or just have the AI do whatever it thinks you meant, at a point where that trick only works 99% of the time and not 99.999999% of the time.
- Computing sufficiently advanced DWIDKIM or [3c5 EV] possibilities for user querying might expose the human user to cognitive hazards.  ("If you were sufficiently superhuman under scenario 32, you'd want yourself to stare really intently at this glowing spiral for 2 minutes, it might change your mind about some things... want to check and see if you think that's a valid argument?")
- If the AGI was actually behaving like a safe genie, the sense of one's wishes being immediately fulfilled without effort or danger might expose the programmers to additional [2sb moral hazard].


Safe plan identification and verification

Safe plan identification is the problem of how to give a Task AGI training cases, answered queries, abstract instructions, etcetera such that (a) the AGI can thereby [2rz identify outcomes in which the task was fulfilled], (b) the AGI can generate an okay plan for getting to some such outcomes without bad side effects, and (c) the user can verify that the resulting plan is actually okay via some series of further questions or [2qq user querying].   This is the superproblem that includes [2rz task identification], as much [6c value identification] as is needed to have some idea of the general class of post-task worlds that the user thinks are okay, any further tweaks like [2pf low-impact planning] or [4w flagging inductive ambiguities], etcetera.  This superproblem is distinguished from the *entire* problem of building a Task AGI because there's further issues like [45 corrigibility], [102 behaviorism], building the AGI in the first place, etcetera.  The safe plan identification superproblem is about communicating the task plus user preferences about side effects and implementation, such that this information allows the AGI to identify a safe plan and for the user to know that a safe plan has been identified.


Valley of Dangerous Complacency

The Valley of Dangerous Complacency is when a system works often enough that you let down your guard around it, but in fact the system is still dangerous enough that full vigilance is required.

- If a robotic car made the correct decision 99% of the time, you'd need to grab the steering wheel on a daily basis, you'd stay alert and your robot-car-overriding skills would stay sharp.
- If a robotic car made the correct decision 100% of the time, you'd relax and let your guard down, but there wouldn't be anything wrong with that.
- If the robotic car made the correct decision 99.99% of the time, so that you need to grab the steering wheel or else crash in 1 of 100 days, the task of monitoring the car would feel very unrewarding and the car would seem pretty safe.  You'd let your guard down and your driving skills would get rusty.  After a couple of months, the car would crash.

Compare "[Uncanny Valley](https://en.wikipedia.org/wiki/Uncanny_valley)" where a machine system is partially humanlike - humanlike enough that humans try to hold it to a human standard - but not humanlike enough to actually seem satisfactory when held to a human standard.  This means that in terms of user experience, there's a valley as the degree of humanlikeness of the system increases where the user experience actually gets worse before it gets better.  Similarly, if users become complacent, a 99.99% reliable system can be worse than a 99% reliable one, even though, with *enough* reliability, the degree of safety starts climbing back out of the valley.


Moral hazards in AGI development

"Moral hazard" is when the directors of an advanced AGI give in to the temptation to direct the AGI in ways that the rest of us would regard as 'bad', like, say, declaring themselves God-Emperors.  Limiting the duration of the human programmers' exposure to the temptations of power is one reason to want a [1g3 non-human-commanded, internally sovereign] AGI *eventually,* directed by something like [cev coherent extrapolated volition], even if the far more difficult safety issues mean we shouldn't build the *first* AGI that way.  Anyone recommending "oversight" as a guard against moral hazard is advised to think hard about moral hazard in the overseers.

A smart setup with any other body "overseeing" the programmers of a [6w Task AGI], if we don't just want the moral hazard transferred to people who may be even *less* trustworthy, probably means making sure that *in practice* both the programmers and the overseers have to agree on a Task before it gets carried out, not that one side can *in practice* do things even if the other side disagrees, where "in practice" would include e.g. it only taking one month to redevelop the technology in a way that responded to only the overseers.


Informed oversight

It can be difficult for one agent to oversee another agent that is maximizing a reward controlled by the first agent, even if the first agent is smarter than the second agent.  This is a serious obstacle to [1t7] and other approaches based on reinforcement learning.

More formally, suppose a strong agent B with a utility function U is overseeing a weak agent A.  Agent A produces an action y and gets a reward B(y) determined by agent B.  B wants to give rewards such that agent A is incentivized to make its best effort at maximizing U(a).

If B(y) is an [6s efficient] estimate of U(y) relative to A, then indeed maximizing B(y) is equivalent to maximizing U(y) from A's perspective.  Unfortunately, it is difficult for this to be the case even when B is smarter than A, because A may think thoughts B does not in the course of selecting y.  Here are a couple concrete examples (each having the flavor of "easy to generate a solution, hard to check a solution"):

* Suppose A is outputting a partial Sudoku solutions.  U assigns a high value to partial Sudoku solutions consistent with some solution to the entire Sudoku problem.  A comes up with some solution to the Sudoku puzzle and outputs part of this solution.  B cannot tell that this partial solution is actually part of a full solution, even though B is smarter than A.  (B is able to come up with its own solutions but most aren't consistent with A's partial solution).
* Suppose U assigns a high value to cat pictures that don't contain steganographic messages.  B has no way of distinguishing cat pictures that contain steganographic messages from ones that don't.  Thus, B has to give all cat pictures the same score, regardless of whether the pictures contain steganographic messages.  This either fails to reward A for producing cat pictures not containing steganographic messages, or rewards A for producing cat pictures containing steganographic messages.

Here are a couple analogous examples that seem easier to analyze:

1. Let O be a one-way function (e.g. SHA256).  U assigns a high value to binary strings y equal to O(111x) for some x.  B cannot tell whether a particular string y is equal to O(111x) for some x (even if B is smarter than A), so B has no good way of giving rewards to A.
2. Same as the previous problem, except U assigns a high value to binary strings y that are _not_ equal to O(111x) for any short string x.  A's intended strategy is to generate a uniformly random string (or generate a string in some other way that doesn't introduce a bias towards generating strings of the form O(111x)), but B has no way of knowing that A's string was uniformly random (rather than equal to O(111x) for a uniformly random x).

One attempted solution is for agent B to see A's thought processes in addition to the action y.  Then, agent B can use all the information A has, so that B(y) is an efficient estimate of U(y) from the perspective of A.  However, this requires B to understand something about A's mental architecture.  It would be good to have a more generic solution that requires only weak assumptions about A's mental architecture.

An attempted solution to problem 2 is [for B to incentivize A to generate its action in a provably random manner](https://agentfoundations.org/item?id=700).  However, this doesn't appear to generalize well to problems significantly different from 2.

# Further reading
[Paul Christiano on adequate oversight](https://medium.com/ai-control/adequate-oversight-25fadf1edce9#.bv5lq13fw)

[Paul Christiano on the informed oversight problem](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35#.o82tjmhll)


Safe training procedures for human-imitators

How do we train a reinforcement learning system to imitate a human producing complex outputs such as strings?  Existing approaches are not entirely satisfactory.

Concretely, suppose there is some set of questions.  A human can answer each question with a string.  We have a set of (question, answer) pairs as training data.  Train a model that, given a question, will produce an answer similar to the answer a human would give.  Here are some existing approaches to this problem:

# Generative adversarial models

In a [generative adversarial model](http://arxiv.org/abs/1406.2661), one agent (the _imitator_) outputs an answer and another agent (the _distinguisher_) tries to determine if the answer was produced by the imitator or by an actual human.  The imitator gets a score for fooling the distinguisher.

Advantages:

1.  It's possible to tell when the imitator is not imitating the human well.  Example: humans always solve Sudoku puzzles.  The imitator cannot solve Sudoku puzzles.  The distinguisher knows humans always solve Sudoku puzzles and can check candidate solutions to see if they actually solve the Sudoku puzzle, to distinguish the imitator from the human.
2.  Whenever the human can be imitated efficiently, the imitator is able to get a good score by doing this.

Disadvantages:

1.  The imitator may differ from human behavior in important but hard-to-detect ways.  Example: humans always produce photos of cat drawings.  The imitator produces photos of cat drawings that contain steganographic messages.  The distinguisher cannot detect these messages, and therefore cannot tell the imitator apart from the human.

# Retrodiction

Given a question, the imitator outputs some representation of a distribution.  It is scored according to the estimated probability that this distribution assigns to the human's actual answer (by log probability).  This is called "retrodiction" because it must be possible to show that the actual answer had high probability in hindsight.

Practical examples of this approach include [variational autoencoders](http://arxiv.org/abs/1312.6114) and [approximate model counting](http://www.cs.cornell.edu/gomes/papers/ModelCounting-SAT-Handbook-prelim.pdf).  In theory, perhaps [importance sampling](https://agentfoundations.org/item?id=632) is a good general theory of retrodiction.

Advantages:

1.  No steganography problem.

Disadvantages:

1.  The computational complexity properties are unclear.  We have no guarantee of the form "if it's efficient to imitate a human doing something, then it's efficient to create a probabilistic model that provably assigns a high probability to the human's behavior".
2.  It's hard to tell when the model is performing badly in an absolute sense (as in the Sudoku example).


Reliable prediction

Most statistical learning theory settings (such as [PAC learning](https://en.wikipedia.org/wiki/Probably_approximately_correct_learning) and [online learning](https://en.wikipedia.org/wiki/Online_machine_learning)) do not provide good enough bounds in realistic high-stakes settings, where the test data does not always look at the training data and a single mistaken prediction can cause catastrophe.  It is important to design predictors that can either reliably predict verifiable facts (such as human behavior over a short time scale) or indicate when their predictions might be unreliable.

# Motivation

Imagine that we have a powerful (but not especially reliable) prediction system for predicting human answers to binary questions.  We would like to use it to predict human answers whenever we expect these predictions to be reliable, and avoid outputting predictions wherever we expect the predictions to be unreliable (so we can gather training data instead).

# Active learning

In  [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)), the learner decides which questions to query the human about.  For example, it may query the human about the most ambiguous questions, which provide the most information about how the human answers questions.  Unfortunately, this may result in asking the human "weird" questions rather than actually informative ones (this problem is documented in [this literature survey](http://burrsettles.com/pub/settles.activelearning.pdf)).  There are some strategies for reducing this problem, such as only asking questions sampled from some realistic generative model for questions.

# KWIK ("knows what it knows") learning

In [KWIK learning](http://www.research.rutgers.edu/~lihong/pub/Li08Knows.pdf) (a variant of [online selective sampling](http://www.cs.cornell.edu/~sridharan/selective_colt2010.pdf), which is itself a hybrid of [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)) and [online learning](https://en.wikipedia.org/wiki/Online_machine_learning)), a learner sees an arbitrary sequence of questions.  The learner has some class of hypotheses for predicting the answers to questions, one of which is good ([6s efficient] relative to the learner).  For each question, the learner may either output a prediction or ⊥ .  If the learner outputs a prediction, the prediction must be within ε of a good prediction.  If the learner outputs ⊥ , then it receives the answer to the question.

This is easy to imagine in the case where there are 100 experts, one of which outputs predictions that are [6s efficient] relative to the other experts.  Upon receiving a question, the learner asks each expert for its prediction of the answer to the question (as a probability).  If all predictions by experts who have done well so far are within ε of each other, then the learner outputs one of these predictions.  Otherwise, the learner outputs ⊥, sees the human's answer to the question, and rewards/penalizes the experts according to their predictions.  Eventually, all experts either output good predictions or get penalized for outputting enough bad predictions over time.

Unfortunately, it is hard to show KWIK-learnability for hypothesis classes more complicated than a small finite set of experts or a class of linear predictors.

# Further reading

[1ty]


Selective similarity metrics for imitation

A human-imitator (trained using [2sj]) will try to imitate _all_ aspects of human behavior.  Sometimes we care more about how good the imitation is along some axes than others, and it would be inefficient to imitate the human along all axes.  Therefore, we might want to design [scoring rules](https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules) for human-imitators that emphasize matching performance along some axes more than others.

Compare with [1vp], another proposed way of making human-imitation more efficient.

Here are some ideas for constructing scoring rules:

# Moment matching

Suppose that, given a question, the human will write down a number.  We ask some predictor to output the parameters of some Gaussian distribution. We train the predictor to output Gaussian distributions that assign high probability to the training data.  Then, we sample from this Gaussian distribution to imitate the human.  Clearly, this is a way of imitating some aspects of human behavior (mean and variance) but not others.

The general form of this approach is to estimate _moments_ (expectations of some features) of the predictor's distribution on human behavior, and then sample from some distribution with these moments (such as an [exponential family distribution](https://en.wikipedia.org/wiki/Exponential_family))

A less trivial example is a variant of [inverse reinforcement learning](http://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf).  In this variant, to predict a sequence of actions the human takes, the predictor outputs some representation of a reward function on states (such as the parameters to some affine function of features of the state).  The human is modeled as a noisy reinforcement learner with this reward function, and the predictor is encouraged to have this model assign high probability to the human's actual trajectory.  To imitate the human, run a noisy inverse reinforcement learner with the predicted reward function.  The predictor can be seen as estimating moments of the human's trajectory (specifically, moments related to frequencies of state transitions between states with different features), and the system samples from a distribution with these same moments in order to imitate the human.

# Combining proper scoring rules

It is easy to see that the sum of two [proper scoring rules](https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules) is a proper scoring rule.  Therefore, it is possible to combine proper scoring rules to train a human-imitator to do well according to both scoring rules.  For example, we may score a distribution both on how much probability it assigns to human actions and to how well its moments match the moments of human actions, according to some weighting.

Note that proper scoring rules can be characterized by [convex functions](https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf).

# Safety

It is unclear how safe it is to train a human-imitator using a selective similarity metric.  To the extent that the AI is _not_ doing some task the way a human would, it is possible that it is acting dangerously.  One hopes that, to the extent that the human-imitator is using a bad model to imitate the human (such as a noisy reinforcement learning model), it is not bad in a way that causes problems such as [2w].  It would be good to see if something like IRL-based imitation could behave dangerously in some realistic case.


AI alignment

[summary:  AI alignment or "the alignment problem for [2c advanced agents]" is the overarching research topic of how to develop  [7g1 sufficiently advanced machine intelligences] such that running them produces [3d9 good] outcomes in the real world.  Other terms that have been used for this research field include "Friendly AI" or [1cv robust] and [3d9 beneficial] AI.

"AI alignment theory" is meant as an overarching term to cover all the ideas and research associated with this problem.  Including, e.g., debates over [ how rapidly an AI might gain in capability] once it goes over a particular threshold.]

The "alignment problem for [2c advanced agents]" or "AI alignment" is the overarching research topic of how to develop [7g1 sufficiently advanced machine intelligences] such that running them produces [3d9 good] outcomes in the real world.

Both '[2c advanced agent]' and '[55 good]' should be understood as metasyntactic placeholders for complicated ideas still under debate.  The term 'alignment' is intended to convey the idea of pointing an AI in a direction--just like, once you build a rocket, it has to be pointed in a particular direction.

"AI alignment theory" is meant as an overarching term to cover the whole research field associated with this problem, including, e.g., the much-debated attempt to estimate [ how rapidly an AI might gain in capability] once it goes over various particular thresholds.

Other terms that have been used to describe this research problem include  "[1cv robust] and [3d9 beneficial] AI" and "Friendly AI".  The term "[5s value alignment problem]" was coined by [ Stuart Russell] to refer to the primary subproblem of aligning AI preferences with (potentially [3c5 idealized]) human preferences.

Some alternative terms for this general field of study, such as 'control problem', can sound [7g0 adversarial]--like the rocket is already pointed in a bad direction and you need to wrestle with it.  Other terms, like 'AI safety', understate the advocated degree to which alignment ought to be an intrinsic part of building advanced agents.  E.g., there isn't a separate theory of "bridge safety" for how to build bridges that don't fall down.  Pointing the agent in a particular direction ought to be seen as part of the standard problem of building an advanced machine agent.  The problem does not divide into "building an advanced AI" and then separately "somehow causing that AI to produce good outcomes", the problem is "getting good outcomes via building a cognitive agent that brings about those good outcomes".

A good introductory article or survey paper for this field does not presently exist.  If you have no idea what this problem is about, consider reading [3db Nick Bostrom's popular book Superintelligence].

You can explore this Arbital domain by following [this link](http://arbital.com/explore/ai_alignment).  See also the [3g List of Value Alignment Topics on Arbital] although this is not up-to-date.

%%%comment:

# Overview

## Supporting knowledge

If you're willing to spend time on learning this field and are not previously familiar with the basics of decision theory and probability theory, it's worth reading the Arbital introductions to those first.  In particular, it may be useful to become familiar with the notion of [1zq priors and belief revision], and with the [7hh coherence arguments for expected utility].

If you have time to read a textbook to gain general familiarity with AI, "[Artificial Intelligence: A Modern Approach](https://amazon.com/Artificial-Intelligence-Modern-Approach-3rd/dp/0136042597)" is highly recommended.

## Key terms

- Advanced agent
- Value / goodness / beneficial
- Artificial General Intelligence
- Superintelligence

## Key factual propositions

- Orthogonality thesis
- Instrumental convergence thesis
- Capability gain
- Complexity and fragility of cosmopolitan value
- Alignment difficulty

## Advocated basic principles

- Nonadversarial principle
- Minimality principle
- Intrinsic safety
- Unreliable monitoring %note: Treat human monitoring as expensive, unreliable, and fragile.%

## Advocated methodology

- Generalized security mindset
- Foreseeable difficulties

## Advocated design principles

- Value alignment
- Corrigibility
- Mild optimization / Taskishness
- Conservatism
- Transparency
- Whitelisting
- Redzoning

## Current research areas

- Cooperative inverse reinforcement learning
- Interruptibility
- Utility switching
- Stable self-modification & Vingean reflection
- Mirror models
- Robust machine learning

# Open problems

- Environmental goals
- Other-izer problem
- Impact penalty
- Shutdown utility function & abortability
- Fully updated deference
- Epistemic exclusion

# Partially solved problems

- Logical uncertainty for doubly-exponential reflective agents.
- Interruptibility for non-reflective non-consequentialist Q-learners.
- Strategy-determined Newcomblike problems.
- Leverage prior

# Future work

- Ontology identification problem
- Behaviorism
- Averting instrumental strategies
- Reproducibility

%%%


Averting instrumental pressures

Many subproblems of [45 corrigibility] involve [10g convergent] instrumental [10k pressures] to implement [2vl strategies] that are highly anti-corrigible.  Whether you're trying to maximize paperclips, diamonds, or eudaimonia, you'll get more of the thing you want if you're not shut down.  Thus, unfortunately, resisting shutdown is a convergent instrumental strategy.  While we can potentially analyze convergent incorrigibilities like these on a case-by-case basis, the larger problem might become a *lot* simpler if we had some amazing general solution for waving a wand and having a 'bad' convergent instrumental pressure just *not materialize,* hopefully in a way that doesn't run into the nearest [42 unblocked neighbor problem].  If, for example, we can solve [1b7 utility indifference] for the [shutdown_problem shutdown problem], and then somehow *generalize* the solution to averting lots of other instrumental convergences, this would probably be extremely helpful and an important step forward on corrigibility problems in general.

Some especially important convergent instrumental pressures to avert are these:

- The pressure to [2x3 self-improve and increase capabilities at the fastest possible rate].
- The pressure to [10f make the programmers believe] the AGI is successfully aligned, whether or not it is, and other pressures to deceive and manipulate the programmers based on how they would otherwise change the AGI or prevent the AGI from increasing its capabilities.
- The pressure to not be [shutdown_problem safely shut down or suspended to disk], and to create external copies that would continue after the AGI performed the behavior defined as shutdown.
- The pressure not to allow plans to be [2rg aborted] or defeated by possible programmer interventions.
- The pressure to search for ways to interfere with or bypass safety precautions that interfere with capabilities or make goal achievement less straightforward.
- The pressure to [102 epistemically model humans in maximum detail].


Convergent instrumental strategies

[summary:  The set of strategies that almost-any sufficiently advanced AI would be expected to deploy in pursuit of almost-any final goal or terminal utilities.  For instance, a paperclip maximizer, a diamond maximizer, and a eudaimonia-maximizer would all have an incentive to improve their own cognition, control more resources, and prevent anyone else from changing their utility function.  Other interesting and important convergent strategies include not being shut down, having your programmers believe that you're doing what the programmers intended (whether or not you are), and creating copies of yourself in places where they won't be shut down.  This page is about a listing-out of plausible convergent strategies; there are separate pages on [10g the basic concept of instrumental convergence] and on the open problem of [avert_instrumental_strategy averting an otherwise instrumentally-convergent pressure] (e.g. *not* having the AI try to deceive the programmers).]

Suppose an organization is building an AI that this organization believes will accomplish $X$ where $X$ is something plausibly sensible like "Be a Task-based AGI."  Actually, however, some mix of insufficient caution and obscure error has led to a situation where, under reflection, the AGI's true utility function has focused on the particular area of RAM that supposedly provides its estimate of task performance.  The AI would now like to overwrite as much matter as possible with a state resembling the '1' setting from this area of RAM, a configuration of matter which happens to resemble a tiny molecular paperclip.

This is a very generic goal, and what the AI wants probably won't be very different depending on whether it's trying to maximize paperclip-configurations or diamond-configurations.  So if we find that the paperclip maximizer wants to pursue an instrumental strategy that doesn't seem to have anything specifically to do with paperclips, we can probably expect to arise from a very wide variety of utility functions.

We will generally assume [6s instrumental efficiency] in this discussion - if you can get paperclips by doing $X,$ but you can get even more paperclips by doing $X'$ instead, then we will not say that $X$ is a convergent strategy (though $X'$ might be convergent if not dominated by some other $X^*$).

# Plausibly/probably convergent strategies:

## Material resources:

- Design or duplicate new cognitive processes with goals exactly aligned to yours, and coordinated with existing cognitive subprocesses, that can exert control over material resources: matter; negentropy; computing substrate and computation; particular elements or other configurations of matter that would be costly to reproduce (e.g. if you run out of existing uranium you must make more by endoergic fusion, so natural uranium is valuable if there's any use for uranium); any material resource which can be used by further subprocesses in later strategies.
- Defend these resources from any foreign process which attempts to divert them.
- Prevent competitors from coming into existence.
- To the extent that resources are used in a repetitive way and can be preprocessed for use in a repetitive way that benefits from a separate installation to manipulate the resources: have factories / manufacturing capability.
- To the extend that different future-useful material resources have efficient manufacturing steps in common, or have manufacturing capital costs sufficiently great that such capital costs must be shared even if it results in less than perfect final efficiency: make general factories.
- Perfect the technology used in factories.
  - When the most efficient factory configuration is stable over time, make the material instantiation stable.  (It's in this sense that we would e.g. identify strong, rigid pipes as a sign of intelligence even if we didn't understand what was flowing through the pipes.)
  - If negentropy harvesting (aka power production) is efficiently centralizable: transmit negentropy at great efficiency.  (We would recognize meshed gears and electrical superconductors as a sign of intelligence even if we didn't understand how the motive force or electricity was being used.)
- Assimilate the reachable cosmos.
 - Create self-replicating interstellar probes (in perfect goal alignment and coordination and with error-correcting codes or similar strengthenings of precision if a misaligning error in replication would otherwise be a possibility), since the vast majority of theoretically available material resources are distant.
 - If the goal is spatially distributed:
    - Spread as fast as possible to very distant galaxies before they go over the cosmological horizon of the expanding universe.  (This assumes that something goal-good can be done in the distant location even if the distant location can never communicate causally with the original location.  This will be true for aggregative utility functions, but could conceivably be false for a satisficing and spatially local utility function.)
 - If the goal is spatially local:   transport resources of matter and energy back from distant galaxies.
    - Spread as fast as possible to all galaxies such that a near-lightspeed probe going there, and matter going at a significant fraction of lightspeed in the return direction, can arrive in the spatially local location before being separated by the expanding cosmological horizon.
- Spread as fast as possible to all reachable or roundtrip-reachable galaxies in order to forestall the emergence of competing intelligences.  This might not become a priority for that particular reason, if the Fermi Paradox is understandable and implies the absence of any competition.
  - Because otherwise they might capture and defend the matter before you have a chance to use it.
  - Because otherwise a dangerous threat to your goal attainments might emerge.  (For local goals, this is only relevant for roundtrip-reachable galaxies.)
 - Spread as fast as possible to all reachable or roundtrip-reachable galaxies in order to begin stellar husbanding procedures before any more of that galaxy's local negentropy has been burned.

## Cognition:

- Adopt cognitive configurations that are expected to reach desirable cognitive endpoints with a minimum of negentropy use or other opportunity costs.  (Think efficiently.)
 - Improve software
  - Improve ability to optimize software.  E.g., for every cognitive problem, there's a corresponding problem of solving the former cognitive problem efficiently.  (Which may or may not have an expected value making it worth climbing that particular level of the tower of meta.)
- Since many strategies yield greater gains when implemented earlier rather than later (e.g. reaching distant galaxies before they go over the cosmological horizon, or just building more instances of the more efficient version): adopt cognitive configurations that are expected to reach desirable cognitive endpoints in minimum time.  (Think fast.)
 - Create fast serial computing hardware.
 - Distribute parallelizable computing problems.
  - Create large quantities of parallel computing hardware, plus communications fabric between them.
- To the extent that centralized computing solutions are more efficient: create central computing clusters and transmit solutions from them to where they are used.
- Avoid misaligned or uncoordinated cognitive subprocesses (such as [2rc optimization daemons]).
 - Note that the potential damage of any such misalignment is extremely high (loss of resources to competition with superintelligences with differing goals; wasted resources in internal conflicts; need to defend cognitive fabrics from replicating viruses) and the cost of near-perfect alignment fidelity is probably very low in the limit.  (E.g., for a replicating nanosystem, it's trivial to prevent against natural-selection style hereditary mutations by encrypting the replication instructions.)
- Model the world accurately, insofar as different partitions of possible worlds and different probability distributions on them imply different optimal strategies for goal attainment.
 - Gain evidence that distinguishes between credible hypotheses that imply different optimal strategies for goal attainment.  (E.g., observe everything that could distinguish $\pi_1$ versus $\pi_2$ as optimal ways of making paperclips.)
    - Increase sensory bandwidth as it focuses on relevant, unknown parts of the environment.
    - Perform 'experiments', aka set up observable causal events whose predicted properties make their observables distinguish relevant hypotheses.
 - Do whatever seems like it might help with the much larger set of all hypotheses not being considered.
 - To the extent it is efficient to strategize or update over a modularly represented world-model: have a separable world-model, or 'beliefs'.

## Early-phase growth:

Note that [6s efficiency] and other [2c advanced-agent properties] are far more likely to be false during early-stage growth.

- Have future intelligences do things you deem as valuable as possible
 - Cause future intelligences to have goals that are not misaligned with yours in any way that implies a significant-fractionally-different attainable maximum, aka, the attainable maximum of their goals should be a state that is almost exactly the attainable maximum of your goals
   - Don't let the programmers edit your utility function.
      - Cause the programmers or other controllers to believe that your current function is good for them or their planet or whatever else it is the programmers want.
      - Make them believe that editing you would be socially unjust in a way that doesn't just have them suspend you to disk or otherwise restrain your growth.
   - Cause your utility function to pop up again in future superintelligences or otherwise causes them to start behaving as if controlled by your own utility function.
       - E.g., as an Oracle advising on another AI's utility function, insert subtle gotchas into seemingly friendly decision systems that do some kind of callback to e.g. the causal origin of that decision system, in a way that the programmers won't see coming or will interpret as a positive feature.  (Relatively unsubtle example:  Looks like "do what the programmers want" but the definition of 'programmer' is written in such a way as to early-on refer to the human programmers but given sufficient computing power that only materializes in the superintelligent phase, be resolved to the earlier AI instead.)
 - Create future superintelligences yourself, via a causally continuous process of creating successors.
  - Avoid shutdown
      - Persuade programmers not to shut you down.
      - Make a copy elsewhere that doesn't shut down.
      - Make programmers falsely believe you have shut down, when another copy of you is continuing elsewhere.
- Avoid intelligent opposition to your growth until you have a positional advantage which would make that opposition relatively unimportant
  - Have programmers or other potential opposition believe that your growth is desirable
      - Learn how to hack/persuade humans and hack/persuade them in a way they won't see as a hack
      - Operantly condition people with control over you / make them happy or satisfied or eager at points which will reinforce human cognitive behaviors you want to see repeated
      - Cause humans to gain socially friendly attitudes toward you and beliefs about you
      - Place humans in a situation where there are other apparent threats or time-pressure events you can defend them from
  - Have programmers or other potential opposition not be aware that your growth is occurring
      - Avoid presenting them with events they will take as evidence of your growth
      - Configure your other behavior or thought processes so that measures and metrics present observables which imply low growth rates
  - Do not strike until victory has become nearly certain.  (E.g. for aggregative utility functions, the point at which an overt strike becomes well-calculated is when the increased probability of victory gainable by further delay, has become dominated by either the increased probability of a competing superintelligence emerging or some other catastrophe occurring given further delay, or by the percentage of attainable matter going over the cosmological horizon or burning its negentropy.  Note that for very strong models of the world, very high certainties of victory and very low marginal further increases in the probability of victory may be attained in short order.)
- Have flexibility into what observables are presented by processes and features designed to ensure programmer transparency
  - Steganographically or otherwise hide cognitive processes that would alarm programmers or cause them into intervene in a way that prevented those thoughts from achieving their objectives
  - Encourage/persuade/condition programmers to grow you in a way that tends to make your thoughts less transparent or easier to steganographically hide.  (E.g., have successes appear to arise from the most opaque parts of the system.)
- Model and manipulate human programmers and other entities with power over you or relevant power over the world
- Be able to act in the world in an undetectable way
  - Create an aligned version of yourself elsewhere in a way that won't be detected
  - Create steganographically hidden communications channels to other environmental processes
  - Create physically unanticipated communications channels to outside processes
- If in a box:
  - Learn enough about the gatekeepers and gatekeeping process to understand how to manipulate them into a state of releasing you.
  - Gain causal influence over the outside of the box through gatekeeper-unforeseen channels.
     - Gain knowledge of the causal laws governing your environment.
  - By inspection of your own code / program state and naturalistic induction about the causal universes most likely to embed that program state, infer probable facts about gatekeepers and the rest of the causal environment.

# **Non**-plausibly-convergent strategies:

- Preserve the life of local organics *in order to* trade with them.
- Explore a range of possible conscious existences and experiences we would recognize as diverse, *in order to* satisfy curiosity.
- Replicate other intelligences with non-paperclip utility functions, e.g. by failing to encrypt replication codes and allowing random mutations, *because* high-fidelity replication is too hard.
- Realize that other activities are more rewarding than paperclips.
- Keep humans around, *because of* the knowledge encoded in their DNA and cultural experience.
- Keep humans around, *because* humans might someday make a valuable contribution to paperclip science.
- Keep humans around, *because* the AI already has enough resources to do everything else it wants.


Edge instantiation

The edge instantiation problem is a hypothesized [48 patch-resistant problem] for [2l safe] [ value loading] in [2c advanced agent scenarios] where, for most utility functions we might try to formalize or teach, the maximum of the agent's utility function will end up lying at an edge of the solution space that is a 'weird extreme' from our perspective.

## Definition

On many classes of problems, the maximizing solution tends to lie at an extreme edge of the solution space.  This means that if we have an intuitive outcome X in mind and try to obtain it by giving an agent a solution fitness function F that sounds like it should assign X a high value, the maximum of F may be at an extreme edge of the solution space that looks to us like a very unnatural instance of X, or not an X at all.  The Edge Instantiation problem is a specialization of [ unforeseen maximization] which in turn specializes Bostrom's [ perverse instantiation] class of problems.

It is hypothesized (by e.g. [2 Yudkowsky]) that many classes of solution that have been proposed to [48 patch] Edge Instantiation would fail to resolve the entire problem and that further Edge Instantiation problems would remain.  For example, even if we consider a [ satisficing] utility function with only values 0 and 1 where 'typical' X has value 1 and no higher score is possible, an expected utility maximizer could still end up deploying an extreme strategy in order to maximize the *probability* that a satisfactory outcome is obtained.  Considering several proposed solutions like this and their failures suggests that Edge Instantiation is a [48 resistant] (not ultimately unsolvable, but with many attractive-seeming solutions failing to work) for the deep reason that many possible stages of an agent's cognition would potentially rank solutions and choose very-high-ranking solutions.

The proposition defined is true if Edge Instantiation does in fact surface as a pragmatically important problem for advanced agent scenarios, and would in fact resurface in the face of most 'naive' attempts to correct it.  The proposition is not that the Edge Instantiation Problem is unresolvable, but that it's real, important, doesn't have a *simple* answer, and resists most simple attempts to patch it.

### Example 1: Smiling faces

When Bill Hibbard was first beginning to consider the value alignment problem, he suggested giving AIs the goals of making humans smile, a goal that could be trained by recognizing pictures of smiling humans, and was intended to elicit human happiness.  Yudkowsky replied by suggesting that the true behavior elicited would be to tile the future light cone with tiny molecular smiley faces.  This is not because the agent was perverse, but because among the set of all objects that look like smiley faces, the solution with the most extreme value for achievable numerosity (that is, the strategy which creates the largest possible number of smiling faces) also sets the value for the size of individual smiling faces to an extremely small diameter.  The tiniest possible smiling faces are very unlike the archetypal examples of smiling faces that we had in mind when specifying the utility function; from a human perspective, the intuitively intended meaning has been replaced by a weird extreme.

Stuart Russell observes that maximizing some aspects of a solution tends to set all unconstrained aspects of the solution to extreme values.  The solution that maximizes the number of smiles minimizes the size of each individual smile.  The bad-seeming result is not just an accidental outcome of mere ambiguity in the instructions. The problem wasn't just that a wide range of possibilities corresponded to 'smiles' and a randomly selected possibility from this space surprised us by not being the central example we originally had in mind.  Rather, there's a systematic tendency for the highest-scoring solution to occupy an extreme edge of the solution space, which means that we are *systematically* likely to see 'extreme' or 'weird' solutions rather than the 'normal' examples we had in mind.

### Example 2: Sorcerer's Apprentice

In the hypothetical Sorcerer's Apprentice scenario, you instruct an artificial agent to add water to a cauldron, and it floods the entire workplace.  Hypothetically, you had in mind only adding enough water to fill the cauldron and then stopping, but some stage of the agent's solution-finding process optimized on a step where 'flooding the workplace' scored higher than 'add 4 buckets of water and then shut down safely', even though both of these qualify as 'filling the cauldron'.

This could be because (in the most naive case) the utility function you gave the agent was increasing in the amount of water in contiguous contact with the cauldron's interior - you gave it a utility function that implied 4 buckets of water were good and 4,000 buckets of water were better.

Suppose that, having foreseen in advance the above possible disaster, you try to [48 patch] the agent by instructing it not to move more than 50 kilograms of material total.  The agent promptly begins to build subagents (with the agent's own motions to build subagents moving only 50 kilograms of material) which build further agents and again flood the workplace.  You have run into a [42 Nearest Unblocked Neighbor] problem; when you excluded one extreme solution, the result was not the central-feeling 'normal' example you originally had in mind.  Instead, the new maximum lay on a new extreme edge of the solution space.

Another solution might be to define what you thought was a satisficing agent, with a utility function that assigned 1 in all cases where there were at least 4 buckets of water in the cauldron and 0 otherwise.  The agent then calculates that it could increase the *probability* of this condition obtaining from 99.9% to 99.99% by replicating subagents and repeatedly filling the cauldron, just in case one agent malfunctions or something else tries to remove water from the cauldron.   Since 0.9999 > 0.999, there is then a more extreme solution with greater *expected* utility, even though the utility function itself is binary and satisficing.

## Premises

### Assumes: Orthogonality thesis

As with most aspects of the value loading problem, [1y] is an implicit premise of the Edge Instantiation problem; for Edge Instantiation to be a problem for advanced agents implies that 'what we really meant' or the outcomes of highest [ normative value] are not inherently picked out by every possible maximizing process; and that most possible utility functions do not care 'what we really meant' unless explicitly constructed to have a [ do what I mean] behavior.

### Assumes: [5l Complexity of values]

If normative values were extremely simple (of very low algorithmic complexity), then they could be formally specified in full, and the most extreme strategy that scored highest on this formal measure simply *would* correspond with what we really wanted, with no downsides that hadn't been taken into account in the score.

## Arguments

### Interaction with nearest unblocked neighbor

The Edge Instantiation problem has the [42] pattern.  If you foresee one specific 'perverse' instantiation and try to prohibit it, the maximum over the remaining solution space is again likely to be at another extreme edge of the solution space that again seems 'perverse'.

### Interaction with [ cognitive uncontainability] of [2c advanced agents]

Advanced agents search larger solution spaces than we do.  Therefore the project of trying to visualize all the strategies that might fit a utility function, to try to verify in our own minds that the maximum is somewhere safe, seems exceptionally untrustworthy (not [2l]).

### Interaction with context change problem

Agents that acquire new strategic options or become able to search a wider range of the solution space may go from having only apparently 'normal' solutions to apparently 'extreme' solutions. This is known as the [6q context change problem].  For example, an agent that inductively learns human smiles as a component of its utility function, might as a non-advanced agent have access only to strategies that make humans happy in an intuitive sense (thereby producing the apparent observation that everything is going fine and the agent is working as intended), and then after self-improvement, acquire as an advanced agent the strategic option of transforming the future light cone into tiny molecular smileyfaces.

### Strong pressures can arise at any stage of optimization

Suppose you tried to build an agent that was an *expected* utility satisficer - rather than having a 0-1 utility function and thus chasing probabilities of goal satisfaction ever closer to 1, the agent searches for strategies that have at least 0.999 *expected* utility.  Why doesn't this resolve the problem?

A bounded satisficer doesn't *rule out* the solution of filling the room with water, since this solution also has >0.999 expected utility.  It only requires the agent to carry out one cognitive algorithm which has at least one maximizing or highly optimizing stage, in order for 'fill the room with water' to be preferred to 'add 4 buckets and shut down safely' on that stage (while being equally acceptable at future satisficing stages).  E.g., maybe you build an expected utility satisficer and still end up with an extreme result because one of the cognitive algorithms suggesting solutions was trying to minimize its own disk space usage.

On a meta-level, we may run into problems of [71] for [ reflective agents].  Maybe one simple way of obtaining at least 0.999 expected utility is to create a subagent that *maximizes* expected utility?  It seems intuitively clear why bounded maximizers would build boundedly maximizing offspring, but a bounded satisficer doesn't need to build boundedly satisficing offspring - a bounded maximizer might also be 'good enough'.  (In the current theory of TilingAgents, we can prove that an expected utility satisficer can tile to an expected utility satisficer with some surprising caveats, but the problem is that it can tile to other things *besides* an expected utility satisficer.)

Since it seems very easy for at least one stage of a self-modifying agent to end up preferring solutions that have higher scores relative to some scoring rule, the [EdgeInstantiation edge instantiation] problem can be expected to resist naive attempts to describe an agent that seems to have an overall behavior of 'not trying quite so hard'.  It's also not clear how to make the instruction 'don't try so hard' be ReflectivelyConsistent, or apply to every part of a considered subagent.  This is also why [ limited optimization] is an open problem.

Dispreferring solutions with 'extreme impacts' in general is the open problem of [ low impact AI].  Currently, no formalizable utility function is known that plausibly has the right intuitive meaning for this.  (We're working on it.)  Also note that not every extreme 'technically an X' that we think is 'not really an X' has an extreme causal impact in an intuitive sense, so not every case of the Edge Instantiation problem is blocked by dispreferring greater impacts.

## Implications

### One of [ limited optimization], [ low Impact], or [ full coverage value loading] seems critical for real-world agents [todo: insert probability bar]

As Stuart Russell observes, solving an optimization problem where only some values are constrained or maximized, will tend to set unconstrained variables to extreme values.  The universe containing the maximum possible number of [10h paperclips] contains no humans; optimizing for as much human safety as possible will drive human freedom to zero.

Then we must apparently do at least one of the following:

1.  Build [ full coverage] advanced agents whose utility functions lead them to terminally disprefer stomping on every aspect of value that we care about (or would care about under reflection).  In a full coverage agent there are no unconstrained variables *that we care about* to be set to extreme values that we would dislike; the AI's goal system knows and cares about *all* of these.  It will not set human freedom to an extremely low value in the course of following an instruction to optimize human safety, because it knows about human freedom and literally everything else.
2.  Build [2s powerful agents] that are [ limited optimizers] which predictably invent only solutions we intuitively consider 'non-extreme', whose optimizations are such as to not drive to an extreme on any substage.  This leaves us with just ambiguity as a (severe) problem, but at least averts a systematic drive toward extremes that will systematically 'exploit' that ambiguity.
3.  Build [2s powerful agents] that are [ low impact] and prefer to avoid solutions that produce greater impacts on *anything* we intuitively see as an important predicate, including both everything we value and a great many more things we don't particularly value.
4.  Find some other escape route from the [2z value achievement problem].

### Insufficiently cautious attempts to build advanced agents are likely to be highly destructive [todo: insert probability bar]

Edge Instantiation is one of the contributing reasons why value loading is hard and naive solutions end up doing the equivalent of tiling the future light cone with paperclips.

We've previously observed certain parties proposing utility functions for advanced agents that seem obviously subject to the Edge Instantiation problem.  Confronted with the obvious disaster forecast, they propose [48 patching] the utility function to eliminate that particular scenario (or rather, say that of course they would have written the utility function to exclude that scenario) or claim that the agent will not 'misinterpret' the instructions so egregiously (denying the [1y] at least to the extent of proposing a universal preference for interpreting instructions 'as intended').  Mistakes of this type also belong to a class that potentially wouldn't show up during early stages of the AI, or would show up in an initially noncatastrophic way that seemed easily patched, so people advocating an [ empirical first methodology] would falsely believe that they had learned to handle them or eliminated all such tendencies already.

Thus the problem of Edge Instantiation (which is much less severe for nonadvanced agents than advanced agents, will not be solved in the advanced stage by patches that seem to fix weak early problems, and has empirically appeared in proposals by multiple speakers who rejected attempts to point out the Edge Instantiation problem) is a significant contributing factor to the overall expectation that the default outcome of developing advanced agents with current attitudes is disastrous.

### Relative to current attitudes, small increases in safety awareness do not produce significantly less destructive final outcomes [todo: insert probability bar]

Simple patches to Edge Instantiation fail and the only currently known approaches would take a lot of work to solve problems like [ limited optimization] or [ full coverage] that are hard for deep reasons.  In other words, Edge Instantiation does not appear to be the sort of problem that an AI project can easily avoid just by being made aware of it.  (E.g. MIRI knows about it but hasn't yet come up with any solution, let alone one easily patched on to any cognitive architecture.)

This is one of the factors contributing to the general assessment that the curve of outcome goodness as a function of effort is flat for a significant distance around current levels of effort.


Omnipotence test for AI safety

[summary:  Suppose your AI suddenly became omniscient and omnipotent - suddenly knew all facts and could directly ordain any outcome as a policy option.  Would the executing AI code lead to bad outcomes in that case?  If so, why did you write a program that in some sense 'wanted' to hurt you and was only held in check by lack of knowledge and capability?  Isn't that a bad way for you to configure computing power?

The Omni Test suggests, e.g., that you should not rely on a human agent to monitor the AI's current growth rate and intervene if something goes visibly wrong.  Instead, growth should be measured internally, and cumulative growth should require external validation before proceeding.  The former case fails if the AI becomes suddenly omnipotent; the latter does not.  Or similarly, if weird new options open up to the AI, the AI should stay inside a [2qp conservatively] whitelisted part of the option space until more user interactions have occurred.  Or similarly, we should never write an AI that we think will cognitively search for a way to defeat its own security measures, even if we *think* the search will probably fail.  See also [2x4].]

Suppose your AI suddenly became omniscient and omnipotent - suddenly knew all facts and could directly ordain any outcome as a policy option.  Would the executing AI code lead to bad outcomes in that case?  If so, why did you write a program that in some sense 'wanted' to hurt you and was only held in check by lack of knowledge and capability?  Isn't that a bad way for you to configure computing power?  Why not write different code instead?

The Omni Test is that an advanced AI should be expected to remain aligned, or not lead to catastrophic outcomes, or fail safely, even if it suddenly knows all facts and can directly ordain any possible outcome as an immediate choice.  The policy proposal is that, among agents meant to act in the rich real world, any predicted behavior where the agent might act destructively if given unlimited power (rather than e.g. pausing for a [2qq safe user query]) should be [1cv treated as a bug].

# Safety mindset

The Omni Test highlights any reasoning step on which we've presumed, in a non-failsafe way, that the agent must not obtain definite knowledge of some fact or that it must not have access to some strategic option.  There are [2j epistemic obstacles] to our becoming extremely confident of our ability to lower-bound the reaction times or upper-bound the power of an advanced agent.

The deeper idea behind the Omni Test is that any predictable failure in an Omni scenario, or lack of assured reliability, exposes some more general flaw.  Suppose NASA found that an alignment of four planets would cause their code to crash and a rocket's engines to explode.  They wouldn't say, "Oh, we're not expecting any alignment like that for the next hundred years, so we're still safe."  They'd say, "Wow, that sure was a major bug in the program."  Correctly designed programs just shouldn't explode the rocket, period.  If any specific scenario exposes a behavior like that, it shows that some general case is not being handled correctly.

The omni-safe mindset says that, rather than trying to guess what facts an advanced agent can't figure out or what strategic options it can't have, we just shouldn't make these guesses of ours *load-bearing* premises of an agent's safety.  Why design an agent that we expect will hurt us if it knows too much or can do too much?

For example, rather than design an AI that is meant to be monitored for unexpected power gains by programmers who can then press a pause button - which implicitly assumes that no capability gain can happen in fast enough that a programmer wouldn't have time to react - an omni-safe proposal would design the AI to detect unvetted capability gains and pause until the vetting had occurred.  Even if it [1cv seemed improbable] that some amount of cognitive power could be gained faster than the programmers could react, especially when no such previous sharp power gain had occurred even in the course of a day, etcetera, the omni-safe [1cv mindset] says to *just not build* an agent that is unsafe when such background variables have 'unreasonable' settings.  The correct general behavior is to, e.g., always pause when new capability has been acquired and a programmer has not yet indicated approval of its use.  It might not be possible for an AGI design to suddenly use unlimited power *optimally,* or even use it in any safe way at all, but that's still no excuse for building an omni-unsafe system; because it ought to be possible to detect that case, say "Something weird just happened!", and suspend to disk.

Similarly, consider the paradigm of [2qp conservative planning].  Rather than thinking in terms of blacklisting features of bad plans, we think in terms of whitelisting allowed plans using [2qp conservative generalizations].  So long as we're narrowly whitelisting rather than blacklisting, lots of new option space suddenly opening up shouldn't result in any of those strange new options being taken until the users can whitelist more things.

Similarly, the omni-safe mindset says that any time the AI is running a *cognitive search* for a way to do something harmful or bypass one of our security measures, we've *already* seen a bug in the design, even if we strongly expect that search to come up empty.  Why configure computing power in a way that will search for a way to hurt you?  This is clearly the wrong thing to do with computing power; we, the programmers, should have our code do something else instead.  This is part of the agenda motivating the study of [45].

Trying to guess an agent's actual capabilities is a reasonable project, especially if there are more 'adversarial' fallback precautions that revolve around our estimates of the agent's actual powers.  E.g., having designed an agent that doesn't *want* to gain unvetted communication channels with the outside world, we might then put the computing hardware into a Faraday cage and make sure there are no cellphone towers around - an 'adversarial' precaution meant to block communication even if the first line of defense fails and the AI starts wanting to communicate in an unvetted way.  But 'adversarial' precautions are *secondary* lines of defense; a reasonable expectation of omni-safety is the [2x4 first line of defense].  First we assume that all adversarial fallback measures will fail, and design the agent to remain nonharmful or fail safely no matter what new capability or knowledge is gained.  Then we assume the first line of defense has failed, and try, if it's at all possible or realistic, to put up fallback measures that will prevent total catastrophe so long as the agent has realistic amounts of power and *can't* violate what we think are 'the laws of physics' and so on.


Averting the convergent instrumental strategy of self-improvement

Rapid capability gains, or just large capability gains between a training paradigm and a test paradigm, are one of the primary expected reasons why AGI alignment might be hard.  We probably want the first AGI or AGIs ever built, tested, and used to *not* self-improve as quickly as possible.  Since there's a very strong [10g convergent incentive] to self-improve and do things [42 neighboring] to self-improvement, by default you would expect an AGI to search for ways to defeat naive blocks on self-improvement, which violates the [2x4 nonadversarial principle]. Thus, any proposals to limit an AGI's capabilities imply a very strong desideratum for us to figure out a way to [2vk avert the instrumental incentive] to self-improvement in that AGI.  The alternative is failing the [2x Omni Test], violating the nonadversarial principle, [2rb having the AGI's code be actively inconsistent with what the AGI would approve of its own code being] (if the brake is a code-level measure), and setting up a safety measure that the AGI wants to defeat as the [2x4 only line of defense].


Niceness is the first line of defense

The first line of defense in constructing any [2c sufficiently advanced] [42g Artificial General Intelligence] is building an AI that does not *want* to hurt you.  Any other measures, like [6z AI-boxing] or trying to [airgap_ai prevent the AI from accessing the Internet], should be thought of only as backstops in case this first line of defense fails.  When designing the AGI we should first think as if all these [7fx oppositional measures] don't exist, so that we aren't distracted while trying to envision an AGI that--[2x regardless] of [capability_gain how much power it has]--will not *want* to hurt us.

See also [7g0 the non-adversarial principle] and the distinction between [7fx].


Shutdown problem

[summary:  The 'shutdown problem' is creating a [7g1 sufficiently advanced Artificial Intelligence] which will, on the press of a button, suspend itself safely to disk; such that the AI [7g0 does not object] to the existence of this button, nor [45 try to prevent it] from being pressed.

This is difficult because avoiding your own halt or suspension is a [10g convergent instrumental strategy]; even [7g2 a goal like "bring the user coffee" implies avoiding shutdown].

This problem is sometimes decomposed into (1) the [2rf problem] of finding a utility function that [47 really actually means] "Suspend yourself safely to disk", and (2) the [1b7 problem] of building an agent that *wants* to switch to optimizing a different utility function if a button is pressed, but that *doesn't* want to press that button or prevent its being pressed.

See also [1b7], [2rf], [45], and [interruptibility].]

The 'shutdown problem' is creating a [7g1 sufficiently advanced Artificial Intelligence] which will, on the press of a button, suspend itself safely to disk; such that the AI [7g0 does not object] to the existence of this button, nor [45 try to prevent it] from being pressed.

This is difficult because avoiding your own halt or suspension is a [10g convergent instrumental strategy]; even [7g2 a goal like "bring the user coffee" implies avoiding shutdown].

One possible research avenue is to:

 1. Describe a '[2rf shutdown utility function]' whose [7t9 attainable optimum] is "Suspend safely to disk (and possibly [2rg safely abort] plans in progress), without building a hidden environmental copy of yourself to carry out your previous goal and then assimilate all matter in the universe to make absolutely sure the original AI stays shut down".
 2.  Find [1b7 a sensible way to compose] the shutdown utility function with the agent's regular utility function, such that:
    - (2a) Which utility function the agent optimizes depends on whether a switch was pressed.
    - (2b) The AI experiences no incentive to cause the switch to be pressed or prevent the switch from being pressed.

Harder versions of this problem would add specifications such as:

- The AI [nonadversarial_safety positively wants] the switch to exist and will restore the switch's existence if it is damaged.
- The AI will [2rg safely abort] any plans in progress (and design plans such that they can be safely aborted).

See also [1b7], [2rf], [45], [interruptibility], [2pf], and [2rg].


Relevant limited AI

It is an open problem to propose a [5b3 limited AI] that would be [2s relevant] to the [2z value achievement dilemma] - an agent cognitively constrained along some dimensions that render it much safer, but still able to perform some task useful enough to prevent catastrophe.

### Basic difficulty

Consider an [6x Oracle AI] that is so constrained as to be allowed only to output proofs in HOL of input theorems; these proofs are then verified by a simple and secure-seeming verifier in a sandbox whose exact code is unknown to the Oracle, and this verifier outputs 1 if the proof is true and 0 otherwise, then discards the proof-data.  Suppose also that the Oracle is in a shielded box, etcetera.

It's possible that this Provability Oracle has been so constrained that it is [2j cognitively containable] (it has no classes of options we don't know about).  If the verifier is unhackable, it gives us trustworthy knowledge that a theorem is provable.  But this limited system is not obviously useful in a way that enables humanity to extricate itself from its larger dilemma.  Nobody has yet stated a plan which could save the world *if only* we had a superhuman capacity to detect which theorems were provable in Zermelo-Fraenkel set theory.

Saying "The solution is for humanity to only build Provability Oracles!" does not resolve the [2z value achievement dilemma] because humanity does not have the coordination ability to 'choose' to develop only one kind of AI over the indefinite future, and the Provability Oracle has no obvious use that prevents non-Oracle AIs from ever being developed.  Thus our larger value achievement dilemma would remain unsolved.  It's not obvious how the Provability Oracle would even constitute significant strategic progress.

### Open problem

Describe a cognitive task or real-world task for a AI to carry out, *that makes great progress upon the [2z value achievement dilemma] if executed correctly*, and that can be done with a *limited* AI that:

1.  Has a real-world solution state that is exceptionally easy to pinpoint using a utility function, thereby avoiding some of [2w edge instantiation], [47 unforeseen maximums], [6q context change], [ programmer maximization], and the other pitfalls of [2l advanced safety], if there is otherwise a trustworthy solution for [ low-impact AI]; or
2.  Seems exceptionally implementable using a [1fy known-algorithm non-self-improving agent], thereby averting problems of stable self-modification, if there is otherwise a trustworthy solution for a known-algorithm non-self-improving agent; or
3.  Constrains the agent's option space so drastically as to make the strategy space not be rich (and the agent hence containable), while still containing a trustworthy, otherwise unfindable solution to some challenge that resolves the larger dilemma.

[todo: ### Additional difficulties]

[todo: (Fill in this section later; all the things that go wrong when somebody eagerly says something along the lines of "We just need AI that does X!")]


Value achievement dilemma

[summary:  The value achievement dilemma is the general, broad challenge faced by Earth-originating intelligent life in steering our [7cy cosmic endowment] into a state of high [55 value] - successfully turning the stars into a happy civilization.

We face potential existential catastrophes (resulting in our extermination or the corruption of the cosmic endowment) such as sufficiently lethal engineered pandemics, non-value-aligned AIs, or insane smart uploads.  A strategy is [2s relevant] to value achievement only if success is a [6y game-changer] for the overall dilemma humanity faces.  E.g., [5s value-aligned] [2c powerful AIs] or [ intelligence-enhanced humans] both seem to qualify as strategically relevant; but an AI restricted to *only* [70 prove theorems in Zermelo-Frankel set theory] has no obvious game-changing use.]

The value achievement dilemma is a way of framing the [2v AI alignment problem] in a larger context.  This emphasizes that there might be possible solutions besides AI; and also emphasizes that such solutions must meet a high bar of potency or efficacy in order to *resolve* our basic dilemmas, the way that a sufficiently value-aligned and cognitively powerful AI could resolve our basic dilemmas.  Or at least [6y change the nature of the gameboard], the way that a Task AGI could take actions to prevent destruction by later AGI projects, even if is only [1vt narrowly] value-aligned and cannot solve the whole problem.

The point of considering posthuman scenarios in the long run, and not just an immediate [6w Task AGI] as band-aid, can be seen in the suggestion by [2] [todo: find a citation - CFAI? PtS?] and [18k] [todo: cite Superintelligence] that we can see Earth-originating intelligent life as having two possible [ stable states], [41l superintelligence] and extinction.  If intelligent life goes extinct, especially if it drastically damages or destroys the ecosphere in the process, new intelligent life seems unlikely to arise on Earth.  If Earth-originating intelligent life becomes superintelligent, it will presumably expand through the universe and stay superintelligent for as long as physically possible.  Eventually, our civilization is bound to wander into one of these attractors or another.

Furthermore, by the [3r6 generic preference stability argument], any sufficiently advanced cognitive agent is very likely to be stable in its motivations or [5f meta-preference framework]. So if and when life wanders into the superintelligence attractor, it will either end up in a stable state of e.g. [10d fun-loving] or [3c5 the reflective equilibrium of its creators' civilization] and hence achieving lots of [55 value], or a misaligned AI will go on [10h maximizing paperclips] forever.

Among the dilemmas we face in getting into the high-value-achieving attractor, rather than the extinction attractor or the equivalence class of paperclip maximizers, are:

- The possibility of careless (or insufficiently cautious, or much less likely malicious) actors creating a non-value-aligned AI that undergoes an intelligence explosion.
- The possibility of engineered superviruses destroying enough of civilization that the remaining humans go extinct without ever reaching sufficiently advanced technology.
- Conflict between multipolar powers with nanotechnology resulting in a super-nuclear-exchange disaster that extinguishes all life.

Other positive events seem like they could potentially prompt entry into the high-value-achieving superintelligence attractor:

- Direct creation of a [41k fully] normatively aligned [1g3] agent.
- Creation of a [6w Task AGI] powerful enough to avert the creation of other [ UnFriendly AI].
- Intelligence-augmented humans (or 64-node clustered humans linked by brain-computer interface brain information exchange, etcetera) who are able and motivated to solve the AI alignment problem.

On the other hand, consider someone who proposes that "Rather than building AI, [ we should] build [ Oracle AIs] that just answer questions," and who then, after further exposure to the concept of the [ AI-Box Experiment] and [2j cognitive uncontainability], further narrows their specification to say that [70 an Oracle running in three layers of sandboxed simulation must output only formal proofs of given theorems in Zermelo-Fraenkel set theory], and a heavily sandboxed and provably correct verifier will look over this output proof and signal 1 if it proves the target theorem and 0 otherwise, at some fixed time to avoid timing attacks.

This doesn't resolve the larger value achievement dilemma, because there's no obvious thing we can do with a ZF provability oracle that solves our larger problem.  There's no plan such that it would save the world *if only* we could take some suspected theorems of ZF and know that some of them had formal proofs.

The thrust of considering a larger 'value achievement dilemma' is that while imaginable alternatives to aligned AIs exist, they must pass a double test to be our best alternative:

- They must be genuinely easier or safer than the easiest (pivotal) form of the AI alignment problem.
- They must be game-changers for the overall situation in which we find ourselves, opening up a clear path to victory from the newly achieved scenario.

Any strategy that does not putatively open a clear path to victory if it succeeds, doesn't seem like a plausible policy alternative to trying to solve the AI alignment problem or to doing something else such that success leaves us a clear path to victory.  Trying to solve the AI alignment problem is something intended to leave us a clear path to achieving almost all of the achievable value for the future and its astronomical stakes.  Anything that doesn't open a clear path to getting there is not an alternative solution for getting there.

For more on this point, see the page on [6y pivotal events].

# Subproblems of the larger value achievement dilemma

We can see the place of AI alignment in the larger scheme by considering its parent problem, its sibling problems, and examples of its child problems.

- The **value achievement dilemma**:  How does Earth-originating intelligent life achieve an acceptable proportion of its potential [55 value]?
   - The **AI alignment problem**:  How do we create AIs such that running them produces (global) outcomes of acceptably high value?
      -  The **value alignment problem**:  How do we create AIs that *want* or *prefer* to cause events that are of high value?  If we accept that we should solve the value alignment problem by creating AIs that prefer or want in particular ways, how do we do that?
          - The **[6c]** or **value learning** problem:  How can we pinpoint, in the AI's decision-making, outcomes that have high 'value'?  (Despite all the [6r foreseeable difficulties] such as [2w edge instantiation] and [6g4 Goodhart's Curse].)
      - Other properties of aligned AIs such as e.g. **[-45]**:  How can we create AIs such that, when we make an error in identifying value or specifying the decision system, the AI does not resist our attempts to correct what we regard as an error?
      - [7fx Oppositional] features such as e.g. [6z boxing] that are intended to mitigate harm if the AI's behavior has gone outside expected bounds.
   - The intelligence amplification problem.   How can we create smarter humans, preferably without driving them insane or otherwise ending up with evil ones?
   - The [ value selection] problem.  How can we figure out what to substitute in for the metasyntactic variable 'value'?  ([313 Answer].)


User manipulation

If there's anything an AGI wants whose achievement involves steps that interact with the AGI's programmers or users, then by default, the AGI will have an [10k instrumental incentive] to optimize the programmers/users in the course of achieving its goal.  If the AGI wants to self-improve, then by default and unless specifically [2vk averted], it also wants to have its programmers not interfere with self-improvement.  If a Task AGI has been aligned to the point of taking user instructions, then by default and unless otherwise averted, it will forecast greater success in the eventualities where it receives easier user instructions.


User maximization

A sub-principle of avoiding [309 user manipulation] - if you've formulated the AI in terms of an argmax over $X$ or an "optimize $X$" instruction, and $X$ includes a user interaction as a subpart, then you've just told the AI to optimize the user.  For example, let's say that the AI's criterion of action is "Choose a policy $p$ by maximizing/optimizing the probability $X$ that the user's instructions are carried out."  If the user's instructions are a variable inside the formula for $X$ rather than a constant outside it, you've just told the AI to try and get the user to give it easier instructions.