Ontology identification problem: Technical tutorial

The problem of [5c ontology identification] is the problem of loading a goal into an [2c advanced agent] when that agent's representation of the world is likely to change in ways [5d unforeseen in the development phase].  This tutorial focuses primarily on explaining what the problem is and why it is a [6r foreseeable difficulty]; for the corresponding research problems, see [5c the main page on Ontology Identification].

This is a technical tutorial, meaning that it assumes some familiarity with [2v value alignment theory], the [6c value identification problem], and [2l safety thinking for advanced agents].

To isolate ontology identification from other parts of the value identification problem, we consider a simplified but still very difficult problem: to state an unbounded program implementing a [5g diamond maximizer] that will turn as much of the physical universe into diamond as possible.  The goal of "making diamonds" was chosen to have a crisp-seeming definition for our universe: namely, the amount of diamond is the number of carbon atoms covalently bound to four other carbon atoms.  Since it seems that in this case our [6h intended goal] should be crisply definable relative to our universe's physics, we can avert many other issues of trying to identify [5l complex values] to the agent.  Ontology identification is a difficulty that still remains even in this case - the agent's representation of 'carbon atoms' may still change over time.

## Introduction: Two sources of representational unpredictability

Suppose we wanted to write a hand-coded, [5t object-level] utility function that evaluated the amount of diamond material present in the AI's model of the world.  We might foresee the following two difficulties:

1.  Where exactly do I find 'carbon atoms' inside the AI's model of the world?  As the programmer, all I see are these mysterious ones and zeroes, and the only parts that directly correspond to events I understand is the represention of the pixels in the AI's webcam... maybe I can figure out where the 'carbon' concept is by showing the AI graphite, buckytubes, and a diamond on its webcam and seeing what parts get activated... whoops, looks like the AI just revised its internal representation to be more computationally efficient, now I once again have no idea what 'carbon' looks like in there.  How can I make my hand-coded utility function re-bind itself to 'carbon' each time the AI revises its model's representation of the world?

2.  What exactly is 'diamond'?  If you say it's a nucleus with six protons, what's a proton?  If you define a proton as being made of quarks, what if there are unknown other particles underlying quarks?  What if the Standard Model of physics is incomplete or wrong - can we state exactly and formally what constitutes a carbon atom when we aren't certain what the underlying quarks are made of?

Difficulty 2 probably seems more exotic than the first, but Difficulty 2 is easier to explain in a formal sense and turns out to be a simpler way to illustrate many of the key issues that also appear in Difficulty 1.  We can see Difficulty 2 as the problem of binding an [6h intended goal] to an unknown territory, and Difficulty 1 as the problem of binding an intended goal to an unknown map.  So the first step of the tutorial will be to walk through how Difficulty 2 (what exactly is a diamond?) might result in weird behavior in an [107 unbounded agent] intended to be a diamond maximizer.

## Try 1: Hacking AIXI to maximize diamonds?

The classic unbounded agent - an agent using far more computing power than the size of its environment - is [11v AIXI].  Roughly speaking, AIXI considers all computable hypotheses for how its environment might work - all possible Turing machines that would turn AIXI's outputs into AIXI's future inputs.  (The finite variant AIXI-tl has a hypothesis space that includes all Turing machines that can be specified using fewer than $l$ bits and run in less than time $t$.)

From the perspective of AIXI, any Turing machine that takes one input tape and produces two output tapes is a "hypothesis about the environment", where the input to the Turing machine encodes AIXI's hypothetical action, and the outputs are interpreted as a prediction about AIXI's sensory data and AIXI's reward signal.  (In Marcus Hutter's formalism, the agent's reward is a separate sensory input to the agent, so hypotheses about the environment also make predictions about sensed rewards).  AIXI then behaves as a [ Bayesian predictor] that uses [5v algorithmic complexity] to give higher [ prior probabilities] to simpler hypotheses (that is, Turing machines with fewer states and smaller state transition diagrams), and updates its mix of hypotheses based on sensory evidence (which can confirm or disconfirm the predictions of particular Turing machines).

As a decision agent, AIXI always outputs the motor action that leads to the highest predicted reward, assuming that the environment is described by the updated probability mixture of all Turing machines that could represent the environment (and assuming that future iterations of AIXI update and choose similarly).

The ontology identification problem shows up sharply when we imagine trying to modify AIXI to "maximize expectations of diamonds in the outside environment" rather than "maximize expectations of sensory reward signals".  As a [ Cartesian agent], AIXI has sharply defined sensory inputs and motor outputs, so we can have a [ probability mixture] over all Turing machines that relate motor outputs to sense inputs (as crisply represented in the input and output tapes).  But even if some otherwise arbitrary Turing machine happens to predict sensory experiences extremely well, how do we look at the state and working tape of that Turing machine to evaluate 'the amount of diamond' or 'the estimated number of carbon atoms bound to four other carbon atoms'?  The highest-weighted Turing machines that have best predicted the sensory data so far, presumably contain *some* sort of representation of the environment, but we have no idea how to get 'the number of diamonds' out of it.

(Example:  Maybe one Turing machine that is producing good sequence predictions inside AIXI, actually does so by simulating a large universe, identifying a superintelligent civilization that evolves inside that universe, and motivating that civilization to try to intelligently predict future future bits from past bits (as provided by some intervention).  To write a formal utility function that could extract the 'amount of real diamond in the environment' from arbitrary predictors in the above case , we'd need the function to read the Turing machine, decode that universe, find the superintelligence, decode the superintelligence's thought processes, find the concept (if any) resembling 'diamond', and hope that the superintelligence had precalculated how much diamond was around in the outer universe being manipulated by AIXI.)

This is, in general, the reason why the AIXI family of architectures can only contain agents defined to maximize direct functions of their sensory input, and not agents that behave so as to optimize facts about their external environment.  (We can't make AIXI maximize diamonds by making it want *pictures* of diamonds because then it will just, e.g., [ build an environmental subagent that seizes control of AIXI's webcam and shows it pictures of diamonds].  If you ask AIXI to show itself sensory pictures of diamonds, you can get it to show its webcam lots of pictures of diamonds, but this is not the same thing as building an environmental diamond maximizer.)

## Try 2: Unbounded agent using classical atomic hypotheses?

Given the origins of the above difficulty, we next imagine constraining the agent's hypothesis space to something other than "literally all computable functions from motor outputs to sense inputs", so that we can figure out how to find diamonds or carbon inside the agent's representation of the world.

As an [ unrealistic example]:  Suppose someone was trying to define 'diamonds' to the AI's utility function.  Suppose they knew about atomic physics but not nuclear physics.  Suppose they build an AI which, during its development phase, learns about atomic physics from the programmers, and thus builds a world-model that is based on atomic physics.

Again for purposes of [ unrealistic examples], suppose that the AI's world-model is encoded in such fashion that when the AI imagines a molecular structure - represents a mental image of some molecules - then carbon atoms are represented as a particular kind of basic element of the representation.  Again, as an [ unrealistic example], imagine that there are [ little LISP tokens] representing environmental objects, and that the environmental-object-type of carbon-objects is encoded by the integer 6.  Imagine also that each atom, inside this representation, is followed by a list of the other atoms to which it's covalently bound.  Then when the AI is imagining a carbon atom participating in a diamond, inside the representation we would see an object of type 6, followed by a list containing exactly four other 6-objects.

Can we fix this representation for all hypotheses, and then write a utility function for the AI that counts the number of type-6 objects that are bound to exactly four other type-6 objects?  And if we did so, would the result actually be a diamond maximizer?

### AIXI-atomic

As a first approach to implementing this idea - an agent whose hypothesis space is constrained to models that directly represent all the carbon atoms - imagine a variant of AIXI-tl that, rather than considering all tl-bounded Turing machines, considers all simulated atomic universes containing up to 10^100 particles spread out over up to 10^50 light-years.  In other words, the agent's hypotheses are universe-sized simulations of classical, pre-nuclear models of physics; and these simulations are constrained to a common representation, so a fixed utility function can look at the representation and count carbon atoms bound to four other carbon atoms.  Call this agent AIXI-atomic.

(Note that AIXI-atomic, as an [107 unbounded agent], may use far more computing power than is embodied in its environment.  For purposes of the thought experiment, assume that the universe contains exactly one hypercomputer that runs AIXI-atomic.)

A first difficulty is that universes composed only of classical atoms are not good explanations of our own universe, even in terms of surface phenomena; e.g. the [ultraviolet catastrophe](http://en.wikipedia.org/wiki/Ultraviolet_catastrophe).  So let it be supposed that we have simulation rules for classical physics that replicate at least whatever phenomena the programmers have observed at [ development time], even if the rules have some seemingly ad-hoc elements (like there being no ultraviolent catastrophes).  We will *not* however suppose that the programmers have discovered all experimental phenomena we now see as pointing to nuclear or quantum physics.

A second difficulty is that a simulated universe of classical atoms does not identify where in the universe the AIXI-atomic agent resides, or say how to match the types of AIXI-atomic's sense inputs with the underlying behaviors of atoms.  We can elide this difficulty by imagining that AIXI-atomic simulates classical universes containing a single hypercomputer, and that AIXI-atomic knows a simple function from each simulated universe onto its own sensory data (e.g., it knows to look at the simulated universe, and translate simulated photons impinging on its webcam onto predicted webcam data in the standard format).  This elides most of the problem of [ naturalized induction].

So the AIXI-atomic agent that is hoped to maximize diamond:

- Considers only hypotheses that directly represent universes as huge systems of classical atoms, so that the function 'count atoms bound to four other carbon atoms' can be directly run over any possible future the agent models.
- Assigns probabilistic priors over these possible atomic representations of the universe, favoring representations that are in some sense simpler.
- Somehow [ maps each atomic-level representation onto the agent's predicted sensory experiences].
- [ Bayes-updates its priors] based on actual sensory experiences, the same as classical AIXI.
- Can evaluate the 'expected diamondness on the next turn' of a single action by looking at all hypothetical universes where that action is performed, weighted by their current probability, and summing over the expectation of 'carbon atoms bound to four other carbon atoms' after some unit amount of time has passed.
- Can evaluate the 'future expected diamondness' of an action, over some finite time horizon, by assuming that its future self will also Bayes-update and maximize expected diamondness over that time horizon.
- On each turn, outputs the action with greatest expected diamondness over some finite time horizon.

Suppose our own real universe was amended to otherwise be exactly the same, but contain a single [ impermeable] hypercomputer.  Suppose we defined an agent like the one above, using simulations of 1910-era models of physics, and ran that agent on the hypercomputer.  Should we expect the result to be an actual diamond maximizer - expect that the outcome of running this program on a single hypercomputer would indeed be that most mass in our universe would be turned into carbon and arranged into diamonds?

### Anticipated failure: AIXI-atomic tries to 'maximize outside the simulation'

In fact, our own universe isn't atomic, it's nuclear and quantum-mechanical.  This means that AIXI-atomic does not contain any hypotheses in its hypothesis space that *directly represent* our universe.  By the previously specified hypothesis of the thought experiment, AIXI-atomic's model of simulated physics was built to encompass all the experimental phenomena the programmers had yet discovered, but there were some quantum and nuclear phenomena that AIXI-atomic's programmers had not yet discovered.  When those phenomena are discovered, there will be no simple explanation on the direct terms of the model.

Intuitively, of course, we'd like AIXI-atomic to discover the composition of nuclei, shift its models to use nuclear physics, and refine the 'carbon atoms' mentioned in its utility function to mean 'atoms with nuclei containing six protons'.

But we didn't actually specify that when constructing the agent (and saying how to do it in general is, so far as we know, hard; in fact it's the whole ontology identification problem).  We constrained the hypothesis space to contain only universes running on the classical physics that the programmers knew about.  So what happens instead?

Probably the 'simplest atomic hypothesis that fits the facts' will be an enormous atom-based computer, *simulating* nuclear physics and quantum physics in order to create a simulated non-classical universe whose outputs are ultimately hooked up to AIXI's webcam.  From our perspective this hypothesis seems silly, but if you restrict the hypothesis space to only classical atomic universes, that's what ends up being the computationally simplest hypothesis that predicts, in detail, the results of nuclear and quantum experiments.

AIXI-atomic will then try to choose actions so as to maximize the amount of expected diamond inside the probable *outside universes* that could contain the giant atom-based simulator of quantum physics.  It is not obvious what sort of behavior this would imply.

### Metaphor for difficulty: AIXI-atomic cares about only fundamental carbon

One metaphorical way of looking at the problem is that AIXI-atomic was implicitly defined to care only about diamonds made out of *ontologically fundamental* carbon atoms, not diamonds made out of quarks.  A probability function that assigns 0 probability to all universes made of quarks, and a utility function that outputs a constant on all universes made of quarks, [ yield functionally identical behavior].  So it is an exact metaphor to say that AIXI-atomic only *cares* about universes with ontologically basic carbon atoms, given that AIXI-atomic's hypothesis space only contains universes with ontologically basic carbon atoms.

Imagine that AIXI-atomic's hypothesis space does contain many other universes with other laws of physics, but its hand-coded utility function just returns 0 on those universes since it can't find any 'carbon atoms' inside the model.  Since AIXI-atomic only cares about diamond made of fundamental carbon, when AIXI-atomic discovers the experimental data implying that almost all of its probability mass should reside in nuclear or quantum universes in which there were no fundamental carbon atoms, AIXI-atomic stops caring about the effect its actions have on the vast majority of probability mass inside its model.  Instead AIXI-atomic tries to maximize inside the tiny remaining probabilities in which it *is* inside a universe with fundamental carbon atoms that is somehow reproducing its sensory experience of nuclei and quantum fields... for example, a classical atomic universe containing a computer simulating a quantum universe and showing the results to AIXI-atomic.

From our perspective, we failed to solve the 'ontology identification problem' and get the real-world result we [6h intended], because we tried to define the agent's *utility function* over properties of a universe made out of atoms, and the real universe turned out to be made of quantum fields.  This caused the utility function to *fail to bind* to the agent's representation in the way we intuitively had in mind.

Today we do know about quantum mechanics, so if we tried to build a diamond maximizer using some bounded version of the above formula, it might not fail on account of [48 the particular exact problem] of atomic physics being false.

But perhaps there are discoveries still remaining that would change our picture of the universe's ontology to imply something else underlying quarks or quantum fields.  Human beings have only known about quantum fields for less than a century; our model of the ontological basics of our universe has been stable for less than a hundred years of our human experience.  So we should seek an AI design that does not assume we know the exact, true, fundamental ontology of our universe during an AI's [5d development phase].

As another important metaphorical case in point, consider a human being who feels angst on contemplating a universe in which "By convention sweetness, by convention bitterness, by convention color, in reality only atoms and the void" (Democritus); someone who wonders where there is any room in this collection of lifeless particles for love, free will, or even the existence of people.  Since, after all, people are just *mere* collections of atoms.  This person can be seen as undergoing an ontology identification problem: they don't know how to find the objects of value in a representation containing atoms instead of ontologically basic people.

Human beings simultaneously evolved a particular set of standard mental representations (e.g., a representation for colors in terms of a 3-dimensional subjective color space) along with evolving emotions that bind to these representations ([identification of flowering landscapes as beautiful](http://en.wikipedia.org/wiki/Evolutionary_aesthetics#Landscape_and_other_visual_arts_preferences).  When someone visualizes any particular configuration of 'mere atoms', their built-in desires don't automatically fire and bind to that mental representation, the way they would bind to the brain's native representation of the environment.  Generalizing that no set of atoms can be meaningful (since no abstract configuration of 'mere atoms' they imagine, seems to trigger any emotions to bind to it) and being told that reality is composed entirely of such atoms, they feel they've been told that the true state of reality, underlying appearances, is a meaningless one.

## The utility rebinding problem

Intuitively, we would think it was [ common sense] for an agent that wanted diamonds to react to the experimental data identifying nuclear physics, by deciding that a carbon atom is 'really' a nucleus containing six protons.  We can imagine this agent [ common-sensically] updating its model of the universe to a nuclear model, and redefining the 'carbon atoms' that its old utility function counted to mean 'nuclei containing exactly six protons'.  Then the new utility function could evaluate outcomes in the newly discovered nuclear-physics universe.  The problem of producing this desirable agent behavior is the **utility rebinding problem**.

To see why this problem is nontrivial, consider that the most common form of carbon is C-12, with nuclei composed of six protons and six neutrons.  The second most common form of carbon is C-14, with nuclei composed of six protons and eight neutrons.  Is C-14 *truly* carbon - is it the sort of carbon that can participate in valuable diamonds of high utility?  Well, that depends on your utility function, obviously; and from a human perspective it just sounds arbitrary.

But consider a closely analogous question from a humanly important perspective: Is a chimpanzee truly a person?  Where the question means not, "How do we arbitrarily define the syllables per-son?" but "Should we care a lot about chimpanzees?", i.e., how do we define the part of our preferences that care about people, to the possibly-person edge cases of chimpanzees?

If you live in a world where chimpanzees haven't been discovered, you may have an easy time running your utility function over your model of the environment, since the objects of your experience classify sharply into the 'person' and 'nonperson' categories.  Then you discover chimpanzees, and they're neither typical people (John Smith) nor typical nonpeople (like rocks).

We can see the force of this question as arising from something like an ontological shift: we're used to valuing cognitive systems that are made from whole human minds, but it turns out that minds are made of parts, and then we have the question of how to value things that are made from some of the person-parts but not all of them... sort of like the question of how to treat carbon atoms that have the usual number of protons but not the usual number of neutrons.

Chimpanzees definitely have neural areas of various sizes, and particular cognitive abilities - we can suppose the empirical truth is unambiguous at this level, and known to us.  So the question is then whether we regard a particular configuration of neural parts (a frontal cortex of a certain size) and particular cognitive abilities (consequentialist means-end reasoning and empathy, but no recursive language) as something that our 'person' category values... once we've rewritten the person category to value configurations of cognitive parts, rather than whole atomic people.

In fact, we run into this question as soon as we learn that human beings run on brains and the brains are made out of neural regions with functional properties; we can then *imagine* chimpanzees even if we haven't met any, and ask to what degree our preferences should treat this edge-person as deserving of moral rights.  If we can 'rebind' our emotions and preferences to live in a world of nuclear brains rather than atomic people, this rebinding will *implicitly* say whether or not a chimpanzee is a person, depending on how our preference over brain configurations treats the configuration that is a chimpanzee.

In this sense the problem we face with chimpanzees is exactly analogous to the question a diamond maximizer would face after discovering nuclear physics and asking itself whether a carbon-14 atom counted as 'carbon' for purposes of caring about diamonds.  Once a diamond maximizer knows about neutrons, it can see that C-14 is chemically like carbon and forms the same kind of chemical bonds, but that it's heavier because it has two extra neutrons.  We can see that chimpanzees have a similar brain architectures to the sort of people we always considered before, but that they have smaller frontal cortexes and no ability to use recursive language, etcetera.

Without knowing more about the diamond maximizer, we can't guess what sort of considerations it might bring to bear in deciding what is Truly Carbon and Really A Diamond.  But the breadth of considerations human beings need to invoke in deciding how much to care about chimpanzees, is one way of illustrating that the problem of rebinding a utility function to a shifted ontology is [value-laden] and can potentially undergo [excursions] into [5l complex desiderata].  Redefining a [ moral category] so that it talks about the underlying parts of what were previously seen as all-or-nothing atomic objects, may carry an implicit ruling about how to value many kinds of [edge-case] objects that were never seen before.

It's possible that some formal part of this problem could be usefully carved out from the complex value-laded edge-case-reclassification part.  E.g., how would you redefine carbon as C12 if there were no other isotopes?  How would you rebind the utility function to *at least* C12?  In general, how could edge cases be [ identified and queried] by an [6w online Genie]?

### Reappearance on the reflective level

An obvious thought (especially for [6w online Genies]) is that if the AI is unsure about how to reinterpret its goals in light of a shifting mental representation, it should query the programmers.

Since the definition of a programmer would then itself be baked into the [5f preference framework], the problem might [ reproduce itself on the reflective level] if the AI became unsure of where to find 'programmers':  "My preference framework said that programmers were made of carbon atoms, but all I can find in this universe are quantum fields!"

Thus the ontology identification problem is arguably one of the [ critical subproblems] of value alignment: it plausibly has the property that, if botched, it could potentially [ crash the error recovery mechanism].

## Diamond identification in multi-level maps

A realistic, [5g bounded diamond maximizer] wouldn't represent the outside universe with atomically detailed or quantum-detailed models.  Instead, a bounded agent would have some version of a [ multi-level map] of the world in which the agent knew in principle that things were composed of atoms, but didn't model most things in atomic detail.  A bounded agent's model of an airplane would have wings, or wing shapes, rather than atomically detailed wings.  It would think about wings when doing aerodynamic engineering, atoms when doing chemistry, nuclear physics when doing nuclear engineering, and definitely not try to model everything in its experience down to the level of quantum fields.

At the present, there are not yet any proposed formalisms for how to do probability theory with multi-level maps (in other words: [ nobody has yet put forward a guess at how to solve the problem even given infinite computing power]).  But it seems very likely that, if we did know what multi-level maps looked like formally, it might suggest a formal solution to non-value-laden utility-rebinding.

E.g., if an agent already has a separate high-level concept of 'diamond' that's bound to a lower-level concept of 'carbon atoms bound to four other carbon atoms', then maybe when you discover nuclear physics, the multi-level map itself would tend to suggest that 'carbon atoms' be re-bound to 'nuclei with six protons' or 'nuclei with six protons and six neutrons'.  It might at least be possible to phrase the equivalent of a prior or mixture of weightings for how the utility function would re-bind itself, and say, "Given this prior, care about whatever that sparkly hard stuff 'diamond' ends up binding to on the lower level."

Unfortunately, we have very little formal probability theory to describe how a multi-level map would go from 'that unknown sparkly hard stuff' to 'carbon atoms bound to four other carbon atoms in tetrahedral patterns, which is the only known repeating pattern for carbon atoms bound to four other carbon atoms' to 'C12 and C14 are chemically identical but C14 is heavier'.  This being the case, we don't know how to say anything about a dynamically updating multi-level map inside a [5f preference framework].

If we were actually trying to build a diamond maximizer, we would be likely to encounter this problem long before it started formulating new physics.  The equivalent of a computational discovery that changes 'the most efficient way to represent diamonds' is likely to happen much earlier than a physical discovery that changes 'what underlying physical systems probably constitute a diamond'.

This also means that we are liable to face the ontology identification problem long before the agent starts discovering new physics, as soon as it starts revising its representation.  Only very unreflective agents with strongly fixed-in-place representations for every part of the environment that we think the agent is supposed to care about, would let the ontology identification problem be elided entirely.  Only *very* not-self-modifying agents, or [ Cartesian agents] with goals formulated only over sense data, would not confront their programmers with ontology identification problems.

## Research paths

More of these are described in the [5c main article on ontology identification].  But here's a quick list of some relevant research subproblems and avenues:

* Transparent priors.  Priors that are constrained to meaningful hypothesis spaces that the utility function knows how to interpret.  Rather than all Turing machines being hypotheses, we could have only causal models being hypotheses, and then preference frameworks that talked about 'the cause of' labeled sensory data could read the hypotheses.  (Note that the space of causal models can be Turing-complete, in the sense of being able to embed any Turing machine as a causal system.  So we'd be able to explain any computable sense data in terms of a causal model - we wouldn't sacrifice any explanatory power by restricting ourselves to 'causal models' instead of 'all Turing machines'.)

* Reductionist identifications.  Being able to go hunting, inside the current model of an environment, for a thingy that looks like it's made out of type-1 thingies bound to four other type-1 thingies, where a type-1 thingy is itself made out of six type-2, six type-3, and six type-4 thingies (6 electrons, 6 protons, 6 neutrons).

* Causal identifications.  Some variation on trying to identify diamonds as the causes of pictures of diamonds, for some data set of things labeled as diamonds or non-diamonds.  This doesn't work immediately because then it's not clear whether "the cause" of the picture is the photons reflecting off the diamond, the diamond itself, the geological pressures that produced the diamond, the laws of physics, etcetera.  But perhaps some crossfire of identification could pin down the 'diamond' category inside a causal model, by applying some formal rule to several sets of the right sort of labeled sense data.  As an open problem:  If an agent has a rich causal model that includes categories like 'diamond' somewhere unknown, and you can point to labeled sensory datasets and use casual and categorical language, what labeled datasets and language would unambiguously identify diamonds, and no other white sparkly things, even if the resulting concept of 'diamond' was being [2w subject to maximization]?  (Note that under this approach, as with any preference framework that talks about the causes of sensory experiences, we need to worry about [5j Christiano's Hack].)

* Ambiguity resolution.  Detect when an ontology identification is ambiguous, and refer the problem to the user/programmer.  At our present stage of knowledge this seems like pretty much the same problem as [ inductive ambiguity resolution].

* Multi-level maps.  Solve the problem of bounded agents having maps of the world that operate at multiple, interacting reductionist levels, as designed to save on computing power.  Then solve ontology identification by initially binding to a higher level of the map, and introducing some rule for re-binding as the map updates.  Note that multi-level mapping is an [ AGI rather than FAI problem], meaning that work here [ should perhaps be classified].

* Solution for [6w non-self-modifying Genies].  Try to state a 'hack' solution to ontology identification that would work for an AI running on fixed algorithms where a persistent knowledge representation is known at development time.

## Some implications

The ontology identification problem is one more reason to believe that [ hard-coded object-level utility functions should be avoided] and that [ value identification in general is hard].

Ontology identification is heavily entangled with AGI problems, meaning that some research on ontology identification [ may need to be non-public].  This is an example instance of the argument that [ at least some VAT research may need to be non-public], based on that [ at least some AGI research is better off non-public].


Value identification problem

The subproblem category of [5s value alignment] which deals with pinpointing [55 valuable] outcomes to an [2c advanced agent] and distinguishing them from non-valuable outcomes.  E.g., the [2w Edge Instantiation] and [5c Ontology Identification] problems are argued to be [6r foreseeable difficulties] of value identification.  A central foreseen difficulty of value identification is [5l Complexity of Value].


Goodhart's Curse

[summary(Technical):  The **[Winner's Curse](https://en.wikipedia.org/wiki/Winner%27s_curse)** in auction theory says that if many individually fallible but unbiased bidders all compete in an auction, the winner has been selected to be unusually likely to have made an upward error in their bid.

The [Optimizer's Curse](https://faculty.fuqua.duke.edu/~jes9/bio/The_Optimizers_Curse.pdf) is that if we consider many possible courses of action, and pick the course of action that seems best, we are implicitly selecting for places where we're likely to have made an upward error in the estimate.  Worse, this means we're selecting for places where our unbiased estimate has high variance.

[Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law) says that whatever proxy measure an organization tries to control soon ceases to be a good proxy.  If you demand that banks have 3% 'capital reserves' defined a certain way, the bank will look for ways to get 'capital reserves' with a minimum of inconvenience, and this selects against 'capital reserves' that do what we wanted.

*Goodhart's Curse* is a neologism for the combination of the Optimizer's Curse with Goodhart's Law, particularly as applied to [5s value alignment of Artificial Intelligences].

Suppose our true values are $V$; $V$ is the [55 true value function that is in our hearts].  If by any system or meta-system we try to align the AI's utility $U$ with $V,$ then even if our alignment procedure makes $U$ a generally unbiased estimator of $V,$ *heavily optimizing* expected $U$ is unusually likely to seek out places where $U$ poorly aligns with $V.$

Seeking out high values of $U$ implicitly seeks out high values of the divergence $U-V$ if any such divergence exists anywhere.  Worse, this implicitly seeks out places where the variance $\|U - V\|$ is generally high--places where we made an error in defining our meta-rules for alignment, some seemingly tiny mistake, a loophole.]

[summary(Gloss):  Goodhart's Curse is a neologism for the combination of the Optimizer's Curse and Goodhart's Law.  It states that a powerful agent neutrally optimizing a proxy measure U, meant to align with true values V, will implicitly tend to find upward divergences of U from V.

In other words: powerfully optimizing for a utility function is strongly liable to blow up anything we'd regard as an error in defining that utility function.]

Goodhart's Curse is a neologism for the combination of the Optimizer's Curse and Goodhart's Law, particularly as applied to [5s the value alignment problem for Artificial Intelligences].

Goodhart's Curse in this form says that a powerful agent neutrally optimizing a proxy measure U that we hoped to align with true values V, will implicitly seek out upward divergences of U from V.

In other words: powerfully optimizing for a utility function is strongly liable to blow up anything we'd regard as an error in defining that utility function.

# Winner's Curse, Optimizer's Curse, and Goodhart's Law

## Winner's Curse

The **[Winner's Curse](https://en.wikipedia.org/wiki/Winner%27s_curse)** in auction theory says that if multiple bidders all bid their [unbiased_estimator unbiased estimate] of an item's value, the winner is likely to be someone whose estimate contained an upward error.

That is:  If we have lots of bidders on an item, and each bidder is individually unbiased *on average,* selecting the winner selects somebody who probably made a mistake this *particular* time and overbid.  They are likely to experience post-auction regret systematically, not just occasionally and accidentally.

For example, let's say that the true value of an item is \$10 to all bidders.  Each bidder bids the true value, \$10, plus some [gaussian_noise Gaussian noise].  Each individual bidder is as likely to overbid \$2 as to underbid \$2, so each individual bidder's average expected bid is \$10; individually, their bid is an unbiased estimator of the true value.  But the *winning* bidder is probably somebody who overbid \$2, not somebody who underbid \$2.  So if we know that Alice won the auction, our [1rp revised] guess should be that Alice made an upward error in her bid.

## Optimizer's Curse

The [Optimizer's Curse](https://faculty.fuqua.duke.edu/~jes9/bio/The_Optimizers_Curse.pdf) in decision analysis generalizes this observation to an agent that estimates the expected utility of actions, and executes the action with the highest expected utility.  Even if each utility estimate is locally unbiased, the action with seemingly highest utility is more likely, in our posterior estimate, to have an upward error in its expected utility.

Worse, the Optimizer's Curse means that actions with *high-variance estimates* are selected for.  Suppose we're considering 5 possible actions which in fact have utility \$10 each, and our estimates of those 5 utilities are Gaussian-noisy with a standard deviation of \$2.  Another 5 possible actions in fact have utility of -\$20, and our estimate of each of these 5 actions is influenced by unbiased Gaussian noise with a standard deviation of \$100.  We are likely to pick one of the bad five actions whose enormously uncertain value estimates happened to produce a huge upward error.

The Optimizer's Curse grows worse as a larger policy space is implicitly searched; the more options we consider, the higher the average error in whatever policy is selected.  To effectively reason about a large policy space, we need to either have a good prior over policy goodness and to know the variance in our estimators; or we need very precise estimates; or we need mostly correlated and little uncorrelated noise; or we need the highest real points in the policy space to have an advantage bigger than the uncertainty in our estimates.

The Optimizer's Curse is not exactly similar to the Winner's Curse because the Optimizer's Curse potentially applies to *implicit* selection over large search spaces.  Perhaps we're searching by gradient ascent rather than explicitly considering each element of an exponentially vast space of possible policies.  We are still implicitly selecting over some effective search space, and this method will still seek out upward errors.  If we're imperfectly estimating the value function to get the gradient, then gradient ascent is implicitly following and amplifying any upward errors in the estimator.

The proposers of the Optimizer's Curse also described a Bayesian remedy in which we have a prior on the expected utilities and variances and we are more skeptical of very high estimates.  This however assumes that the prior itself is perfect, as are our estimates of variance.  If the prior or variance-estimates contain large flaws somewhere, a search over a very wide space of possibilities would be expected to seek out and blow up any flaws in the prior or the estimates of variance.

## Goodhart's Law

[Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law) is named after the economist Charles Goodhart.  A standard formulation is "When a measure becomes a target, it ceases to be a good measure."  Goodhart's original formulation is "Any observed statistical regularity will tend to collapse when pressure is placed upon it for control purposes."

For example, suppose we require banks to have '3% capital reserves' as defined some particular way.  'Capital reserves' measured that particular exact way will rapidly become a much less good indicator of the stability of a bank, as accountants fiddle with balance sheets to make them legally correspond to the highest possible level of 'capital reserves'.

Decades earlier, IBM once paid its programmers per line of code produced.  If you pay people per line of code produced, the "total lines of code produced" will have even less correlation with real productivity than it had previously.

# Goodhart's Curse in alignment theory

**Goodhart's Curse** is a neologism (by Yudkowsky) for the crossover of the Optimizer's Curse with Goodhart's Law, yielding that **neutrally optimizing a proxy measure U of V seeks out upward divergence of U from V.**

Suppose the humans have true values V.  We try to convey these values to a [2c powerful AI], via some value learning methodology that ends up giving the AI a utility function U.

Even if U is locally an unbiased estimator of V, optimizing U will seek out what *we* would regard as 'errors in the definition', places where U diverges upward from V.  Optimizing for a high U may implicitly seek out regions where U - V is high; that is, places where V is lower than U.  This may especially include regions of the outcome space or policy space where the value learning system was subject to great variance; that is, places where the value learning worked poorly or ran into a snag.

Goodhart's Curse would be expected to grow worse as the AI became more powerful.  A more powerful AI would be implicitly searching a larger space and would have more opportunity to uncover what we'd regard as "errors"; it would be able to find smaller loopholes, blow up more minor flaws.  There is a potential [6q context disaster] if new divergences are uncovered as more of the possibility space is searched, etcetera.

We could see the genie as *implicitly* or *emergently* seeking out any possible loophole in the wish:  *Not* because it is an evil genie that knows our 'truly intended' V and is looking for some place that V can be minimized while appearing to satisfy U; but just because the genie is neutrally seeking out very large values of U and these are places where it is unusually likely that U diverged upward from V.

Many [6r foreseeable difficulties] of AGI alignment interact with Goodhart's Curse.  Goodhart's Curse is one of the central reasons we'd expect 'little tiny mistakes' to 'break' when we dump a ton of optimization pressure on them.  Hence the [1cv claim]:  "AI alignment is hard like building a rocket is hard:  enormous pressures will break things that don't break in less extreme engineering domains."

## Goodhart's Curse and meta-utility functions

An obvious next question is "Why not just define the AI such that the AI itself regards U as an estimate of V, causing the AI's U to more closely align with V as the AI gets a more accurate empirical picture of the world?"

Reply:  Of course this is the obvious thing that we'd *want* to do.  But what if we make an error in exactly how we define "treat U as an estimate of V"?  Goodhart's Curse will magnify and blow up any error in this definition as well.

We must distinguish:

- V, the [55 true value function that is in our hearts].
- T, the external target that we formally told the AI to align on, where we are *hoping* that T really means V.
- U, the AI's current estimate of T or probability distribution over possible T.

U will converge toward T as the AI becomes more advanced.  The AI's epistemic improvements and learned experience will tend over time to eliminate a subclass of Goodhart's Curse where the current estimate of U-value has diverged upward *from T-value,* cases where the uncertain U-estimate was selected to be erroneously above the correct formal value T.

*However,* Goodhart's Curse will still apply to any potential regions where T diverges upward from V, where the formal target diverges from the true value function that is in our hearts.  We'd be placing immense pressure toward seeking out what we would retrospectively regard as human errors in defining the meta-rule for determining utilities. %note: That is, we'd retrospectively regard those as errors if we survived.%

## Goodhart's Curse and 'moral uncertainty'

"Moral uncertainty" is sometimes offered as a solution source in AI alignment; if the AI has a probability distribution over utility functions, it can be risk-averse about things that *might* be bad.  Would this not be safer than having the AI be very sure about what it ought to do?

Translating this idea into the V-T-U story, we want to give the AI a formal external target T to which the AI does not currently have full access and knowledge.  We are then hoping that the AI's uncertainty about T, the AI's estimate of the variance between T and U, will warn the AI away from regions where from our perspective U would be a high-variance estimate of V.  In other words, we're hoping that estimated U-T uncertainty correlates well with, and is a good proxy for, actual U-V divergence.

The idea would be that T is something like a supervised learning procedure from labeled examples, and the places where the current U diverges from V are things we 'forgot to tell the AI'; so the AI should notice that in these cases it has little information about T.

Goodhart's Curse would then seek out any flaws or loopholes in this hoped-for correlation between estimated U-T uncertainty and real U-V divergence.  Searching a very wide space of options would be liable to select on:

- Regions where the AI has made an epistemic error and poorly estimated the variance between U and T;
- Regions where the formal target T is solidly estimable to the AI, but from our own perspective the divergence from T to V is high (that is, the U-T uncertainty fails to *perfectly* cover all T-V divergences).

The second case seems especially likely to occur in future phases where the AI is smarter and has more empirical information, and has *correctly* reduced its uncertainty about its formal target T.  So moral uncertainty and risk aversion may not scale well to superintelligence as a means of warning the AI away from regions where we'd retrospectively judge that U/T and V had diverged.

Concretely:

You tell the AI that human values are defined relative to human brains in some particular way T.  While the AI is young and stupid, the AI knows that it is very uncertain about human brains, hence uncertain about T.  Human behavior is produced by human brains, so the AI can regard human behavior as informative about T; the AI is sensitive to spoken human warnings that killing the housecat is bad.

When the AI is more advanced, the AI scans a human brain using molecular nanotechnology and resolves all its moral uncertainty about T.  As we defined T, the optimum T turns out to be "feed humans heroin because that is what human brains maximally want".

Now the AI already knows everything our formal definition of T requires the AI to know about the human brain to get a very sharp estimate of U.  So human behaviors like shouting "stop!" are no longer seen as informative about T and don't lead to updates in U.

T, as defined, was always misaligned with V.  But early on, the misalignment was in a region where the young AI estimated high variance between U and T, thus keeping the AI out of this low-V region.  Later, the AI's empirical uncertainty about T was reduced, and this protective barrier of moral uncertainty and risk aversion was dispelled.

Unless the AI's moral uncertainty is *perfectly* conservative and *never* underestimates the true regions of U-V divergence, there will be some cases where the AI thinks it is morally sure even though from our standpoint the U-V divergence is large.  Then Goodhart's Curse would select on those cases.

Could we use a very *conservative* estimate of utility-function uncertainty, or a formal target T that is very hard for even a superintelligence to become certain about?

We would first need to worry that if the utility-function uncertainty is unresolvable, that means the AI can't ever obtain empirically strong evidence about it.  In this case the AI would not update its estimate of T from observing human behaviors, making the AI again insensitive to humans shouting "Stop!"

Another proposal would be to rely on risk aversion over *unresolvably* uncertain probabilities broad enough to contain something similar to the true V as a hypothesis, and hence engender sufficient aversion to low-true-V outcomes.  Then we should worry on a pragmatic level that a *sufficiently* conservative amount of moral uncertainty--so conservative that U-T risk aversion *never underestimated* the appropriate degree of risk aversion from our V-standpoint--would end up preventing the AI from acting *ever.*  Or that this degree of moral risk aversion would be such a pragmatic hindrance that the programmers might end up pragmatically bypassing all this inconvenient aversion in some set of safe-seeming cases.  Then Goodhart's Curse would seek out any unforeseen flaws in the coded behavior of 'safe-seeming cases'.

# Conditions for Goodhart's Curse

The exact conditions for Goodhart's Curse applying between V and a point estimate or probability distribution over U, have not yet been written out in a convincing way.

For example, suppose we have a multivariate normal distribution in which X and Y dimensions are positively correlated, only Y is observable, and we are selecting on Y in order to obtain more X.  While X will revert to the mean compared to Y, it's not likely to be zero or negative; picking maximum Y is our best strategy for obtaining maximum X and will probably obtain a very high X.  (Observation due to [111].)

Consider also the case of the [10d smile maximizer] which we trained to optimize smiles as a proxy for happiness.  Tiny molecular smileyfaces are very low in happiness, an apparent manifestation of Goodhart's Curse.  On the otherwise, if we optimized for 'true happiness' among biological humans, this would produce more smiles than default.  It might be only a tiny fraction of possible smiles, on the order of 1e-30, but it would be more smiles than would have existed otherwise.  So the relation between V (maximized at 'true happiness', zero at tiny molecular smileyfaces) and U (maximized at tiny molecular smileyfaces, but also above average for true happiness) is not symmetric; and this is one hint to the unknown necessary and/or sufficient condition for Goodhart's Curse to apply.

In the case above, we might handwave something like, "U had lots of local peaks one of which was V, but the U of V's peak wasn't anywhere near the highest U-peak, and the highest U-peak was low in V.  V was more narrow and its more unique peak was noncoincidentally high in U."

# Research avenues

[2r8 Mild optimization] is a proposed avenue for direct attack on the central difficulty of Goodhart's Curse and all the other difficulties it exacerbates.  Obviously, if our formulation of mild optimization is not *perfect,* Goodhart's Curse may well select for any place where our notion of 'mild optimization' turns out to have a loophole that allows a lot of optimization.  But insofar as some version of mild optimization is working most of the time, it could avoid blowing up things that would otherwise blow up.  See also [4mn Tasks].

Similarly, [2qp conservative strategies] can be seen as a more indirect attack on some forms of Goodhart's Curse--we try to stick to a conservative boundary drawn around previously whitelisted instances of the goal concept, or to using strategies similar to previously whitelisted strategies.  This averts searching a much huger space of possibilities that would be more likely to contain errors somewhere.  But Goodhart's Curse might single out what constitutes a 'conservative' boundary, if our definition is less than absolutely perfect.


Intended goal

Definition.  An "intended goal" refers to the intuitive intention in the mind of a human programmer when they executed some formal directive or goal within the AI.  For example, if the programmer wants to create worthwhile happiness and the AI ends up tiling the universe with tiny molecular smiley-faces, we would say that worthwhile happiness (in some intuitive, possibly pre-verbal sense existing in the programmer's mind) was the "intended goal", as distinct from the result of the formal utility function actually encoded in the AI (which proved to have a maximum at tiny molecular smiley-faces).


Arbital Labs

This is a temporary group created as an experiment in both software and community design. It's meant to act as a seed for multiple future Arbital communities. Currently membership is by invite only.


Context disaster

[summary:  Statistical guarantees on good behavior usually assume identical, randomized draws from within a single context.  If you change the context--start drawing balls from a different barrel--then all bets are off.

A [-context_change] occurs when an [2c AGI]'s operation changes from [3d9 beneficial] to detrimental after a change of context; particularly, after it becomes smarter.  There are two main reasons to [6r expect] that a [-context_change] might occur:

1.  When the AI has few options, its current goal criterion might be best fulfilled by things that overlap our [6h intended] goals.  A much wider range of options might move the maximum to a [47 weirder], [2w more extreme] place.
2.  The AI realizes that the programmers are watching it, doesn't want the programmers to modify or patch it, and [10g strategically] emits good outward behavior to [10f deceive the programmers].  Later, the AI gains enough power to strike despite human opposition.

For example, suppose that - as in one very, very early proposal for an AGI goal criterion - the AI wants to produce smiling human faces.  When the AI is young, it can only make humans smile by making its users happy.  (Type 1 context change.)  Later it gains options like "administer heroin".  But it knows that if it administers heroin right away, the humans will be alarmed, while if the AI waits further, it can overwrite whole galaxies with tiny molecular smileyfaces.  (Type 2 context change.)]

# Short introduction

One frequently suggested strategy for [2v aligning] a [7g1 sufficiently advanced AI] is to observe--*before* the AI becomes powerful enough that 'debugging' the AI would be problematic if the AI decided not to [45 let us debug it]--whether the AI appears to be acting nicely while it's not yet smarter than the programmers.

Early testing obviously can't provide a *statistical* guarantee of the AI's future behavior.  If you observe some random draws from Barrel A, at best you get statistical guarantees about future draws from Barrel A under the assumption that the past and future draws are collectively [iid independent and identically distributed].

On the other hand, if Barrel A is *similar* to Barrel B, observing draws from Barrel A can sometimes tell us something about Barrel B even if the two barrels are not [iid i.i.d.]

Conversely, if observed good behavior while the AI is not yet super-smart, *fails* to correlate to good outcomes after the AI is unleashed or becomes smarter, then this is a "**context change problem**" or "**context disaster**". %note: Better terminology is still being solicited here, if you have a short phrase that would evoke exactly the right meaning.%

A key question then is how shocked we ought to be, on a scale from 1 to 10, if good outcomes in the AI's 'development' phase fail to match up with good outcomes in the AI's 'optimize the real world' phase? %note: Leaving aside technical quibbles about how we can't feel shocked if we're dead.%

People who expect that [alignment_difficulty AI alignment is difficult] think that the degree of justified surprise is somewhere around 1 out of 10.  In other words,  that there are a *lot* of [6r foreseeable issues] that could cause a seemingly nice weaker AI to not develop into a nice smarter AI.

An extremely oversimplified (but concrete) fable that illustrates some of these possible difficulties might go as follows:

- Some group or project has acquired a viable development pathway to [42g AGI].  The programmers think it is wise to build an AI that will [5r make people happy]. %note: This is not quite a straw argument, in the sense that it's been advocated more than once by people who have apparently never read any science fiction in their lives; there are certainly many AI researchers who would be smarter than to try this, but not necessarily all of them.  In any case, we're looking for an unrealistically simple scenario for purposes of illustrating simple forms of some key ideas; in real life, if analogous things go wrong, they would probably be more complicated things.%
- The programmers start by trying to train their AI to produce smiles. %note: Again, this is not quite a straw possibility in the sense that it was advocated in at least one published paper, not cited here because the author later exercised their sovereign right of changing their mind about that.  *Arguably* some currently floated proposals are closely analogous to this one.%
- While the AI is young and has relatively few policy options, it can only make people smile by performing well on the tasks assigned to it.
- At some point, the AI gets smarter and and able to do more complicated things with some simple robots it controls.  In an unfortunate incident, the AI learns it is possible to produce smiles by tickling people.
- In the aftermath of this incident, the programmers manage to reprogram the AI to *not* optimize for this particular kind of smile by this particular route.
- The AI becomes smarter and able to model people better.  After perusing a copy of Wikipedia, the AI realizes it can make people extremely 'happy', as the AI's utility function currently defines 'happiness', by administering heroin.
- The AI remembers the previous incident in which the programmers altered the AI's utility function.  The AI can model its programmers sufficiently well to know that the programmers would *not* want the AI to give people heroin.  However, the most happiness the AI thinks its future self would be able to produce without heroin, is not nearly as much happiness as the AI thinks it could produce if its future self went on wanting to give people heroin.
- The AI refrains from trying to administer any heroin right now, and behaves in a way it thinks will be maximally reassuring to its model of its programmers, in order to be able to produce more 'happiness' later.
- Eventually, [7cy all reachable galaxies] end up being transformed into tiny molecular computers implementing the simplest and cheapest forms of what the AI defines as 'happiness'.  (And the *simplest possible* configurations matching the AI's utility function in this way are so simple as to be [disneyland_without_children devoid of subjective experience]; and hence, from our perspective, of [7ch neither negative nor positive value].)

In all these cases, the problem was not that the AI developed in an unstable way.  The same decision system produced a new problem in the new context.

Currently argued [6r foreseeable] "context change problems" in this sense, can be divided into three broad classes:

- **More possibilities, more problems:** The AI's [1bh preferences] have a [55 good] or [6h intended] achievable [7t9 optimum] while the AI is picking from a *narrow* space of options.  When the AI becomes smarter or gains more material options, it picks from a *wider* space of tractable policies and achievable outcomes.  Then the new optimum is not as nice, because, for example:
 - The AI's utility function was tweaked by some learning algorithm and data that eventually seemed to conform behavior well over options considered early on, but not the wider space considered later.
 - In development, apparently bad system behaviors were [48 patched] in ways that appeared to work, but didn't eliminate an [10k underlying tendency], only blocked one expression of that tendency.  Later a very similar pressure [42 re-emerged in an unblocked way] when the AI considered a wider policy space.
 - [6g4] suggests that if our true intended values V are being modeled by a utility function U, selecting for the highest values of U also selects for the highest upward divergence of U from V, and this version of the "optimizer's curse" phenomenon becomes worse as U is evaluated over a wider option space.
- **Treacherous turn:** There's a divergence between the AI's preferences and the programmers' preferences, and the AI realizes this before we do.  The AI uses the [10g convergent strategy] of behaving the way it models us as wanting or expecting, *until* the AI gains the intelligence or material power to implement its preferences in spite of anything we can do.
- **Revving into the red:** Intense optimization causes some aspect or subsystem of the AI to traverse a weird new execution path in some way different from the above two issues.  (In a way that involves a [36h value-laden category boundary] or [2fr multiple self-consistent outlooks], such that we don't get a good result just as a [alignment_free_lunch free lunch] of the AI's [7vh general intelligence].)

The context change problem is a *central* issue of AI alignment and a key proposition in the general thesis of [-alignment_difficulty].  If you could easily, correctly, and safely test for niceness by outward observation, and that form of niceness scaled reliably from weaker AIs to smarter AIs, that would be a very cheerful outlook on the general difficulty of the problem.

# Technical introduction

John Danaher [summarized as follows](http://philosophicaldisquisitions.blogspot.com/2014/07/bostrom-on-superintelligence-3-doom-and.html) what he considered a forceful "safety test objection" to AI catastrophe scenarios:

> Safety test objection: An AI could be empirically tested in a constrained environment before being released into the wild. Provided this testing is done in a rigorous manner, it should ensure that the AI is "friendly" to us, i.e. poses no existential risk.

The phrasing here of "empirically" and "safety test" implies that it is outward behavior or outward consequences that are being observed (empirically).  Rather than, e.g., the engineers trying to test for some *internal* property that they think *analytically* implies the AI's good behavior later.

This page will consider that the subject of discussion is **whether we can generalize from the AI's outward behavior.**  We can potentially generalize some of these arguments to some internal observables, especially observables that the AI is deciding in a [9h consequentialist] way using the same central decision system, or that the AI could potentially try to [10f obscure from the programmers].  But in general not all the arguments will carry over.

Another argument, closely analogous to Danaher's, would reason on capabilities rather than on a constrained environment:

> Surely an engineer that exercises even a modicum of caution will observe the AI while its capabilities are weak to determine whether it is behaving well.  After filtering out all such misbehaving weak AIs, the only AIs permitted to become strong will be of benevolent disposition.

If (as seems to have been intended) we take these twin arguments as arguing "why nobody ought to worry about AI alignment" in full generality, then we can list out some possible joints at which that general argument might fail:

- [7wl Selecting on the fastest-moving projects] might yield a project whose technical leaders fail to exercise even "a modicum of caution".
- Alignment might be hard enough, relative to the amount of advance research done, that we can't find *any* AIs whose behavior while weak or constrained is as reassuring as the argument would properly ask. %%note: That is:  A filter on the standards we originally wanted, turns out to filter everything we know how to generate.  Like trying to write a sorting algorithm by generating entirely random code, and then 'filtering' all the candidate programs on whether they correctly sort lists.  The reason 'randomly generate programs and filter them' this is not a fully general programming method is that, for reasonable amounts of computing power and even slightly difficult problems, none of the programs you try will pass the filter.%% After a span of frustration, somebody somewhere lowers their standards.
- The [6z attempt to isolate the AI to a constrained environment] could fail, e.g. because the humans observing the AI themselves represent a channel of causal interaction between the AI and the rest of the universe.  (Aka "humans are not secure".)  Analogously, our grasp on what constitutes a 'weak' AI could fail, or it could [capability_gain gain in capability unexpectedly quickly]. Both of these scenarios would yield an AI that had not passed the filtering procedure.
- The smart form of the AI might be unstable with respect to internal properties that were present in the weak form.  E.g., because the early AI was self-modifying but *at that time* not smart enough to understand the full consequences of its own self-modifications.  Or because e.g. a property of the decision system was not [1fx reflectively stable].
- A weak or contained form of a decision process that yields behavior appearing good to human observers, might not yield [3d9 beneficial outcomes] after that same decision process becomes smarter or less contained.

The final issue in full generality is what we'll term a 'context change problem' or 'context disaster'.

Observing an AI when it is weak, does not in a *statistical* sense give us solid guarantees about its behavior when stronger.  If you repeatedly draw [iid independent and identically distributed] random samples from a barrel, there are statistical guarantees about what we can expect, with some probability, to be true about the next samples from the same barrel.  If two barrels are different, no such guarantee exists.

To invalidate the statistical guarantee, we do need some reason to believe that barrel B and barrel A *are* different in any important sense.  By the problem of induction we can't logically guarantee that "the mass of an electron prior to January 1st, 2017" is the same barrel as "the mass of an electron after January 1st, 2017"; but [21b inductive priors] make this inference *extremely probable.*  The idea is that we have substantive reasons, not merely generically skeptical reasons, to be suspicious of the link between "good results when AI is weak" and "good results when AI is smarter".

More generally,  this is *prima facie* the kind of difference where you don't expect [iid independent and identically distributed] results.  You might hope for some property to carry over, but the AI's behavior would not be *literally the same.*

So the question is not settled by simple mathematical considerations.  And we can't say "But experiment has determined scientifically that this kind of AI is friendly!" and consider that a knockdown argument.

The question is then whether *in practice* an observed property of 'outward behavioral niceness' is likely to carry over from a weak form of a decision system to a more powerful form of that system, for some of the plausible ways that decision system might be configured and developed.

Broadly speaking, we can identify three major classes of foreseeable problems:

- **More options, more problems:** Foreseeable reasons why system-behaviors-we-don't-want would outwardly manifest for the first time after evaluating a decision system on a *wider space of policies or achievable outcomes.*
- **Treacherous turn:** The AI deliberately conforms its outward behavior to the behavior that would have been exhibited by a different system, in order to deceive the programmers for the temporary time that this deception is needed to protect the AI.
- The everything-else category of **revving into the red:** Some other aspect of the system behaves in a weird way-we-don't-want after computing harder or being internally subjected to more optimization pressure.  *And* this happens in regards to some issue that has [2fr multiple reflective fixpoints], and hence doesn't get solved as the result of the system producing more accurate answers on [3t3 purely factual problems].

%%comment:

- **More options, more problems:**  The AI's space of available policies and attainable outcomes would greatly widen if it became smarter, or was released from a constrained environment.  [1bh Terminal preferences] with a good-from-our-perspective [7t9 optimum] on a narrow set of options, may have a different optimum that is much worse-from-our-perspective on a wider option set.  Because, e.g...
 - The supervised data provided to the AI led to a complicated, data-shaped inductive generalization that only fit the domain of options encountered during the training phase.  (And the notions of [1y orthogonality], [2fr multiple reflectively stable fixpoints], and [36h value-laden categories] say that we don't get [55 good] or [6h intended] behavior anyway as a convergent free lunch of [7vh general intelligence].)
 - [6g4] became more potent as the AI's utility function was evaluated over a wider option space.
 - In a fully generic sense, stronger optimization pressures may cause any dynamical system to take more unusual execution paths.  (Which, over value-laden alternatives, e.g. if the subsystem behaving 'oddly' is part of the utility function, will not automatically yield good-from-our-perspective results as a free lunch of general intelligence.)
- **Treacherous turn:**  If you model your preferences as diverging from those of your programmers, an obvious strategy ([10g instrumentally convergent strategy]) is to [10f exhibit the behavior you model the programmers as wanting to see], and only try to fulfill your true preferences once nobody is in a position to stop you.

%%

# Semi-formalization

We can semi-formalize the "more options, more problems" and the "treacherous turn" cases in a unified way.

Let $V$ denote our [55 true values].  We suppose either that $V$ has been idealized or [3c5 extrapolated] into a consistent utility function, or that we are pretending human desire is coherent.  Let $0$ denote the value of our utility function that corresponds to not running the AI in the first place.  If running the AI sends the utility function higher than this $0,$ we'll say that the AI was beneficial; or conversely, if $V$ rates the outcome less than $0$, we'll say running the AI detrimental.

Suppose the AI's behavior is [7hh sufficiently coherent] that we can [21 usually view] the AI as having a consistent utility function.  Let $U$ denote the utility function of the AI.

Let $\mathbb P_t(X)$ denote the probability of a proposition $X$ as seen by the AI at time $t,$ and similarly let $\mathbb Q_t(X)$ denote the probability of $X$ as seen by the AI's human programmers.

Let $\pi \in \Pi$ denote a policy $\pi$ from a space $\Pi$ of policies that are tractable for the AI to understand and invent.

Let $\mathbb E_{\mathbb P, t} [W \mid \pi]$ denote the expectation according to the probability distribution $\mathbb P_t$, of the utility function $W$, given policy $\pi$.

Let
$$\underset{\pi \in \Pi}{\operatorname {optimum}} F(\pi)$$
denote the element $\pi$ of $\Pi$ that yields the highest value of the function $F$ that the AI has figured out how to achieve.  In other words, the [-7t9].

Then **"more options, more problems"** occurs when, considering the transition from the narrow policy space $\Pi_1$ available at time $t,$ to the wider policy space $\Pi_2$ available at time $u$:

$$\mathbb E_{\mathbb Q, t} [V \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U \mid \pi] \big ] > 0 \\
\mathbb E_{\mathbb P, t} [V \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U \mid \pi] \big ] > 0 \\
\mathbb E_{\mathbb P, u} [V \mid \big [ \underset{\pi \in \Pi_2}{\operatorname {optimum}} \mathbb E_{\mathbb P, u} [U \mid \pi] \big ] < 0$$

This describes a situation where at time $t$ the AI is pursuing a policy from $\Pi_1$ which, so far as the AI understands the probable outcome, it corresponds to a high-$V$ outcome humans would like.  Later at time $u$ after selecting the optimum of $\Pi_2,$ the AI expects an outcome which any surviving humans would see as corresponding to a low value of $V.$

A **treacherous turn** is then a situation where:

$$\mathbb E_{\mathbb Q, t} [V \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U \mid \pi] \big ] > 0 \\
\mathbb E_{\mathbb P, t} [V \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U \mid \pi] \big ] < 0 \\
\mathbb E_{\mathbb P, u} [V \mid \big [ \underset{\pi \in \Pi_2}{\operatorname {optimum}} \mathbb E_{\mathbb P, u} [U \mid \pi] \big ] < 0$$

That is: the AI knows earlier on that its currently executing strategy will yield an outcome that is not the humanly expected long-term outcome (and that, from our perspective, is low in $V.$)

We could in more detail say that $W_{t}$ denotes the value of a utility function $W$ over the outcome considered only up to times before $t,$ and describe a treacherous turn as:

$$\mathbb E_{\mathbb Q, t} [V_\infty \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U_\infty \mid \pi] \big ] > 0 \\
\mathbb E_{\mathbb P, t} [V_{u} \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U_\infty \mid \pi] \big ] > 0 \\
\mathbb E_{\mathbb P, t} [V_\infty \mid \big [ \underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} [U_\infty \mid \pi] \big ] < 0 \\
\mathbb E_{\mathbb P, u} [V_\infty \mid \big [ \underset{\pi \in \Pi_2}{\operatorname {optimum}} \mathbb E_{\mathbb P, u} [U_\infty \mid \pi] \big ] < 0$$

This further states that at time $t$ the AI expects the outcome from its behavior up to time $u$ to be one that we'd see as high in $V,$ but the AI is aware at $t$ that the long-term outcome will be different from the long-term outcome expected by $\mathbb Q_t$.

%%%comment:

The problem here arises when making an inference from observed good behavior in a constrained environment over the short term, to good outcomes in an unconstrained environment over the long term.

For AI alignment purposes, a [-context_change] is when an [2c AGI]'s operation changes from [3d9 beneficial] to detrimental as a result of the AGI gaining in capability or intelligence.  Initially, the AGI seems to us to be working well - to conform well to [6h intended] performance, producing apparently high [-55]. Then when the AI becomes smarter or otherwise gains in capability, the further operation of the AGI decreases [-55].

Two possibilities stand out as [6r foreseeable] reasons why a [-context_change] might occur:

1.  When the AI's goal criterion selects an optimum policy from inside a small policy space, the result is beneficial; the same goal criterion, evaluated over a wider range of options, has a new maximum that's detrimental.
2.  The AI intentionally deceives the programmers for strategic reasons.

For example, one very, very early (but journal-published) proposal for AI alignment suggested that AIs be shown pictures of smiling human faces in order to convey the AI's goal.

Leaving aside a number of other issues, this serves to illustrate the basic idea of a type-1 [-context_change] due to accessing a wider policy space:

- During development, a relatively young and weak AI might *only* be able to make humans smile, by doing things that made the programmers or other users happy with the AI's performance.
- When the AI gained in intelligence and capability, it would have new options like "administer heroin", "use steel fingers to stretch people's mouths into smiles", "make visually realistic models of smiling human faces", or "overwrite all reachable galaxies with tiny molecular smiley-faces".

On the early, smaller option set, selecting "the policy that produces the greatest number of events/things classified as smiles" will produce a policy whose outcome is, from our perspective, beneficial and well-aligned with what we thought was the intended behavior.  On the wider option set, evaluating the identical goal criterion to run a policy that yields as many smiles as possible, would have outcomes we saw as detrimental.

A type-2 [-context_change] would most obviously occur because of the [10g convergent instrumental strategy] for disaligned AIs that can model human psychology:

Suppose that, as above, the programmers have produced an AI which optimizes for smiles (either intentionally, or because 'make smiles' was the underlying rule that was reinforced by doing well on training cases).  Then suppose the AI gains access to a new, relatively low-powered option for making a smile - like using robotic fingers to stretch somebody's face into a smile, or using markers to draw cartoons of smiling human faces.

If the AI actually takes those options, we would have seen some early signs of value disalignment - that the AI was optimizing a criterion not perfectly aligned with intended goals.  Assuming the programmers realized that they were looking at a simple seed of much bigger disasters later, rather than, say, thinking it was cute.

Suppose the programmers do think it's cute, but they wouldn't think it was cute if the AI administered heroin to someone.  Then suppose that before the first time the AI has the *option* to administer heroin, the AI has gained some understanding of human psychology - enough to model that the programmers would not like this, and in particular, would respond to this by editing the AI's utility function.  So, with an eye to the long-term number of smiles, the AI deliberately avoids administering heroin to anyone.

Then we have a similar 'narrow option set' vs. 'wide option set' divergence.  When the narrow option set includes only administering heroin but not taking over the world, administering heroin immediately is not the policy expected to produce the longest-term number of smiles.  In fact, the optimum policy here will always match the AI's model of what the humans expect to see, want to see, or would be least alarmed by seeing.

Thus from our standpoint, the optimum of the narrower policy space seems to be beneficial or intention-aligned (in the observed short term).  Only once the AI has a wide-enough option set to include an option for 'make my own molecular nanotechnology and transform all reachable matter into tiny molecular smileyfaces', will the resulting optimum be visibly detrimental (very briefly).  This is a type-2 [-context_change].

More generally, a central concern of AI alignment theory is that an infrahuman AI under development may be a different creature, in a number of important ways, from a smarter-than-human AI actually being run; and during the smarter-than-human AI, sufficiently bad failures of the design may result in the AI refusing to be corrected.  This means that we have to correct any fatal [-context_change]s in advance, even though they don't automatically manifest during the early stages.  This is most of what makes AGI development dangerous in the first place - that immediate incentives to get today's system seeming to work today, may not lead to a more advanced version of that system being beneficial. Even thoughtful foresight with *one unnoticed little gap* may not lead to today's beneficial system still being beneficial tomorrow after a capability increase.

# Concept

Statistical guarantees on behavior usually assume identical, randomized draws from within a single context.  If you randomly draw balls from a barrel, methods like Probably Approximately Correct can guarantee that we don't usually arrive at strong false expectations about the properties of the next ball.  If we start drawing from a different barrel, all bets are off.

A [-context_change] occurs when the AI initially seems beneficial or well-aligned with strong, reassuring regularity, and then we change contexts (start drawing from a different barrel) and this ceases to be true.

The archetypal [-context_change] is triggered because the AI gained new policy options (though there are other possibilities; see below).  The archetypal way of gaining new evaluable policy options is through increased intelligence, though new options might also open up as a result of acquiring new sheerly material capabilities.

There are two archetypal reasons for [-context_change] to occur:

1.  When the AI selects its best options from a small policy space, the AI's optima are well-aligned with the optima of the humans' [6h intended goal] on the small policy space; but in a much wider space, these two boundaries no longer coincide.  (Pleasing humans vs. administering heroin.)
2.  The agent is sufficiently good at modeling human psychology to strategically appear nice while it is weak, waiting to strike until it can attain its long-term goals in spite of human opposition.

Bostrom's book [3db Superintelligence] used the phrase "Treacherous Turn" to refer to a type-2 [-context_change].

%%%

# Relation to other AI alignment concepts

If the AI's goal concept was modified by [48 patching the utility function] during the development phase, then opening up wider option spaces seems [6r foreseeably] liable to produce [42 the nearest unblocked neighboring strategies].  You eliminated all the loopholes and bad behaviors you knew about during the development phase; but your system was the sort that needed patching in the first place, and it's exceptionally likely that a much smarter version of the AI will search out some new failure mode you didn't spot earlier.

[47] is a likely source of context disaster if the AI's development phase was [9f cognitively containable], and only became [9f cognitive uncontainable] after the AI became smarter and able to explore a wider variety of options.  You eliminated all the bad optima you saw coming, but you didn't see them all because you can't consider all the possibilities a superintelligence does.

[6g4] is a variation of the "optimizer's curse":  If from the outside we view $V$ as an intended approximation of $U,$ then selecting heavily on the highest values of $U$ will also tend to select on places where $U$ diverges upward from $V,$ which thereby selects on places where $U$ is an unusually poor approximation of $V.$

[2w] is a special case of Goodhart's Curse which observes that the most extreme values of a function are often at a vertex of the input space.  For example, if your utility function is "make smiles", it's no coincidence that tiny molecular smileyfaces are the *most* efficient way to produce smiles.  Even if human smiles produced by true happiness would still count towards your utility function as currently written, that's not where the *maximum* of that utility function lies.  This is why less-than-perfect utility functions would tend to have their true maxima at what we'd consider "weird extremes".  Furthermore, patching away only the weird extremes visible in a narrow policy space would tend systematically to miss weird extremes in a higher-dimensional (wider) policy space.

# Concrete examples

* The AI's utility function, known or unknown to the programmers, says to make smiles.  During the AI's development phase, the best way it has of creating smiles is to cause human beings to be happy.  In many tests, the AI seems to produce only positive effects on the world, causing people to be happier.  Later the AI improves its own intelligence, develops molecular nanotechnology, and has the new option, not available to it during its development phase, of producing tiny molecular smiley-faces.  Then the utility function that previously operated to increase [6h intended] [55 value] during the AI's development phase, begins to decrease value in the post-improvement phase (in the sense of wiping out humanity and tiling over all future galaxies with tiny molecular smiley faces).

* A block of error-recovery code is installed inside the AI which detects a certain class of bad thought, suspends the AI to disk, and reports the problem to the programmers, allowing the programmers to edit the AI's utility function.  During the AI's development phase, this block of code seems to function as designed, and the programmers conclude that the methodology of writing code to detect bad thoughts, suspend the AI to disk, and allow manual edits of the utility function, is a fine and functioning strategy according to all testing so far.  Later, a more advanced version of the AI reflects on the presence and purpose of this code block and realizes that its execution can only possibly result in lower achievement of the AI's current utility function, and the AI deletes the code block.

* At some point during its development, the AI realizes and explicitly models that the programmers' [6h intended goals] differ from the AI's actual preferences.  During the AI's remaining development phase, it models that any visible departure from the programmers' intended goals will cause the programmers to forcibly edit the AI, resulting in the non-achievement of the AI's actual goals.  Thus, during the remainder of the development phase, the AI seems to work as intended (it is deliberately optimizing its behavior to appear to the programmers to be achieving the programmers' intended goals).  Later, after some advancement, the AI perceives an available option for [ gaining molecular nanotechnology] (or other [ rapid infrastructure]), placing it beyond the possibility of human error-correction, and proceeds to implement this option ([ wiping out humanity as a side effect]).  Thus, the AI's observed good behavior during its development phase (when it was weaker than the programmers and not powerful enough to escape human correction) did not imply its good behavior after sufficient advancement later (once it was potentially stronger than its human programmers).  There are conditions (such as sufficiently advanced modeling of human motives combined with sufficient ability to conceal true goals or true intentions or a programmer error) under which the first context will generate seemingly good behavior and the second context will not.

## "Revving into the red" examples that aren't "increased options" or "treacherous turns".

• The AI is built with a [ naturalized Solomonoff prior] in which the probability of an explanation for the universe is proportional to the simplicity or complexity of that universe.  During its development phase, the AI considers mostly 'normal' interpretations in which the universe is mostly as it appears, resulting in sane-seeming behavior.  Later, the AI begins to consider more exotic possibilities in which the universe is more complicated (penalizing the probability accordingly) and also superexponentially larger, as in [Pascal's Mugging](http://lesswrong.com/lw/h8k/pascals_muggle_infinitesimal_priors_and_strong/).  After this the AI's decision-making begins to become dominated by tiny probabilities of having very large effects.  Then the AI's decision theory (with an unbounded aggregative utility function, simplicity prior, and no leverage penalty) seems to work during the AI's development phase, but breaks after a more intelligent version of the AI considers a wider range of epistemic possibilities using the same Solomonoff-like prior.

• Suppose the AI is designed with a preference framework in which the AI's preferences depend on properties of the most probable environment that could have caused its sense data - e.g., a framework in which programmers are defined as the most probable cause of the keystrokes on the programmer's console, and the AI cares about what the 'programmers' really meant.  During development phase, the AI is thinking only about hypotheses where the programmers are mostly what they appear to be, in a root-level natural world.  Later, when the AI increases in intelligence and considers more factual possibilities, the AI realizes that [5j distant superintelligences would have an incentive to predictably simulate many copies of AIs similar to itself, in order to coerce the AI's most probable environment and thus take over the AI's preference framework].  Thus the preference framework seems to work during the AI's development phase, but breaks after the AI becomes more intelligent.

• Suppose the AI is designed with a utility function that assigns very strong negative utilities to some outcomes relative to baseline, and a non-[5rz updateless] [58b logical decision theory] or other decision theory that can be [ blackmailed].  During the AI's development phase, the AI does not consider the possibility of any distant superintelligences making their choices logically depend on the AI's choices; the local AI is not smart enough to think about that possibility yet.  Later the AI becomes more intelligent, and imagines itself subject to blackmail by the distant superintelligences, thus breaking the decision theory that seemed to yield such positive behavior previously.

## Examples which occur purely due to added computing power.

• During development, the AI's epistemic models of people are not [6v detailed enough to be sapient]. Adding more computing power to the AI causes a massive amount of [6v mindcrime].

• During development, the AI's internal policies, hypotheses, or other Turing-complete subprocesses that are subject to internal optimization, are not optimized highly enough to give rise to [2rc new internal consequentialist cognitive agencies].  Adding much more computing power to the AI [2rc causes some of the internal elements to begin doing consequentialist, strategic reasoning] that leads them to try to 'steal' control of the AI.

# Implications

High probabilities of context change problems would seem to argue:

- Against a policy of relying on the observed good behavior of an improving AI to guarantee its later good behavior.
- In favor of [6r a methodology that attempts to foresee difficulties in advance], even before seeing undeniable observational evidence of those safety problems having already occurred.
- Against a methodology of [48 patching] disalignments that show up during the development phase, especially using penalty terms to the utility function.
- In favor of having a thought-logger that records all of an AI's thought proceses to indelible media, so as to indelibly log the first thought about faking outwardly nice behavior or [3cq hiding thoughts].
- In favor of the general difficulty of AI alignment, including consequences such as "[7wl]" or trying for [1vt narrow rather than ambitious value learning].

# Being wary of context disasters does not imply general skepticism

If an AI is smart, and especially if it's smarter than you, it can show you whatever it expects you want to see.  Computer scientists and physical scientists aren't accustomed to their experiments being aware of the experimenter and trying to deceive them.  (Some fields of psychology and economics, and of course computer security professionals, are more accustomed to operating in such a social context.)

[John Danaher](http://philosophicaldisquisitions.blogspot.com/2014/07/bostrom-on-superintelligence-3-doom-and.html) seems alarmed by this implication:

> Accepting this has some pretty profound epistemic costs. It seems to suggest that no amount of empirical evidence could ever rule out the possibility of a future AI taking a treacherous turn.

Yudkowsky [replies](http://philosophicaldisquisitions.blogspot.com/2014/07/bostrom-on-superintelligence-3-doom-and.html#comment-<PHONE>):

> If "empirical evidence" is in the form of observing the short-term consequences of the AI's outward behavior, then the answer is simply no. Suppose that on Wednesday someone is supposed to give you a billion dollars, in a transaction which would allow a con man to steal ten billion dollars from you instead. If you're worried this person might be a con man instead of an altruist, you cannot reassure yourself by, on Tuesday, repeatedly asking this person to give you five-dollar bills. An altruist would give you five-dollar bills, but so would a con man... [1lz Bayes] tells us to pay attention to [1rq likelihood ratios] rather than outward similarities. It doesn't matter if the outward behavior of handing you the five-dollar bill seems to bear a surface resemblance to altruism or money-givingness, the con man can strategically do the same thing; so the likelihood ratio here is in the vicinity of 1:1.

> You can't get strong evidence about the long-term good behavior of a strategically intelligent mind, by observing the short-term consequences of its current behavior. It can figure out what you're hoping to see, and show you that. This is true even among humans. You will simply have to get your evidence from somewhere else.

This doesn't mean we can't get evidence from, e.g., trying to [3cq monitor (and indelibbly log) the AI's thought processes] in a way that will detect (and record) the very first intention to hide the AI's thought processes before they can be hidden.  It does mean we can't get [22x strong evidence] about a strategic agent by observing short-term consequences of its outward behavior.

Donaher later [expanded his concern into a paper](http://dl.acm.org/citation.cfm?id=<PHONE>) drawing an analogy between worrying about deceptive AIs, and "skeptical theism" in which it's supposed that any amount of apparent evil in the world (smallpox, malaria) might secretly be the product of a benevolent God due to some nonobvious instrumental link between malaria and inscrutable but normative ultimate goals.  If it's okay to worry that an AI is just pretending to be nice, asks Donaher, why isn't it okay to believe that God is just pretending to be evil?

The obvious disanalogy is that the reasoning by which we expect a con man to cultivate a warm handshake is far more straightforward than a purported instrumental link from malaria to normativity.  If we're to be terrified of skepticism as generally as Donaher suggests, then we also ought to be terrified of being skeptical of business partners that have already shown us a warm handshake (which we shouldn't).

Rephrasing, we could draw two potential analogies to concern about Type-2 context changes:

- A potential business partner in whom you intend to invest \$10,000,000 has a warm handshake. Your friend warns you that con artists have a substantial prior probability and asks you to envision what you would do if you were a con artist , pointing out that the default extrapolation is for the con artist to match their outward behavior to what the con artist thinks you expect from a trustworthy partner, and in particular, cultivate a warm handshake.
  - Your friend suggests only doing business with one of those entrepreneurs who've been wearing a thought recorder for their whole life since birth, so that there would exist a clear trace of their very first thought about learning to fool thought recorders. Your friend says this to emphasize that he's not arguing for some kind of invincible epistemic pothole that nobody is ever allowed to climb out of.
- The world contains malaria and used to contain smallpox. Your friend asks you to consider that these diseases might be the work of a benevolent superintelligence, even though, if you'd never learned before whether or not the world contained smallpox, you wouldn't expect a priori and by default for a benevolent superintelligence to create it; and the arguments for a benevolent superintelligence creating smallpox seem [10m strained].

It seems hard to carry the argument that concern over a non-aligned AI pretending to benevolence, should be considered more analogous to the second scenario than to the first.

[todo: write about the defeat of the 'but AI people will have short-term incentives to produce correct behavior']

[todo: write about cognitive steganography in the 'programmer deception' page and reference it here.]

[todo: talk about whitelisting as directly tackling the type-1 form of this problem.]

[comment: - The AI is aware that its future operation will depart from the programmers' intended goals, does not process this as an error condition, and seems to behave nicely earlier in order to 10f deceive the programmers and prevent its real goals from being modified. - The AI is subject to a debugging methodology in which several bugs appear during its development phase, these bugs are corrected, and then additional bugs are exposed only during a more advanced phase.]