Ontology identification problem

[summary:  It seems likely that for advanced agents, the agent's representation of the world will [5d change in unforeseen ways as it becomes smarter].  The ontology identification problem is to create a [5f preference framework] for the agent that optimizes the same external facts, even as the agent modifies its representation of the world.  [6b For example], if the [6h intended goal] were to [5g create large amounts of diamond material], one type of ontology identification problem would arise if the programmers thought of carbon atoms as primitive during the AI's development phase, and then the advanced AI discovered nuclear physics.]

[toc:]

# Introduction: The ontology identification problem for unreflective diamond maximizers

A simplified but still very difficult open problem in [2v] is to state an unbounded program implementing a [5g diamond maximizer] that will turn as much of the physical universe into diamond as possible.  The goal of "making diamonds" was chosen to have a crisp-seeming definition for our universe (the amount of diamond is the number of carbon atoms covalently bound to four other carbon atoms).  If we can crisply define exactly what a 'diamond' is, we can avert issues of trying to convey [5l complex values] into the agent.  (The [5g unreflective diamond maximizer] putatively has [ unlimited computing power], runs on a [ Cartesian processor], and confronts no other agents [ similar to itself].  This averts many other problems of [71 reflectivity], [18s decision theory] and [2v value alignment].)

Even with a seemingly crisp goal of "make diamonds", we might still run into two problems if we tried to write a [5t hand-coded object-level utility function] that [ identified] the amount of diamond material:

- Unknown substrate: We might not know the true, fundamental ontology of our own universe, hence not know what stuff diamonds are really made of.  (What exactly is a carbon atom?  If you say it's a nucleus with six protons, what's a proton?  If you define a proton as being made of quarks, what if there are unknown other particles underlying quarks?)
 - It seems intuitively like there ought to be some way to identify carbon atoms to an AI in some way that doesn't depend on talking about quarks.  Doing this is part of the ontology identification problem.
- Unknown representation:  We might crisply know what diamonds are in our universe, but not know how to find diamonds inside the agent's model of the environment.
 - Again, it seems intuitively like it ought to be possible to identify diamonds in the environment, even if we don't know details of the agent's exact internal representation.  Doing this is part of the ontology identification problem.

To introduce the general issues in ontology identification, we'll try to walk through the [ anticipated difficulties] of constructing an unbounded agent that would maximize diamonds, by trying specific methods and suggesting [ anticipated difficulties] of those methods.

## Difficulty of making AIXI-tl maximize diamond

The classic unbounded agent - an agent using far more computing power than the size of its environment - is [11v].  Roughly speaking, AIXI considers all computable hypotheses for how its environment might be turning AIXI's motor outputs into AIXI's sensory inputs and rewards.  We can think of AIXI's hypothesis space as including all Turing machines that, sequentially given AIXI's motor choices as inputs, will output a sequence of predicted sense items and rewards for AIXI.  The finite variant AIXI-tl has a hypothesis space that includes all Turing machines that can be specified using fewer than $l$ bits and run in less than time $t$.

One way of seeing the difficulty of ontology identification is considering why it would be difficult to make an AIXI-tl variant that maximized 'diamonds' instead of 'reward inputs'.

The central difficulty here is that there's no way to find 'diamonds' inside the implicit representations of AIXI-tl's sequence-predicting Turing machines.  Given an arbitrary Turing machine that is successfully predicting AIXI-tl's sense inputs, there is no general rule for how to go from the representation of that Turing machine to a statement about diamonds or carbon atoms.  The highest-weighted Turing machines that have best predicted the sensory data so far, presumably contain *some* sort of representation of the environment, but we have no idea how to get 'the number of diamonds' out of it.

If AIXI has a webcam, then the final outputs of the Turing machine are predictions about the stream of bits produced by the webcam, going down the wire into AIXI.  We can understand the meaning of that Turing machine's output predictions; those outputs are meant to match types with the webcam's input.  But we have no notion of anything else that Turing machine is representing.  Even if somewhere in the Turing machine happens to be an atomically detailed model of the world, we don't know what representation it uses, or what format it has, or how to look inside it for the number of diamonds that will exist after AIXI's next motor action.

This difficulty ultimately arises from AIXI being constructed around a [ Cartesian] paradigm of [ sequence prediction], with AIXI's sense inputs and motor outputs being treated as sequence elements, and the Turing machines in its hypothesis space having inputs and outputs matched to the sequence elements and otherwise being treated as black boxes.  This means we can only get AIXI to maximize direct functions of its sensory input, not any facts about the outside environment.

(We can't make AIXI maximize diamonds by making it want *pictures* of diamonds because then it will just, e.g., [ build an environmental subagent that seizes control of AIXI's webcam and shows it pictures of diamonds].  If you ask AIXI to show itself sensory pictures of diamonds, you can get it to show its webcam lots of pictures of diamonds, but this is not the same thing as building an environmental diamond maximizer.)

## Agent using classical atomic hypotheses

As an [ unrealistic example]:  Suppose someone was trying to define 'diamonds' to the AI's utility function, and suppose they knew about atomic physics but not nuclear physics.  Suppose they build an AI which, during its development phase, learns about atomic physics from the programmers, and thus builds a world-model that is based on atomic physics.

Again for purposes of [ unrealistic examples], suppose that the AI's world-model is encoded in such fashion that when the AI imagines a molecular structure - represents a mental image of some molecules - carbon atoms are represented as a particular kind of basic element of the representation.  Again, as an [ unrealistic example], imagine that there are [ little LISP tokens] representing environmental objects, and that the environmental-object-type of carbon-objects is encoded by the integer 6.  Imagine also that each atom, inside this representation, is followed by a list of the other atoms to which it's covalently bound.  Then when the AI is imagining a carbon atom participating in a diamond, inside the representation we would see an object of type 6, followed by a list containing exactly four other 6-objects.

Can we fix this representation for all hypotheses, and then write a utility function for the AI that counts the number of type-6 objects that are bound to exactly four other type-6 objects?  And if we did so, would the result actually be a diamond maximizer?

### AIXI-atomic

We can imagine formulating a variant of AIXI-tl that, rather than all tl-bounded Turing machines, considers tl-bounded simulated atomic universes - that is, simulations of classical, pre-nuclear physics.  Call this AIXI-atomic.

A first difficulty is that universes composed only of classical atoms are not good explanations of our own universe, even in terms of surface phenomena; e.g., the [ultraviolet catastrophe](http://en.wikipedia.org/wiki/Ultraviolet_catastrophe).  So let it be supposed that we have simulation rules for classical physics that replicate at least whatever phenomena the programmers have observed at [ development time], even if the rules have some seemingly ad-hoc elements (like there being no ultraviolent catastrophes).

A second difficulty is that a simulated universe of classical atoms does not identify where in the universe the AIXI-atomic agent resides, or that AIXI-atomic's sense inputs don't have types commensurate with the types of atoms.  We can elide this difficulty by imagining that AIXI-atomic simulates classical universes containing a single hypercomputer, and that AIXI-atomic knows a simple function from each simulated universe onto its own sensory data (e.g., it knows to look at the simulated universe, and translate simulated photons impinging on its webcam onto predicted webcam data in the received format).  This elides most of the problem of [ naturalized induction], by fixing the ontology of all hypotheses and standardizing their hypothetical [ bridging laws].

So the analogous AIXI-atomic agent that maximizes diamond:

- Considers only hypotheses that directly represent universes as huge systems of classical atoms, so that the function 'count atoms bound to four other carbon atoms' can be directly run over any possible future the agent considers.
- Assigns probabilistic priors over these possible atomic representations of the universe.
- Somehow [ maps each atomic representation onto the agent's sensory experiences and motor actions].
- [Bayes-updates its priors] based on actual sensory experiences, the same as classical AIXI.
- Can evaluate the 'expected diamondness on the next turn' of a single action by looking at all hypothetical universes where that action is performed, weighted by their current probability, and summing over the expectation of diamond-bound carbon atoms on their next clock tick.
- Can evaluate the 'future expected diamondness' of an action, over some finite time horizon, by assuming that its future self will also Bayes-update and maximize expected diamondness over that time horizon.
- On each turn, outputs the action with greatest expected diamondness over some finite time horizon.

Suppose our own real universe was amended to otherwise be exactly the same, but contain a single [ impermeable] hypercomputer.  Suppose we defined an agent like the one above, using simulations of 1900-era classical models of physics, and ran that agent on the hypercomputer.  Should we expect the result to be an actual diamond maximizer - that most mass in the universe will be turned into carbon and arranged into diamonds?

### Anticipated failure of AIXI-atomic in our own universe: trying to maximize diamond outside the simulation

Our own universe isn't atomic, it's nuclear and quantum-mechanical.  This means that AIXI-atomic does not contain any hypotheses in its hypothesis space that *directly represent* the universe.  By 'directly represent', we mean that carbon atoms in AIXI-atomic's best representations do not correspond to carbon atoms in our own world).

Intuitively, we would think it was [ common sense] for an agent that wanted diamonds to react to the experimental data identifying nuclear physics, by deciding that a carbon atom is 'really' a nucleus containing six protons, and atomic binding is 'really' covalent electron-sharing.  We can imagine this agent [ common-sensically] updating its model of the universe to a nuclear model, and redefining the 'carbon atoms' that its old utility function counted to mean 'nuclei containing exactly six protons'.  Then the new utility function could evaluate outcomes in the newly discovered nuclear-physics universe.  We will call this the **utility rebinding problem**.

We don't yet have a crisp formula that seems like it would yield commonsense behavior for utility rebinding.  In fact we don't yet have any candidate formulas for utility rebinding, period. Stating one is an open problem.  See below.

For the 'classical atomic AIXI' agent we defined above, what happens instead is that the 'simplest atomic hypothesis that fits the facts' will be an enormous atom-based computer, simulating nuclear physics and quantum physics in order to control AIXI's webcam, which is still believed to be composed of atoms in accordance with the prespecified bridging laws.  From our perspective this hypothesis seems silly, but if you restrict the hypothesis space to only classical atomic universes, that's what ends up being the computationally simplest hypothesis to explain the results of quantum experiments.

AIXI-atomic will then try to choose actions so as to maximize the amount of expected diamond inside the probable *outside universes* that could contain the giant atom-based simulator of quantum physics.  It is not obvious what sort of behavior this would imply.

### Metaphor for difficulty: AIXI-atomic cares about only fundamental carbon

One metaphorical way of looking at the problem is that AIXI-atomic was implicitly defined to care only about diamonds made out of *ontologically fundamental* carbon atoms, not diamonds made out of quarks.  A probability function that assigns 0 probability to all universes made of quarks, and a utility function that outputs a constant on all universes made of quarks, [ yield functionally identical behavior].  So it is an exact metaphor to say that AIXI-atomic only *cares* about universes with ontologically basic carbon atoms, given that AIXI-atomic only *believes* in universes with ontologically basic carbon atoms.

Since AIXI-atomic only cares about diamond made of fundamental carbon, when AIXI-atomic discovered the experimental data implying that almost all of its probability mass should reside in nuclear or quantum universes in which there were no fundamental carbon atoms, AIXI-atomic stopped caring about the effect its actions had on the vast majority of probability mass inside its model.  Instead AIXI-atomic tried to maximize inside the tiny remaining probabilities in which it *was* inside a universe with fundamental carbon atoms that was somehow reproducing its sensory experience of nuclei and quantum fields; for example, a classical atomic universe with an atomic computer simulating a quantum universe and showing the results to AIXI-atomic.

From our perspective, we failed to solve the 'ontology identification problem' and get the real-world result we wanted, because we tried to define the agent's *utility function* in terms of properties of a universe made out of atoms, and the real universe turned out to be made of quantum fields.  This caused the utility function to *fail to bind* to the agent's representation in the way we intuitively had in mind.

### Advanced-nonsafety of hardcoded ontology identifications

Today we do know about quantum mechanics, so if we tried to build an unreflective diamond maximizer using the above formula, it might not fail on account of [48 the particular exact problem] of atomic physics being false.

But perhaps there are discoveries still remaining that would change our picture of the universe's ontology to imply something else underlying quarks or quantum fields.  Human beings have only known about quantum fields for less than a century; our model of the ontological basics of our universe has been stable for less than a hundred years of our human experience.  So we should seek an AI design that does not assume we know the exact, true, fundamental ontology of our universe during an AI's [5d development phase].  Or if our failure to know the exact laws of physics causes catastrophic failure of the AI, we should at least heavily mark that this is a [ relied-on assumption].

## Beyond AIXI-atomic: Diamond identification in multi-level maps

A realistic, bounded diamond maximizer wouldn't represent the outside universe with atomically detailed models.  Instead, it would have some equivalent of a [ multi-level map] of the world in which the agent knew in principle that things were composed of atoms, but didn't model most things in atomic detail.  E.g., its model of an airplane would have wings, or wing shapes, rather than atomically detailed wings.  It would think about wings when doing aerodynamic engineering, atoms when doing chemistry, nuclear physics when doing nuclear engineering.

At the present, there are not yet any proposed formalisms for how to do probability theory with multi-level maps (in other words: [ nobody has yet put forward a guess at how to solve the problem even given infinite computing power]).  Having some idea for how an agent could reason with multi-level maps, would be a good first step toward being able to define a bounded expected utility optimizer with a utility function that could be evaluated on multi-level maps.  This in turn would be a first step towards defining an agent with a utility function that could rebind itself to *changing* representations in an *updating* multi-level map.

If we were actually trying to build a diamond maximizer, we would be likely to encounter this problem long before it started formulating new physics.  The equivalent of a computational discovery that changes 'the most efficient way to represent diamonds' is likely to happen much earlier than a physical discovery that changes 'what underlying physical systems probably constitute a diamond'.

This also means that, on the actual [ value loading problem], we are liable to encounter the ontology identification problem long before the agent starts discovering new physics.

# Discussion of the generalized ontology identification problem

If we don't know how to solve the ontology identification problem for maximizing diamonds, we probably can't solve it for much more complicated values over universe-histories.

### View of human angst as ontology identification problem

Argument:  A human being who feels angst on contemplating a universe in which "By convention sweetness, by convention bitterness, by convention color, in reality only atoms and the void" (Democritus), or wonders where there is any room in this cold atomic universe for love, free will, or even the existence of people - since, after all, people are just *mere* collections of atoms - can be seen as undergoing an ontology identification problem: they don't know how to find the objects of value in a representation containing atoms instead of ontologically basic people.

Human beings simultaneously evolved a particular set of standard mental representations (e.g., a representation for colors in terms of a 3-dimensional subjective color space, a representation for other humans that simulates their brain via [empathy]) along with evolving desires that bind to these representations ([identification of flowering landscapes as beautiful](http://en.wikipedia.org/wiki/Evolutionary_aesthetics#Landscape_and_other_visual_arts_preferences), a preference not to be embarrassed in front of other objects designated as people).  When someone visualizes any particular configurations of 'mere atoms', their built-in desires don't automatically fire and bind to that mental representation, the way they would bind to the brain's native representation of other people.  Generalizing that no set of atoms can be meaningful, and being told that reality is composed entirely of such atoms, they feel they've been told that the true state of reality, underlying appearances, is a meaningless one.

Arguably, this is structurally similar to a utility function so defined as to bind only to true diamonds made of ontologically basic carbon, which evaluates as unimportant any diamond that turns out to be made of mere protons and neutrons.

## Ontology identification problems may reappear on the reflective level

An obvious thought (especially for [6w online genies]) is that if the AI is unsure about how to reinterpret its goals in light of a shifting mental representation, it should query the programmers.

Since the definition of a programmer would then itself be baked into the [5f preference framework], the problem might [ reproduce itself on the reflective level] if the AI became unsure of where to find [9r programmers].  ("My preference framework said that programmers were made of carbon atoms, but all I can find in this universe are quantum fields.")

## Value lading in category boundaries

Taking apart objects of value into smaller components can sometimes create new moral [ edge cases].  In this sense, rebinding the terms of a utility function decides a [ value-laden] question.

Consider chimpanzees.  One way of viewing questions like "Is a chimpanzee truly a person?" - meaning, not, "How do we arbitrarily define the syllables per-son?" but "Should we care a lot about chimpanzees?" - is that they're about how to apply the 'person' category in our desires to things that are neither typical people nor typical nonpeople.  We can see this as arising from something like an ontological shift: we're used to valuing cognitive systems that are made from whole human minds, but it turns out that minds are made of parts, and then we have the question of how to value things that are made from some of the person-parts but not all of them.

Redefining the value-laden category 'person' so that it talked about brains made out of neural regions, rather than whole human beings, would implicitly say whether or not a chimpanzee was a person.  Chimpanzees definitely have neural areas of various sizes, and particular cognitive abilities - we can suppose the empirical truth is unambiguous at this level, and known to us.  So the question is then whether we regard a particular configuration of neural parts (a frontal cortex of a certain size) and particular cognitive abilities (consequentialist means-end reasoning and empathy, but no recursive language) as something that our 'person' category values... once we've rewritten the person category to value configurations of cognitive parts, rather than whole atomic people.

In this sense the problem we face with chimpanzees is exactly analogous to the question a diamond maximizer would face after discovering nuclear physics and asking itself whether a carbon-14 atom counted as 'carbon' for purposes of caring about diamonds.  Once a diamond maximizer knows about neutrons, it can see that C-14 is chemically like carbon and forms the same kind of chemical bonds, but that it's heavier because it has two extra neutrons.  We can see that chimpanzees have a similar brain architectures to the sort of people we always considered before, but that they have smaller frontal cortexes and no ability to use recursive language, etcetera.

Without knowing more about the diamond maximizer, we can't guess what sort of considerations it might bring to bear in deciding what is Truly Carbon and Really A Diamond.  But the breadth of considerations human beings need to invoke in deciding how much to care about chimpanzees, is one way of illustrating that the problem of rebinding a utility function to a shifted ontology is [value-laden] and potentially undergo [ excursions] into [ arbitrarily complicated desiderata].  Redefining a [ moral category] so that it talks about the underlying parts of what were previously seen as all-or-nothing atomic objects, may carry an implicit ruling about how to value many kinds of [ edge case] objects that were never seen before.

 A formal part of this problem may need to be carved out from the edge-case-reclassification part: e.g., how would you redefine carbon as C12 if there were no other isotopes, or how would you rebind the utility function to *at least* C12, or how would edge cases be identified and queried.

# Potential research avenues

## 'Transparent priors' constrained to meaningful but Turing-complete hypothesis spaces

The reason why we can't bind a description of 'diamond' or 'carbon atoms' to the hypothesis space used by [11v] or [ AIXI-tl] is that the hypothesis space of AIXI is all Turing machines that produce binary strings, or probability distributions over the next sense bit given previous sense bits and motor input.  These Turing machines could contain an unimaginably wide range of possible contents

(Example:  Maybe one Turing machine that is producing good sequence predictions inside AIXI, actually does so by simulating a large universe, identifying a superintelligent civilization that evolves inside that universe, and motivating that civilization to try to intelligently predict future future bits from past bits (as provided by some intervention).  To write a formal utility function that could extract the 'amount of real diamond in the environment' from arbitrary predictors in the above case , we'd need the function to read the Turing machine, decode that universe, find the superintelligence, decode the superintelligence's thought processes, find the concept (if any) resembling 'diamond', and hope that the superintelligence had precalculated how much diamond was around in the outer universe being manipulated by AIXI.)

This suggests that to solve the ontology identification problem, we may need to constrain the hypothesis space to something [ less general] than 'an explanation is any computer program that outputs a probability distribution on sense bits'.  A constrained explanation space can still be Turing complete (contain a possible explanation for every computable sense input sequence) without every possible computer program constituting an explanation.

An [ unrealistic example] would be to constrain the hypothesis space to Dynamic Bayesian Networks.  DBNs can represent any Turing machine with bounded memory,[todo: Not sure where to look for a citation, but I'd be very surprised if this wasn't true.] so they are very general, but since a DBN is a causal model, they make it possible for a preference framework to talk about 'the cause of a picture of a diamond' in a way that you couldn't look for 'the cause of a picture of a diamond' inside a general Turing machine.  Again, this might fail if the DBN has no 'natural' way of representing the environment except as a DBN simulating some other program that simulates the environment.

Suppose a rich causal language, such as, e.g., a [ dynamic system] of objects with [ causal relations] and [ hierarchical categories of similarity].  The hope is that in this language, the *natural* hypothesis representing the environment - the simplest hypotheses within this language that well predict the sense data, or those hypotheses of highest probability under some simplicity prior after updating on the sense data - would be such that there was a natural 'diamond' category inside the most probable causal models.  In other words, the winning hypothesis for explaining the universe would already have postulated diamondness as a [ natural category] and represented it as Category #803,844, in a rich language where we already know how to look through the enviromental model and find the list of categories.

Given some transparent prior, there would then exist the further problem of developing a utility-identifying preference framework that could look through the most likely environmental representations and identify diamonds.  Some likely (interacting) ways of binding would be, e.g., to "the causes of pictures of diamonds", to "things that are bound to four similar things", querying ambiguities to programmers, or direct programmer inspection of the AI's model (but in this case the programmers might need to re-inspect after each ontological shift).  See below.

(A bounded value loading methodology would also need some way of turning the bound preference framework into the estimation procedures for expected diamond and the agent's search procedures for strategies high in expected diamond, i.e., the bulk of the actual AI that carries out the goal optimization.)

## Matching environmental categories to descriptive constraints

Given some transparent prior, there would exist a further problem of how to actually bind a preference framework to that prior.  One possible contributing method for pinpointing an environmental property could be if we understand the prior well enough to understand what the described object ought to look like - the equivalent of being able to search for 'things W made of six smaller things X near six smaller things Y and six smaller things Z, that are bound by shared Xs to four similar things W in a tetrahedral structure' in order to identify carbon atoms and diamond.

We would need to understand the representation well enough to make a guess about how carbon or diamond would be represented inside it.  But if we could guess that, we could write a program that identifies 'diamond' inside the hypothesis space without needing to know in advance that diamondness will be Category #823,034.  Then we could rerun the same utility-identification program when the representation updates, so long as this program can reliably identify diamond inside the model each time, and the agent acts so as to optimize the utility identified by the program.

One particular class of objects that might plausibly be identifiable in this way is 'the AI's programmers' (aka the agents that are causes of the AI's code) if there are parts of the preference framework that say to query programmers to resolve ambiguities.

A toy problem for this research avenue might involve:

- One of the richer representation frameworks that can be inducted as of the time, e.g., a simple Dynamic Bayes Net.
- An agent environment that can be thus represented.
- A goal over properties relatively distant from the agent's sensory experience (e.g., the goal is over the cause of the cause of the sensory data).
- A program that identifies the objects of utility in the environment, within the model thus freely inducted.
- An agent that optimizes the identified objects of utility, once it has inducted a sufficiently good model of the environment to optimize what it is looking for.

Further work might add:

- New information that can change the model of the environment.
- An agent that smoothly updates what it optimizes for in this case.

And further:

- Environments complicated enough that there is real structural ambiguity (e.g., dependence on exact initial conditions of the inference program) about how exactly the utility-related parts are modeled.
- Agents that can optimize through a probability distribution about environments that differ in their identified objects of utility.

A potential agenda for unbounded analysis might be:

- An [ unbounded analysis] showing that a utility-identifying [5f preference framework] is a generalization of a [ VNM utility] and can [ tile] in an architecture that tiles a generic utility function.
- A [45] analysis showing that an agent is not motivated to try to cause the universe to be such as to have utility identified in a particular way.
- A [45] analysis showing that the identity and category boundaries of the objects of utility will be treated as a [ historical fact] rather than one lying in the agent's [ decision-theoretic future].

## Identifying environmental categories as the causes of labeled sense data.

Another potential approach, given a prior transparent enough that we can find causal data inside it, would be to try to identify diamonds as the causes of pictures of diamonds.

[todo: expand]

### Security note

[5j Christiano's hack]: if your AI is advanced enough to model distant superintelligences, it's important to note that distant superintelligences can make 'the most probable cause of the AI's sensory data' be anything they want by making a predictable decision to simulate AIs such that your AI doesn't have info distinguishing itself from the distant AIs your AI imagines being simulated

## Ambiguity resolution

Both the description-matching and cause-inferring methods might produce ambiguities.  Rather than having the AI optimize for a probabilistic mix over all the matches (as if it were uncertain of which match were the true one), it would be better to query the ambiguity to the programmers (especially if different probable models imply different strategies).  This problem shares structure with [ inductive inference with ambiguity resolution] as a strategy for resolving [ unforeseen inductions].

[todo: if you try to solve the reflective problem by defining the queries in terms of sense data, you might run into Cartesian problems.  if you try to ontologically identify the programmers in terms more general than a particular webcam, so that the AI can have new webcams, the ontology identification problem might reproduce itself on the reflective level.  you have to note it down as a dependency either way.]

## Multi-level maps

Being able to describe, in purely theoretical principle, a prior over epistemic models that have at least two levels and can switch between them in some meaningful sense, would constitute major progress over the present state of the art.

[todo: try this with just two level.  half adders as potential models?  requirements: that the lower level be only partially realized rather than needing to be fully modeled; that it can describe probabilistic things; that we can have a language for things like this and prior over them that gets updated on the evidence, rather than just a particular handcrafted two-level map.]

# Implications

[todo:
if the programmers can read through updates to the AI's representation fast enough, or if most of the routine ones leave certain levels intact or imply a defined relation between old and new models, then it might be possible to solve this problem programmatically for genies.  especially if it's a nonrecursive genie with known algorithms, because then it might have a known representation that might be known not to change suddenly, and be corrigible-by-default while the representation is being worked out.
so this is one of the problems more likely to be averted in practice
but understanding it does help to see one more reason why You Cannot Just Hardcode the Utility Function By Hand.]

[todo: Hard to solve entire problem because it has at least some entanglement with the full AGI problem.]

The problem of using sensory data to build computationally efficient probabilistic maps of the world, and to efficiently search for actions that are predicted by those maps to have particular consequences, could be identified with the entire problem of AGI.  So the research goal of ontology identification is not to publish a complete bounded system like that (i.e. an AGI), but to develop an unbounded analysis of utility rebinding that seems to say something useful specifically about the ontology-identification part of the problem.)


Development phase unpredictable

Several proposed problems in [2l advanced safety] are alleged to be difficult because they depend on some property of a mature [2c agent] that is allegedly hard to predict in advance at the time we are designing, teaching, or testing the agent.  We say that such properties are allegedly 'development phase unpredictable'.  For example, the [47 Unforeseen Maximums problem] arises when [2j we can't search a rich solution space as widely as an advanced agent], making it development-phase unpredictable which real-world strategy or outcome state will maximize some formal utility function.


Preference framework

[summary:  A 'preference framework' is a way of deciding which outcomes an agent [1bh terminally] prefers.  'Preference framework' is a broader term than '[1fw utility function]', since 'preference framework' would also include structurally complicated [meta_utility meta-utility] functions, such as those which appear in some proposals for [1b7] or [7s2].]

A 'preference framework' refers to a fixed algorithm that updates, or potentially changes in other ways, to determine what the agent [ prefers] for [1bh terminal] outcomes.  'Preference framework' is a term more general than '[1fw utility function]' which includes structurally complicated generalizations of utility functions.

As a central example, the [1b7 utility indifference] proposal has the agent switching between utility functions $U_X$ and $U_Y$ depending on whether a switch is pressed.  We can call this meta-system a 'preference framework' to avoid presuming in advance that it embodies a [7hh VNM-coherent] utility function.

An even more general term would be [decision_algorithm] which doesn't presume that the agent operates by [9h preferring outcomes].


Diamond maximizer

A difficult, far-reaching open problem in [2v] is to specify an [ unbounded] formula for an agent that would, if run on an [ unphysically large finite computer], create as much diamond material as possible.  The goal of 'diamonds' was chosen to make it physically crisp as to what constitutes a 'diamond'.  Supposing a crisp goal plus hypercomputation avoids some problems in value alignment, while still invoking many others, making it an interesting intermediate problem.

### Importance

The diamond maximizer problem is to give an [ unbounded] description of a computer program such that, if it were instantiated on a sufficiently powerful but [ physical computer], the result of running the program would be the creation of an immense amount of diamond - around as much diamond as is physically possible for an agent to create.

The fact that this problem is still extremely hard shows that the value alignment problem is not just due to the [5l].  As a thought experiment, it helps to distinguish value-complexity-laden difficulties from those that arise even for simple goals.

It also helps to [ illustrate the difficulty of value alignment] by making the more clearly visible point that we can't even figure out how to create lots of diamond using unlimited computing power, never mind creating [55 value] using [ bounded computing power].

### Problems avoided

If we can crisply define exactly what a 'diamond' is, in theory it seems like we should be able to avoid issues of [2w Edge Instantiation], [47 Unforeseen Maximums], and trying to convey [5l complex values] into the agent.

The amount of diamond is defined as the number of carbon atoms that are covalently bonded, by electrons, to exactly four other carbon atoms.  A carbon atom is any nucleus containing six protons and any number of neutrons, bound by the strong force.  The utility of a universal history is the total amount of Minkowskian interval spent by all carbon atoms being bound to exactly four other carbon atoms.  More precise definitions of 'bound', or the amount of measure in a quantum system that is being bound, are left to the reader - any crisp definition will do, so long as we are confident that it has no [47 unforeseen maximum] at things we don't intuitively see as diamonds.

### Problems challenged

Since this diamond maximizer would hypothetically be implemented on a very large but physical computer, it would confront [ reflective stability], the [ anvil problem], and the problems of making [ subagents].

To the extent the diamond maximizer might need to worry about other agents in the environment that have a good ability to model, or that it may need to cooperate with other diamond maximizers, it must resolve [ Newcomblike problems] using some [ logical decision theory].  This would also require it to confront [ logical uncertainty] despite possessing immense amounts of computing power.

To the extent the diamond maximizer must work well in a rich real universe that might operate according to any number of possible physical laws, it faces a problem of [ naturalized induction] and [5c ontology identification].  See the article on [5c ontology identification] for the case that even for the goal of 'make diamonds', the problem of [ goal identification] remains difficult.

### Unreflective diamond maximizer

As a further-simplified but still unsolved problem, an **unreflective diamond maximizer** is a diamond maximizer implemented on a [ Cartesian hypercomputer] in a [ causal universe] that does not face any [ Newcomblike problems].  This further avoids problems of reflectivity and logical uncertainty.  In this case, it seems plausible that the primary difficulty remaining is *just* the [5c ontology identification problem].  Thus the open problem of describing an unreflective diamond maximizer is a central illustration for the difficulty of ontology identification.


Distant superintelligences can coerce the most probable environment of your AI

A distant superintelligence can change 'the most likely environment' for your AI by simulating many copies of AIs similar to your AI, such that your local AI doesn't know it's not one of those simulated AIs.  This means that, e.g., if there is any reference in your AI's [5f preference framework] to the [ causes] of [ sense data] - like, programmers being the cause of sensed keystrokes - then a distant superintelligence can try to hack that reference.  This would place us in an [ adversarial security context versus a superintelligence], and should be avoided if at all possible.

### Difficulty

Some proposals for AI preference frameworks involve references to the AI's *causal environment* and not just the AI's immediate *sense events*.  For example, a [ DWIM] preference framework would putatively have the AI identify 'programmers' in the environment, model those programmers, and care about what its model of the programmers 'really wanted the AI to do'.  In other words, the AI would care about the causes behind its immediate sense experiences.

This potentially opens our AIs to a remote root attack by a distant superintelligence.  A distant superintelligence has the power to simulate lots of copies of our AI, or lots of AIs such that our AI doesn't think it can introspectively distinguish itself from those AIs.  Then it can force the 'most likely' explanation of the AI's apparent sensory experiences to be that the AI is in such a simulation.  Then the superintelligence can change arbitrary features of the most likely facts about the environment.

This problem was observed in a security context by [3], and precedented by a less general suggestion from [http://www.sl4.org/archive/0708/16600.html Rolf Nelson].

"Probable environment hacking" depends on the local AI trying to model distant superintelligences.  The actual proximal harm is done by the local AI's *model of* distant superintelligences, rather than by the superintelligences themselves.  However, a distant superintelligence that uses a [ logical decision theory] may model its choices as logically correlated to the local AI's model of the distant SI's choices.  Thus, a local AI that models a distant superintelligence that uses a logical decision theory may model that distant superintelligence as behaving as though it could control the AI's model of its choices via its choices.  Thus, the local AI would model the distant superintelligence as probably creating lots of AIs that it can't distinguish from itself, and update accordingly on the most probable cause of its sense events.

This hack would be worthwhile, from the perspective of a distant superintelligence, if e.g. it could gain control of the whole future light cone of 'naturally arising' AIs like ours, in exchange for expending some much smaller amount of resource (small compared to our future light cone) in order to simulate lots of AIs.  (Obviously, the distant SI would prefer even more to 'fool' our AI into expecting this, while not actually expending the resources.)

This hack would be expected to go through by default if: (1) a local AI uses [ naturalized induction] or some similar framework to reason about the [ causes] of sense events, (2) the local AI models distant superintelligences as being likely to use logical decision theories and to have utility functions that would vary with respect to outcomes in our local future light cone, and (3) the local AI has a preference framework that can be 'hacked' via induced beliefs about the environment.

### Implications

For any AI short of a full-scale autonomous Sovereign, we should probably try to get our AI to [1g4 not think at all about distant superintelligences], since this creates a host of [ adversarial security problems] of which "probable environment hacking" is only one.

We might also think twice about DWIM architectures that seem to permit catastrophe purely as a function of the AI's beliefs about the environment, without any check that goes through a direct sense event of the AI (which distant superintelligences cannot control the AI's beliefs about, since we can directly hit the sense switch).

We can also hope for any number of miscellaneous safeguards that would sound alarms at the point where the AI begins to imagine distant superintelligences imagining how to hack itself.


Complexity of value

[summary:  The proposition that there's no [5v algorithmically simple] [5t object-level goal] we can give to an [2c advanced AI] that yields a future of high [55 value].  Or: Any formally simple goal given to an AI, that talks directly about what sort of world to create, will produce disaster.  Or: If you're trying to talk directly about what events or states of the world you want, then any sort of programmatically simple utility function, of the sort a programmer could reasonably hardcode, will lead to a bad end.  (The non-simple alternative would be, e.g., an induction rule that can learn complicated classification rules from labeled instances, or a preference framework that explicitly models humans in order to learn complicated facts about what humans want.)]

## Introduction

"Complexity of value" is the idea that if you tried to write an AI that would do right things (or maximally right things, or adequately right things) *without further looking at humans* (so it can't take in a flood of additional data from human advice, the AI has to be complete as it stands once you're finished creating it), the AI's preferences or utility function would need to contain a large amount of data ([Kcomplexity algorithmic complexity]).  Conversely, if you try to write an AI that directly wants *simple* things or try to specify the AI's preferences using a *small* amount of data or code, it won't do acceptably right things in our universe.

Complexity of value says, "There's no simple and non-meta solution to AI preferences" or "The things we want AIs to want are complicated in the [5v Kolmogorov-complexity] sense" or "Any simple goal you try to describe that is All We Need To Program Into AIs is almost certainly wrong."

Complexity of value is a further idea above and beyond the [1y orthogonality thesis] which states that AIs don't automatically do the right thing and that we can have, e.g., [10h paperclip maximizers].  Even if we accept that paperclip maximizers are possible, and simple and nonforced, this wouldn't yet imply that it's very *difficult* to make AIs that do the right thing.  If the right thing is very simple to encode - if there are [55 value] optimizers that are scarcely more complex than [5g diamond maximizers] - then it might not be especially hard to build a nice AI even if not all AIs are nice.  Complexity of Value is the further proposition that says, no, this is forseeably quite hard - not because AIs have 'natural' anti-nice desires, but because niceness requires a lot of work to specify.

### Frankena's list

As an intuition pump for the complexity of value thesis, consider William Frankena's list of things which many cultures and people seem to value (for their own sake rather than their external consequences):

> "Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."

When we try to list out properties of a human or galactic future that seem like they'd be very nice, we at least *seem* to value a fair number of things that aren't reducible to each other.  (What initially look like plausible-sounding "But you do A to get B" arguments usually fall apart when we look for [ third alternatives] to doing A to get B.  Marginally adding some freedom can marginally increase the happiness of a human, so a happiness optimizer that can only exert a small push toward freedom might choose to do so.  That doesn't mean that a *pure, powerful* happiness maximizer would instrumentally optimize freedom.  If an agent cares about happiness but not freedom, the outcome that *maximizes* their preferences is a large number of brains set to maximum happiness.  When we don't just seize on one possible case where a B-optimizer might use A as a strategy, but instead look for further C-strategies that might maximize B even better than A, then the attempt to reduce A to an instrumental B-maximization strategy often falls apart.  It's in this sense that the items on Frankena's list don't seem to reduce to each other as a matter of pure preference, even though humans in everyday life often seem to pursue several of the goals at the same time.

Complexity of value says that, in this case, the way things seem is the way they are: Frankena's list is *not* encodable in one page of Python code.  This proposition can't be established definitely without settling on a sufficiently well-specified [ metaethics], such as [ reflective equilibrium], to make it clear that there is indeed no a priori reason for normativity to be algorithmically simple.  But the basic intuition for Complexity of Value is provided just by the fact that Frankena's list was more than one item long, and that many individual terms don't seem likely to have algorithmically simple definitions that distinguish their valuable from non-valuable forms.

### Lack of a central core

We can understand the idea of complexity of value by contrasting it to the situation with respect to [ epistemic reasoning] aka truth-finding or answering simple factual questions about the world.  In an ideal sense, we can try to compress and reduce the idea of mapping the world well down to algorithmically simple notions like "Occam's Razor" and "Bayesian updating".  In a practical sense, natural selection, in the course of optimizing humans to solve factual questions like "Where can I find a tree with fruit?" or "Are brightly colored snakes usually poisonous?" or "Who's plotting against me?", ended up with enough of the central core of epistemology that humans were later able to answer questions like "How are the planets moving?" or "What happens if I fire this rocket?", even though humans hadn't been explicitly selected on to answer those exact questions.

Because epistemology does have a central core of simplicity and Bayesian updating, selecting for an organism that got some pretty complicated epistemic questions right enough to reproduce, also caused that organism to start understanding things like General Relativity.  When it comes to truth-finding, we'd expect by default for the same thing to be true about an Artificial Intelligence; if you build it to get epistemically correct answers on lots of widely different problems, it will contain a core of truthfinding and start getting epistemically correct answers on lots of other problems - even problems completely different from your training set, the way that humans understanding General Relativity wasn't like any hunter-gatherer problem.

The complexity of value thesis is that there *isn't* a simple core to normativity, which means that if you hone your AI to do normatively good things on A, B, and C and then confront the AI with very different problem D, the AI may do the wrong thing on D.  There's a large number of independent ideal "gears" inside the complex machinery of value, compared to epistemology that in principle might only contain "prefer simpler hypotheses" and "prefer hypotheses that match the evidence".

The [1y Orthogonality Thesis] says that, contra to the intuition that [10h maximizing paperclips] feels "stupid", you can have arbitrarily cognitively powerful entities that maximize paperclips, or arbitrarily complicated other goals.  So while intuitively you might think it would be simple to avoid paperclip maximizers, requiring no work at all for a sufficiently advanced AI, the Orthogonality Thesis says that things will be more difficult than that; you have to put in some work to have the AI do the right thing.

The Complexity of Value thesis is the next step after Orthogonality; it says that, contra to the feeling that "rightness ought to be simple, darn it", normativity turns out not to have an algorithmically simple core, not the way that correctly answering questions of fact has a central tendency that generalizes well.  And so, even though an AI that you train to do well on problems like steering cars or figuring out General Relativity from scratch, may hit on a core capability that leads the AI to do well on arbitrarily more complicated problems of galactic scale, we can't rely on getting an equally generous bonanza of generalization from an AI that seems to do well on a small but varied set of moral and ethical problems - it may still fail the next problem that isn't like anything in the training set.  To the extent that we have very strong reasons to have prior confidence in Complexity of Value, in fact, we ought to be suspicious and worried about an AI that seems to be pulling correct moral answers from nowhere - it is much more likely to have hit upon the convergent instrumental strategy "say what makes the programmers trust you", rather than having hit upon a simple core of all normativity.

## Key sub-propositions

Complexity of Value requires [1y Orthogonality], and would be implied by three further subpropositions:

The **intrinsic complexity of value** proposition is that the properties we want AIs to achieve - whatever stands in for the metasyntactic variable '[55 value]' - have a large amount of intrinsic information in the sense of comprising a large number of independent facts that aren't being generated by a single computationally simple rule.

A very bad example that may nonetheless provide an important intuition is to imagine trying to pinpoint to an AI what constitutes 'worthwhile happiness'.  The AI suggests a universe tiled with tiny Q-learning algorithms receiving high rewards.  After some explanation and several labeled datasets later, the AI suggests a human brain with a wire stuck into its pleasure center.  After further explanation, the AI suggests a human in a holodeck.  You begin talking about the importance of believing truly and that your values call for apparent human relationships to be real relationships rather than being hallucinated.  The AI asks you what constitutes a good human relationship to be happy about.  The series of questions occurs because (arguendo) the AI keeps running into questions whose answers are not AI-obvious from the previous answers already given, because they involve new things you want such that your desire of them wasn't obvious from answers you'd already given.  The upshot is that the specification of 'worthwhile happiness' involves a long series of facts that aren't reducible just to the previous facts, and some of your preferences may involve many fine details of surprising importance.  In other words, the specification of 'worthwhile happiness' would be at least as hard to code by hand into the AI as it would be difficult to hand-code a formal rule that could recognize which pictures contained cats.  (I.e., impossible.)

The second proposition is **incompressibility of value** which says that attempts to reduce these complex values into some incredibly simple and elegant principle fail (much like early attempts by e.g. Bentham to reduce all human value to pleasure); and that no simple instruction given an AI will happen to target outcomes of high value either.  The core reason to expect a priori that all such attempts will fail, is that most 1000-byte strings aren't compressible down to some incredibly simple pattern no matter how many clever tricks you try to throw at them; fewer than 1 in 1024 such strings can be compressible to 990 bytes, never mind 10 bytes.  Due to the tremendous number of different proposals for why some simple instruction to an AI should end up achieving high-value outcomes or why all human value can be reduced to some simple principle, there is no central demonstration that all these proposals *must* fail, but there is a sense in which *a priori* we should strongly expect all such clever attempts to fail.  Many disagreeable attempts at reducing value A to value B, such as [ Juergen Schmidhuber's attempt to reduce all human value to increasing the compression of sensory information], stand as a further cautionary lesson.

The third proposition is **[fragility of value](fragility-1)** which says that if you have a 1000-byte *exact* specification of worthwhile happiness, and you begin to mutate it, the [55 value] created by the corresponding AI with the mutated definition falls off rapidly.  E.g. an AI with only 950 bytes of the full definition may end up creating 0% of the value rather than 95% of the value.  (E.g., the AI understood all aspects of what makes for a life well-lived... *except* the part about requiring a conscious observer to experience it.)

Together, these propositions would imply that to achieve an *adequate* amount of value (e.g. 90% of potential value, or even 20% of potential value) there may be no simple handcoded object-level goal for the AI that results in that value's realization.  E.g., you can't just tell it to 'maximize happiness', with some hand-coded rule for identifying happiness.

## Centrality

Complexity of Value is a central proposition in [2v value alignment theory].  Many [6r foreseen difficulties] revolve around it:

- Complex values can't be hand-coded into an AI, and require [ value learning] or [ Do What I Mean] preference frameworks.
- Complex /fragile values may be hard to learn even by induction because the labeled data may not include distinctions that give all of the 1000 bytes a chance to cast an unambiguous causal shadow into the data, and it's very bad if 50 bytes are left ambiguous.
- Complex / fragile values require error-recovery mechanisms because of the worry about getting some single subtle part wrong and this being catastrophic.  (And since we're working inside of highly intelligent agents, the recovery mechanism has to be a [45 corrigible preference] so that the agent accepts our attempts at modifying it.)

More generally:

- Complex values tend to be implicated in [48 patch-resistant problems] that wouldn't be resistant if there was some obvious 5-line specification of *exactly* what to do, or not do.
- Complex values tend to be implicated in the [6q context change problems] that wouldn't exist if we had a 5-line specification that solved those problems once and for all and that we'd likely run across during the development phase.

### Importance

Many policy questions strongly depend on Complexity of Value, mostly having to do with the overall difficulty of developing value-aligned AI, e.g.:

- Should we try to develop [ Sovereigns], or restrict ourselves to [6w Genies]?
- How likely is a moderately safety-aware project to succeed?
- Should we be more worried about malicious actors creating AI, or about well-intentioned errors?
- How difficult is the total problem and how much should we be panicking?
- How attractive would be any genuinely credible [2z game-changing alternative] to AI?

It has been advocated that there are [ psychological biases] and [ popular mistakes] leading to beliefs that directly or by implication deny Complex Value.  To the extent one credits that Complex Value is probably true, one should arguably be concerned about the number of early assessments of the value alignment problem that seem to rely on Complex Value being false (like just needing to hardcode a particular goal into the AI, or in general treating the value alignment problem as not panic-worthily difficult).

## Truth condition

The Complexity of Value proposition is true if, relative to viable and acceptable real-world [ methodologies] for AI development, there isn't any reliably knowable way to specify the AI's [ object-level preferences] as a structure of low [ algorithmic complexity], such that the result of running that AI is [2z achieving] [ enough] of the possible [55 value], for reasonable definitions of [55 value].

Caveats:

### Viable and acceptable computation

Suppose there turns out to exist, in principle, a relatively simple Turing machine (e.g. 100 states) that picks out 'value' by re-running entire evolutionary histories, creating and discarding a hundred billion sapient races in order to pick out one that ended up relatively similar to humanity.  This would use an unrealistically large amount of computing power and *also* commit an unacceptable amount of [6v mindcrime].


Immediate goods

One of the potential views on 'value' in the value alignment problem is that what we should want from an AI is a list of immediate goods or outcome features like 'a cure for cancer' or 'letting humans make their own decisions' or 'preventing the world from being wiped out by  a paperclip maximizer'.  (Immediate Goods as a criterion of 'value' isn't the same as saying we should give the AI those explicit goals; calling such a list 'value' means it's the real criterion by which we should judge how well the AI did.)

# Arguments

## Immaturity of view deduced from presence of instrumental goods

It seems understandable that Immediate Goods would be a very common form of expressed want when people first consider the [2v value alignment problem]; they would look for valuable things an AI could do.

But such a quickly produced list of expressed wants will often include [ instrumental goods] rather than [ terminal goods].  For example, a cancer cure is (presumably) a means to the end of healthier or happier humans, which would then be the actual grounds on which the AI's real-world 'value' was evaluated from the human speaker's standpoint.  If the AI 'cured cancer' in some technical sense that didn't make people healthier, the original person making the wish would probably not see the AI as having achieved value.

This is a reason for suspecting the maturity of such expressed views, and to suspect that the stated list of immediate goods will probably evolve into a more [ terminal] view of value from a human standpoint, given further reflection.

### Mootness of immaturity

Irrespective of the above, so far as technical issues like [2w Edge Instantiation] are concerned, the 'value' variable could still apply to someone's spontaneously produced list of immediate wants, and that all the standard consequences of the value alignment problem usually still apply.  It means we can immediately say (honestly) that e.g. [2w Edge Instantiation] would be a problem for whatever want the speaker just expressed, without needing to persuade them to some other stance on 'value' first.  Since the same technical problems will apply both to the immature view and to the expected mature view, we don't need to dispute the view of 'value' in order to take it at face value and honestly explain the standard technical issues that would still apply.

## Moral imposition of short horizons

Arguably, a list of immediate goods may make some sense as a stopping-place for evaluating the performance of the AI, if either of the following conditions obtain:

- There is much more agreement (among project sponsors or humans generally) about the goodness of the instrumental goods, than there is about the terminal values that make them good.  E.g., twenty project sponsors can all agree that freedom is good, but have nonoverlapping concepts about why it is good, and it is hypothetically the case that these people would continue to disagree in the limit of indefinite debate or reflection.  Then if we want to collectivize 'value' from the standpoint of the project sponsors for purposes of talking about whether the AI methodology achieves 'value', maybe it would just make sense to talk about how much (intuitively evaluated) freedom the AI creates.
- It is in some sense morally incumbent upon humanity to do its own thinking about long-term outcomes and achieve them through immediate goods, or it is in some sense morally incumbent for humanity to arrive at long-term outcomes via its own decisions or optimization starting from immediate goods.  In this case, it might make sense to see the 'value' of the AI as being realized only in terms of the AI getting to those immediate goods, because it would be morally wrong for there to be optimization by the AI of consequences beyond that.

To the knowledge of [2] as of May 2015, neither of these views have yet been advocated by anyone in particular as a defense of an immediate-goods theory of value.


Value alignment problem

Disambiguation:  For the research subject that includes the entire edifice of how and why to produce good AIs, see [2v].

[summary: The 'value alignment problem' is to produce [7g1 sufficiently advanced machine intelligences] that *want* to do [3d9 beneficial] things and not do harmful things.  The largest-looming subproblem is [6c 'value identification' or 'value learning'] (sometimes considered synonymous with value alignment) but this also includes subproblems like [45], that is, AI values such that it doesn't *want* to interfere with you correcting what you see as an error in its code.]

The 'value alignment problem' is to produce [7g1 sufficiently advanced machine intelligences] that *want* to do [3d9 beneficial] things and not do harmful things.  The largest-looming subproblem is [6c 'value identification' or 'value learning'] (sometimes considered synonymous with value alignment) but this also includes subproblems like [45], that is, AI values such that it doesn't *want* to interfere with you correcting what you see as an error in its code.


Object-level vs. indirect goals

An 'object-level goal' is a goal that involves no indirection, doesn't require any further computation or observation to be fully specified, and is evaluated directly on events or things in the agent's model of the universe.  Contrast to meta-level or indirect goals.

Some examples of object-level goals might be these:

- Eat apples.
- Win a chess game.
- Cause a pawn to advance to the eighth row, in order to win a chess game.
- [10h Create paperclips.]
- [10d Maximize the amount of human 'happiness' for some fixed definition of happiness.]
- [5g The utility of a possible history of the universe is the amount of interval spent by carbon atoms covalently bound to four other carbon atoms.]

Here are some example cases of a goal or [5f preference framework] with properties that make them *not* object level:

- "Observe Alice, model what she wants, then do that."
  - This framework is *indirect* because it doesn't directly say what events or things in the universe are good or bad, but rather gives a recipe for deciding which events are good or bad (namely, model Alice).
  - This framework is *not fully locally specified*, because we have to observe Alice, and maybe compute further based on our observations of her, before we find out the actual evaluator we run to weigh events as good or bad.
- "Induce a compact category covering the proximal causes of sensory data labeled as positive instances and not covering sensory data labeled as negative instances.  The utility of an outcome is the number of events classified as positive by the induced category."
  - This framework is not fully specified because it is based on a supervised dataset, and the actual goals will vary with the dataset obtained.  We don't know what the agent wants when we're told about the induction algorithm; we also have to be told about the dataset, and we haven't been told about that yet.
- "Compute an average of what all superintelligences will want, relative to some prior distribution over the origins of superintelligences, and then do that."
  - This framework might be computable locally without any more observations, but it still involves a level of indirection, and is not locally complete in the sense that it would take a whole lot more computation before the agent knew what it ought to think of eating an apple.

The object-level vs. meta-level distinction should not be confused with the [1bh terminal vs. instrumental] distinction.


Opinion page

Opinion pages represent one view or opinion on a topic, and are not necessarily balanced or a reflection of consensus.