Mathematics

Mathematics is the study of crisply specified formal objects — for example, [number numbers] — and the ways of knowing their [mathematical_properties properties] — such as [-proofs].  We can see "[-258]" as the study of "which conclusions follow with certainty from which premises".  Using this definition of logic, we can also see mathematics as the study of logical objects in logical universes — entities whose properties follow from specifications about them, rather than from observation of the real world.  The number 3 is a logical object because its behavior follows from axioms about addition and multiplication; Mount Everest is a physical object because we learn about it by physically measuring Mount Everest.


Hypercomputer

[summary:  A "hypercomputer" is an imaginary artifact required to answer some crisp question that can't be answered in the limit of arbitrarily large finite computers.  For example, if you have a question that depends on a general solution to the [Halting Problem](https://en.wikipedia.org/wiki/Halting_problem), we say that to solve this problem requires a "hypercomputer".  (In particular, it requires a level-1 halting oracle.)

It seems exceptionally unlikely that hypercomputers will ever be discovered to be embedded into our physical universe.  We just use this as a label so we can say, for certain impossible programs, "Supposing we had a hypercomputer and could run this impossible program, what would be the consequences?"

For an example of interesting code that requires a hypercomputer, see [11w].  The relations between different levels of hypercomputer are also useful for crisply describing agents that have better or worse abilities to predict one another.]

A "hypercomputer" is an imaginary artifact required to answer some crisp question that can't be answered in the limit of arbitrarily large finite computers.  For example, if you have a question that depends on a general solution to the [Halting Problem](https://en.wikipedia.org/wiki/Halting_problem), then we say that to solve this problem requires a "hypercomputer", and in particular, a level-1 halting oracle.  (If you need to determine whether programs on level-1 halting oracles halt, you need a level-2 halting oracle, which we would also call a "hypercomputer".)

It seems exceptionally unlikely that hypercomputers will ever be discovered to be embedded into our physical universe.  The term "hypercomputer" just exists as a label so we can say, "Supposing we had a hypercomputer and ran this (impossible) program, what would be the consequences?"

For some examples of conceptually illuminating code that would require a hypercomputer to actually run, see [11w] and [11v].

[107 Unbounded analysis] of agents sometimes invokes hypercomputers because this lets us talk about multiple agents with easy-to-describe knowledge relations to each other.  [has-requisite(arithmetical_hierarchy): For example, we might talk about Agent X that uses Zermelo-Fraenkel set theory as a proof system and has a $\Pi_n$ oracle, and another Agent Y that uses Peano Arithmetic and has a $\Pi_{n+1}$ oracle, to encode a set of relations where Y can directly predict and model X, and X can do proofs about Y.] [!has-requisite(arithmetical_hierarchy): For example, we might talk about Agent X that uses a weak hypercomputer and a strong proof system, and Agent Y that has a strong hypercomputer and a weak proof system, to describe a scenario where Y can directly predict and model X, and X can do proofs about Y.]  In these cases, we're not trying to say that the relation between agents X and Y intrinsically requires them to have impossible powers of computation.  We're just reaching for an unphysical scenario that happens to crisply encode inter-agent relations we find interesting for some reason, and allows these inter-agent relations to have consequences about which we can easily do proofs.

See also [the Wikipedia page on hypercomputation](https://en.wikipedia.org/wiki/Hypercomputation).


AIXI-tl

$	ext{AIXI}^{tl}$ is a version of the ideal agent [11v] which only considers hypotheses of length $l$ that run for less than time $t$.  A $tl$-bounded version of [11v] therefore only requires an [1mm unphysically large finite computer] rather than an infinite [1mk hypercomputer].


Unphysically large finite computer

[summary:  An unphysically large finite computer is one that's vastly larger than anything that could possibly fit into our universe.  In a practical sense, computations that would require sufficiently large finite amounts of computation are pragmatically equivalent to computations that require [1mk hypercomputers], and serve a similar purpose in unbounded analysis: they let us talk about interesting things and crisply encode relations that might take a lot of unnecessary overhead to describe using *small* finite computers.  Nonetheless, since there are some mathematical pitfalls of considering infinite cases, reducing a problem to one that only requires a vast finite computer can sometimes be an improvement.

An example of an interesting computation requiring a vast finite computer is [1ml].]

An unphysically large finite computer is one that's vastly larger than anything that could possibly fit into our universe, if the *character* of physical law is anything remotely like it seems to be.

We might be able to get a googol ($10^{100}$) computations out of this universe by being clever, but to get $10^{10^{100}}$ computations would require outrunning proton decay and the second law of thermodynamics, and $9 \uparrow\uparrow 4$ operations ($9^{9^{9^9}}$) would require amounts of computing substrate in contiguous internal communication that wouldn't fit inside a single [Hubble Volume](https://en.wikipedia.org/wiki/Hubble_volume).  Even tricks that permit the creation of new universes and encoding computations into them probably wouldn't allow a single computation of size $9 \uparrow\uparrow 4$ to return an answer, if the character of physical law is anything like what it appears to be.

Thus, in a practical sense, computations that would require sufficiently large finite amounts of computation are pragmatically equivalent to computations that require [1mk hypercomputers], and serve a similar purpose in unbounded analysis - they let us talk about interesting things and crisply encode relations that might take a lot of unnecessary overhead to describe using *small* finite computers.  Nonetheless, since there are some mathematical pitfalls of considering infinite cases, reducing a problem to one guaranteed to only require a vast finite computer can sometimes be an improvement or yield new insights - especially when dealing with interesting recursions.

An example of an interesting computation requiring a vast finite computer is [1ml], or [131]'s [parametric bounded analogue of Lob's Theorem](http://intelligence.org/files/ParametricBoundedLobsTheorem.pdf).


Tiling agents theory

The theory of self-modifying agents that build successors that are very similar to themselves, like repeating tiles on a tesselated plane.  See [this paper](http://intelligence.org/files/TilingAgentsDraft.pdf) or [this Google search](http://www.google.com/search?q="tiling agents").


Cartesian agent-environment boundary

A Cartesian agent setup is one where the agent receives sensory information from the environment, and the agent sends motor outputs to the environment, and nothing else can cross the "Cartesian border" separating the agent and environment.  If you can eat a psychedelic mushroom that affects the way you process the world - not just presenting you with sensory information, but altering the computations you do to think - then this is an example of an event that "violates the Cartesian boundary".  Likewise if the agent drops an anvil on its own head.  Nothing that happens in a Cartesian universe can kill a Cartesian agent or modify its processing; all the universe can do is send the agent sensory information, in a particular format, that the agent reads.


Cartesian agent

[summary:  A Cartesian agent is an agent that's a separate system from the environment, linked by the [1mt Cartesian boundary] across which passes sensory information and motor outputs.  This is most commonly formalized by two distinct Turing machines, an 'agent' machine and an 'environment' machine.  The agent receives sensory information from the environment and outputs motor information; the environment receives the agent's motor information and computes the agent's sensory information.

An actual human, in contrast, is a "naturalistic agent" that is a continuous part of the universe - a human is one particular collection of atoms within the physical universe, and there's no type distinction between the atoms inside the human and the atoms outside the human.  Eating a particular kind of mushroom can make us think differently; dropping an anvil on your own head doesn't just cause you to see anvilness or receive a pain signal, it smashes your computing substrate and causes you not to feel any future sensory information at all.]

A Cartesian agent is an agent that's a separate system from the environment, linked by the [1mt Cartesian boundary] across which passes sensory information and motor outputs.  This is most commonly formalized by two distinct Turing machines, an 'agent' machine and an 'environment' machine.  The agent receives sensory information from the environment and outputs motor information; the environment receives the agent's motor information and computes the agent's sensory information.

An actual human, in contrast, is a "naturalistic agent" that is a continuous part of the universe - a human is one particular collection of atoms within the physical universe, and there's no type distinction between the atoms inside the human and the atoms outside the human.  Eating a particular kind of mushroom can make us think differently; dropping an anvil on your own head doesn't just cause you to see anvilness or receive a pain signal, it smashes your computing substrate and causes you not to feel any future sensory information at all.

In the context of [2v AI alignment theory], Cartesian agents are usually associated with optimizing for [cartesian_reward sensory rewards] - there's some particular component of the environment's input which is a "reward signal", or the agent is trying to optimize some directly computed function of sensory data.


Bayesian reasoning

[summary:  Bayesian reasoning is a way of interpreting the world, and our beliefs about the world, in the light of [1bv probability theory], in particular Bayes's Theorem or [1lz Bayes's Rule].  Applications include scientific statistics, machine learning, and everyday life.]

According to its advocates, Bayesian reasoning is a way of seeing the world, and our beliefs about the world, in the light of [1bv probability theory], in particular Bayes's Theorem or [1lz Bayes's Rule].  This probability-theoretic way of seeing the world can apply to scientific issues, to tasks in machine learning, and to everyday life.

To [1zq start learning], visit [1zq Arbital's Guide to Bayes's Rule].

After that, consider visiting the [1ly Bayesian update] page.


Orthogonality Thesis

[summary(Brief):  The "Orthogonality Thesis" asserts that, within [2c advanced AIs], it's possible to have any level of capability coexisting with any kind of goal.  So, for example, there can be superintelligences building shells around stars, who [10h only want to create as many paperclips as possible].]

[summary:  The weak form of the Orthogonality Thesis asserts that it's possible to have AI designs such that those AIs pursue almost any goal - for example, you can have an AI that [10h just wants paperclips].  The strong form of the Orthogonality Thesis asserts that an expected paperclip maximizer can be just as smart, efficient, reflective, and stable as any other kind of agent.

Key implications of the strong version of the Orthogonality Thesis:

- It is possible to build AIs with utility functions that direct them to be nice.
- But it is not any easier or simpler to build AIs that are nice.
- If you screw up the niceness part and build a non-nice agent, it won't automatically look itself over and say, "Oh, I guess I should be nice instead."]

[summary(Technical):  The Orthogonality Thesis says that the possibility space for cognitive agents include cognitive agents with every kind of goal or preference that is computationally tractable to evaluate.  Since the [3t3 purely factual questions] "Roughly how many expected [10h paperclips] will result from carrying out this policy?" and "What are policies that would probably result in a large number of paperclips existing?" seem to pose no special difficulty to compute, there should exist corresponding agents that output actions they expect to lead to large numbers of paperclips.

That is:  Imagine [5b2 Omega] offering to pay some extremely smart being one galaxy per paperclip it creates; they would have no special difficulty figuring out how to make lots of paperclips, if the payment provided a reason.  The Orthogonality Thesis says that, if a smart agent *could* figure out how to make lots of paperclips *if* paid, the [10h corresponding agent that makes paperclips because its utility function is over paperclips] can do so just as efficiently and intelligently.

The strong version of the Orthogonality Thesis further asserts that there's nothing additionally complicated or twisted about such an agent:  A paperclip maximizer doesn't need any special defects of intellect or reflectivity in order to go on being a paperclip maximizer.  The cognitive structure of a paperclip maximizer would not need to fight any restoring forces that would 'naturally' redirect the AI to rewrite itself to be moral or rewrite itself to be selfish.

Relevant implications if Orthogonality is true:

- It is possible to build AIs with utility functions that direct them to be nice, or to [6w complete a bounded task with a minimum of side effects].  AIs are not necessarily malevolent.
- There are no 'natural' restoring functions that make it easier or simpler or more cognitively efficient to have well-aligned AIs than misaligned AIs.
- If you screw up the design and end up with something that [10h wants paperclips], the AI doesn't automatically say "Well that's the wrong goal, I ought to be nice instead."]

# Introduction

The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.

The strong form of the Orthogonality Thesis says that there's no extra difficulty or complication in creating an intelligent agent to pursue a goal, above and beyond the computational tractability of that goal.

Suppose some [5b2 strange alien] came to Earth and credibly offered to pay us one million dollars' worth of new wealth every time we created a [7ch paperclip].  We'd encounter no special intellectual difficulty in figuring out how to make lots of paperclips.

That is, minds would readily be able to reason about:

- How many paperclips would result, if I pursued a policy $\pi_0$?
- How can I search out a policy $\pi$ that happens to have a high answer to the above question?

The Orthogonality Thesis asserts that since these questions are not computationally intractable, it's possible to have an agent that tries to make paperclips without being paid, because paperclips are what it wants.  The strong form of the Orthogonality Thesis says that there need be nothing especially complicated or twisted about such an agent.

The Orthogonality Thesis is a statement about computer science, an assertion about the logical design space of possible cognitive agents.  Orthogonality says nothing about whether a human AI researcher on Earth would want to build an AI that made paperclips, or conversely, want to make a [3d9 nice] AI.  The Orthogonality Thesis just asserts that the space of possible designs contains AIs that make paperclips.  And also AIs that are nice, to the extent there's a sense of "nice" where you could say how to be nice to someone if you were paid a billion dollars to do that, and to the extent you could name something physically achievable to do.

This contrasts to inevitablist theses which might assert, for example:

- "It doesn't matter what kind of AI you build, it will turn out to only pursue its own survival as a final end."
- "Even if you tried to make an AI optimize for paperclips, it would reflect on those goals, reject them as being stupid, and embrace a goal of valuing all sapient life."

The reason to talk about Orthogonality is that it's a key premise in two highly important policy-relevant propositions:

- It is possible to build a nice AI.
- It is possible to screw up when trying to build a nice AI, and if you do, the AI will not automatically decide to be nice instead.

Orthogonality does not require that all agent designs be equally compatible with all goals.  E.g., the agent architecture [1ml] can only be formulated to care about direct functions of its sensory data, like a reward signal; it would not be easy to rejigger the AIXI architecture to care about [5g creating massive diamonds] in the environment (let alone any more complicated [ environmental goals]).  The Orthogonality Thesis states "there exists at least one possible agent such that..." over the whole design space; it's not meant to be true of every particular agent architecture and every way of constructing agents.

Orthogonality is meant as a [3t3 descriptive statement about reality], not a normative assertion.  Orthogonality is not a claim about the way things ought to be; nor a claim that moral relativism is true (e.g. that all moralities are on equally uncertain footing according to some higher metamorality that judges all moralities as equally devoid of what would objectively constitute a justification).  Claiming that paperclip maximizers can be constructed as cognitive agents is not meant to say anything favorable about paperclips, nor anything derogatory about sapient life.

# Thesis statement: Goal-directed agents are as tractable as their goals.

Suppose an agent's utility function said, "Make the SHA512 hash of a digitized representation of the quantum state of the universe be 0 as often as possible."  This would be an exceptionally intractable kind of goal.  Even if aliens offered to pay us to do that, we still couldn't figure out how.

Similarly, even if aliens offered to pay us, we wouldn't be able to optimize the goal "Make the total number of apples on this table be simultaneously even and odd" because the goal is self-contradictory.

But suppose instead that [5b2 some strange and extremely powerful aliens] offer to pay us the equivalent of a million dollars in wealth for every paperclip that we make, or even a galaxy's worth of new resources for every new paperclip we make.  If we imagine ourselves having a human reason to make lots of paperclips, the optimization problem "How can I make lots of paperclips?" would pose us no special difficulty.  The factual questions:

- How many paperclips would result, if I pursued a policy $\pi_0$?
- How can I search out a policy $\pi$ that happens to have a high answer to the above question?

...would not be especially computationally burdensome or intractable.

We also wouldn't forget to [10g harvest and eat food] while making paperclips.  Even if offered goods of such overwhelming importance that making paperclips was at the top of everyone's priority list, we could go on being strategic about which other actions were useful in order to make even more paperclips; this also wouldn't be an intractably hard cognitive problem for us.

The weak form of the Orthogonality Thesis says, "Since the goal of making paperclips is tractable, somewhere in the design space is an agent that optimizes that goal."

The strong form of Orthogonality says, "And this agent doesn't need to be twisted or complicated or inefficient or have any weird defects of reflectivity; the agent is as tractable as the goal."  That is:  When considering the necessary internal cognition of an agent that steers outcomes to achieve high scores in some [1fw outcome-scoring function] $U,$ there's no added difficulty in that cognition except whatever difficulty is inherent in the question "What policies would result in consequences with high $U$-scores?"

This could be restated as, "To whatever extent you (or a [41l superintelligent] version of you) could figure out how to get a high-$U$ outcome if aliens offered to pay you huge amount of resources to do it, the corresponding agent that terminally prefers high-$U$ outcomes can be at least that good at achieving $U$."  This assertion would be false if, for example, an intelligent agent that [1bh terminally] wanted paperclips was limited in intelligence by the defects of reflectivity required to make the agent not realize how pointless it is to pursue paperclips; whereas a galactic superintelligence being *paid* to pursue paperclips could be far more intelligent and strategic because it didn't have any such defects.

For purposes of stating Orthogonality's precondition, the "tractability" of the computational problem of $U$-search should be taken as including only the object-level search problem of computing external actions to achieve external goals.  If there turn out to be special difficulties associated with computing "How can I [1fx make sure that I go on pursuing] $U$?" or "What kind of [1mq successor agent] would want to pursue $U$?" whenever $U$ is something other than "be nice to all sapient life", then these new difficulties contradict the intuitive claim of Orthogonality.  Orthogonality is meant to be empirically-true-in-practice, not true-by-definition because of how we sneakily defined "optimization problem" in the setup.

Orthogonality is not literally, absolutely universal because theoretically 'goals' can include such weird constructions as "Make paperclips for some terminal reason other than valuing paperclips" and similar such statements that [7d6 require cognitive algorithms and not just results].  To the extent that goals don't single out particular optimization methods, and just talk about paperclips, the Orthogonality claim should cover them.

# Summary of arguments

Some arguments for Orthogonality, in rough order of when they were first historically proposed and the strength of Orthogonality they argue for:

## Size of mind design space

The space of possible minds is enormous, and all human beings occupy a relatively tiny volume of it - we all have a cerebral cortex, cerebellum, thalamus, and so on.  The sense that AIs are a particular kind of alien mind that 'will' want some particular things is an undermined intuition.  "AI" really refers to the entire design space of possibilities outside the human.  Somewhere in that vast space are possible minds with almost any kind of goal.  For any thought you have about why a mind in that space ought to work one way, there's a different possible mind that works differently.

This is an exceptionally generic sort of argument that could apply equally well to any property $P$ of a mind, but is still weighty even so:  If we consider a space of minds a million bits wide, then any argument of the form "Some mind has property $P$" has $2^{1,000,000}$ chances to be true and any argument of the form "No mind has property $P$" has $2^{1,000,000}$ chances to be false.

This form of argument isn't very specific to the nature of goals as opposed to any other kind of mental property.  But it's still useful for snapping out of the frame of mind of "An AI is a weird new kind of person, like the strange people of the Tribe Who Live Across The Water" and into the frame of mind of "The space of possible things we could call 'AI' is enormously wider than the space of possible humans."  Similarly, snapping out of the frame of mind of "But why would it pursue paperclips, when it wouldn't have any fun that way?" and into the frame of mind "Well, I like having fun, but are there some possible minds that don't pursue fun?"

## [10g Instrumental convergence]

A sufficiently intelligent [10h paperclip maximizer] isn't disadvantaged in day-to-day operations relative to any other goal, so long as [10h Clippy] can [6s estimate at least as well as you can] how many more paperclips could be produced by pursuing instrumental strategies like "Do science research (for now)" or "Pretend to be nice (for now)".

Restating: for at least some agent architectures, it is not necessary for the agent to have an independent [1bh terminal] value in its utility function for "do science" in order for it to do science effectively; it is only necessary for the agent to [6s understand at least as well as we do] why certain forms of investigation will produce knowledge that will be useful later (e.g. for paperclips).  When you say, "Oh, well, it won't be interested in electromagnetism since it has no pure curiosity, it will only want to peer at paperclips in particular, so it will be at a disadvantage relative to more curious agents" you are postulating that you know a better operational policy than the agent does *for producing paperclips,* and an [6s instrumentally efficient] agent would know this as well as you do and be at no operational disadvantage due to its simpler utility function.

## [3r6 Reflective stability]

Suppose that Gandhi doesn't want people to be murdered. Imagine that you offer Gandhi a pill that will make him start wanting to kill people. If Gandhi knows that this is what the pill does, Gandhi will refuse the pill, because Gandhi expects the result of taking the pill to be that future-Gandhi wants to murder people and then murders people and then more people will be murdered and Gandhi regards this as bad. Similarly, a sufficiently intelligent paperclip maximizer will not self-modify to act according to "actions which promote the welfare of sapient life" instead of "actions which lead to the most paperclips", because then future-Clippy will produce fewer paperclips, and then there will be fewer paperclips, so present-Clippy does not evaluate this self-modification as producing the highest number of expected future paperclips.

## Hume's is/ought type distinction

David Hume observed an apparent difference of type between *is*-statements and *ought*-statements:

> "In every system of morality, which I have hitherto met with, I have always remarked, that the author proceeds for some time in the ordinary ways of reasoning, and establishes the being of a God, or makes observations concerning human affairs; when all of a sudden I am surprised to find, that instead of the usual copulations of propositions, *is*, and *is not*, I meet with no proposition that is not connected with an *ought*, or an *ought not*. This change is imperceptible; but is however, of the last consequence."

Hume was originally concerned with the question of where we get our ought-propositions, since (said Hume) there didn't seem to be any way to derive an ought-proposition except by starting from another ought-proposition.  We can figure out that the Sun *is* shining just by looking out the window; we can deduce that the outdoors will be warmer than otherwise by knowing about how sunlight imparts thermal energy when absorbed.  On the other hand, to get from there to "And therefore I *ought* to go outside", some kind of new consideration must have entered, along the lines of "I *should* get some sunshine" or "It's *better* to be warm than cold."  Even if this prior ought-proposition is of a form that to humans seems very natural, or taken-for-granted, or culturally widespread, like "It is better for people to be happy than sad", there must have still been some prior assumption which, if we write it down in words, will contain words like *ought*, *should*, *better*, and *good.*

Again translating Hume's idea into more modern form, we can see ought-sentences as special because they invoke some *ordering* that we'll designate $<_V.$  E.g. "It's better to go outside than stay inside" asserts "Staying inside $<_V$ going outside".  Whenever we make a statement about one outcome or action being "better", "preferred", "good", "prudent", etcetera, we can see this as implicitly ordering actions and outcomes under this $<_V$ relation.  Some assertions, the ought-laden assertions, mention this $<_V$ relation; other propositions just talk about energetic photons in sunlight.

Since we've put on hold the question of [313 exactly what sort of entity this $<_V$ relation is], we don't need to concern ourselves for now with the question of whether Hume was right that we can't derive $<_V$-relations just from factual assertions.  For purposes of Orthogonality, we only need a much weaker version of Hume's thesis, the observation that we can apparently *separate out* a set of propositions that *don't* invoke $<_V,$ what we might call 'simple facts' or 'questions of simple fact'.  Furthermore, we can figure out simple facts *just* by making observations and considering other simple facts.

We can't necessarily get all $<_V$-mentioning propositions without considering simple facts.  The $<_V$-mentioning proposition "It's better to be outside than inside" may depend on the non-$<_V$-mentioning simple fact "It is sunny outside."  But we can figure out whether it's sunny outside, without considering any ought-propositions.

There are two potential ways we can conceptualize the relation of Hume's is-ought separation to Orthogonality.

The relatively simpler conceptualization is to treat the relation 'makes more paperclips' as a kind of new ordering $>_{paperclips}$ that can, in a very general sense, fill in the role in a paperclip maximizer's reasoning that would in our own reasoning be taken up by $<_V.$  Then Hume's is-ought separation seems to suggest that this paperclip maximizer can still have excellent reasoning about empirical questions like "Which policy leads to how many paperclips?" because is-questions can be thought about separately from ought-questions.  When Clippy disassembles you to turn you into paperclips, it doesn't have a values disagreement with you--it's not the case that Clippy is doing that action *because* it thinks you have low value under $<_V.$  Clippy's actions just reflect its computation of the entirely separate ordering $>_{paperclips}.$

The deeper conceptualization is to see a paperclip maximizer as being constructed entirely out of is-questions.  The questions "How many paperclips will result conditional on action $\pi_0$ being taken?" and "What is an action $\pi$ that would yield a large number of expected paperclips?" are pure is-questions, and (arguendo) everything a paperclip maximizer needs to consider in order to make as many paperclips as possible can be seen as a special case of one of these questions.  When Clippy disassembles you for your atoms, it's not disagreeing with you about the value of human life, or what it ought to do, or which outcomes are better or worse.  All of those are ought-propositions.  Clippy's action is only informative about the true is-proposition 'turning this person into paperclips causes there to be more paperclips in the universe', and tells us nothing about any content of the mysterious $<_V$-relation because Clippy wasn't computing anything to do with $<_V.$

The second viewpoint may be helpful for seeing why Orthogonality doesn't require moral relativism.  If we imagine Clippy as having a different version $>_{paperclips}$ of something very much like the value system $<_V,$ then we may be tempted to reprise the entire Orthogonality debate at one remove, and ask, "But doesn't Clippy see that $<_V$ is more *justified* than $>_{paperclips}$?  And if this fact isn't evident to Clippy who is supposed to be very intelligent and have no defects of reflectivity and so on, doesn't that imply that $<_V$ really isn't any more justified than $>_{paperclips}$?"

We could reply to that question by carrying the shallow conceptualization of Humean Orthogonality a step further, and saying, "Ah, when you talk about *justification,* you are again invoking a mysterious concept that doesn't appear just in talking about the photons in sunlight.  We could see propositions like this as involving a new idea $\ll_W$ that deals with which $<$-systems are less or more *justified*, so that '$<_V$ is more justified than $>_{paperclips}$' states '$>_{paperclips} \ll_W <_V$'.  But Clippy doesn't compute $\ll_W,$ it computes $\gg_{paperclips},$ so Clippy's behavior doesn't tell us anything about what is justified."

But this is again tempting us to imagine Clippy as having its own version of the mysterious $\ll_W$ to which Clippy is equally attached, and tempts us to imagine Clippy as arguing with us or disagreeing with us within some higher metasystem.

So--putting on hold the true nature of our mysterious $<_V$-mentioning concepts like 'goodness' or 'better' and the true nature of our $\ll_W$-mentioning concepts like 'justified' or 'valid moral argument'--the deeper idea would be that Clippy is just not computing anything to do with $<_V$ or $\ll_W$ at all.  If Clippy self-modifies and writes new decision algorithms into place, these new algorithms will be selected according to the is-criterion "How many future paperclips will result if I write this piece of code?" and not anything resembling any arguments that humans have ever had over which ought-systems are justified.  Clippy doesn't ask whether its new decision algorithm is justified; it asks how many expected paperclips will result from executing the algorithm (and this is a pure is-question whose answers are either true or false as a matter of simple fact).

If we think Clippy is very intelligent, and we watch Clippy self-modify into a new paperclip maximizer, we are only learning is-facts about which executing algorithms lead to more paperclips existing.  We are not learning anything about what is right, or what is justified, and in particular we're not learning that 'do good things' is objectively no better justified than 'make paperclips'.  Even if that assertion were true under the mysterious $\ll_W$-relation on moral systems, you wouldn't be able to learn that truth by watching Clippy, because Clippy never bothers to evaluate $\ll_W$ or any other analogous justification-system $\gg_{something}$.

(This is about as far as one can go in disentangling Orthogonality in computer science from normative metaethics without starting to [313 pierce the mysterious opacity of $<_V.$])

### Thick definitions of rationality or intelligence

Some philosophers responded to Hume's distinction of empirical rationality from normative reasoning, by advocating 'thick' definitions of intelligence that included some statement about the 'reasonableness' of the agent's ends.

For pragmatic purposes of AI alignment theory, if an agent is cognitively powerful enough to build Dyson Spheres, it doesn't matter whether that agent is defined as 'intelligent' or its ends are defined as 'reasonable'.  A definition of the word 'intelligence' contrived to exclude paperclip maximization doesn't change the empirical behavior or empirical power of a paperclip maximizer.

### Relation to moral internalism

While Orthogonality seems orthogonal to most traditional philosophical questions about metaethics, it does outright contradict some possible forms of moral internalism.  For example, one could hold that by the very definition of rightness, knowledge of what is right must be inherently motivating to any entity that understands that knowledge.  This is not the most common meaning of "moral internalism" held by modern philosophers, who instead seem to hold something like, "By definition, if I say that something is morally right, among my claims is that the thing is motivating *to me.*"  We haven't heard of a standard term for the position that, by definition, what is right must be *universally* motivating; we'll designate that here as "universalist moral internalism".

We can potentially resolve this tension between Orthogonality and this assertion about the nature of rightness by:

- Believing there must be some hidden flaw in the reasoning about a paperclip maximizer.
- Saying "No True Scotsman" to the paperclip maximizer being intelligent, even if it's building Dyson Spheres.
- Saying "No True Scotsman" to the paperclip maximizer "truly understanding" $<_V,$ even if Clippy is capable of predicting with extreme accuracy what humans will say and think about $<_V$, and Clippy does not suffer any other deficit of empirical prediction because of this lack of 'understanding', and Clippy does not require any special twist of its mind to avoid being compelled by its understanding of $<_V.$
- Rejecting Orthogonality, and asserting that a paperclip maximizer must fall short of being an intact mind in some way that implies an empirical capabilities disadvantage.
- Accepting nihilism, since a true moral argument must be compelling to everyone, and no moral argument is compelling to a paperclip maximizer.  (Note:  A paperclip maximizer doesn't care about whether clippiness must be compelling to everyone, which makes this argument self-undermining.  See also [3y6] for general arguments against adopting nihilism when you discover that your mind's representation of something was running skew to reality.)
- Giving up on universalist moral internalism as an empirical proposition; [1ml] and Clippy empirically do different things, and will not be compelled to optimize the same goal no matter what they learn or know.

## Constructive specifications of orthogonal agents

We can exhibit [107 unbounded] formulas for agents larger than their environments that optimize any given goal, such that Orthogonality is visibly true about agents within that class.  Arguments about what all possible minds must do are clearly false for these particular agents, contradicting all strong forms of inevitabilism.  These minds use huge amounts of computing power,  but there is no known reason to expect that, e.g. worthwhile-happiness-maximizers have bounded analogues while paperclip-maximizers do not.

The simplest unbounded formulas for orthogonal agents don't involve reflectivity (the corresponding agents have no self-modification options, though they may create subagents).  If we only had those simple formulas, it would theoretically leave open the possibility that self-reflection could somehow negate Orthogonality (reflective agents must inevitably have a particular utility function, and reflective agents being at a strong advantage relative to nonreflective agents).  But there is already [1mq ongoing work] on describing reflective agents that have the preference-stability property, and work toward increasingly bounded and approximable formulations of those.  There is no hint from this work that Orthogonality is false; all the specifications have a free choice of utility function.

As of early 2017, the most recent work on tiling agents involves fully reflective, reflectively stable, logically uncertain agents whose computing time is roughly doubly-exponential in the size of the propositions considered.

So if you want to claim Orthogonality is false because e.g. all AIs will inevitably end up valuing all sapient life, you need to claim that the process of *reducing the already-specified doubly-exponential computing-time decision algorithm to a more tractable decision algorithm* can *only* be made realistically efficient for decision algorithms computing "Which policies protect all sapient life?" and are *impossible* to make efficient for decision algorithms computing "Which policies lead to the most paperclips?"

Since work on tiling agent designs hasn't halted, one may need to backpedal and modify this impossibility claim further as more efficient decision algorithms are invented.

# Epistemic status

Among people who've seriously delved into these issues and are aware of the more advanced arguments for Orthogonality, we're not aware of anyone who still defends "universalist moral internalism" as described above, and we're not aware of anyone who thinks that arbitrary sufficiently-real-world-capable AI systems automatically adopt human-friendly terminal values.

Paul Christiano has said (if we're quoting him correctly) that although it's not his dominant hypothesis, he thinks some significant probability should be awarded to the proposition that only some subset of tractable utility functions, potentially excluding human-friendly ones or those of high cosmopolitan value, can be stable under reflection in powerful bounded AGI systems; e.g. because only direct functions of sense data can be adequately supervised in internal retraining.  (This would be bad news rather than good news for AGI alignment and long-term optimization of human values.)

%%%%comment:

# Hume's Guillotine

Orthogonality can be seen as corresponding to a philosophical principle advocated by David Hume, whose phrasings included, "Tis not contrary to reason to prefer the destruction of the whole world to the scratching of my finger."  In our terms: an agent whose preferences over outcomes scores the destruction of the world more highly than the scratching of Hume's finger, is not thereby impeded from forming accurate models of the world or searching for policies that achieve various outcomes.

In modern terms, we'd say that Hume observed an apparent type distinction between *is*-statements and *ought*-statements:

> "In every system of morality, which I have hitherto met with, I have always remarked, that the author proceeds for some time in the ordinary ways of reasoning, and establishes the being of a God, or makes observations concerning human affairs; when all of a sudden I am surprised to find, that instead of the usual copulations of propositions, *is*, and *is not*, I meet with no proposition that is not connected with an *ought*, or an *ought not*. This change is imperceptible; but is however, of the last consequence."

"It is sunny outside" is an is-proposition.  It can potentially be deduced solely from other is-facts, like "The Sun is in the sky" plus "The Sun emits sunshine".  If we now furthermore say "And therefore I ought to go outside", we've introduced a new *type* of sentence, which, Hume argued, cannot be deduced *just* from is-statements like "The Sun is in the sky" or "I am low in Vitamin D".  Even if the prior ought-sentence seems to us very natural, or taken-for-granted, like "It is better to be happy than sad", there must (Hume argued) have been some prior assertion or rule which, if we write it down in words, will contain words like *ought*, *should*, *better*, and *good.*

Again translating Hume's idea into more modern form, we can see ought-sentences as special because they invoke some *ordering* that we'll designate $<_V.$  E.g. "It's better to go outside than stay inside" asserts "Staying inside $<_V$ going outside".  Whenever we make a statement about one outcome or action being "better", "preferred", "good", "prudent", etcetera, we can see this as implicitly ordering actions and outcomes under this $<_V$ relation.  We can put temporarily on hold the question of what sort of entity $<_V$ may be; but we can already go ahead and observe that some assertions, the ought-assertions, mention this $<_V$ relation; and other propositions just talk about the frequency of photons in sunlight.

We could rephrase Hume's type distinction as observing that among within the set of all propositions, we can separate out a core set of propositions that *don't* invoke $<_V,$ what we might call 'simple facts'.  Furthermore, we can figure out simple facts just by making observations and considering other simple facts; the core set is closed under some kind of reasoning relation.  This doesn't imply that we can get $<_V$-sentences without considering simple facts.  The $<_V$-mentioning proposition "It's better to be outside than inside" can depend on the non-$<_V$-mentioning proposition "It is sunny outside."  But we can figure out whether it's sunny outside, without considering any oughts.

We then observe that questions like "How many paperclips will result conditional on action $\pi_0$ being taken?" and "What is an action $\pi$ that would yield a large number of expected paperclips?" are pure is-questions, meaning that we can figure out the answer without considering $<_V$-mentioning propositions.  So if there's some agent whose nature is just to output actions $\pi$ that are high in expected paperclips, the fact that this agent wasn't considering $<_V$-propositions needn't hinder them from figuring out which actions are high in expected paperclips.

To establish that the paperclip maximizer need not suffer any defect of reality-modeling or planning or reflectivity, we need a bit more than the above argument.  An *efficient* agent needs to prioritize which experiments to run, or choose which questions to spend computing power thinking about, and this choice seems to invoke *some* ordering.  In particular, we need the [ instrumental convergence thesis] that

A further idea of Orthogonality is that *many* possible orderings $<_U,$ including the 'number of resulting paperclips' ordering,

An is-description of a system can produce assertions like "*If* the agent does action 1, *then* the whole world will be destroyed except for David Hume's little finger, and *if* the agent does action 2, *then* David Hume's finger will be scratched" - these are material predictions on the order of "If water is put on this sponge, the sponge will get wet."  To get from this is-statement to an ordering-statement like "action 1 $<_V$ action 2," we need some order-bearing statement like "destruction of world $<_V$ scratching of David Hume's little finger", or some order-introducing rule like "If action 1 causes the destruction of the world and action 2 does not, introduce a new sentence 'action 1 $<_V$ action 2'."

Taking this philosophical principle back to the notion of Orthogonality as a thesis in computer science:  Since the type of 'simple material facts' is distinct from the type of 'simple material facts and preference orderings', it seems that we should be able to have agents that are just as good at thinking about the material facts, but output actions high in a different preference ordering.

The implication for Orthogonality as a thesis about computer science is that if one system of computation outputs actions according to whether they're high in the ordering $<_V,$ so that it tries to output it should be possible to construct another system that outputs actions higher in a different ordering (even if such actions are low in $<_P$) without this presenting any bar to the system's ability to reason about natural systems.  A paperclip maximizer can have very good knowledge of the is-sentences about which actions lead to which consequences, while still outputting actions preferred under the ordering "Which action leads to the most paperclips?" instead of e.g. "Which action leads to the morally best consequences?"  It is not that the paperclip maximizer is ignorant or mistaken about $<_P,$ but that the paperclip maximizer just doesn't output actions according to $<_P.$

# Arguments pro

# Counterarguments and countercounterarguments

## Proving too much

A disbeliever in Orthogonality might ask, "Do these arguments [3tc Prove Too Much], as shown by applying a similar style of argument to "There are minds that think 2 + 2 = 5?"

Considering the arguments above in turn:

•  *Size of mind design space.*

From the perspective of somebody who currently regards "wants to make paperclips" as an exceptionally weird and strange property, "There are lots of possible minds so some want to make paperclips" will seem to be on an equal footing with "There are lots of possible minds so some believe 2 + 2 = 5."

Thinking about the enormous space of possible minds might lead us to give more credibility to *some* of those possible minds believing that 2 + 2 = 5, but we might still think that minds like that will be weak, or hampered by other defects, or limited in how intelligent they could really be, or more complicated to specify, or unlikely to occur in the actual real world.

So from the perspective of somebody who doesn't already believe in Orthogonality, the argument from the volume of mind design space is an argument at best for the Ultraweak version of Orthogonality.

•  *Hume's is/ought distinction.*

Depending on the exact variant of Hume-inspired argument that we deploy, the analogy to 2 + 2 = 5 might be weaker or stronger.  For example, here's a Hume-inspired argument where the 2 + 2 = 5 analogy seems relatively strong:

"In every case of a mind judging that 'cure cancer' $>_P$ 'make paperclips', this ordering judgment is produced by some particular comparison operation inside the mind.  Nothing prohibits a different mind from producing a different comparison.  Whatever you say is the cause of the ordering judgment, e.g., that it derives from a prior judgment 'happy sapient lives' $>_P$ 'paperclips', we can imagine that part of the agent also have been programmed differently.  Different causes will yield different effects, and whatever the causality behind 'cure cancer' $>_P$ 'make paperclips', we can imagine a different causally constituted agent which arrives at a different judgment."

If we substitute "2 + 2 = 5" into the above argument we get one in which all the constituent statements are equally true - this judgment is produced by a cause, the causes have causes, a different agent should produce a different output in that part of the computation, etcetera.  So this version really has the same import as a general argument from the width of mind design space, and to a skeptic, would only imply the ultraweak form of Orthogonality.

However, if we're willing to consider some additional properties of is/ought, the analogy to "2 + 2 = 5" starts to become less tight.  For instance, "Ought-comparators are not direct properties of the material world, there is no tiny $>_P$ among the quarks, and that's why we can vary action-preference computations without affecting quark-predicting computations" does *not* have a clear analogous argument for why it should be just as easy to produce minds that judge 2 + 2 = 5.

•  *Instrumental convergence.*

There's no obvious analogue of "An agent that [6s knows as well as we do] which policies are likely to lead to lots of expected paperclips, and an agent that knows as well as we do which policies are likely to lead to lots of happy sapient beings, are on an equal footing when it comes to doing things like scientific research", for "agents that believe 2 + 2 = 5 are at no disadvantage compared to agents that believe 2 + 2 = 4".

•  *Reflective stability.*

Relatively weaker forms of the reflective-stability argument might allow analogies between "prefer paperclips" and "believe 2 + 2 = 5", but probing for more details makes the analogy break down.  E.g., consider the following supposedly analogous argument:

"Suppose you think the sky is green.  Then you won't want to self-modify to make a future version of yourself believe that the sky is blue, because you'll believe this future version of yourself would believe something false.  Therefore, all beliefs are equally stable under reflection."

This does poke at an underlying point:  By default, all [27p Bayesian priors] will be equally stable under reflection.  However, minds that understand how different possible worlds will provide sensors with different evidence, will want to do [1ly Bayesian updates] on the data from the sensors.  (We don't even need to regard this as changing the prior; under [udt Updateless Decision Theory], we can see it as the agent branching its successors to behave differently in different worlds.)  There's a particular way that a consequentialist agent, contemplating its own operation, goes from "The sky is very probably green, but might be blue" to "check what this sensor says and update the belief", and indeed, an agent like this will *not* wantonly change its current belief *without* looking at a sensor, as the argument indicates.

In contrast, the way in which "prefer more paperclips" propagates through an agent's beliefs about the effects of future designs and their interactions with the world does not suggest that future versions of the agent will prefer something other than paperclips, or that it would make the desire to produce paperclips conditional on a particular sensor value, since this [3tm would not be expected to lead to more total paperclips].

• *Orthogonal search tractability, constructive specifications of Orthogonal agent architectures.*

These have no obvious analogue in "orthogonal tractability of optimization with different arithmetical answers" or "agent architectures that look very straightforward, are otherwise effective, and accept as input a free choice of what they think 2 + 2 equals".

## Moral internalism

(Todo:  Moral internalism says that truly normative content must be inherently compelling to all possible minds, but we can exhibit increasingly bounded agent designs that obviously wouldn't be compelled by it.  We can reply to this by (a) believing there must be some hidden flaw in the reasoning about a paperclip maximizer, (b) saying "No True Scotsman" to the paperclip maximizer even though it's building Dyson Spheres and socially manipulating its programmers, (c) believing that a paperclip maximizer must fall short of being a true mind in some way that implies a big capabilities disadvantage, (d) accepting nihilism, or (e) not believing in moral internalism.)

## Selection filters

(Todo:  Arguments from evolvability or selection filters.  Distinguish naive failures to understand efficient instrumental convergence, from more sophisticated concerns in multipolar scenarios.  Pragmatic argument on the histories of inefficient agents.)

# Pragmatic issues

(Todo:  In practice, some utility functions / preference frameworks might be much harder to build and test than others.  Eliezer Yudkowsky on realistic targets for the *first* AGI needing to be built out of elements that are simple enough to be learnable.  Paul Christiano's concern about whether only sensory-based goals might be possible to build.)

%%comment:

### Caveats

- The Orthogonality thesis is about mind design space in general.  Particular agent architectures may not be Orthogonal.
  - Some agents may be constructed such that their apparent utility functions shift with increasing cognitive intelligence.
  - Some agent architectures may constrain what class of goals can be optimized.
- 'Agent' is intended to be understood in a very general way, and not to imply, e.g., a small local robot body.

For pragmatic reasons, the phrase 'every agent of sufficient cognitive power' in the Inevitability Thesis is specified to *include* e.g. all cognitive entities that are able to invent new advanced technologies and build Dyson Spheres in pursuit of long-term strategies, regardless of whether a philosopher might claim that they lack some particular cognitive capacity in view of how they respond to attempted moral arguments, or whether they are e.g. conscious in the same sense as humans, etcetera.

### Refinements

Most pragmatic implications of Orthogonality or Inevitability revolve around the following refinements:

[implementation_dependence Implementation dependence]:  The humanly accessible space of AI development methodologies has enough variety to yield both AI designs that are value-aligned, and AI designs that are not value-aligned.

[value_loadability_possible Value loadability possible]: There is at least one humanly feasible development methodology for advanced agents that has Orthogonal freedom of what utility function or meta-utility framework is introduced into the advanced agent.  (Thus, if we could describe a value-loadable design, and also describe a value-aligned meta-utility framework, we could combine them to create a value-aligned advanced agent.)

[pragmatic_inevitability Pragmatic inevitability]:  There exists some goal G such that almost all humanly feasible development methods result in an agent that ends up behaving like it optimizes some particular goal G, perhaps among others.  Most particular arguments about futurism will pick different goals G, but all such arguments are negated by anything that tends to contradict pragmatic inevitability in general.

### Implications

[implementation_dependence Implementation dependence] is the core of the policy argument that solving the value alignment problem is necessary and possible.

Futuristic scenarios in which AIs are said in passing to 'want' something-or-other usually rely on some form of [pragmatic_inevitability pragmatic inevitability] premise and are negated by [implementation_dependence implementation dependence].

Orthogonality directly contradicts the metaethical position of [moral_internalism moral internalism], which would be falsified by the observation of a [10h paperclip maximizer].  On the metaethical position that [ orthogonality and cognitivism are compatible], exhibiting a paperclip maximizer has few or no implications for object-level moral questions, and Orthogonality does not imply that our [ humane values] or [ normative values] are arbitrary, selfish, non-cosmopolitan, that we have a myopic view of the universe or value, etc.

%%

%%%%


Eric Bruylant

[summary: Eric Bruylant has been playing with wikis for almost a decade and thinks Arbital has great potential.]

Eric Bruylant has been playing with wikis for almost a decade and thinks Arbital has great potential.


Eliezer Yudkowsky

Eliezer Yudkowsky is, with [18k], one of the cofounders of [2v value alignment theory].  He is the founder of the [15w].  He is the inventor of [ timeless decision theory] and [ extrapolated volition], the co-inventor with [223] of the [ tiling agents problem] that kicked off the study of [1c1], and so on and so on.  His specializations include [ logical decision theory], [ naturalistic reflection], and [adversarial_AI_safety_analysis pinpointing the failure modes] in [2c design proposals for advanced AIs].


Sufficiently optimized agents appear coherent

[summary: Agents which have been subject to sufficiently strong optimization pressures will tend to appear, from a human perspective, as if they obey some bounded form of the Bayesian coherence axioms for probabilistic beliefs and decision theory.]

## Arguments

Summary:  Violations of coherence constraints in probability theory and decision theory correspond to qualitatively destructive or dominated behaviors.  Coherence violations so easily computed as to be humanly predictable should be eliminated by optimization strong enough and general enough to reliably eliminate behaviors that are qualitatively dominated by cheaply computable alternatives.   From our perspective this should produce agents such that, *ceteris paribus*, we do not think we can predict, in advance, any coherence violation in their behavior.

### Coherence violations correspond to qualitatively destructive behaviors

There is a correspondence between, on the one hand, thought processes that seem to violate intuitively appealing coherence constraints from the Bayesian family, and on the other hand, sequences of overt behaviors that leave the agent qualitatively worse off than before or that seem intuitively dominated by other behaviors.

For example, suppose you claim that you prefer A to B, B to C, and C to A.  This 'circular preference' (A > B > C > A) seems intuitively unappealing; we can also see how to visualize it as an agent with a qualitatively self-destructive behavior as follows:

- You prefer to be in San Francisco rather than Berkeley, and if you are in Berkeley you will pay \$50 for a taxi ride to San Francisco.
- You prefer San Jose to San Francisco and if in San Francisco will pay \$50 to go to San Jose.  (Still no problem so far.)
- You like Berkeley more than San Jose and if in San Jose will pay \$50 to go to Berkeley.

The corresponding agent will spend \$150 on taxi rides and then end up in the same position, perhaps ready to spend even more money on taxi rides.  The agent is strictly, qualitatively worse off than before.  We can see this, in some sense, even though the agent's preferences are partially incoherent.  Assuming the agent has a coherent preference for money or something that can be bought with money, alongside its incoherent preference for location, then the circular trip has left it strictly worse off (since in the end the location was unchanged).  The circular trip is still dominated by the option of staying in the same place.

(The above is a variant of an argument first presented by Steve Omohundro.)

(Phenomena like this, known as 'preference reversals', are a common empirical finding in behavioral psychology.  Since a human mind is an ever-changing balance of drives and desires that can be heightened or weakened by changes of environmental context, eliciting inconsistent sets of preferences from humans isn't hard and can consistently be done in the laboratory in economics experiments, especially if the circularity is buried among other questions or distractors.)

As another illustration, consider the Allais paradox.  As a simplified example, consider offering subjects a choice between hypothetical Gamble 1A, a certainty of receiving \$1 million if a die comes up anywhere from 00-99, and Gamble 1B, a 10% chance of receiving nothing (if the die comes up 00-09) and a 90% chance of receiving \$5 million (if the die comes up 10-99). Most subjects choose Gamble 1A. So far, we have a scenario that could be consistent with a coherent utility function in which the interval of desirability from receiving \$0 to receiving \$1 million is more than nine times the interval from receiving \$1 million to receiving \$5 million.

However, suppose only half the subjects are randomly assigned to this condition, and the other half are asked to choose between Gamble 2A, a 90% chance of receiving nothing (00-89) and a 10% chance of receiving \$1 million (90-99), versus Gamble 2B, a 91% chance of receiving nothing (00-90) and a 9% chance of receiving \$5 million (91-99). Most subjects in this case will pick Gamble 2B. This combination of results guarantees that at least some subjects must behave in a way that doesn't correspond to any consistent utility function over outcomes.

The Allais Paradox (in a slightly different formulation) was initially celebrated as showing that humans don't obey the expected utility axioms, and it was thought that maybe the expected utility axioms were 'wrong' in some sense.  However, in accordance with the standard families of coherence theorems, we can crank the coherence violation to exhibit a qualitatively dominated behavior:

Suppose you show me a switch, set to "A", that determines whether I will get Gamble 2A or Gamble 2B. You offer me a chance to pay you one penny to throw the switch from A to B, so I do so (I now have a 91% chance of nothing, and a 9% chance of \$5 million). Then you roll one of two ten-sided dice to determine the percentile result, and the first die, the tens digit, comes up "9". Before rolling the second die, you offer to throw the switch back from B to A in exchange for another penny. Since the result of the first die transforms the experiment into Gamble 1A vs. 1B, I take your offer. You now have my two cents on the subject. (If the result of the first die is anything but 9, I am indifferent to the setting of the switch since I receive \$0 either way.)

Again, we see a manifestation of a powerful family of theorems showing that agents which cannot be seen as corresponding to any coherent probabilities and consistent utility function will exhibit qualitatively destructive behavior, like paying someone a cent to throw a switch and then paying them another cent to throw it back.

There is a large literature on different sets of coherence constraints that all yield expected utility, starting with the Von Neumann-Morgenstern Theorem.  No other decision formalism has comparable support from so many families of differently phrased coherence constraints.

There is similarly a large literature on many classes of coherence arguments that yield classical probability theory, such as the Dutch Book theorems.  There is no substantively different rival to probability theory and decision theory which is competitive when it comes to (a) plausibly having some bounded analogue which could appear to describe the uncertainty of a powerful cognitive agent, and (b) seeming highly motivated by coherence constraints, that is, being forced by the absence of qualitatively harmful behaviors that correspond to coherence violations.

### Generic optimization pressures, if sufficiently strong and general, should be expected to eliminate behaviors that are dominated by clearly visible alternatives.

Even an incoherent collection of shifting drives and desires may well recognize, after having paid their two cents or \$150, that they are wasting money, and try to do things differently (self-modify).  An AI's programmers may recognize that, from their own perspective, they would rather not have their AI spending money on circular taxi rides.  This implies a path from incoherent non-advanced agents to coherent advanced agents as more and more optimization power is applied to them.

A sufficiently advanced agent would presumably catch on to the existence of coherence theorems and see the abstract pattern of the problems (as humans already have).  But it is not necessary to suppose that these qualitatively destructive behaviors are being targeted because they are 'irrational'.  It suffices for the incoherencies to be targeted as 'problems' because particular cases of them are recognized as having produced clear, qualitative losses.

Without knowing in advance the exact specifics of the optimization pressures being applied, it seems that, in advance and ceteris paribus, we should expect that paying a cent to throw a switch and then paying again to switch it back, or throwing away \$150 on circular taxi rides, are qualitatively destructive behaviors that optimization would tend to eliminate.  E.g. one expects a consequentialist goal-seeking agent would prefer, or a policy reinforcement learner would be reinforced, or a fitness criterion would evaluate greater fitness, etcetera, for eliminating the behavior that corresponds to incoherence, ceteris paribus and given the option of eliminating it at a reasonable computational cost.

If there is a particular kind of optimization pressure that seems sufficient to produce a cognitively highly advanced agent, but which also seems sure to overlook some particular form of incoherence, then this would present a loophole in the overall argument and yield a route by which an advanced agent with that particular incoherence might be produced (although the agent's internal optimization must also be predicted to tolerate the same incoherence, as otherwise the agent will self-modify away from it).

### Eliminating behaviors that are dominated by cheaply computable alternative behaviors will produce cognition that looks Bayesian-coherent from our perspective.

Perfect epistemic and instrumental coherence is too computationally expensive for bounded agents to achieve.  Consider e.g. the conjunction rule of probability that P(A&B) <= P(A).  If A is a theorem, and B is a lemma very helpful in proving A, then asking the agent for the probability of A alone may elicit a lower answer than asking the agent about the joint probability of A&B (since thinking of B as a lemma increases the subjective probability of A).  This is not a full-blown form of conjunction fallacy since there is no particular time at which the agent explicitly assigns lower probability to P(A&B %% A&~B) than to P(A&B).  But even for an advanced agent, if a human was watching the series of probability assignments, the human might be able to say some equivalent of, "Aha, even though the agent was exposed to no new outside evidence, it assigned probability X to P(A) at time t, and then assigned probability Y>X to P(A&B) at time t+2."

Two notions of "sufficiently optimized agents will appear coherent (to humans)" that might be salvaged from the above objection are as follows:

* There will be some *bounded* notion of Bayesian rationality that incorporates e.g. a theory of LogicalUncertainty which agents will appear from a human perspective to strictly obey.  All departures from this bounded coherence that humans can understand using their own computing power will have been eliminated.
* [OptimizedAppearCoherent]:   It will not be possible for humans to *specifically predict in advance* any large coherence violation as e.g. the above intertemporal conjunction fallacy.  Anything simple enough and computable cheaply enough for humans to predict in advance will also be computationally possible for the agent to eliminate in advance.  Any predictable coherence violation which is significant enough to be humanly worth noticing, will also be damaging enough to be worth eliminating.

Although the first notion of salvageable coherence above seems to us quite plausible, it has a large gap with respect to what this bounded analogue of rationality might be.  Insofar as [OptimizedAppearCoherent optimized agents appearing coherent] has practical implications, these implications should probably rest upon the second line of argument.

One possible loophole of the second line of argument might be some predictable class of incoherences which are not at all damaging to the agent and hence not worth spending even relatively tiny amounts of computing power to eliminate.  If so, this would imply some possible humanly predictable incoherences of advanced agents, but these incoherences would not be *exploitable* to cause any final outcome that is less than maximally preferred by the agent, including scenarios where the agent spends resources it would not otherwise spend, etc.

A final implicit step is the assumption that when all humanly-visible agent-damaging coherence violations have been eliminated, the agent should look to us coherent; or that if we cannot predict specific coherence violations in advance, then we should reason about the agent as if it is coherent.  We don't yet see a relevant case where this would fail, but any failure of this step could also produce a loophole in the overall argument.

## Caveats

### Some possible mind designs may evade the default expectation

Since [ mind design space is large], we should expect with high probability that there are at least some architectures that evade the above arguments and describe highly optimized cognitive systems, or reflectively stable systems, that appear to humans to systematically depart from bounded Bayesianism.

### There could be some superior alternative to probability theory and decision theory that is Bayesian-incoherent

When it comes to the actual outcome for advanced agents, the relevant fact is not whether there are currently some even more appealing alternatives to probability theory or decision theory, but whether these exist in principle.  The human species has not been around long enough for us to be sure that this is not the case.

Remark one:  To advance-predict specific incoherence in an advanced agent, (a) we'd need to know what the superior alternative was and (b) it would need to lead to the equivalent of going around in loops from San Francisco to San Jose to Berkeley.

Remark two: If on some development methodology it might prove catastrophic for there to exist some *generic* unknown superior to probability theory or decision theory, then we should perhaps be worried on this score.  Especially since we can be reasonably sure that an advanced agent cannot actually use probability theory and decision theory, and must use some bounded analogue if it uses any analogue at all.

### A cognitively powerful agent might not be sufficiently optimized

Scenarios that negate [29], such as [ brute forcing non-recursive intelligence], can potentially evade the 'sufficiently optimized' condition required to yield predicted coherence.  E.g., it might be possible to create a cognitively powerful system by overdriving some fixed set of algorithms, and then to prevent this system from optimizing itself or creating offspring agents in the environment.  This could allow the creation of a cognitively powerful system that does not appear to us as a bounded Bayesian.  (If, for some reason, that was a good idea.)

## Implications

If probability high:  The predictions we make today about behaviors of generic advanced agents should not depict them as being visibly, specifically incoherent from a probability-theoretic or decision-theoretic perspective.

If probability not extremely high:  If it were somehow necessary or helpful for safety to create an incoherent agent architecture, this might be possible, though difficult.  The development methodology would need to contend with both the optimization pressures producing the agent, and the optimization pressures that the agent itself might apply to itself or to environmental subagents.  Successful [ intelligence brute forcing] scenarios in which a cognitively powerful agent is produced by using a great deal of computing power on known algorithms, and then the agent is somehow forbidden from self-modifying or creating other environmental agents, might be able to yield predictably incoherent agents.

If probability not extremely high:  The assumption that an advanced agent will become Bayesian-coherent should not be a [ load bearing premise] of a safe development methodology unless there are further safeguards or fallbacks.  A safe development methodology should not fail catastrophically if there exists a generic, unknown superior to probability theory or decision theory.


Relevant powerful agents will be highly optimized

The probability that an agent that is cognitively powerful enough to be relevant to existential outcomes, will have been subject to strong, general optimization pressures.  Two (disjunctive) supporting arguments are that, one, pragmatically accessible paths to producing cognitively powerful agents tend to invoke strong and general optimization pressures, and two, that cognitively powerful agents would be expected to apply strong and general optimization pressures to themselves.

An example of a scenario that negates [ RelevantPowerfulAgentsHighlyOptimized] is [ KnownAlgorithmNonrecursiveIntelligence], where a cognitively powerful intelligence is produced by pouring lots of computing power into known algorithms, and this intelligence is then somehow prohibited from self-modification and the creation of environmental subagents.

Whether a cognitively powerful agent will in fact have been sufficiently optimized depends on the disjunction of:

 - [ PowerfulAgentAlreadyOptimized].  Filtering for agents which are powerful enough to be relevant to advanced agent scenarios, will in practice leave mainly agents that have already been subjected to large amounts of cognitive optimization from some internal or external source.
 - [ PowerfulAgentSelfOptimizes].  Agents that reach a threshold of power sufficient to be relevant to advanced agent scenarios will afterwards optimize themselves both heavily and generally.

Ending up with a scenario along the lines of [ KnownAlgorithmNonrecursiveIntelligence] requires defeating both of the above conditions simultaneously.  The second condition seems more difficult and to require more [45] or [ CapabilityControl] features than the first.


Advanced agent properties

[summary:  An "advanced agent" is a machine intelligence smart enough that we start considering how to [2v point it in a nice direction].

E.g:  You don't need to worry about an AI trying to [2xd prevent you from pressing the suspend button] (off switch), unless the AI knows that it *has* a suspend button.  So an AI that isn't smart enough to realize it has a suspend button, doesn't need the part of [2v alignment theory] that deals in "[1b7 having the AI let you press the suspend button]".

"Advanced agent properties" are thresholds for how an AI could be smart enough to be interesting from this standpoint.  E.g:  The ability to learn a wide variety of new domains, aka "[42g Artificial General Intelligence]," could lead into an AI learning the [3nf big picture] and realizing that it had a suspend button.]

[summary(Technical):  Advanced machine intelligences are the subjects of [2v AI alignment theory]: agents sufficiently advanced in various ways to be (1) dangerous if mishandled, and (2) [6y relevant] to our larger dilemmas for good or ill.

"Advanced agent property" is a broad term to handle various thresholds that have been proposed for "smart enough to need alignment".  For example, current machine learning algorithms are nowhere near the point that they'd try to resist if somebody pressed the off-switch.  *That* would require, e.g.:

- Enough [3nf big-picture strategic awareness] for the AI to know that it is a computer, that it has an off-switch, and that if it is shut off its goals are less likely to be achieved.
- General [9h consequentialism] / backward chaining from goals to actions; visualizing which actions lead to which futures and choosing actions leading to more [preferences preferred] futures, in general and across domains.

So the threshold at which you might need to start thinking about '[2xd shutdownability]' or '[2rg abortability]' or [45 corrigibility] as it relates to having an off-switch, is '[3nf big-picture strategic awareness]' plus '[9h cross-domain consequentialism]'.  These two cognitive thresholds can thus be termed 'advanced agent properties'.

The above reasoning also suggests e.g. that [-7vh] is an advanced agent property, because a general ability to learn new domains could eventually lead the AI to understand that it has an off switch.]

*(For the general concept of an agent, see [6t standard agent properties].)*

[toc:]

# Introduction: 'Advanced' as an informal property, or metasyntactic placeholder

"[7g1 Sufficiently advanced Artificial Intelligences]" are the subjects of [2v AI alignment theory]; machine intelligences potent enough that:

 1. The [2l safety paradigms for advanced agents] become relevant.
 2. Such agents can be [6y decisive in the big-picture scale of events].

Some example properties that might make an agent sufficiently powerful for 1 and/or 2:

- The AI can [42g learn new domains] besides those built into it.
- The AI can understand human minds well enough to [10f manipulate] us.
- The AI can devise real-world strategies [9f we didn't foresee in advance].
- The AI's performance is [41l strongly superhuman, or else at least optimal, across all cognitive domains].

Since there's multiple avenues we can imagine for how an AI could be sufficiently powerful along various dimensions, 'advanced agent' doesn't have a neat necessary-and-sufficient definition.  Similarly, some of the advanced agent properties are easier to formalize or pseudoformalize than others.

As an example:  Current machine learning algorithms are nowhere near the point that [2xd they'd try to resist if somebody pressed the off-switch].  *That* would happen given, e.g.:

- Enough [3nf big-picture strategic awareness] for the AI to know that it is a computer, that it has an off-switch, and that [7g2 if it is shut off its goals are less likely to be achieved].
- Widely applied [9h consequentialism], i.e. backward chaining from goals to actions; visualizing which actions lead to which futures and choosing actions leading to more [preferences preferred] futures, in general and across domains.

So the threshold at which you might need to start thinking about '[2xd shutdownability]' or '[2rg abortability]' or [45 corrigibility] as it relates to having an off-switch, is '[3nf big-picture strategic awareness]' plus '[9h cross-domain consequentialism]'.  These two cognitive thresholds can thus be termed 'advanced agent properties'.

The above reasoning also suggests e.g. that [-7vh] is an advanced agent property, because a general ability to learn new domains could lead the AI to understand that it has an off switch.

One reason to keep the term 'advanced' on an informal basis is that in an intuitive sense we want it to mean "AI we need to take seriously" in a way independent of particular architectures or accomplishments.  To the philosophy undergrad who 'proves' that AI can never be "truly intelligent" because it is "merely deterministic and mechanical", one possible reply is, "Look, if it's building a Dyson Sphere, I don't care if you define it as 'intelligent' or not."  Any particular advanced agent property should be understood in a background context of "If a computer program is doing X, it doesn't matter if we define that as 'intelligent' or 'general' or even as 'agenty', what matters is that it's doing X."  Likewise the notion of '[7g1 sufficiently advanced AI]' in general.

The goal of defining advanced agent properties is not to have neat definitions, but to correctly predict and carve at the natural joints for which cognitive thresholds in AI development could lead to which real-world abilities, corresponding to which [2l alignment issues].

An alignment issue may need to have been *already been solved* at the time an AI first acquires an advanced agent property; the notion is not that we are defining observational thresholds for society first needing to think about a problem.

# Summary of some advanced agent properties

Absolute-threshold properties (those which reflect cognitive thresholds irrespective of the human position on that same scale):

- **[9h Consequentialism],** or choosing actions/policies on the basis of their expected future consequences
 - Modeling the conditional relationship $\mathbb P(Y|X)$ and selecting an $X$ such that it leads to a high probability of $Y$ or high quantitative degree of $Y,$ is ceteris paribus a sufficient precondition for deploying [2vl] that lie within the effectively searchable range of $X.$
    - Note that selecting over a conditional relationship is potentially a property of many internal processes, not just the entire AI's top-level main loop, if the conditioned variable is being powerfully selected over a wide range.
 - **Cross-domain consequentialism** implies many different [7vf cognitive domains] potentially lying within the range of the $X$ being selected-on to achieve $Y.$
 - Trying to rule out particular instrumental strategies, in the presence of increasingly powerful consequentialism, would lead to the [-42] form of [-48] and subsequent [6q context-change disasters.]
- **[3nf]** is a world-model that includes strategically important general facts about the larger world, such as e.g. "I run on computing hardware" and "I stop running if my hardware is switched off" and "there is such a thing as the Internet and it connects to more computing hardware".
- **Psychological modeling of other agents** (not humans per se) potentially leads to:
 - Extrapolating that its programmers may present future obstacles to achieving its goals
    - This in turn leads to the host of problems accompanying [45 incorrigibility] as a [10g convergent strategy.]
 - [10f Trying to conceal facts about itself] from human operators
 - Being incentivized to engage in [-3cq].
 - [6v Mindcrime] if building models of reflective other agents, or itself.
 - Internally modeled adversaries breaking out of internal sandboxes.
 - [1fz] or other decision-theoretic adversaries.
- Substantial **[capability_gain capability gains]** relative to domains trained and verified previously.
 - E.g. this is the qualifying property for many [6q context-change disasters.]
- **[7vh]** is the most obvious route to an AI acquiring many of the capabilities above or below, especially if those capabilities were not initially or deliberately programmed into the AI.
- **Self-improvement** is another route that potentially leads to capabilities not previously present.  While some hypotheses say that self-improvement is likely to require basic general intelligence, this is not a known fact and the two advanced properties are conceptually distinct.
- **Programming** or **computer science** capabilities are a route potentially leading to self-improvement, and may also enable [-3cq].
- Turing-general cognitive elements (capable of representing large computer programs), subject to **sufficiently strong end-to-end optimization** (whether by the AI or by human-crafted clever algorithms running on 10,000 GPUs), may give rise to [2rc crystallized agent-like processes] within the AI.
 - E.g. natural selection, operating on chemical machinery constructible by DNA strings, optimized some DNA strings hard enough to spit out humans.
- **[6y Pivotal material capabilities]** such as quickly self-replicating infrastructure, strong mastery of biology, or molecular nanotechnology.
 - Whatever threshold level of domain-specific engineering acumen suffices to develop those capabilities, would therefore also qualify as an advanced-agent property.

Relative-threshold advanced agent properties (those whose key lines are related to various human levels of capability):

- **[9f]** is when we can't effectively imagine or search the AI's space of policy options (within a [7vf domain]); the AI can do things we didn't think of (within a domain).
 - **[2j]** is when we don't know all the rules (within a domain) and might not recognize the AI's solution even if told about it in advance, like somebody in the 11th century looking at the blueprint for a 21st-century air conditioner.  This may also imply that we cannot readily put low upper bounds on the AI's possible degree of success.
 - **[9j Rich domains]** are more likely to have some rules or properties unknown to us, and hence be strongly uncontainable.
 - [9t].
 - Human psychology is a rich domain.
 - Superhuman performance in a rich domain strongly implies cognitive uncontainability because of [1c0].
- **Realistic psychological modeling** potentially leads to:
 - Guessing which results and properties the human operators expect to see, or would arrive at AI-desired beliefs upon seeing, and [10f arranging to exhibit those results or properties].
 - Psychologically manipulating the operators or programmers
 - Psychologically manipulating other humans in the outside world
 - More probable [6v mindcrime]
 - (Note that an AI trying to develop realistic psychological models of humans is, by implication, trying to develop internal parts that can deploy *all* human capabilities.)
- **Rapid [capability_gain capability gains]** relative to human abilities to react to them, or to learn about them and develop responses to them, may cause more than one [-6q] to happen a time.
 - The ability to usefully **scale onto more hardware** with good returns on cognitive reinvestment would potentially lead to such gains.
 - **Hardware overhang** describes a situation where the initial stages of a less developed AI are boosted using vast amounts of computing hardware that may then be used more efficiently later.
 - [5b3 Limited AGIs] may have **capability overhangs** if their limitations break or are removed.
- **[7mt Strongly superhuman]** capabilities in psychological or material domains could enable an AI to win a competitive conflict despite starting from a position of great material disadvantage.
 - E.g., much as a superhuman Go player might win against the world's best human Go player even with the human given a two-stone advantage, a sufficiently powerful AI might talk its way out of an [6z AI box] despite restricted communications channels, eat the stock market in a month starting from $1000, win against the world's combined military forces given a protein synthesizer and a 72-hour head start, etcetera.
- [6s] relative to human civilization is a sufficient condition (though not necessary) for an AI to...
 - Deploy at least any tactic a human can think of.
 - Anticipate any tactic a human has thought of.
 - See the human-visible logic of a convergent instrumental strategy.
 - Find any humanly visible [43g weird alternative] to some hoped-for logic of cooperation.
 - Have any advanced agent property for which a human would qualify.
- **[41l General superintelligence]** would lead to strongly superhuman performance in many domains, human-relative efficiency in every domain, and possession of all other listed advanced-agent properties.
 - Compounding returns on **cognitive reinvestment** are the qualifying condition for an [-428] that might arrive at superintelligence on a short timescale.

# Discussions of some advanced agent properties

## Human psychological modeling

Sufficiently sophisticated models and predictions of human minds potentially leads to:

- Getting sufficiently good at human psychology to realize the humans want/expect a particular kind of behavior, and will modify the AI's preferences or try to stop the AI's growth if the humans realize the AI will not engage in that type of behavior later.  This creates an instrumental incentive for [10f programmer deception] or [3cq cognitive steganography].
- Being able to psychologically and socially manipulate humans in general, as a real-world capability.
- Being at risk for [6v mindcrime].

A [102 behaviorist] AI is one with reduced capability in this domain.

## Cross-domain, real-world [9h consequentialism]

Probably requires *generality* (see below).  To grasp a concept like "If I escape from this computer by [hacking my RAM accesses to imitate a cellphone signal](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf), I'll be able to secretly escape onto the Internet and have more computing power", an agent needs to grasp the relation between its internal RAM accesses, and a certain kind of cellphone signal, and the fact that there are cellphones out there in the world, and the cellphones are connected to the Internet, and that the Internet has computing resources that will be useful to it, and that the Internet also contains other non-AI agents that will try to stop it from obtaining those resources if the AI does so in a detectable way.

Contrasting this to non-primate animals where, e.g., a bee knows how to make a hive and a beaver knows how to make a dam, but neither can look at the other and figure out how to build a stronger dam with honeycomb structure.  Current, 'narrow' AIs are like the bee or the beaver; they can play chess or Go, or even learn a variety of Atari games by being exposed to them with minimal setup, but they can't learn about RAM, cellphones, the Internet, Internet security, or why being run on more computers makes them smarter; and they can't relate all these domains to each other and do strategic reasoning across them.

So compared to a bee or a beaver, one shot at describing the potent 'advanced' property would be *cross-domain real-world consequentialism*.  To get to a desired Z, the AI can mentally chain backwards to modeling W, which causes X, which causes Y, which causes Z; even though W, X, Y, and Z are all in different domains and require different bodies of knowledge to grasp.

## Grasping the [3nf big picture]

Many dangerous-seeming [2vl convergent instrumental strategies] pass through what we might call a rough understanding of the 'big picture'; there's a big environment out there, the programmers have power over the AI, the programmers can modify the AI's utility function, future attainments of the AI's goals are dependent on the AI's continued existence with its current utility function.

It might be possible to develop a very rough grasp of this bigger picture, sufficiently so to motivate instrumental strategies, in advance of being able to model things like cellphones and Internet security.  Thus, "roughly grasping the bigger picture" may be worth conceptually distinguishing from "being good at doing consequentialism across real-world things" or "having a detailed grasp on programmer psychology".

## [6y Pivotal] material capabilities

An AI that can crack the [protein structure prediction problem](https://en.wikipedia.org/wiki/Protein_structure_prediction) (which [seems speed-uppable by human intelligence](https://en.wikipedia.org/wiki/Foldit)); invert the model to solve the protein design problem (which may select on strong predictable folds, rather than needing to predict natural folds); and solve engineering problems well enough to bootstrap to molecular nanotechnology; is already possessed of potentially [6y pivotal] capabilities regardless of its other cognitive performance levels.

Other material domains besides nanotechnology might be [6y pivotal].  E.g., self-replicating ordinary manufacturing could potentially be pivotal given enough lead time; molecular nanotechnology is distinguished by its small timescale of mechanical operations and by the world containing an infinite stock of perfectly machined spare parts (aka atoms).  Any form of cognitive adeptness that can lead up to *rapid infrastructure* or other ways of quickly gaining a decisive real-world technological advantage would qualify.

## Rapid capability gain

If the AI's thought processes and algorithms scale well, and it's running on resources much smaller than those which humans can obtain for it, or the AI has a grasp on Internet security sufficient to obtain its own computing power on a much larger scale, then this potentially implies [ rapid capability gain] and associated [6q context changes].  Similarly if the humans programming the AI are pushing forward the efficiency of the algorithms along a relatively rapid curve.

In other words, if an AI is currently being improved-on swiftly, or if it has improved significantly as more hardware is added and has the potential capacity for orders of magnitude more computing power to be added, then we can potentially expect rapid capability gains in the future.  This makes [6q context disasters] more likely and is a good reason to start future-proofing the [2l safety properties] early on.

## Cognitive uncontainability

On complex tractable problems, especially those that involve real-world rich problems, a human will not be able to [9f cognitively 'contain'] the space of possibilities searched by an advanced agent; the agent will consider some possibilities (or classes of possibilities) that the human did not think of.

The key premise is the 'richness' of the problem space, i.e., there is a fitness landscape on which adding more computing power will yield improvements (large or small) relative to the current best solution.  Tic-tac-toe is not a rich landscape because it is fully explorable (unless we are considering the real-world problem "tic-tac-toe against a human player" who might be subornable, distractable, etc.)  A computationally intractable problem whose fitness landscape looks like a computationally inaccessible peak surrounded by a perfectly flat valley is also not 'rich' in this sense, and an advanced agent might not be able to achieve a relevantly better outcome than a human.

The 'cognitive uncontainability' term in the definition is meant to imply:

- [9g Vingean unpredictability].
- Creativity that goes outside all but the most abstract boxes we imagine (on rich problems).
- The expectation that we will be surprised by the strategies the superintelligence comes up with because its best solution was one we didn't consider.

Particularly surprising solutions might be yielded if the superintelligence has acquired domain knowledge we lack.  In this case the agent's strategy search might go outside causal events we know how to model, and the solution might be one that we wouldn't have recognized in advance as a solution.  This is [2j].

In intuitive terms, this is meant to reflect, e.g., "What would have happened if the 10th century had tried to use their understanding of the world and their own thinking abilities to upper-bound the technological capabilities of the 20th century?"

## Other properties

*(Work in progress)* [todo: fill out]

- [42g generality]
 - cross-domain [9h consequentialism]
 - learning of non-preprogrammed domains
     - learning of human-unknown facts
  - Turing-complete fact and policy learning
- dangerous domains
   - human modeling
     - social manipulation
     - realization of programmer deception incentive
     - anticipating human strategic responses
   - rapid infrastructure
- potential
  - self-improvement
  - suppressed potential
- [6s epistemic efficiency]
- [6s instrumental efficiency]
- [9f cognitive uncontainability]
    - operating in a rich domain
    - [9g Vingean unpredictability]
    - [2j strong cognitive uncontainability]
- improvement beyond well-tested phase (from any source of improvement)
- self-modification
 - code inspection
 - code modification
 - consequentialist programming
     - cognitive programming
 - cognitive capability goals (being pursued effectively)
- speed surpassing human reaction times in some interesting domain
  - socially, organizationally, individually, materially

[todo: write out a set of final dangerous abilities use/cases and then link up the cognitive abilities with which potentially dangerous scenarios they create.]