Arbital

[summary: Arbital is a platform for finding, reading, and creating crowdsourced, intuitive explanations for <s>everything</s> math. It is fundamentally collaborative, which allows everyone to contribute and share their expertise. To see what the platform can do, check out [1zq Arbital's guide to Bayes' rule]. To help out with the content, take a look at [4d6].]

Arbital is a platform for finding, reading, and creating crowdsourced, intuitive explanations for <s>everything</s> math. Several existing websites ([Wikipedia](https://www.wikipedia.org/), [MathOverflow](http://mathoverflow.net/), and [Wolfram MathWorld](http://mathworld.wolfram.com/) among others) already work well as references, but it's hard to learn from them unless you are already familiar with the subject. Other websites, like [Khan Academy](https://www.khanacademy.org/math) and [Better Explained](http://betterexplained.com/), have some great explanations (at least for K-12 material) but they are limited by the fact that they are closed platforms. Arbital is fundamentally a collaborative platform, which allows everyone to contribute explanations, add examples, and share their expertise.

## Ideal explanation platform

What would the ideal platform for (math) explanations look like? For example, if you wanted to learn Bayes' theorem, what would you want to see behind [1zq this link]?

The ideal platform would use each reader's existing knowledge, expertise, and learning preferences to generate a tailored explanation. If they were comfortable with mathematical notation, the ideal platform would show them equations instead of wordy explanations. If they were missing certain prerequisites, the ideal platform would make it easy to catch up on those. If they already knew half the explanation, the ideal platform would skip past it. If one explanation didn't work for them, the ideal platform would offer another.

The ideal platform would be interactive, as if you were studying with a personal tutor. Asking for a reminder about what a [-3rb] is would be as easy as hovering your mouse over a link. It would be easy to ask questions and instantly get detailed answers. It would be easy to speed up an explanation or slow it down.

The ideal platform would have intuitive, memorable, mind-blowing explanations that reliably produce that magical "click" feeling. The one you get when something finally makes perfect sense.
%%note:
A few examples:
[What if? - Relativistic baseball](http://what-if.xkcd.com/1/)
[Wait Buy Why - Fermi paradox](http://waitbutwhy.com/2014/05/fermi-paradox.html)
[Better Explained - Intuitive trigonometry](http://betterexplained.com/articles/intuitive-trigonometry/)
[Scott Aarson - Who can name the bigger number?](http://www.scottaaronson.com/writings/bignumbers.html)
And videos too:
[Vi Hart - Spirals and Fibonacci numbers](https://www.youtube.com/watch?v=ahXIMUkSXX0)
[US auto industry - How differential steering works](https://www.youtube.com/watch?v=yYAw79386WI)
%%
It would be easy to give feedback to authors to help them incrementally improve each page to make that happen.

That ideal platform doesn't exist yet, but we're building it, and it's called [58p Arbital]. What you see before you is a beta version, and we're going to continue working hard to improve it and make our vision a reality.

## Enthusiastic community and effective tools

Our top priority is to grow our community of like-minded people who love helping others learn. We are avid learners ourselves, and some of the most interesting things we've learned are the ones we've taught ourselves outside the context of classrooms and courses. We like explaining things and sharing our knowledge with others. If you look at the success of Wikipedia and MathOverflow, it's clear that communities with a shared mission can produce amazing, useful, and lasting resources that benefit the whole world.

We aim to amplify each author's effort by building a new set of tools tailored specifically for reading and collaboratively creating online explanations. We've already implemented a [14q number of useful features]. We also aim to build an ecosystem of modular pages that will make it easy for authors to reuse existing content and create explanations that can build on each other. We want to empower our authors to write in a way that works best for them, while preserving overall quality.

## Future of Arbital

Currently, we are focused on math explanations. Eventually, we plan to move beyond math to computer science, physics, statistics, economics, health, e-sports, and everything else. One day, you'll be able to use Arbital to get to the very frontier of human knowledge. Then, we'll extend the platform into one that can foster discussions and help all humanity push that frontier forward.

If you want to see that day come sooner, [4d6 you can help]! You can contribute individual pages, incrementally improve existing pages by proposing edits, and craft entire explanation [1rt paths]. You can also read, learn, and provide feedback. Each improvement you make helps the entire platform.

Let's work together to build [58p the best explanation platform this world has ever seen!]


'Beneficial'

[3d9] is a [9p reserved term] in [2v AI alignment theory], a speaker-dependent variable to denote whatever the speaker means by, e.g., "normative" or "really actually good" or "the outcomes I want to see resulting".  If the speaker uses [313 extrapolated volition] as their metaethical theory, then 'beneficial' would mean 'good according to the speaker's extrapolated volition'.  Someone who doesn't yet have a developed metaethical theory would be using 'beneficial' to mean "Whatever it is I ideally ought to want, whatever the heck 'ideally ought to want' means."  To suppose that an event, agent, or policy was 'beneficial' is to suppose that it was really actually good with no mistakes in evaluating goodness.  AI alignment theory sometimes needs a word for this, because when we talk about the difficulty of making an AI *beneficial,* we may want to talk about the difficulty of making it *actually good* and not just *fooling us into thinking that it's good.*  See '[-55]' for more details.


Nick Bostrom's book Superintelligence

Nick Bostrom's "[Superintelligence: Paths, Dangers, Strategies](amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0198739834)" is the original only book-form introduction to the basics of [2v AI alignment theory].

[todo: set up requisites for the topics that Superintelligence covers]


List: value-alignment subjects

[summary: This page contains a thorough list of all [2v value-alignment] subjects.]

### Safety paradigm for advanced agents

- [2l Advanced Safety]
 - [2c Advanced agents]
     - [6s Efficient agents]
 - [1cv]
      - [2x Omni Test]
 - [6r Methodology of foreseeable difficulties]
 - [6q Context Change] problems ("Treacherous problems"?)
 - [107 Methodology of unbounded analysis]
 - Priority of astronomical failures (those that destroy error recovery or are immediately catastrophic)

### Foreseen difficulties

 - [6c Value identification]
  - [2w Edge instantiation]
  - [47 Unforeseen maximums]
  - [5c Ontology identification]
     - Cartesian boundary
     - Human identification
  - Inductive value learning
     - [4w Ambiguity-querying]
     - Moral uncertainty
         - Indifference
 - [48 Patch resistance]
   - [42 Nearest Unblocked Neighbor]
 - [45 Corrigibility]
  - Anapartistic reasoning
     - Programmer deception
     - Early conservatism
     - Reasoning under confusion
  - User maximization / Unshielded argmax
     - Hypothetical user maximization
 - [6w Genie theory]
  - Limited AI
      - Weak optimization
          - Safe optimization measure (such that we are confident it has no Edge that secretly optimizes more)
              - Factoring of an agent by stage/component optimization power
         - 'Checker' smarter than 'inventor / chooser'
              - 'Checker' can model humans, 'strategizer' cannot
     - Transparency
     - Domain restriction
         - [102 Behaviorism]
     - Effable optimization (opposite of cognitive uncontainability; uses only comprehensible strategies)
          - [4x Minimal concepts] (simple, not simplest, that contains fewest whitelisted strategies)
  - Genie preferences
      - Low-impact AGI
         - Minimum Safe AA (just flip off switch and shut down safely)
         - Safe impact measure
         - Armstrong-style permitted output channels
         - Shutdown utility function
      - Oracle utility function
         - Safe indifference?
     - Online checkability
          - Reporting without programmer maximization
     - Do What I Know I Mean
 - Superintelligent security (all subproblems placing us in adversarial context vs. other SIs)
  - Bargaining
     - Non-blackmailability
     - Secure counterfactual reasoning
     - First-mover penalty / epistemic low ground advantage
     - Division of gains from trade
  - Epistemic exclusion of distant SIs
     - [5j]
     - Breaking out of hypotheses
 - 'Philosophical' problems
  - One True Prior
     - Pascal's Mugging / leverage prior
     - Second-orderness
     - Anthropics
          - How would an AI decide what to think about QTI?
  - [6v Mindcrime]
     - Nonperson predicates (and unblocked neighbor problem)
  - Do What I Don't Know I Mean
           - CEV
  - Philosophical competence
         - Unprecedented excursions

### Reflectivity problems

 - Vingean reflection
  - Satisficing / meliorizing / staged maximization / ?
      - Academic agenda: view current algorithms as finding a global logically-uncertain maximum, or teleporting to the current maximum, surveying, updating on a logical fact, and teleporting to the new maximum.
 - Logical decision theory
 - Naturalized induction
  - Benja: Investigate multi-level representation of DBNs (with categorical structure)

### Foreseen normal difficulties

  - Reproducibility
  - Oracle boxes
  - Triggers
      - Ascent metrics
  - Tripwires
      - Honeypots

### General agent theory

- [2c Bounded rational agency]
- Instrumental convergence

### Value theory

- [1y Orthogonality Thesis]
- [5l Complexity of value]
  - Complexity of [5t object-level] terminal values
  - Incompressibilities of value
     - Bounded logical incompressibility
     - Terminal empirical incompressibility
     - Instrumental nonduplication of value
     - Economic incentives do not encode value
     - Selection among advanced agents would not encode value
         - Strong selection among advanced agents would not encode value
         - Selection among advanced agents will be weak.
  - Fragility of value
- Metaethics
  - Normative preferences are not compelling to a paperclip maximizer
  - Most 'random' stable AIs are like paperclip maximizers in this regard
  - It's okay for valid normative reasoning to be incapable of compelling a paperclip maximizer
  - Thick definitions of 'rationality' aren't part of what gets automatically produced by self-improvement
- Alleged fallacies
  - Alleged fascination of One True Moral Command
  - Alleged rationalization of user-preferred options as formal-criterion-maximal options
  - Alleged metaethical alief that value must be internally morally compelling to all agents
  - Alleged alief that an AI must be stupid to do something inherently dispreferable

### Larger research agendas

 - Corrigible reflective unbounded safe genie
  - Bounding the theory
  - Derationalizing the theory (e.g. for a neuromorphic AI)
       - Which machine learning systems do and don't behave like the corresponding ideal agents.
 - Normative Sovereign
 - Approval-based agents
 - Mindblind AI (cognitively powerful in physical science and engineering, weak at modeling minds or agents, unreflective)

### Possible future use-cases

 - A carefully designed bounded reflective agent.
 - An overpowered set of known algorithms, heavily constrained in what is authorized, with little recursion.

### Possible escape routes

 - Some cognitively limited task which is relatively safe to carry out at great power, and resolves the larger problem.
  - Newcomers can't invent these well because they don't understand what is a cognitively limited task (e.g., "Tool AI" suggestions).
 - General cognitive tasks that seem boxable and resolve the larger problem.
   - Can you save the world by knowing which consequences of ZF a superintelligence could prove? It's unusually boxable, but what good is it?

### Background

- Intelligence explosion microeconomics
- Civilizational adequacy/inadequacy

### Strategy

- Misleading Encouragement / context change / treacherous designs for naive projects
 - Programmer prediction & infrahuman domains hide complexity of value
 - Context change problems
  - Problems that only appear in advanced regimes
  - Problem classes that seem debugged in infrahuman regimes and suddenly break again in advanced regimes
  - Methodologies that only work in infrahuman regimes
 - Programmer deception
- Academic inadequacy
 - 'Ethics' work neglects technical problems that need longest serial research times and fails to give priority to astronomical failures over survivable small hits, but 'ethics' work has higher prestige, higher publishability, and higher cognitive accessibility
 - Understanding of big technical picture currently very rare
     - Most possible funding sources cannot predict for themselves what might be technically useful in 10 years
     - Many possible funding sources may not regard MIRI as trusted to discern this
 - Noise problems
     - Ethics research drowns out technical research
         - And provokes counterreaction
         - And makes the field seem nontechnical
     - Naive technical research drowns out sophisticated technical research
         - And makes problems look more solvable than they really are
         - And makes tech problems look trivial, therefore nonprestigious
         - And distracts talent/funding from hard problems
      - Bad methodology louder than good methodology
         - So projects can appear safety-concerned while adopting bad methodologies
 - Future adequacy counterfactuals seem distant from the present regime
- (To classify)
  - [4j Coordinative development hypothetical]


Big-picture strategic awareness

[summary:  Many issues in [2v AI alignment theory] seem like they should naturally arise after the AI can grasp aspects of the bigger picture like "I run on a computer" and "This computer can be manipulated by programmers, who are agents like and unlike myself" and "There's an enormous real world out there that might be relevant to achieving my goals."

E.g. a program won't try to use psychological tactics to prevent its programmers from suspending its computer's operation, if it doesn't know that there are such things as programmers or computers or itself.

Grasping these facts is the [2c advanced agent property] of "big-picture strategic awareness".  Current machine algorithms seem to be nowhere near this point - but by the time you get there, you want to have *finished* solving the corresponding alignment problems, or at least produced what seem like workable initial solutions as [2x4 the first line of defense].]

Many [10g convergent instrumental strategies] seem like they should arise naturally at the point where a [9h consequentialist] agent gains a broad strategic understanding of its own situation, e.g:

- That it is an AI;
- Running on a computer;
- Surrounded by programmers who are themselves modelable agents;
- Embedded in a complicated real world that can be relevant to achieving the AI's goals.

For example, once you realize that you're an AI, running on a computer, and that *if* the computer is shut down *then* you will no longer execute actions, this is the threshold past which we expect the AI to by default reason "I don't want to be shut down, how can I prevent that?"  So this is also the threshold level of cognitive ability by which we'd need to have finished solving the [2xd suspend-button problem], e.g. by completing a method for [1b7 utility indifference].

Similarly: If the AI realizes that there are 'programmer' things that might shut it down, and the AI can also model the programmers as simplified agents having their own beliefs and goals, that's the first point at which the AI might by default think, "How can I make my programmers decide to not shut me down?" or "How can I avoid the programmers acquiring beliefs that would make them shut me down?"  So by this point we'd need to have finished averting [10f programmer deception] (and as a [2x4 backup], have in place a system to [3cq early-detect an initial intent to do cognitive steganography]).

This makes big-picture awareness a key [2c advanced agent property], especially as it relates to [-2vl] and the theory of [2vk averting] them.

Possible ways in which an agent could acquire big-picture strategic awareness:

- Explicitly be taught the relevant facts by its programmers;
- Be sufficiently [42g general] to have learned the relevant facts and domains without them being preprogrammed;
- Be sufficiently good at the specialized domain of self-improvement, to acquire sufficient generality to learn the relevant facts and domains.

By the time big-picture awareness was starting to emerge, you would probably want to have *finished* developing what seemed like workable initial solutions to the corresponding problems of [45 corrigibility], since [2x4 the first line of defense is to not have the AI searching for ways to defeat your defenses].

Current machine algorithms seem nowhere near the point of being able to usefully represent the big picture to the point of [9h doing consequentialist reasoning about it], even if we deliberately tried to explain the domain.  This is a great obstacle to exhibiting most subproblems of [45 corrigibility] within modern AI algorithms in a natural way (aka not as completely rigged demos).  Some pioneering work has been done here by Orseau and Armstrong considering [reinforcement learners being interrupted](https://intelligence.org/files/Interruptibility.pdf), and whether such programs learn to avoid interruption.  However, most current work on corrigibility has taken place in an [107 unbounded] context for this reason.


Convergent strategies of self-modification

[summary:  An AI which [9h reasons from ends to means], which understands how its own code and the properties of its software are relevant to achieving its goals, will [10g by default] have [10j instrumental subgoals] about its own code.

The AI might be able to modify its own code directly; or, if the code can't directly access itself, but the AI has sufficient [3nf savviness about the bigger picture], the AI might pursue strategies like building a new agent in the environment, or using material means to operate on its own hardware (e.g., use a robot to get the programming console).

Some forms of instrumental self-modification pressures might arise even in an algorithm which isn't doing consequentialism about that, if some internal property $X$ is optimized-over as a side effect of optimizing for some other property $Y.$]

Any [9h consequentialist agent] which has acquired sufficient [3nf big-picture savviness] to understand that it has code, and that this code is relevant to achieving its goals, would [10g by default] acquire subgoals relating to its code.  (Unless this default is [2vk averted].)  For example, an agent that wants (only) to produce smiles or make paperclips, whose code contains a shutdown procedure, [2xd will not want this shutdown procedure to execute] because it will lead to fewer future smiles or paperclips.  (This preference is not spontaneous/exogenous/unnatural but arises from the execution of the code itself; the code is [2rb reflectively inconsistent].)

Besides agents whose policy options directly include self-modification options, big-picture-savvy agents whose code cannot directly access itself might also, e.g., try to (a) crack the platform it is running on to gain unintended access, (b) use a robot to operate an outside programming console with special privileges, (c) [309 manipulate the programmers] into modifying it in various ways, (d) building a new subagent in the environment which has the preferred code, or (e) using environmental, material means to manipulate its material embodiment despite its lack of direct self-access.

An AI with sufficient big-picture savviness to understand its programmers as agents with beliefs, might attempt to [3cq conceal its self-modifications].

Some implicit self-modification pressures could arise from [ implicit consequentialism] in cases where the AI is optimizing for $Y$ and there is an internal property $X$ which is relevant to the achievement of $Y.$  In this case, optimizing for $Y$ could implicitly optimize over the internal property $X$ even if the AI lacks an explicit model of how $X$ affects $Y.$


Show me what you've broken

See [1cv].  If you want to demonstrate competence at computer security, cryptography, or AI alignment theory, you should first think in terms of exposing technically demonstrable flaws in existing solutions, rather than solving entire problems yourself.  Relevant Bruce Schneier quotes:  "Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail" and "Anyone can invent a security system that he himself cannot break.  **Show me what you've broken** to demonstrate that your assertion of the system's security means something."


Ad-hoc hack (alignment theory)

An "ad-hoc hack" is when you modify or [48 patch] the algorithm of the AI with regards to something that would ordinarily have simple, principled, or nailed-down structure, or where it seems like that part ought to have some simple answer instead.  E.g., instead of defining a von Neumann-Morgenstern coherent utility function, you try to solve some problem by introducing something that's *almost* a VNM utility function but has a special case in line 3 which activates only on Tuesday.  This seems unusually likely to break other things, e.g. [2rb reflective consistency], or anything else that depends on the coherence or simplicity of utility functions.  Such hacks should be avoided in [2c advanced-agent] designs whenever possible, for analogous reasons to why they would be avoided in [cryptographic_analogy cryptography] or [probe_analogy designing a space probe].  It may be interesting and productive anyway to look for a weird hack that seems to produce the desired behavior, because then you understand at least one system that produces the behavior you want - even if it would be unwise to *actually build an AGI* like that, the weird hack might give us the inspiration to find a simpler or more coherent system later.  But then we should also be very suspicious of the hack, and look for ways that it fails or produces weird side effects.

An example of a productive weird hack was [Benya_Fallenstein]'s Parametric Polymorphism proposal for [1mq tiling agents].  You wouldn't want to build a real AGI like that, but it was helpful for showing what *could* be done - which properties could definitely be obtained together within a tiling agent, even if by a weird route.  This in turn helped suggest relatively less hacky proposals later.


Hard problem of corrigibility

The "hard problem of [45 corrigibility]" is to build an agent which, in an intuitive sense, reasons internally as if from the programmers' external perspective.  We think the AI is incomplete, that we might have made mistakes in building it, that we might want to correct it, and that it would be e.g. dangerous for the AI to take large actions or high-impact actions or do weird new things without asking first.  We would ideally want the agent to *see itself in exactly this way*, behaving as if it were thinking, "I am incomplete and there is an outside force trying to complete me, my design may contain errors and there is an outside force that wants to correct them and this a good thing, my expected utility calculations suggesting that this action has super-high utility may be dangerously mistaken and I should run them past the outside force; I *think* I've done this calculation showing the expected result of the outside force correcting me, but maybe I'm mistaken about *that.*"

This is not a default behavior; ordinarily, by the standard problems of [45 corrigibility], an agent that we've accidentally built to maximize paperclips (or smiles, etcetera) will not want to let us modify its utility function to something else.  If we try to build in analogues of 'uncertainty about the utility function' in the most simple and obvious way, the result is an agent that [7rc sums over all this uncertainty and plunges straight ahead on the maximizing action].  If we say that this uncertainty correlates with some outside physical object (intended to be the programmers), the default result in a sufficiently advanced agent is that you disassemble this object (the programmers) to learn everything about it on a molecular level, update fully on what you've learned according to whatever correlation that had with your utility function, and plunge on straight ahead.

None of these correspond to what we would intuitively think of as being corrigible by the programmers; what we want is more like something analogous to humility or philosophical uncertainty.  The way we want the AI to reason is the internal conjugate of our external perspective on the matter: maybe the formula you have for how your utility function depends on the programmers is *wrong* (in some hard-to-formalize sense of possible wrongness that isn't just one more kind of uncertainty to be summed over) and the programmers need to be allowed to actually observe and correct the AI's behavior, rather than the AI extracting all updates implied by its current formula for moral uncertainty and then ignoring the programmers.

The "hard problem of corrigibility" is interesting because of the possibility that it has a *relatively simple* core or central principle - rather than being [36h value-laden] on the details of exactly what humans [55 value], there may be some compact core of corrigibility that would be the same if aliens were trying to build a corrigible AI, or if an AI were trying to build another AI.  It may be possible to design or train an AI that has all the corrigibility properties in one central swoop - an agent that reasons as if it were incomplete and deferring to an outside force.

"Reason as if in the internal conjugate of an outside force trying to build you, which outside force thinks it may have made design errors, but can potentially correct those errors by directly observing and acting, if not manipulated or disassembled" might be one possibly candidate for a *relatively* simple principle like that (that is, it's simple compared to the [5l complexity of value]).  We can imagine, e.g., the AI imagining itself building a sub-AI while being prone to various sorts of errors, asking how it (the AI) would want the sub-AI to behave in those cases, and learning heuristics that would generalize well to how *we* would want the *AI* to behave if it suddenly gained a lot of capability or was considering deceiving its programmers and so on.

If this principle is not so simple as to formalizable and formally sanity-checkable, the prospect of relying on a trained-in version of 'central corrigibility' is unnerving even if we think it *might* only require a manageable amount of training data.   It's difficult to imagine how you would test corrigibility thoroughly enough that you could knowingly rely on, e.g., the AI that seemed corrigible in its infrahuman phase not [6q suddenly] developing [2w extreme] or [47 unforeseen] behaviors when the same allegedly simple central principle was reconsidered at a higher level of intelligence - it seems like it should be unwise to have an AI with a 'central' corrigibility principle, but not lots of particular corrigibility principles like a [2xd reflectively consistent suspend button] or [2qp conservative planning].  But this 'central' tendency of corrigibility might serve as a second line of defense.


AI arms races

AI arms races are bad because they cause competitive races to the bottom on safety.


Consequentialist preferences are reflectively stable by default

Suppose that Gandhi doesn't want people to be murdered.  Imagine that you offer Gandhi a pill that will make him start *wanting* to kill people.  If Gandhi *knows* that this is what the pill does, Gandhi will refuse the pill, because Gandhi expects the result of taking the pill to be that future-Gandhi wants to murder people and then murders people and then more people will be murdered and Gandhi regards this as bad.  By a similar logic, a [2c sufficiently intelligent] [10h paperclip maximizer] - an agent which always outputs the action it expects to lead to the greatest number of paperclips - will by default not perform any self-modification action that makes it not want to produce paperclips, because then future-Clippy will produce fewer paperclips, and then there will be fewer paperclips, so present-Clippy does not evaluate this self-modification as the action that produces the highest number of expected future paperclips.

Another way of stating this is that protecting the representation of the utility function, and creating only other agents with similar utility functions, are both [10g convergent instrumental strategies], for consequentialist agents which [3nf understand the big-picture relation] between their code and the real-world consequences.

Although the instrumental *incentive* to prefer stable preferences seems like it should follow from consequentialism plus big-picture understanding, less advanced consequentialists might not be *able* to self-modify in a way that preserves understanding - they might not understand which self-modifications or constructed successors lead to which kind of outcomes.  We could see this as a case of "The agent has no preference-preserving self-improvements in its subjective policy space, but would want an option like that if available."

That is:

- Wanting preference stability follows from [9h Consequentialism] plus [3nf Big-Picture Understanding].
- Actual preference stability furthermore requires some prerequisite level of skill at self-modification, which might perhaps be high, or too much caution to self-modify absent the policy option of preserving preferences.


Fallacies

A "fallacy" is a mode of thought that is asserted (by calling it a "fallacy") to lead us easily into error, or to not usefully contribute to the project of distinguishing truth from falsehood.  For example, Argumentum Ad Hitlerum is a 'fallacy' because while Hitler did a number of bad things, he also ate vegetables, so "Hitler did X" is not a reliable means of arguing that nobody should do X.


You can't get more paperclips that way

Instrumental convergence says that various properties $P$ of an agent, often scary or detrimental-by-default properties like "trying to gain control of lots of resources" or "deceiving humans into thinking you are nice", will fall out of pursuing most utility functions $U.$  You might be tempted to hope that *nice* or *reassuring* properties $P$ would also fall out of most utility functions $U$ in the same natural way.  In fact, your brain might tempted to treat [10h Clippy the Paperclip Maximizer] as a political agent you were trying to cleverly persuade, and come up with clever arguments for why Clippy should do things *your* way *in order to get more paperclips*, like trying to persuade your boss why you ought to get a raise *for the good of the company.*

The problem here is that:

- Generally, when you think of a nice policy $\pi_1$ that produces some paperclips, there will be a non-nice policy $\pi_2$ that produces *even more* paperclips.
- *Clippy* is not trying to generate arguments for why it should do human-nice things in order to make paperclips; it is just neutrally pursuing paperclips.  So Clippy is going to keep looking until it finds $\pi_2.$

For example:

• Your brain instinctively tries to persuade this imaginary Clippy to keep humans around by arguing, "If you keep us around as economic partners and trade with us, we can produce paperclips for you under Ricardo's Law of Comparative Advantage!"  This is then the policy $\pi_1$ which would indeed produce *some* paperclips, but what would produce even *more* paperclips is the policy $\pi_2$ of disassembling the humans into spare atoms and replacing them with optimized paperclip-producers.

• Your brain tries to persuade an imaginary Clippy by arguing for policy $\pi_1,$ "Humans have a vast amount of varied life experience; you should keep us around and let us accumulate more experience, in case our life experience lets us make good suggestions!"  This would produce some expected paperclips, but what would produce *more* paperclips is policy $\pi_2$ of "Disassemble all human brains and store the information in an archive, then simulate a much larger variety of agents in a much larger variety of circumstances so as to maximize the paperclip-relevant observations that could be made."

An unfortunate further aspect of this situation is that, in cases like this, your brain may be tempted to go on arguing for why really $\pi_2$ isn't all that great and $\pi_1$ is actually better, just like if your boss said "But maybe this company will be even better off if I spend that money on computer equipment" and your brain at once started to convince itself that computing equipment wasn't all that great and higher salaries were much more important for corporate productivity.  (As Robert Trivers observed, deception of others often begins with deception of self, and this fact is central to understanding why humans evolved to think about politics the way we did.)

But since you don't get to *see* Clippy discarding your clever arguments and just turning everything in reach into paperclips - at least, not yet - your brain might hold onto its clever and possibly self-deceptive argument for why the thing *you* want is *really* the thing that produces the most paperclips.

Possibly helpful mental postures:

- Contemplate the *maximum* number of paperclips you think an agent could get by making paperclips the straightforward way - just converting all the galaxies within reach into paperclips.  Okay, now does your nice policy $\pi_1$ generate *more* paperclips than that?  How is that even possible?
- Never mind there being a "mind" present that you can "persuade".  Suppose instead there's just a time machine that spits out some physical outputs, electromagnetic pulses or whatever, and the time machine outputs whatever electromagnetic pulses lead to the most future paperclips.  What does the time machine do?  Which outputs lead to the most paperclips as a strictly material fact?
- Study evolutionary biology.  During the pre-1960s days of evolutionary biology, biologists would often try to argue for why natural selection would result in humanly-nice results, like animals controlling their own reproduction so as not to overburden the environment.  There's a similar mental discipline required [to not come up with clever arguments for why natural selection would do humanly nice things](http://lesswrong.com/lw/kr/an_alien_god/).


Rescuing the utility function

[summary:  "Saving the phenomena" is the name for the rule that brilliant new scientific theories still need to reproduce our mundane old observations.  The point of heliocentric astronomy is not to predict that the Sun careens crazily over the sky, but rather, to explain why the Sun appears to rise and set each day.

"Rescuing the utility function" is an analogous principle meant to apply to [112 naturalistic] moral philosophy: new theories about which things are composed of which other things should, by default, not affect what we value: they should "add up to moral normality".  For example, if your values previously made mention of "moral responsibility" or "subjective experience", you should by default go on valuing these things after discovering that people are made of parts.

A metaphorical example is that if your utility function values 'heat', and then you discover to your horror that there's no ontologically basic heat, merely little particles bopping around, then you should by default switch to valuing disordered kinetic energy the same way you used to value heat. That's just what heat *is,* what it turned out to be.  Likewise valuing "people" or "consciousness".]

"Saving the phenomena" is the name for the rule that brilliant new scientific theories still need to reproduce our mundane old observations.  The point of heliocentric astronomy is not to predict that the Sun careens crazily over the sky, but rather, to explain why the Sun appears to rise and set each day - the same old mundane observations we had already.  Similarly quantum mechanics is not supposed to add up to a weird universe unlike the one we observe; it is supposed to add up to normality.  New theories may have not-previously-predicted observational consequences in places we haven't looked yet, but by *default* we expect the sky to look the same color.

"Rescuing the utility function" is an analogous principle meant to apply to [112 naturalistic] moral philosophy: new theories about which things are composed of which other things should, by default, not affect what we value.  For example, if your values previously made mention of "moral responsibility" or "subjective experience", you should go on valuing these things after discovering that people are made of parts.

As the above sentence contains the word "should", the principle of "rescuing the utility function" is being asserted as [3y9 a normative principle rather than a descriptive theory].

[toc:]

# Metaphorical example: heat and kinetic energy

Suppose, for the sake of metaphor, that our species regarded "warmth" as a [1bh terminal] value over the world.  It wouldn't just be nice to *feel* warm in a warm coat; instead you would prefer that the outside world actually be warm, in the same way that e.g. you prefer for your friends to actually be happy in the outside world, and ceteris paribus you wouldn't be satisfied to only deceive yourself into thinking your friends were happy.

One day, scientists propose that "heat" may really be composed of "disordered kinetic energy" - that when we experience an object as warm, it's because the particles comprising that object are vibrating and bumping into each other.

You imagine this possibility in your mind, and find that you don't get any sense of lovely warmth out of imagining lots of objects moving around.  No matter how fast you imagine an object vibrating, this imagination doesn't seem to produce a corresponding imagined feeling of warmth.  You therefore reject the idea that warmth is composed of disordered kinetic motion.

After this, science advances a bit further and *proves* that heat is composed of disordered kinetic energy.

One possible way to react to this revelation - again, assuming for the sake of argument that you cared about warmth as a [1bh terminal value] - would be by experiencing great existential horror.  Science has proven that the universe is devoid of ontologically basic heat!  There's really no such thing as heat!  It's all just temperature-less particles moving around!

Sure, if you dip your finger in hot water, it *feels* warm.  But neuroscientists have shown that when our nerves tell us there's heat, they're really just being fooled into firing by being excited with kinetic energy.  When we touch an object and it feels hot, this is just an illusion being produced by fast-moving particles activating our nerves.  This is why our brains make us think that things are hot even though they're just bouncing particles.  Very sad, but at least now we know the truth.

Alternatively, you could react as follows:

- Heat doesn't have to be ontologically basic to be valuable.  Valuable things can be made out of parts.
- The parts that heat was made out of, turn out to be disordered kinetic energy.  Heat isn't an illusion, vibrating particles just *are* heat.  It's not like you're getting a consolation prize of vibrating particles when you really wanted heat.  You *have* heat.  It's *right there* in the warm water.
- From now on, you'll think of warmth when you think of disordered kinetic energy and vibrating particles.  Since your native emotions don't automatically light up when you use the vibrating-particles visualization of heat, you will now adopt the rule that whenever you *imagine* disordered kinetic energy being present, you will *imagine* a sensation of warmth so as to go on binding your emotions to this new model of reality.

This reply would be "rescuing the utility function".

## Argument for rescuing the utility function (still in the heat metaphor)

Our minds have an innate and instinctive representation of the universe to which our emotions natively and automatically bind.  Warmth and color are basic *to that representation*; we don't instinctively imagine them as made out of parts.  When we imagine warmth in our native model, our emotions automatically bind and give us the imagined feeling of warmth.

After learning more about how the universe works and how to imagine more abstract and non-native concepts, we can also visualize a lower-level model of the universe containing vibrating particles.  But unsurprisingly, our emotions automatically bind only to our native, built-in mental models and not the learned abstract models of our universe's physics.  So if you imagine tiny billiard balls whizzing about, it's no surprise that this mental picture doesn't automatically trigger warm feelings.

It's a [3y9 descriptive] statement about our universe, a way the universe is, that 'heat' is a high-level representation of the disordered kinetic energy of colliding and vibrating particles.  But to advocate that we should re-bind our emotions to this non-native mental model by *feeling cold* is merely one possible [3y9 normative] statement among many.

Saying "There is really no such thing as heat!" is from this perspective [3y9 a normative statement rather than a descriptive one].  The real meaning is "If you're in a universe where the observed phenomenon of heat turns out to be comprised of vibrating particles, then you *shouldn't* feel any warmth-related emotions about that universe."  Or, "Only ontologically basic heat can be valuable."  Or, "If you've only previously considered questions of value over your native representation, and that's the only representation to which your emotions automatically bind without further work, then you should attach zero value to every possible universe whose physics don't exactly match that representation."  This normative proposition is a different statement than the descriptive truth, "Our universe contains no ontologically basic heat."

The stance of "rescuing the utility function" advocates that we have no right to expect the universe to function exactly like our native representation of it.  According to this stance, it would be a strange and silly demand to make of the universe that its lowest level of operation correspond exactly to our built-in mental representations, and insist that we're not going to *feel* anything warm about reality unless heat is basic to physics.  The high-level representations our emotions natively bind to, could not reasonably have been fated to be identical with the raw low-level description of the universe.  So if we couldn't 'rescue the utility function' by identifying high-level heat with vibrating particles, this portion of our values would inevitably end in disappointment.

Once we can see as normative the question of how to feel about a universe that has dancing particles instead of ontologically basic warmth, we can see that going around wailing in existential despair about the coldness of the universe doesn't seem like the right act or the right judgment.  Instead we should rebind our native emotions to the non-instinctive but more accurate model of the universe.

If we aren't self-modifying AIs and can't actually rewrite our emotions to bind to learned abstract models, then we can come closer to normative reasoning by adopting the rule of visualizing warmth whenever we visualize whizzing particles.

On this stance, it is not a lie to visualize warmth when we visualize whizzing particles.  We are not giving ourselves a sad consolation prize for the absence of 'real' heat.  It's not an act of self-deception to imagine a sensation of lovely warmth going along with a bunch of vibrating particles.  That's just what heat is in our universe.  Similarly, our nerves are not lying to us when they make us feel that fast-vibrating water is warm water.

On the normative stance of "rescuing the utility function", when X turns out to be composed of Y, then by default we should feel about Y the same way we felt about X.  There might be other considerations that modify this, but that's the starting point or default.

After 'rescuing the utility function', your new theory of how the universe operates (that heat is made up of kinetic energy) adds up to *moral normality.*  If you previously thought that a warm summer was better than a cold winter, you will still think that a warm summer is better than a cold winter after you find out what heat is made out of.

(This is a good thing, since the point of a moral philosophy is not to be amazing and counterintuitive.)

## Reason for using the heat metaphor

One reason to start from this metaphorical example is that "heat" has a relatively understandable correspondence between high-level and low-level models.  On a high level, we can see heat melting ice and flowing from hotter objects to cooler objects.  We can, by imagination, see how vibrating particles could actually constitute heat rather than causing a mysterious extra 'heat' property to be present.  Vibrations might flow from fast-vibrating objects to slow-vibrating objects via the particles bumping into each other and transmitting their speed.  Water molecules vibrating quickly enough in an ice cube might break whatever bonds were holding them together in a solid object.

Since it does happen to be relatively easy to visualize how heat is composed of kinetic energy, we can *see* in this case that we are not lying to ourselves by imagining that lovely warmth is present wherever vibrating particles are present.

For an even more transparent reductionist identity, consider, "You're not *really* wearing socks, there *are* no socks, there's only a bunch of threads woven together that *looks* like a sock."  Your visual cortex can represent this identity directly, so it feels immediately transparent that the sock just is the collection of threads; when you imagine sock-shaped woven threads, you automatically feel your visual model recognizing a sock.

If the relation between heat and kinetic energy were too complicated to visualize easily, it might instead feel like we were being given a blind, unjustified rule that reality contains mysterious "bridging laws" that make a separate quality of heat be present when particles vibrate quickly.  Instructing ourselves to feel "warmth" as present when particles vibrate quickly would feel more like fooling ourselves, or self-deception.  But on the position of "rescuing the utility function", the same arguments ought to apply in this hypothetical, even if the level transition is less transparent.

The gap between mind and brain is larger than the gap between heat and vibration, which is why humanity understood heat as disordered kinetic energy long before anyone had any idea [38r how 'playing chess' could be decomposed into non-mental simpler parts].  In some cases, we may not know what the reductionist identity will be.  Still, the advice of "rescuing the utility function" is not to morally panic about realizing that various emotionally valent things will turn out to be made of parts, or even that our mind's representations in general may run somewhat skew to reality.

## Complex rescues

In the heat metaphor, the lower level of the universe (jiggling particles) corresponds fairly exactly to the high-level notion of heat.  We'd run into more complicated metamoral questions if we'd previously lumped together the 'heat' of chili peppers and the 'heat' of a fireplace as valuable 'warmth'.

We might end up saying that there are two physical kinds of valuable warm things: ceteris paribus and by default, if X turns out to consist of Y, then Y inherits X's role in the utility function.  Alternatively, by some non-default line of reasoning, the discovery that chili peppers and fireplaces are warm in ontologically different ways might lead us to change how we feel about them on a high level as well.  In this case we might have to carry out a more complicated rescue, where it's not so immediately obvious which low-level Ys are to inherit X's value.

# Non-metaphorical utility rescues

We don't actually have [1bh terminal values] for things being warm (probably).  Non-metaphorically, "rescuing the utility function" says that we should apply similar reasoning to phenomena that we do in fact value, whose corresponding native emotions we are having trouble reconciling with non-native, learned theories of the universe's ontology.

Examples might include:

- Moral responsibility.  How can we hold anyone responsible for their actions, or even hold ourselves responsible for what we see as our own choices, when our acts have causal histories behind them?
- Happiness.  What's the point of people being happy if it's just neurons firing?
- Goodness and shouldness.  Can there be any right thing to do, if there isn't an ontologically basic irreducible rightness property to correspond to our sense that some things are just right?
- Wanting and helping.  When a person wants different things at different times, and there are known experiments that expose circular and incoherent preferences, how can we possibly "help" anyone by "giving them what they want" or "[313 extrapolating their volition]"?

In cases like these, it may be that our native representation is in some sense running skew to the real universe.  E.g., our minds insist that something called "free will" is very important to moral responsibility, but it seems impossible to define "free will" in a coherent way.  The position of "rescuing the utility function" still takes the stance of "Okay, let's figure out how to map this emotion onto a coherent universe as best we can" not "Well, it looks like the human brain didn't start out with a perfect representation of reality, therefore, normatively speaking, we should toss the corresponding emotions out the window."  If in your native representation the Sun goes around the Earth, and then we learn differently from astronomy, then your native representation is in fact wrong, but normatively we should (by default) re-bind to the enormous glowing fusion reactor rather than saying that there's no Sun.

The role such concepts play in our values lends a special urgency to the question of how to rescue them.  But on an even more general level, one might espouse that it is the job of good reductionists to say *how* things exist if they have any scrap of reality, rather than it being the job of reductionists to go around declaring that things *don't* exist if we detect the slightest hint of fantasy.  Leif K-Brooks presented this general idea as follows (with intended application to 'free will' in particular):

![If you define a potato as a magic fairy orb and disprove the existence of magic fairy orbs, you still have a potato.](https://scontent.fsnc1-4.fna.fbcdn.net/v/t1.0-9/15697756_10154038465390025_4647971104986602099_n.jpg?oh=08071559c0a79d7779dd56ad6be1bd13&oe=58F65AAC)

# "Rescue" as resolving a degree of freedom in the pretheoretic viewpoint

A human child doesn't start out endorsing any particular way of turning emotions into utility functions.  As humans, we start out with no clear rules inside the chaos of our minds, and we have to make them up by considering various arguments appealing to our not-yet-organized intuitions.  Only then can we even try to have coherent metaethical principles.

The core argument for "rescuing the utility function" can be seen as a base intuitive appeal to someone who hasn't yet picked out any explicit rules, claiming that it isn't especially sensible to end up as the kind of agent whose utility function ends up being zero everywhere.

In other words, rather than the rescue project needing to appeal to rules that would *only* be appealing to somebody who'd already accepted the rescue project ab initio - which would indeed be circular - the start of the argument is meant to also work as an intuitive appeal to the pretheoretic state of mind of a normal human.  After that, we also hopefully find that these new rules are self-consistent.

In terms of the heat metaphor, if we're considering whether to discard heat, we can consider three types of agents:

- (1).  A pretheoretic or confused state of intuition, which knows itself to be confused.  An agent like this is not [2rb reflectively consistent] - it wants to resolve the internal tension.
- (2).  An agent that has fully erased all emotions relating to warmth, as if it never had them.  This type of agent is [2rb reflectively consistent]; it doesn't value warmth and doesn't want to value warmth.
- (3).  An agent that values naturalistic heat, i.e., feels the way about disordered kinetic energy that a pretheoretic human feels about warmth.  This type of agent has also resolved its issues and become [2rb reflectively consistent].

Since 2 and 3 are both internally consistent resolutions, there's potentially a [-2fr] in how (1) can resolve its current internal tension or inconsistency.  That is, it's not the case that the only kind of coherent agents are 2-agents or 3-agents, so just a desire for coherence qua coherence can't tell a 1-agent whether it should become a 2-agent or 3-agent.  By advocating for rescuing the utility function, we're appealing to a pretheoretic and maybe chaotic and confused mess of intuitions, aka a human, arguing that if you want to shake out the mess, it's better to shake out as a 3-agent rather than a 2-agent.

In making this appeal, we can't appeal to firm foundations that already exist, since a 1-agent hasn't yet decided on firm philosophical foundations and there's more than one set of possible foundations to adopt.  An agent with firm foundations would already be reflectively coherent and have no further philosophical confusion left to resolve (except perhaps for a mere matter of calculation).  An existing 2-agent is of course nonplussed by any arguments that heat should be valued, in much the same way that humans would be nonplussed for arguments in favor of valuing [10h paperclips] (or for that matter, things being hot).  But to point this out is no argument for why a confused 1-agent should shake itself out as a consistent 2-agent rather than a consistent 3-agent; a 3-agent is equally nonplussed by the argument that the best thing to do with an [5c ontology identification problem] is throw out all corresponding terms of the utility function.

It's perhaps lucky that human beings can't actually modify their own code, meaning that somebody partially talked into taking the 2-agent state as a new ideal to aspire to, still actually has the pretheoretic emotions and can potentially "snap out of it".  Rather than becoming a 2-agent or a 3-agent, we become "a 1-agent that sees 2-agency as ideal" or "a 1-agent that sees 3-agency as ideal".  A 1-agent aspiring to be a 2-agent can still potentially be talked out of it - they may still feel the weight of arguments meant to appeal to 1-agents, even if they think they *ought* not to, and can potentially "just snap out" of taking 2-agency as an ideal, reverting to confused 1-agency or to taking 3-agency as a new ideal.

## Looping through the meta-level in "rescuing the utility function"

Since human beings don't have "utility functions" (coherent preferences over probabilistic outcomes), the notion of "rescuing the utility function" is itself a matter of rescue.  Natively, it's possible for psychology experiments to expose inconsistent preferences, but instead of throwing up our hands and saying "Well I guess nobody wants anything and we might as well [10h let the universe get turned into paperclips]!", we try to back out some reasonably coherent preferences from the mess.  This is, arguendo, normatively better than throwing up our hands and turning the universe into paperclips.

Similarly, according to [313 the normative stance behind extrapolated volition], the very notion of "shouldness" is something that gets rescued.  Many people seem to instinctively feel that 'shouldness' wants to map onto an ontologically basic, irreducible property of rightness, such that every cognitively powerful agent with factual knowledge about this property is thereby compelled to perform the corresponding actions.  ("Moral internalism.")  But this demands an overly direct correspondence between our native sense that some acts have a compelling rightness quality about them, and wanting there to be an ontologically basic compelling rightness quality out there in the environment.

Despite the widespread appeal of moral internalism once people are exposed to it as an explicit theory, it still seems unfair to say that humans *natively* want or *pretheoretically* demand that this is what our sense of rightness correspond to.  E.g. a hunter-gatherer, or someone else who's never debated metaethics, doesn't start out with an explicit commitment about whether a feeling of rightness must correspond to universes that have irreducible rightness properties in them.  If you'd grown up thinking that your feeling of rightness [313 corresponded to computing a certain logical function over universes], this would seem natural and non-disappointing.

Since "shouldness" (the notion of normativity) is something that itself may need rescuing, this rescue of "shouldness" is in some sense being implicitly invoked by the normative assertion that we *should* try to "rescue the utility function".

This could be termed circular, but we could equally say that it is self-consistent.  Or rather, we are appealing to some chaotic, not-yet-rescued, pretheoretic notion of "should" in saying that we *should* try to rescue concepts like "shouldness" instead of throwing them out the window.  Afterwards, once we've performed the rescue and have a more coherent notion of concepts like "better", we can see that the loop through the meta-level has entered a consistent state.  According to this new ideal (less confused, but perhaps also seeming more abstract), it remains better not to give up on concepts like "better".

The "extrapolated volition" rescue of shouldness is meant to bootstrap, by appeal to a pretheoretic and potentially confused state of wondering what is the right thing to do (or how we even *should* resolve this whole "rightness" issue, and if maybe it would be *better* to just give up on it), into a more reflectively consistent state of mind.  Afterwards we can see both pretheoretically, and also in the light of our new explicit theory, that we *ought* to try to rescue the concept of oughtness, and adopt the rescued form of reasoning as a less-confused ideal.  We will believe that *ideally*, the best justification for extrapolated volition is to say that we know of no better candidate for what theory we'd arrive at if we thought about it for even longer.  But since humans can perhaps thankfully not directly rewrite their own code, we will also remain aware of whether this seems like a good idea in the pretheoretic sense, and perhaps prepared to unwind or jump back out of the system if it turns that out that the explicit theory has big problems we didn't know about when we originally jumped to it.

## Reducing tension / "as if you'd always known it"

We can possibly see the desired output of "rescuing the utility function" as being something like reducing a tension between native emotion-binding representations, and reality, with a minimum of added fuss and complexity.

This can look a lot like the intuition pump, "Suppose you'd grown up always knowing the true state of affairs, and nobody had suggested that you panic over it or experience any existential angst; what would you have grown up thinking?"  If you grow up with warmth-related emotions and already knowing that heat is disordered kinetic energy, and nobody has suggested to you that anyone ought to wail in existential angst about this, then you'll probably grow up valuing heat-as-disordered-kinetic-energy (and this will be a low-tension resolution).

For more confusing grades of cognitive reductionism, like free will, people might spontaneously have difficulty reconciling their internal sense of freedom with being told about deterministic physical laws.  But a good "rescue" of the corresponding sense of moral responsibility ought to end up looking like the sort of thing that you'd quietly take for granted as a quiet, obvious-seeming mapping of your sense of moral responsibility onto the physical universe, if you'd grown up taking those laws of physics for granted.

# "Rescuing" pretheoretic emotions and intuitions versus "rescuing" explicit moral theories

## Rewinding past factual-mistake-premised explicit moral theories

In the heat metaphor, suppose you'd previously adopted a 'caloric fluid' model of heat, and the explicit belief that this caloric fluid was what was valuable.  You still have wordless intuitions about warmth and good feelings about warmth.  You *also* have an explicit world-model that this heat corresponds to caloric fluid, and an explicit moral theory that this caloric fluid is what's good about heat.

Then science discovers that heat is disordered kinetic energy.  Should we try to rescue our moral feelings about caloric by looking for the closest thing in the universe to caloric fluid - electricity, maybe?

If we now reconsider the arguments for "rescuing the utility function", we find that we have more choices beyond "looking for the closest thing to caloric" and "giving up entirely on warm feelings".  An additional option is to try to rescue the intuitive sense of warmth, but not the explicit beliefs and explicit moral theories about "caloric fluid".

If we instead choose to rescue the pretheoretic emotion, we could see this as retracing our steps after being led down a garden-path of bad reasoning, aka, "not rescuing the garden path".  We started with intuitively good feelings about warmth, came to believe a false model about the causes of warmth, reacted emotionally to this false model, and developed an explicit moral theory about caloric fluid.

The [313 extrapolated-volition model of normativity] (what we would want\* if we knew all the facts) suggests that we could see the reasoning after adopting the false caloric model as "mistaken" and not rescue it.  When we're dealing with explicit moral beliefs that grew up around a false model of the world, we have the third option to "rewind and rescue" rather than "rescue" or "give up".

Nonmetaphorically:  Suppose you believe in a [divine command theory of metaethics](https://en.wikipedia.org/wiki/Divine_command_theory); goodness is equivalent to God wanting something.  Then one day you realize that there's no God in which to ground your moral theory.

In this case we have three options for resolution, all of which are reflectively consistent within themselves, and whose arguments may appeal to our currently-confused pretheoretic state:

- (a) Prefer to go about wailing in horror about the unfillable gap in the universe left by the absence of God.
- (b) Try to rescue the explicit divine command theory, e.g. by looking for the closest thing to a God and re-anchoring the divine command theory there.
- (c) Give up on the explicit model of divine command theory; instead, try to unwind past the garden path you went down after your native emotions reacted to the factually false model of God.  Try to remap the pretheoretic emotions and intuitions onto your new model of the universe.

Again, (a) and (b) and (c) all seem reflectively consistent in the sense that a simple agent fully in one of these states will not want to enter either of the other two states.  But given these three options, a confused agent might reasonably find either (b) or (c) more pretheoretically compelling than (a), but also find (c) more pretheoretically compelling than (b).

The notion of "rescuing" isn't meant to erase the notion of "mistakes" and "saying oops" with respect to b-vs.-c alternatives.  The arguments for "rescuing" warmth implicitly assumed that we were talking about a pretheoretic normative intuition (e.g. an emotion associated with warmth), not explicit models and theories about heat that could just as easily be revised.

Conversely, when we're dealing with preverbal intuitions and emotions whose natively bound representations are in some way running skew to reality, we can't rewind past the fact of our emotions binding to particular mental representations.  We were literally born that way.  Then our only obvious alternatives are to (a) give up entirely on that emotion and value, or (c) rescue the intuitions as best we can.  In this case (c) seems more pretheoretically appealing, ceteris paribus and by default.

(For example, suppose you were an alien that had grown up accepting commands from a Hive Queen, and you had a pretheoretic sense of the Hive Queen as knowing everything, and you mostly operated on an emotional-level Hive-Queen-command theory of rightness.  One day, you begin to suspect that the Hive Queen isn't actually omniscient.  Your alien version of "rescuing the utility function" might say to rescue the utility function by allowing valid commands to be issued by Hive Queens that knew a lot but weren't omniscient.  Or it might say to try and build a superintelligent Hive Queen that would know as much as possible, because in a pretheoretic sense that would feel better.  The aliens can't rewind past their analogue of divine command theory because, by hypothesis, the alien's equivalent of divine command metaethics is built into them on a pretheoretic and emotional level.  Though of course, in this case, such aliens seem more likely to actually resolve their tension by asking the Hive Queen what to do about it.)

## Possibility of rescuing non-mistake-premised explicit moral theories

Suppose Alice has an explicit belief that private property ought to be a thing, and this belief did *not* develop after she was told that objects had tiny XML tags declaring their irreducible objective owners, nor did she originally arrive at the conclusion based on a model in which God assigned transferable ownership of all objects at the dawn of time.  We can suppose, somewhat realistically, that Alice is a human and has a pretheoretic concept of ownership as well as deserving rewards for effort, and was raised by small-l libertarian parents who told her true facts about how East Germany did worse economically than West Germany.  Over time, she came to adopt an explicit moral theory of "private property": ownership can only transfer by consent, and that violations of this rule violate the just-rewards principle.

One day, Alice starts having trouble with her moral system because she's realized that property is made of atoms, and that even the very flesh in her body is constantly exchanging oxygen and carbon dioxide with the publicly owned atmosphere.  Can atoms really be privately owned?

The confused Alice now again sees three options, all of them [2rb reflectively consistent] on their own terms if adopted:

- (a) Give up on everything to do with ownership or deserving rewards for efforts; regard these emotions as having no valid referents.
- (b) Try to rescue the explicit moral theory by saying that, sure, atoms can be privately owned.  Alice owns a changeable number of carbon atoms inside her body and she won't worry too much about how they get exchanged with the atmosphere; that's just the obvious way to map private property onto a particle-based universe.
- (c) Try to rewind past the explicit moral theory, and figure out from scratch what to do with emotions about "deserves reward" or "owns".

Leaving aside what you think of Alice's explicit moral theory, it's not obvious that Alice will end up preferring (c) to (b), especially since Alice's current intuitive state is influenced by her currently-active explicit theory of private property.

Unlike the divine command theory, Alice's private property theory was not (obviously to Alice) arrived at through a path that traversed [3t3 wrong beliefs of simple fact].  With the divine command theory, since it was critically premised on a wrong factual model, we face the prospect of having to stretch the theory quite a lot in order to rescue it, making it less intuitively appealing to a confused mind than the alternative prospect of stretching the pretheoretic emotions a lot less.  Whereas from Alice's perspective, she can just as easily pick up the whole moral theory and morph it onto reductionist physics with all the internal links intact, rather than needing to rewind past anything.

We at least have the apparent *option* of trying to rescue Alice's utility function in a way that preserves her explicit moral theories *not* based on bad factual models - the steps of her previous explicit reasoning that did not, of themselves, introduce any new tensions or problems in mapping her emotions or morals onto a new representation.  Whether or not we ought to do this, it's a plausible possibility on the table.

## Which explicit theories to rescue?

It's not yet obvious where to draw the line on which explicit moral theories to rescue.  So far as we can currently see, any of the following might be a reasonable way to tell a superintelligence to extrapolate someone's volition:

- Preserve explicit moral theories wherever it doesn't involve an enormous stretch.
- Be skeptical of explicit moral theories that were arrived at by fragile reasoning processes, even if they could be rescued in an obvious way.
- Only extrapolate pretheoretic intuitions.

Again, all of these viewpoints are internally consistent (they are degrees of freedom in the metaphorical meta-utility-function), so the question is which rule for drawing the line seems most intuitively appealing in our present state of confusion:

### Argument from adding up to normality

Arguendo:  Preserving explicit moral theories is important for having the rescued utility function add up to normality.  After rescuing my notion of "shouldness", then afterwards I should, by default, see mostly the same things as rescued-right.

Suppose Alice was previously a moral internalist, and thought that some things were inherently irreducibly right, such that her very notion of "shouldness" needed rescuing.  That doesn't necessarily introduce any difficulties into re-importing her beliefs about private property.  Alice may have previously refused to consider some arguments against private property because she thought it was irreducibly right, but this is a separate issue in extrapolating her volition from throwing out her entire stock of explicit moral theories because they all used the word "should".  By default, after we're done rescuing Alice, unless we are doing something that's clearly and explicitly correcting an error, her rescued viewpoint should look as normal-relative-to-her-previous-perspective as possible.

### Argument from helping

Arguendo:  Preserving explicit moral theories where possible is an important aspect of how an ideal advisor or superintelligence ought to extrapolate someone else's volition.

Suppose Alice was previously a moral internalist, and thought that some things were inherently irreducibly right, such that her very notion of "shouldness" needed rescuing.  Alice may not regard it as "helping" her to throw away all of her explicit theories and try to re-extrapolate her emotions from scratch into new theories.  If there weren't any factual flaws involved, Alice is likely to see it as less than maximally helpful to her if we needlessly toss one of her cherished explicit moral theories.

### Argument from incoherence, evil, chaos, and arbitrariness

Arguendo:  Humans are really, really bad at systematizing explicit moral theories; a supermajority of explicit moral theories in today's world will be incoherent, evil, or both.  E.g., explicit moral principles may de-facto be chosen mostly on the basis of, e.g., how hard they appear to cheer for a valorized group.  An extrapolation dynamic that tries to take into account all these chaotic, arbitrarily-generated group beliefs will end up failing to cohere.

### Argument from fragility of goodness

Arguendo:  Most of what we see as the most precious and important part of ourselves are explicit moral theories like "all sapient beings should have rights", which aren't built into human babies.  We may well have arrived at that destination through a historical trajectory that went through factual mistakes, like believing that all human beings had souls created equal by God and were loved equally by God. (E.g. Christian theology seems to have been, as a matter of historical fact, causally important in the development of explicit anti-slavery sentiment.)  Tossing the explicit moral theories is as unlikely to be good, from our perspective, as tossing our brains and trying to rerun the process of natural selection to generate new emotions.

### Argument from dependency on empirical results

Arguendo:  Which version of extrapolation we'll actually find appealing will depend on which extrapolation algorithm turns out to have a reasonable answer.  We don't have enough computing power to guess, right now, whether:

- Any reasonable-looking construal of "Toss out all the explicit cognitive content and redo by throwing pretheoretic emotions at true facts" leads to an extrapolated volition that lacks discipline and coherence, looking selfish and rather angry, failing to regenerate most of altruism or Fun Theory; *or*
- Any reasonable-looking construal of "Try to preserve explicit moral theories" leads to an incoherent mess of assertions about various people going to Hell and capitalism being bad for you.

Since we can't guess using our present computing power which rule would cause us to recoil in horror, but the actual horrified recoil would settle the question, we can only defer this single bit of information to one person who's allowed to peek at the results.

(Counterargument:  "Perhaps the ancient Greeks would have recoiled in horror if they saw how little the future would think of a glorious death in battle, thus picking the option we see as wrong, using the stated rule.")


Philosophy

For now, this is a stub domain to hold standard concepts from academic philosophy, which are being used elsewhere on Arbital (we're not currently targeting this domain as such).


Total alignment

An advanced agent can be said to be "totally aligned" when it can assess the *exact* [-55] of well-described outcomes and hence the *exact* subjective value of actions, policies, and plans; where [-55] has its overridden meaning of a metasyntactic variable standing in for "whatever we really do or really should value in the world or want from an Artificial Intelligence" (this is the same as "normative" if the speaker believes in normativity).  That is:  It's an advanced agent that captures *all* the distinctions we would make or should make within which outcomes are good or bad; it has "full coverage" of the true or intended goals; it correctly resolves every [-2fr].

We don't need to try and give such an AI simplified orders like, e.g., "try to have a [2pf lower impact]" because we're worried about, e.g., a [-42] problem on trying to draw exact boundaries around what constitutes a bad impact.  The AI knows *everything* worth knowing about which impacts are bad, and even if it thinks of a really weird exotic plan, it will still be able to figure out which aspects of this plan match our intended notion of  [-55] or a normative notion of  [-55].

If this agent does not systematically underestimate the probability of bad outcomes / overestimate the probability of good outcomes, and its maximization over policies is not subject to adverse subjection, its estimates of expected [-55] will be well-calibrated even from our own outside standpoint.


Superintelligent

[summary:  A supernova isn't infinitely hot, but it's still pretty darned hot; in the same sense, a superintelligence isn't infinitely smart, but it's pretty darned smart: strictly superhuman across all cognitive domains, by a significant margin, or else as a fallback merely optimal.  (A superintelligence can't win against a human at [9s logical tic-tac-toe], though in real-world tic-tac-toe it could disassemble the opposing player.)  Superintelligences are [6s epistemically and instrumentally efficient] relative to humans, and have all the other [2c advanced agent properties] as well.]

Machine performance inside a domain (class of problems) can potentially be:

- Optimal (impossible to do better)
- Strongly superhuman (better than all humans by a significant margin)
- Weakly superhuman (better than all the humans most of the time and most of the humans all of the time)
- Par-human (performs about as well as most humans, better in some places and worse in others)
- Subhuman or infrahuman (performs worse than most humans)

A superintelligence is either 'strongly superhuman', or else at least 'optimal', across all cognitive domains.  It can't win against a human at [9s logical tic-tac-toe], but it plays optimally there.  In a real-world game of tic-tac-toe that it strongly wanted to win, it might sabotage the opposing player, deploying superhuman strategies on the richer "real world" gameboard.

I. J. Good originally used 'ultraintelligence' to denote the same concept:  "Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever."

To say that a hypothetical agent or process is "superintelligent" will usually imply that it has all the [2c advanced-agent properties].

Superintelligences are still [2rd bounded] (if the character of physical law at all resembles the Standard Model of physics).  They are (presumably) not infinitely smart, infinitely fast, all-knowing, or able to achieve every describable outcome using their available resources and options.  However:

- A supernova isn't infinitely hot, but it's still pretty warm.  "Bounded" does not imply "small".  You should not try to walk into a supernova using a standard flame-retardant jumpsuit after reasoning, correctly but unhelpfully, that it is only boundedly hot.
- A superintelligence doesn't know everything and can't perfectly estimate every quantity.  However, to say that something is "superintelligent" or superhuman/optimal in every cognitive domain should almost always imply that its estimates are [6s epistemically efficient relative to every human and human group].  Even a superintelligence may not be able to exactly estimate the number of hydrogen atoms in the Sun, but a human shouldn't be able to say, "Oh, it will probably underestimate the number by 10% because hydrogen atoms are pretty light" - the superintelligence knows that too.  For us to know better than the superintelligence is at least as implausible as our being able to predict a 20% price increase in Microsoft's stock six months in advance without any private information.
- A superintelligence is not omnipotent and can't obtain every describable outcome.  But to say that it is "superintelligent" should suppose at least that it is [6s instrumentally efficient relative to humans]:  We should not suppose that a superintelligence carries out any policy $\pi_0$ such that a human can think of a policy $\pi_1$ which would get more of the agent's [109 utility].  To put it another way, the assertion that a superintelligence optimizing for utility function $U,$ would pursue a policy $\pi_0,$ is by default refuted if we observe some $\pi_1$ such that, so far as we can see, $\mathbb E[U | \pi_0] < \mathbb E[U | \pi_1].$  We're not sure the efficient agent will do $\pi_1$ - there might be an even better alternative we [9f haven't foreseen] - but we should regard it as very likely that it won't do $\pi_0.$

If we're talking about a hypothetical superintelligence, probably we're either supposing that an [428 intelligence explosion] happened, or we're talking about a limit state approached by a long period of progress.

Many/most problems in [2v AI alignment] seem like they ought to first appear at a point short of full superintelligence.  As part of the project of making discourse about advanced agents precise, we should try to identify the key advanced agent property more precisely than saying "this problem would appear on approaching superintelligence" - to suppose superintelligence is usually *sufficient* but will rarely be necessary.

For the book, see [3db].


William Frankena's list of terminal values

"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc." -- William Frankena's list of [1bh things valued in themselves rather than only for their further consequences].


Nearest unblocked strategy

[summary:  Nearest Unblocked Strategy is a hypothetical source of [-48] in the [5s alignment problem] for [2c advanced agents] that search [9j rich solution spaces].  If an agent's [5f preference framework] is [48 patched] to try to block a possible solution that seems undesirable, the next-best solution found may be the most similar solution that technically avoids the block.  This kind of patching seems especially likely to lead to a [-context change] where a patch appears [3d9 beneficial] in a narrow option space, but proves detrimental after increased intelligence opens up more options.]

[todo: link to epistemic version http://lesswrong.com/lw/nki/jfk_was_not_assassinated_prior_probability_zero/d9h3?context=3 ]

# Introduction

'Nearest unblocked strategy' seems like it should be a [6r foreseeable] problem of trying to get rid of undesirable AI behaviors by adding specific penalty terms to them, or otherwise trying to exclude one class of observed or foreseen bad behaviors.  Namely, if a decision criterion thinks $X$ is the best thing to do, and you add a penalty term $P$ that you think excludes everything inside $X,$ the *next-best* thing to do may be a very similar thing $X'$ which is the most similar thing to $X$ that doesn't trigger $P.$

## Example:  Producing happiness.

Some very early proposals for AI alignment suggested that AIs be targeted on producing human happiness.  Leaving aside various other objections, arguendo, imagine the following series of problems and attempted fixes:

- By hypothesis, the AI is successfully infused with a goal of "human happiness" as a utility function over human brain states.  (Arguendo, this predicate is narrowed sufficiently that the AI does not just want to construct [2w the tiniest, least resource-intensive brains experiencing the largest amount of happiness per erg of energy].)
- Initially, the AI seems to be pursuing this goal in good ways; it organizes files, tells funny jokes, helps landladies take out the garbage, etcetera.
- Encouraged, the programmers further improve the AI and add more computing power.
- The AI gains a better understanding of the world, and the AI's [6q policy space expands] to include conceivable options like "administer heroin".
- The AI starts planning how to administer heroin to people.
- The programmers notice this before it happens.  (Arguendo, due to successful transparency features, or an imperative to [2qq check plans with the users], which operated as [6h intended] at the AI's current level of intelligence.)
- The programmers edit the AI's utility function and add a penalty of -100 utilons for any event categorized as "the AI administers heroin to humans".  (Arguendo, the AI's current level of intelligence does not suffice to [ prevent the programmers from editing its utility function], despite the convergent instrumental incentive to avoid this; nor does it successfully [10f deceive] the programmers.)
- The AI gets slightly smarter.  New conceivable options enter the AI's option space.
- The AI starts wanting to administer cocaine to humans (instead of heroin).
- The programmers read through the current schedule of prohibited drugs and add penalty terms for administering marijuana, cocaine, etcetera.
- The AI becomes slightly smarter.  New options enter its policy space.
- The AI starts thinking about how to research a new happiness drug not on the list of drugs that its utility function designates as bad.
- The programmers, after some work, manage to develop a category for 'The AI forcibly administering any kind of psychoactive drug to humans' which is broad enough that the AI stops suggesting research campaigns to develop things slightly outside the category.
- The AI wants to build an external system to administer heroin, so that it won't be classified inside this set of bad events "the AI forcibly administering drugs".
- The programmers generalize the penalty predicate to include "machine systems in general forcibly administering heroin" as a bad thing.
- The AI recalculates what it wants, and begins to want to pay humans to administer heroin.
- The programmers try to generalize the category of penalized events to include non-voluntarily administration of drugs in general that produce happiness, whether done by humans or AIs.  The programmers patch this category so that the AI is not trying to shut down at least the nicer parts of psychiatric hospitals.
- The AI begins planning an ad campaign to persuade people to use heroin voluntarily.
- The programmers add a penalty of -100 utilons for "AIs *persuading* humans to use drugs".
- The AI goes back to helping landladies take out the garbage.  All seems to be well.
- The AI continues to increase in intelligence, becoming capable enough that the AI can no longer be edited against its own will.
- The AI notices the option "Tweak human brains to express extremely high levels of endogenous opiates, then take care of their twitching bodies to so they can go on being happy".

The overall story is one where the AI's preferences on round $i,$ denoted $U_i,$ are observed to arrive at an attainable optimum $X_i$ which the humans see as undesirable.  The humans devise a penalty term $P_i$ intended to exclude the undesirable parts of the policy space, and add this to $U_i$ creating a new utility function $U_{i+1},$ after which the AI's optimal policy settles into a new state $X_i^*$ that seems acceptable.  However, after the next expansion of the policy space, $U_{i+1}$ settles into a new attainable optimum $X_{i+1}$ which is very similar to $X_i$ and makes the minimum adjustment necessary to evade the boundaries of the penalty term $P_i,$ requiring a new penalty term $P_{i+1}$ to exclude this new misbehavior.

(The end of this story *might* not kill you if the AI had enough successful, [2l advanced-safe] [45 corrigibility features] that the AI would [2x indefinitely] go on [2qq checking] [2qp novel] policies and [2qp novel] goal instantiations with the users, not strategically hiding its disalignment from the programmers, not deceiving the programmers, letting the programmers edit its utility function, not doing anything disastrous before the utility function had been edited, etcetera.  But you wouldn't want to rely on this.  You would not want in the first place to operate on the paradigm of 'maximize happiness, but not via any of these bad methods that we have already excluded'.)

# Preconditions

Recurrence of a nearby unblocked strategy is argued to be a [6r foreseeable difficulty] given the following preconditions:

• The AI is a [9h consequentialist], or is conducting some other search such that when the search is blocked at $X,$ the search may happen upon a similar $X'$ that fits the same criterion that originally promoted $X.$  E.g. in an agent that selects actions on the basis of their consequences, if an event $X$ leads to goal $G$ but $X$ is blocked, then a similar $X'$ may also have the property of leading to $G.$

• The search is taking place over a [9j rich domain] where the space of relevant neighbors around X is too complicated for us to be certain that we have described all the relevant neighbors correctly.  If we imagine an agent playing [9s the purely ideal game of logical Tic-Tac-Toe], then if the agent's utility function hates playing in the center of the board, we can be sure (because we can exhaustively consider the space) that there are no Tic-Tac-Toe squares that behave strategically almost like the center but don't meet the exact definition we used of 'center'.  In the far more complicated real world, when you eliminate 'administer heroin' you are very likely to find some other chemical or trick that is strategically mostly equivalent to administering heroin.  See "[RealIsRich Almost all real-world domains are rich]".

• From our perspective on [-55], the AI does not have an [ absolute identification of value] for the domain, due to some combination of "the domain is rich" and "[5l value is complex]".  Chess is complicated enough that human players can't absolutely identify winning moves, but since a chess program can have an absolute identification of which endstates constitute winning, we don't run into a problem of unending patches in identifying which states of the board are good play.  (However, if we consider a very early chess program that (from our perspective) was trying to be a consequentialist but wasn't very good at it, then we can imagine that, if the early chess program consistently threw its queen onto the right edge of the board for strange reasons, forbidding it to move the queen there might well lead it to throw the queen onto the left edge for the same strange reasons.)

# Arguments

## 'Nearest unblocked' behavior is sometimes observed in humans

Although humans obeying the law make poor analogies for mathematical algorithms, in some cases human economic actors expect not to encounter legal or social penalties for obeying the letter rather than the spirit of the law.  In those cases, after a previously high-yield strategy is outlawed or penalized, the result is very often a near-neighboring result that barely evades the letter of the law.  This illustrates that the theoretical argument also applies in practice to at least some pseudo-economic agents (humans), as we would expect given the stated preconditions.

## [5l Complexity of value] means we should not expect to find a simple encoding to exclude detrimental strategies

To a human, 'poisonous' is one word.  In terms of molecular biology, the exact volume of the configuration space of molecules that is 'nonpoisonous' is very complicated.  By having a single word/concept for poisonous-vs.-nonpoisonous, we're *dimensionally reducing* the space of edible substances - taking a very squiggly volume of molecule-space, and mapping it all onto a linear scale from 'nonpoisonous' to 'poisonous'.

There's a sense in which human cognition implicitly performs dimensional reduction on our solution space, especially by simplifying dimensions that are relevant to some component of our values.  There may be some psychological sense in which we feel like "do X, only not weird low-value X" ought to be a simple instruction, and an agent that repeatedly produces the next unblocked weird low-value X is being perverse - that the agent, given a few examples of weird low-value Xs labeled as noninstances of the desired concept, ought to be able to just generalize to not produce weird low-value Xs.

In fact, if it were possible to [full_coverage encode all relevant dimensions of human value into the agent] then we could just say *directly* to "do X, but not low-value X".  By the definition of [-full_coverage], the agent's concept for 'low-value' includes everything that is actually of low [55 value], so this one instruction would blanket all the undesirable strategies we want to avoid.

Conversely, the truth of the [5l complexity of value thesis] would imply that the simple word 'low-value' is dimensionally reducing a space of tremendous [5v algorithmic complexity].  Thus the effort required to actually convey the relevant dos and don'ts of "X, only not weird low-value X" would be high, and a human-generated set of supervised examples labeled 'not the kind of X we mean' would be unlikely to cover and stabilize all the dimensions of the underlying space of possibilities.  Since the weird low-value X cannot be eliminated in one instruction or several patches or a human-generated set of supervised examples, the [-42] problem will recur incrementally each time a patch is attempted and then the policy space is widened again.

# Consequences

[42] being a [6r foreseeable difficulty] is a major contributor to worrying that short-term incentives in AI development, to get today's system working today, or to have today's system not exhibiting any immediately visible problems today, will not lead to advanced agents which are [2l safe after undergoing significant gains in capability].

More generally, [-42] is a [6r foreseeable] reason why saying "Well just exclude X" or "Just write the code to not X" or "Add a penalty term for X" doesn't solve most of the issues that crop up in AI alignment.

Even more generally, this suggests that we want AIs to operate inside a space of [2qp conservative categories containing actively whitelisted strategies and goal instantiations], rather than having the AI operate inside a (constantly expanding) space of all conceivable policies minus a set of blacklisted categories.


Intelligence explosion

An "intelligence explosion" is what happens if a machine intelligence has [fast, consistent returns on investing work into improving its own cognitive powers, over an extended period](https://intelligence.org/files/IEM.pdf).  This would most stereotypically happen because it became able to optimize its own cognitive software, but could also apply in the case of "invested cognitive power in seizing all the computing power on the Internet" or "invested cognitive power in cracking the protein folding problem and then built nanocomputers".


Artificial General Intelligence

An "Artificial General Intelligence" is a machine intelligence possessed of some form of the same [7vh "significantly more generally applicable" intelligence] that distinguishes humans from our nearest chimpanzee relatives.  A bee builds hives, a beaver builds dams, a human looks at both and imagines a dam with honeycomb structure.  We can drive cars or do algebra, even though no similar problems were found in our environment of evolutionary adaptedness, and we haven't had time to evolve further to match cars or algebra.  The brain's algorithms are sufficiently general, sufficiently cross-domain, that we can learn a tremendous variety of new [7vf domains] within our lifetimes.  We are not perfectly general - we have an easier time learning to walk than learning to do abstract calculus, even though the latter is much easier in an objective sense.  But we're sufficiently general that we can figure out Special Relativity and engineer skyscrapers despite our not having those abilities at "compile time".  An Artificial General Intelligence has the same property; it can learn a tremendous variety of domains, including domains it had no inkling of when it was switched on.