Methodology of foreseeable difficulties

Much of the current literature about value alignment centers on purported reasons to expect that certain problems will require solution, or be difficult, or be more difficult than some people seem to expect.  The subject of this page's approval rating is this practice, considered as a policy or methodology.

The basic motivation behind trying to foresee difficulties is the large number of predicted [6q Context Change] problems where an AI seems to behave nicely up until it reaches some threshold level of cognitive ability and then it behaves less nicely.  In some cases the problems are generated without the AI having formed that intention in advance, meaning that even transparency of the AI's thought processes during its earlier state can't save us.  This means we have to see problems of this type in advance.

(The fact that Context Change problems of this type can be *hard* to see in advance, or that we might conceivably fail to see one, doesn't mean we can skip this duty of analysis.  Not trying to foresee them means relying on observation, and it seems *predictable* that trying to eyeball the AI and rejecting theory *definitely* doesn't catch important classes of problem.)

[todo: # Examples]

[todo: ...most of value alignment theory, so try to pick 3 cases that illustrate the point in different ways.  Pick from Context Change?]

# Arguments

For:  it's sometimes possible to strongly foresee a difficulty coming in a case where you've observed naive respondents to seem to think that no difficulty exists, and in cases where the development trajectory of the agent seems to imply a potential [6q Treacherous Turn].  If there's even one real Treacherous Turn out of all the cases that have been argued, then the point carries that past a certain point, you have to see the bullet coming before it actually hits you.  The theoretical analysis suggests really strongly that blindly forging ahead 'experimentally' will be fatal.  Someone with such a strong commitment to experimentalism that they want to ignore this theoretical analysis... it's not clear what we can say to them, except maybe to appeal to the normative principle of not predictably destroying the world in cases where it seems like we could have done better.

Against:  no real arguments against in the actual literature, but it would be surprising if somebody didn't claim that the foreseeable difficulties program was too pessimistic, or inevitably ungrounded from reality and productive only of bad ideas even when refuted, etcetera.

Primary reply: look, dammit, people actually are way too optimistic about FAI, we have them on the record, [todo: find 3 prestigious examples] and it's hard to see how humanity could avoid walking directly into the whirling razor blades without better foresight of difficulty.  One potential strategy is enough academic respect and consensus on enough really obvious foreseeable difficulties that the people claiming it will all be easy are actually asked to explain why the foreseeable difficulty consensus is wrong, and if they can't explain that well, they lose respect.

Will interact with the arguments on [108 empiricism vs. theorism is a false dichotomy].


Epistemic and instrumental efficiency

[summary:  An agent that is "efficient", relative to you, within a domain, never makes a real error that you can predict.

For example:  A [41l superintelligence] might not be able to count the exact number of atoms in a star.  But you shouldn't be able to say, "I think it will overestimate the number of atoms by 10%, because hydrogen atoms are so light."  It knows that too.  For you to foresee a predictable directional error in a superintelligence's estimates is at least as impossible as you predicting a 10% rise in Microsoft's stock, over the next week, using only public information.

- *Epistemic* efficiency is your inability to predict any directional error in the agent's estimates and probabilities.
- *Instrumental* efficiency is your inability to think of a policy that would achieve more of the agent's utility than whatever policy it actually uses.]

An agent that is "efficient", relative to you, within a domain, is one that never makes a real error that you can systematically predict in advance.

- Epistemic efficiency (relative to you):  You cannot predict directional biases in the agent's estimates (within a domain).
- Instrumental efficiency (relative to you):  The agent's strategy (within a domain) always achieves at least as much utility or expected utility, under its own preferences, as the best strategy you can think of for obtaining that utility (while staying within the same domain).

If an agent is epistemically and instrumentally efficient relative to all of humanity across all domains, we can just say that it is "efficient" (and almost surely [41l superintelligent]).

## Epistemic efficiency

A [41l superintelligence] cannot be assumed to know the exact number of hydrogen atoms in a star; but we should not find ourselves believing that we ourselves can predict in advance that a superintelligence will overestimate the number of hydrogen atoms by a factor of 10%.  Any thought process we can use to predict this overestimate should also be accessible to the superintelligence, and it can apply the same corrective factor itself.

The main analogy from present human experience would be the Efficient Markets Hypothesis as applied to short-term asset prices in highly-traded markets.  Anyone who thinks they have a reliable, repeatable ability to predict 10% changes in the price of S&P 500 companies over one-month time periods is mistaken.  If someone has a story to tell about how the economy works that requires advance-predictable 10% changes in the asset prices of highly liquid markets, we infer that the story is wrong.  There can be sharp corrections in stock prices (the markets can be 'wrong'), but not humans who can reliably predict those corrections (over one-month timescales).  If e.g. somebody is consistently making money by selling options using some straightforward-seeming strategy, we suspect that such options will sometimes blow up and lose all the money gained ("picking up pennies in front of a steamroller").

An 'efficient agent' is epistemically strong enough that we apply at least the degree of skepticism to a human proposing to outdo their estimates that, e.g., an experienced proponent of the Efficient Markets Hypothesis would apply to your uncle boasting about how he made a lot of money by predicting how General Motors's stock would rise.

Epistemic efficiency implicitly requires that an advanced agent can always learn a model of the world at least as predictively accurate as used by any human or human institution.  If our hypothesis space were usefully wider than that of an advanced agent, such that the truth sometimes lay in our hypothesis space while being outside the agent's hypothesis space, then we would be able to produce better predictions than the agent.

## Instrumental efficiency

This is the analogue of epistemic advancement for instrumental strategizing:  By definition, humans cannot expect to imagine an improved strategy compared to an efficient agent's selected strategy (relative to the agent's preferences, and given the options the agent has available).

If someone argues that a [2c cognitively advanced] [10h paperclip maximizer] would do X yielding M expected paperclips, and we can think of an alternative strategy Y that yields N expected paperclips, N > M, then while we cannot be confident that a PaperclipMaximizer will use strategy Y, we strongly predict that:

- (1) a [10h paperclip maximizer] will not use strategy X, or
- (2a) if it does use X, strategy Y was unexpectedly flawed, or
- (2b) if it does use X, strategy X will yield unexpectedly high value

...where to avoid [ privileging the hypothesis] or [ fighting a rearguard action] we should usually just say, "No, a Paperclip Maximizer wouldn't do X because Y would produce more paperclips."  In saying this, we're implicitly making an appeal to a version of instrumental efficiency; we're supposing the Paperclip Maximizer isn't stupid enough to miss something that seems obvious to a human thinking about the problem for five minutes.

Instrumental efficiency implicitly requires that the agent is always able to conceptualize any useful strategy that humans can conceptualize; it must be able to search at least as wide a space of possible strategies as humans could.

### Instrumentally efficient agents are presently unknown

From the standpoint of present human experience, instrumentally efficient agents are unknown outside of very limited domains.  There are perfect tic-tac-toe players; but even modern chess-playing programs, with ability far in advance of any human player, are not yet so advanced that every move that *looks* to us like a mistake *must therefore* be secretly clever.  We don't dismiss out of hand the notion that a human has thought of a better move than the chess-playing algorithm, the way we dismiss out of hand a supposed secret to the stock market that predicts 10% price changes of S&P 500 companies using public information.

There is no analogue of 'instrumental efficiency' in asset markets, since market prices do not directly select among strategic options.  Nobody has yet formulated a use of the EMH such that we could spend a hundred million dollars to guarantee liquidity, and get a well-traded asset market to directly design a liquid fluoride thorium nuclear plant, such that if anyone said before the start of trading, "Here is a design X that achieves expected value M", we would feel confident that either the asset market's final selected design would achieve at least expected value M or that the original assertion about X's expected value was wrong.

By restricting the meaning even further, we get a valid metaphor in chess: an ordinary person such as you, if you're not an International Grandmaster with hours to think about the game, should regard a modern chess program as instrumentally efficient relative to you.  The chess program will not make any mistake that you can understand as a mistake.  You should expect the reason why the chess program moves anywhere to be only understandable as 'because that move had the greatest probability of winning the game' and not in any other terms like 'it likes to move its pawn'.  If you see the chess program move somewhere unexpected, you conclude that it is about to do exceptionally well or that the move you expected was surprisingly bad.  There's no way for you to find any better path to the chess program's goals by thinking about the board yourself.  An instrumentally efficient agent would have this property for humans in general and the real world in general, not just you and a chess game.

### Corporations are not [41l superintelligences]

For any reasonable attempt to define a corporation's utility function (e.g. discounted future cash flows), it is not the case that we can confidently dismiss any assertion by a human that a corporation could achieve 10% more utility under its utility function by doing something differently.  It is common for a corporation's stock price to rise immediately after it fires a CEO or renounces some other mistake that many market actors knew was a mistake but had been going on for years - the market actors are not able to make a profit on correcting that error, so the error persists.

Standard economic theory does not predict that any currently known economic actor will be instrumentally efficient under any particular utility function, including corporations.  If it did, we could maximize any other strategic problem if we could make that actor's utility function conditional on it, e.g., reliably obtain the best humanly imaginable nuclear plant design by paying a corporation for it via a sufficiently well-designed contract.

We have sometimes seen people trying to label corporations as [41l superintelligences], with the implication that corporations are the real threat and equally severe, as threats, compared to machine superintelligences.  But epistemic or instrumental decision-making efficiency of individual corporations is just not predicted by standard economic theory.  Most corporations do not even use internal prediction markets, or try to run conditional stock-price markets to select among known courses of action.  Standard economic history includes many accounts of corporations making 'obvious mistakes' and these accounts are not questioned in the way that e.g. a persistent large predictable error in short-run asset prices would be questioned.

Since corporations are not instrumentally efficient (or epistemically efficient), they are not superintelligences.


Standard agent properties

### Boundedly rational agents

- Have probabilistic models of the world.
- Update those models in response to sensory information.
  - The ideal algorithm for updating is Bayesian inference, but this requires too much computing power and a bounded agent must use some bounded alternative.
  - Implicitly, we assume the agent has some equivalent of a complexity-penalizing prior or Occam's Razor.  Without this, specifying Bayesian inference does not much constrain the end results of epistemic reasoning.
- Have preferences over events or states of the world, quantifiable by a utility function that maps those events or states onto a scalar field.
  - These preferences must be quantitative, not just ordered, in order to combine with epistemic states of uncertainty (probabilities).
- Are consequentialist: they evaluate the expected consequences of actions and choose among actions based on preference among their expected consequences.
  - Bounded agents cannot evaluate all possible actions and hence cannot obtain literal maximums of expected utility except in very simple cases.
- Act in real time in a noisy, uncertain environment.

For the arguments that sufficiently intelligent agents will appear to us as boundedly rational agents in some sense, see:

- [29 Relevant powerful agents will be highly optimized]
- [21 Sufficiently optimized agents appear coherent]

### Economic agents

- Achieve their goals by efficiently allocating limited resources, including, e.g., time, money, or negentropy;
- Try to find new paths that route around obstacles to goal achievement;
- Predict the actions of other agents;
- Try to coordinate with, manipulate, or hinder other agents (in accordance with the agent's own goals or utilities);
- Respond to both negative incentives (penalties) and positive incentives (rewards) by planning accordingly, and may also consider strategies to avoid penalties or gain rewards that were unforeseen by the creators of the incentive framework.

### Naturalistic agents

- Naturalistic agents are embedded in a larger universe and are made of the same material as other things in the universe (wavefunction, on our current beliefs about physics).
- A naturalistic agent's uncertainty about the environment is uncertainty about which natural universe embeds them (what material structure underlies their available sensory and introspective data).
- Some of the actions available to naturalistic agents potentially alter their sensors, actuators, or computing substrate.
- Sufficiently powerful naturalistic agents may construct other agents out of resources available to them internally or in their environment, or extend their intelligence into outside computing resources.
- A naturalistic agent's sensing, cognitive, and decision/action capabilities may be distributed over space, time, and multiple substrates; the applicability of the 'agent' concept does not require a small local robot body.


Mindcrime

[summary(Gloss):  A huge amount of harm could occur if a [2c machine intelligence] turns out to contain lots of [18j conscious subprograms] enduring poor living conditions.  One worry is that this might happen if an AI models humans in too much detail.]

[summary:  'Mindcrime' is [18k]'s suggested term for the moral catastrophe that occurs if a [2c machine intelligence] contains enormous numbers of [18j conscious beings trapped inside its code].

This could happen as a result of self-awareness being a natural property of computationally efficient subprocesses.  Perhaps more worryingly, the best model of a person may be a person itself, even if they're not the same person.  This means that AIs trying to model humans might be unusually likely to create hypotheses and simulations that are themselves conscious.]

[summary(Technical):  'Mindcrime' is [18k]'s term for mind designs producing moral harm by their internal operation, particularly through embedding sentient subprocesses.

One worry is that mindcrime might arise in the course of an agent trying to predict or manipulate the humans in its environment, since this implies a pressure to model the humans in faithful detail.  This is especially concerning since several value alignment proposals would explicitly call for modeling humans in detail, e.g. [3c5 extrapolated volition] and [44z imitation-based agents].

Another problem scenario is if the natural design for an efficient subprocess involves independent consciousness (though it is a separate question if this optimal design involves pain or suffering).

Computationally powerful agents might contain vast numbers of trapped conscious subprocesses, qualifying this as a [ global catastrophic risk].]

"Mindcrime" is [18k]'s suggested term for scenarios in which an AI's cognitive processes are intrinsically doing moral harm, for example because the AI contains trillions of suffering conscious beings inside it.

Ways in which this might happen:

- Problem of sapient models (of humans):  Occurs naturally if the best predictive model for humans in the environment involves models that are detailed enough to be people themselves.
- Problem of sapient models (of civilizations):  Occurs naturally if the agent tries to simulate, e.g., alien civilizations that might be simulating it, in enough detail to include conscious simulations of the aliens.
- Problem of sapient subsystems:  Occurs naturally if the most efficient design for some cognitive subsystems involves creating subagents that are self-reflective, or have some other property leading to consciousness or personhood.
- Problem of sapient self-models:  If the AI is conscious or possible future versions of the AI are conscious, it might run and terminate a large number of conscious-self models in the course of considering possible self-modifications.

# Problem of sapient models (of humans):

An [10k instrumental pressure] to produce high-fidelity predictions of human beings (or to predict [ decision counterfactuals] about them, or to [ search] for events that lead to particular consequences, etcetera) may lead the AI to run computations that are unusually likely to possess personhood.

An [107 unrealistic] example of this would be [11w], where predictions are made by means that include running many possible simulations of the environment and seeing which ones best correspond to reality.  Among current machine learning algorithms, particle filters and Monte Carlo algorithms similarly involve running many possible simulated versions of a system.

It's possible that a sufficiently advanced AI to have successfully arrived at detailed models of human intelligence, would usually also be advanced enough that it never tried to use a predictable/searchable model that engaged in brute-force simulations of those models.  (Consider, e.g., that there will usually be many possible settings of a variable inside a model, and an efficient model might manipulate data representing a probability distribution over those settings, rather than ever considering one exact, specific human in toto.)

This, however, doesn't make it certain that no mindcrime will occur.  It may not take exact, faithful simulation of specific humans to create a conscious model.  An efficient model of a (spread of possibilities for a) human may still contain *enough* computations that resemble a person *enough* to create consciousness, or whatever other properties may be deserving of personhood.  Consider, in particular, an agent trying to use

Just as it almost certainly isn't necessary to go all the way down to the neural level to create a sapient being, it may be that even with some parts of a mind considered abstractly, the remainder would be computed in enough detail to imply consciousness, sapience, personhood, etcetera.

The problem of sapient models is not to be confused with [ Simulation Hypothesis] issues.  An efficient model of a human need not have subjective experience indistinguishable from that of the human (although it will be a model *of* a person who doesn't believe themselves to be a model).  The problem occurs if the model *is a person*, not if the model is *the same person* as its subject, and the latter possibility plays no role in the implication of moral harm.

Besides problems that are directly or obviously about modeling people, many other practical problems and questions can benefit from modeling other minds - e.g., reading the directions on a toaster oven in order to discern the intent of the mind that was trying to communicate how to use a toaster.  Thus, mindcrime might result from a sufficiently powerful AI trying to solve very mundane problems.

# Problem of sapient models (of civilizations)

A separate route to mindcrime comes from an advanced agent considering, in sufficient detail, the possible origins and futures of intelligent life on other worlds.  (Imagine that you were suddenly told that this version of you was actually embedded in a superintelligence that was imagining how life might evolve on a place like Earth, and that your subprocess was not producing sufficiently valuable information and was about to be shut down.  You would probably be annoyed!  We should try not to annoy other people in this way.)

Three possible origins of a [10g convergent instrumental pressure] to consider intelligent civilizations in great detail:

- Assigning sufficient probability to the existence of non-obvious extraterrestrial intelligences in Earth's vicinity, perhaps due to considering the [ Fermi Paradox].
- [ Naturalistic induction], combined with the AI considering the hypothesis that it is in a simulated environment.
- [ Logical decision theories] and utility functions that care about the consequences of the AI's decisions via instances of the AI's reference class that could be embedded inside alien simulations.

With respect to the latter two possibilities, note that the AI does not need to be considering possibilities in which the whole Earth as we know it is a simulation.  The AI only needs to consider that, among the possible explanations of the AI's current sense data and internal data, there are scenarios in which the AI is embedded in some world other than the most 'obvious' one implied by the sense data.  See also [5j] for a related hazard of the AI considering possibilities in which it is being simulated.

([2] has advocated that we shouldn't let any AI short of *extreme* levels of safety and robustness assurance consider distant civilizations in lots of detail in any case, since this means our AI might embed (a model of) a hostile superintelligence.)

# Problem of sapient subsystems:

It's possible that the most efficient system for, say, allocating memory on a local cluster, constitutes a complete reflective agent with a self-model.  Or that some of the most efficient designs for subprocesses of an AI, in general, happen to have whatever properties lead up to consciousness or whatever other properties are important to personhood.

This might possibly constitute a relatively less severe moral catastrophe, if the subsystems are sentient but [ lack a reinforcement-based pleasure/pain architecture] (since the latter is not obviously a property of the most efficient subagents).  In this case, there might be large numbers of conscious beings embedded inside the AI and occasionally dying as they are replaced, but they would not be suffering.  It is nonetheless the sort of scenario that many of us would prefer to avoid.

# Problem of sapient self-models:

The AI's models of *itself*, or of other AIs it could possibly build, might happen to be conscious or have other properties deserving of personhood.  This is worth considering as a separate possibility from building a conscious or personhood-deserving AI ourselves, when [ we didn't mean to do so], because of these two additional properties:

- Even if the AI's current design is not conscious or personhood-deserving, the current AI might consider possible future versions or subagent designs that would be conscious, and those considerations might themselves be conscious.
 - This means that even if the AI's current version doesn't seem like it has key personhood properties on its own - that we've successfully created the AI itself as a nonperson - we still need to worry about other conscious AIs being embedded into it.
- The AI might create, run, and terminate very large numbers of potential self-models.
 - Even if we consider tolerable the potential moral harm of creating *one* conscious AI (e.g. the AI lacks all of the conditions that a responsible parent would want to ensure when creating a new intelligent species, but it's just one sapient being so it's okay to do that in order to save the world), we might not want to take on the moral harm of creating *trillions* of evanescent, swiftly erased conscious beings.

# Difficulties

Trying to consider these issues is complicated by:

- [ Philosophical uncertainty] about what properties are constitutive of consciousness and which computer programs have them;
- [ Moral uncertainty] about what ([ idealized] versions of) (any particular person's) morality would consider to be the key properties of personhood;
- Our present-day uncertainty about what efficient models in advanced agents would look like.

It'd help if we knew the answers to these questions, but the fact that we don't know doesn't mean we can thereby conclude that any particular model is not a person.  (This would be some mix of [ argumentum ad ignorantiem], and [ availability bias] making us think that a scenario is unlikely when it is hard to visualize.)  In the limit of infinite computing power, the epistemically best models of humans would almost certainly involve simulating many possible versions of them; superintelligences would have [ very large amounts of computing power] and we don't know at what point we come close enough to this [ limiting property] to cross the threshold.

## Scope of potential disaster

The prospect of mindcrime is an especially alarming possibility because sufficiently advanced agents, *especially* if they are using computationally efficient models, might consider *very large numbers* of hypothetical possibilities that would count as people.  There's no limit that says that if there are seven billion people, an agent will run at most seven billion models; the agent might be considering many possibilities per individual human.  This would not be an [ astronomical disaster] since it would not (by hypothesis) wipe out our posterity and our intergalactic future, but it could be a disaster orders of magnitude larger than the Holocaust, the Mongol Conquest, the Middle Ages, or all human tragedy to date.

## Development-order issue

If we ask an AI to predict what we would say if we had a thousand years to think about the problem of defining personhood or think about which causal processes are 'conscious', this seems unusually likely to cause the AI to commit mindcrime in the course of answering the question.  Even asking the AI to think abstractly about the problem of consciousness, or predict by abstract reasoning what humans might say about it, seems unusually likely to result in mindcrime.  There thus exists a [ development order issue] preventing us from asking a Friendly AI to solve the problem for us, since to file this request safely and without committing mindcrime, we would need the request to already have been completed.

The prospect of enormous-scale disaster mitigates against 'temporarily' tolerating mindcrime inside a system, while, e.g., an [ extrapolated-volition] or [ approval-based] agent tries to compute the code or design of a non-mindcriminal agent.  Depending on the agent's efficiency, and secondarily on its computational limits, a tremendous amount of moral harm might be done during the 'temporary' process of computing an answer.

## Weirdness

Literally nobody outside of MIRI or FHI ever talks about this problem.

# Nonperson predicates

A [1fv nonperson predicate] is an [ effective] test that we, or an AI, can use to determine that some computer program is definitely *not* a person.  In principle, a nonperson predicate needs only two possible outputs, "Don't know" and "Definitely not a person".  It's acceptable for many actually-nonperson programs to be labeled "don't know", so long as no people are labeled "definitely not a person".

If the above was the only requirement, one simple nonperson predicate would be to label everything "don't know".  The implicit difficulty is that the nonperson predicate must also pass some programs of high complexity that do things like "acceptably model humans" or "acceptably model future versions of the AI".

Besides addressing mindcrime scenarios, Yudkowsky's [original proposal](http://lesswrong.com/lw/x4/nonperson_predicates/) was also aimed at knowing that the AI design itself was not conscious, or not a person.

It seems likely to be very hard to find a good nonperson predicate:

- Not all philosophical confusions and computational difficulties are averted by asking for a partial list of unconscious programs instead of a total list of conscious programs.  Even if we don't know which properties are sufficient, we'd need to know something solid about properties that are necessary for consciousness or sufficient for nonpersonhood.
- We can't pass once-and-for-all any class of programs that's Turing-complete. We can't say once and for all that it's safe to model gravitational interactions in a solar system, if enormous gravitational systems could encode computers that encode people.
- The [42] problem seems particularly worrisome here.  If we block off some options for modeling humans directly, the *next best* option is unusually likely to be conscious.  Even if we rely on a whitelist rather than a blacklist, this may lead to a whitelisted "gravitational model" that secretly encodes a human, and so on.

# Research avenues

+ [102 Behaviorism]:  Try to create a [5b3 limited AI] that does not model other minds or possibly even itself, except using some narrow class of agent models that we are pretty sure will not be sentient.  This avenue is potentially motivated for other reasons as well, such as avoiding [5j probable environment hacking] and averting [ programmer manipulation].

+ Try to define a nonperson predicate that whitelists enough programs to carry out some [6y pivotal achievement].

+ Try for an AI that can bootstrap our understanding of consciousness and tell us about what we would define as a person, while committing a relatively small amount of mindcrime, with all computed possible-people being stored rather than discarded, and the modeled agents being entirely happy, mostly happy, or non-suffering.  E.g., put a happy person at the center of the approval-directed agent, and try to oversee the AI's algorithms and ask it not to use Monte Carlo simulations if possible.

+ Ignore the problem in all pre-interstellar stages because it's still relatively small compared to astronomical stakes and therefore not worth significant losses in success probability.  (This may [ backfire] under some versions of the Simulation Hypothesis.)

+ Try to [112 finish] the philosophical problem of understanding which causal processes experience sapience (or are otherwise objects of ethical value), in the next couple of decades, to sufficient detail that it can be crisply stated to an AI, with sufficiently complete coverage that it's not subject to the [42] problem.


Task-directed AGI

[summary:  A task-based AGI or "genie" is an AGI [6h intended] to follow a series of human orders, rather than autonomously pursuing long-term goals.  A Task AGI might be easier to render safe, since:

- It's possible to [2qq query the user] before and during a Task.
- [4mn Tasks are satisficing] - they're of limited scope and can be fully accomplished using a limited effort.  (In other words, Tasks should not become more and more accomplished as more and more effort is put into them.)
- Adequately [2rz identifying] what it means to safely "cure cancer" might be simpler than adequately identifying [55 all normative value].
- Task AGIs can be limited in various ways, rather than self-improving as far as possible, so long as they can still carry out at least some [6y pivotal] Tasks.

The obvious disadvantage of a Task AGI is [2sb moral hazard] - it may tempt the users in ways that an autonomous AI would not.

The problem of making a safe Task AGI invokes numerous subtopics such as [2pf low impact], [2r8 mild optimization], and [2qp conservatism] as well as numerous standard AGI safety problems like [ goal identification] and [1fx reflective stability].]

A task-based AGI is an AGI [6h intended] to follow a series of human-originated orders, with these orders each being of limited scope - "satisficing" in the sense that they can be [4mn accomplished using bounded amounts of effort and resources] (as opposed to the goals being more and more fulfillable using more and more effort).

In [1g0 Bostrom's typology], this is termed a "Genie".  It contrasts with a "Sovereign" AGI that acts autonomously in the pursuit of long-term real-world goals.

Building a safe Task AGI might be easier than building a safe Sovereign for the following reasons:

- A Task AGI can be "online"; the AGI can potentially [2qq query the user] before and during Task performance.  (Assuming an ambiguous situation arises, and is successfully identified as ambiguous.)
- A Task AGI can potentially be [5b3 limited] in various ways, since a Task AGI doesn't need to be *as powerful as possible* in order to accomplish its limited-scope Tasks.  A Sovereign would presumably engage in all-out self-improvement.  (This isn't to say Task AGIs would automatically not self-improve, only that it's possible *in principle* to limit the power of a Task AGI to only the level required to do the targeted Tasks, *if* the associated safety problems can be solved.)
- Tasks, by assumption, are limited in scope - they can be accomplished and done, inside some limited region of space and time, using some limited amount of effort which is then complete.  (To gain this advantage, a state of Task accomplishment should not go higher and higher in preference as more and more effort is expended on it open-endedly.)
- Assuming that users can figure out [6h intended goals] for the AGI that are [55 valuable] and [6y pivotal], the [2rz identification problem] for describing what constitutes a safe performance of that Task, might be simpler than giving the AGI a [ complete description] of [55 normativity in general].  That is, the problem of communicating to an AGI an adequate description of "cure cancer" (without killing patients or causing other side effects), while still difficult, might be simpler than an adequate description of all normative value.  Task AGIs fall on the narrow side of [1vt].

Relative to the problem of building a Sovereign, trying to build a Task AGI instead might step down the problem from "impossibly difficult" to "insanely difficult", while still maintaining enough power in the AI to perform [6y pivotal acts].

The obvious disadvantage of a Task AGI is [2sb moral hazard] - it may tempt the users in ways that a Sovereign would not.  A Sovereign has moral hazard chiefly during the development phase, when the programmers and users are perhaps not yet in a position of special relative power.  A Task AGI has ongoing moral hazard as it is used.

[2] has suggested that people only confront many important problems in value alignment when they are thinking about Sovereigns, but that at the same time, Sovereigns may be impossibly hard in practice.  Yudkowsky advocates that people think about Sovereigns first and list out all the associated issues before stepping down their thinking to Task AGIs, because thinking about Task AGIs may result in premature pruning, while thinking about Sovereigns is more likely to generate a complete list of problems that can then be checked against particular Task AGI approaches to see if those problems have become any easier.

Three distinguished subtypes of Task AGI are these:

- **[6x Oracles]**, an AI that is intended to only answer questions, possibly from some restricted question set.
- **[1fy Known-algorithm AIs]**, which are not self-modifying or very weakly self-modifying, such that their algorithms and representations are mostly known and mostly stable.
- **[102 Behaviorist Genies]**, which are meant to not model human minds or model them in only very limited ways, while having great material understanding (e.g., potentially the ability to invent and deploy nanotechnology).

# Subproblems

The problem of making a safe genie invokes numerous subtopics such as [2pf low impact], [2r8 mild optimization], and [2qp conservatism] as well as numerous standard AGI safety problems like [1fx reflective stability] and safe [6c identification] of [6h intended goals].

([2mx See here for a separate page on open problems in Task AGI safety that might be ready for current research.])

Some further problems beyond those appearing in the page above are:

- **Oracle utility functions** (that make the Oracle not wish to leave its box or optimize its programmers)
- **Effable optimization** (the opposite of [9f cognitive uncontainability])
- **Online checkability**
 - Explaining things to programmers [30b without putting the programmers inside an argmax] for how well you are 'explaining' things to them
- **Transparency**
- [2s1 Do What I Mean]


Oracle

Oracles are a subtype of [6w Genies] putatively designed to safely answer questions, and *only* to answer questions.  Oracles are often assumed to be [6z Boxed] and the study of Oracles is sometimes taken to be synonymous with the study of Boxed Oracles.


Pivotal act

The term 'pivotal act' in the context of [2v AI alignment theory] is a [10l guarded term] to refer to actions that will make a large positive difference a billion years later. Synonyms include 'pivotal achievement' and 'astronomical achievement'.

We can contrast this with *existential catastrophes* (or 'x-catastrophes'), events that will make a large *negative* difference a billion years later. Collectively, this page will refer to pivotal acts and existential catastrophes as *astronomically significant events* (or 'a-events').

'Pivotal *event*' is a deprecated term for referring to astronomically significant events, and 'pivotal catastrophe' is a deprecated term for existential catastrophes. 'Pivotal' was originally used to refer to the superset (a-events), but AI alignment researchers kept running into the problem of lacking a crisp way to talk about 'winning' actions in particular, and their distinctive features.

Usage has therefore shifted such that (as of late 2021) researchers use 'pivotal' and 'pivotal act' to refer to *good* events that upset the current gameboard - events that decisively settle a [55 win], or drastically increase the probability of a win.

### Reason for guardedness

[10l Guarded definitions] are deployed where there is reason to suspect that a concept will otherwise be over-extended.  The case for having a guarded definition of 'pivotal act' (and another for 'existential catastrophe') is that, after it's been shown that event X is maybe not as important as originally thought, one side of that debate may be strongly tempted to go on arguing that, wait, really it could be "relevant" (by some [10m strained] line of possibility).

**Example 1**:  In the [70 Zermelo-Fraenkel provability oracle] dialogue, Alice and Bob consider a series of possible ways that an untrusted [6x oracle] could break an attempt to [6z box] it. We end with an extremely boxed oracle that can only output machine-checkable proofs of predefined theorems in Zermelo-Fraenkel set theory, with the proofs themselves being thrown away once machine-verified.

The dialogue then observes that there isn't currently any obvious way to save the world by finding out that particular pre-chosen theorems are provable.

It may then be tempting to argue that this device could greatly advance the field of mathematics, and that math is relevant to the AI alignment problem.  However, at least given that particular proposal for using the ZF oracle, the basic rules of the AI-development playing field would remain the same, the AI alignment problem would not be *finished* nor would it have moved on to a new phase, the world would still be in danger (neither safe nor destroyed), etcetera.

(This doesn't rule out that tomorrow some reader will think of some spectacularly clever use for a ZF oracle that *does* upset the chessboard and get us on a direct path to winning where we know what we need to do from there - and in this case MIRI would reclassify the ZF oracle as a high-priority research avenue!)

**Example 2**:  Suppose a funder, worried about the prospect of advanced AIs wiping out humanity, starting offering grants for "AI safety".  It may then be tempting to try to write papers that you know you can finish, like a paper on robotic cars [3b causing unemployment] in the trucking industry, or a paper on who holds legal liability when a factory machine crushes a worker. These have the advantage of being much less difficult problems than those involved in making something actually smarter than you be safe.

But while it's true that crushed factory workers and unemployed truckers are both, ceteris paribus, bad, they are not *existential catastrophes that transform all galaxies inside our future light cone into paperclips*, and the latter category seems worth distinguishing.

This definition needs to be guarded because there may then be a temptation for the grantseeker to argue, "Well, if AI causes unemployment, that could slow world economic growth, which will make countries more hostile to each other, which would make it harder to prevent an AI arms race."  The possibility of something ending up having a *non-zero impact* on astronomical stakes is not the same concept as events that have a *game-changing impact* on astronomical stakes.

The question is what are the largest lowest-hanging fruit in astronomical stakes, not whether something can be argued as defensible by pointing to a non-zero astronomical impact.

**Example 3**:  Suppose a [102 behaviorist genie] is restricted from modeling human minds in any great detail, but is still able to build and deploy molecular nanotechnology.  Moreover, the AI is able to understand the instruction, "Build a device for scanning human brains and running them at high speed with minimum simulation error", and is able to work out a way to do this without simulating whole human brains as test cases.  The genie is then used to upload a set of, say, fifty human researchers, and run them at 10,000-to-1 speeds.

This accomplishment would not of itself save the world or destroy it - the researchers inside the simulation would still need to solve the alignment problem, and might not succeed in doing so.

But it would (positively) *upset the gameboard* and change the major determinants of winning, compared to the default scenario where the fifty researchers are in an equal-speed arms race with the rest of the world, and don't have practically-unlimited time to check their work.  The event where the genie was used to upload the researchers and run them at high speeds would be a critical event, a hinge where the optimum strategy was drastically different before versus after that pivotal act.

**Example 4**:  Suppose a paperclip maximizer is built, self-improves, and converts everything in its future light cone into paperclips.  The fate of the universe is then settled in the negative direction, so building the paperclip maximizer was an existential catastrophe.

**Example 5**:  A mass simultaneous malfunction of robotic cars causes them to deliberately run over pedestrians in many cases.  Humanity buries its dead, picks itself up, and moves on.  This was not an existential catastrophe, even though it may have nonzero influence on future AI development.

**Discussion**: Many [10m strained arguments] for X being a pivotal act have a step where X is an input into a large pool of goodness that also has many other inputs.  A ZF provability oracle would advance mathematics, and mathematics can be useful for alignment research, but there's nothing obviously game-changing about a ZF oracle that's specialized for advancing alignment work, and it's unlikely that the effect on win probabilities would be large relative to the many other inputs into total mathematical progress.

Similarly, handling trucker disemployment would only be one factor among many in world economic growth.

By contrast, a genie that uploaded human researchers putatively would *not* be producing merely one upload among many; it would be producing the only uploads where the default was otherwise no uploads.  In turn, these uploads could do decades or centuries of unrushed serial research on the AI alignment problem, where the alternative was rushed research over much shorter timespans; and this can plausibly make the difference by itself between an AI that achieves ~100% of value versus an AI that achieves ~0% of value.  At the end of the extrapolation where we ask what difference everything is supposed to make, we find a series of direct impacts producing events qualitatively different from the default, ending in a huge percentage difference in how much of all possible value gets achieved.

By having narrow and guarded definitions of 'pivotal acts' and 'existential catastrophes', we can avoid bait-and-switch arguments for the importance of research proposals, where the 'bait' is raising the apparent importance of 'AI safety' by discussing things with large direct impacts on astronomical stakes (like a paperclip maximizer or Friendly sovereign) and the 'switch' is to working on problems of dubious astronomical impact that are inputs into large pools with many other inputs.

### 'Dealing a deck of cards' metaphor

There's a line of reasoning that goes, "But most consumers don't want general AIs, they want voice-operated assistants.  So companies will develop voice-operated assistants, not general AIs."  But voice-operated assistants are themselves not astronomically significant events; developing them doesn't prevent general AIs from being developed later.  So even though this non-astronomically-significant event precedes a more significant event, it doesn't mean we should focus on the earlier event instead.

No matter how many non-game-changing 'AIs' are developed, whether playing great chess or operating in the stock market or whatever, the underlying research process will keep churning and keep turning out other and more powerful AIs.

Imagine a deck of cards which has some aces (superintelligences) and many more non-aces.  We keep dealing through the deck until we get a black ace, a red ace, or some other card that *stops the deck from dealing any further*.

A non-ace Joker card that permanently prevents any aces from being drawn would be 'astronomically significant' (not necessarily good, but definitely astronomically significant).

A card that shifts the further distribution of aces in the deck from 10% red to 90% red would be pivotal; we could see this as a metaphor for the hoped-for result of Example 3 (uploading the researchers), even though the game is not then stopped and assigned a score.

A card that causes the deck to be dealt 1% slower or 1% faster, that eliminates a non-ace card, that adds a non-ace card, that changes the proportion of red non-ace cards, etcetera, would not be astronomically significant.  A card that raises the probability of a red ace from 50% to 51% would be highly desirable, but not pivotal - it would not qualitatively change the nature of the game.

Giving examples of non-astronomically-significant events that could precede or be easier to accomplish than astronomically significant ones doesn't change the nature of the game where we keep dealing until we get a black ace or red ace.

### Examples of possible events

Existential catastrophes:

- non-value-aligned AI is built, takes over universe
- a complete and detailed synaptic-vesicle-level scan of a human brain results in cracking the cortical and cerebellar algorithms, which rapidly leads to non-value-aligned neuromorphic AI

Potential pivotal acts:

- human intelligence enhancement powerful enough that the best enhanced humans are qualitatively and significantly smarter than the smartest non-enhanced humans
- a limited [6w Task AGI] that can:
 - upload humans and run them at speeds more comparable to those of an AI
 - prevent the origin of all hostile superintelligences (in the nice case, only temporarily and via strategies that cause only acceptable amounts of collateral damage)
 - design or deploy nanotechnology such that there exists a direct route to the operators being able to do one of the other items on this list (human intelligence enhancement, prevent emergence of hostile SIs, etc.)

Non-astronomically-significant events:

- curing cancer (good for you, but it didn't resolve the alignment problem)
- proving the Riemann Hypothesis (ditto)
- an extremely expensive way to augment human intelligence by the equivalent of 5 IQ points that doesn't work reliably on people who are already very smart
- making a billion dollars on the stock market
- robotic cars devalue the human capital of professional drivers, and mismanagement of aggregate demand by central banks plus burdensome labor market regulations is an obstacle to their re-employment

Borderline-astronomically-significant cases:

- unified world government with powerful monitoring regime for 'dangerous' technologies
- widely used gene therapy that brought anyone up to a minimum equivalent IQ of 120

### Centrality to limited AI proposals

We can view the general problem of Limited AI as having the central question: **What is a pivotal act, such that an AI which does that thing and not some other things is therefore a whole lot safer to build?**

This is not a trivial question because it turns out that most interesting things require general cognitive capabilities, and most interesting goals can require arbitrarily complicated value identification problems to pursue safely.

It's trivial to create an "AI" which is absolutely safe and can't be used for any pivotal acts.  E.g. Google Maps, or a rock with "2 + 2 = 4" painted on it.

(For arguments that Google Maps could potentially help researchers drive to work faster or that a rock could potentially be used to bash in the chassis of a hostile superintelligence, see the pages on [10l guarded definitions] and [10m strained arguments].)

### Centrality to concept of 'advanced agent'

We can view the notion of an advanced agent as "agent with enough cognitive capacity to cause an astronomically significant event, positive or negative"; the [2c advanced agent properties] are either those properties that might lead up to participation in an astronomically significant event, or properties that might play a critical role in determining the AI's trajectory and hence how the event turns out.

### Policy of focusing effort on enacting pivotal acts or preventing existential catastrophes

Obvious utilitarian argument: doing something with a big positive impact is better than doing something with a small positive impact.

In the larger context of [ effective altruism] and [ adequacy theory], the issue is a bit more complicated.  Reasoning from [ adequacy theory] says that there will often be barriers (conceptual or otherwise) to the highest-return investments.  When we find that hugely important things seem *relatively neglected* and hence promising of high marginal returns if solved, this is often because there's some conceptual barrier to running ahead and doing them.

For example: tackling the hardest problems is often much scarier (you're not sure if you can make any progress on describing a self-modifying agent that provably has a stable goal system) than 'bouncing off' to some easier, more comprehensible problem (like writing a paper about the impact of robotic cars on unemployment, where you're very sure you can in fact write a paper like that at the time you write the grant proposal).

The obvious counterargument is that perhaps you can't make progress on your problem of self-modifying agents, perhaps it's too hard.  But from this it doesn't follow that the robotic-cars paper is what we should be doing instead - the robotic cars paper only makes sense if there are *no* neglected tractable investments that have bigger relative marginal inputs into astronomically significant events.

If there are in fact some neglected tractable investments in gameboard-flipping acts, then we can expect a search for gameboard-flipping acts to turn up superior places to invest effort.  But a failure mode of this search is if we fail to cognitively guard the concept of 'pivotal act'.

In particular, if we're allowed to have indirect arguments for 'relevance' that go through big common pools of goodness like 'friendliness of nations toward each other', then the pool of interventions inside that concept is so large that it will start to include things that are optimized to be appealing under more usual metrics, such as papers that don't seem unnerving and that somebody knows they can write.

So if there's no guarded concept of research on 'pivotal' things, we will end up with very standard research being done, the sort that would otherwise be done by academia anyway, and our investment will end up having a low expected marginal impact on the final outcome.

This sort of qualitative reasoning about what is or isn't 'pivotal' wouldn't be necessary if we could put solid numbers on the impact of each intervention on the probable achievement of astronomical goods.  But that is an unlikely 'if'.  Thus, there's some cause to reason qualitatively about what is or isn't 'pivotal', as opposed to just calculating out the numbers, when we're trying to pursue [ astronomical altruism].


Boxed AI

AI-boxing is the theory that deals in machine intelligences that are allegedly safer due to allegedly having extremely restricted manipulable channels of causal interaction with the outside universe.

AI-boxing theory includes:

- The straightforward problem of building elaborate sandboxes (computers and simulation environments designed not to have any manipulable channels of causal interaction with the outside universe).
- [6r Foreseeable difficulties] whereby the remaining, limited channels of interaction may be exploited to manipulate the outside universe, especially the human operators.
- The attempt to design [5f preference frameworks] that are not incentivized to go outside the Box, not incentivized to manipulate the outside universe or human operators, and incentivized to answer questions accurately or perform whatever other activity is allegedly to be performed inside the box.

The central difficulty of AI boxing is to describe a channel which cannot be used to manipulate the human operators, but which provides information relevant enough to be [6y pivotal or game-changing] relative to larger events.  For example, it seems not unthinkable that [70 we could safely extract, from a boxed AI setup, reliable information that prespecified theorems had been proved within Zermelo-Fraenkel set theory], but there is no known way to save the world if only we could sometimes know that prespecified theorems had been reliably proven in Zermelo-Fraenkel set theory.


Zermelo-Fraenkel provability oracle

[summary: The Zermelo-Fraenkel provability [6x oracle] is a [ thought experiment] illustrating the safety-capability tradeoff: a narrowly [6z Boxed] advanced agent, wrapped in multiple layers of software simulation, that can only output purported proofs of prespecified theorems in Zermelo-Fraenkel set theory, which proofs are then routed to a Boxed [proof-verifier] which checks the proof to make sure all the derivations follow syntactically.  Then the verifier outputs a 0 or 1 saying whether the prespecified theorem was proven from the Zermelo-Fraenkel axioms.  While this *might* be restrictive enough to actually be safe, and the result trustworthy, it's not clear what [6y pivotal achievement] could be carried out thereby.]

The Zermelo-Fraenkel provability [6x oracle] is the endpoint of the following conversation:

Alice:  Let's build [6x an AI that only answers questions].  That sounds pretty safe to me. In fact, I don't understand why all you lot keep talking about building AIs that *don't* just answer questions, when the question-answering systems sound so much safer to me.

Bob:  Suppose you screw up the goal system and the AI starts wanting to do something besides answer questions?  This system is only as safe as your ability to make an AI with a [71 stable goal system] that actually implements your [6h intended goal] of "not doing anything except answering questions", which is [oracle_utility_function difficult to formalize].

Alice:  But surely it's *safer*? Surely it's *easier* to develop a goal system that just safely answers questions?

Bob:  How much safer?  How much easier?  If we hypothesize restricting the abilities of the AI in this way, we're reducing the potential payoff if we actually succeed.  An AI that can act in the world and build its own nanotechnology can potentially protect us from other AIs becoming superintelligent, at least as a temporary fix to the immediate dilemma and the immediate danger of death, if not as an eternal solution.  An AI that can act in the real world using nanotechnology would be, in our parlance, a [6y pivotal achievement].  Getting from a question-answering AI to a safer world seems a lot harder, unless the question you ask the AI is "How to do I build an intelligent, directed nanotechnological system?" in which case you might as not well bother pretending it's an Oracle.  So we're losing something by supposing that the system only answers questions - we're losing some of the pivotal-ness of the hypothetical success.  If you'll pardon the extreme example, if the Oracle system is only 0.01% safer or 99.9% as difficult to build as a Genie, then it definitely wouldn't be worth the decreased probability of being able to win the larger dilemma using a successful Oracle instead of a successful Genie.  So the question is, *how* useful is the Oracle compared to the Genie, and given that you're going to use it that way, *how* much safer is it, or easier to build, compared to the Genie?  An "Oracle" that immediately writes the code for a Genie is equally useful, but it's definitely not any easier to build safely.

Alice:  But there *are* substantial gains in safety, or in ease of building the system to adequate safety, from adopting a question-answering design.  Like, imagine that the computer is in a sealed chamber on the Moon, without any attached robots or robot arms, and just a single radio downlink to send the answers to the questions we ask.  Then we'll be safe even if the goal design fails and it turns against us, because, you know, it'll be stuck on the Moon without any hands.  To me this obviously seems a *lot* safer than, say, a Genie running around in the world with its own nanotechnology.  Less useful assuming it works, yes, but also a lot less dangerous assuming it fails.

Bob:  Are you sure you're properly [respecting the intelligence of something vastly smarter than you](respect_intelligence-1)?  We're not smart enough to [consider all the strategies it might imagine](cognitive_uncontainability-1).  What if there's some creative thing you can do even if you're in a sealed chamber on the Moon?  There are radio waves leaving the chamber that depend on the flows of electrons in transistors -

Alice:  Okay, a sealed chamber *and* we're running it in three layers of sandboxed simulation *and* we haven't told the AI about the hardware it's running on *or* about any of the emulation layers.  I think it'd be pretty hard for the AI to manipulate the radio waves in that condition.  It just wouldn't be able to figure out a relation between anything it could control, and the radio waves going out, and even if it could modulate the radio waves some tiny amount, do you really think that could somehow end the world?

[todo: finish the conversation]

**[4v]**

bob:  it can manipulate the human

alice:  we won't tell it about humans

bob:  it may be able to deduce an awful lot by looking at its own code.

alice: okay it gives a bare answer, like just a few bits.

bob:  timing problem

alice:  time it, obviously.

bob:  what the heck do you think you can do with just a few bits that's interesting?

alice:  we'll ask it which of two policies is the better policy for getting the world out of the AI hole.

bob:  but how can you trust it?

alice:  it can't coordinate with future AIs, but humans are trustworthy because we're honorable.

bob:  cognitive uncontainability zaps you!  it can totally coordinate.  we have proof thereof.  so any question it answers could just be steering the future toward a hostile AI.  you can't trust it if you screwed up the utility function.

alice: okay fine, it outputs proofs of predetermined theorems in ZF that go into a verifier which outputs 0 or 1 and is then destroyed by thermite.

bob:  I agree that this complete system might actually work.

alice:  yay!

bob:  but unfortunately it's now useless.

alice:  useless?  we could ask it if the Riemann Hypothesis is provable in ZF -

bob:  and how is that going to save the world, exactly?

alice:  it could generally advance math progress.

bob:  and now we're back at the "contributing one more thingy to a big pool" that people do when they're trying to promote their non-pivotal exciting fun thing to pivotalness.

alice:  that's not fair.  it could advance math a lot.  beyond what humans could do for two hundred years.

bob:  but the kind of math we do at MIRI frankly doesn't depend on whether RH is true.  it's very rare that we know exactly what theorem we might want to prove, and we don't really know whether it's a consequence of ZF, and if we do know, we can move on.  that's basically never happened in our whole history, except for maybe 7 days at a time.  if we'd had your magic box the whole time, there are certain fields of math that would be decades ahead, but our field would be maybe a week or a month ahead.  most of our work is in thinking up the theorems, not proving them.

alice:  so what if it could suggest theorems to prove?  like show you the form of a tiling agent?

bob:  if it's showing us the outline of a framework for AI and proving something about it, I'm now very worried that it's going to manage to smuggle in some deadly trick of decision theory that lets it take over somehow.  we've found all sorts of gotchas that could potentially do that, like Pascal's Mugging, [5j probable environment hacking], blackmail... by unrestricting it to be able to suggest which theorems it can prove, you've given it a rich output channel and a wide range of options for influencing future AI design.

points:

- supports methodology of foreseeable difficulties and its rule of explicitization and critique: when something looks easy or safe to someone, it often isn't.
- intuitive notions of what sort of restricted or limited AI should be "much easier to build" are often wrong, and this screws up the first intuition of how much safety we're gaining in exchange for putatively relinquishing some capabilities
  - by the time you correct this bias, the amount of capability you actually have to relinquish in or order to get a bunch of safety and development ease, often makes the putative AI not have any obvious use
  - it is wise to be explicit about how you intend to use the restricted AI to save the world, because trying to think of concrete details here will often reveal that anything you think of either (1) isn't actually very safe or (2) isn't actually very useful.


Stub

Stub-Class pages are tiny, often a single paragraph, or a couple of sentences.  Stubs may have been created to provide a [-5xs] for parenting, tagging, or reference purposes in a section under construction.  An editor expanding a Stub page might start over from scratch.

Contrast:

- [3rk], which is a larger page, but still visibly incomplete or disorganized.
- [4gs], for brief pages which only contain a formal definition and don't try to explain the topic.

**Also** use the tag [4v] if the page is being actively edited.

**[4yg Quality scale]**

* [4ym]
* [4gs]
* [5xq]
* [72]
* [3rk]
* [4y7]
* [4yd]
* [4yf]
* [4yl]


Real-world domain

A 'domain', in Artificial Intelligence or machine learning, is a problem class on which an AI program can be targeted.  Face recognition is a 'domain', as is playing chess, or driving a car.  Each of these domains presents characteristic options and modes of action for accomplishing the task: a Go game can be won by outputting good Go moves, a car can be moved safely to its definition by turning, accelerating, and braking.  The *real-world domain* is the superdomain that contains all of reality.  In reality, you could try to win a chess game by making funny faces to distract the opponent, and a driving problem has solutions like 'work for a salary and use the money to pay for an Uber'.  If you can reason like this then you are acting in the 'real-world domain'.

Being able to act in the real-world domain is distinct from *realistic* domains.  Driving a robotic car is a *realistic* domain: solutions must be computed in real-time, everything is continuous rather than turn-based, a deer could wander across the road at any time, sensors and knowledge are imperfect, and sometimes a tire blows out.  Even so, a standard robotic driver that faces this class of events will act through the medium of steering and acceleration.  If instead the robotic driver switched to running a hedge fund and hired a human to drive the car using the money it earned, it would have taken advantage of such a breadth of options that the only domain wide enough to contain those options is 'the real world', reality itself where anything goes so long as it works.

Acting in the real world is an [2c advanced agent property] and would likely follow from the advanced agent property of [42g Artificial General Intelligence]: the ability to learn new domains would lead to the ability to act through modalities covering many different kinds of real-world options.


Paperclip

[summary:  A 'paperclip', in the context of [2v AI alignment], is any configuration of matter which would seem very boring and pointless even from a very [7cl cosmopolitan] perspective.  If some bizarre physics catastrophe, spreading out at the speed of light, permanently transformed all matter it touched into paperclips, this would be morally equivalent to a physics catastrophe that destroys the reachable universe outright.  There is no deep missing moral insight we could have, no broadening of perspective and understanding, that would make us realize that little bent pieces of metal are a wonderful use of a cosmic endowment that could otherwise be transformed into intelligent life.  This is why we consider actual non-metaphorical paperclips as the case in point of 'paperclips'.]

A 'paperclip', in the context of [2v AI alignment], is any configuration of matter which would seem boring and [55 valueless] even from a very [7cl cosmopolitan] perspective.

If some bizarre physics catastrophe, spreading out at the speed of light, permanently transformed all matter it touched into paperclips, this would be morally equivalent to a physics catastrophe that destroys the reachable universe outright.  There is no deep missing moral insight we could have, no broadening of perspective and understanding, that would make us realize that little bent pieces of metal without any thought or internal experiences are the best possible use of our [7cy cosmic endowment].  It's true that we don't know what epiphanies may lie in the future for us, but that *particular* paperclip-epiphany seems *improbable.*  If you are tempted to argue with this statement as it applies to actual non-metaphorical paperclips, you are probably being overly contrary.  This is why we consider actual non-metaphorical paperclips as the case in point of 'paperclips'.

From our perspective, any entity that did in fact go around transforming almost all reachable matter into literal actual nonmetaphorical paperclips, would be doing something incredibly pointless; there would almost certainly be no hidden wisdom in the act that we could perceive on deeper examination or further growth of our own intellectual capacities.  By the definition of the concept, this would be equally true of anything more generally termed a [10h paperclip maximizer].  Anything claimed about 'paperclips' or a [10h 'paperclip' maximizer] (such as the claim that [1y such an entity can exist without having any special intellectual defects]) must go through without any change for actual paperclips.  Actual paperclips are meant to be a central example of 'paperclips'.

The only distinction between paperclips and 'paperclips' is that the category 'paperclips' is far *wider* than the category 'actual non-metaphorical paperclips' and includes many more specific configurations of matter.  Pencil erasers, tiny molecular smileyfaces, and [5g enormous diamond masses] are all 'paperclips'.  Even under the [1y Orthogonality Thesis], an AI maximizing actual non-metaphorical paperclips would be an improbable *actual* outcome of screwing up on [5s value alignment]; but only because there are so many other possibilities.  A 'red actual-paperclip maximizer' would be even more improbable than an actual-paperclip maximizer to find in real life; but this is not because redness is antithetical to the nature of intelligent goals.  The 'redness' clause is one more added piece of complexity in the specification that drives down the probability of that exact outcome.

The popular press has sometimes distorted the notion of a paperclip maximizer into a story about an AI running a paperclip factory that takes over the universe.  (Needless to say, the kind of AI used in a paperclip-manufacturing facility is unlikely to be a frontier research AI.)  The concept of a 'paperclip' is not that it's an explicit goal somebody foolishly gave an AI, or even a goal comprehensible in human terms at all.  To imagine a central example of a supposed paperclip maximizer, imagine a research-level AI that did not stably preserve what its makers thought was supposed to be its utility function, or an AI with a poorly specified value learning rule, etcetera; such that the configuration of matter that [47 actually happened to max out the AI's utility function] looks like a [2w tiny] string of atoms in the shape of a paperclip.


Cosmopolitan value

[summary:  'Cosmopolitan', lit. "of the city of the cosmos", intuitively refers to a broad, widely embracing standpoint that tolerates and appreciates other people (entities) whose ways may at first seem very strange to us; trying to step out of our small, parochial, local instincts.  An alien civilization might at first seem completely bizarre to us, but if we really understood what was going on and tried to open our minds and hearts, we'd see it was a galaxy no less to be valued than our own.

People who feel strongly about this 'citizen of the cosmos' perspective often start out with a strong [1rm prior] that anyone talking about *not* just letting AIs do their own thing must be taking a parochial, humans-first, carbon-chauvinist viewpoint.  To which at least some AI alignment theorists reply: "No!  You don't understand!  We *are* cosmopolitans!  We *also* grew up reading science fiction about aliens that turned out to have their own perspectives, and AIs willing to extend a hand in friendship being mistreated by carbon chauvinists!  But [10h paperclip maximizers] are really genuinely different from that!  We predict that if you got to see the use a paperclip maximizer would make of the cosmic endowment, you'd be as horrified as we are; we have a difference of empirical predictions about what happens when you run a paperclip maximizer, not a values difference about how far to widen the circle of concern."]

'Cosmopolitan', lit. "of the city of the cosmos", intuitively implies a very broad, embracing standpoint that is tolerant of other people (entities) and ways that may at first seem strange to us; trying to step out of our small, parochial, local standpoint and adopt a broader one.

From the perspective of [313 volitional metaethics], this would [3y9 normatively] cover a case where what we humans *currently* value doesn't cover as much as what we would predictably come to value\* in the limit of better knowledge, greater comprehension, longer thinking, higher intelligence, or better understanding our own natures and changing ourselves in directions we thought were right.  An alien civilization might at first seem completely bizarre to us, and hence scarce in events that we intuitively understood how to value; but if we really understood what was going on, and tried to take additional steps toward widening our circle of concern, we'd see it was a galaxy no less to be valued than our own.

From outside the perspective of any particular metaethics, the notion of 'cosmopolitan' may be viewed as more like a historical generalization about moral progress: many times in human history, we get a first look at people different from us, find their ways repugnant or just confusing, and then later on we bring these people into our circle of concern and learn that they had their own nice things even if we didn't understand those nice things.  Afterwards, in these cases, we look back and say 'moral progress has occurred'.  Anyone pointing at people and claiming they are *not* to be valued as our fellow sapients, or asserting that their ways are objectively inferior to our own, is refusing to learn this lesson of history and unable to appreciate what we would see if we could really adopt their perspective.  To be 'cosmopolitan' is to learn from this generalization, and accept in advance that other beings may have valuable lives and ways even if we don't find them immediately easy to understand.

People who've adopted this viewpoint often start out with a strong prior that anyone talking about *not* just letting AIs do their own thing, figure out their own path, and create whatever kind of intergalactic civilization they want, must have failed to learn the cosmopolitan lesson.  To which at least some AI alignment theorists reply: "No!  You don't understand!  You're completely failing to pass our [7cm Ideological Turing Test]!  We *are* cosmopolitans!  We *also* grew up reading science fiction about aliens that turned out to have their own perspectives, and AIs willing to extend a hand in friendship but being mistreated by carbon chaunivists!  We'd be *fine* with a weird and wonderful intergalactic civilization full of non-organic beings appreciating their own daily life in ways we wouldn't understand.  But [10h paperclip maximizers] *don't do that!*  We predict that if you got to see the use a paperclip maximizer would make of the cosmic endowment, if you really understood what was going on inside that universe, you'd be as horrified as we are.  You and I have a difference of empirical predictions about the consequences of running a paperclip maximizer, not a values difference about how far to widen the circle of concern."

"Fragility of Cosmopolitan Value" could denote the form of the [ Fragility of Value] / [5l] thesis that is relevant to intuitive cosmopolitans:  Agents with [7cp random utility functions] wouldn't use the cosmic endowment in ways that achieve a tiny fraction of the achievable [55 value], even in the limit of our understanding exactly what was going on and trying to take a very embracing perspective.


Random utility function

[summary:  A 'random utility function' is a utility function that's been drawn according to some simple probability measure over a logical space of formal, compact specifications for utility functions.  For example, we might say that a random utility function is a utility function specified by a random program drawn from the [5v algorithmic complexity] [27p prior] on programs.]

A 'random utility function' is a utility function selected according to some simple probability measure over a logical space of formal, compact specifications of utility functions.

For example: suppose utility functions are specified by computer programs (e.g. a program that maps an output description to a rational number).  We then draw a random computer program from the standard [4mr universal] [27p prior] on computer programs:  $2^{-\operatorname K(U)}$ where $\operatorname K(U)$ is the algorithmic complexity ([5v Kolmogorov complexity]) of the utility-specifying program $U.$

This obvious measure could be amended further to e.g. take into account non-halting programs; to not put almost all of the probability mass on extremely simple programs; to put a satisficing criterion on whether it's computationally tractable and physically possible to optimize for $U$ (as assumed in the [1y Orthogonality Thesis]); etcetera.

[5l] is the thesis that the attainable optimum of a random utility function has near-null [55 goodness] with very high probability.  That is: the attainable optimum configurations of matter for a random utility function are, with very high probability, the moral equivalent of [7ch paperclips].  This in turn implies that a [41l superintelligence] with a random utility function is with very high probability the moral equivalent of a [10h paperclip maximizer].

A 'random utility function' is *not:*

-  A utility function randomly selected from whatever distribution of utility functions may actually exist among agents within the generalized universe.  That is, a random utility function is not the utility function of a random actually-existing agent.
- A utility function with maxentropy content.  That is, a random utility function is not one that independently assigns a uniform random value between 0 and 1 to every distinguishable outcome.  (This utility function would not be tractable to optimize for--we couldn't optimize it ourselves even if somebody paid us--so it's not covered by e.g. the [1y Orthogonality Thesis].)


Cosmic endowment

The 'cosmic endowment' comprises the $\approx 4 \times 10^{20}$ estimated stars that could potentially be reached by probes originating from modern Earth, before the expanding universe carries those stars over the [cosmological horizon](https://en.wikipedia.org/wiki/Hubble_volume).  It has been [estimated](https://www.gwern.net/docs/ai/1999-bradbury-matrioshkabrains.pdf) that simply surrounding a star with solar panels would yield on the order of enough energy to perform $\approx 10^{42}$ computer operations per second.  Using 100 billion neurons x 1000 synapses per neuron x 100 operations per synapse per second as a standard handwavy estimate of the computing power required to support a human-equivalent existence--although it is doubtful that humans are using this computational power very efficiently, or that 100 ops/second could faithfully simulate one synapse--this suggests that a star could support, at minimum, $\approx 10^{25}$ human-equivalent sapient lives.  Glossing a star's lifetime as 1 billion years, and assuming the quality of posthuman existence to count for no more than 1 Quality-Adjusted Life Year (QALY), this suggests that the cosmic endowment could be used (extreme handwaving) to realize *at least* $\approx 10^{54}$ QALYs.


Directing, vs. limiting, vs. opposing

[summary:  With respect to the [2v theory] of constructing [7g1 sufficiently advanced AIs] in ways that yield [3d9 good outcomes]:

- **Direction** is when it's okay for the AI to compute plans, because the AI will end up choosing rightly;
- **Limitation** is shaping an insufficiently-aligned AI so that it doesn't run a computation, if we expect that computation to produce bad results;
- **Opposition**  is when we try to prevent the AI from successfully doing something we don't like, *assuming* the AI would act wrongly given the power to do so.

For example:

- A successfully **directed** AI, given full Internet access, will do [3d9 beneficial] things given that Internet access;
- A **limited AI**, suddenly given an Internet feed, will not do *anything* with that Internet access, because its programmers haven't [whitelisting whitelisted] this new domain as okay to think about;
- **Opposition** is [airgapping](https://en.wikipedia.org/wiki/Air_gap_(networking)) the AI from the Internet and then putting the AI's processors inside a [Faraday cage](https://en.wikipedia.org/wiki/Faraday_cage), in the hope that even if the AI *wants* to get to the Internet, the AI won't be able to, say, [produce GSM cellphone signals by modulating its memory accesses](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf).]

'Directing' versus 'limiting' versus 'opposing' is a proposed conceptual distinction between 3 ways of getting [3d9 good] outcomes and avoiding [450 bad] outcomes, when running a [7g1 sufficiently advanced Artificial Intelligence]:

- **Direction** means the AGI wants to do the right thing in a domain;
- **Limitation** is the AGI not thinking or acting in places where it's not aligned;
- **Opposition**  is when we try to prevent the AGI from successfully doing the wrong thing, *assuming* that it would act wrongly given the power to do so.

For example:

- A successfully **directed** AI, given full Internet access, will do [3d9 beneficial] things rather than [450 detrimental] things using Internet access, because it wants to do good and understands sufficiently well which actions have good or bad outcomes;
- A **limited AI**, suddenly given an Internet feed, will not do *anything* with that Internet access, because its programmers haven't [whitelisting whitelisted] this new domain for being okay to think about;
- **Opposition** is [airgapping](https://en.wikipedia.org/wiki/Air_gap_(networking)) the AI from the Internet and then putting the AI's processors inside a [Faraday cage](https://en.wikipedia.org/wiki/Faraday_cage), in the hope that even if the AI *wants* to get to the Internet, the AI won't be able to [produce GSM cellphone signals by modulating its memory accesses](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf).

A fourth category not reducible to the other three might be **stabilizing,** e.g. numerical stability of floating-point algorithms, not having memory leaks in the code, etcetera.  These are issues that a sufficiently advanced AI would fix in itself automatically, but an [1fy insufficiently advanced] AI might not, which causes problems either if early errors introduce changes that are reflectively stable later, or if we are intending to run the AI in insufficiently-advanced mode.

# Not running adversarial searches

Under the [agents_as_searches] view, we can see an agent as embodying computations that search out actions within a domain.  The [7g0 non-adversarial principle] then says that we should not be running a search which is looking for ways to hurt us.

We can achieve this desideratum in one of two ways:

- The AI is sufficiently nice / [5s value-aligned] that it's okay for the AI to think about this domain / run searches here; whatever the AI is searching for, we don't mind if that search succeeds.
- The AI is *not* aligned in this domain and therefore should not be *running searches* here--we didn't finish the alignment here, and therefore, we shouldn't be running this computation.

If both of these measures fail, then as a [strictly fallback second line of defense], we could try to make a bad search return a null result (the AI doesn't find any way to do the harmful thing).

For example:  The [7g0 non-adversarial principle] implies that if you build a [2xd shutdown button] for the AI that suspends the AI to disk when pressed, you must *also* ensure:

- That the AI *wants* there to be a shutdown button;
- That the AI *wants* the state of this shutdown switch to reflect the dynamic decisions of the human operators;
- That the AI does not *want* to influence the operators to press the switch or not press it;
- That the AI does *not* want to cause anything besides an orderly suspend-to-disk to happen after this switch is pressed.

*Or:*

- That the AI doesn't *want* to think about the shutdown button at all (e.g. because it's not inside a [whitelisting whitelisted cognitive domain]).

The first set of measures would be 'direction'--if those measures work, it's okay for the AI to think here because we think those thoughts are pointed in the right direction.  The second approach would be [7fx limitation]--we don't think the AI's computations are aligned inside this domain yet, so until then, we shouldn't be running computations there.

As a [2x4 final, strictly fallback] line of defense, you could put a big red lever in the computing center that shut off all the electricity even if the AI was failing to suspend to disk.  (But you're not relying on this working; it's just there as a maybe-it-might-work measure that might buy you a little more probable safety even if the primary lines of defense failed.)

# Relation to other non-adversarial ideas

The direction/limitation/opposition distinction can help state other ideas from the [1cv].  For example:

The principle [2x4 niceness is the first line of defense] can be rephrased as follows:  When designing an AGI, we should imagine that all 'oppositional' measures are absent or failed, and think only about 'direction' and 'limitation'.  Any oppositional measures are then added on top of that, just in case.

Similarly, the [2x] says that when thinking through our primary design for alignment, we should think as if the AGI just *will* get Internet access on some random Tuesday.  This says that we should design an AGI that is limited by [whitelisting not wanting to act in newly opened domains without some programmer action], rather than relying on the AI to be *unable* to reach the Internet until we've finished aligning it.


Non-adversarial principle

[summary:  The 'non-adversarial principle' states:  *By design, the human operators and the AGI should never come into conflict.*

Since every event inside an AI is ultimately the causal result of choices by the human programmers, we should not choose so as to run computations that are searching for a way to hurt us.  At the point the AI is even *trying* to outwit us, we've already screwed up the design; we've made a foolish use of computing power.

E.g., according to this principle, if the AI's server center has [2xd a switch that shuts off the electricity], our first thought should not be, "How do we have guards with guns defending this off-switch so the AI can't destroy it?"  Our first thought should be, "How do we make sure the AI *wants* this off-switch to exist?"]

The 'Non-Adversarial Principle' is a proposed design rule for [7g1 sufficiently advanced Artificial Intelligence] stating that:

*By design, the human operators and the AGI should never come into conflict.*

Special cases of this principle include [2x4] and [ai_wants_security The AI wants your safety measures].

According to this principle, if the AI has an off-switch, our first thought should not be, "How do we have guards with guns defending this off-switch so the AI can't destroy it?" but "How do we make sure the AI *wants* this off-switch to exist?"

If we think the AI is not ready to act on the Internet, our first thought should not be "How do we [airgapping airgap] the AI's computers from the Internet?" but "How do we construct an AI that wouldn't *try* to do anything on the Internet even if it got access?"  Afterwards we may go ahead and still not connect the AI to the Internet, but only as a fallback measure.  Like the containment shell of a nuclear power plant, the *plan* shouldn't call for the fallback measure to ever become necessary.  E.g., nuclear power plants have containment shells in case the core melts down.  But this is not because we're planning to have the core melt down on Tuesday and have that be okay because there's a containment shell.

# Why run code that does the wrong thing?

Ultimately, every event inside an AI--every RAM access and CPU instruction--is an event set in motion by our own design.  Even if the AI is modifying its own code, the modified code is a causal outcome of the original code (or the code that code wrote etcetera).  Everything that happens inside the computer is, in some sense, our fault and our choice.  Given that responsibility, we should not be constructing a computation that is *trying* to hurt us.  At the point that computation is running, we've already done something foolish--willfully shot ourselves in the foot.  Even if the AI doesn't find any way to do the bad thing, we are, at the very least, wasting computing power.

No aspect of the AI's design should ever put us in an adversarial position vis-a-vis the AI, or pit the AI's wits against our wits.  If a computation starts *looking* for a way to outwit us, then the design and methodology has *already* failed.  We just shouldn't be putting an AI in a box and then having the AI search for ways to get out of the box.  If you're building a toaster, you don't build one element that heats the toast and then add a tiny refrigerator that cools down the toast.

# Not running searches for harmful strategies

Under the [agents_as_searches] viewpoint, we can see an AI as embodying computations that search out strategies within a domain.  E.g., an AI carrying out the "[one_damn_tomato put one damn strawberry on a damn plate]" task is searching the space of imaginable strategies for a strategy such that the consequence of executing that strategy is one strawberry being on the plate.

The non-adversarial principle then says that we should at no point be running a computation which is *searching* for ways to hurt us.

Suppose you don't want the AI to go on the Internet.  If your first thought is [airgapping airgapping] the AI's hardware from the Internet or otherwise [6z boxing the AI], then what you're trying to do is ensure that *a computation searching for strategies to reach the Internet returns no result.*

If the primary plan is "the AI searches, but doesn't find anything", then we need to worry that the AI's search [9f considers some class of strategies we didn't imagine].  We are pitting our own wits against a potentially [9f weird] or [41l superhuman] search.  Even if we did successfully rule out all possibilities for escape, it would be hard for us to ever be confident that we had done so.  It would be very hard to rule out the *conceptual* possibility of some entirely unexpected side-channel attack like the AI [producing GSM cellphone signals by cleverly modulating its memory accesses](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf), or the AI [aibox_experiment persuading its human operators to let it out of the box].  (Except that neither of those two particular possibilities would be *unexpected;* any specific strategy we write down will be the kind of possibility that humans consider.)

Even if the AI's search does come up empty, you were tempting fate and wasting computing power by instantiating that search in the first place.

# Oppositional measures are fallbacks

*After* we ensure that the escape-search computation is not running in the first place, we can think about fallback [7fx oppositional] lines of defense; e.g. putting the AI's processors inside a [Faraday cage](https://en.wikipedia.org/wiki/Faraday_cage).

But the *plan* is not that the AI tries to escape and we successfully prevent the escape.  If we imagine the AI trying to escape, we ought to [rule_of_surprise feel very surprised in our imagination].  If the AI actually does try to escape, we should be crying 'What?' and tracking down which strong-seeming assumption has failed.

No matter what clever design we use or how strong it seems, we'd probably *still* be wise to [airgapping airgap] an AI under development, not say "Oh it's supposed to be safe!" and connect it to the Internet and saunter on.  But that doesn't mean we *rely* on the airgap.  It doesn't mean the plan calls for the airgap to be necessary.

Nuclear power plants have concrete containment units in case the core melts down; but the *design* is not that they melt down.  The master plan doesn't say "And then on Tuesday the core melts down, but that's fine because of the containment unit."  By design, that enormous concrete shell isn't supposed to actually ever become necessary.  And then we build it anyway, because the best-laid plans etcetera.

Similarly, when designing an AI, we should pretend that the airgap doesn't exist or that the AI will [2x suddenly get Internet access anyway] on Tuesday; our *primary* thought should be to design AI that doesn't need an airgap to be safe.  And *then* we add the airgap, making sure that we're not thinking the equivalent of "Oh, it doesn't *really* matter if the core melts down, because we've got a containment structure there anyway."

# Challenges in implementing non-adversarialism

The main difficulties foreseen so far for implementing the non-adversarial principle, tend to center around [10g] plus [42] behavior.

For example, if you build a [2xd shutdown button] for a [6w Task AGI] that suspends the AI to disk when pressed, the nonadversarial principle implies you must also ensure:

- That the AI *wants* there to be a shutdown button;
- That the AI *wants* to be suspended to disk after this button is pressed;
- That the AI *wants* the state of this shutdown button to reflect the dynamic decisions of the human operators;
- That the AI does not *want* to influence the operators to decide to not press the switch, or to press it;
- That the AI does *not* want anything *besides* an orderly suspend-to-disk to happen, or not happen, after this button is pressed.

*Or:*

- The AI does not think about or make plans involving the shutdown button, e.g. because that domain was not [whitelisting whitelisted] for cognition.
- None of the AI's other models end up reflecting the existence of the shutdown button or none of its other plans end up taking into account that part of the model.

The difficulties here center around [7g2 "You can't fetch the coffee if you're dead"].  This reasoning is very general, so [48 even if we try to make it not apply at one point, it tends to pop up somewhere else]:

- If you naively try to add in a special-case clause to the utility function for wanting a shutdown button to exist, the AI wants the shutdown button to not be pressed.
- If you successfully add a special case saying that the AI wants the button to be pressed if the humans want that button to be pressed, the AI wants the humans to not want to press the button.
- If you naively try to add in a special clause for the AI wanting to shut down after the button is pressed, the AI wants to create a [environmental_subagent subagent] to make sure the coffee gets pressed anyway.
- If you try to make an AI that [1g4 doesn't think about the shutdown button] or model it at all, this seems potentially difficult because in reality the best hypothesis to explain the world *does* contain a shutdown button.  A general search for good hypotheses may tend to create cognitive tokens that represent the shutdown button, and it's not clear (yet) how this could in general be prevented by trying to divide the world into domains.

More generally: by default a lot of *high-level* searches we do want to run, [10g have *subsearches* we'd prefer *not* to run].  If we run an agent that searches *in general* for ways to fetch the coffee, that search would, by default and if smart enough, also search for ways to prevent itself from being shut down.

How exactly to implement the non-adversarial principle is thus a major open problem.  We may need to be more clever about shaping which computations give rise to which other computations than the default "Search for any action in any domain which achieves X."

# See also

- [2x4]
- [2x The omnipotence/omniscience test]
- [nonadversarial_safety The AI should not want to defeat your safety measures]
- [7fx]


Sufficiently advanced Artificial Intelligence

[summary:  A 'sufficiently advanced' Artificial Intelligence is one smart enough that we need to start thinking about some potential difficulty being discussed.

For example:  We probably don't need to worry about an AI that tries to prevent us from pressing its off-switch, until the AI knows that (a) it has an off-switch and (b) [7g2 pressing the off-switch will prevent the AI from achieving its other goals].

In turn, this knowledge is a special case of [3nf], which might follow from learning many facts about many domains via [42g].

See [2c] for a list of some different ways a cognitive agent or algorithm could be smart enough that we would need to start doing [2v AI alignment theory] to it.]

A 'sufficiently advanced Artificial Intelligence' is a cognitive [6t agent] or cognitive algorithm with capabilities great enough that we need to think about it in a qualitatively different way from robotic cars; a machine intelligence smart enough that we need to start doing [2v AI alignment theory] to it.

For example, we probably don't need to worry about an AI that tries to prevent us from pressing its off-switch, until the AI knows that (a) it has an off-switch and (b) [7g2 pressing the off-switch will prevent the AI from achieving its other goals].

In turn, this knowledge is a special case of [3nf], which might follow from learning many facts about many domains via [42g].

The page on [2c advanced agent properties] starts to list out some of the different ways that an AI could be 'smart enough' in this sense, along with the particular problems that might be encountered with an AI that smart.


You can't get the coffee if you're dead

"You can't get the coffee if you're dead" is [18m Stuart Russell's] capsule description of why almost any agent goal implies an instrumental strategy of surviving / not being shut down--by default and barring other measures to prevent this from happening; and assuming the agent has sufficient [3nf big-picture awareness] to understand the relation between the agent's non-operation and the coffee not being brought.  See [2xd] and [10g].