Corporations vs. superintelligences

It is sometimes suggested that corporations are relevant analogies for [41l superintelligences].  To evaluate this analogy without simply falling prey to the continuum fallacy, we need to consider which specific thresholds from the standard list of [2c advanced agent properties] can reasonably be said to apply in full force to corporations.  This suggests roughly the following picture:

- Corporations generally exhibit [7mt infrahuman, par-human, or high-human] levels of ability on non-heavily-parallel tasks.  On cognitive tasks that parallelize well across massive numbers of humans being paid to work on them, corporations exhibit [7mt superhuman] levels of ability compared to an individual human.
 - In order to try and grasp the overall performance boost from organizing into a corporation, consider a Microsoft-sized corporation trying to play Go in 2010.  The corporation could potentially pick out its strongest player and so gain high-human performance, but would probably not play very far above that individual level, and so would not be able to defeat the individual world champion.  Consider also the famous chess game of Kasparov vs. The World, which Kasparov ultimately won.
 - On massively parallel cognitive tasks, corporations exhibit strongly superhuman performance; the best passenger aircraft designable by Boeing seems likely to be far superior to the best passenger aircraft that could be designed by a single engineer at Boeing.
- In virtue of being composed of humans, corporations have most of the advanced-agent properties that humans themselves do:
 - They can deploy **[7vh general intelligence]** and **[9h cross-domain consequentialism].**
 - They possess **[-3nf]** and operate in the **[-78k].**
 - They can deploy **realistic psychological models** of humans and try to deceive them.
- Also in virtue of being composed of humans, corporations are not in general **[1c0 Vingean-unpredictable,]** hence not systematically **[9f cognitively uncontainable.]**  Without constituent researchers who know secret phenomena of a domain, corporations are not **[2j strongly cognitively uncontainable.]**
- Corporations are not [6s epistemically efficient] relative to humans, except perhaps in limited domains for the extremely few such that have deployed internal prediction markets with sufficiently high participation and subsidy.  (The *stock prices* of large corporations are efficient, but the corporations aren't; often the stock price tanks after the corporation does something stupid.)
- Corporations are not [6s instrumentally efficient.]  No currently known method exists for aggregating human strategic acumen into an instrumentally efficient conglomerate the way that prediction markets try to do for epistemic predictions about near-term testable events.  It is often possible for a human to see a better strategy for accomplishing the corporation's pseudo-goals than the corporation is pursuing.
- Corporations generally exhibit little interest in fundamental cognitive self-improvement, e.g. extremely few of them have deployed internal prediction markets (perhaps since the predictions of these internal prediction markets are often embarrassing to overconfident managers).  Since corporate intelligence is almost entirely composed of humans, most of the basic algorithms running a corporation are not subject to improvement by the corporation.  Attempts to do crude analogues of this tend to, e.g., bog down the entire corporation in bureaucracy and internal regulations, rather than resulting in genetic engineering of better executives or an [428 intelligence explosion].
- Corporations have no basic speed advantage over their constituent humans, since speed does not parallelize.

Sometimes discussion of analogies between corporations and hostile superintelligences focuses on a purported misalignment with human values.

As mentioned above, corporations are  *composed of* consequentialist agents, and can often deploy consequentialist reasoning to this extent.  The humans inside the corporation are not all always pulling in the same direction, and this can lead to non-consequentialist behavior by the corporation considered as a whole; e.g. an executive may not maximize financial gain for the company out of fear of personal legal liability or just other life concerns.

On many occasions some corporations have acted psychopathically with respect to the outside world, e.g. tobacco companies.  However, even tobacco companies are still composed entirely of humans who might balk at being e.g. [10h turned into paperclips].  It is possible to *imagine* circumstances under which a Board of Directors might wedge itself into pressing a button that turned everything including themselves into paperclips.  However, acting in a unified way to pursue an interest of *the corporation* that is contrary to the non-financial personal interests of all executives *and* directors *and* employees *and* shareholders, does not well-characterize the behavior of most corporations under most circumstances.

The conditions for [7hh the coherence theorems implying consistent expected utility maximization] are not met in corporations, as they are not met in the constituent humans.  On the whole, the *strategic acumen* of big-picture corporate strategy seems to behave more like Go than like airplane design, and indeed corporations are usually strategically dumber than their smartest employee and often seem to be strategically dumber than their CEOs.  Running down the list of [2vl] suggests that corporations exhibit some such behaviors sometimes, but not all of them nor all of the time.  Corporations sometimes act like they wish to survive; but sometimes act like their executives are lazy in the face of competition.  The directors and employees of the company will not go to literally any lengths to ensure the corporation's survival, or protect the corporation's (nonexistent) representation of its utility function, or converge their decision processes toward optimality (again consider the lack of internal prediction markets to aggregate epistemic capabilities on near-term resolvable events; and the lack of any known method for agglomerating human instrumental strategies into an efficient whole).

Corporations exist in a strongly multipolar world; they operate in a context that includes other corporations of equal size, alliances of corporations of greater size, governments, an opinionated public, and many necessary trade partners, all of whom are composed of humans running at equal speed and of equal or greater intelligence and strategic acumen.  Furthermore, many of the resulting compliance pressures are applied directly to the individual personal interests of the directors and managers of the corporation, i.e., the decision-making CEO might face individual legal sanction or public-opinion sanction independently of the corporation's expected average earnings.  Even if the corporation did, e.g., successfully assassinate a rival's CEO, not all of the resulting benefits to the corporation would accrue to the individuals who had taken the greatest legal risks to run the project.

Potential strong disanalogies to a [-10h] include the following:

- A paperclip maximizer can get much stronger returns on cognitive investment and reinvestment owing to being able to optimize its own algorithms at a lower level of organization.
- A paperclip maximizer can operate in much faster serial time.
- A paperclip maximizer can scale single-brain algorithms (rather than hiring more humans to try to communicate with each other across verbal barriers, a paperclip maximizer can potentially solve problems that require one BIG brain using high internal bandwidth).
- A paperclip maximizer can scale continuous, perfectly cooperative and coordinated copies of itself as more computational power becomes available.
- Depending on the returns on cognitive investment, and the timescale on which it occurs, a paperclip maximizer undergoing an intelligence explosion can end up with a strong short-term intelligence lead on the nearest rival AI projects (e.g. because the times separating the different AI projects were measured on a human scale, with the second-leading project 2 months behind the leading project, and this time difference was amplified by many orders of magnitude by fast serial cognition once the leading AI became capable of it).
- Strongly superhuman cognition potentially leads the paperclip maximizer to rapidly overcome initial material disadvantages.
 - E.g. a paperclip maximizer that can e.g. crack protein folding to develop its own biological organisms or bootstrap nanotechnology, or that develops superhuman psychological manipulation of humans, potentially acquires a strong positional advantage over all other players in the system and can ignore game-theoretic considerations (you don't have to play the Iterated Prisoner's Dilemma if you can simply disassemble the other agent and use their atoms for something else).
- Strongly superhuman strategic acumen means the paperclip maximizer can potentially deploy tactics that literally no human has ever imagined.
- Serially fast thinking and serially fast actions can take place faster than humans (or corporations) can react.
- A paperclip maximizer is *actually* motivated to *literally* kill all opposition including all humans and turn everything within reach into paperclips.

To the extent one credits the dissimilarities above as relevant to whatever empirical question is at hand, arguing by analogy from corporations to superintelligences--especially under the banner of "corporations *are* superintelligences!"--would be an instance of the [noncentral_fallacy noncentral fallacy] or [-reference_class_tennis].  Using the analogy to argue that "superintelligences are no more dangerous than corporations" would be the "precedented therefore harmless" variation of the [-7nf].  Using the analogy to argue that "corporations are the real danger," without having previously argued out that superintelligences are harmless or that superintelligences are sufficiently improbable, would be [-derailing].


Difficulty of AI alignment

This page attempts to list basic propositions in computer science which, if they are true, would be ultimately responsible for rendering difficult the task of getting good outcomes from a [7g1 sufficiently advanced Artificial Intelligence].

[auto-summary-to-here]

# "Difficulty."

By saying that these propositions would, if true, seem to imply "difficulties", we don't mean to imply that these problems are unsolvable.  We could distinguish possible levels of "difficulty" as follows:

- The problem is straightforwardly solvable, but must in fact be solved.
- The problem is straightforwardly solvable if foreseen in advance, but does not *force* a general solution in its early manifestations--if the later problems have not been explicitly foreseen, early solutions may fail to generalize.  Projects which are not exhibiting sufficient foresight may fail to future-proof for the problem, even though it is in some sense easy.
- The problem seems solvable by applying added effort, but the need for this effort will contribute *substantial additional time or resource requirements* to the aligned version of the AGI project; implying that unsafe clones or similar projects would have an additional time advantage.  E.g., computer operating systems can be made more secure, but it adds rather more than 5% to development time and requires people willing to take on a lot of little inconveniences instead of doing things the most convenient way.  If there are enough manifested difficulties like this, and the sum of their severity is great enough, then...
 - If there is strongly believed to be a great and unavoidable resource requirement even for safety-careless AGI projects, then we have a worrisome situation in which coordination among the leading five AGI projects is required to avoid races to the bottom on safety, and arms-race scenarios where the leading projects don't trust each other are extremely bad.
 - If the probability seems great enough that "A safety-careless AGI project can be executed using few enough resources, relative to every group in the world that might have those resources and a desire to develop AGI, that there would be dozens or hundreds of such projects" then a sufficiently great [7wl added development for AI alignment] *forces* [closed_is_cooperative closed AI development scenarios].  (Because open development would give projects that skipped all the safety an insuperable time advantage, and there would be enough such projects that getting all of them to behave is impossible.  (Especially in any world where, like at present, there are billionaires with great command of computational resources who don't seem to understand [1y Orthogonality].))
- The problem seems like it should in principle have a straightforward solution, but it seems like there's a worrisome probability of screwing up along the way, meaning...
 - It requires substantial additional work and time to solve this problem reliably and know that we have solved it (see above), or
 - Feasible amounts of effort still leave a worrying residue of probability that the attempted solution contains a land mine.
- The problem seems unsolvable using realistic amounts of effort, it which case aligned-AGI designs are constrained to avoid confronting it and we must find workarounds.
- The problem seems like it ought to be solvable somehow, but we are not sure exactly how to solve it.  This could imply that...
 - Novel research and perhaps genius is required to avoid this type of failure, even with the best of good intentions;
 - This might be a kind of conceptual problem that takes a long serial time to develop, and we should get started on it sooner;
 - We should start considering alternative design pathways that would work around or avoid the difficulty, in case the problem is not solved.


Instrumental goals are almost-equally as tractable as terminal goals

[summary:  One counterargument to the [1y Orthogonality Thesis] asserts that agents with terminal preferences for goals like e.g. resource acquisition will always be much better at those goals than agents which merely try to acquire resources on the way to doing something else, like making paperclips.  A reply is that any competent agent optimizing a utility function $U_0$ must have the ability to execute many subgoals and sub-subgoals $W_1, W_2, ... W_25$ that are all conditioned on arriving at the same future, and it is not especially easier to optimize $W_25$ if you promote $W_1$ to an unconditional terminal goal.  E.g. it is not much harder to design a battery to power interstellar probes to gather resources to make paperclips than to design a battery to power interstellar probes to gather resources and hoard them.]

One counterargument to the Orthogonality Thesis asserts that agents with terminal preferences for goals like e.g. resource acquisition will always be much better at those goals than agents which merely try to acquire resources on the way to doing something else, like making paperclips.  Therefore, by filtering on real-world competent agents, we filter out all agents which do not have terminal preferences for acquiring resources.

A reply is that "figuring out how to do $W_4$ on the way to $W_3$, on the way to $W_2$, on the way to $W_1$, without that particular way of doing $W_4$ stomping on your ability to later achieve $W_2$" is such a ubiquitous idiom of cognition or supercognition that (a) any competent agent must already do that all the time, and (b) it doesn't seem like adding one more straightforward target $W_0$ to the end of the chain should usually result in greatly increased computational costs or greatly diminished ability to optimize $W_4$.

E.g. contrast the necessary thoughts of a paperclip maximizer acquiring resources in order to turn them into paperclips, and an agent with a terminal goal of acquiring and hoarding resources.

The paperclip maximizer has a terminal utility function $U_0$ which counts the number of paperclips in the universe (or rather, paperclip-seconds in the universe's history).  The paperclip maximizer then identifies a sequence of subgoals and sub-subgoals $W_1, W_2, W_3...W_N$ corresponding to increasingly fine-grained strategies for making paperclips, each of which is subject to the constraint that it doesn't stomp on the previous elements of the goal hierarchy.  (For simplicity of exposition we temporarily pretend that each goal has only one subgoal rather than a family of conjunctive and disjunctive subgoals.)

More concretely, we can imagine that $W_1$ is "get matter under my control (in a way that doesn't stop me from making paperclips with it)", that is, if we were to consider the naive or unconditional description $W_1'$ "get matter under my 'control' (whether or not I can make paperclips with it)", we are here interested in a subset of states $W_1 \subset W_1'$ such that $\mathbb E[U_0|W_1]$ is high.  Then $W_2$ might be "explore the universe to find matter (in such a way that it doesn't interfere with bringing that matter under control or turning it into paperclips)", $W_3$ might be "build interstellar probes (in such a way that ...)", and as we go further into the hierarchy we will find $W_{10}$ "gather all the materials for an interstellar probe in one place (in such a way that ...)", $W_{20}$ "lay the next 20 sets of rails for transporting the titanium cart", and $W_{25}$ "move the left controller upward".

Of course by the time we're that deep in the hierarchy, any efficient planning algorithm is making some use of real independences where we can reason relatively myopically about how to lay train tracks without worrying very much about what the cart of titanium is being used for.  (Provided that the strategies are constrained enough in domain to not include any strategies that stomp distant higher goals, e.g. the strategy "build an independent superintelligence that just wants to lay train tracks"; if the system were optimizing that broadly it would need to check distant consequences and condition on them.)

The reply would then be that, in general, any feat of superintelligence requires making a ton of big, medium-sized, and little strategies all converge on a single future state in virtue of all of those strategies having been selected sufficiently well to optimize the expectation $\mathbb E[U|W_1,W_2,...].$ for some $U.$  A ton of little and medium-sized strategies must have all managed not to collide with each other or with larger big-picture considerations.  If you can't do this much then you can't win a game of Go or build a factory or even walk across the room without your limbs tangling up.

Then there doesn't seem to be any good reason to expect an agent which is instead optimizing directly the utility function $U_1$ which is "acquire and hoard resources" to do a very much better job of optimizing $W_{10}$ or $W_{25}.$  When $W_{25}$ already needs to be conditioned in such a way as to not stomp on all the higher goals $W_2, W_3, ...$ it just doesn't seem that much less constraining to target $U_1$ versus $U_0, W_1.$  Most of the cognitive labor in the sequence does not seem like it should be going into checking for $U_0$ at the end instead of checking for $U_1$ at the end.  It should be going into, e.g., figuring out how to make any kind of interstellar probe and figuring out how to build factories.

It has not historically been the case that the most computationally efficient way to play chess is to have competing agents inside the chess algorithm trying to optimize different unconditional utility functions and bidding on the right to make moves in order to pursue their own local goal of "protect the queen, regardless of other long-term consequences" or "control the center, regardless of other long-term consequences".  What we are actually trying to get is the chess move such that, conditioning on that chess move and the sort of future chess moves we are likely to make, our chance of winning is the highest.  The best modern chess algorithms do their best to factor in anything that affects long-range consequences whenever they know about those consequences.  The best chess algorithms don't try to factor things into lots of colliding unconditional urges, because sometimes that's not how "the winning move" factors.  You can extremely often do better by doing a deeper consequentialist search that conditions multiple elements of your strategy on longer-term consequences in a way that prevents your moves from stepping on each other.  It's not very much of an exaggeration to say that this is why humans with brains that can imagine long-term consequences are smarter than, say, armadillos.

Sometimes there are subtleties we don't have the computing power to notice, we can't literally actually condition on the future.  But "to make paperclips, acquire resources and use them to make paperclips" versus "to make paperclips, acquire resources regardless of whether they can be used to make paperclips" is not subtle.  We'd expect a superintelligence that was [6s efficient relative to humans] to understand and correct at least those divergences between $W_1$ and $W_1'$ that a human could see, using at most the trivial amount of computing power represented by a human brain.  To the extent that particular choices are being selected-on over a domain that is likely to include choices with huge long-range consequences, one expends the computing power to check and condition on the long-range consequences; but a supermajority of choices shouldn't require checks of this sort; and even choices about how to design train tracks that do require longer-range checks are not going to be very much more tractable depending on whether the distant top of the goal hierarchy is something like "make paperclips" or "hoard resources".

Even supposing that there could be 5% more computational cost associated with checking instrumental strategies for stepping on "promote fun-theoretic eudaimonia", which might ubiquitously involve considerations like "make sure none of the computational processes you use to do this are themselves sentient", this doesn't mean you can't have competent agents that go ahead and spend 5% more computation.  Iit's simply the correct choice to build subagents that expend 5% more computation to maintain coordination on achieving eudaimonia, rather than building subagents that expend 5% less computation to hoard resources and never give them back.  It doesn't matter if the second kind of agents are less "costly" in some myopic sense, they are vastly less useful and indeed actively destructive.  So nothing that is choosing so as to optimize its expectation of $U_0$ will build a subagent that generally optimizes its own expectation of $U_1.$


Separation from hyperexistential risk

A principle of [2v AI alignment] that does not seem reducible to [7v8 other principles] is "The AGI design should be widely separated in the design space from any design that would constitute a hyperexistential risk".  A hyperexistential risk is a "fate worse than death", that is, any AGI whose outcome is worse than quickly killing everyone and filling the universe with [7ch paperclips].

As an example of this principle, suppose we could write a first-generation AGI which contained an explicit representation of our exact true value function $V,$ but where we were not in this thought experiment absolutely sure that we'd solved the problem of getting the AGI to align on that explicit representation of a utility function.  This would violate the principle of hyperexistential separation, because an AGI that optimizes $V$ is near in the design space to one that optimizes $-V.$  Similarly, suppose we can align an AGI on $V$ but we're not certain we've built this AGI to be immune to decision-theoretic extortion.  Then this AGI distinguishes the global minimum of $V$ as the most effective threat against it, which is something that could increase the probability of $V$-minimizing scenarios being realized.
[auto-summary-to-here]

The concern here is a special case of [ shalt-not backfire] whereby identifying a negative outcome to the system moves us closer in the design space to realizing it.

One seemingly obvious patch to avoid disutility maximization might be to give the AGI a utility function $U = V + W$ where $W$ says that the absolute worst possible thing that can happen is for a piece of paper to have written on it the SHA256 hash of "Nopenopenope" plus 17.  Then if, due to otherwise poor design permitting single-bit errors to have vast results, a cosmic ray flips the sign of the AGI's effective utility function, the AGI tiles the universe with pieces of paper like that; this is no worse than ordinary paperclips.  Similarly, any extortion against the AGI would use such pieces of paper as a threat.  $W$ then functions as a honeypot or distractor for disutility maximizers which prevents them from minimizing our own true utility.

This patch would not actually work because this is a rare special case of a utility function *[3r6 not]* being [2rb reflectively consistent].  By the same reasoning we use to add $W$ to the AI's utility function $U,$ we might expect the AGI to realize that the only thing causing this weird horrible event to happen would be that event's identification by its representation of $U,$ and thus the AGI would be motivated to delete its representation of $W$ from its successor's utility function.

A patch to the patch might be to have $W$ single out a class of event which we didn't otherwise care about, but would otherwise happen at least once on its own over the otherwise expected history of the universe.  If so, we'd need to weight $W$ relative to $V$ within $U$ such that $U$ still motivated expending only a small amount of effort on easily preventing the $W$-disvalued event, rather than all effort being spent on averting $W$ to the neglect of $V.$

A deeper solution for an early-generation [6w Task AGI] would be to *never try to explicitly represent complete human values,* especially the parts of $V$ that identify things we dislike more than death.  If you avoid [2pf impacts in general] except for operator-whitelisted impacts, then you would avoid negative impacts along the way, rather than the AI containing an explicit description of what is the worst sort of impact that needs to be avoided.  In this case, the AGI just doesn't contain the information needed to compute states of the universe that we'd consider worse than death; flipping the sign of the utility function $U,$ or subtracting components from $U$ and then flipping the sign, doesn't identify any state we consider worse than paperclips.  The AGI no longer *neighbors a hyperexistential risk in the design space;* there is no longer a short path we can take in the design space, by any simple negative miracle, to get from the AGI to a fate worse than death.

Since hyperexistential catastrophes are narrow special cases (or at least it seems this way and we sure hope so), we can avoid them much more widely than ordinary existential risks.  A Task AGI powerful enough to do anything [6y pivotal] seems unavoidably very close in the design space to something that would destroy the world if we took out all the internal limiters.  By the act of having something powerful enough to destroy the world lying around, we are closely neighboring the destruction of the world within an obvious metric on possibilities.  Anything powerful enough to save the world can be transformed by a simple negative miracle into something that (merely) destroys it.

But we don't fret terribly about how a calculator that can add 17 + 17 and get 34 is very close in the design space to a calculator that gets -34; we just try to prevent the errors that would take us there.  We try to constrain the state trajectory narrowly enough that it doesn't slop over into any "neighboring" regions.  This type of thinking is plausibly the best we can do for ordinary existential catastrophes, which occupy very large volumes of the design space near any AGI powerful enough to be helpful.

By contrast, an "I Have No Mouth And I Must Scream" scenario requires an AGI that specifically wants or identifies particular very-low-value regions of the outcome space.  Most simple utility functions imply reconfiguring the universe in a way that merely kills us; a hyperexistential catastrophe is a much smaller target.  Since hyperexistential risks can be extremely bad, we prefer to avoid even very tiny probabilities of them; and since they are narrow targets, it is reasonable to try to avoid *being anywhere near them* in the state space.  This can be seen as a kind of Murphy-proofing; we will naturally try to rigidify the state trajectory and perhaps succeed, but errors in our reasoning are likely to take us to nearby-neighboring possibilities despite our best efforts.  You would still need bad luck on top of that to end up in the particular neighborhood that denotes a hyperexistential catastrophe, but this is the type of small possibility that seems worth minimizing further.

This principle implies that *general* inference of human values should not be a target of an early-generation Task AGI.  If a [7t8 meta-utility] function $U'$ contains all of the information needed to identify all of $V,$ then it contains all of the information needed to identify minima of $V.$  This would be the case if e.g. an early-generation AGI was explicitly identifying a meta-goal along the lines of "learn all human values".  However, this consideration weighing against general value learning of true human values might not apply to e.g. a Task AGI that was learning inductively from human-labeled examples, if the labeling humans were not trying to identify or distinguish within "dead or worse" and just assigned all such cases the same "bad" label.  There are still subtleties to worry about in a case like that, by which simple negative miracles might end up identifying the true $V$ anyway in a goal-valent way.  But even on the first step of "use the same label for death and worse-than-death as events to be avoided, likewise all varieties of bad fates better than death as a type of consequence to notice and describe to human operators", it seems like we would have moved substantially further away in the design space from hyperexistential catastrophe.


Cognitive uncontainability

[summary:  An [2s agent] is cognitively uncontainable in a [ domain], relative to us, when we can't hold all of the agent's possible strategies inside our own minds, and we can't make sharp predictions about what it can and can't do.  For example, the most powerful modern chess programs, playing a human novice, would be cognitively uncontainable on the chess board (the novice can't imagine everything the chess program might do), but easily cognitively containable in the context of the larger world (the novice knows the chess program won't suddenly reach out and upset the board).  One of the most critical [2c advanced agent properties] is if an agent is cognitively uncontainable in the real world.]

[9g Vingean unpredictability] is when an agent is cognitively uncontainable because it is smarter than us: if you could predict in advance exactly where [ Deep Blue] would move, you could play chess at least as well as Deep Blue yourself by doing whatever you predicted Deep Blue would do in your shoes.

Although Vingean unpredictability is the classic way in which cognitive uncontainability can arise, other possibilities are [9d imaginable].  For instance, the AI could be operating in a [9j rich domain] and searching a different part of the search space that humans have difficulty handling, while still being dumber or less competent overall than a human.  In this case the AI's strategies might still be unpredictable to us, even while it was less effective or competent overall.  Most [ anecdotes about AI algorithms doing surprising things] can be viewed from this angle.

An extremely narrow, exhaustibly searchable domain may yield cognitive containability even for intelligence locally superior to a human's.  Even a perfect Tic-Tac-Toe player can only draw against a human who knows the basic strategies, because humans can also play perfect Tic-Tac-Toe.  Of course this is only true so long as the agent can't modulate some transistors to form a wireless radio, escape onto the Internet, and offer a nearby bystander twenty thousand dollars to punch the human in the face - in which case the agent's strategic options would have included, in retrospect, things that affected the real world; and the real world is a much more complicated domain than Tic-Tac-Toe.  There's some sense in which [9j richer domains] seem likely to feed into increased cognitive uncontainability, but it's worth remembering that every game and every computer is embedded into the extremely complicated real world.

[2j Strong cognitive uncontainability] is when the agent knows some facts we don't, that it can use to formulate strategies that we wouldn't be able to recognize in advance as successful.  From the perspective of e.g. the 11th century C.E. trying to cool their house, bringing in cool water from the nearby river to run over some nearby surfaces might be an understandable solution; but if you showed them the sketch of an air conditioner, without running the air conditioner or explaining how it worked, they wouldn't recognize this sketch as a smart solution because they wouldn't know the further facts required to see why it would work.  When an agent can win using options that we didn't imagine, couldn't invent, and wouldn't understand even if we caught a glimpse of them in advance, it is strongly cognitively uncontainable in the same way that the 21st century is strongly uncontainable from the standpoint of the 11th century.


Vingean uncertainty

[summary:  [1c0] says that you (usually) can't predict *exactly* what an entity smarter than you will do, because if you knew exactly what a smart agent would do, you would be at least that smart yourself.  If you can predict exactly what move [1bx Deep Blue] will make on a chessboard, you can play chess as well as Deep Blue by moving to the same place you predict Deep Blue would.

This doesn't mean Deep Blue's programmers were ignorant of all aspects of their creation.  They understood where Deep Blue was working to steer the board's future - that Deep Blue was trying to win (rather than lose) chess games.

"Vingean uncertainty" is the epistemic state we enter into when we consider an agent too smart for us to predict its exact actions.  In particular, we will probably become *more* confident of the agent achieving its goals - that is, become more confident of which final outcomes will result from the agent's actions - even as we become *less* confident of which exact actions the agent will take.]

> Of course, I never wrote the "important" story, the sequel about the first amplified human. Once I tried something similar. John Campbell's letter of rejection began: "Sorry—you can't write this story. Neither can anyone else."...
> "Bookworm, Run!" and its lesson were important to me. Here I had tried a straightforward extrapolation of technology, and found myself precipitated over an abyss. It's a problem writers face every time we consider the creation of intelligences greater than our own. When this happens, human history will have reached a kind of singularity—a place where extrapolation breaks down and new models must be applied—and the world will pass beyond our understanding.
> -- [Vernor Vinge](https://books.google.com/books?id=tEMQpbiboH0C&pg=PA44&lpg=PA44&dq=vinge+%22pass+beyond+our+understanding%22+%22john+campbell%22&source=bl&ots=UTTxJ7Pndr&sig=88zngfy45_he2nJePP5dd0CTuR4&hl=en&sa=X&ved=0ahUKEwjD34_wrubJAhUHzWMKHVXYAocQ6AEIHTAA#v=onepage&q=vinge%20%22pass%20beyond%20our%20understanding%22%20%22john%20campbell%22&f=false), True Names and other Dangers, p. 47.

Vingean unpredictability is a key part of how we think about a [9h consequentialist intelligence] which we believe is smarter than us in a domain.  In particular, we usually think we can't predict exactly what a smarter-than-us agent will do, because if we could predict that, we would be that smart ourselves ([1c0]).

If you could predict exactly what action [1bx Deep Blue] would take on a chessboard, you could play as well as Deep Blue by making whatever move you predicted Deep Blue would make.  It follows that Deep Blue's programmers necessarily sacrificed their ability to intuit Deep Blue's exact moves in advance, in the course of creating a superhuman chessplayer.

But this doesn't mean Deep Blue's programmers were confused about the criterion by which Deep Blue chose actions.  Deep Blue's programmers still knew in advance that Deep Blue would try to *win* rather than lose chess games.  They knew that Deep Blue would try to steer the chess board's future into a particular region that was high in Deep Blue's preference ordering over chess positions.  We can predict the *consequences* of Deep Blue's moves better than we can predict the moves themselves.

"Vingean uncertainty" is the peculiar epistemic state we enter when we're considering sufficiently intelligent programs; in particular, we become less confident that we can predict their exact actions, and more confident of the final outcome of those actions.

(Note that this rejects the claim that we are epistemically helpless and can know nothing about beings smarter than ourselves.)

Furthermore, our ability to think about agents smarter than ourselves is not limited to knowing a particular goal and predicting its achievement.  If we found a giant alien machine that seemed very well-designed, we might be able to infer the aliens were superhumanly intelligent even if we didn't know the aliens' ultimate goals.   If we saw metal pipes, we could guess that the pipes represented some stable, optimal mechanical solution which was made out of hard metal so as to retain its shape.  If we saw superconducting cables, we could guess that this was a way of efficiently transporting electrical work from one place to another, even if we didn't know what final purpose the electricity was being used for.  This is the idea behind [10g]: if we can recognize that an alien machine is efficiently harvesting and distributing energy, we might recognize it as an intelligently designed artifact in the service of *some* goal even if we don't know the goal.

# Noncontainment of belief within the action probabilities

When reasoning under Vingean uncertainty, due to our [ lack of logical omniscience], our beliefs about the consequences of the agent's actions are not fully contained in our probability distribution over the agent's actions.

Suppose that on each turn of a chess game playing against Deep Blue, I ask you to put a probability distribution on Deep Blue's possible chess moves.  If you are a rational agent you should be able to put a [1bw well-calibrated] probability distribution on these moves - most trivially, by assigning every legal move an equal probability (if Deep Blue has 20 legal moves, and you assign each move 5% probability, you are guaranteed to be well-calibrated).

Now imagine a randomized game player RandomBlue that, on each round, draws randomly from the probability distribution you'd assign to Deep Blue's move from the same chess position.  In every turn, your belief about where you'll observe RandomBlue move, is equivalent to your belief about where you'd see Deep Blue move.  But your belief about the probable end of the game is very different.  (This is only possible due to your lack of logical omniscience - you lack the computing resources to map out the complete sequence of expected moves, from your beliefs about each position.)

In particular, we could draw the following contrast between your reasoning about Deep Blue and your reasoning about RandomBlue:

- When you see Deep Blue make a move to which you assigned a low probability, you think the rest of the game will go worse for you than you expected (that is, Deep Blue will do better than you previously expected).
- When you see RandomBlue make a move that you assigned a low probability (i.e., a low probability that Deep Blue would make that move in that position), you expect to beat RandomBlue sooner than you previously expected (things will go worse for RandomBlue than your previous average expectation).

This reflects our belief in something like the [6s instrumental efficiency] of Deep Blue.  When we estimate the probability that Deep Blue makes a move $x$, we're estimating the probability that, as Deep Blue estimated each move $y$'s expected probability of winning $EU[y]$, Deep Blue found $orall y
eq x: EU[x] > EU[y]$ (neglecting the possibility of exact ties, which is unlikely with deep searches and floating-point position-value estimates).  If Deep Blue picks $z$ instead of $x$, we know that Deep Blue estimated $orall y
eq z: EU[z] > EU[y]$ and in particular that Deep Blue estimated $EU[z] > EU[x]$.  This could be because the expected worth of $x$ to Deep Blue was less than expected, but for low-probability move $z$ to be better than all other moves as well implies that $z$ had an unexpectedly high value relative to our own estimates.  Thus, when Deep Blue makes a very unexpected move, we mostly expect that Deep Blue saw an unexpectedly good move that was better than what we thought was the best available move.

In contrast, when RandomBlue makes an unexpected move, we think the random number generator happened to land on a move that we justly assigned low worth, and hence we expect to defeat RandomBlue faster than we otherwise would have.

# Features of Vingean reasoning

Some interesting features of reasoning under Vingean uncertainty:

- We may find ourselves more confident of the predicted consequences of an action than of the predicted action.
- We may be more sure about the agent's instrumental strategies than its goals.
- Due to our lack of logical omniscience, our beliefs about the agent's action-mediated relation to the environment are not screened off by our probability distribution over the system's probable next actions.
    - We update on the probable consequence of an action, and on the probable consequences of other actions not taken, after observing that the agent actually outputs that action.
- If there is a compact way to describe the previous consequences of the agent's previous actions, we might try to infer that this consequence is a *goal* of the agent.  We might then predict similar consequences in the future, even without being able to predict the agent's specific next actions.

Our expectation of Vingean unpredictability in a domain may break down [9j if the domain is extremely simple and sufficiently closed].  In this case there may be an optimal play that we already know, making superhuman (unpredictable) play impossible.

# Cognitive uncontainability

Vingean unpredictability is one of the core reasons to expect [9f cognitive uncontainability] in [2c sufficiently intelligent agents].

# Vingean reflection

[1c1] is reasoning about cognitive systems, especially cognitive systems very similar to yourself (including your actual self), under the constraint that you can't predict the exact future outputs.  Deep Blue's programmers, by reasoning about the way Deep Blue was searching through game trees, could arrive at a well-justified but abstract belief that Deep Blue was 'trying to win' (rather than trying to lose) and reasoning effectively to that end.

In [1c1] we need to make predictions about the consequence of operating an agent in an environment, without knowing the agent's exact future actions - presumably via reasoning on some more abstract level, somehow.  In [-1mq], [1c0] appears in the rule that we should talk about our successor's specific actions only inside of quantifiers.

"Vingean reflection" may be a much more general issue in the design of advanced cognitive systems than it might appear at first glance.  An agent reasoning about the consequences of *its current code*, or considering what will happen if it *spends another minute thinking,* can be viewed as doing Vingean reflection.  Vingean reflection can also be seen as the study of how a given agent *wants* thinking to occur in cognitive computations, which may be importantly different from how the agent *currently* thinks.  (If these two coincide, we say the agent is [1fx reflectively stable].)

[1mq] is presently the main line of research trying to (slowly) get started on formalizing Vingean reflection and reflective stability.


Consequentialist cognition

[summary(Gloss):  "Consequentialism" is picking out immediate actions on the basis of which future outcomes you predict will result.

E.g: Going to the airport, not because you really like airports, but because you predict that if you go to the airport now you'll be in Oxford tomorrow.  Or throwing a ball in the direction that your cerebellum predicts will lead to the future outcome of a soda can being knocked off the stump.

An extremely basic and ubiquitous idiom of cognition.]

[summary:  "Consequentialism" is the name for the backward step from preferring future outcomes to selecting current actions.

E.g:  You don't go to the airport because you really like airports; you go to the airport so that, in the future, you'll be in Oxford.  (If this sounds extremely basic and obvious, it's meant to be.)   An air conditioner isn't designed by liking metal that joins together at right angles, it's designed such that the future consequence of running the air conditioner will be cold air.

Consequentialism requires:

- Being able to predict or guess the future outcomes of different actions or policies;
- Having a way to order outcomes, ranking them from lowest to highest;
- Searching out actions that are predicted to lead to high-ranking futures;
- Outputting those actions.

One might say that humans are empirically more powerful than mice because we are better consequentialists.  If we want to eat, we can envision a spear and throw it at prey.  If we want the future consequence of a well-lit room, we can envision a solar power panel.

Many of the issues in [2v AI alignment] and the [2l safety of advanced agents] arise when a machine intelligence starts to be a consequentialist across particular interesting domains.]

Consequentialist reasoning selects policies on the basis of their predicted consequences - it does action $X$ because $X$ is forecasted to lead to preferred outcome $Y$.  Whenever we reason that an agent which prefers outcome $Y$ over $Y'$ will therefore do $X$ instead of $X',$ we're implicitly assuming that the agent has the cognitive ability to do consequentialism at least about $X$s and $Y$s.  It does means-end reasoning; it selects means on the basis of their predicted ends plus a preference over ends.

E.g:  When we [2vl infer] that a [10h paperclip maximizer] would try to [3ng improve its own cognitive abilities] given means to do so, the background assumptions include:

- That the paperclip maximizer can *forecast* the consequences of the policies "self-improve" and "don't try to self-improve";
- That the forecasted consequences are respectively "more paperclips eventually" and "less paperclips eventually";
- That the paperclip maximizer preference-orders outcomes on the basis of how many paperclips they contain;
- That the paperclip maximizer outputs the immediate action it predicts will lead to more future paperclips.

(Technically, since the forecasts of our actions' consequences will usually be uncertain, a coherent agent needs a [1fw utility function over outcomes] and not just a preference ordering over outcomes.)

The related idea of "backward chaining" is one particular way of solving the cognitive problems of consequentialism: start from a desired outcome/event/future, and figure out what intermediate events are likely to have the consequence of bringing about that event/outcome, and repeat this question until it arrives back at a particular plan/policy/action.

Many narrow AI algorithms are consequentialists over narrow domains.  A chess program that searches far ahead in the game tree is a consequentialist; it outputs chess moves based on the expected result of those chess moves and your replies to them, into the distant future of the board.

We can see one of the critical aspects of human intelligence as [cross_consequentialism cross-domain consequentialism].  Rather than only forecasting consequences within the boundaries of a narrow domain, we can trace chains of events that leap from one domain to another.  Making a chess move wins a chess game that wins a chess tournament that wins prize money that can be used to rent a car that can drive to the supermarket to get milk.  An Artificial General Intelligence that could learn many domains, and engage in consequentialist reasoning that leaped across those domains, would be a [2c sufficiently advanced agent] to be interesting from most perspectives on interestingness.  It would start to be a consequentialist about the real world.

# Pseudoconsequentialism

Some systems are [-pseudoconsequentialist] - they in some ways *behave as if* outputting actions on the basis of their leading to particular futures, without using an explicit cognitive model and explicit forecasts.

For example, natural selection has a lot of the power of a cross-domain consequentialist; it can design whole organisms around the consequence of reproduction (or rather, inclusive genetic fitness).  It's a fair approximation to say that spiders weave webs *because* the webs will catch prey that the spider can eat.  Natural selection doesn't actually have a mind or an explicit model of the world; but millions of years of selecting DNA strands that did in fact previously construct an organism that reproduced, gives an effect *sort of* like outputting an organism design on the basis of its future consequences.  (Although if the environment changes, the difference suddenly becomes clear: natural selection doesn't immediately catch on when humans start using birth control.  Our DNA goes on having been selected on the basis of the *old* future of the ancestral environment, not the *new* future of the actual world.)

Similarly, a reinforcement-learning system learning to play Pong might not actually have an explicit model of "What happens if I move the paddle here?" - it might just be re-executing policies that had the consequence of winning last time.  But there's still a future-to-present connection, a pseudo-backwards-causation, based on the Pong environment remaining fairly constant over time, so that we can sort of regard the Pong player's moves as happening *because* it will win the Pong game.

# Ubiquity of consequentialism

Consequentialism is an extremely basic idiom of optimization:

- You don't go to the airport because you really like airports; you go to the airport so that, in the future, you'll be in Oxford.
- An air conditioner is an artifact selected from possibility space such that the future consequence of running the air conditioner will be cold air.
- A butterfly, by virtue of its DNA having been repeatedly selected to *have previously* brought about the past consequence of replication, will, under stable environmental conditions, bring about the future consequence of replication.
- A rat that has previously learned a maze, is executing a policy that previously had the *consequence* of reaching the reward pellets at the end:  A series of turns or behavioral rule that was neurally reinforced in virtue of the future conditions to which it led the last time it was executed.  This policy will, given a stable maze, have the same consequence next time.
- Faced with a superior chessplayer, we enter a state of [9g Vingean uncertainty] in which we are more sure about the final consequence of the chessplayer's moves - that it wins the game - than we have any surety about the particular moves made.  To put it another way, the main abstract fact we know about the chessplayer's next move is that the consequence of the move will be winning.
- As a chessplayer becomes strongly superhuman, its play becomes [6s instrumentally efficient] in the sense that *no* abstract description of the moves takes precedence over the consequence of the move.  A weak computer chessplayer might be described in terms like "It likes to move its pawn" or "it tries to grab control of the center", but as the chess play improves past the human level, we can no longer detect any divergence from "it makes the moves that will win the game later" that we can describe in terms like "it tries to control the center (whether or not that's really the winning move)".  In other words, as a chessplayer becomes more powerful, we stop being able to describe its moves that will ever take priority over our beliefs that the moves have a certain consequence.

Anything that Aristotle would have considered as having a "final cause", or teleological explanation, without being entirely wrong about that, is something we can see through the lens of cognitive consequentialism or pseudoconsequentialism.  A plan, a design, a reinforced behavior, or selected genes:  Most of the complex order on Earth derives from one or more of these.

# Interaction with advanced safety

Consequentialism or pseudoconsequentialism, over various domains, is an [2c advanced agent property] that is a key requisite or key threshold in several issues of AI alignment and advanced safety:

- You get [47 unforeseen maxima] because the AI connected up an action you didn't think of, with a future state it wanted.
- It seems [6r foreseeable] that some issues will be [48 patch-resistant] because of the [-42] effect: after one road to the future is blocked off, the next-best road to that future is often a very similar one that wasn't blocked.
- Reasoning about [-2vl] generally relies on at least pseudoconsequentialism - they're strategies that *lead up to* or would be *expected to lead up to* improved achievement of other future goals.
   - This means that, by default, lots and lots of the worrisome or problematic convergent strategies like "resist being shut off" and "build subagents" and "deceive the programmers" arise from some degree of consequentialism, combined with some degree of [3nf grasping the relevant domains].

Above all:  The human ability to think of a future and plan ways to get there, or think of a desired result and engineer technologies to achieve it, is *the* source of humans having enough cognitive capability to be dangerous.  Most of the magnitude of the impact of an AI, such that we'd want to align in the first place, would come in a certain sense from that AI being a sufficiently good consequentialist or solving the same cognitive problems that consequentialists solve.

# Subverting consequentialism?

Since consequentialism seems tied up in so many issues, some of the proposals for making alignment easier have in some way tried to retreat from, limit, or subvert consequentialism.  E.g:

- [6x Oracles] are meant to "answer questions" rather than output actions that lead to particular goals.
- [44z Imitation-based] agents are meant to imitate the behavior of a reference human as perfectly as possible, rather than selecting actions on the basis of their consequences.

But since consequentialism is so close to the heart of why an AI would be [6y sufficiently useful] in the first place, getting rid of it tends to not be that straightforward.  E.g:

- Many proposals for [6y what to actually do] with Oracles involve asking them to plan things, with humans then executing the plans.
- An AI that [44z imitates] a human doing consequentialism must be [1v0 representing consequentialism inside itself somewhere].

Since 'consquentialism' or 'linking up actions to consequences' or 'figuring out how to get to a consequence' is so close to what would make advanced AIs useful in the first place, it shouldn't be surprising if some attempts to subvert consequentialism in the name of safety run squarely into [42k an unresolvable safety-usefulness tradeoff].

Another concern is that consequentialism may to some extent be a convergent or default outcome of optimizing anything hard enough.  E.g., although natural selection is a pseudoconsequentialist process, it optimized for reproductive capacity so hard that [2rc it eventually spit out some powerful organisms that were explicit cognitive consequentialists] (aka humans).

We might similarly worry that optimizing any internal aspect of a machine intelligence hard enough would start to embed consequentialism somewhere - policies/designs/answers selected from a sufficiently general space that "do consequentialist reasoning" is embedded in some of the most effective answers.

Or perhaps a machine intelligence might need to be consequentialist in some internal aspects in order to be [6y smart enough to do sufficiently useful things] - maybe you just can't get a sufficiently advanced machine intelligence, sufficiently early, unless it is, e.g., choosing on a consequential basis what thoughts to think about, or engaging in consequentialist engineering of its internal elements.

In the same way that [18t expected utility] is the only coherent way of making certain choices, or in the same way that natural selection optimizing hard enough on reproduction started spitting out explicit cognitive consequentialists, we might worry that consequentialism is in some sense central enough that it will be hard to subvert - hard enough that we can't easily get rid of [10g instrumental convergence] on [2vl problematic strategies] just by getting rid of the consequentialism while preserving the AI's usefulness.

This doesn't say that the research avenue of subverting consequentialism is automatically doomed to be fruitless.  It does suggest that this is a deeper, more difficult, and stranger challenge than, "Oh, well then, just build an AI with all the consequentialist aspects taken out."


Rich domain

A [7vf domain] is 'rich', relative to our own intelligence, to the extent that (1) its [ search space] is too large and irregular for all of the best strategies to be searched by our own intelligence, and (2) the known mechanics of the domain do not permit us to easily place absolute bounds against some event occurring or some goal of interest being strategically obtainable.

For the pragmatic implications, see [9f] and [9t].

**[4v]**

%%comment:

spectrum: Logical tic-tac-toe, logical chess, logical go, real-world tic-tac-toe, human brain, Internet, real world

since this is about advanced safety, we have to assume that there's a smarter-than-us intelligence that could seek out and exploit the tiniest loophole in our reasoning about how safe we are, and accordingly be safe and conservative in what we think is a 'narrow' domain, and flag every time we rely on the assumption that the agent's strategy space is narrow and therefore we can cognitively contain something smarter than us

talk about that old example of the evolutionary algorithm that used transistors to form a radio to form a clock circuit; gravity means that everything interacts with everything

paradigmatic example: That Alien Message

%%


Glossary (Value Alignment Theory)

The parent page for the definitions of words that are given a special or unusual meaning inside [2v value alignment theory].

If a word or phrase could be mistaken for ordinary English (e.g. 'value' or 'utility function'), then [9p you should create a glossary page] indicating its special meaning inside VAT, and the first use of that word in any potentially confusing context should be linked here.  While other special phrases (like Value Alignment Theory) should also be linked to their concept pages for understandability, they do not need glossary definitions apart from their existing concept pages.  However, an overloaded word like [55 'value'] needs its own brief (or lengthy) page that can be quickly consulted by somebody wondering what that word is taken to mean in the context of VAT.

See also [5b].


Programmer

The 'programmers' are the human beings creating whatever [2c advanced agent] is being talked about.

The word 'programmer' isn't meant to imply that the AI is hardcoded as a fixed algorithm and then run; it's just the compact word we use to refer to the AI's creators.  (E.g., the creation process could include explicitly coding algorithms, teaching the agent, exposing it to particular experiences, labeling its experiences for purposes of supervised learning, or any number of other nurturing processes extending over years.)

Many proposals for an AI's [ preference framework], or solving the [ value identification problem], require that the AI explicitly model the programmers and, even before then, figure out which objects inside the AI's beliefs about the environment *are* the programmers.  These are respectively the [ programmer modeling] and [ programmer identification] problems.


Logical game

In the context of [2v AI Alignment Theory], a 'logical' game is one that we are, for purposes of thought experiment, treating as having only the mathematical structure of the game as usually understood.  In real-world chess, you can potentially bribe the opposing player, drug them, shoot them, or rearrange the board when they're not looking.  In logical chess, we consider the entire universe to have shrunken to the size of the chess board plus two [Cartesian](Cartesian_agent-1) players, and we imagine that the conventional rules of chess are the absolute and unalterable laws of physics.

Thus, real-world chess is a [9j rich domain], and logical chess is not.  In fact, since everything in the real universe is constantly interacting (e.g., [ a pebble thrown on the Earth exerts a gravitational influence on the Moon]), to consider a conceptual example of something that is definitely, indisputably a narrow domain, we must generally resort to imagining *logical* (not real-world) Tic-Tac-Toe.


Almost all real-world domains are rich

The proposition that almost all real-world problems occupy rich domains, or *could* occupy rich domains so far as we know, due to the degree to which most things in the real world entangle with many other real things.

If playing a *real-world* game of chess, it's possible to:

- make a move that is especially likely to fool the opponent, given their cognitive psychology
- annoy the opponent
- try to cause a memory error in the opponent
- bribe the opponent with an offer to let them win future games
- bribe the opponent with candy
- drug the opponent
- shoot the opponent
- switch pieces on the game board when the opponent isn't looking
- bribe the referees with money
- sabotage the cameras to make it look like the opponent cheated
- [ force some poorly designed circuits to behave as a radio] so that you can break onto a nearby wireless Internet connection and build a smarter agent on the Internet who will create molecular nanotechnology and optimize the universe to make it look just like you won the chess game
- or accomplish whatever was meant to be accomplished by 'winning the game' via some entirely different path.

Since 'almost all' and 'might be' are not precise, for operational purposes this page's assertion will be taken to be, "Every [41l superintelligence] with options more complicated than those of a [70 Zermelo-Fraenkel provability Oracle], should be taken from our subjective perspective to have an at least a 1/3 probability of being [9f cognitively uncontainable]."

%%comment:
*(Work in progress)*

[todo: fill this out]

a central difficulty of one approach to Oracle research is to so drastically constrain the Oracle's options that the domain becomes strategically narrow from its perspective (and we can know this fact well enough to proceed)

gravitational influence of pebble thrown on Earth on moon, but this *seems* not usefully controllable because we *think* the AI can't possibly isolate *any* controllable effect of this entanglement.

when we build an agent based on our belief that we've found an exception to this general rule, we are violating the Omni Test.

central examples: That Alien Message, the Zermelo-Frankel oracle

%%


List

This meta tag is for pages that are basically just a list of things.


Probability theory

Probability theory is the mathematics governing quantitative degrees of belief.


Well-calibrated probabilities

A system of [1bv probability] assignments is "well-calibrated" if things to which it assigns 70% probability, happen around 70% of the time.  This is the goal to which [ bounded rationalists] aspire - not to perfectly predict everything, but to be no more or less certain than our information warrants.


Ability to read algebra

This requisite asks whether you can read a sentence that throws in algebra or a mathematical concept, without slowing down too much.  For instance, a sentence remarking that in the limit of flipping N coins each with a 1/N probability of coming up heads, the chance of never getting heads is 1/*e*.  If that *kind* of sentence is one you can read, you should mark yourself as understanding this requisite, so you will automatically be shown Arbital pages and tabs that invoke math of that level.


Bayesian update

[summary:  *Bayesian updating* or *Bayesian belief revision* is a way of changing [1rf probabilistic] beliefs, in response to evidence, in a way that obeys [-1bv] and in particular [1lz].

When we observe a piece of evidence that is more likely to be seen if X is true than if X is false, we should assign more credence to X.  For example, if I take a random object from your kitchen, and then I tell you this object is sharp, you should assign more credence than you did previously that it's a knife (even though forks are also sharp, so you won't be certain).

For more on this subject see [1zq the Arbital guide to Bayes's rule].]

A Bayesian *update* or *belief revision* is a change in [1rf probabilistic] beliefs after gaining new knowledge.  For example, after observing a patient's test result, we might revise our probability that a patient has a certain disease.  If this belief revision obeys [1lz Bayes's Rule], then it is called [1r8 Bayesian].

Bayesian belief updates have a number of other interesting properties, and exemplify many key principles of clear reasoning or [9l rationality].  Mapping Bayes's Rule onto real-life problems of encountering new evidence allows us to reproduce many intuitive features that have been suggested for "how to revise beliefs in the face of new evidence".

- [21v (Extra)ordinary claims require (extra)ordinary evidence.]
- The scientific virtues of [220 falsifiability, advance prediction, boldness, precision, and falsificationism] can be seen in a Bayesian light.


Bayes' rule

[summary:  When an observation is more likely given one state of the world than another, we should increase our credence that we're in the world that was more likely to have produced that observation.

Suppose that Professor Plum and Miss Scarlet are two suspects in a murder, and that we start out thinking that Professor Plum is twice as likely to have committed the murder as Miss Scarlet.  We then discover that the victim was poisoned.  We think that, on occasions where they do commit murders, Professor Plum is around one-fourth as likely to use poison as Miss Scarlet.  Then after observing the victim was poisoned, we should think Professor Plum is around half as likely to have committed the murder as Miss Scarlet: $2 	imes \dfrac{1}{4} = \dfrac{1}{2}.$

The quantitative rule at work here, in its various forms, is known as Bayes' rule.]

[summary(Technical): Bayes' rule (aka Bayes' theorem) is the quantitative [1bv law] governing how to [1ly revise probabilistic beliefs in response to observing new evidence].  Suppose we previously thought that the probability of $h_1$, denoted $\mathbb {P}(h_1)$, was twice as great as $\mathbb {P}(h_2)$.  Now we see a new piece of evidence, $e_0$, such that the probability of our seeing $e_0$ if $h_1$ is true (denoted by $\mathbb {P}(e_0\mid h_1)$) is one-fourth as great as $\mathbb {P}(e_0\mid h_2)$ (the probability of seeing $e_0$ if $h_2$ is true).  After observing $e_0$, we should think that $h_1$ is now half as likely as $h_2$:

$$rac{\mathbb {P}(h_1\mid e_0)}{\mathbb {P}(h_2\mid e_0)} = rac{\mathbb {P}(h_1)}{\mathbb {P}(h_2)} \cdot rac{\mathbb {P}(e_0\mid h_1)}{\mathbb {P}(e_0\mid h_2)}$$

[1zj More generally], Bayes' rule states:  $\mathbb P(\mathbf{H}\mid e) \propto \operatorname{\mathbb {P}}(e\mid \mathbf{H}) \cdot \operatorname{\mathbb {P}}(\mathbf{H}).$]

Bayes' rule (aka Bayes' theorem) is the quantitative law of [1bv probability theory] governing how to [1ly revise] probabilistic beliefs in response to observing new evidence.

You may want to start at the [1zq Guide] or the [693 Fast Intro].

# The laws of reasoning

Imagine that, as part of a clinical study, you're being tested for a rare form of cancer, which affects 1 in 10,000 people. You have no reason to believe that you are more or less likely than average to have this form of cancer. You're administered a test which is 99% accurate, both in terms of [-specificity] and [-sensitivity]: It _correctly_ detects the cancer (in patients who have it) 99% of the time, and it _incorrectly_ detects cancer (in patients who don't have it) only 1% of the  time. The test results come back positive. What's the chance that you have cancer?

Bayes' rule says that the answer is precisely a 1 in 102 chance, which is a probability a little below 1%. The remarkable thing about this is that there is only one answer: the odds of you having that type of cancer, given the above information, is _exactly_ 1 in 102; no more, no less.

%comment: (999,900 * 0.99 + 100 * 0.99) / (100 * 0.99) = (10098 / 99) = 102.  Please leave this comment here so the above paragraph is not edited to be wrong.%

This is one of the key insights of Bayes' rule: Given what you knew, and what you saw, the maximally accurate state of belief for you to be in is completely pinned down. While that belief state is quite difficult to find in practice, we know how to find it in principle. If you want your beliefs to become more accurate as you observe the world, Bayes' rule gives some hints about what you need to do.

# Learn Bayes' rule

- [1x5 __Bayes' rule: Odds form.__] Bayes' rule is simple, if you think in terms of relative odds.
- [1zm __Bayes' rule: Proportional form.__] The fastest way to say something both convincing and true about belief-updating.
- [1zh __Bayes' rule: Log-odds form.__] A simple transformation of Bayes' rule reveals tools for measuring degree of belief, and strength of evidence.
- [554 __Bayes' rule: Probabilistic form.__] The original formulation of Bayes' rule.
- [1zj __Bayes' rule: Functional form.__] Bayes' rule for continuous variables.
- [1zg __Bayes' rule: Vector form.__] For when you want to apply Bayes' rule to lots of evidence and lots of variables, all in one go.

# Implications of Bayes' rule

- [220 __A Bayesian view of scientific virtues.__] Why is it that science relies on bold, precise, and falsifiable predictions? Because of Bayes' rule, of course.
- [update_by_inches __Update by inches.__] It's virtuous to change your mind in response to overwhelming evidence. It's even more virtuous to shift your beliefs a little bit at a time, in response to *all* evidence (no matter how small).
- [1y6 __Belief revision as probability elimination.__] Update your beliefs by throwing away large chunks of probability mass.
- [552 __Shift towards the hypothesis of least surprise.__] When you see new evidence, ask: which hypothesis is *least surprised?*
- [21v __Extraordinary claims require extraordinary evidence.__] The people who adamantly claim they were abducted by aliens do provide *some* evidence for aliens. They just don't provide quantitatively *enough* evidence.
- [ __Ideal reasoning via Bayes' rule.__] Bayes' rule is to reasoning as the [https://en.wikipedia.org/wiki/Carnot_cycle Carnot cycle] is to engines: Nobody can be a perfect Bayesian, but Bayesian reasoning is still the theoretical ideal.

# Related content

- [4vr __Subjective probability.__] Probability is in the mind, not the world. If you don't know whether a tossed coin came up heads or tails, that's a fact about you, not a fact about the coin.
- [1bv __Probability theory.__] The quantification and study of objects that represent uncertainty about the world, and methods for making those representations more accurate.
- [3qq __Information theory.__] The quantification and study of information, communication, and what it means for one object to tell us about another.

# Other articles and introductions

- [http://en.wikipedia.org/wiki/Bayes%27_rule Wikipedia]
- [http://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/ Better Explained]


Arithmetical hierarchy

[summary:  The arithmetical hierarchy classifies logical statements by the number of nested clauses saying "for every object" and "there exists an object".  Statements with one "for every object" clause belong in $\Pi_1$, and statements with one "there exists an object" clause belong in $\Sigma_1$.  Saying "There exists an object x such that (some $\Pi_n$ statement treating x as a constant)" creates a $\Sigma_{n+1}$ statement.  Similarly, adding a "For every x" clause outside a $\Sigma_n$ statement creates a $\Pi_{n+1}$ statement.  Statements that can be formulated in both $\Pi_n$ and $\Sigma_n$ are said to lie in $\Delta_n$.  Some interesting consequences are that $\Pi_1$ statements are falsifiable by observation, $\Sigma_1$ statements are verifiable by observation, and statements strictly in higher classes can only be probabilistically verified by observation.]

[summary(Technical):  The arithmetical hierarchy classifies statements by the number of nested, unbounded quantifiers they contain.  The classes $\Delta_0$, $\Pi_0$, and $\Sigma_0$ are equivalent and include statements containing only bounded quantifiers, e.g. $orall x < 10: \exists y < x: x + y < 10$.  If, treating $x, y, z...$ as constants, a statement $\phi(x, y, z...)$ would be in $\Sigma_n,$ then adjoining the unbounded universal quantifiers $orall x: orall y: orall z: ... \phi(x, y, z...)$ creates a $\Pi_{n+1}$ statement.  Similarly, adjoining existential quantifiers to a $\Pi_n$ statement creates a $\Sigma_{n+1}$ statement.  Statements that can be equivalently formulated to be in both $\Pi_n$ and $\Sigma_n$ are said to lie in $\Delta_n$.  Interesting consequences include, e.g., $\Pi_1$ statements are falsifiable by simple observation, $\Sigma_1$ statements are verifiable by observation, and statements strictly in higher classes can only be probabilistically verified by observation.]

The arithmetical hierarchy classifies statements according to the number of unbounded $orall x$ and $\exists y$ quantifiers, treating adjacent quantifiers of the same type as a single quantifier.

The formula $\phi(x, y) \leftrightarrow [(x + y) = (y + x)],$ treating $x$ and $y$ as constants, contains no quantifiers and would occupy the lowest level of the hierarchy, $\Delta_0 = \Pi_0 = \Sigma_0.$  (Assuming that the operators $+$ and $=$ are themselves considered to be in $\Delta_0$, or from another perspective, that for any particular $c$ and $d$ we can verify whether $c + d = d + c$ in bounded time.)

Adjoining any number of $orall x_1: orall x_2: ...$ quantifiers to a statement that would be in $\Sigma_n$ if the $x_i$ were considered as constants, creates a statement in $\Pi_{n+1}.$  Thus, the statement $orall x: (x + 3) = (3 + x)$ is in $\Pi_1.$

Similarly, adjoining $\exists x_1: \exists x_2: ...$ to a statement in $\Pi_n$ creates a statement in $\Sigma_{n+1}.$  Thus, the statement $\exists y: orall x: (x + y) = (y + x)$ is in $\Sigma_2$, while the statement $\exists y: \exists x: (x + y) = (y + x)$ is in $\Sigma_1.$

Statements in both $\Pi_n$ and $\Sigma_n$ (e.g. because they have provably equivalent formulations belonging to both classes) are said to lie in $\Delta_n.$

Quantifiers that can be bounded by $\Delta_0$ functions of variables already introduced are ignored by this classification schema: the sentence $orall x: \exists y < x: (x + y) = (y + x)$ is said to lie in $\Pi_1$, not $\Pi_2$.  We can justify this by observing that for any particular $c,$ the statement $orall x < c: \phi(x)$ can be expanded into the non-quantified statement $\phi(0) \wedge \phi(1) ... \wedge \phi(c)$ and similarly $\exists x < c: \phi(x)$ expands to $\phi(0) ee \phi(1) ee ...$

This in turn justifies collapsing adjacent quantifiers of the same type inside the classification schema.  Since, e.g., we can uniquely encode every pair (x, y) in a single number $z = 2^x \cdot 3^y$, to say "there exists a pair (x, y)" or "for every pair (x, y)" it suffices to quantify over z encoding (x, y) with x and y less than z.

We say that $\Delta_{n+1}$ includes the entire sets $\Pi_n$ and $\Sigma_n$, since from a $\Pi_{n}$ statement we can produce a $\Pi_{n+1}$ statement just by adding an inner $\exists$ quantifier and then ignoring it, and we can obtain a $\Sigma_{n+1}$ statement from a $\Pi_{n}$ statement by adding an outer $orall$ quantifier and ignoring it, etcetera.

This means that the arithmetic hierarchy talks about *power sufficient to resolve statements*.  To say $\phi \in \Pi_n$ asserts that if you can resolve all $\Pi_n$ formulas then you can resolve $\phi$, which might potentially also be doable with less power than $\Pi_n$, but can definitely not require more power than $\Pi_n.$

# Consequences for epistemic properties

All and only statements in $\Sigma_1$ are *verifiable by observation*.  If $\phi \in \Delta_0$ then the sentence $\exists x: \phi(x)$ can be positively known by searching for and finding a single example.  Conversely, if a statement involves an unbounded universal quantifier, we can never be sure of it through simple observation because we can't observe the truth for every possible number.

All and only statements in $\Pi_1$ are *falsifiable by observation*.  If $\phi$ can be tested in bounded time, then we can falsify the whole statement $orall x: \phi(x)$ by presenting some single x of which $\phi$ is false.  Conversely, if a statement involves an unbounded existential quantifier, we can never falsify it directly through a bounded number of observations because there could always be some higher, as-yet untested number that makes the sentence true.

This doesn't mean we can't get [1ly probabilistic confirmation and disconfirmation] of sentences outside $\Sigma_1$ and $\Pi_1.$  E.g. for a $\Pi_2$ statement, "For every x there is a y", each time we find an example of a y for another x, we might become a little more confident, and if for some x we fail to find a y after long searching, we might become a little less confident in the entire statement.


Ability to read logic

This requisite asks whether you can read a sentence that throws in logical ideas and notation, without slowing down too much.  If the statement $(\exists v: orall w > v: orall x>0, y>0, z>0: x^w + y^w
eq z^w)
ightarrow ((1 = 0) ee (1 + 0 = 0 + 1))$ makes sense after a bit of staring, you should mark yourself as having this requisite.  You will then automatically be shown Arbital pages and tabs containing such notation.


Arithmetical hierarchy: If you don't read logic

The arithmetical hierarchy is a way of stratifying statements by how many "for every number" and "there exists a number" clauses they contain.

Suppose we say "2 + 1 = 1 + 2".  Since this is only a statement about three specific numbers, this statement would occupy the lowest level of the arithmetical hierarchy, which we can equivalently call $\Delta_0,$ $\Pi_0,$ or $\Sigma_0$ (the reason for using all three terms will soon become clear).

Next, suppose we say, "For all numbers x: (1 + x) = (x + 1)."  This generalizes over *all* numbers - a universal quantifier - and then makes a statement about each particular number x, that (1 + x) = (x + 1), that involves no further quantifiers and can be verified immediately.  This statement is said to be in $\Pi_1.$

Suppose we say, "There exists a y such that $y^9 = 9^y.$"  This is a single existential quantifier.  To verify it by sheer brute force, we'd need to start from 0 and then consider successive integers y, checking for each particular y whether it was true that $y^9 = 9^y.$  Since the statement has a single existential quantifier over y, surrounding a statement that for any particular y is in $\Delta_0,$ it is said to be in $\Sigma_1.$

Suppose we say, "For every number x, there exists a prime number y that is greater than x."  For any particular $c$ the statement "There is a prime number x that is greater than $c$" lies in $\Sigma_1.$  Universally quantifying over all $c,$ outside of the $\Sigma_1$ statement about any particular $c,$ creates a statement in $\Pi_2.$

Similarly, the statement "There exists a number x such that, for every number y, $(x + y) > 10^9$ would be in $\Sigma_2,$ since it adjoins a "there exists a number x..." to a statement that lies in $\Pi_1$ for any particular $x.$

Generalizing, putting a "There exists an x..." quantifier outside a $\Pi_n$ statement creates a $\Sigma_{n+1}$ statement, and putting a "For all y" quantifier outside a $\Sigma_n$ statement about y creates a $\Pi_{n+1}$ statement.

If there are equivalent ways of formulating a sentence such that it can be seen to occupy both $\Sigma_n$ and $\Pi_n$, we say that it belongs to $\Delta_n.$

# Consequences for epistemic reasoning

Statements in $\Sigma_1$ are *verifiable*.  Taking "There exists $y$ such that $y^9 = 9^y$" as the example, soon as we find any one particular $y$ such that $y^9 = 9^y,$ we can verify the central formula $y^9 = 9^y$ for that particular $y$ immediately, and then we're done.

Statements in $\Pi_1$ are *falsifiable*.  We can decisively demonstrate them to be wrong by finding a particular example where the core statement is false.

Sentences in $\Delta_1$ are those which are both falsifiable and verifiable in finite time.

$\Pi_2$ and $\Sigma_2$ statements are not definitely verifiable or falsifiable by brute force.  E.g. for a $\Pi_2$ statement, "For every x there is a y", even after we've found a y for many particular x, we haven't tested all the x; and even if we've searched some particular x and not yet found any y, we haven't yet searched all possible y.  But statements in this class can still be probabilistically supported or countersupported by examples; each time we find an example of a y for another x, we might become a little more confident, and if for some x we fail to find a y after a long time searching, we might become a little less confident.

# Subtleties

## Bounded quantifiers don't count

The statement, "For every number $x,$ there exists a prime number $y$ smaller than $x^x$" is said to lie in $\Pi_1$, not $\Pi_2$.  Since the existence statement is bounded by $x^x$, a function which can itself be computed in bounded time, in principle we could just search through every possible $y$ that is less than $x^x$ and test it in bounded time.  For any particular $x,$ such as $x = 2,$ we could indeed replace the statement "There exists a prime number $y$ less than $2^2$" with the statement "Either 0 is a prime number, or 1 is a prime number, or 2 is a prime number, or 3 is a prime number" which contains no quantifiers at all.  Thus, in general within the arithmetical hierarchy, bounded quantifiers don't count.

We similarly observe that the statement  "For every number $x,$ there exists a prime number $y$ smaller than $x^x$" is *falsifiable* - we could falsify it by exhibiting some particular constant $c,$ testing all the numbers smaller than $c^c,$ and failing to find any primes.  (As would in fact be the case if we tested $c=1.$)

## Similar adjacent quantifiers can be collapsed into a single quantifier

Since bounded quantifiers don't count, it follows more subtly that we can combine adjacent quantifiers of the same type, since there are bounded ways to *encode* multiple numbers in a single number.  For example, the numbers x and y can be encoded into a single number $z = 2^x \cdot 3^y$.  So if I want to say, "For every nonzero integers x, y, and z, it is not the case that $x^3 + y^3 = z^3$" I can actually just say, "There's no number $w$ such that there exist nonzero x, y, and z *less than w* with $w = 2^x \cdot 3^y \cdot 5^z$ and $x^3 + y^3 = z^3.$"  Thus, the three adjacent universal quantifiers over all x, y, and z can be combined.  However, if the sentence is "for all x there exists y", there's no way to translate that into a statement about a single number z, so only alike quantifiers can be collapsed in this way.

With these subtleties in hand, we can see, e.g., that Fermat's Last Theorem belongs in $\Pi_1,$ since FLT says, "For *every* w greater than 2 and x, y, z greater than 0, it's not the case that $x^w + y^w = z^w.$  This implies that like any other $\Pi_1$ statement, Fermat's Last Theorem should be falsifiable by brute force but not verifiable by brute force.  If a counterexample existed, we could eventually find it by brute force (even if it took longer than the age of the universe) and exhibit that example to decisively disprove FLT; but there's no amount of brute-force verification of particular examples that can prove the larger theorem.

## How implications interact with falsifiability and verifiability

In general, if the implication $X
ightarrow Y$ holds, then:

- If $Y$ is falsifiable, $X$ is falsifiable.
- If $X$ is verifiable, $Y$ is verifiable.

The converse implications do not hold.

As an example, consider the $\Pi_2$ statement "For every prime $x$, there is a larger prime $y$".  Ignoring the existence of proofs, this statement is unfalsifiable by direct observation.  The falsifiable $\Pi_1$ statement, "For every prime $x$, there is a larger prime $y = f(x) = 4x+1$ would if true imply the $\Pi_2$ statement."  But this doesn't make the $\Pi_2$ statement falsifiable.  Even if the $\Pi_1$ assertion about the primeness of $4x+1$ in particular is false, the $\Pi_2$ statement can still be true (as is indeed the case).  [comment:  Patrick, is there a particular reason we want this knowledge to be accessible to people who don't natively read logic?  I.e. were you making something else rely on it?]


Math 0

A reader at the **Math 0** level has only a grasp of basic arithmetic and solving problems that employ basic arithmetical concepts. Generally, they will not have any experience with [-algebra].

There's no shame in being Math 0! We all started learning at some point, and Arbital aims to provide explanations of important mathematical concepts for readers of _all_ backgrounds, including people at a Math 0 level. Even if you learned math to a Math 1 level in high school and forgot it all afterwards, we're here to help you remember what you need to know, and understand it better than you did before.

## Writing for a Math 0 audience

When writing for readers at a Math 0 level, avoid the use of any variables or special symbols whenever possible, unless you're introducing the new symbol to them in the first place. Also try to keep article text at a readability level of a seventh or eighth grade student.

Use plenty of images — giving visualizations of what's going on is a very helpful tool in teaching.

If you're having a hard time wrangling down the subject to that level of readability, perhaps reconsider the level of your tutorial — the subject might not be accessible to Math 0 readers anyway.


Math 1

A reader at the **Math 1** level has enough mathematical ability to encompass "good at math" in a colloquial sense. They know at least some basic algebra, and how to apply algebraic thinking to problems in some contexts. If you threw a simple to moderate math puzzle at them, they could probably figure out what they were supposed to do without balking and saying "nope, I don't know how to do this".

## Writing for a Math 1 audience

At this level, you can start to use letters to represent numbers as variables and manipulate them directly.


Math 2

Do you work with math on a fairly routine basis?  Do you have little trouble grasping new mathematical ideas that use language you already know?  Having this requisite would be typical of a computer programmer, a physical engineer, or someone else who routinely works with mathematically-structured ideas.  At this level, you start to see LaTeX formulas in passing, but they'll have explanations attached.


Math 3

A reader at the **Math 3** level can read the sorts of things that a research-level mathematician could — if you're Math 3, it's okay to throw LaTeX formulas straight at you, using standard notation, with a minimum of handholding.

At the Math 3 level, different schools of mathematics may have their own standard notation, so somebody who is Math 3 in one discipline or subject may not necessarily be Math 3 in another.

## Writing for a Math 3 audience

When writing for a Math 3 audience, all bets are off on readability. You can use as much formal notation as you like in order to define your point properly and clearly.


Ability to read calculus

Check off this requisite if you can read sentences containing integrals and differentiations.


Odds

[summary:  Odds are a tool for expressing *relative* chances.  If the odds of a tree in a forest being sick versus healthy are 2 : 3, this says that there are 2 sick trees for every 3 healthy trees.  (The *probability* of a tree being sick, in this case, is 2/5 or 40%.)

![](https://i.imgur.com/GVZnz2c.png?0)]

[summary(Technical): Odds express relative chances.  If the odds for X versus Y are 2 : 3, this expresses that we think that X is 2/3 = 0.666... times as likely as Y.  Clearly, odds of 6 : 9 express the same idea; odds are invariant up to multiplication by a positive factor.  When an odds ratio [1rd exhausts all the possibilities], then we can convert its components to probabilities by [1rk normalizing] them so that they sum to 1.  In the example above, the probabilities would be $2:3 = rac{2}{3+2}:rac{3}{3+2} = 0.4:0.6.$

![](https://i.imgur.com/GVZnz2c.png?0)]

Odds are a tool for expressing relative [1rf chances].  If the odds of a tree in a forest being sick versus healthy are 2 : 3, this says that there are 2 sick trees for every 3 healthy trees.  (The probability of a tree being sick, in this case, is 2/5 or 40%.)

![](https://i.imgur.com/GVZnz2c.png?0)

Odds are expressed in the form "X to Y", e.g. "7 to 9 for X versus Y", more compactly written as $7:9$.

The representation of chances as odds is often used in gambling and [-statistics].


Mutually exclusive and exhaustive

A set of propositions is "mutually exclusive and exhaustive" when exactly one of the propositions must be true.  For example, of the two propositions "The sky is blue" and "It is not the case that the sky is blue", exactly one of those must be the case.  Therefore, the [1rf probabilities] of those propositions must sum to exactly 1.

If a set $X$ of propositions is "mutually exclusive", this states that for every two distinct propositions, the probability that both of them will be true simultaneously is zero:

$orall i: orall j: i
eq j \implies \mathbb{P}(X_i \wedge X_j) = 0.$

This implies that for every two distinct propositions, the probability of their union equals the sum of their probabilities:

$\mathbb{P}(X_i ee X_j) = \mathbb{P}(X_i) + \mathbb{P}(X_j) - \mathbb{P}(X_j \wedge X_j) = \mathbb{P}(X_i) + \mathbb{P}(X_j).$

The "exhaustivity" condition states that the union of all propositions in $X,$ has probability $1$ (the probability of at least one $X_i$ happening is $1$):

$\mathbb{P}(X_1 ee X_2 ee \dots ee X_N) = 1.$

Therefore mutual exclusivity and exhaustivity imply that the probabilities of the propositions sum to 1:

$\displaystyle \sum_i \mathbb{P}(X_i) = 1.$


Probability

*Probabilities* are the central subject of the discipline of [1bv].  $\mathbb{P}(X)$ denotes our level of belief, or someone's level of belief, that the proposition $X$ is true.  In the classical and canonical representation of probability, 0 expresses absolute incredulity, and 1 expresses absolute credulity. %%knows-requisite([1mh]):  Furthermore, [1rd mutually exclusive] events have additive classical probabilities: $\mathbb{P}(X \wedge Y) = 0 \implies \mathbb{P}(X ee Y) = \mathbb{P}(X) + \mathbb{P}(Y).$%%

For the standard probability axioms, see https://en.wikipedia.org/wiki/Probability_axioms. [todo: write up page on arbital about probability axioms.]

# Notation

$\mathbb{P}(X)$ is the probability that X is true.

$\mathbb{P}(
eg X) = 1  - \mathbb{P}(X)$ is the probability that X is false.

$\mathbb{P}(X \wedge Y)$ is the probability that both X and Y are true.

$\mathbb{P}(X ee Y)$ is the probability that X or Y or both are true.

$\mathbb{P}(X|Y) := rac{\mathbb{P}(X \wedge Y}{\mathbb{P}(Y)}$ is the **[1rj conditional probability] of X given Y.**  That is, $\mathbb{P}(X|Y)$ is **the degree to which we would believe X, assuming Y to be true.**  $\mathbb{P}(yellow|banana)$ expresses "The probability that a banana is yellow." $\mathbb{P}(banana|yellow)$ expresses "The probability that a yellow thing is a banana".

# Centrality of the classical representation

While there are other ways of expressing quantitative degrees of belief, such as [1rb odds ratios], there are several especially useful properties or roles of classical probabilities that give them a central  / convergent / canonical status among possible ways of representing credence.

[1rb Odds ratios] are isomorphic to probabilities - we can readily go back and forth between a probability of 20%, and odds of 1:4.  But unlike odds ratios, probabilities have the further appealing property of being able to add the probabilities of two [1rd mutually exclusive] possibilities to arrive at the probability that one of them occurs.  The 1/6 probability of a six-sided die turning up 1, plus the 1/6 probability of a die turning up 2, equals the 1/3 probability that the die turns up 1 or 2.  The odds ratios 1:5, 1:5, and 1:2 don't have this direct relation (though we could convert to probabilities, add, and then convert back to odds ratios).

Thus, classical probabilities are uniquely the quantities that must appear in the [18v expected utilities] to weigh how much we proportionally care about the uncertain consequences of our decisions.  When an outcome has classical probability 1/3, we multiply the degree to which we care by a factor of 1/3, not by, e.g., the odds ratio 1:2.

If the amount you'd pay for a lottery ticket that paid out on 1 or 2 was more or less than twice the price you paid for a lottery ticket that only paid out on 1, or a lottery ticket that paid out on 2, then I could buy from you and sell to you a combination of lottery tickets such that you would end up with a certain loss.  This is an example of a [dutch_book Dutch book argument], which is one kind of coherence theorem that underpins classical probability and its role in choice.  (If we were dealing with actual betting and gambling, you might reply that you'd just refuse to bet on disadvantageous combinations; but in the much larger gamble that is life, "doing nothing" is just one more choice with an uncertain, probabilistic payoff.)

The combination of several such coherence theorems, most notably including the [Dutch Book arguments](http://plato.stanford.edu/entries/dutch-book/), [Cox's Theorem](https://en.wikipedia.org/wiki/Cox%27s_theorem) and its variations for probability theory, and the [Von Neumann-Morgenstern theorem](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem) (VNM) and its variations for expected utility, together give the classical probabilities between 0 and 1 a *central* status in the theory of [9l epistemic and instrumental rationality].  Other ways of representing scalar probabilities, or alternatives to scalar probability, would need to be converted or munged back into classical probabilities in order to animate [agents agents] making coherent choices.

This also suggests that bounded agents which *approximate* coherence, or at least manage to avoid blatantly self-destructive violations of coherence, might have internal mental states which can be *approximately* viewed as corresponding to classical probabilities.  Perhaps not in terms of such agents necessarily containing floating-point numbers that directly represent those probabilities internally, but at least in terms of our being able to look over the agent's behavior and deduce that they were "behaving as if" they had assigned some coherent classical probability.


Joint probability

To write "the chance that both X and Y are true" using standard [1rf] notation, we write $\mathbb{P}(X \wedge Y)$ or $\mathbb{P}(X, Y)$.