Frequency diagrams: A first look at Bayes

Bayesian reasoning is about how to revise our beliefs in the light of evidence.

We'll start by considering one scenario in which the strength of the evidence has clear numbers attached.

(Don't worry if you don't know how to solve the following problem.  We'll see shortly how to solve it.)

Suppose you are a nurse screening a set of students for a sickness called Diseasitis.%note: Literally "inflammation of the disease".%

- You know, from past population studies, that around 20% of the students will have Diseasitis at this time of year.

You are testing for Diseasitis using a color-changing tongue depressor, which usually turns black if the student has Diseasitis.

- Among patients with Diseasitis, 90% turn the tongue depressor black.
- However, the tongue depressor is not perfect, and also turns black 30% of the time for healthy students.

One of your students comes into the office, takes the test, and turns the tongue depressor black.  What is the probability that they have Diseasitis?

(*If* you think you see how to do it, you can try to solve this problem before continuing.  To quickly see if you got your answer right, you can expand the "Answer" button below; the derivation will be given shortly.)

%hidden(Answer): The probability a student with a blackened tongue depressor has Diseasitis is 3/7, roughly 43%. %

This problem can be solved a hard way or a clever easy way.  We'll walk through the hard way first.

First, we imagine a population of 100 students, of whom 20 have Diseasitis and 80 don't.%%note:Multiple studies show that thinking about concrete numbers such as "20 out of 100 students" or "200 out of 1000 students" is more likely to produce correct spontaneous reasoning on these problems than thinking about percentages like "20% of students." E.g. "[Probabilistic reasoning in clinical medicine](https://faculty.washington.edu/jmiyamot/p548/eddydm%20prob%20reas%20i%20clin%20medicine.pdf)" by David M. Eddy (1982).%%

![prior frequency](https://i.imgur.com/gvcoqQN.png)

90% of sick students turn their tongue depressor black, and 30% of healthy students turn the tongue depressor black.  So we see black tongue depressors on 90% * 20 = 18 sick students, and 30% * 80 = 24 healthy students.

![posterior frequency](https://i.imgur.com/0XDbrYi.png)

What's the probability that a student with a black tongue depressor has Diseasitis? From the diagram, there are 18 sick students with black tongue depressors.  18 + 24 = 42 students in total turned their tongue depressors black.  Imagine reaching into a bag of all the students with black tongue depressors, and pulling out one of those students at random; what's the chance a student like that is sick?

![conditional probability](http://i.imgur.com/W6avfHO.png?1)

The final answer is that a patient with a black tongue depressor has an 18/42 = 3/7 = 43% probability of being sick.

Many medical students have at first found this answer counter-intuitive:  The test correctly detects Diseasitis 90% of the time! If the test comes back positive, why is it still less than 50% likely that the patient has Diseasitis? Well, the test also incorrectly "detects" Diseasitis 30% of the time in a healthy patient, and we start out with lots more healthy patients than sick patients.

The test does provide *some* evidence in favor of of the patient being sick.  The probability of a patient being sick goes from 20% before the test, to 43% after we see the tongue depressor turn black.  But this isn't conclusive, and we need to perform further tests, maybe more expensive ones.

*If* you feel like you understand this problem setup, consider trying to answer the following question before proceeding: What's the probability that a student who does *not* turn the tongue depressor black - a student with a negative test result - has Diseasitis?  Again, we start out with 20% sick and 80% healthy students, 70% of healthy students will get a negative test result, and only 10% of sick students will get a negative test result.

%%hidden(Answer):Imagine 20 sick students and 80 healthy students.  10% * 20 = 2 sick students have negative test results.  70% * 80 = 56 healthy students have negative test results.  Among the 2+56=58 total students with negative test results, 2 students are sick students with negative test results.  So 2/58 = 1/29 = 3.4% of students with negative test results have Diseasitis.%%

%if-before([1x1]): Now let's turn to a faster, easier way to solve the same problem.%
%!if-before([1x1]): For a more clever way to perform the same calculation, see [1x1].%


Frequency diagram

[summary:  Frequency diagrams, like [1wy waterfall diagrams], provide a way of visualizing [1lz Bayes' Rule].  For example, if 20% of the patients in the screening patient are sick (red) and 80% are healthy (blue); and 90% of the sick patients get positive test results; and 30% of the healthy patients get positive test results, we could visualize the probabilities as proportions of a large population, as in the following frequency diagram:

![Frequency diagram](https://i.imgur.com/0XDbrYi.png)

Then, when asking about the probability that a patient with a black tongue depressor is sick, we can divide the number of sick patients with black tongue depressors (18 in this case) by the number of total patients with black tongue depressors (18 + 24, in this case).]

Frequency diagrams, like [1wy waterfall diagrams], provide a way of visualizing [1lz Bayes' Rule].  For example, if 20% of the patients in the screening patient are sick (red) and 80% are healthy (blue); and 90% of the sick patients get positive test results; and 30% of the healthy patients get positive test results, we could visualize the probabilities as proportions of a large population, as in the following frequency diagram:

![Frequency diagram](https://i.imgur.com/0XDbrYi.png)

See [55z] for a walkthrough of the diagram.


Odds: Introduction

Lets say we have a bag containing twice as many blue marbles as red marbles. Then, if you reach in without looking and pick out a marble at random, the odds are 2 : 1 in favor of drawing a blue marble as opposed to a red one.

Odds express *relative* quantities. 2 : 1 odds are the same as 4 : 2 odds are the same as 600 : 300 odds. For example, if the bag contains 1 red marble and 2 blue marbles, or 2 red marbles and 4 blue marbles, then your chance of pulling out a red marble is the same in both cases:

![](https://i.imgur.com/IcsOXl0.png?0)

In other words, given odds of $(x : y)$ we can scale it by any positive number $lpha$ to get equivalent odds of $(lpha x : lpha y).$

# Converting odds to probabilities

If there were also green marbles, the *relative odds* for red *versus* blue would still be (1 : 2), but the *probability* of drawing a red marble would be lower.

![](https://i.imgur.com/ooyn9Py.png?0)

If red, blue, and green are the only kinds of marbles in the bag, then we can turn odds of $(r : b : g)$ into probabilities $(p_r : p_b : p_g)$ that say the probability of drawing each kind of marble.  Because red, blue, and green are the only possibilities, $p_r + p_g + p_b$ must equal 1, so $(p_r : p_b : p_g)$ must be odds equivalent to $(r : b : g)$ but "normalized" such that it sums to one. For example, $(1 : 2 : 1)$ would normalize to $rac{1}{4} : rac{2}{4} : rac{1}{4},$ which are the probabilities of drawing a red / blue / green marble (respectively) from the bag on the right above.

Note that if red and blue are not the only possibilities, then it doesn't make sense to convert the odds $(r : b)$ of red vs blue into a probability. For example, if there are 100 green marbles, one red marble, and two blue marbles, then the odds of red vs blue are 1 : 2, but the probability of drawing a red marble is much lower than 1/3! Odds can only be converted into probabilities if its terms are [-1rd].

Imagine a forest with some sick trees and some healthy trees, where the odds of a tree being sick (as opposed to heathy) are (2 : 3), and every tree is either sick or healthy (there are no in-between states). Then the probability of randomly picking a sick tree from among *all* trees is 2 / 5, because 2 out of every (2 + 3) trees is sick.

![](https://i.imgur.com/GVZnz2c.png?0)

In general, the operation we're doing here is taking relative odds like $(a : b : c \ldots)$ and dividing each term by the sum $(a + b + c \ldots)$ to produce $$\left(rac{a}{a + b + c \ldots} : rac{b}{a + b + c \ldots} : rac{c}{a + b + c \ldots}\ldots
ight)$$ Dividing each term by the sum of all terms gives us an equivalent set of odds (because each element is divided by the same amount) whose terms sum to 1.

This process of dividing a set of odds by the sum of its terms to get a set of probabilities that sum to 1 is called [1rk normalization].

# Converting probabilities to odds

Let's say we have two events R and B, which might be things like "I draw a red marble" and "I draw a blue marble." Say $\mathbb P(R) = rac{1}{4}$ and $\mathbb P(B) = rac{1}{2}.$ What are the odds of R vs B? $\mathbb P(R) : \mathbb P(B) = \left(rac{1}{4} : rac{1}{2}
ight),$ of course.

Equivalently, we can take the odds $\left(rac{\mathbb P(R)}{\mathbb P(B)} : 1
ight)$, because $rac{\mathbb P(R)}{\mathbb P(B)}$ is how many more times likely R is than B. In this example, $rac{\mathbb P(R)}{\mathbb P(B)} = rac{1}{2},$ because R is half as likely as B. Sometimes, the quantity $rac{\mathbb P(R)}{\mathbb P(B)}$ is called the "odds ratio of R vs B," in which case it is understood that the odds for R vs B are $\left(rac{\mathbb P(R)}{\mathbb P(B)} : 1
ight).$

# Odds to ratios

When there are only two terms $x$ and $y$ in a set of odds, the odds can be written as a ratio $rac{x}{y}.$ The odds *ratio* $rac{x}{y}$ refers to the *odds* $(x : y),$ or, equivalently, $\left(rac{x}{y} : 1
ight).$


Odds: Refresher

Let's say that, in a certain forest, there are 2 sick trees for every 3 healthy trees. We can then say that the odds of a tree being sick (as opposed to healthy) are $(2 : 3).$

Odds express *relative* chances.  Saying "There's 2 sick trees for every 3 healthy trees" is the same as saying "There's 10 sick trees for every 15 healthy trees."  If the original odds are $(x : y)$ we can multiply by a positive number $lpha$ and get a set of equivalent odds $(lpha x : lpha y).$

If there's 2 sick trees for every 3 healthy trees, and every tree is either sick or healthy, then the *probability* of randomly picking a sick tree from among *all* trees is 2/(2+3):

![Odds v probabilities](https://i.imgur.com/GVZnz2c.png?0)

If the set of possibilities $A, B, C$ are [1rd mutually exclusive and exhaustive], then the probabilities $\mathbb P(A) + \mathbb P(B) + \mathbb P(C)$ should sum to $1.$  If there's no further possibilities $d,$ we can convert the relative odds $(a : b : c)$ into the probabilities $(rac{a}{a + b + c} : rac{b}{a + b + c} : rac{c}{a + b + c}).$  The process of dividing each term by the sum of terms, to turn a set of proportional odds into probabilities that sum to 1, is called [1rk normalization].

When there are only two terms $x$ and $y$ in the odds, they can be expressed as a single ratio $rac{x}{y}.$  An odds ratio of $rac{x}{y}$ refers to odds of $(x : y),$ or, equivalently, odds of $\left(rac{x}{y} : 1
ight).$ Odds of $(x : y)$ are sometimes called odds ratios, where it is understood that the actual ratio is $rac{x}{y}.$


Conditional probability: Refresher

[summary: $\mathbb P(	ext{left} \mid 	ext{right})$ is the probability that the thing on the left is true _assuming_ the thing on the right is true, and it's defined as $rac{\mathbb P(	ext{left} \land 	ext{right})}{\mathbb P(	ext{right})}.$

Thus, $\mathbb P(yellow \mid banana)$ is the probability that a banana is yellow ("the probability of yellowness given banana"), while $\mathbb P(banana \mid yellow)$ is the probability that a yellow object is a banana ("the probability of banana, given yellowness").]

$\mathbb P(	ext{left} \mid 	ext{right})$ is the probability that the thing on the left is true _assuming_ the thing on the right is true, and it's defined as $rac{\mathbb P(	ext{left} \land 	ext{right})}{\mathbb P(	ext{right})}.$

Thus, $\mathbb P(yellow \mid banana)$ is the probability that a banana is yellow ("the probability of yellowness given banana"), while $\mathbb P(banana \mid yellow)$ is the probability that a yellow object is a banana ("the probability of banana, given yellowness").%%note: In general, $\mathbb P(v)$ is an abbreviation of $\mathbb P(V = v)$ for some variable $V$, which is assumed to be known from the context. For example, $\mathbb P(yellow)$ might stand for $\mathbb P({ColorOfNextObjectInBag}=yellow)$ where $ColorOfNextObjectInBag$ is a [random_variable variable] in our [-probability_distribution] $\mathbb P,$ and $yellow$ is one of the [random_variable_value values] that that variable can take on.%%

$\mathbb P(x \land y)$ is used to denote the probability of both $x$ and $y$ being true simultaneously (according to some [-probability_distribution] $\mathbb P$). $\mathbb P(x\mid y)$, pronounced "the conditional probability of x, given y", is defined to be the quantity

$$rac{\mathbb P(x \wedge y)}{\mathbb P(y)}.$$

For example, in the [22s Diseasitis] problem, $\mathbb P({sick}\mid {positive})$ is the probability that a patient is sick _given_ a positive test result, and it's calculated by taking the 18% patients who are sick *and* have positive test results, and dividing by all 42% of the patients who got positive test results.  That is, $\mathbb P({sick}\mid {positive})$ $=$ $rac{\mathbb P({sick} \land {positive})}{\mathbb P({positive})}.$

Using a [bayes_frequency_diagrams frequency diagram], we can visualize $\mathbb P(sick \mid positive)$ as the probability of drawing a $sick$ result from a bag of only those people in the population who got a $positive$ result.

![diseasitis frequency](https://i.imgur.com/0XDbrYi.png?0)

![bag of 18 and 24 patients](https://i.imgur.com/W6avfHO.png?0)

The "given" operator in $\mathbb P(x\mid y)$ tells us to assume that $y$ is true, to restrict our attention to only possible cases where $y$ is true, and then ask about the probability of $x$ *within* those cases.

Note that $\mathbb P(positive \mid sick)$ is _not_ the same as $\mathbb P(sick \mid positive).$ To find the probability that a patient has a positive result given that they're sick, we can visualize taking the 20 sick patients and putting them in a group, and then asking the probability that a randomly selected one will have a positive result, which will be $rac{18}{20} = 0.9$ — so $\mathbb P(positive \mid sick) = 90\%,$ while $\mathbb P(sick \mid positive) pprox 43\%.$ Mixing up which one is which is an unfortunate source of of many practical errors when you're trying to do these calculations using only the formal notation, at least until you get used to it. Just remember that $\mathbb P(	ext{left} \mid 	ext{right})$ is the probability of the thing on the left _given_ that the thing on the right is true.


The n-th root of m is either an integer or irrational

There is an intuitive way to see that for any natural numbers $m$ and $n$, $\sqrt[n]m$ will always either be an integer or an irrational number.

Suppose that there was some $\sqrt[n]m$ that was a rational number $rac{a}{b}$ that was not an integer. Suppose further that $rac ab$ is written as a [-reduced] fraction, such that the [-greatest_common_divisor] of $a$ and $b$ is $1$. Then, since $rac{a}{b}$ is not an integer, $b > 1$.

Since $rac ab = \sqrt[n]m$, we have conversely that $(rac ab)^n = m$, which is a natural number by our hypothesis. But let's take a closer look at $(rac ab)^n$. It evaluates to $rac{a^n}{b^n}$, which is still a reduced fraction. [todo: Proof of gcd(a^n, b^n) = 1.]

But since $b > 1$ before, we have that $b^n > 1$ as well, meaning that $(rac ab)^n$ cannot be an integer, contradicting the fact that it equals $m$, a natural number.


Probability distribution: Motivated definition

When discussing probabilities, people will often (informally) say things like "well, the probability $\mathbb P(sick)$ of the patient being sick is about 20%." What does this $\mathbb P(sick)$ notation mean?

Intuitively, $\mathbb P(sick)$ is supposed to denote the probability that a particular person is sick (on a scale from 0 to 1). But how is $\mathbb P(sick)$ defined? Is there an objective probability of sickness? If not, where does the number come from?

At first you might be tempted to say $\mathbb P(sick)$ is defined by the surrounding population: If 1% of people are sick at any given time, then maybe $\mathbb P(sick)$ should be 1%. But what if this person is currently running a high fever and complaining about an upset stomach? Then we should probably assign a probability higher than 1%.

Next you might be tempted to say that the _true_ probability of the person being sick is either 0 or 1 (because they're either sick or they aren't), but this observation doesn't really help us manage our _own_ uncertainty. It's all well and good to say "either they sick or they aren't," but if you're a doctor who has to choose which medication to prescribe (and different ones have different drawbacks), then you need some way of talking about how sick they _seem_ to be (given what you've seen).

This leads us to the notion of [4vr subjective probability]. _Your_ probability that a person is sick is a fact about _you._ They are either sick or healthy, and as you observe more facts about them (such as "they're running a fever"), your _personal_ belief in their health vs sickness changes. This is the idea that used to define notation like $\mathbb P(sick).$

Formally, $\mathbb P(sick)$ is defined to be the probability that $\mathbb P$ assigns to $sick,$ where $\mathbb P$ is a type of object known as a "probability distribution", which is an object designed for keeping track of (and managing) uncertainty. Specifically, probability distributions are objects that distribute a finite amount of "stuff" across a large number of "states," and $\mathbb P(sick)$ measures how much stuff $\mathbb P$ _in particular_ puts on $sick$-type states. For example, the states could be cups with labels on them, and the stuff could be water, in which case $\mathbb P(sick)$ would be the proportion of all water in the $sick$-labeled cups.

The "stuff" and "states" may be arbitrary: you can build a probability distribution out of water in cups, clay in cubbyholes, abstract numbers represented in a computer, or weightings between neurons in your head. The stuff is called "probability mass," the states are called "possibilities."

To be even more concrete, imagine you build $\mathbb P$ out of cups and water, and that you give some of the cups suggestive labels like $sick$ and $healthy$. Then you can talk about the proportion of all probability-water that's in the $sick$ cup vs the $healthy$ cup. This is a probability distribution, but it's not a very useful one. In practice, we want to model more than one thing at a time. Let's say that you're a doctor at an immigration center who needs to assess a person's health, age, and country of origin. Now the set of possibilities that you want to represent aren't just $sick$ and $healthy,$ they're _all combinations_ of health, age, and origin:

$$
egin{align}
sick, 	ext{age }1, 	ext{Afghanistan} \
healthy, 	ext{age }1, 	ext{Afghanistan} \
sick, 	ext{age }2, 	ext{Afghanistan} \
dots \
sick, 	ext{age }29, 	ext{Albania} \
healthy, 	ext{age }29, 	ext{Albania} \
sick, 	ext{age }30, 	ext{Albania} \
dots
\end{align}
$$

and so on. If you build this probability distribution out of cups, you're going to need a lot of cups. If there are 2 possible health states ($sick$ and $healthy$), 150 possible ages, and 196 possible countries, then the total number of cups you need in order to build this probability distribution is $2 \cdot 150 \cdot 196 = 58800,$ which is rather excessive. (There's a reason we do probabilistic reasoning using transistors and/or neurons, as opposed to cups with water in them).

In order to make this proliferation of possibilities manageable, the possibilities are usually arranged into columns, such as the "Health", "Age", and "Country" columns above. This columns are known as "[random_variable variables]" of the distribution. Then, $\mathbb P(sick)$ is an abbreviation for $\mathbb P(	ext{Health}=sick),$ which counts the proportion of all probability mass (water) allocated to possibilities (cups) that have $sick$ in the Health column of their label.

What's the point of doing all this setup? Once we've made a probability distribution, we can hook it up to the outside world such that, when the world interacts with the probability distribution, the probability mass is shifted around inside the cups. For example, if you have a rule which says "whenever a person shows me a passport from country X, I throw out all water except the water in cups with X in the Country column", then, whenever you see a passport, the probability distribution will get more accurate.

The natural question here is, what are the best ways to manipulate the probability mass in $\mathbb P$ (in response to observations), if the goal is to have $\mathbb P$ get more and more accurate over time? That's exactly the sort of question that [-1bv] can be used to answer (and it has implications both for artificial intelligence, and for understanding human intelligence — after all, _we ourselves_ are a physical system that manages uncertainty, and updates beliefs in response to observations).

At this point, there are two big objections to answer. First objection:

> Whoa now, the number of cups in $\mathbb P$ got pretty big pretty quickly, and this was a simple example. In a realistic probability distribution $\mathbb P$ intended to represent the real world (which has way more than 3 variables worth tracking), the number of necessary possibilities would be _ridiculous._ Why do we define probabilities in terms of these huge impractical "probability distributions"?

This is an important question, which is answered by three points:

1. In practice, there are a number of tricks for exploiting regularities in the structure of the world in order to drastically reduce the number of cups you need to track. We won't be covering those tricks in this guide, but you can check out [ Realistic probabilities] and [ Arbital's guide to Bayes nets] if you're interested in the topic.
2. Even so, full-fledged probabilistic reasoning _is_ computationally infeasible on complex problems. In practice, physical reasoning systems (such as brains or artificial intelligence algorithms) use lots of approximations and shortcuts.
3. Nevertheless, reasoning according to a full probability distribution is the _theoretical ideal_ for how to do good reasoning. [ You can't do better than probabilistic reasoning] (unless you're born knowing the right answers to everything), and [ insofar as you don't use probabilistic reasoning, you can be exploited]. Even if complex probability distribution are too big to manage in practice, they tell give lots of hints about how to reason right or wrong that we can follow in our day-to-day lives.

Second objection:

> You basically just said "given a bunch of cups and some water, we define the probability of a person being sick as the amount of water in some suggestively-labeled cups." How does that have anything to do with whether or not the person is actually sick? Just because you put a $sick$ label on there doesn't magically give the water meaning!

This is an important point. For $\mathbb P$ to be useful, we want to design a reasoning procedure such that the more we interact with a person, the more probability mass starts to reflect how healthy the person actually is. That is, we want the water to go into $sick$ cups if they're sick, and $healthy$ cups if they're healthy. If our reasoning procedure has that property, and we have $\mathbb P$ interact with the world for a while, then its probabilities will get pretty accurate — at which point $\mathbb P$ can be used to answer questions and/or make decisions. (This is the principle that makes brains and artificial intelligence algorithms tick.)

How do we design reasoning mechanisms that cause $\mathbb P$ to become more accurate the more it interacts with the world? That's a [how_to_get_accurate_probabilities big question], and the answer has many parts. One of the most important parts of the answer, though, is a law of probability theory which tells us the correct way to move the probability mass around in response to new observations (assuming the goal is to make $\mathbb P$ more accurate). For more on that law, see [1lz Bayes' rule].


Math 3 example statements

If you're at a Math 3 level, you'll probably be familiar with at least some of these sentences and formulas, or you would be able to understand what they meant on a surface level if you were to look them up. Note that you don't necessarily have to understand the *proofs* of these statements (that's what we're here for, to teach you what they mean), but your eyes shouldn't gloss over them either.

> In a [3gd group] $G$, the [-4bj] of an element $g$ is the set of elements that can be written as $hgh^{-1}$ for all $h \in G$.

> The [ rank-nullity theorem] states that for any [-linear_mapping] $f: V 	o W$, the [-dimension] of the [-image] of $f$ plus the dimension of the [-kernel] of $f$ is equal to the dimension of $V$.

> A [ Baire space] is a space that satisfies [ Baire's Theorem] on [complete_metric_space complete metric spaces]: For a [-topological_space] $X$, if ${F_1, F_2, F_3, \ldots}$ is a [countable_set countable] collection of open sets that are [dense_set dense] in $X$, then $igcap_{n=1}^\infty F_n$ is also dense in $X$.

> The [riemann_hypothesis] asserts that every non-trivial zero of the [riemann_zeta_function] $\zeta(s) = \sum_{n=1}^\infty rac{1}{s^n}$ when $s$ is a complex number has a real part equal to $rac12$.

> $
ewcommand{\pd}[2]{rac{\partial #1}{\partial #2}}$ The [jacobian_matrix] of a [vector_valued_function vector-valued function] $f: \mathbb{R}^m 	o \mathbb{R}^n$ is the matrix of [-partial_derivatives] $\left[ egin{matrix} \pd{y_1}{x_1} & \pd{y_1}{x_2} & \cdots & \pd{y_1}{x_m} \ \pd{y_2}{x_1} & \pd{y_2}{x_2} & \cdots & \pd{y_2}{x_m} \ dots & dots & \ddots & dots \ \pd{y_n}{x_1} & \pd{y_n}{x_2} & \cdots & \pd{y_n}{x_m} \end{matrix}
ight]$ between each component of the argument vector $x = (x_1, x_2, \ldots, x_m)$ and each component of the result vector $y = f(x) = (y_1, y_2, \ldots, y_n)$. It is notated as $\displaystyle rac{d\mathbf{y}}{d\mathbf{x}}$ or $\displaystyle rac{d(y_1, y_2, \ldots, y_n)}{d(x_1, x_2, \ldots, x_m)}$.


Proof of Bayes' rule: Probability form

Let $\mathbf H$ be a [random_variable variable] in $\mathbb P$ for the true hypothesis, and let $H_k$ be the possible values of $\mathbf H,$ such that $H_k$ is [-1rd]. Then, Bayes' theorem states:

$$\mathbb P(H_i\mid e) = \dfrac{\mathbb P(e\mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)},$$

with a proof that runs as follows. By the definition of [-1rj],

$$\mathbb P(H_i\mid e) = \dfrac{\mathbb P(e \wedge H_i)}{\mathbb P(e)} = \dfrac{\mathbb P(e \mid  H_i) \cdot \mathbb P(H_i)}{\mathbb P(e)}$$

By the law of [law_of_marginal_probability marginal probability]:

$$\mathbb P(e) = \sum_{k} \mathbb P(e \wedge H_k)$$

By the definition of conditional probability again:

$$\mathbb P(e \wedge H_k) = \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)$$

Done.

Note that this proof of Bayes' rule is less general than the [1xr proof] of the [1x5 odds form of Bayes' rule].

## Example

Using the [22s Diseasitis] example problem, this proof runs as follows:

$$egin{array}{c}
\mathbb P({sick}\mid {positive}) = \dfrac{\mathbb P({positive} \wedge {sick})}{\mathbb P({positive})} \[0.3em]
= \dfrac{\mathbb P({positive} \wedge {sick})}{\mathbb P({positive} \wedge {sick}) + \mathbb P({positive} \wedge
eg {sick})} \[0.3em]
= \dfrac{\mathbb P({positive}\mid {sick}) \cdot \mathbb P({sick})}{(\mathbb P({positive}\mid {sick}) \cdot \mathbb P({sick})) + (\mathbb P({positive}\mid
eg {sick}) \cdot \mathbb P(
eg {sick}))}
\end{array}
$$

Numerically:

$$3/7 = \dfrac{0.18}{0.42} = \dfrac{0.18}{0.18 + 0.24} = \dfrac{90\% * 20\%}{(90\% * 20\%) + (30\% * 80\%)}$$

Using red for sick, blue for healthy, and + signs for positive test results, the proof above can be visually depicted as follows:

![bayes theorem probability](https://i.imgur.com/H9im04o.png?0)

%todo: if we replace the other Venn diagram for the proof of Bayes' rule, we should probably update this one too.%


Math 2 example statements

If you're at a Math 2 level, you'll probably be familiar with most or all of these sentences and formulas, or you would be able to understand what they meant on a surface level if you were to look them up.

> The [-quadratic_formula] states that the roots of every quadratic expression $ax^2 + bx + c$ are equal to $\displaystyle rac{-b \pm \sqrt{b^2 - 4ac}}{2a}$. The expression under the [-square_root], $b^2 - 4ac$, is called the [-discriminant], and determines how many roots there are in the equation.

> The imaginary number $i$ is defined as the primary root of the quadratic equation $x^2 + 1 = 0$.

> To solve the system of linear equations $egin{matrix}ax + by = c \ dx + ey = f\end{matrix}$ for $x$ and $y$, the value of $x$ can be computed as $\displaystyle rac{bf - ce}{bd - ae}$.

> The [-power_rule] in calculus states that $rac{d}{dx} x^n = nx^{n-1}$.

> All the solutions to the equation $m^n = n^m$ where $m < n$ are of the form $m = (1 + rac 1x)^x$ and $n = (1 + rac 1x)^{x+1}$, where $x$ is any positive real number.


Binary notation

[summary: Binary notation is a way to write [54y numbers] using [math_power multiples] of two instead of the usual multiples of ten. It's especially useful for working with computers, which use binary notation exclusively.]

When you were first taught how to write [54y numbers] and [addition add] them together, you were probably told something about the "ones place", the "tens place", the "hundreds place", and so on.  Each [-digit] further to the left represents a larger [math_power multiple] of ten, and the multiples are added together to get the number you want--so $8207$ is counted as $(7 	imes 10^0) + (0 	imes 10^1) + (2 	imes 10^2) + (8 	imes 10^3)$, or "seven ones, zero tens, two hundreds, and eight thousands".  But why use powers of ten?  What's so special about that number?

Well, nothing, actually. We probably only use powers of ten because we happen to have evolved with ten fingers--any number bigger than one will work just as well, although it may look strange at first.  Since two is the smallest [-48l] larger than one, a *binary* number notation--a notation that uses powers of two--is one of the simplest possible. %note:The word "binary" has the same root as the word "bicycle"--*bi* meaning "two".%

Binary notation uses only two digits, $0$ and $1$, and each "place" to the left goes up by a power of two instead of ten (in other words, it doubles).  For example, in binary notation, the number $11010$ is counted as $(0 	imes 2^0) + (1 	imes 2^1) + (0 	imes 2^2) + (1 	imes 2^3) + (1 	imes 2^4)$, or "zero ones, one two, zero fours, one eight, and one sixteen".  Translating back to the familiar [number_bases base] ten, we would write it as $26$.

You may notice that binary notation tends to be a bit longer than [4sl decimal] (for example, "11010" takes more characters to write than "26").  It's also more difficult to read, unless you have a lot of practice with it.  So why would anyone use it?  Well, for one thing, binary notation is often very convenient for talking about powers of two, for instance when working with base-two [3nd logarithms] or log [-1rb], or when working with some quantity measured in [3y2 bits].  It's also essential for working with computers, as all modern computers store and manipulate data, on the lowest level, exclusively using binary notation.


Likelihood function

Let's say you have a piece of evidence $e$ and a set of hypotheses $\mathcal H.$ Each $H_i \in \mathcal H$ assigns some [56v likelihood] to $e.$ The function $\mathcal L_{e}(H_i)$ that reports this likelihood for each $H_i \in \mathcal H$ is known as a "likelihood function."

For example, let's say that the evidence is $e_c$ = "Mr. Boddy was killed with a candlestick," and the hypotheses are $H_S$ = "Miss Scarlett did it," $H_M$ = "Colonel Mustard did it," and $H_P$ = "Mrs. Peacock did it." Furthermore, if Miss Scarlett was the murderer, she's 20% likely to have used a candlestick. By contrast, if Colonel Mustard did it, he's 10% likely to have used a candlestick, and if Mrs. Peacock did it, she's only 1% likely to have used a candlestick. In this case, the likelihood function is

$$\mathcal L_{e_c}(h) =
egin{cases}
0.2 & 	ext{if $h = H_S$} \
0.1 & 	ext{if $h = H_M$} \
0.01 & 	ext{if $h = H_P$} \
\end{cases}
$$

For an example using a continuous function, consider a possibly-biased coin whose bias $b$ to come up heads on any particular coinflip might be anywhere between $0$ and $1$.  Suppose we observe the coin to come up heads, tails, and tails. We will denote this evidence $e_{HTT}.$ The likelihood function over each hypothesis $H_b$ =  "the coin is biased to come up heads $b$ portion of the time" for $b \in [0, 1]$ is:

$$\mathcal L_{e_{HTT}}(H_b) = b\cdot (1-b)\cdot (1-b).$$

There's no reason to [1rk normalize] likelihood functions so that they sum to 1 — they aren't probability distributions, they're functions expressing each hypothesis' propensity to yield the observed evidence. For example, if the evidence was really obvious ($e_s$ = "the sun rose this morning,") it might be the case that almost all hypotheses have a very high likelihood, in which case the sum of the likelihood function will be much more than 1.

Likelihood functions carry _absolute_ likelihood information, and therefore, they contain information that [1rq relative likelihoods] do not. Namely, absolute likelihoods can be used to check a hypothesis for [227 strict confusion].


Likelihood ratio

[summary: Given a piece of evidence $e$ and two hypothsese $H_i$ and $H_j,$ the likelihood ratio between them is the ratio of the [-56v] each hypothesis assigns to $e$ For example, let $e$ = "Mr. Boddy was knifed", and say that Professor Plum is 25% likely to use a knife while Mrs. White is only 5% likely to use a knife. Then the likelihood ratio of $e$ between the hypotheses "Plum did it" and "Mrs. White did it" is 25/5 = 5/1. See also [1rq].]

Given a piece of evidence $e_0$ and two hypothsese $H_i$ and $H_j,$ the likelihood ratio between them is the ratio of the [-56v] each hypothesis assigns to $e_0.$

For example, imagine the evidence is $e$ = "Mr. Boddy was knifed", and the hypotheses are $H_P$ = "Professor Plum did it" and $H_W$ = "Mrs. White did it." Let's say that, if Professor Plum were the killer, we're 25% sure he would have used a knife. Let's also say that, if Mrs. White were the killer, there's only a 5% chance she would have used a knife. Then the likelihood ratio of $e_0$ between $H_P$ and $H_W$ is 25/5 = 5, which says that $H_P$ assigns five times as much likelihood to $e$ as does $H_W,$ which means that the evidence supports the "Plum did it" hypothesis five times as much as it supports the "Mrs. White did it" hypothesis.

A likelihood ratio of 5 denotes [1rq relative likelihoods] of $(5 : 1).$ Relative likelihoods can be multiplied by [1rb odds] in order to update those odds, as per [1lz].


Likelihood

[summary:  "Likelihood", when speaking of Bayesian reasoning, denotes *the probability of an observation, supposing some hypothesis to be correct.*

Suppose our piece of evidence $e$ is that "Mr. Boddy was shot."  One of our suspects is Miss Scarlett, and we denote by $H_S$ the hypothesis that Miss Scarlett shot Mr. Boddy.  Suppose that if Miss Scarlett *were* the killer, we'd have predicted in advance a 20% probability she would use a gun, and an 80% chance she'd use some other weapon.

Then the *likelihood* from the evidence, to Miss Scarlett being the killer, is 0.20.  Using [1rj conditional probability notation], $\mathbb P(e \mid H_S) = 0.20.$

This doesn't mean Miss Scarlett has a 20% chance of being the killer; it means that if she is the killer, our observation had a probability of 20%.

Relative likelihoods are a key ingredient for [1ly Bayesian reasoning] and one of the quantities plugged into [1lz Bayes's Rule].]

Consider a piece of evidence $e,$ such as "Mr. Boddy was shot." We might have a number of different hypotheses that explain this evidence, including $H_S$ = "Miss Scarlett killed him", $H_M$ = "Colonel Mustard killed him", and so on.

Each of those hypotheses assigns a different probability to the evidence. For example, imagine that _if_ Miss Scarlett _were_ the killer, there's a 20% chance she would use a gun, and an 80% chance she'd use some other weapon. In this case, the "Miss Scarlett" hypothesis assigns a *likelihood* of 20% to $e.$

When reasoning about different hypotheses using a [-probability_distribution probability distribution] $\mathbb P$, the likelihood of evidence $e$ given hypothesis $H_i$ is often written using the [1rj conditional probability] $\mathbb P(e \mid H_i).$ When reporting likelihoods of many different hypotheses at once, it is common to use a [-likelihood_function,] sometimes written [51n $\mathcal L_e(H_i)$].

[1rq Relative likelihoods] measure the degree of support that a piece of evidence $e$ provides for different hypotheses. For example, let's say that if Colonel Mustard were the killer, there's a 40% chance he would use a gun. Then the absolute likelihoods of $H_S$ and $H_M$ are 20% and 40%, for _relative_ likelihoods of (1 : 2). This says that the evidence $e$ supports $H_M$ twice as much as it supports $H_S,$ and that the amount of support would have been the same if the absolute likelihoods were 2% and 4% instead.

According to [1lz Bayes' rule], relative likelihoods are the appropriate tool for measuring the [22x strength of a given piece evidence]. Relative likelihoods are one of two key constituents of belief in [bayesian_reasoning Bayesian reasoning], the other being [1rm prior probabilities].

While absolute likelihoods aren't necessary when updating beliefs by Bayes' rule, they are useful when checking for [227 confusion]. For example, say you have a coin and only two hypotheses about how it works: $H_{0.3}$ = "the coin is random and comes up heads 30% of the time", and $H_{0.9}$ = "the coin is random and comes up heads 90% of the time." Now let's say you toss the coin 100 times, and observe the data HTHTHTHTHTHTHTHT... (alternating heads and tails). The _relative_ likelihoods strongly favor $H_{0.3},$ because it was less wrong. However, the _absolute_ likelihood of $H_{0.3}$ will be much lower than expected, and this deficit is a hint that $H_{0.3}$ isn't right. (For more on this idea, see [227].)


Subtraction of rational numbers (Math 0)

[summary: Subtraction is "adding anti-apples", or equivalently removing some apples you already have.]

So far, we have met [4zx the idea of a rational number], treating them as chunks of apples, and [55m how to add them together].
Now we will discover how the idea of the anti-apple (by analogy with the [53r integers' anti-cow]) must work.

# The anti-apple

Just as we had an anti-cow, so we can have an anti-apple.
If we combine an apple with an anti-apple, they both annihilate, leaving nothing behind.
We write this as $1 + (-1) = 0$.

A very useful thing for you to ponder for thirty seconds (though I will give you the answer soon): given that $rac{a}{n}$ means "divide an apple into $n$ equal pieces, then take $a$ copies of the resulting little-piece", what would $rac{-1}{n}$ mean? And what would $-rac{1}{n}$ mean?

%%hidden(Show possible solution):
$rac{-1}n$ would mean "divide an apple into $n$ equal pieces, then take $-1$ copies of the resulting little-piece".
That is, turn it into an anti-little-piece.
This anti-little-piece will annihilate one little-piece of the same size: $rac{-1}{n} + rac{1}{n} = 0$.

$-rac{1}{n}$, on the other hand, would mean "divide an anti-apple into $n$ equal pieces, then take $1$ copy of the resulting little-anti-piece".
But this is the same as $rac{-1}{n}$: it doesn't matter whether we do "convert to anti, then divide up the apple" or "divide up the apple, then convert to anti".
That is, "little-anti-piece" is the same as "anti-little-piece", which is very convenient.
%%

What about chunks of apple?
If we combine half an apple with half an anti-apple, they should also annihilate, leaving nothing behind.
We write this as $rac{1}{2} + \left(-rac{1}{2}
ight) = 0$.

How about a bit more abstract?
If we combine an apple with half an anti-apple, what should happen?
Well, the apple can be made out of two half-chunks (that is, $1 = rac{1}{2} + rac{1}{2}$); and we've just seen that half an apple will annihilate half an anti-apple; so we'll be left with just one of the two halves of the apple.
More formally, $1 + \left(-rac{1}{2}
ight) = rac{1}{2}$; or, writing out the calculation in full, $$1 + \left(-rac{1}{2}
ight) = rac{1}{2} + rac{1}{2} + \left(-rac{1}{2}
ight) = rac{1}{2}$$

Let's go the other way round: if we combine an anti-apple with half an apple, what happens?
It's pretty much the same as the opposite case except flipped around: the anti-apple is made of two anti-half-chunks, and the half apple will annihilate one of those chunks, leaving us with half an anti-apple: that is, $(-1) + rac{1}{2} = -rac{1}{2}$.

We call all of these things **subtraction**: "subtracting" a quantity is defined to be the same as the addition of an anti-quantity.

# General procedure

Since we already know how to add, we might hope that subtraction will be easier (since subtraction is just a slightly different kind of adding).

In general, we write $-rac{1}{n}$ for "the $rac{1}{n}$-sized building block, but made by dividing an *anti*-apple (instead of an apple) into $n$ equal pieces".
Remember from the pondering above that this is actually the same as $rac{-1}{n}$, where we have divided an apple into $n$ equal pieces but then taken $-1$ of the pieces.

Then in general, we can just use the instant addition rule that we've already seen. %%note:Recall: this was $rac{a}{m} + rac{b}{n} = rac{a	imes n + b 	imes m}{m 	imes n}$. Remember that the order of operations in the integers is such that in the numerator, we calculate both the products first; then we add them together.%%
In fact, $$rac{a}{m} - rac{b}{n} = rac{a}{m} + \left(rac{-b}{n}
ight) = rac{a 	imes n + (-b) 	imes m}{m 	imes n} = rac{a 	imes n - b 	imes m}{m 	imes n}$$

Why are we justified in just plugging these numbers into the formula, without justification?
You're quite right if you are dubious: you should not be content merely to *learn* this formula%%note:This goes for all of maths! It's not simply a collection of arbitrary rules, but a proper process that we use to model our thoughts. Behind every pithy, unmemorable formula is a great edifice of motivation and reason, if you can only find it.%%.
In the rest of this page, we'll go through why it works, and how you might construct it yourself if you forgot it.
I took the choice here to present the formula first, because it's a good advertisement for why we use the $rac{a}{n}$ notation rather than talking about "$rac{1}{n}$-chunks" explicitly: it's a very compact and neat way of expressing all this talk of anti-apples, in the light of what we've already seen about addition.

Very well: what should $rac{a}{m} - rac{b}{n}$ be?
We should first find a smaller chunk out of which we can build both the $rac{1}{m}$ and $rac{1}{n}$ chunks.
We've seen already that $rac{1}{m 	imes n}$ will work as a smaller chunk-size.

Now, what is $rac{a}{m}$ expressed in $rac{1}{m 	imes n}$-chunks?
Each $rac{1}{m}$-chunk is $n$ of the $rac{1}{m 	imes n}$-chunks, so $a$ of the $rac{1}{m}$-chunks is $a$ lots of "$n$ lots of $rac{1}{m 	imes n}$-chunks": that is, $a 	imes n$ of them.

Similarly, $rac{b}{n}$ is just $b 	imes m$ lots of $rac{1}{m 	imes n}$-chunks.

So, expressed in $rac{1}{m 	imes n}$-chunks, we have $a 	imes n$ lots of positive chunks, and $b 	imes m$ lots of anti-chunks.
Therefore, when we put them together, we'll get $a 	imes n - b 	imes m$ chunks (which might be negative or positive or even zero - after all the annihilation has taken place we might end up with either normal or anti-chunks or maybe no chunks at all - but it's still an integer, being a number of chunks).

So the total amount of apple we have is $rac{a 	imes n - b 	imes m}{m 	imes n}$, just like we got out of the instant formula.

[todo: the rest of the page, including examples; make an exercises page, including subtracting negatives and so on]


Transitive relation

A binary [-3nt] $R$ is **transitive** if whenever $aRb$ and $bRc$, $aRc$.

The most common examples or transitive relations are [3rb partial orders] (if $a \leq b$ and $b \leq c$, then $a \leq c$) and [53y equivalence relations] (if $a \sim b$ and $b \sim c$, then $a \sim c$).

A transitive relation that is also [5dy reflexive] is called a [-preorder].

A [-transitive_set] $S$ is a set on which the element-of relation $\in$ is transitive; whenever $a \in x$ and $x \in S$, $a \in S$.


Lattice: Examples

Here are some additional examples of lattices. $
ewcommand{
subg}{\mathcal N \mbox{-} Sub~G}$

A familiar example
---------------------------------

Consider the following lattice.

![Suspicious Lattice Hasse Diagram](http://i.imgur.com/7HefDKm.png)

Does this lattice look at all familiar to you? From some other area of mathematics, perhaps?

%%hidden(Reveal the truth):

In fact, this lattice corresponds to boolean logic, as can be seen when we replace b with true and a with false in the following "truth table".

![lattice truth table](http://i.imgur.com/hpThsTk.png)

%%comment:

Latex source:

egin{tabular} {| c | c | c | c |}
  \hline
  $x$ & $y$ & $x ee y$ & $x \wedge y$ \ \hline
  $a$ & $a$ & $a$ & $a$ \ \hline
  $a$ & $b$ & $b$ & $a$ \ \hline
  $b$ & $a$ & $b$ & $a$ \ \hline
  $b$ & $b$ & $b$ & $b$ \ \hline
\end{tabular}

%%

%%

Normal subgroups
---------------------------------

Let $G$ be a group, and let $
subg$ be the set of all [4h6 normal subgroups] of $G$. Then $\langle
subg, \subseteq
angle$ is a lattice where for $H, K \in
subg$, $H \wedge K = H \cap K$, and $H ee K = HK = \{ hk \mid h \in H, k \in K \}$.

%%hidden(Proof):

Let $H,K \in
subg$. Then $H \wedge K = H \cap K$. We first note that $H \cap K$ is a [576 subgroup] of $G$. For let $a,b \in H \cap K$. Since $H$ is a group, $a \in H$, and $b \in H$, we have $ab \in H$. Likewise, $ab \in K$. Combining these, we have $ab \in H \cap K$, and so $H \cap K$ is satisfies the closure requirement for subgroups. Since $H$ and $K$ are groups, $a \in H$, and $a \in K$, we have $a^{-1} \in H$ and $a^{-1} \in K$. Hence, $a^{-1} \in H \cap K$, and so $H \cap K$ satisfies the inverses requirement for subgroups. Since $H$ and $K$ are subgroups of $G$, we have $e \in H$ and $e \in K$. Hence, we have $e \in H \cap K$, and so $H \cap K$ satisfies the identity requirement for subgroups.

Furthermore, $H \cap K$ is a normal subgroup, because for all $a \in G$, $a^{-1}(H \cap K)a = a^{-1}Ha \cap a^{-1}Ka = H \cap K$. It's clear from the definition of intersection that $H$ and $K$ do not share a common subset larger than $H \cap K$.

For $H, K \in
subg$, we have $H ee K = HK = \{ hk \mid h \in H, k \in K \}$.

First we will show that $HK$ is a group. For $hk, h'k' \in HK$, since $kH = Hk$, there is some $h'' \in H$ such that $kh' = h''k$. Hence, $hkh'k' = hh''kk' \in HK$, and so $HK$ is closed under $G$'s group action. For $hk \in HK$, we have $(hk)^{-1} = k^{-1}h^{-1} \in k^{-1}H = Hk^{-1} \subseteq HK$, and so $HK$ is closed under inversion. Since $e \in H$ and $e \in K$, we have $e = ee \in HK$. Finally, $HK$ inherits its associativity from $G$.

To see that $HK$ is a normal subgroup of $G$, let $a \in G$. Then $a^{-1}HKa = Ha^{-1}Ka = HKa^{-1}a = HK$.

There is no subgroup $F$ of $G$ smaller than $HK$ which contains both $H$ and $K$. If there were such a subgroup, there would exist some $h \in H$ and some $k \in K$ such that $hk
ot\in F$. But $h \in F$ and $k \in F$, and so from $F$'s group closure we conclude $hk \in F$, a contradiction.

%%


Subgroup

A **subgroup** of a [-3gd] $(G,*)$ is a group of the form $(H,*)$, where $H \subset G$. We usually say simply that $H$ is a subgroup of $G$.

For a subset of a group $G$ to be a subgroup, it needs to satisfy all of the group axioms itself: [3gy closure], [3h4 associativity], [54p identity], and [inverse_element inverse]. We get associativity for free because $G$ is a group. So the requirements of a subgroup $H$ are:

1. **[3gy Closure]:** For any $x, y$ in $H$, $x*y$ is in $H$.
2. **[54p Identity]:** The identity $e$ of $G$ is in $H$.
3.  **[inverse_element Inverses]:** For any $x$ in $H$, $x^{-1}$ is also in $H$.

A subgroup is called [4h6 normal] if it is closed under [4gk conjugation].

The subgroup [-3nt] is [573 transitive]: if $H$ is a subgroup of $G$, and $I$ is a subgroup of $H$, then $I$ is a subgroup of $G$.

#Examples

Any group is a subgroup of itself. The [-trivial_group] is a subgroup of every group.

For any [-48l] $n$, the set of multiples of $n$ is a subgroup of the integers (under [-addition]).


Boolean

[summary: A boolean is a mathematical quantity that can only take on one of two values: "[-true]" or "[-false]". Booleans are to [-258] what [48l integers] are to [-arithmetic].]

In formal [-258], a boolean variable is a [-variable] that can take on one of only two possible values: "[-true]" or "[-false]".  [2v8] can then be said to [-evaluation evaluate] to one of these two values, in the same way that ordinary [algebra algebraic] expressions evaluate to a [-54y].

Also as in algebraic expressions, boolean values can be manipulated using certain operators such as [and_operator $\land$ (and)], [or_operator $\lor$ (or)], [negation_operator $
eg$ ([negation)], and [implication_operator $
ightarrow$ (implication)].  This field is called, surprisingly, [-boolean_algebra].

Because booleans can only express [4mq absolute truth or falsity], when working with measures of uncertainty you must use other representations, such as [-1rf].


Disambiguation

Several distinct concepts with comparable importance use this page's name, this page helps readers find what they're looking for.

[todo: Add guidelines for use.]
[todo: Add best practices for good disambig pages.]


Proof of Löb's theorem

[summary:
You can prove [55w] by assuming $\square A	o A$ and constructing a sentence $S$ such that $PAdash S\leftrightarrow (\square S 	o A)$
]

The intuition behind the proof of Löb's theorem invokes the formalization of the following argument:

Let $S$ stand for the sentence "If $S$ is provable, then I am Santa Claus".

As it is standard, when trying to prove a If...then theorem we can start by supposing the antecedent and checking that the consequent follows. But if $S$ is provable, then we can chain its proof to an application of modus ponens of $S$ to itself in order to get a proof of the consequent "I am Santa Claus". If we suppose that showing the existence of a proof of a sentence implies that the sentence is true, then "I am Santa Claus" is true.

Thus we have supposed that $S$ is provably true then it follows that "I am Santa Claus", which is a proof of what $S$ states! Therefore, $S$ is provably true, and we can apply the same reasoning again (this time without supposing a counterfactual) to get that it is true that "I am Santa Claus".

Holy Gödel! Have we broken logic? If this argument worked, then we could prove any nonsensical thing we wanted. So where does it fail?

One suspicious point is our implicit assumption that $S$ can be constructed in the first place as a first order sentence. But we know that there exists a [5j7 provability predicate] encoding the meaning we need, and then we can apply the [59c diagonal lemma] to the formula $Prv(x)\implies$ "I am Santa Claus" to get our desired $S$.

(We will ignore the fact that expressing "I am Santa Claus" in the language of first order logic is rather difficult, since we could easily replace it with something simple like $0=1$.)

It turns out that the culprit of this paradox is the apparently innocent supposition that when "I am Santa Claus" is provable, then it is true.

How can this be false?, you may ask. Well, the standard provability predicate $Prv$ is something of the form $\exists y Proof(x,y)$, where $Proof(x,y)$ is true iff $y$ encodes a valid proof of $x$. But $y$ might be a [ nonstandard number] encoding a [ nonstandard proof] of infinitely many steps. If the only proofs of $x$ are nonstandard then certainly you will never be able to prove $x$ from the axioms of the system using standard methods.

We can restate this result as Löb's theorem: If $Prv(A)\implies A$ is provable, then $A$ is provable.

-----
Now we go for the formal proof.

Let $T$ be an [ axiomatizable] extension of [ minimal arithmetic], and $P$ a one-place predicate satisfying the [5j7 Bernais-Hilbert derivability conditions], which are:

1. If $Tdash A$, then $Tdash P(A)$.
2. $Tdash P(A\implies B) \implies (P(A)\implies P(B))$
3. $Tdash P(A)\implies P(P(A))$.

For example, $PA$ and the standard provability predicate [ satisfy those conditions].

Let us suppose that $A$ is such that $Tdash P(A)\implies A$.

As $T$ extends [ minimal arithmetic], the [-59c] is applicable, and thus there exists $S$ such that $Tdash S \iff (P(S)\implies A)$.

By condition 1, $Tdash P(S \implies (P(S)\implies A))$.

By condition 2, $Tdash P(S) \implies P(P(S) \implies A)$.

Also by condition 2, $Tdash P(P(S) \implies A) \implies (P(P(S)) \implies P(A))$.

Combining these, $Tdash P(S) \implies (P(P(S)) \implies P(A))$.

By condition 3, $Tdash P(S)\implies P(P(S))$ and thus $Tdash P(S)\implies P(A)$.

By our initial assumption, this means that $Tdash P(S)\implies A$.

But by the construction of $S$, then $Tdash S$!

By condition 1 again, then $Tdash P(S)$, and since we already had shown that $Tdash P(S)\implies A$, we have that $Tdash A$, which completes the proof.

We remark that $P$ can be any predicate satisfying the conditions, such as the verum predicate $x=x$.


Diagonal lemma

[summary: If F(x) is a one place formula of First Order Logic and $T$ is a theory extending [ minimal arithmetic] then there exists a sentence $S$ such that $T\vdash S\iff F(\ulcorner S \urcorner)$.]

The **diagonal lemma** shows how to construct self-referential sentences in the language of arithmetic from formulas $\phi(x)$.

In its standard form it says that is $T$ is a theory extending [ minimal arithmetic] (so that it can talk about [ recursive] expressions with full generality) and $\phi(x)$ is a formula with one free variable $x$ then there exists a sentence $S$ such that $T\vdash S\leftrightarrow \phi(\ulcorner S\urcorner)$.

This can be further [ generalized] for cases with multiples formulas and variables.

The diagonal lemma is important because it allows the formalization of all kind of self-referential and apparently paradoxical sentences.

Take for example the liar's sentence affirming that "This sentence is false". Since [ there is no truth predicate], we will have to adapt it to our language to say "This sentence is not provable". Since there is a [5gt predicate of arithmetic expressing provability] we can construct a formula $\neg \square_{PA} (x)$, which is true iff there is no proof in [3ft $PA$] of the sentence [31z encoded] by $x$.

By invoking the diagonal's lemma, we can see that there exists a sentence $G$ such that $PA\vdash G\leftrightarrow \neg \square_{PA} (\ulcorner G\urcorner)$, which reflects the spirit of our informal sentence. The [ weak form of Gödel's first incompleteness theorem] follows swiftly from there.

The equivalent in computation of the diagonal lemma is called [322 quining], and refers to computer programs which produce their own source code as part of their computation.

Indeed, the key insight of the diagonal lemma and quining is that you can have subexpressions in your program that when "expanded" are identical to the original expression that contains them.


Gödel's first incompleteness theorem

[summary: Every [ consistent] and [ axiomatizable] extension of [ minimal arithmetic] is incomplete]

The **first incompleteness theorem** is a result about the existence of sentences of arithmetic that cannot be proved or disproved, no matter what axioms we take as true.

What Gödel originally proved was that every [ $\omega$-consistent] and [ axiomatizable] extension of Peano Arithmetic is incomplete, but the result was later refined to weaken the requirement of $\omega$-consistency to simple [-5km] and the set of theorems that the extension had to prove to that of [ minimal arithmetic].

The heart of both proofs is the [-59c], which allows us express self-referential sentences in the language of arithmetic.

This put an end to the dream of building a [-complete] [-5hh] that axiomatized all of mathematics, since as soon as one was expressive enough to talk about arithmetic, incompleteness would kick in.

##Interpretation from model theory
The first incompleteness theorem highlights the impossibility of defining the [45h natural numbers] with the usual operations of [ addition] and [ multiplication] in [ first order logic].

We already knew from Lowenhëim-Skolem theorem that there would be models of $PA$ which are not isomorphic to the usual arithmetic, but the first incompleteness theorem implies that some of those models disagree in the truth value of some theorems of this language (those are the undecidable sentences).


Proof of Gödel's first incompleteness theorem

##Weak form
The weak Gödel's first incompleteness theorem states that every [ $\omega$-consistent] [ axiomatizable] extension of minimal arithmetic is incomplete.

Let $T$ extend [-minimal_arithmetic], and let $Prv_{T}$ be the [5gt standard provability predicate] of $T$.

Then we apply the [59c diagonal lemma] to get $G$ such that $T\vdash G\iff \neg Prv_{T}(G)$.

We assert that the sentence $G$ is undecidable in $T$. We prove it by contradiction:

Suppose that $T\vdash G$. Then $Prv_ {T}(G)$ is correct, and as it is a $\exists$-rudimentary sentence then it is [every_true_e_rudimentary_sentence_is_provable_in_minimal_arithmetic provable in minimal arithmetic], and thus in $T$. So we have that $T\vdash Prv_ {T}(G)$ and also by the construction of $G$ that $T\vdash \neg Prv_{T}(G)$, contradicting that $T$ is consistent.

Now, suppose that $T\vdash \neg G$. Then $T\vdash  Prv_{T}(G)$. But then as $T$ is consistent there cannot be a standard proof of $G$, so if $Prv_{T}(x)$ is of the form $\exists y Proof_{T}(x,y)$ then for no natural number $n$ it can be that $T\vdash Proof_ {T}(\ulcorner G\urcorner,n)$, so $T$ is $\omega$-inconsistent, in contradiction with the hypothesis.

##Strong form

> Every [5km consistent] and [-axiomatizable] extension of [-minimal_arithmetic] is [complete incomplete].

This theorem follows as a consequence of the [ undecidability of arithmetic] combined with the lemma stating that [ any complete axiomatizable theory is undecidable]


Multiplication of rational numbers (Math 0)

[summary: Multiplication is making a number by "doing to some quantity what would usually be done to the quantity $1$".]

We've seen how to [55m add] and [56x subtract] pairs of [4zq rational numbers].
But the [45h natural numbers] have another operation on them: multiplication.

Remember, a given rational number represents what we get when we cut an apple into pieces all of the same size, then take some number %%note:Possibly *more* than we actually made, and possibly negative!%% of the little pieces.
The **product** of $rac{a}{m}$ and $rac{b}{n}$ %%note: Recall that $rac{a}{m}$ is "$a$ copies of the little-piece we get when we cut an apple into $m$ equal pieces.%% is what we call "$rac{a}{m}$ **multiplied by** $rac{b}{n}$", and it answers the question "What happens if we do the procedure that would make $rac{b}{n}$, but instead of starting by cutting one apple into $n$ pieces, we started by cutting $rac{a}{m}$ apples into $n$ pieces?".

We write the product of $rac{a}{m}$ and $rac{b}{n}$ as $rac{a}{m} 	imes rac{b}{n}$.

# Example

It's hopefully easy to see that $1 	imes rac{b}{n} = rac{b}{n}$.
Indeed, the definition is "what do we get if we would make $rac{b}{n}$, but instead of starting by cutting one apple, we started by cutting $1$ apple?"; but that's just the same!
It's like saying "What if, instead of putting bread around my sandwich filling, I tried putting bread?" - I haven't actually changed anything, and I'll still get the same old sandwich %%note:or $rac{b}{n}$%% out at the end.

How about $2 	imes rac{3}{5}$? (Strictly speaking, I should probably be writing $rac{2}{1}$ instead of $2$, but this way saves a bit of writing. $rac{2}{1}$ means "two copies of the thing I get when I cut an apple into one piece"; but an apple cut into one piece is just that apple, so $rac{2}{1}$ just means two apples.)
Well, that says "instead of cutting one apple, we cut two apples" into $rac{3}{5}$-sized pieces.

From now on, my pictures of apples will get even worse: rather than being circles, they'll now be squares.
It just makes the diagrams easier to understand.

![Two times three-fifths](http://i.imgur.com/tW78Nys.png)

In the picture, we have two apples (squares) which I've drawn next to each other, separated by a dashed line.
Then I've taken $rac{3}{5}$ of the whole shape (shaded in red): that is, to the group of two apples I have done the procedure that would create $rac{3}{5}$ if it were done to one apple alone.

Notice, though, that this divides neatly into $rac{3}{5}$ of the left-hand apple, and $rac{3}{5}$ of the right-hand apple.
So the red-shaded area comes to $rac{3}{5} + rac{3}{5}$, which you already know how to calculate: it is $rac{6}{5}$.

# General integer times fraction

Can you work out, from the case of $2 	imes rac{3}{5}$ above, what $m 	imes rac{a}{n}$ is, where $m$ is an integer?

%hidden(Show solution):
It is $rac{a 	imes m}{n}$.

Indeed, the procedure to get $rac{a}{n}$ is: we split $1$ into $n$ equal pieces, and then take $a$ of them.
So the procedure to get $m 	imes rac{a}{n}$ is: we split $m$ into $n$ equal pieces, and then take $a$ of them.

But each of the pieces we've just made by splitting $m$—that is, those demarcated by the longer solid lines in the $2 	imes rac{3}{5}$ diagram above—can be viewed as being $m$ copies of what we get by splitting $1$.
(In the diagram above, we have $2$ copies of that which we get by splitting $1$: namely the two copies indicated by the dashed line.)
So we can view the second procedure as: we split $1$ into $n$ equal pieces %%note:In the diagram above, there are $5$ such equal pieces, and right now we're looking only at one square, not at both squares joined together.%%, and then take $a$ of them %%note:In the diagram above, $a$ is $3$: this has given us the red shaded bit of one of the squares.%%, and then do this $m$ times. %%note: In the diagram above, $m$ is $2$: we're finally looking at the two squares joined together into a rectangle.%%

This produces $a 	imes m$ pieces, each of size $rac{1}{n}$, and hence the rational number $rac{a 	imes m}{n}$.
%

You should check that you get the right answer for a different example: $-5 	imes rac{2}{3}$.
%%hidden(Show solution):
This is "do the procedure that makes $rac{2}{3}$, but instead of starting with $1$, start with $-5$".

So we take five anti-apples, and divide them into thirds (obtaining $15$ anti-chunks of size $rac{1}{3}$ each, grouped as five groups of three); and then we take two chunks out of each group of three, obtaining $10$ anti-chunks of apple in total.

So $-5 	imes rac{2}{3} = rac{-10}{3}$, in accordance with the rule of $n 	imes rac{a}{n} = rac{a 	imes m}{n}$.
%%

# General fraction times fraction

[todo: examples and pictures]
[todo: instant rule]

# Order doesn't matter

Notice that while it was fairly obvious that order doesn't matter during addition (that is, $rac{a}{m} + rac{b}{n} = rac{b}{n} + rac{a}{m}$), because it's simply "putting two things next to each other and counting up what you've got", it's not all that obvious that the product of two fractions should be independent of the order we multiplied in.
However, you should check, from the general expression above, that it actually *is* independent of the order.

Why is this?
Why should it be that "do the procedure that made $rac{b}{n}$, but starting from $rac{a}{m}$ instead of $1$" and "do the procedure that made $rac{a}{m}$, but starting from $rac{b}{n}$ instead of $1$" give the same answer?

Well, remember the diagram we had for $2 	imes rac{3}{5}$ (remembering that that is "do the procedure that would make $rac{3}{5}$, but instead of doing it to $1$, we do it to $2$):

![Two times three-fifths](http://i.imgur.com/tW78Nys.png)

What would we get if we rotated this diagram by a quarter-turn?

![Two times three-fifths, rotated](https://imgur.com/C61M0f2.png)

But wait! The shaded bit is just what we get when we do the procedure that makes $2$ (namely "put two copies of the shape next to each other"), but instead of doing it on the single (upper-most) square, we do it to the version of the number $rac{3}{5}$ that is represented by the shaded bit of the upper-most square! And that is exactly what we would do to get $rac{3}{5} 	imes 2$.

In general, $rac{a}{m} 	imes rac{b}{n}$ is the same as $rac{b}{n} 	imes rac{a}{m}$, because the two just "come from the same diagram, rotated by a quarter-turn".
They are measuring the same amount of stuff, because the amount of stuff in a diagram doesn't change simply because we rotated it.

## Another example

We'll do $rac{-5}{7} 	imes rac{2}{3}$.

[todo: this example]

# Meditation: why the notation makes sense

At this point, a digression is in order.
We have already seen the notation $rac{a}{n}$ for "take an apple; divide it into $n$ pieces, each $rac{1}{n}$-sized; and then take $a$ of the chunks".
In the language of multiplication that we've now seen, that is "do what we would do to make $a$, but do it starting from a $rac{1}{n}$-chunk instead of $1$".
That is, $rac{a}{n}$ is just $rac{1}{n} 	imes a$.

And we can do that in a different way: we can take $a$ apples, divide each into $n$ chunks, and then just draw one of the chunks from each apple.
In the language of multiplication, that is just "do what we would do to make a $rac{1}{n}$-chunk, but do it to $a$ instead of $1$".
That is, $rac{a}{n} = a 	imes rac{1}{n}$.

Recalling that $a$ is just $rac{a}{1}$, our notation $rac{a}{n}$ is simply the same as $rac{a}{1} 	imes rac{1}{n}$, as an instance of the "instant rule" $rac{a}{1} 	imes rac{1}{n} = rac{a 	imes 1}{1 	imes n} = rac{a}{n}$.

# Inverses: putting things in reverse

Remember that we had "anti-apples" as a way of making nothing ($0$) by adding to some quantity of apples.
In a similar vein, we can "invert" multiplication.

Whenever $a$ is not $0$, we can find a rational number $rac{c}{d}$ such that $rac{a}{b} 	imes rac{c}{d} = 1$.
(Notice that we've got $1$ as our "base point" now, rather than the $0$ that addition had.)

Indeed, using the instant rule, we see that $rac{a}{b} 	imes rac{c}{d} = rac{a 	imes c}{b 	imes d}$, so to make $1$ we want $a 	imes c$ to be the same as $b 	imes d$.

But we can do that: if we let $c = b$ and $d = a$, we get the right thing, namely $rac{a 	imes b}{b 	imes a} = rac{a 	imes b}{a 	imes b} = rac{1}{1} = 1$.

So $rac{b}{a}$ works as an inverse to $rac{a}{b}$.
And this is why we needed $a$ not to be $0$: because $rac{b}{a}$ isn't actually a rational number unless $a$ is nonzero.

## Intuition

We've seen how this definition follows from the instant rule.
Where does it *actually* come from, though?

[todo: intuition]


Bijective Function: Intro (Math 0)

##Comparing Amounts##
Consider the Count von Count. He cares only about counting things. He doesn't care what they are, just how many there are. He decides that he wants to collect items into plastic crates, and he considers two crates equal if both contain the same number of items.

![Equivalent Crates](http://i.imgur.com/sG7Qfyv.jpg)

Now Elmo comes to visit, and he wants to impress the Count, but Elmo is not great at counting. Without counting them explicitly, how can Elmo tell if two crates contain the same number of items?

Well, he can take one item out of each crate and put the pair to one side.

![Pairing one pair of items](https://i.imgur.com/myTqIrRl.jpg)

 He continues pairing items up in this way and when one crate runs out he checks if there are any left over in the other crate. If there aren't any left over, then he knows there were the same number of items in both crates.

![Pairing all items in one crate with items in the other](https://i.imgur.com/S87JO2nl.jpg)

Since the Count von Count only cares about counting things, the two crates are basically equivalent, and might as well be the same crate to him. Whenever two objects are the same from a certain perspective, we say that they are **isomorphic**.

In this example, the way in which the crates were the same is that each item in one crate could be paired with an item in the other.

![Pairing all items in one box with items in the other](https://i.imgur.com/S87JO2nl.jpg)

![Bijection between crates](https://i.imgur.com/53YbraFl.jpg)

This wouldn't have been possible if the crates had different numbers of items in them.

![Different numbers of items](https://i.imgur.com/F5rLwAsl.jpg)

![No way to pair items](https://i.imgur.com/KCI9UOvl.jpg)

Whenever you can match each item in one collection with exactly one item in another collection, we say that the collections are **[-499 bijective]** and the way you paired them is a **[-499 bijection]**. A bijection is a specific kind of isomorphism.

![Bijection between crates](https://i.imgur.com/53YbraFl.jpg)

Note that there might be many different bijections between two bijective things.

![Another bijection between crates](http://i.imgur.com/Q6Si1ZX.png)

In fact, all that counting involves is pairing up the things you want to count, either with your fingers, or with the concepts of 'numbers' in your head. If there are as many objects in one crate as there are numbers from one to seven, and there are as many objects in another crate as numbers from one to seven, then both crates contain the same number of objects.


Convex set

A convex set is a [-3jz] of [3w0 vectors] that contains all line segments between vectors in the set.  Consider the following shape:

![a convex set](https://upload.wikimedia.org/wikipedia/commons/thumb/6/6b/Convex_polygon_illustration1.svg/634px-Convex_polygon_illustration1.svg.png)

As shown, a line segment between two points $x$ and $y$ in this shape lies entirely within the shape.  In fact, this is true for any pair of points in the shape.  Therefore, this shape is convex.  For comparison, the following shape is not convex:

![a non-convex set](https://upload.wikimedia.org/wikipedia/commons/thumb/6/6c/Convex_polygon_illustration2.svg/634px-Convex_polygon_illustration2.svg.png)

The fact that part of the line segment between $x$ and $y$ (both inside the shape) lies outside the shape proves that this shape is not convex.

Formally, a set $S$ is convex if

$$orall x, y \in S, 	heta \in [0, 1]: 	heta x + (1 - 	heta) y \in S$$

(images are from Wikipedia: [here](https://en.wikipedia.org/wiki/File:Convex_polygon_illustration1.svg) and [here](https://en.wikipedia.org/wiki/File:Convex_polygon_illustration2.svg))

[todo: use non-Wikipedia images and remove the attribution]


Factorial

Factorial is most simply defined as a [-3jy] on positive [48l integers]. 5 factorial (written as $5!$) means $1*2*3*4*5$. In general then, for a positive integer $n$, $n!=\prod_{i=1}^{n}i$. For applications to [ combinatorics], it will also be useful to define $0! = 1$.

## Applications to Combinatorics ##

$n!$ is the number of possible orders for a set of $n$ objects. For example, if we arrange the letters $A$, $B$, and $C$, here are all the options:
$$ABC$$
$$ACB$$
$$BAC$$
$$BCA$$
$$CAB$$
$$CBA$$
You can see that there are $6$ possible orders for $3$ objects, and $6 = 3*2*1 = 3!$. Why does this work? We can [5fz prove this by induction]. First, we'll see pretty easily that it works for $1$ object, and then we can show that if it works for $n$ objects, it will work for $n+1$. Here's the case for $1$ object.
$$A$$
$$1 = \prod_{i=1}^{1}i = 1!$$
Now we have the objects $\{A_{1},A_{2},...,A_{n},A_{n+1}\}$, and $n+1$ slots to put them in. If $A_{n+1}$ is in the first slot, now we're ordering $n$ remaining objects in $n$ remaining slots, and by our [5fz induction hypothesis], there are $n!$ ways to do this. Now let's suppose $A_{n+1}$ is in the second slot. Any orderings that result from this will be completely unique from the orderings where $A_{n+1}$ was in the first slot. Again, there are $n$ remaining slots, and $n$ remaining objects to put in them, in an arbitrary order. There are another $n!$ possible orderings. We can put $A_{n+1}$ in each slot, one by one, and generate another $n!$ orderings, all of which are unique, and by the end, we will have every possible ordering. We know we haven't missed any because $A_{n+1}$ has to be somewhere. The total number of orderings we get is $n!*(n+1)$, which equals $(n+1)!$.

## Extrapolating to [50d Real Numbers] ##

The factorial function can be defined in a different way so that it is defined for all real numbers (and in fact for complex numbers too).

%%hidden(Definition):
We define $x!$ as follows:
$$x! = \Gamma (x+1),$$
where $\Gamma $ is the [ gamma function]:
$$\Gamma(x)=\int_{0}^{\infty}t^{x-1}e^{-t}\mathrm{d} t$$
Why does this correspond to the factorial function as defined previously? We can prove by induction that for all positive integers $x$:
$$\prod_{i=1}^{x}i = \int_{0}^{\infty}t^{x}e^{-t}\mathrm{d} t$$
First, we verify for the case where $x=1$. Indeed:
$$\prod_{i=1}^{1}i = \int_{0}^{\infty}t^{1}e^{-t}\mathrm{d} t$$
$$1=1$$
Now we suppose that the equality holds for a given $x$:
$$\prod_{i=1}^{x}i = \int_{0}^{\infty}t^{x}e^{-t}\mathrm{d} t$$
and try to prove that it holds for $x + 1$:
$$\prod_{i=1}^{x+1}i = \int_{0}^{\infty}t^{x+1}e^{-t}\mathrm{d} t$$
We'll start with the induction hypothesis, and manipulate until we get the equality for $x+1$.
$$\prod_{i=1}^{x}i = \int_{0}^{\infty}t^{x}e^{-t}\mathrm{d} t$$
$$(x+1)\prod_{i=1}^{x}i = (x+1)\int_{0}^{\infty}t^{x}e^{-t}\mathrm{d} t$$
$$\prod_{i=1}^{x+1}i = (x+1)\int_{0}^{\infty}t^{x}e^{-t}\mathrm{d} t$$
$$= 0+\int_{0}^{\infty}(x+1)t^{x}e^{-t}\mathrm{d} t$$
$$= \left (-t^{x+1}e^{-t})
ight]_{0}^{\infty}+\int_{0}^{\infty}(x+1)t^{x}e^{-t}\mathrm{d} t$$
$$= \left (-t^{x+1}e^{-t})
ight]_{0}^{\infty}-\int_{0}^{\infty}(x+1)t^{x}(-e^{-t})\mathrm{d} t$$
By the product rule of integration:
$$=\int_{0}^{\infty}t^{x+1}e^{-t}\mathrm{d} t$$
This completes the proof by induction, and that's why we can define factorials in terms of the gamma function.
%%


Convex function

A convex [-3jy] "only curves upwards."  To check whether a function is convex, use the following procedure:

1. Draw the graph of the function.

![graph of a function](http://i.imgur.com/bJAKquW.png)

2. Select any two points in the graph.

![graph with two points](http://i.imgur.com/j3MsPWR.png)

3. Draw a line segment connecting the two points.

![graph with two points connected](http://i.imgur.com/MoZTIK1.png)

4. See whether this line segment is ever lower than the graph.

![graph with part of line segment highlighted](http://i.imgur.com/4piUSZJ.png)

If the line segment is ever lower than the graph, the function is not convex.  The function graphed above is not convex, as can be seen by looking at the red part of the line segment.  On the other hand, if the line segment never goes below the graph (no matter which two initial points you selected), then the function is convex.

Equivalently, a function is convex if its [-epigraph] is a [-5bs].


Needs accessible summary

This tag is for pages which have a summary of some kind but don't have a summary which is accessible to a relevant audience.

Some pages may require a [4pt technical] or [4q2 brief] summary.


Reflexive relation

A binary [3nt relation] over some set is **reflexive** when every element of that set is related to itself. (In symbols, a relation $R$ over a set $X$ is reflexive if $orall a \in X, aRa$.) For example, the relation $\leq$ defined over the real numbers is reflexive, because every number is less than or equal to itself.

A relation is **anti-reflexive** when *no* element of the set over which it is defined is related to itself. $<$ is an anti-reflexive relation over the real numbers. Note that a relation doesn't have to be either reflexive or anti-reflexive; if Alice likes herself but Bob doesn't like himself, then the relation "_ likes _" over the set $\{Alice, Bob\}$ is neither reflexive nor anti-reflexive.

The **reflexive closure** of a relation $R$ is the union of $R$ with the [Identity_relation identity relation]; it is the smallest relation that is reflexive and that contains $R$ as a subset. For example, $\leq$ is the reflexive closure of $<$.

Some other simple properties that can be possessed by binary relations are [Symmetric_relation symmetry] and [573 transitivity].

A reflexive relation that is also transitive is called a [Preorder preorder].


Algebraic structure tree

Some classes of [-3gx] are given special names based on the properties of their sets and operations. These terms grew organically over the history of modern mathematics, so the overall list of names is a bit arbitrary (and in a few cases, some authors will use slightly different assumptions about certain terms, such as whether a semiring needs to have identity elements). This list is intended to clarify the situation to someone who has some familiarity with what an algebraic structure is, but not a lot of experience with using these specific terms.

%%comment:Tree is the wrong word; this should be more of an algebraic structure collection of disjoint directed acyclic graphs? But this is what other pages seem to have chosen to link to, so here we are!%%

# One set, one binary operation

* [Groupoid], sometimes known as a magma. This is the freebie. Have a set and a binary operation? That's a groupoid.
    * A [-semigroup] is a groupoid where *the operation is [3h4 associative]*.
        * A [-3h3] is a semigroup where *the operation has an [-54p]*.
            * A [-3gd] is a monoid where *every element has an [-inverse_element] under the binary operation*.
                * An [3h2] is a group where *the binary operation is [3jb commutative]*.
        * A [-semilattice] is a semigroup where *the operation is [Idempotent_operation idempotent] and commutative*.
    * A [-quasigroup] is a groupoid where *every element has a [quotient_algebra left and right quotient] under the binary operation* (sometimes called the [Latin_square_property]).
        * A [Algebraic_loop loop] is a quasigroup *with identity*.
        * A **group**, as defined above, can also be defined as a (non-empty) quasigroup where *the operation is associative* ([ quotients and associativity give a two-sided identity and two-sided inverses], provided there's at least one element to be that identity).

# One set, two binary operations

For the below, we'll use $*$ and $\circ$ to denote the two binary operations in question. It might help to think of $*$ as "like addition" and $\circ$ as "like multiplication", but be careful—in most of these structures, properties of addition and multiplication like commutativity won't be assumed!

* [Ringoid] assumes only that $\circ$ distributes over $*$—in other words, $a \circ (b * c) = (a \circ b) * (a \circ c)$ and $(a * b) \circ c = (a \circ c) * (b \circ c)$.
    * A [-semiring] is a ringoid where *both $*$ and $\circ$ define semigroups*.
        * An [-additive_semiring] is a semiring where $*$ *is commutative*.
            * A [-rig] is an additive semiring where $*$ *has an identity element*. (It's almost a ring! It's just missing <b>n</b>egatives.)
                * A [-3gq] is a rig where *every element has an inverse element under* $*$. (Some authors also require $\circ$ to have an identity to call the structure a ring.)
                    * A **ring with unity** is a ring where $\circ$ *has an identity*. (Some authors just use the word "ring" for this; others use variations like "unit ring".)
                        * A [-division_ring] is a ring with unity where *every element (except for the identity of $*$) has an inverse element under* $\circ$.
                            * A [481 field] is a division ring where $\circ$ *is commutative*.
    * A [46c lattice] is a ringoid where *both $*$ and $\circ$ define semilattices, and satisfy the absorption laws ($a \circ (a * b) = a * (a \circ b) = a$)*. (While we'll continue to use $*$ and $\circ$ here, the two operations of a lattice are almost always denoted with [3rc $\wedge$ and $ee$].)
        * A [-bounded_lattice] is a lattice where *both operations have identities*.


Bayes' rule: Beginner's guide

This arc teaches basic understanding of [1lz] via [1x5]. This arc should work for [1r5]-type readers.


Lattice: Exercises

Try these exercises to test your knowledge of lattices.

## Distributivity

Does the lattice meet operator distribute over joins? In other words, for all lattices $L$ and all $p, q, r \in L$, is it necessarily true that $p \wedge (q ee r) = (p \wedge q) ee (p \wedge r)$? Prove your answer.

%%hidden(Solution):
The following counterexample shows that lattice meets do not necessarily distribute over joins.

![A non-distributive lattice called as M3](http://i.imgur.com/SKJirZx.png)

%%%comment:
dot source:

digraph G {
  node [width = 0.1, height = 0.1]
  edge [arrowhead = "none"]

  rankdir = BT;
  t -> p
  t -> q
  t -> r
  p -> s
  q -> s
  r -> s
}
%%%

In the above lattice, $p \wedge (q ee r) = p
eq t = (p \wedge q) ee (p \wedge r)$.
%%

## Common elements

Let $L$ be a lattice, and let $J$ and $K$ be two finite subsets of $L$ with a non-empty intersection. Prove that $igwedge J \leq igvee K$.

%%hidden(Solution):
If $J$ and $K$ have a non-empty intersection, then there exists some lattice element $p$ such that $p \in J$ and $p \in K$. Since $igwedge J$ is a lower bound of $J$, we have $igwedge J \leq p$. Since $igvee K$ is an upper bound of $K$, we have $p \leq igvee K$. By transitivity, we have $igwedge J \leq p \leq igvee K$.
%%

## Another inequality

Let $L$ be a lattice, and let $J$ and $K$ be two finite subsets of $L$ such that for all $j \in J$ and $k \in K$, $j \leq k$. Prove that $igvee J \leq igwedge K$.

%%hidden(Solution):
Rephrasing the problem statement, we have that every element of $J$ is a lower bound of $K$ and that every element of $K$ is an upper bound of $J$. It the follows that for $j \in J$, $j \leq igwedge K$. Hence, $igwedge K$ is an upper bound of $J$, and therefore it is greater than or equal to the *least* upper bound of $J$: $igvee J \leq igwedge K$.
%%

## The minimax theorem

Let $L$ be a lattice and $A$ an $m 	imes n$ matrix of elements of $L$. Prove the following inequality: $$igvee_{i=1}^m igwedge_{j=1}^n A_{ij} \leq igwedge_{j=1}^n igvee_{i=1}^m A_{ij}$$.

%%hidden(Solution):
To get an intuitive feel for this theorem, it helps to first consider a small concrete instantiation. Consider the $3 	imes 3$ matrix depicted below, with elements $a,b,c,d,e,f,g,h$, and $i$. The inequality instantiates to $(a \wedge b \wedge c) ee (d \wedge e \wedge f) ee (g \wedge h \wedge i) \leq (a ee d ee g) \wedge (b ee e ee h) \wedge (c ee f ee i)$. Why would this inequality hold?

$$\left[ egin{array}{ccc}
a & b & c \
d & e & f \
g & h & i \
\end{array}
ight]$$

Notice that each parenthesized expression on the left hand side of the inequality shares an element with each parenthesized expression on the right hand side of the inequality.This is true because the parenthesized expressions on the left hand side correspond to rows and the parenthesized expressions on the right hand side correspond to columns; each row of a matrix shares an element with each of its columns. The theorem proven in the *Common elements* exercise above then tells us that each parenthesized expression on the left hand side is less than or equal to each parenthesized expression on the right hand side.

Let $J = \{ a \wedge b \wedge c, d \wedge e \wedge f, g \wedge h \wedge i \}$ and $K = \{ a ee d ee g, b ee e ee h, c ee f ee i \}$. Then the hypothesis for the theorem proven in the *Another inequality* exercise holds, giving us $igvee J \leq igwedge K$, which is exactly what we wanted to prove. Extending this approach to the general case is straightforward.
%%


Mathematical induction

The **principle of mathematical induction** is a proof technique in which a statement, $P(n)$, is proven about a set of [45h natural numbers] $n$. It may be best understood as treating the statements like dominoes: a statement $P(n)$ is true if the $n$-th domino is knocked down. We must knock down a first domino, or prove that a **base case** $P(m)$ is true. Next we must make sure the dominoes are close enough together to fall, or that the **inductive step** holds; in other words, we prove that if $k \geq m$ and $P(k)$ is true, $P(k+1)$ is true. Then since $P(m)$ is true, $P(m+1)$ is true; and since $P(m+1)$ is true, $P(m+2)$ is true, and so on.

An example
=======

We'll do an example to build our intuition before giving the proper definition of the principle. We'll provide yet another proof that
$$ 1 + 2 + \cdots + n = rac{n(n+1)}{2}$$
for all integers $n \ge 1$. First, let's check the base case, where $n=1$:
$$ 1 = rac{1(1+1)}{2} = rac{2}{2} = 1.$$
This is (fairly obviously) true, so we can move forward with the inductive step. The inductive step includes an assumption, namely that the statement is true for some integer $k$ that is larger than the base integer. For our example, if $k\ge1$, we assume that
$$1 + 2 + \cdots + k = rac{k(k+1)}{2}$$
and try to prove that
$$ 1 + 2 + \cdots + k + (k+1) = rac{(k+1)([k+1]+1)}{2}.$$
Take our assumption and add $k+1$ to both sides:
$$1+2+\cdots + k + (k+1) = rac{k(k+1)}{2} + k + 1.$$
Now the left-hand sides of what we know and what we want are the same. Hopefully the right-hand side will shake out to be the same. Get common denominators so that the right-hand side of the above equation is
$$rac{k(k+1)}{2} + rac{2(k+1)}{2} = rac{(k+2)(k+1)}{2} = rac{(k+1)([k+1]+1)}{2}.$$
Therefore,
$$ 1 + 2 + \cdots + k + (k+1) = rac{(k+1)([k+1]+1)}{2}$$
as desired.

Let $P(n)$ be the statement for $n \ge 1$ that the sum of all integers between 1 and $n$ is $n(n+1)/2$. At the beginning we showed that the base case, $P(1)$, is true. Next we showed the inductive step, that if $k \ge 1$ and $P(k)$ is true, then $P(k+1)$ is true. This means that since $P(1)$ is true, $P(2)$ is true; and since $P(2)$ is true, $P(3)$ is true; etc., so that $P(n)$ is true for all integers $n \ge 1$.

Definition for the natural numbers
======

We are ready to properly define mathematical induction.

Weak mathematical induction
-------

Let $P(n)$ be a statement about the natural numbers. Then $P$ is true for all $n \ge m$ if

 1. $P(m)$ is true, and
 2. For all $k \ge m$, $P(k+1)$ is true if $P(k)$ is.

Strong mathematical induction
-----

Let $P(n)$ be a statement about the natural numbers. Then $P$ is true for all $n \ge m$ if

 1. $P(m)$ is true, and
 2. For all $k \ge m$, $P(k)$ is true so long as $P(\ell)$ is true for all $m \le \ell < k$.

A note: **strong mathematical induction** is a variant on mathematical induction by requiring a stronger inductive step, namely that the statement is true for *all* smaller indices, not just the immediate predecessor.

Induction on a well-ordered set
=====

Well done if your immediate response to the above material was, "Well, am I only restricted to this technique on the natural numbers?" No. As long as your index set is [55r well-ordered], then strong mathematical induction will work.

However, if your ordered set is not well-ordered, you can prove properties 1 and 2 above, and still not have it hold for all elements beyond the base case. For instance, consider the set of nonnegative real numbers, and let $P(x)$ be the claim $x\leq 1$. Then $P(0)$ is true, and for all real numbers $x\ge 0$, $P(x)$ is true whenever $P(y)$ is true for all $0 \le y < x$. But of course $P(2)$ is false.


Proposed A-Class

Post pages which you think should be A-Class in a comment here. Subscribe to this page to get updates about new proposals.

Once the number of approvals from [5ft reviewers] reaches $3-rejections*2$ the page can be given an [4yf A-Class] tag.


Proposed B-Class

Post pages which you think should be B-Class in a comment here. Subscribe to this page to get updates about new proposals.

Once the number of approvals from [5ft reviewers] reaches $1-rejections*2$ the page can be given an [4yd B-Class] tag.


Standard provability predicate

[summary: In theories extending [-minimal_arithmetic] there exists a $\exists_1$-formula $\square_T$(x) which defines the existence of a proof in an [ axiomatizable theory] $T$.

The formula satisfies [5j7 some nice propereties], but fails to capture some intuitions about provability due to non-standard weirdness.]

We know that every theory as [expressiveness_mathematics expressive] as [-minimal_arithmetic] (i.e., [Peano_Arithmetic $PA$]) is capable of talking about [statement_mathematics statements] about itself via [encoding encodings] of [sentence_mathematics sentences] of the language of [-arithmetic] into the [45h natural numbers].

Of particular interest is figuring out whether it is possible to write a formula $Prv(x)$ which is true [-46m] there exists a proof of $x$ from the axioms and rules of inference of our theory.

If we reflect on what a proof is, we will come to the conclusion that a proof is a sequence of symbols which follows certain rules. Concretely, it is a sequence of strings such that either:

1. The string is an [-axiom] of the system or...
2. The string is a theorem that can be deduced from previous strings of the system by applying a [ rule of inference].

And the last string in the sequence corresponds to the theorem we want to prove.

If the axioms are a [-computable_set], and the rules of inference are also effectively computable, then checking whether a certain string is a proof of a given theorem is also computable.

Since every computable set can be defined by a [ $\Delta_1$-formula] in arithmetic %%note:[ Proof]%% then we can define the $\Delta_1$-formula $Prv(x,y)$ such that $PA\vdash Prv(n,m)$ iff $m$ encodes a proof of the sentence of arithmetic encoded by $n$.

Then a sentence $S$ is a theorem of $PA$ if $PA\vdash \exists y Prv(\ulcorner S \urcorner, y)$.

This formula is the standard provability predicate, which we will abbreviate as $\square_{PA}(x)$, and has some nice properties which correspond to what one would expect of a [-5j7].

However, due to [non_standard_model non-standard models], there are some intuitions about provability that the standard provability predicate fails to capture. For example, it turns out that $\square_{PA}A\rightarrow A$ [55w is not always a theorem of $PA$].

There are non-standard models of $PA$ which contain numbers other than $0,1,2,..$ (called [non_standard_models_of_arithmetic non-standard models of arithmetic]). In those models, the standard predicate $\square_{PA}x$ can be true even if for no natural number $n$ it is the case that $Prv(x,n)$. %%note:This condition is called [ $\omega$-inconsistency]%% This means that there is a non-standard number which satisfies the formula, but nonstandard numbers do not encode standard proofs!


Löb's theorem and computer programs

The close relationship between [ logic and computability] allows us to frame Löb's theorem in terms of a computer program which is systematically looking for proofs of mathematical statements, `ProofSeeker(X)`.

ProofSeeker can be something like this:

    ProofSeeker(X):
        n=1
        while(True):
            if Prv(X,n): return True
            else n = n+1

Where `Prv(X,n)` is true if $n$ [ encodes] a proof of $X$%%note:See [-5gt] for more info on how to talk about provablity%%.

Now we form a special sentence called a *reflection principle*, of the form $L(X)$= "*If `ProofSeeker(X)` halts, then X is true*". (This requires a [322 quine] to construct.)

Reflection principles are intuitively true, since ProofSeeker clearly halts iff it finds a proof of $X$, and if there is a proof of $X$, then $X$ must be true if we have chosen an appropriate [-5hh] to search for proofs. For example, let's say that `ProofSeeker` is looking for proofs within [3ft].

The question now becomes, what happens when we call `ProofSeeker` on $L(X)$? Is $PA$ capable of proving that the reflection principle for any given $X$ is true, and therefore `ProofSeeker` will eventually halt? Or will it run forever?

Several possibilities appear:

1. If $PAdash X$, then certainly $PAdash L(X)$, since if the consequent of $L(X)$ is provable, then the whole sentence is provable.
2. If $PAdash
eg X$, then we cannot assert that $PAdash L(X)$, for that would imply asserting that $PAdash$"There is no proof of X". This is tantamount to $PA$ asserting the [-5km] of $PA$, which is forbidden by [ Gödel's second incompleteness theorem].
3. If $X$ is undecidable in $PA$, then if it were the case that $PAdash L(X)$ it would be inconsistent that $PAdash
eg X$ for the same reason as when $X$ is disprovable, and thus $PAdash X$, contradicting that it was undecidable.

**Löb's theorem** is the assertion that $PA$ proves the reflection principle for $X$ only if $PA$ proves $X$.

Or conversely, Löb's theorem states that if $PA
otdash X$ then $PA
otdash \square_{PA} X
ightarrow X$.

It can be [ proved] in $PA$ and stronger systems. It has a very strong link with Gödel's second incompleteness theorem, and in fact [ they both are equivalent].


Gödel II and Löb's theorem

[summary: [ Gödel's second incompleteness theorem] and [ Löb's theorem] are equivalent to each other. ]

The abstract form of [ Gödel's second incompleteness theorem] states that if $P$ is a provability predicate in a [5km consistent], [-axiomatizable] theory $T$ then $T\not\vdash \neg P(\ulcorner S\urcorner)$ for a disprovable $S$.

On the other hand, [55w Löb's theorem] says that in the same conditions and for every sentence $X$, if $T\vdash P(\ulcorner X\urcorner)\rightarrow X$, then $T\vdash X$.

It is easy to see how GII follows from Löb's. Just take $X$ to be $\bot$, and since $T\vdash \neg \bot$ (by definition of $\bot$), Löb's theorem tells that if $T\vdash \neg P(\ulcorner \bot\urcorner)$ then $T\vdash \bot$. Since we assumed $T$ to be consistent, then the consequent is false, so we conclude that $T\neg\vdash \neg P(\ulcorner \bot\urcorner)$.

The rest of this article exposes how to deduce Löb's theorem from GII.

Suppose that $T\vdash P(\ulcorner X\urcorner)\rightarrow X$.

Then $T\vdash \neg X \rightarrow \neg P(\ulcorner X\urcorner)$.

Which means that $T + \neg X\vdash \neg P(\ulcorner X\urcorner)$.

From Gödel's second incompleteness theorem, that means that $T+\neg X$ is inconsistent, since it proves $\neg P(\ulcorner X\urcorner)$ for a disprovable $X$.

Since $T$ was consistent before we introduced $\neg X$ as an axiom, then that means that $X$ is actually a consequence of $T$. By completeness, that means that we should be able to prove $X$ from $T$'s axioms, so $T\vdash X$ and the proof is done.


Provability predicate

[summary:
A provability predicate of a theory $T$ is a formula $P(x)$ with one free variable $x$ such that:

1. If $T\vdash S$, then $T\vdash P(\ulcorner S \urcorner)$
2. $T\vdash P(\ulcorner A\rightarrow B \urcorner)\rightarrow (P(\ulcorner A \urcorner)\rightarrow P(\ulcorner B \urcorner))$
3. $T\vdash P(\ulcorner S \urcorner)\rightarrow P(\ulcorner P(\ulcorner S \urcorner) \urcorner)$
]

A provability predicate is a formula $P$ of a theory satisfying the Hilbert-Bernais derivability conditions. If the diagonal theorem is applicable in the theory as well, then [55w] and [ Gödel's second incompleteness theorem] are provable for $P$.

The Hilbert-Bernais derivability conditions are as follows:

1. (**Necessitation**) If $T\vdash S$, then $T\vdash P(\ulcorner S \urcorner)$
2. (**Provability of modus ponens / distributive axioms**) $T\vdash P(\ulcorner A\rightarrow B \urcorner)\rightarrow (P(\ulcorner A \urcorner)\rightarrow P(\ulcorner B \urcorner))$
3. (**Provability of renecessitation**) $T\vdash P(\ulcorner S \urcorner)\rightarrow P(\ulcorner P(\ulcorner S \urcorner) \urcorner)$

The derivability conditions are tighlty related to the axioms and rules of inference of [-534]. In fact, the normal systems of provability are defined as those that have necessitation as a rule and the distributive axioms. %%note:They also have to be closed under substitution%% On the other hand, D3 is the axiom that defines the system [ K4], and it is also satisfied by [GL].

##Examples

The **verum** formula $x=x$ trivially satisfies the derivability conditions.

The [-5gt] of arithmetic is a provability predicate.


Normal system of provability logic

Between the modal systems of provability, the normal systems distinguish themselves by exhibiting nice properties that make them useful to reason.

A normal system of provability is defined as satisfying the following conditions:

1. Has **necessitation** as a rule of inference. That is, if $Ldash A$ then $Ldash \square A$.
2. Has **modus ponens** as a rule of inference: if $Ldash A
ightarrow B$ and $Ldash A$ then $Ldash B$.
3. Proves all **tautologies** of propositional logic.
4. Proves all the **distributive axioms** of the form $\square(A
ightarrow B)
ightarrow (\square A
ightarrow \square B)$.
5. It is **closed under substitution**. That is, if $Ldash F(p)$ then $Ldash F(H)$ for every modal sentence $H$.

The simplest normal system, which only has as axioms the tautologies of propositional logic and the distributive axioms, it is known as the [ K system].

##Normality
The good properties of normal systems are collectively called **normality**.

Some theorems of normality are:

* $Ldash \square(A_1\wedge ... \wedge A_n)\leftrightarrow (\square A_1 \wedge ... \wedge \square A_n)$
* Suppose $Ldash A
ightarrow B$. Then $Ldash \square A
ightarrow \square B$ and $Ldash \diamond A
ightarrow \diamond B$.
* $Ldash \diamond A \wedge \square B
ightarrow \diamond (A\wedge B)$

##First substitution theorem
Normal systems also satisfy the first substitution theorem.
>(**First substitution theorem**) Suppose $Ldash A\leftrightarrow B$, and $F(p)$ is a formula in which the sentence letter $p$ appears. Then $Ldash F(A)\leftrightarrow F(B)$.

##The hierarchy of normal systems
The most studied normal systems can be ordered by extensionality:

![Hierarchy of normal systems](http://i.imgur.com/1yrL9FU.png)

Those systems are:

* The system K
* The system K4
* The system [5l3 GL]
* The system T
* The system S4
* The system B
* The system S5