Logarithm

[summary: The logarithm base $b$ of a number $n,$ written $\log_b(n),$ is the answer to the question "how many times do you have to multiply 1 by $b$ to get $n$?" For example, $\log_{10}(1000)=3,$ because $10 \cdot 10 \cdot 10 = 1000,$ and $\log_2(16)=4$ because $2 \cdot 2 \cdot 2 \cdot 2 = 16.$]

[summary(Technical): $\log_b(n)$ is defined to be the number $x$ such that $b^x = n.$ Thus, logarithm functions satisfy the following properties, among others:

- $\log_b(1) = 0$
- $\log_b(b) = 1$
- $\log_b(x\cdot y) = log_b(x) + \log_b(y)$
- $\log_b(rac{x}{y}) = \log_b(x) - \log_b(y)$
- $\log_b(x^n) = n\log_b(x)$
- $\log_b(\sqrt[n]{x}) = rac{\log_b(x)}{n}$
- $\log_b(n) = rac{\log_a(n)}{\log_a(b)}$]

[summary(Inverse exponentials): Logarithms are the inverse of [exponential exponentials]. That is, for any base $b$ and number $n,$ $\log_b(b^n) = n$ and $b^{\log_b(n)} = n.$]

[summary(Measure of data): A message that singles out one thing from a set of $n$ carries $\log(n)$ units of data, where the unit of information depends on the base of the logarithm. For example, a message singling out one thing from 1024 carries about three [3wn decits] of data (because $\log_{10}(1024) pprox 3$), or exactly ten [3p0 bits] of data (because $\log_2(1024)=10$). For details, see [+3p0].]

[summary(Generalized lengths):  A quick way of approximating the logarithm base 10 is to look at the length of a number: 103 is a 3-digit number but it's _almost_ a 2-digit number, so its logarithm (base ten) is a little higher than 2 (it's about 2.01). 981 is also a three-digit number, and it's using nearly all three of those digits, so its logarithm (base ten) is just barely lower than 3 (it's about 2.99). In this way, logarithms generalize the notion of "length," and in particular, $\log_b(n)$ measures the generalized length of the number $n$ when it's written in $b$-ary notation.]

The logarithm base $b$ of a number $n,$ written $\log_b(n),$ is the answer to the question "how many times do you have to multiply 1 by $b$ to get $n$?" For example, $\log_{10}(100)=2,$ and $\log_{10}(316) pprox 2.5,$ because $316 pprox$ $10 \cdot 10 \cdot \sqrt{10},$ and [ multiplying by $\sqrt{10}$ corresponds to multiplying by 10 "half a time"].

In other words, $\log_b(x)$ counts the number of $b$-factors in $x$. For example, $\log_2(100)$ counts the number of "doublings" in the number 100, and $6 < \log_2(100) < 7$ because scaling an object up by a factor of 100 requires more than 6 (but less than 7) doublings. For an introduction to logarithms, see the [3wj Arbital logarithm tutorial]. For an advanced introduction, see the [advanced_log_tutorial advanced logarithm tutorial].

Formally, $\log_b(n)$ is defined to be the number $x$ such that $b^x = n,$ where $b$ and $n$ are numbers. $b$ is called the "base" of the logarithm, and has a relationship to the [number_base base of a number system]. For a discussion of common and useful bases for logarithms, see the page on [logarithm_bases logarithm bases]. $x$ is unique if by "number" we mean [4bc $\mathbb R$], but may not be unique if by "number" we mean [complex_number $\mathbb C$]. For details, see the page on [complex_logarithm complex logarithms].

# Basic properties

Logarithms satisfy a number of desirable properties, including:

- $\log_b(1) = 0$ for any $b$
- $\log_b(b) = 1$ for any $b$
- $\log_b(x\cdot y) = log_b(x) + \log_b(y)$
- $\log_b(x^n) = n\log_b(x)$
- $\log_a(n) = rac{\log_b(n)}{\log_b(a)}$

For an expanded list of properties, explanations of what they mean, and the reasons for why they hold, see [+3wp].

# Interpretations

- Logarithms can be interpreted as a generalization of the notion of the [number_length length of a number]: 103 and 981 are both three digits long, but, intuitively, 103 is only barely using three digits, whereas 981 is pushing its three digits to the limit. Logarithms quantify this intuition: the [common_logarithm common logarithm] of 103 is approximately 2.01, and the common log of 981 is approximately 2.99. Logarithms give rise to a notion of exactly how many digits a number is "actually" making use of, and give us a notion of "fractional digits." For more on this interpretation (and why it is 316, not 500, that is two and a half digits long), see [+416].

- Logarithms can be interpreted as a measure of how much data it takes to carry a message. Imagine that you and I are both facing a collection of 100 different objects, and I'm thinking of one of them in particular. If I want to tell you which one I'm thinking of, how many digits do I need to transmit to you? The answer is $\log_{10}(100)=2,$ assuming that by "digit" we mean "some method of encoding one of the symbols 0-9 in a physical medium." Measuring data in this way is the cornerstone of [3qq information theory].

- Logarithms are the inverse of exponentials. The function $\log_b(\cdot)$ inverts the function $b^{\ \cdot}.$ In other words, $\log_b(n) = x$ implies that $b^x = n,$ so $\log_b(b^x)=x$ and $b^{\log_b(n)}=n.$ Thus, logarithms give us tools for analyzing anything that grows exponentially. If a population of bacteria doubles each day, then logarithms measure days in terms of bacteria — that is, they can tell you how long it will take for the population to reach a certain size. For more on this idea, see [+3wr].

# Applications

Logarithms are ubiquitous in many fields, including mathematics, physics, computer science, cognitive science, and artificial intelligence, to name a few. For example:

- In mathematics, the most natural logarithmic base is [mathematics_e $e$] ([log_e_is_natural Why?]) and the log base $e$ of $x$ is written $\ln(x)$, pronounced "[natural_logarithm natural log] of x." The natural logarithm of a number gives one notion of the "intrinsic length" of a number, a concept that proves useful when reasoning about other properties of that number. For example, the quantity of prime numbers smaller than $x$ is approximately $rac{x}{\ln(x)},$ this is the [prime_number_theorem prime number theorem].

- Logarithms also give us tools for measuring the runtime (or memory usage) of algorithms. When an algorithm uses a [divide_and_conquer divide and conquer] approach, the amount of time (or memory) used by the algorithm increases logarithmically as the input size grows linearly. For example, the amount of time that it takes to perform a [binary_search binary search] through $n$ possibilities is $\log_2(n),$ which means that the search takes one unit longer to run every time the set of things to search through doubles in size.

- Logarithms give us tools for studying the tools we use to represent numbers. For example, humans tend to use ten different symbols to represent numbers (0, 1, 2, 3, 4, 5, 6, 7, 8, and 9), while computers tend to use two digits (0 and 1). Are some representations better or worse than others? What are the pros and cons of using more or fewer symbols? For more on these questions, see [number_base Number bases].

- The human brain encodes various perceptions logarithmically. For example, the perceived tone of a sound goes up by one octave every time the frequency of air vibrations doubles. Your perception of tone is proportional to the logarithm (base 2) of the frequency at which the air is vibrating. See also [https://en.wikipedia.org/wiki/Hick%27s_law Hick's law].


Relation

%%comment:
I do not want to be shortened. The motivation for this is that I would prefer that someone has the ability to learn everything they need to know about relations just by reading the popup summary.
%%

[summary: A **relation** is a [3jz set] of [tuple_mathematics tuples], all of which have the same [tuple_arity arity]. The inclusion of a tuple in a relation indicates that the components of the tuple are related. A set of $n$-tuples is called an $n$*-ary relation*. Sets of pairs are called binary relations, sets of triples are called ternary relations, etc.

Examples of binary relations include the equality relation on natural numbers $\{ (0,0), (1,1), (2,2), ... \}$ and the predecessor relation $\{ (0,1), (1,2), (2,3), ... \}$. When a symbol is used to denote a specific binary relation ($R$ is commonly used for this purpose),  that symbol can be used with infix notation to denote set membership: $xRy$ means that the pair $(x,y)$ is an element of the set $R$.]

A **relation** is a [3jz set] of [tuple_mathematics tuples], all of which have the same [todo: generalize the function_arity page to include general arity][tuple_arity arity]. The inclusion of a tuple in a relation indicates that the components of the tuple are related. A set of $n$-tuples is called an $n$*-ary relation*. Sets of pairs are called binary relations, sets of triples are called ternary relations, etc.

Examples of binary relations include the equality relation on natural numbers $\{ (0,0), (1,1), (2,2), ... \}$ and the predecessor relation $\{ (0,1), (1,2), (2,3), ... \}$. When a symbol is used to denote a specific binary relation ($R$ is commonly used for this purpose),  that symbol can be used with infix notation to denote set membership: $xRy$ means that the pair $(x,y)$ is an element of the set $R$.


Bit (of data)

[summary: A bit of data is the amount of data required to single out one message from a set of two. Equivalently, it is the amount of data required to cut the set of possible messages in half. If you want to single out one of the hands of one of your biological grandparents (such as the left hand of your paternal grandmother), you can do that by transmitting three bits of data: One to single out "left hand" or "right hand", one to single out "maternal" or "paternal", and one to single out "grandfather" or "grandmother". We can also speak of storing data in physical systems which can be put into multiple states: A coin can be used to store a single bit of data (by placing it either heads or tails); a normal six-sided die can be used to store a little over [3ty 2.5 bits] of data. In general, a message that singles out one message from a set of $n$ is defined to contain [3nd $\log_2(n)$] bits of data.]

A bit of data (not to be confused with a [shannon], an [3y3 abstract bit], or a [evidence_bit bit of evidence]) is the amount of data required to single out one message from a set of two. If you want to single out one of the hands of one of your biological grandparents (such as the left hand of your paternal grandmother), you can do that with three bits of data: One to single out "left hand" or "right hand", one to single out "maternal" or "paternal", and one to single out "grandfather" or "grandmother". In order for someone to look at the data and know which hand you singled out, they need to know what method you used to [encoding encode] the message into the data.

Data can be stored in physical systems which can be put into multiple states: A coin can be used to store a single bit of data (by placing it either heads or tails); a normal six-sided die can be used to store a little over [3ty 2.5 bits] of data. In general, a message that singles out one thing from a set of $n$ is defined to contain [3nd $\log_2(n)$] bits of data.

"Bit" is a portmanteau of "binary digit": "binary" because it is the amount of information required to single out one message from a set of 2. The 2 is arbitrary; analogous units of data exist for every other number. For example, a [3ww] is the amount of data required to single out one message from a set of three, a [3wn] is the amount of data it takes to single out one message from a set of ten, and so on. It is easy to convert between units, for example, a decit is $\log_2(10) pprox 3.32$ bits, because it takes a little over three bits to pick one thing out from a set of 10. See also the pages on [ converting between units of data] and [3ty fractional bits].

[fixme: Talk about how we want to define data such that two objects hold twice as much.]

The amount of data you can transmit (or store) grows [exponential exponentially] in the number of bits at your disposal. For example, consider a punch card that can hold ten bits of data. You can use one punch card to pick out a single thing from a set of $2^{10}=1024.$ From two punch cards you can pick out a single thing from a set of $2^{20}=<PHONE>.$ The number of _things you can distinguish_ using two punch cards is 1024 times larger than the number of things you can distinguish with one punch card, and the amount of data you can encode using two punch cards is precisely twice as much (20 bits) as the amount of information you can encode using one punch card (10 bits). In other words, you can single out one object from a collection of $n$ (or store a number between 1 and $n$) using [3nd $\log_2(n)$] bits of data. For more on why the amount of data is logarithmic in the number of possible messages, see [ Data is logarithmic].

[fixme: Add a formalization section.]


Information theory

The study (and quantificaiton) of information, and its communication and storage.


Partially ordered set

[summary: A partially ordered set (also called a poset) is a pair $\langle P, \leq
angle$ of a set $P$ and a [3nt binary relation] $\leq$ on $P$ such that for all $p,q,r \in P$, the following properties are satisfied:

- [5dy Reflexivity]: $p \leq p$
- [573 Transitivity]: $p \leq q$ and  $q \leq r$ implies $p \leq r$
- [5lt Anti-symmetry]: $p \leq q$ and $q \leq p$ implies $p = q$

$P$ is referred to as the poset's underlying set and $\leq$
 is referred to as its order relation. When the order relation of a poset $\langle P, \leq
angle$ is clear from context, the poset can be referred to merely as $P$. Posets are the central object of study in [3dt order theory].]

A **partially ordered set** (also called a **poset**) is a pair $\langle P, \leq
angle$ of a set $P$ and a [3nt binary relation] $\leq$ on $P$ such that for all $p,q,r \in P$, the following properties are satisfied:

- [5dy Reflexivity]: $p \leq p$
- [573 Transitivity]: $p \leq q$ and  $q \leq r$ implies $p \leq r$
- [5lt Anti-symmetry]: $p \leq q$ and $q \leq p$ implies $p = q$

$P$ is referred to as the poset's underlying set and $\leq$
 is referred to as its order relation. Posets are the central object of study in [3dt order theory].

Notations and fundamentals
=======================

Greater than
--------------------

A greater than relation $\geq$ can be derived as the transpose of a poset's less than relation: $a \geq b$ means that $b \leq a$.

Incomparability
-------------------------

For poset elements $p$ and $q$, if neither $p \leq q$ nor $q \leq p$ then we say that $p$ and $q$ are *incomparable*, and write $p \parallel q$.

Strict orders
--------------------

From any poset $\langle P, \leq
angle$, we can derive an underlying strict order $<$. For $p, q \in P$, $p < q$ means that the following conditions are satisfied:

- 1.) $p \leq q$
- 2.) $p
eq q$

The cover relation
-----------------------------

From any poset $\langle P, \leq
angle$, we can derive an underlying *cover relation* $\prec$, defined such that for $p,q \in P$, $p \prec q$ whenever the following two conditions are satisfied:

- 1.) $p < q$
- 2.) For all $r \in P$, $p \leq r < q$ implies $p = r$

Simply put, $p \prec q$ means that $q$ is the smallest element of $P$ which is strictly greater than $p$.
$p \prec q$ is pronounced "$p$ is covered by $q$", or "$q$ covers $p$", and $q$ is said to be a *cover* of $p$.

Hasse diagrams
--------------------------

Let $\langle P, \leq
angle$ be the poset such that $P = \{ p, q, r \}$ and $\leq = \{(p,p),(p,q)(p,r),(q,q),(q,r),(r,r) \}$. The order relation $\leq$, being binary, can be depicted as a directed graph, though the resulting image leaves something to be desired.

![Naive first attempt at representing P with a directed graph](http://i.imgur.com/fOkTwXW.png)

%%%comment:
dot source:

digraph G {
  node [width = 0.1, height = 0.1]
  rankdir = BT;
  p -> q;
  q -> r;
  p -> p;
  r -> r;
  q -> q;
  p -> r;
}
%%%

Including an edge for every pair in $\leq$ results in a rather noisy looking graph. In particular, as long as we are under assumption that the graph we are viewing depicts a poset, placing a reflexive edge on each node is tedious and redundant. Our first step toward cleaning up this graph is to remove all reflexive edges, instead leaving reflexivity as an implicit assumption. Likewise, we can leave those edges which are implied by transitivity — $(p,r)$ is the only such edge in this poset — implicit. As a final step, we remove the arrow heads from our edges, leaving their directions implied by the y-coordinates of the nodes they connect: an edge between two nodes means that the poset element corresponding to the lower node is less than the poset element corresponding to the higher node. The simplified depiction of $\langle P, \leq
angle$ then follows.

![Hasse diagram for P](http://i.imgur.com/W8erVrC.png)

%%%comment:
dot source:

digraph G {
  node [width = 0.1, height = 0.1];
  edge [arrowhead = "none"];
  rankdir = BT;
  p -> q;
  q -> r;
}
%%%

Such a depiction is called a *Hasse diagram*; drawing a Hasse diagram is the standard method for visually depicting a poset. Now that we understand what Hasse diagrams are, we can provide a concise definition: a Hasse diagram is a graph-based visual depiction of a poset $\langle P, \leq
angle$, where elements of $P$ are represented as nodes, and for all $(p,q) \in P^2$ such that $p \prec q$, an upward edge is drawn from $p$ to $q$.

$p \leq q$ requires that $p$ must appear lower in a Hasse diagram than $q$. However, the converse is not true.  In the following Hasse diagram, $t \parallel r$ even though $t$ is positioned lower than $r$.

![t is lower than r but incomparable to r](http://i.imgur.com/hmyUZv9.png)

%%%comment:
dot source:

digraph G {
  node [width = 0.1, height = 0.1];
  edge [arrowhead = "none"];
  rankdir = BT;
  p -> t;
  p -> r;
  t -> q;
  q -> s;
  r -> s;
}
%%%

Duality
-----------

Every poset $\langle P, \leq
angle$ has a corresponding *dual* poset $\langle P^{\partial}, \geq
angle$, where $P^{\partial}$ (pronounced "P op") is a set whose elements are in correspondence with those of $P$, and $\geq$ is the transpose of $\leq$ discussed previously. The Hasse diagram of $P^{\partial}$ is obtained by flipping the Hasse diagram of $P$ upside down. Whenever $\phi$ is a propsition about posets, we obtain $\phi$'s *dual proposition* $\phi^{\partial}$ by replacing all occurences of $\leq$ with $\geq$ and all occurrences of $\geq$ with $\leq$.

The existence of dual posets and dual propositions gives rise to the *duality principle*: if a proposition $orall P. \phi$, quantified over all posets, is true then its dual $orall P. \phi^{\partial}$ is also true. The duality principle works because instantiating $orall P. \phi$ with a poset $P$ is equivalent to instantiating $orall P. \phi^{\partial}$ with $P^{\partial}$. This is due to the fact that $a \leq b$ in $P$ iff $a \geq b$ in $P^{\partial}$. Theorems involving posets often come with a free dual theorem thanks to the duality principle.

%%%knows-requisite([4c7]):
Poset as category
==============

Any poset $\langle P, \leq
angle$ can be viewed as a [category category] in which the objects are the elements of $P$, and for all $p,q \in P$, there is a unique morphism from $p$ to $q$ iff $p \leq q$. This category has closure under morphism composition due to the transitivity of $\leq$ and identity morphisms due to the reflexivity of $\leq$. Associativity of morphism composition stems from the uniqueness of arrows between any pair of objects. The functors between poset categories are [poset_monotone_map monotone maps].
%%%

Additional Material
----------------------------------

For some additional examples of posets, see [43s].
To test your knowledge of posets, try these [4l1 exercises].
The most useful operators on elements of a poset are called the [3rc join and meet].
We can relate the elements of one poset to another through the use of [5jg monotone functions].


Join and meet

[summary: Let $\langle P, \leq
angle$ be a [3rb poset], and let $S \subseteq P$. The **join** of $S$ in $P$, denoted by $igvee_P S$, is an element $p \in P$ satisfying the following two properties:

* p is an *upper bound* of $S$; that is, for all $s \in S$, $s \leq p$.
* For all upper bounds $q$ of $S$ in $P$, $p \leq q$.

$igvee_P S$ does not necessarily exist, but if it does then it is unique. The notation $igvee S$ is typically used instead of $igvee_P S$ when $P$ is clear from context. Joins are often called *least upper bounds* or *supremums*. For $a, b$ in $P$, the join of $\{a,b\}$ in $P$ is denoted by $a ee_P b$, or $a ee b$ when $P$ is clear from context. **Meets** are greatest lower bounds, and are related to joins by duality.
]

Let $\langle P, \leq
angle$ be a [-3rb], and let $S \subseteq P$. The **join** of $S$ in $P$, denoted by $igvee_P S$, is an element $p \in P$ satisfying the following two properties:

* p is an *upper bound* of $S$; that is, for all $s \in S$, $s \leq p$.
* For all upper bounds $q$ of $S$ in $P$, $p \leq q$.

$igvee_P S$ does not necessarily exist, but if it does then it is unique. The notation $igvee S$ is typically used instead of $igvee_P S$ when $P$ is clear from context. Joins are often called *least upper bounds* or *supremums*. For $a, b$ in $P$, the join of $\{a,b\}$ in $P$ is denoted by $a ee_P b$, or $a ee b$ when $P$ is clear from context.

The dual concept of the join is that of the meet. The **meet** of $S$ in $P$, denoted by $igwedge_P S$, is defined an element $p \in P$ satisfying.

* p is a *lower bound* of $S$; that is, for all $s$ in $S$, $p \leq s$.
* For all lower bounds $q$ of $S$ in $P$, $q \leq p$.

Meets are also called *infimums*, or *greatest lower bounds*. The notations $igwedge S$, $p \wedge_P q$, and $p \wedge q$ are all have meanings that are completely analogous to the aforementioned notations for joins.

Basic example
--------------------------

![Joins Failing to exist in a finite lattice](http://i.imgur.com/sx1Ss9w.png)

The above Hasse diagram represents a poset with elements $a$, $b$, $c$, and $d$. $igvee \{a,b\}$ does not exist because the set $\{a,b\}$ has no upper bounds. $igvee \{c,d\}$ does not exist for a different reason: although $\{c, d\}$ has upper bounds $a$ and $b$, these upper bounds are incomparable, and so $\{c, d\}$ has no *least* upper bound. There do exist subsets of this poset which possess joins; for example, $a ee c = a$, $igvee \{b,c,d\} = b$, and $igvee \{c\} = c$.

Now for some examples of meets. $igwedge \{a, b, c, d\}$ does not exist because $c$ and $d$ have no common lower bounds. However, $igwedge \{a,b,d\} = d$ and $a \wedge c = c$.

Additional Material
---------------------------------

* [3v4 Examples]
* [4ll Exercises]

Further reading
---------------
* [46c Lattices]


Start

Start-Class pages are visibly incomplete, or need significant work to be useful.  This covers pages missing essential information, pages without coherent structure, pages lacking [1kl summaries] (or first paragraphs that serve as a summary), pages with sufficiently severe prose or stylistic issues, and any other problem that prevents pages from being useful to readers or fitting into their domains.  Start-Class pages have significant content, but are still in the category "should be seen by editors" rather than "should be seen by readers."

Contrast:

- [72], for tiny pages containing only a paragraph or a couple of sentences of text.
- [4gs], for pages which only contain a formal definition and don't try to explain the topic.

**Also** use the tag [4v] if the page is being actively edited.

**[4yg Quality scale]**

* [4ym]
* [4gs]
* [72]
* [3rk]
* [4y7]
* [4yd]
* [4yf]
* [4yl]


Type theory

Hub page for type theory explanations


Group: Examples

[summary:
Examples of [-3gd groups], including the [-497 symmetric groups] and [-general_linear_group general linear groups].
]

# The symmetric groups

For every positive integer $n$ there is a group $S_n$, the [497 symmetric group] of order $n$, defined as the group of all permutations (bijections) $\{ 1, 2, \dots n \} 	o \{ 1, 2, \dots n \}$ (or any other [-3jz] with $n$ elements). The symmetric groups play a central role in group theory: for example, a [3t9 group action] of a group $G$ on a set $X$ with $n$ elements is the same as a [47t homomorphism] $G 	o S_n$.

Up to [4bj conjugacy], a permutation is determined by its [4cg cycle type].

# The dihedral groups

The [4cy dihedral groups] $D_{2n}$ are the collections of symmetries of an $n$-sided regular polygon. It has a [5j9 presentation] $\langle r, f \mid r^n,  f^2,  (rf)^2
angle$, where $r$ represents rotation by $	au/n$ degrees, and $f$ represents reflection.

For $n > 2$, the dihedral groups are non-commutative.

# The general linear groups

For every [481 field] $K$ and positive integer $n$ there is a group $GL_n(K)$, the [general_linear_group general linear group] of order $n$ over $K$. Concretely, this is the group of all invertible $n 	imes n$ [matrix matrices] with entries in $K$; more abstractly, this is the [automorphism automorphism group] of a [3w0 vector space] of [vector_space_dimension dimension] $n$ over $K$.

If $K$ is [algebraically_closed_field algebraically closed], then up to conjugacy, a matrix is determined by its [Jordan_normal_form Jordan normal form].


Sample space

[summary: A sample space is the [3jz set] $\Omega$ of possible ways that a part of the world might be, as far as you know.]

Motivation
===

If we are uncertain about some part of the world, then there are multiple different things that might happen there, as far as we know.
The sample space for a part of the world is the set of things that could possibly happen  there. We can use a sample space to reason using [1bv probability theory], so that we assign probabilities to [event_probability events] in a consistent way.

Definition
===

A sample space is a [3jz set], usually denoted $\Omega$.

[todo: examples]
[todo: further points]


Group theory: Examples

# Even and odd functions

Recall that a function $f : \mathbb{R} \to \mathbb{R}$ is [even_function even] if $f(-x) = f(x)$, and [odd_function odd] if $f(-x) = - f(x)$. A typical example of an even function is $f(x) = x^2$ or $f(x) = \cos x$, while a typical example of an odd function is $f(x) = x^3$ or $f(x) =   \sin x$.

We can think about evenness and oddness in terms of [3g8 group theory] as follows. There is a group called the [cyclic_group cyclic group] $C_2$ of [3gg order] $2$ acting on the set of functions $\mathbb{R} \to \mathbb{R}$: in other words, each element of the group describes a function of [3sz type]

$$ (\mathbb{R} \to \mathbb{R}) \to (\mathbb{R} \to \mathbb{R}) $$

meaning that it takes as input a function $\mathbb{R} \to \mathbb{R}$ and returns as output another function $\mathbb{R} \to \mathbb{R}$.

$C_2$ has two elements which we'll call $1$ and $-1$. $1$ is the identity element: it acts on functions by sending a function $f(x)$ to the same function $f(x)$ again. $-1$ sends a function $f(x)$ to the function $f(-x)$, which visually corresponds to reflecting the graph of $f(x)$ through the y-axis. The group multiplication is what the names of the group elements suggests, and in particular $(-1) \times (-1) = 1$, which corresponds to the fact that $f(-(-x)) = f(x)$.

Any time a group $G$ [3t9 acts] on a set $X$, it's interesting to ask what elements are [invariant_under_a_group_action invariant] under that group action. Here the invariants of functions under the action of $C_2$ above are the even functions, and they form a [subspace] of the [vector_space vector space] of all functions.

It turns out that every function is uniquely the sum of an even and an odd function, as follows:

$$f(x) = \underbrace{\frac{f(x) + f(-x)}{2}}_{\text{even}} + \underbrace{\frac{f(x) - f(-x)}{2}}_{\text{odd}}.$$

This is a special case of various more general facts in [3tn representation theory], and in particular can be thought of as the simplest case of the [discrete_Fourier_transform discrete Fourier transform], which in turn is a [mathematical_toy_model toy model] of the theory of [Fourier_series Fourier series] and the [Fourier_transform Fourier transform].

It's also interesting to observe that the cyclic group $C_2$ shows up in lots of other places in mathematics as well. For example, it is also the group describing how even and odd numbers add1 (where even corresponds to $1$ and odd corresponds to $-1$); this is the simplest case of [modular_arithmetic modular arithmetic].

1That is: an even plus an even make an even, an odd plus an odd make an even, and an even plus an odd make an odd.


Programming in  Dependent Type Theory

All code in this article was written in the Lean theorem prover, which means you can copy any of it and paste it [here](https://leanprover.github.io/tutorial/?live) to try it out.

## Arithmetic and Interaction
While Lean is nominally an interactive theorem prover, much of the power of type theory comes from the fact that you can treat it like a programming language.

There are a few different commands for interacting with Lean.
The one I make the most use of is the `check` command, which prints out the type of the expression following it.
So, for example, `check 3` outputs `num`, and `check (3 : nat)` outputs `nat`.

We can also make definitions

    definition five : num := 3 + 2

This declares a new constant `five` of type `num` to which we give the value `3 + 2`.
We can define functions with similar syntax

    definition num.const₀ : num → num → num := λ x y, x -- this is a comment
    -- Lean can infer types whenever they're unique
    definition num.const₁                    := λ (x y : num), x
    -- we can also name the arguments in the definition rather than the function body
    definition num.const₂ (x y : num)        := x

The definition of polymorphic functions becomes the first point where we get a hint about what makes programming in dependent type theory different from, say, Haskell.
In dependent type theory, the term and type languages are unified, so in order to write a polymorphic function we must take the type as an argument.

    definition poly_id (A : Type) := λ (a : A), a
    -- or, equivalently
    definition poly_id₁ := λ A (a : A), a
    -- applied to arguments
    check poly_id num 1 -- num
    check poly_id (num → num → num) num.const -- num → num → num

Exercise: write a polymorphic version of `num.const₀`.

%%hidden(Show solution):
    definition poly_const (A B : Type) (a : A) (b : B) := a
%%

Having to explicitly indicate types everywhere is a pain.
In order to get around that, most proof assistants provide support for implicit arguments, which let you leave out arguments that only have one valid value.
In Lean, the syntax for implicits looks like this:

    definition id {A : Type} := λ (a : A), a

## Inductive types ##
Of course, none of this would be that useful if we couldn't define *new* types.
There are lots of ways to craft new types in dependent type theory, but among the most fundamental is the creation of inductive types.

To define a new inductive type, you give a list of constructor tags, each associated with a type representing the arguments it takes.
The simplest ones are just enumerations. For example, the days of the week:

    inductive weekday : Type :=
    | mon : weekday
    | tue : weekday
    | wed : weekday
    | thu : weekday
    | fri : weekday
    | sat : weekday
    | sun : weekday

This creates a new type `weekday`, and seven new constants (`weekday.mon`, `weekday.tue`, `weekday.wed`...) of type `weekday`.
If you're familiar with Haskell, you'll correctly notice that this looks an awful lot like GADT declarations.

Just like in Haskell, we can parametrize our types over other types, making new types like `either`:

    inductive either (A B : Type) : Type :=
    | inl {} : A → either A B``
    | inr {} : B → either A B

From this declaration, we get a new constant `either : Type → Type → Type`.
This represents the union of the types `A` and `B`, the type of values that belong either to `A` or `B`.

We can also define recursive types, such as natural numbers

    inductive nat : Type :=
    | zero : nat
    | succ : nat → nat

The easiest way to define functions over `nat`s is recursively.
For example, we can define addition as

    definition add (n : nat) : nat -> nat
    | nat.zero := n
    | (nat.succ m) := nat.succ (add m) -- n is constant at every recursive call

Bringing both of these together we can define the type of linked lists

    inductive list (A : Type) : Type :=
    | nil {} : list A
    | cons : A → list A → list A

We can also define functions over lists by pattern matching

    definition map {A B : Type} (f : A → B) : list A → list B
    | list.nil := list.nil
    | (list.cons x xs) := list.cons (f x) (map xs) -- f is constant at every recursive call

Exercise: write `foldr` and `foldl` by pattern matching

%%hidden(Show solution):
    definition foldr {A B : Type} (r : A → B → B) (vnil : B) : list A → B
    | list.nil := vnil
    | (list.cons x xs) := r x (foldr xs)

    definition foldl {A B : Type} (r : B → A → B) : B → list A → B :=
    | b list.nil := b
    | b (list.cons x xs) := foldl (r b x) xs
%%


Group action

An action of a [-3gd] $G$ on a [-3jz] $X$ is a function $lpha : G 	imes X 	o X$ ([3vl colon-to notation]), which is often written $(g, x) \mapsto gx$ ([3vm mapsto notation]), with $lpha$ omitted from the notation, such that

1. $ex = x$ for all $x \in X$, where $e$ is the identity, and
2. $g(hx) = (gh)x$ for all $g, h \in G, x \in X$, where $gh$ implicitly refers to the group operation in $G$ (also omitted from the notation).

Equivalently, via [currying], an action of $G$ on $X$ is a [47t group homomorphism] $G 	o 	ext{Aut}(X)$, where $	ext{Aut}(X)$ is the [automorphism_group automorphism group] of $X$ (so for sets, the group of all bijections $X 	o X$, but phrasing the definition this way makes it natural to generalize to other [category_theory categories]). It's a good exercise to verify this; Arbital [49c has a proof].

Group actions are used to make precise the notion of "symmetry" in mathematics.

# Examples

Let $X = \mathbb{R}^2$ be the [Euclidean_geometry Euclidean plane]. There's a group acting on $\mathbb{R}^2$ called the [Euclidean_group Euclidean group] $ISO(2)$ which consists of all functions $f : \mathbb{R}^2 	o \mathbb{R}^2$ preserving distances between two points (or equivalently all [isometry isometries]). Its elements include translations, rotations about various points, and reflections about various lines.


Probability distribution (countable sample space)

[summary: A probability distribution on a countable [3t4] $\Omega$ is a [3jy] $\mathbb{P}: \Omega 	o [0,1]$ such that $\sum_{\omega \in \Omega} \mathbb{P}(\omega) = 1$.]

Definition
===

A probability distribution on a countable [3t4] $\Omega$ is a [3jy] $\mathbb{P}: \Omega 	o [0,1]$ such that $\sum_{\omega \in \Omega} \mathbb{P}(\omega) = 1$.

Intuition
=====

We express a belief that "$x\in \Omega$ happens with probability $r$" by setting $\mathbb{P}(x) = r$. So a probability distribution divides up our anticipation of what will happen, out of the set $\Omega$ of things that might possibly happen.

[todo: examples, futher points]


Sample spaces are too large

[summary: Sample spaces enumerate everything that could possibly happen, so that we can reason consistently under uncertainty. But this enumeration usually enumerates too many things, making it difficult to do computations using the raw sample space.]

When we are reasoning about an uncertain world using [1bv probability theory], we have a [3t4 sample space] of possible ways the world could be, and we assign
probabilities to outcomes in the sample space with a [3tb probability distribution]. Unfortunately, often the sample space (which is supposed to contain every possible way the world might turn out) is very large and possibly infinite.

For example, if we are thinking about a box with some particles in it, our sample space is the enormous set of all possible arrangements of the particles.
To reason usefully about the box, we'd have to do computations using a gigantic table specifying a separate number for every single distinct arrangement of particles.
Two arrangements that are pretty much the same, as far as we care, take up just as much of our computational resources as two importantly different arrangements.
So, even though we could in principle reason consistently under uncertainty just using a probability distribution over a sample space, we cannot do so easily,
because we are trying to keep track of too much.

[todo: examples]


Group: Exercises

[summary:
Test your understanding of the definition of a [-3gd group] with these exercises.
]

# Preliminaries

1. Show that the identity element in a group is unique. That is, if $G$ is a group and two elements $e_1, e_2 \in G$ both satisfy the axioms describing the identity element, then $e_1 = e_2$.

%%hidden(Show solution):
By definition, an identity element $e$ satisfies $eg = ge = g$ for all $g \in G$. Hence if $e_1$ is an identity, then $e_1 e_2 = e_2 e_1 = e_1$. And if $e_2$ is an identity, then $e_2 e_1 = e_1 e_2 = e_2$. Hence $e_1 = e_2$. Note that this argument makes no use of inverses, and so is valid for [3h3 monoids].
%%

2. Show that inverses are also unique. That is, if $g \in G$ is an element of a group and $h_1, h_2 \in G$ both satisfy the axioms describing the inverse of $g$, then $h_1 = h_2$.

%%hidden(Show solution):
By definition, an inverse $h$ of $g$ satisfies $hg = gh = e$. So $h_1 g = g h_1 = e$ and $h_2 g = g h_2 = e$. Hence, on the one hand,

$$h_1 g h_2 = (h_1 g) h_2 = (e) h_2 = h_2$$

and, on the other hand,

$$h_1 g h_2 = h_1 (g h_2) = h_1 (e) = h_1.$$

Hence $h_1 = h_2$.
%%

# Examples involving numbers

Determine whether the following sets equipped with the specified binary operations are groups. If so, describe their identity elements (which by the previous exercise must be unique) and how to take inverses.

1. The real numbers $\mathbb{R}$ together with the addition operation $(x, y) \mapsto x + y$.

%%hidden(Show answer):
Yes, this is a group. The identity element is $0$, and inverse is given by $x \mapsto -x$.
%%

2. The real numbers $\mathbb{R}$ together with the multiplication operation $(x, y) \mapsto xy$.

%%hidden(Show answer):
No, this is not a group. $0 \in \mathbb{R}$ has the property that $0 	imes x = 0$ for all real numbers $x$, so it can't be invertible no matter what the identity is.
%%

3. The positive real numbers $\mathbb{R}_{>0}$ together with the multiplication operation $(x, y) \mapsto xy$.

%%hidden(Show answer):
Yes, this is a group. The identity is $1$, and inverse is given by $x \mapsto rac{1}{x}$. In fact this group is [49x isomorphic] to $(\mathbb{R}, +)$; can you name the isomorphism?
%%

4. The real numbers $\mathbb{R}$ together with the operation $(x, y) \mapsto x + y - 1$.

%%hidden(Show answer):
Yes, this is a group (in fact [49x isomorphic] to $(\mathbb{R}, +)$; can you name the isomorphism?). The identity element is $1$, and inverse is given by $x \mapsto 2 - x$ (can you explain why, conceptually?).
%%

5. The real numbers $\mathbb{R}$ together with the operation $(x, y) \mapsto rac{x + y}{1 + xy}$.

%%hidden(Show answer):
No, this is not a group. It's easy to be tricked into thinking it is, because if you just work through the algebra, it seems that all of the group axioms hold. However, this operation is not an operation! It's not defined if the denominator is $0$, because then we'd be [division_by_zero dividing by zero].

This operation is interesting and useful, though, when it is defined. It shows up in [special_relativity special relativity], where it describes how velocities add relativistically (in units where the speed of light is $1$).
%%


Uncountable sample spaces are way too large

[summary:
If the sample space $\Omega$ is [2w0 uncountable], then in general we can't even define a probability distribution over $\Omega$ in the same way we defined
[3tb probability distributions] over countable sample spaces, i.e. by just assigning numbers to each point in the sample space.  Any function $f: \Omega 	o [0,1]$ with $\sum_{\omega \in \Omega} f(\omega) = 1$ can only assign positive values to at most countably many elements of $\Omega$. ]

If the sample space $\Omega$ is [2w0 uncountable], then in general we can't even define a probability distribution over $\Omega$ in the same way we defined
[3tb probability distributions] over countable sample spaces, i.e. by just assigning numbers to each point in the sample space.  Any function $f: \Omega 	o [0,1]$ with $\sum_{\omega \in \Omega} f(\omega) = 1$ can only assign positive values to at most countably many elements of $\Omega$. But this means we can't, for example, talk about a
[uniform_distribution uniform distribution] over the interval $[0,2]$, which intuitively should assign equal probability to everywhere in the interval.


Linear algebra

The study of [linear_transformation linear transformations] and [vector_space vector spaces].


Bit (of data): Examples

In the game "20 questions", one player (the "leader") thinks of a concept, and the other players ask 20 yes-or-no questions in attempts to guess that concept. In this game, the players extract 20 bits of data from the leader, which (if the questions are selected wisely) is enough data to single out one concept from a set of roughly one million concepts (<PHONE>, to be precise).

Three bits of data are required to specify one of the hands of one of your biological grandparents: One bit for "maternal" or "paternal", one bit for "male" or "female," one bit for "left" or "right."

The president of the United States of America is selected from a set of about 320,000,000 people. This selection encodes roughly 28 bits of data. Where do those bits come from? Between 1/4 and 1/2 of the population is eligible for the presidency, so 1-2 bits come from constraints on who is allowed to be president. Another ~1 bit comes from the voters in the presidential election (when one candidate is selected from a set of two finalists). 2-4 more bits come from the primary elections (when the two finalists are selected from a set of ~4-32). However, most of the bits that go into choosing the president — 21 or more of them — come from self-selection and private politicking.


Fractional bits

It takes $\log_2(8) = 3$ [3p0 bits] of data to [3y1 carry] one message from a set of 8 possible messages. Similarly, it takes $\log_2(1024) = 10$ bits to carry one message from a set of 1024 possibilities. How many bits does it take to carry one message from a set of 3 possibilities? By the definition of "bit," the answers is $\log_2(3) pprox 1.58.$ What does this mean? That it takes about one and a half yes-or-no questions to single out one thing from a set of three? What is "half of a yes-or-no question"?

Fractional bits can be interpreted as telling about the [4b5 expected] cost of transmitting information: If you want to single out one thing from a set of 3 using bits (in, e.g., the [+3tz] thought experiment) then you'll have to purchase two bits, but sometimes you'll be able to sell one of them back, which means you can push your expected cost lower than 2. How low can you push it? The lower bound is $\log_2(3),$ which is the minimum expected cost of adding a [3v9 3-message] to a long message when encoding your message as cleverly as possible. For more on this interpretation, see [+3v2].

Fractional bits can also be interpreted as conversion rates between types of information: "A 3-message carries about 1.58 bits" can be interpreted as "one [3ww] is worth about 1.58 bits." To understand this, see [427], or [ How many bits is a trit?]

Fractional units of data can also be interpreted as a measure of how much we're using the digits allocated to encoding a number. For example, working with fractional [3wn decits] instead of fractional bits, it only takes about 2.7 decits carry a [3v9 500-message], despite the fact that the number 500 clearly requires 3 decimal digits to write down. What's going on here? Well, we could declare that all numbers that start with a number $n \ge 5$ will be interpreted as if they start with the number $n - 5.$ Then we have two ways of representing each number (for example, 132 can be represented as both 132 and 632). Thus, if we have 3 decits but we only need to encode a 500-message, we have one bit to spare: We can encode one extra bit in our message according to whether we use the low representation or the high representation of the intended number. Thus, the amount of data it takes to communicate a 500-message is one bit lower than the amount of data it takes to encode a 1000-message — for a total cost of 3 decits minus one bit, which comes out to about 2.70 decits (or just short of 9 bits). For more on this interpretation, see [+44l].


GalCom

[summary: In the GalCom thought experiment, you live in the future, and make your money by living in the Deneb star system and sending information to an automated stock trader you own in the Vega star system. Sending interstellar information is possible, but very expensive — and you have a lot of data to send. If you're clever, you can reprogram the computer on Vega in the low-cost off-hours, but to send the key market data (at the critical moment) you have to reserve high-cost precisely-timed bits far in advance. When reserving those bits you have to pick a fixed set of messages and buy enough bits to cover the longest message that you might want to send. If you end up sending a shorter message you can sell some of the bits back. You send a very high volume of data, so small decreases in the average cost of sending a bit are worth a lot of effort.]

[summary(Rules):

1. It costs 1 galcoin per bit to reserve on-peak bits in advance. Galcoins are very expensive.
2. You can reprogram your machine on Vega in advance, using cheap, off-peak bits. Don't worry about the cost of those.
3. You can resell reserved bits that you don't use (including fractional bits).
4. Any decrease in the expected cost of transmitting a message, no matter how small, is worthwhile.]

_The GalCom hypothetical is a thought experiment where sending a bit of information is a clearly defined and very expensive action, which makes it useful for understanding various concepts in [-3qq] and [-1bv]._

The year is 21026. Humanity has become a flourishing inter-stellar civilization. Capitalism never died, and stock markets are thriving in various star systems. Due to the light-speed limitations, the markets are out of sync, and there's a lot of money to be made by setting up an automated trader in one star system, moving to another star system, and sending information about the stock market to your automated trader the instant that that information becomes available.

You set up your own automated trader in the Vega star system, and you currently make a living by transmitting market data from the Deneb star system. To transmit that data, you use the deep-space Galactic Communications network known as "GalCom."

You have a lot of data to transmit, and sending information on GalCom isn't cheap. Fortunately, it _is_ efficient: GalCom is a highly optimized network used by trillions of citizens to send and receive pulses of light across the cosmos. To send information via GalCom, you purchase [3p0 bits]. For each bit you purchase, you are allowed to control a single pulse in the GalCom signal, making it either be present (representing a 1) or absent (representing a 0).

It's relatively cheap to buy bits on non-peak hours (to, e.g., reprogram your machine on Vega), but it's very expensive to reserve bits during peak hours (i.e., just after a juicy earnings report is published). You make your money by being among the very first people to send information about the stock market, so you have to reserve those very expensive precisely-timed bits in advance. This means that you have to know what messages you might want to send in advance, and reserve enough bits for the longest possible message that you might send. Fortunately, there's a secondary market on peak-hour bits, so if you end up reserving 5 bits and then only using 3 of them, you can resell the last two bits.

For example, say you know that you're going to send one of the following messages: `"buy"`, `"sell"`, or `"hold"`. GalCom would require you to purchase 4 bytes of information (enough to transmit four letters), and then if you actually end up transmitting `"buy"` you can sell the last byte back (because you only used 3 of the 4).

Of course, you can send that message using much less than 4 bytes, if you're clever, both by developing efficient [ encoding schemes] and by [ accounting for the fact that some outcomes are more likely than others]. Your profit margins depend on you finding ways to transmit information as efficiently as possible. To do that, you can use [-3qq].


GalCom: Rules

1. It costs 1 galcoin per bit to reserve on-peak bits in advance. (Galcoins are very expensive.)
2. You can reprogram your machine on Vega in advance, using slow, off-peak bits. The costs of reprogramming your machine on Vega to make more efficient use of the on-peak bits are inconsequential compared to the gains when you send on-peak bits, that is, you don't need to worry about the cost of reprogramming your machine.
3. If you wind up not needing all the on-peak bits that you reserved, you can resell the bits you did not use (assuming your computer on Vega uses the appropriate protocols, which is the sort of thing you can set up in advance).
4. The market for bits is very, very competitive; you are allowed to resell [3ty fractional bits] (including [trit trits], [decit decits], and so on).
5. You make your living by sending lots and lots of data, and your profits depend on your efficiency. Due to the volume of information you transmit, you are risk-neutral in the cost of bits, and thus, very small decreases in the [expectation expected cost] of transmitting a message are worth quite a bit of money to you.


Real analysis

Real analysis is a branch of mathematics that deals with real numbers, real-valued functions, sequences, series, differentiation, integration, and related subjects.


Fractional bits: Expected cost interpretation

In the [3tz GalCom] thought experiment, you regularly have to send large volumes of information through deep space to your automated stock trader in the Vega star system, and every bit of information is very expensive. Imagine that a very important earnings report is coming up for Company X, and you want to transmit instructions to your automated trader in the Vega system the instant that the information becomes available. You've decided that you're capable of distinguishing between 7 different outcomes, which each appear equally likely to you. You label them -3, -2, -1, 0, +1, +2, and +3 (according to how the company performed compared to your median estimate). How many bits do you need to reserve on GalCom to ensure that you can send one of those 7 messages the moment that the earnings report comes out?

$\log_2(7)$ is about 2.807, which implies that two bits won't do — with two bits, you could only single out one message from a set of four. So you have to purchase at least 3 bits. (In general, if you want to send a message on GalCom that distinguishes between $n$ outcomes, you have to reserve at least [3vc $\lceil \log_2(n)
ceil$] bits.)

Once you've purchased 3 bits, there are 8 different signals that you'll be able to send via GalCom to the Vega system: 000, 001, 010, 011, 100, 101, 110, and 111. You can program your automated trader to interpret these signals as -3, -2, -1, 0, +1, +2, and +3 respectively, and then you have one code — 111 — left over. Is there any way to make money off the fact that you have an extra code?

(You're invited to pause and attempt to answer this question yourself. Is there any way to program the computer on Vega to use your 8 codes in such a way that you sometimes get to sell one bit back to GalCom?)

You can recoup some of the losses as follows: Program your machine on Vega to recognize both 110 _and_ 111 as +3. Then, if the earnings report for company X comes out as "very good," you get to resell one bit to GalCom. Why? Because both 110 and 111 will be interpreted as +3, so you don't care which one gets sent. Therefore, you can choose to send 110 if the next GalCom bit was going to be a 0, and 111 if the next GalCom bit was going to be a 1, and thereby shave one bit off of the next person's message (assuming your automated trader is programmed to forward that one bit where it needs to go, which is the sort of thing you can set up ahead of time).

![](http://i.imgur.com/mbJ4Apo.png)

_3 bits are enough to single out a leaf in a [binary_tree binary tree] of depth 3. You only have seven messages you want to encode, which means one of the codes gets doubled up — in this case, both 110 and 111 are assigned to +3. Thus, your computer on Vega already knows that the answer is +3 by the time GalCom has transmitted 11, so if the earnings report comes up +3 you can repurpose the third bit._

Because you set up your codes such that you thought each outcome was equally likely, your [4b5 expected cost] of sending the message is 2.875 bits: 7/8ths of the time you pay for 3 bits, but 1/8th of the time you only pay for 2 bits. If you're sending one message from a set of 7, you have to set aside at least 3 bits at the beginning, but every so often you get to sell one of them back.

If you send many, many [3v9 7-messages] using the encoding above, then on average, each 7-message costs 2.875 bits.

But $\log_2(7)
eq 2.875,$ it's actually close to about 2.807. What gives?

In short, the above encoding is not the most efficient way to pack 7-messages. Or, rather, it's the most efficient way to pack one 7-message in isolation, but if we have lots of 7-messages, we can do better on average. Imagine two people teaming up to send two 7-messages. Two 7-messages is the same as one 49-message, where $(m, n)$ is encoded as $7m + n,$ i.e., 17 means "2" to the first person and "3" to the second. Sending a 49 message requires reserving $\lceil \log_2(49)
ceil = 6$ bits on GalCom, thereby purchasing 64 different possible signals (000000 through 111111). $64 - 49 = 15$ of those signals will have a double meaning, which means that 15/49 of the time, one bit can be resold to GalCom. Thus, the expected cost to send two messages from two sets of seven is $6 - rac{15}{49} pprox 5.694$ bits. When the two parties split the cost, they each pay only half that, or roughly 2.847 bits in expectation — slightly better than the 2.875 they would have paid sending one 7-message in isolation.

If you pack three 7-messages together, the expected cost comes out to $(9 - rac{169}{343})pprox 8.507$ bits, for an average cost of $pprox 2.836$ bits per 7-message: slightly closer still to the $2.807$ bound. In the limit, as you pack more and more 7-messages together, $\log_2(7)$ is the smallest average cost you can get. For more on why this is the lower bound, see [trit_in_bits How many bits is a trit?], [427], and [44l].

Thus, we can interpret "a 7-message carries about 2.807 bits of information" as "sending lots of 7-messages costs about 2.807 bits on average, if you pack your message as cleverly as possible."

More generally, the fractional portion of the amount of information carried by a message can be interpreted as the lower bound for the expected cost of sending that message. If we have to send a single $n$-message across GalCom in isolation, then we have to reserve at least $\lceil \log_2(n)
ceil$ bits. But if we append our message to a longer message, and pack our message carefully, then we can push our expected cost lower and lower. What's the lower bound on the expected cost? $\log_2(n).$

Why is $\log_2(n)$ the lower bound? The proof has two parts. First, we have to show that you can in fact get closer and closer to an average cost of $\log_2(n)$ as you send more and more messages. Second, imagine you could send a $b$-message for $x < \log_2(b)$ bits, on average. Then you could use $b$-messages to encode lots and lots of 2-messages, which (by the same method) would cost on average $\log_b(2) \cdot x$ bits per $2$-message — $\log_b(2)$ $b$-messages per $2$-message, times $x$ bits per $b$-message. But $\log_b(2) \cdot \log_2(b) = 1$ for all $b,$ which means that, if you could send a $b$-message for less than $\log_2(b)$ bits, then you could use bits to send more than one 2-message per bit!


Join and meet: Examples

A union of sets and the least common multiple of a set of natural numbers can both be viewed as joins. In addition, joins can be useful to the designers of statically typed programming languages.

%%%knows-requisite([3v1]):
The real numbers
------------------------------

Consider the [-3rb] $\langle \mathbb{R}, \leq
angle$ of all real numbers ordered by the standard comparison relation. For any non-empty $X \subseteq \mathbb{R}$, $igvee X$ exists if and only if $X$ has an upper bound; this fact falls directly out of the definition of the set of real numbers.
%%%

Subtyping
------------------

Statically typed programming languages often define a poset of types ordered by the subtyping relation; Scala is one such language. Consider the following Scala program.

![A scala program with class inheritance](http://i.imgur.com/IUkzT7A.png)

When a programmer defines a class hierarchy in an object-oriented language, they are actually defining a poset of types. The above program defines the simple poset shown in the following Hasse diagram.

![Hasse diagram for the nominal subtyping relation defined by the above program](http://i.imgur.com/dk1EHuA.png)

%%%comment:
dot source:

digraph G {
  node [width = 0.1, height = 0.1];
  edge [arrowhead = "none"];
  rankdir = BT;
  Dog -> Animal;
  Cat -> Animal;
}
%%%

Now consider the expression *if (b) dog else cat*. If b is true, then it evaluates to a value of type Dog. If b is false, then it evaluates to a value of type Cat. What type, then, should *if (b) dog else cat* have? Its type should be the join of Dog and Cat, which is Animal.

Power sets
-------------------

Let $X$ be a [-3jz]. Consider the [-3rb] $\langle \mathcal{P}(X), \subseteq
angle$, the power set of $X$ ordered by inclusion. In this poset, joins are unions: for all $A \subseteq \mathcal{P}(X)$, $igvee A = igcup A$. This can be shown as follows. Let $A \subseteq \mathcal{P}(X)$. Then $igcup A$ is an upper bound of $A$ because a union contains each of its constituent sets. Furthermore, $igcup A$ is the *least* upper bound of $A$. For let $Z$ be an upper bound of $A$. Then $x \in igcup A$ implies $x \in Y$ for some $Y \in A$, and since $Y \subseteq Z$, we have $x \in Y \subseteq Z$. Since $x \in igcup A$ implies $x \in Z$, we have $igcup A \subseteq Z$. Hence, $igvee A = igcup A$.

Divisibility
------------------

Consider the poset $\langle \mathbb Z_+, |
angle$ of divisibility on the positive integers. In this poset, the upper bounds of an integer are exactly its multiples. Thus, the join of a set of positive integers in $\langle \mathbb Z_+, |
angle$ is their [least_common_multiple least common multiple]. Dually, the meet of a set of positive integers in $\langle \mathbb Z_+, |
angle$ is their [greatest_common_divisor greatest common divisor].


n-message

[summary: A message singling out one thing from a set of $n$ is sometimes called an $n$-message. For example, a message singling out one thing from a set of 13 is a 13-message. An $n$-message carries $\log_2(n)$ [3p0 bits] of information.]

A message singling out one thing from a set of $n$ is sometimes called an $n$-message. For example, a message singling out one thing from a set of 13 is a 13-message. An $n$-message carries $\log_2(n)$ [3p0 bits] of information. For a formal account of what it means for a message to "single out one thing from a set of $n$," see [3xd].


Thought experiment

Meta-tag for thought experiments.


Ceiling

The ceiling of a [real_number real number] $x,$ denoted $\lceil x
ceil$ or sometimes $\operatorname{ceil}(x),$ is the first [integer] $n \ge x.$ For example, $\lceil 3.72
ceil = 4, \lceil 4
ceil = 4,$ and $\lceil -3.72
ceil = -3.$ In other words, the ceiling function rounds its input up to the nearest integer.

For the function that rounds its input down to the nearest integer, see [floor]. Ceiling and floor are not to be confused with [fix_towards_zero fix] and [ceilfix], which round towards and away from zero (respectively).

Formally, ceiling is a [3jy function] of type $\mathbb R 	o \mathbb Z.$ The ceiling function can also be defined on [complex_number complex numbers].


Colon-to notation

In mathematics, the notation $f : X 	o Y$ (here, "colon-to notation," because the arrow $	o$ is written "	o" in [LaTeX]) means that $f$ is a [3jy function] with [3js domain] $X$ and [3lg codomain] $Y$. It can be read "$f$, a function from $X$ to $Y$."

This can be thought of as ascribing a function type to the value $f$. The use of a colon to express that a given value has a given type, as is done in type theory, is a generalization of this notation.

# Examples

$f : \mathbb{R} 	o \mathbb{R}$ means that $f$ is a function from the real numbers to the real numbers, such as $x \mapsto x^2$ ([3vm mapsto notation]).

$f : \mathbb{R} 	imes \mathbb{R} 	o \mathbb{R}$ means that $f$ is a function from pairs of real numbers to real numbers. The $	imes$ here refers to the [3xb Cartesian product] of [3jz sets].


Mapsto notation

In mathematics, the arrow $\mapsto$ (which in [LaTeX] is called "\mapsto") is used to describe what a [3jy function] does to an arbitrary input. It is commonly used in combination with [3vl colon-to notation], which describes what a function's [3js domain] and [3lg codomain] are.

By itself, $\mapsto$ can be used to describe a function without naming it, and so is a way to describe [anonymous_function anonymous functions] in mathematics.

# Examples

The function $f(x) = x^2$ from the [4bc real numbers] to the real numbers can be described using a combination of colon-to notation and mapsto notation as

$$f : \mathbb{R} 	o \mathbb{R}$$
$$x \mapsto x^2.$$

In one line, although this is less common,

$$f : \mathbb{R}
i x \mapsto x^2 \in \mathbb{R}.$$

(see [3vq in notation]).


In notation

In mathematics, the notation $x \in X$ (where $\in$ is written "\in" in [5xw]) means that $X$ is a [3jz set] and $x$ is an element of that set.

# Examples

$r \in \mathbb{R}$ means that $r$ is a [4bc real number].


Vector space

A vector space is a [algebraic_field field] $F$ paired with a [3gd] $V$ and a function $\cdot : F 	imes V 	o V$ (called "scalar multiplication") such that $1 \cdot v = v$ and such that scalar multiplication distributes over addition. Elements of the field are called "scalars," elements of the group are called "vectors."

Also there are some nice geometric interpretations and stuff.


Subspace

A subspace $U=(F_U, V_U)$ of a [3w0] $W=(F_W, V_W)$ is a vector space where $F_U = F_W$ and $V_U$ is a [subgroup] of $V_W,$ and $V_U$ is [ closed] under scalar multiplication.


Sum of vector spaces

The sum of two [3w0 vector spaces] $U$ and $W,$ written $U + W,$ is a vector space where the set of vectors is all possible $u + w$ for every $u \in U$ and $w \in W.$


Direct sum of vector spaces

The direct sum of two [3w0 vector spaces] $U$ and $W,$ written $U \oplus W,$ is just the [3w2 sum] of $U$ and $W,$ but it can only be applied when $U$ and $W$ are [ linearly independent].


Logarithm: Examples

$\log_{10}(100)=2.$ $\log_2(4)=2.$ $\log_2(3)pprox 1.58.$ (TODO)


Logarithm: Exercises

Without using a calculator: What is $\log_{10}(4321)$? What integer is it larger than, what integer is it smaller than, and which integer is it closer to? How do you know? (TODO)


Introductory guide to logarithms

Welcome to the Arbital introduction to logarithms! In modern education, logarithms are often mentioned but rarely motivated. At best, students are told that logarithms are just a tool for inverting [4ts exponentials]. At worst, they're told a bunch of properties of the logarithm that they're expected to memorize, just because. The goal of this tutorial is to explore what logarithms are actually doing, and help you build an intuition for how they work.

For example, one motivation we will explore is this: Logarithms measure how long a number is when you write it down, for a generalized notion of "length" that allows fractional lengths. The number 139 is three digits long:

$$\underbrace{139}_\text{3 digits}$$

and the logarithm (base 10) of 139 is pretty close to 3. It's actually closer to 2 than it is to 3, because 139 is closer to the largest 2-digit number than it is to the largest 3-digit number. Specifically, $\log_{10}(139) \approx 2.14$. We can interpret this as saying "139 is three digits long in [-4sl], but it's not really using its third digit to the fullest extent."

You might be thinking "Wait, what do you mean it's closer to 2 digits than it is to 3? It plainly takes three digits: '1', '3', and '9'. What does it mean to say that 139 is 'almost' a 2-digit number?"

You might also be wondering what it means to say that a number is "two and a half digits long," and you might be surprised that it is 316 (rather than 500) that is most naturally seen as 2.5 digits long. Why? What does that mean?

These questions and others will be answered throughout the tutorial, as we explore what logarithms actually do.

%box:
This path contains 9 pages:

1. [40j What is a logarithm?]
2. [416 Log as generalized length]
3. [427 Exchange rates between digits]
4. [44l Fractional digits]
5. [45q Log as the change in the cost of communicating]
6. [47k The characteristic of the logarithm]
7. [4gp The log lattice]
8. [4h0 Life in logspace]
9. [4h2 The End (of the basic log tutorial)]
%


Decit

A decit, or "decimal digit" (also known as a "dit", "ban", or "hartley") is the amount of [3xd information] required to single out one thing from a set of ten. That is, a decit is a measure of information equal to $\log_2(10)pprox 3.32$ [shannon shannons].


Logarithmic identities

[summary:

- [ Inversion of exponentials]: $b^{\log_b(n)} = \log_b(b^n) = n.$
- [ Log of 1 is 0]: $\log_b(1) = 0$
- [ Log of the base is 1]: $\log_b(b) = 1$
- [ Multiplication is addition in logspace]: $\log_b(x\cdot y) = log_b(x) + \log_b(y).$
- [ Exponentiation is multiplication in logspace]: $\log_b(x^n) = n\log_b(x).$
- [ Symmetry across log exponents]: $x^{\log_b(y)} = y^{\log_b(x)}.$
- [ Change of base]: $\log_b(n) = rac{\log_a(n)}{\log_a(b)}$]

Recall that [3nd $\log_b(n)$] is defined to be the (possibly fractional) number of times that you have to multiply 1 by $b$ to get $n.$ Logarithm functions  satisfy the following properties, for any base $b$:

- [ Inversion of exponentials]: $b^{\log_b(n)} = \log_b(b^n) = n.$
- [ Log of 1 is 0]: $\log_b(1) = 0$
- [ Log of the base is 1]: $\log_b(b) = 1$
- [ Multiplication is addition in logspace]: $\log_b(x\cdot y) = log_b(x) + \log_b(y).$
- [ Exponentiation is multiplication in logspace]: $\log_b(x^n) = n\log_b(x).$
- [ Symmetry across log exponents]: $x^{\log_b(y)} = y^{\log_b(x)}.$
- [ Change of base]: $\log_a(n) = rac{\log_b(n)}{\log_b(a)}$


Needs lenses

A tag for pages that have a technical/non-intuitive main lens, and which need lenses that can explain the concept less formally.


Logarithms invert exponentials

The function [3nd $\log_b(\cdot)$] inverts the function $b^{(\cdot)}.$ In other words, $\log_b(n) = x$ implies that $b^x = n,$ so $\log_b(b^x)=x$ and $b^{\log_b(n)}=n.$ (For example, $\log_2(2^3) = 3$ and $2^{\log_2(8)} = 8.$) Thus, logarithms give us tools for analyzing anything that grows [4ts exponentially]. If a population of bacteria grows exponentially, then logarithms can be used to answer questions about how long it will take the population to reach a certain size. If your wealth is accumulating interest, logarithms can be used to ask how long it will take until you have a certain amount of wealth. (TODO)


How many bits to a trit?

[summary: $\log_2(3) pprox 1.585.$ This can be interpreted a few different ways:

1. If you multiply the number of messages you might want to send by 3, then the cost of encoding the message will go up by 1.58 bits on average.
2. If you pack $n$ of independent and equally likely [3v9 3-messages] together into one giant $3^n$ message, then the cost (in bits) per individual 3-message drops as $n$ grows, ultimately converging to $\log_2(3)$ bits per 3-message as $n$ gets very large.
3. The infinite expansion of $\log_2(3) = <PHONE>\ldots$ tells us not just how many bits it takes to send one 3-message $(pprox \lceil 1.585
ceil = 2)$ but also how long it takes to send any number of 3-messages put together. For example, it costs 2 bits to send one 3-message; 16 bits to send 10; 159 bits to send 1000; 1585 to send 10,000; 15850 to send 100,000; 158497 to send 1,000,000; and so on.
]

There are [3nd $\log_2(3) pprox 1.585$] [3p0 bits] to a [3ww]. This can be interpreted a few different ways:

1. If you multiply the number of messages you might want to send by 3, then the cost of encoding the message will go up by 1.58 bits on average. See [+marginal_message_cost] for more on this interpretation.
2. If you pack $n$ of independent and equally likely [3v9 3-messages] together into one giant $3^n$ message, then the cost (in bits) per individual 3-message drops as $n$ grows, ultimately converging to $\log_2(3)$ bits per 3-message as $n$ gets very large. For more on this, see [+average_message_cost] and the [3zn GalCom example of encoding trits using bits].
3. The infinite expansion of $\log_2(3) = <PHONE>\ldots$ tells us not just how many bits it takes to send one 3-message $(pprox \lceil 1.585
ceil = 2)$ but also how long it takes to send any number of 3-messages put together. For example, it costs 2 bits to send one 3-message; 16 bits to send 10; 159 bits to send 1000; 1585 to send 10,000; 15850 to send 100,000; 158497 to send 1,000,000; and so on. For more on this interpretation, see the [log_series_of_ceilings "series of ceilings"] interpretation of logarithms.


Trit

[summary: The trinary analog of the [3y2]. Depending on the context, a trit can mean any of the following things:

1. Abstract trit: one thing from a set of three
2. Trit of data: the amount of data carried by an object that can be put into three states
3. Trit of information: $\log_2(3) pprox 1.58$ [shannon Sh] of information
4. Trit of evidence: $3:1$ evidence in favor of a specific hypothesis]

A trit is the trinary analog of the [3y2]: Where a bit says 2, a trit says 3. "Trit," like "bit," is overloaded:

- An abstract trit (analog of the [3y3 abstract bit]) is an element of a [3jz set] with  three elements. Thus, there are three abstract trits. They are often denoted 0, 1, and 2.
- A trit of data (analog of a [3p0 bit of data]) is the amount of data that can be carried by a physical object that can be placed in any one of three states.
- A trit of information (analog of a [shannon]) is the amount of information someone receives when they start out maximally uncertain about which one of three outcomes happened, and then they learn which one happened. A trit of information is $\log_2(3)pprox 1.58$ bits of information.
- A trit of evidence (analog of a [evidence_bit bit of evidence]) in favor of a hypothesis is an observation that is three times as likely if the hypothesis is true (versus if the hypothesis is false).

"Trit" is a portmanteau of "trinary digit."


Intradependent encodings can be compressed

Given an encoding scheme $E$ which gives an [3x4] of a message $m,$ we can in principle design a more efficient coding $E^\prime$ that gives a shorter encoding of $m.$ For example, $E$ encodes 8-letter English words as a series of letters, $m$ is aardvark, then $E(m)$ is intradependent (because you already know what the message is by the time you've seen "aardv"), so you can define an alternative encoding $E^\prime$ which encodes "aardvark" as just "aardv", thereby saving three letters.

The practice of taking an intradependent encoding and finding a more efficient one is known as [compression].


Intradependent encoding

[summary: An encoding $E(m)$ of a message $m$ is intradependent if the fact that $E(m)$ encodes $m$ can be deduced without looking at the whole encoding. For example, if you know you're going to see an 8-letter English word, and you see "aardv", you know that the message is "aardvark", even without looking at the last three letters.

Formally, intradependence is defined in terms of the mutual information shared by symbols in the encoding. Given an encoding is intradependent, it's possible (in principle) to develop a more efficient encoding that is less intradependent.]

We call an [encoding] $E(m)$ of a message $m$ "intradependent" if the fact that $E(m)$ encodes $m$ can be deduced without looking at the whole encoding. For example, imagine that you know you're going to see an 8-letter English word, and you see the letters "aardv". You can then deduce that the message is  "aardvark" without looking at the last three letters, because that's the only valid message that starts with "aardv".

(Seeing "aard" is not enough, because [https://en.wikipedia.org/wiki/Aardwolf aardwolfs] exist.)

In an intradependent encoding, some parts of the encoding carry information about the other parts. For example, once you've seen "aard", there are $26^4 = 456976$ possible combinations of the next four letters, but "aard" cuts the space down to just two possibilities — "vark" and "wolf". This means that the first four letters carry $\log_2(26^4) - 1 pprox 17.8$ bits of information about the last four. (The fifth letter carries one final bit of information, in the choice between "v" and "w". The last three letters carry no new information.)

Formally, the intradependence of an encoding is defined to be the sum of [mutual_information mutual information] between each symbol in the encoding and all the previous symbols. Given an intradependent encoding, it is possible in principle to design a more efficient encoding; see [+intradependent_compression].


Non-standard terminology

A tag for terminology that is Arbital-specific, Arbital-originated, or just not very common outside of Arbital.


Dependent messages can be encoded cheaply

Say you want to transmit a [3v9 2-message], a 4-message, and a 256-message to somebody. For example, you might want to tell them which way a coin came up, a cardinal direction, and a letter (encoded as one byte of [https://en.wikipedia.org/wiki/ASCII ASCII] text). How many [3p0 bits] of information does it take to transmit those three messages?

At most, it takes 11 bits: One for the coin, two for the direction, and 8 for the ASCII byte. For example, [heads, north, A] might be encoded as <PHONE>, where 0 means "heads", 00 means "north", and <PHONE> means "A" in ASCII.

But what if the messages depend on each other? What if the way that the cardinal direction was picked was by looking at the coin (such that you always say north if the coin lands heads, and south if the coin lands tails), and then the letter is picked by looking at the direction (such that you always say A for north, B for east, Z for south, and Y for west). Then how many bits does it take to transmit the message [heads, north, A]? Only one! Why? Because there are now only two possible messages: [heads, north, A] and [tails, south, Z]. Given two people who know the links between the three messages, all you need to tell them is how the coin came up, and they can figure out the entire message.

Formally, if you want to send multiple messages, and those messages share [mutual_information mutual information], then the amount of [information] it takes to encode all three messages together is less than the amount of information it takes to encode each one separately. (In fact, the amount of information you can save by encoding them both together is at most the amount of mutual information between them).

Looking at the collection of multiple messages as a single message, this fact is an immediate consequence of the fact that [ some messages being more likely than others means you can develop more efficient encodings for them].

Alternatively, this fact can be seen as a corollary of the fact that [intradependent_compression intradependent encodings can be compressed]: Given three messages $m_1, m_2, m_3$ and an encoding scheme $E$, the encoding $E(m_1)E(m_2)E(m_3)$ made by simply putting all three encodings together can be interpreted as a single [3x4 intradependent encoding] of the triplet $(m_1, m_2, m_3)$.


Cartesian product

The Cartesian product of two sets $A$ and $B,$ denoted $A 	imes B,$ is the set of all [ordered_pair ordered pairs] $(a, b)$ such that $a \in A$ and $b \in B.$ For example, if $\mathbb B 	imes \mathbb N$ is the set of all pairs of a [boolean] with a [natural_number natural number], and it contains elements like (true, 0), (false, 17), (true, 17), (true, 100), and (false, 101).

Cartesian product are often referred to as just "products."

Cartesian products can be constructed from more than two sets, for example, $\mathbb B^3 = \mathbb B 	imes \mathbb B 	imes \mathbb B$ is the set of all [boolean] [tuple 3-tuples]. (The $	imes$ operator is [3h4 associative], so we don't need to write parenthesis when using it on a whole chain of sets.) A product of $n$ sets is called an $n$-ary product.


Information

[summary: Information is a measure of how much a message grants an observer the ability to predict the world. Information is observer-dependent: If someone tells both of us your age, you'd learn nothing, while I'd learn how old you are. Information is measured in [shannon shannons]. If you are about to observe a coin that came up either "heads" or "tails," one shannon is the difference between utter uncertainty about which way the coin came up, and total certainty that it came up "heads." Information can be used to quantify both (a) how much uncertainty a person has; and (b) how much uncertainty a message resolves.]

[summary(Technical): Given a probability distribution $\mathrm P$ over a set $O$ of possible observations, an observation $o \in O$ is said to carry $\log rac{1}{\mathrm P(o)}$ units of information (with respect to $\mathrm P$), where the base of the logarithm determines the unit of information. The standard choice is log base 2, in which case the information is measured in [shannon shannons].]

Information is a measure of how much a message grants an observer the ability to predict the world. For a formal description of what this means, see [ Information: Formalization]. Information is observer-dependent: If someone tells both of us your age, you'd learn nothing, while I'd learn how old you are. [+3qq] gives us tools for quantifying and studying information.

Information is measured in [shannon shannons], which are also the units used to measure [uncertainty] and [entropy]. Given that you're about to observe a coin that certainly came up either heads or tails, one shannon is the difference between utter uncertainty about which way the coin came up, and total certainty that the coin came up heads. Specifically, the amount of information in an observation is quantified as the [3nd] of the [reciprocal] of the [1rf] that the observer assigned to that observation.

For a version of the previous sentence written in English, see [measuring_information Measuring information]. For a discussion of why this quantity in particular is called "information," see [ Information: Intro].

# Information vs Data

The word "information" has a precise, technical meaning within the field of [3qq]. Information is not to be confused with [data], which is a measure of how many messages a communication medium can distinguish in principle. For example, a series of three ones and zeros is 3 [3p0 bits of data], but the amount of information those three bits carry depends on the observer. Unfortunately, in colloquial usage (and even in some texts on information theory!) the word "information" is used interchangeably with the word "data"; matters are not helped by the fact that the standard unit of information is sometimes called a "bit" (the name for the standard unit of data), despite the fact that these units are distinct. The proper name for a binary unit of information is a "[shannon]."

That said, there are many links between information and data (and between shannons and bits). For instance:

* An object with a [3y1] of $n$ bits can carry anywhere between 0 and $\infty$ [shannon shannons] of information (depending on the observer), but the maximum amount of information an observer can consistently [expectation _expect_] from observing the object is $n$ shannons. For details, see [+expected_info_capacity].
* The number of shannons an observer gets from an observation is equal to the number of bits in the [encoding encoding] for that observation in their [ideal_encoding]. In other words, shannons measure the number of bits of data you would use to communicate that observation to someone who knows everything you know (except for that one observation). For details, see [+ideal_encoding] and [ Information as encoding length].

For more on the difference between information and data, see [+info_vs_data].

# Information and Entropy

[+entropy] is a measure on probability distributions which, intuitively, measures the total uncertainty of that distribution. Specifically, entropy measures the number of shannons of information that the distribution [expectation expects] to gain by being told the actual state of the world. As such, it can be interpreted as a measure of how much information the distribution says it is missing about the world. See also [ Information and entropy].

# Formalization

The amount of information an observation carries to an observer is the [3nd] of the [reciprocal] of the [1rf] that they assigned to that observation. In other words, given a probability distribution $\mathrm P$ over a set $O$ of possible observations, an observation $o \in O$ is said to carry $\log_2rac{1}{\mathrm P(o)}$ shannons of information with respect to $\mathrm P$. A different choice for the base of the logarithm corresponds to a different unit of information; see also [ Converting between units of information]. For a full formalization, see [ Information: Formalization]. For an understanding of why information is logarithmic, see [ Information is logarithmic]. For a full understanding of why we call this quantity in particular "information," see [ Information: Intro].


Data capacity

The data capacity of an object is defined to be the [-3nd] of the number of different distinguishable states the object can be placed into. For example, a coin that can be placed heads or tails has a data capacity of $\log(2)$ units of [-data]. The choice of [-logarithm_base base] for the logarithm determines the unit of data; common units include the [-3p0], [-nat], and [-3wn] (corresponding to base 2, e, and 10). For example, the coin has a data capacity of $\log_2(2)=1$ bit, and a pair of two dice (which can be placed into 36 distinguishable states) have a data capacity of $\log_2(36) pprox 5.17$ bits. Note that the data capacity of a channel depends on the ability of an observer to distinguish different states of the object: If the coin is a penny, and I'm able to tell whether you placed the image of Abraham Lincoln facing North, South, West, or East (regardless of whether the coin is heads or tails), then that coin has a data capacity of $\log_2(8) = 3$ bits when used to transmit a message from you to me.

The data capacity of an object is closely related to the [channel_capacity channel capacity] of a [communications_channel communications channel]. The difference is that the channel capacity is the amount of data the channel can transmit per unit time (measured, e.g., in bits per second), while the data capacity of an object is the amount of data that can be encoded by putting the object into a particular state (measured, e.g., in bits).

Why is data capacity defined as the logarithm of the number of states an object can be placed into? Intuitively, because a 2GB hard drive is supposed to carry twice as much data as a 1GB hard drive. More concretely, note that if you have $n$ copies of a physical object that can be placed into $b$ different states, then you can use those to encode $b^n$ different messages. For example for example, with three coins, you can encode eight different messages: HHH, HHT, HTH, HTT, THH, THT, TTH, and TTT. The number of messages that the objects can encode grows exponentially with the number of copies. Thus, if we want to define a unit of message-carrying-capacity that grows linearly in the number of copies of an object (such that 3 coins hold 3x as much data as 1 coin, and a 9GB hard drive holds 9x as much data as a 1GB hard drive, and so on) then data [47k must] grow logarithmically with the number of messages.

The data capacity of an object bounds the length of the message that you can send using that object. For example, it takes about 5 bits of data to encode a single letter A-Z, so if you want to transmit an 8-letter word to somebody, you need an object with a data capacity of $5 \cdot 8 = 40$ bits. In other words, if you have 40 coins in a coin jar on your desk, and if we worked out an [encoding_scheme encoding scheme] (such as [https://en.wikipedia.org/wiki/ASCII ASCII]) ahead of time, then you can tell me any 8-letter word using those coins.

What does it mean to say that an object "can" be placed into different states, and what does it mean for those states to be "distinguishable"? [+3qq] is largely agnostic about the answer to those questions. Rather, _given_ a set of states that you claim you could put an object into, which you claim I can distinguish, information theory can tell you how to use those objects in a clever way to send messages to me. For more on what it means for two physical subsystems to communicate with each other by encoding messages into their environment, see [communication Communication].

Note that the amount of [-3xd] carried by an object is not generally equal to the data capacity of the object. For example, say a trusted mutual friend of ours places a coin "heads" if your eyes are blue and "tails" otherwise. If I already know what color your eyes are, then the state of the coin doesn't carry any information for me. If instead I was very sure that your eyes are blue, but actually you've been wearing blue contact lenses and your eyes are really brown, then the coin may carry significantly more than 1 [-452] of information for me. See also [+3xd] and [ Information is subjective]. The amount of information carried by an object to me (measured in [452 shannons]) may be either more or less than its data capacity — More if the message is very surprising to me, less if I already knew the message in advance. The relationship between the data capacity of an object and the amount of information it carries to me is that the maximum amount of information I can [4b5 expect] to gain by observing the object is equal to the data capacity. For more, see [ Information and data capacity].


Bit

[summary: The term "bit" refers to different concepts in different fields. The common theme across all the uses is the number 2. Usually, if you see something measured in bits (such as a bit of data, a bit of evidence, or a bit of information) it means that someone is counting the number of 2-factors in something (the number of possibilities, a likelihood ratio, a probability) using [3nd $\log_2$].]

[summary(set): An element of the set [boolean $\mathbb B$], which has two elements. These elements are sometimes called 0 and 1, or true and false, or yes and no. See [3y3 Bit (abstract)].]

[summary(data): A unit of [-data], namely, the amount of data required to single out one message from a set of two, or, equivalently, the amount of data required to cut a set of possible messages in half. See [3p0 Bit of data].]

[summary(information): A unit of [subjective_information information], namely, the difference in certainty between having no idea which way a coin is going to come up, and being entirely certain that it's going to come up heads. While this unit of information is colloquially known as a "bit" (for historical reasons), it is more properly known as a [452].]

[summary(evidence): A unit of [-bayesian_evidence evidence], namely, a $2 : 1$ [-1rq] in favor of one outcome over another. See [evidence_bit Bit of evidence] and [1zh].]

The term "bit" is [-overloaded]. It can mean any of the following:

1. An element of the set [boolean $\mathbb B$], which has two elements. These elements are sometimes called 0 and 1, or true and false, or yes and no. See [3y3 Bit (abstract)].
2. A unit of [data], namely, the amount of data required to single out one message from a set of two, or, equivalently, the amount of data required to cut a set of possible messages in half. See [3p0 Bit of data].
3. A unit of [subjective_information (subjective) information], namely, the difference in certainty between having no idea which way a coin is going to come up, and being entirely certain that it's going to come up heads. While this unit of information is colloquially known as a "bit" (for historical reasons), it is more properly known as a [452].
4. A unit of [bayesian_evidence evidence], namely, a $2 : 1$ [-1rq] in favor of one outcome over another. See [evidence_bit Bit of evidence] and/or [1zh].

The common theme across all the uses listed above is the number 2. An abstract bit is one of two values. A bit of data is a portion of a representation of a message that cuts the set of possible messages in half (i.e., the $\log_2$ of the number of possible messages). A bit of information (aka a [452]) is the amount of information in the answer to a yes-or-no question about which the observer was maximally uncertain (i.e., the $\log_2$ of a probability). A bit of evidence in favor of A over B is an observation which provides twice as much support for A as it does for B (i.e., the $\log_2$ of a [-1rq]). Thus, if you see someone using bits as units (a bit _of data,_ or a bit _of evidence,_ etc.) you can bet that they took a $\log_2$ of something somewhere along the way.

Unfortunately, abstract bits break this pattern, so if you see someone talking about "bits" without disambiguating what sort of bit they mean, the most you can be sure of is that they're talking about something that has to do with the number 2. Unless they're just using the word "bit" to mean "small piece," in which case you're in a bit of trouble.


Bit (abstract)

An abstract bit is an element of the set [boolean $\mathbb B$], which has two elements. An abstract bit is to $\mathbb B$ as a number is to [natural_number $\mathbb N$]. The set $\mathbb N$ contains a bunch of "numbers", and these numbers sometimes have many different labels, such as "three" and "3". Similarly, the set $\mathbb B$ contains a pair of "bits", which sometimes have many different labels. The abstract bits are sometimes labeled 0 and 1; true and false; yes and no; or + and -. The labels don't matter; the point is that the set $\mathbb B$ contains two elements, and we call those elements (abstract) bits.


Compressing multiple messages

How many [3p0 bits of data] does it take to encode an [3v9 $n$-message]? Naively, the answer is $\lceil \log_2(n)
ceil$ ([n_message_bit_length why?]): For example, it takes 5 bits to encode a 21-message, because 4 bits are only enough to encode 16 different messages, but 5 bits are enough to encode 32. The use of the [3vc] function implies an inefficiency: 2 bits are required to encode a 3-message, but 2 bits are enough to distinguish between four different possibilities. One of those possibilities is being wasted. That inefficiency can be reduced by encoding multiple $n$-messages at the same time. For example, while an individual 3-message requires 2 bits to encode, a series of 10 3-messages requires at most 16 bits to encode: $3^{10} < 2^{16}.$

Why is it that encoding ten 3-messages together (using bits) is cheaper than encoding ten 3-messages separately? Naively, there are three different factors that allow the combined encoding to be shorter than the sum of the separate encodings: The messages could have different likelihoods ([expected_compression allowing the combined message to be compressed in expectation]); the messages could be dependent on each other ([compressing_dependent_messages meaning they can be compressed]); and the mismatch between bits and 3-messages gets washed out as we put more three-messages together (see [+3wv]).

In fact, the first two factors are equivalent: 10 3-messages are equivalent to one $3^{10}$ message, and in general, [n_k_messages $n$ $k$-messages are equivalent to one $n^k$-message]. If the individual n-messages are dependent on each other, then different $n^k$ messages have different likelihoods: For example, if message 3 never follows message 2, then in the combined message, "32" never appears as a substring.

Thus, there are two different ways that an encoding of $k$ $n$-messages can be shorter than $k$ times the encoding of an $n$-message: The various combined messages can have different likelihoods, and the efficiency of the coding might increase. To study the effect of different likelihoods on the encoding length in isolation, we can [assume_maximum_efficiency assume that the codings are maximally efficient] and see how much additional [compression] the different likelihoods get us. To study code efficiency in isolation, we can [assume_equal_likelihood_messages assume each message is equally likely] and see how much additional compression we get as we put more $n$-messages together. In practice, real compression involves using both techniques at once.


Communication: magician example

Imagine that you and I are both magicians, performing a trick where I think of a card from a deck of cards and then leave the room, at which point you enter the room and have to guess which card I picked. To perform this trick, I leave some [3xd] in the room that allows you to figure out which card I picked. I do this by [encoding] a [message] into the environment (using objects that can carry [data]). Let's say that our props table includes (among other things) a coin and two dice (one red, one blue). Assume I have the ability to (without the audience noticing) place the coin and the two dice however I like before leaving the room. Those three objects have a [3y1] of $\log_2(2 	imes 6 	imes 6) pprox 6.17$ bits. I can use those objects to [physical_encoding represent] the encoding of the message I want to send to you.

The messages that I want to send are messages saying which card I'm thinking of. Example messages include the message saying that I'm thinking of the ace of spades, or messages saying that I'm thinking of the king of hearts.

The usual way that humans encode messages like these is to use physical objects like pressure waves in air or pixels on a screen or ink on paper arranged to look like the following patterns: "my card is the ace of spades" or "my card is the king of hearts." ([ Don't confuse the message with the encoding.])

Those sentences are pretty long, though: 28 and 29 letters, respectively. A coin and two dice can't carry 29 characters. Fortunately, we don't need all those characters: I can easily encode a message about which card I'm thinking of using only two characters, a rank and a suit, such as $A♠$ or $K♡.$ and so on. However, the coin and dice don't naturally carry rank and suit symbols on them.

However, there are only 52 possible messages I might want to encode, and the coin and dice can be put into $2 \cdot 6 \cdot 6 = 72$ different configurations. Thus, if we're clever, and [ choose a decoding scheme ahead of time], then I can set those objects in such a way that when you see them, you know which card I'm thinking of. Let's say we've agreed on the following [ decoding scheme]:

1. If the coin is heads, the card is red. If the coin is tails, the card is black.
2. If the red die is a 1, 2, or 3 then the suit is either spades or hearts. Otherwise, the suit is either clubs or diamonds.
3. If the red die is a 1 or 4, the card is either A, 2, 3, 4, 5, or 6. If the red die is a 2 or 5, the card is either 7, 8, 9, 10, J, or Q. If the red die is a 3 or a 6, the card is a K.
4. If the blue die is 1, the card is either A or 7 or K; if the blue die is 2, the card is either 2 or 8 or K; If the blue die is 3, the card is either 3 or 9 or K; and so on.

When you walk in the room, you can glance at the coin and the two dice and know which card I picked. If you see that the coin is tails-up, the red die is 1-up, and the blue die is 2-up, then you know that the card is black, either spades or hearts, between A and 6, and either 2 or 8 — in other words, the card is the 2 of spades.