Report likelihoods, not p-values

[summary: If scientists reported likelihood functions instead of p-values, this could help science avoid [https://en.wikipedia.org/wiki/Data_dredging p-hacking], [https://en.wikipedia.org/wiki/Publication_bias publication bias], [https://en.wikipedia.org/wiki/Decline_effect the decline effect], and other hazards of standard statistical techniques. Furthermore, it could help make it easier to combine results from multiple studies and perfrom meta-analyses, while making statistics intuitively easier to understand. (This is a bold claim, but a claim which is largely supported by [-1bv].)]

This page advocates for a change in the way that statistics is done in standard scientific journals. The key idea is to report [56s likelihood functions] instead of p-values, and this could have many benefits.

_(Note: This page is a personal [60p opinion page].)_

[toc:]

## What's the difference?

The status quo across scientific journals is to test data for "[statistically_significant statistical significance]" using functions such as [p_value p-values]. A p-value is a number calculated from a hypothesis (called the "null hypothesis"), an experiment, a result, and a [-summary_statistic]. For example, if the null hypothesis is "this coin is fair," and the experiment is "flip it 6 times", and the result is HHHHHT, and the summary statistic is "the sequence has at least five H values," then the p-value is 0.11, which means "if the coin were fair, and we did this experiment a lot, then only 11% of the sequences generated would have at least five H values."%%note:This does _not_ mean that the coin is 89% likely to be biased! For example, if the only alternative is that the coin is biased towards tails, then HHHHHT is evidence that it's fair. This is a common source of confusion with p-values.%% If the p-value is lower than an arbitrary threshold (usually $p < 0.05$) then the result is called "statistically significant" and the null hypothesis is "rejected."

This page advocates that scientific articles should report _likelihood functions_ instead of p-values. A likelihood function for a piece of evidence $e$ is a function $\mathcal L$ which says, for each hypothesis $H$ in some set of hypotheses, the probability that $H$ assigned to $e$, written [51n $\mathcal L_e(H)$].%%note: Many authors write $\mathcal L(H \mid e)$ instead. We think this is confusing, as then $\mathcal L(H \mid e) = \mathbb P(e \mid H),$ and it's hard enough for students of statistics to keep "probability of $H$ given $e$" and "probability of $e$ given $H$" straight as it is if the notation _isn't_ swapped around every so often.%% For example, if $e$ is "this coin, flipped 6 times, generated HHHHHT", and the set of hypotheses are $H_{0.25} =$ "the coin only produces heads 25% of the time" and $H_{0.5}$ = "the coin is fair", then $\mathcal L_e(H_{0.25})$ $=$ $0.25^5 \cdot 0.75$ $\approx 0.07\%$ and $\mathcal L_e(H_{0.5})$ $=$ $0.5^6$ $\approx 1.56\%,$ for a [1rq likelihood ratio] of about $21 : 1$ in favor of the coin being fair (as opposed to biased 75% towards tails).

In fact, with a single likelihood function, we can report the amount of support $e$ gives to _every_ hypothesis $H_b$ of the form "the coin has bias $b$ towards heads":%note:To learn how this graph was generated, see [1zj Bayes' rule: Functional form].%

![](http://i.imgur.com/UwwxmCe.png)

Note that this likelihood function is _not_ telling us the probability that the coin is actually biased, it is _only_ telling us how much the evidence supports each hypothesis. For example, this graph says that HHHHHT provides about 3.8 times as much evidence for $H_{0.75}$ over $H_{0.5}$, and about 81 times as much evidence for $H_{0.75}$ over $H_{0.25}.$

Note also that the likelihood function doesn't necessarily contain the _right_ hypothesis; for example, the function above shows the support of $e$ for every possible bias on the coin, but it doesn't consider hypotheses like "the coin alternates between H and T". Likelihood functions, like p-values, are essentially a mere summary of the raw data — there is no substitute for the raw data when it comes to allowing people to test hypotheses that the original researchers did not consider. (In other words, even if you report likelihoods instead of p-values, it's still virtuous to share your raw data.)

Where p-values let you measure (roughly) how well the data supports a _single_ "null hypothesis", with an arbitrary 0.05 "not well enough" cutoff, the likelihood function shows the support of the evidence for lots and lots of different hypotheses at once, without any need for an arbitrary cutoff.

## Why report likelihoods instead of p-values?

__1. Likelihood functions are less arbitrary than p-values.__ To report a likelihood function, all you have to do is pick which hypothesis class to generate the likelihood function for. That's your only degree of freedom. This introduces one source of arbitrariness, and if someone wants to check some other hypothesis they still need access to the raw data, but it is better than the p-value case, where you only report a number for a single "null" hypothesis.

Furthermore, in the p-value case, you have to pick not only a null hypothesis but also an experiment and a summary statistic, and these degrees of freedom can have a huge impact on the final report. These extra degrees of freedom are both unnecessary ([1lz to carry out a probabilistic update, all you need are your own personal beliefs and a likelihood function]) and exploitable, and empirically, they're actively harming scientific research.

__2. Reporting likelihoods would solve p-hacking.__ If you're using p-values, then you can game the statistics via your choice of experiment and summary statistics. In the example with the coin above, if you say your experiment and summary statistic are "flip the coin 6 times and count the number of heads" then the p-value of HHHHHT with respect to $H_{0.5}$ is 0.11, whereas if you say your experiment and summary statistic are "flip the coin until it comes up tails and count the number of heads" then the p-value of HHHHHT with respect to $H_{0.5}$ is 0.03, which is "significant." This is called "[https://en.wikipedia.org/wiki/Data_dredging p-hacking]", and it's a serious problem in modern science.

In a likelihood function, _the amount of support an evidence gives to a hypothesis does not depend on which experiment the researcher had in mind._ Likelihood functions depend only on the data you actually saw, and the hypotheses you chose to report. The only way to cheat a likelihood function is to lie about the data you collected, or refuse to report likelihoods for a particular hypothesis.

If your paper fails to report likelihoods for some obvious hypotheses, then (a) that's precisely analogous to you choosing the wrong null hypothesis to consider; (b) it's just as easily noticeable as when your paper considers the wrong null hypothesis; and (c) it can be easily rectified given access to the raw data. By contrast, p-hacking can be subtle and hard to detect after the fact.

__3. Likelihood functions are very difficult to game.__ There is no analog of p-hacking for likelihood functions. This is a theorem of probability theory known as [-conservation_of_expected_evidence], which says that likelihood functions can't be gamed unless you're falsifying or omitting data (or screwing up the likelihood calculations).%note:Disclaimer: the theorem says likelihood functions can't be gamed, but we still shouldn't underestimate the guile of dishonest researchers struggling to make their results look important. Likelihood functions have not been put through the gauntlet of real scientific practice; p-values have. That said, when p-values were put through that gauntlet, they [https://en.wikipedia.org/wiki/Data_dredging failed] in a [https://en.wikipedia.org/wiki/Replication_crisis spectacular fashion]. When rebuilding, it's probably better to start from foundations that provably cannot be gamed.%

__4. Likelihood functions would help stop the "vanishing effect sizes" phenomenon.__ The [https://en.wikipedia.org/wiki/Decline_effect decline effect] occurs when studies which reject a null hypothesis $H_0$ have effect sizes that get smaller and smaller and smaller over time (the more someone tries to replicate the result). This is [http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ usually evidence] that there is no actual effect, and that the initial "large effects" were a result of publication bias.

Likelihood functions help avoid the decline effect by _treating different effect sizes differently._ The likelihood function for coins of different biases shows that the evidence HHHHHT gives a different amount of support to $H_{0.52},$ $H_{0.61}$, and $H_{0.8}$ (which correspond to small, medium, and large effect sizes, respectively). If three different studies find low support for $H_{0.5},$ and one of them gives all of its support to the large effect, another gives all its support to the medium effect, and the third gives all of its support to the smallest effect, then likelihood functions reveal that something fishy is going on (because they're all peaked in different places).

If instead we only use p-values, and always decide whether or not to "keep" or "reject" the null hypothesis (without specifying how much support goes to different alternatives), then it's hard to notice that the studies are actually contradictory (and that something very fishy is going on). Instead, it's very tempting to exclaim "3 out of 3 studies reject $H_{0.5}$!" and move on.

__5. Likelihood functions would help stop publication bias.__ When using p-values, if the data yields a p-value of 0.11 using a null hypothesis $H_0$, the study is considered "insignificant," and many journals have a strong bias towards positive results. When reporting likelihood functions, there is no arbitrary "significance" threshold. A study that reports a relative likelihoods of $21 : 1$ in favor of $H_a$ vs $H_0,$ that's exactly the same _strength of evidence_ as a study that reports $21 : 1$ odds against $H_a$ vs $H_0.$ It's all just evidence, and it can all be added to the corpus, there's no arbitrary "significance" threshold.

__6. Likelihood functions make it trivially easy to combine studies.__ When combining studies that used p-values, researchers have to perform complex meta-analyses with dozens of parameters to tune, and they often find [https://en.wikipedia.org/wiki/Confirmation_bias exactly what they were expecting to find]. By contrast, the way you combine multiple studies that reported likelihood functions is... (drumroll)
...you just multiply the likelihood functions together. If study A reports that $H_{0.75}$ was favored over $H_{0.5}$ with a [-1rq] of $3.8 : 1$, and study B reports that $H_{0.75}$ was favored over $H_{0.5}$ at $5 : 1$, then the combined likelihood functions of both studies favors $H_{0.75}$ over $H_{0.5}$ at $(3.8 \cdot 5) : 1$ $=$ $19 : 1.$

Want to combine a hundred studies on the same subject? Multiply a hundred functions together. Done. No parameter tuning, no degrees of freedom through which bias can be introduced — just multiply.

__7. Likelihood functions make it obvious when something has gone wrong.__ If, when you multiply all the likelihood functions together, _all_ hypotheses have extraordinarily low likelihoods, then something has gone wrong. Either a mistake has been made somewhere, or fraud has been committed, or the true hypothesis wasn't in the hypothesis class you're considering.

The actual hypothesis that explains all the data will have decently high likelihood across all the data. If none of the hypotheses fit that description, then either you aren't considering the right hypothesis yet, or some of the studies went wrong. (Try looking for one study that has a likelihood function _very very different_ from all the other studies, and investigate that one.)

Likelihood functions won't do your science for you — you still have to generate good hypotheses, and be honest in your data reporting — but they _do_ make it obvious when something went wrong. (Specifically, [227 each hypothesis can tell you how low its likelihood is expected to be on the data], and if _every_ hypothesis has a likelihood far lower than expected, then something's fishy.)

---

A scientific community using likelihood functions would produce scientific research that's easier to use. If everyone's reporting likelihood functions, then all you personally need to do in order to figure out what to believe is take your own personal (subjective) prior probabilities and multiply them by all the likelihood functions in order to get your own personal (subjective) posterior probabilities.

For example, let's say you personally think the coin is probably fair, with $10 : 1$ odds of being fair as opposed to 75% biased in favor of heads. Now let's say that study A reports a likelihood function which favors $H_{0.75}$ over $H_{0.5}$ with a likelihood ratio of $3.8 : 1.$, and study B reports a $5 : 1$ likelihood ratio in the same direction. Multiplying all these together, your personal posterior beliefs should be $19 : 10$ in favor of $H_{0.75}$ over $H_{0.5}$. This is simply [1lz Bayes' rule]. Reporting likelihoods instead of p-values lets science remain objective, while allowing everyone to find their own personal posterior probabilities via a simple application of Bayes' theorem.

## Why should we think this would work?

This may all sound too good to be true. Can one simple change really solve that many problems in modern science?

First of all, you can be assured that reporting likelihoods instead of p-values would not "solve" all the problems above, and it would surely not solve all problems with modern experimental science. Open access to raw data, preregistration of studies, a culture that rewards replication, and many other ideas are also crucial ingredients to a scientific community that zeroes in on truth.

However, reporting likelihoods would _help_ solve lots of different problems in modern experimental science. This may come as a surprise. Aren't likelihood functions just one more statistical technique, just another tool for the toolbox? Why should we think that one single tool can solve that many problems?

The reason lies in [-1bv]. According to the axioms of probability theory, there is only one good way to account for evidence when updating your beliefs, and that way is via likelihood functions. Any other method is subject to inconsistencies and pathologies, as per the [probability_coherence_theorems coherence theorems of probability theory].

If you're manipulating equations like $2 + 2 = 4,$ and you're using methods that may or may not let you throw in an extra 3 on the right hand side (depending on the arithmetician's state of mind), then it's no surprise that you'll occasionally get yourself into trouble and deduce that $2 + 2 = 7.$ The laws of arithmetic show that there is only one correct set of tools for manipulating equations if you want to avoid inconsistency.

Similarly, the laws of probability theory show that there is only one correct set of tools for manipulating _uncertainty_ if you want to avoid inconsistency. According to [1lz those rules], the right way to represent evidence is through likelihood functions.

These laws (and a solid understanding of them) are younger than the experimental science community, and the statistical tools of that community predate a modern understanding of probability theory. Thus, it makes a lot of sense that the existing literature uses different tools. However, now that humanity _does_ possess a solid understanding of probability theory, it should come as no surprise that many diverse pathologies in statistics can be cleaned up by switching to a policy of reporting likelihoods instead of p-values.

## What are the drawbacks?

The main drawback is inertia. Experimental science today reports p-values almost entirely across the board. Modern statistical toolsets have built-in support for p-values (and other related statistical tools) but very little support for reporting likelihood functions. Experimental scientists are trained mainly in [-frequentist_statistics], and thus most are much more familiar with p-value-type tools than likelihood-function-type tools. Making the switch would be painful.

Barring the switching costs, though, making the switch could well be a strict improvement over modern techniques, and would help solve some of the biggest [http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ problems] [http://www.stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf facing] [https://en.wikipedia.org/wiki/Data_dredging science] [https://en.wikipedia.org/wiki/Publication_bias today].

See also the [505 Likelihoods not p-values FAQ] and [4xx].


Report likelihoods not p-values: FAQ

This page answers frequently asked questions about the [4zd] proposal for experimental science.

_(Note: This page is a personal [60p opinion page].)_

[toc:]

### What does this proposal entail?

Let's say you have a coin and you don't know whether it's biased or not. You flip it six times and it comes up HHHHHT.

To report a [p_value p-value], you have to first declare which experiment you were doing — were you flipping it six times no matter what you saw and counting the number of heads, or were you flipping it until it came up tails and seeing how long it took? Then you have to declare a "null hypothesis," such as "the coin is fair." Only then can you get a p-value, which in this case, is either 0.11 (if you were going to toss the coin 6 times regardless) or 0.03 (if you were going to toss until it came up heads). The p-value of 0.11 means "if the null hypothesis _were_ true, then data that has as many H values as the observed data would only occur 11% of the time, if the declared experiment were repeated many many times."

To report a [bayes_likelihood likelihood], you don't have to do any of that "declare your experiment" stuff, and you don't have to single out one special hypothesis. You just pick a whole bunch of hypotheses that seem plausible, such as the set of hypotheses $H_b$ = "the coin has a bias of $b$ towards heads" for $b$ between 0% and 100%. Then you look at the actual data, and report how likely that data is according to each hypothesis. In this example, that yields a graph which looks like this:

![L(e|H)](http://i.imgur.com/UwwxmCe.png)

This graph says that HHHHHT is about 1.56% likely under the hypothesis $H_{0.5}$ saying that the coin is fair, and about 5.93% likely under the hypothesis $H_{0.75}$ that the coin comes up heads 75% of the time, and only 0.17% likely under the hypothesis $H_{0.3}$ that the coin only comes up tails 30% of the time.

That's all you have to do. You don't need to make any arbitrary choice about which experiment you were going to run. You don't need to ask yourself what you "would have seen" in other cases. You just look at the actual data, and report how likely each hypothesis in your hypothesis class said that data should be.

(If you want to compare how well the evidence supports one hypothesis or another, you just use the graph to get a [1rq likelihood ratio] between any two hypotheses. For example, this graph reports that the data HHHHHT supports the hypothesis $H_{0.75}$ over $H_{0.5}$ at odds of $\frac{0.0593}{0.0156}$ = 3.8 to 1.)

For more of an explanation, see [4zd].

### Why would reporting likelihoods be a good idea?

Experimental scientists reporting likelihoods instead of p-values would likely help address many problems facing modern science, including [https://en.wikipedia.org/wiki/Data_dredging p-hacking], [http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ the vanishing effect size problem], and [https://en.wikipedia.org/wiki/Publication_bias publication bias].

It would also make it easier for scientists to combine the results from multiple studies, and it would make it much much easier to conduct meta-analyses.

It would also make scientific statistics more intuitive, and easier to understand.

### Likelihood functions are a Bayesian tool. Aren't Bayesian statistics subjective? Shouldn't science be objective?

Likelihood functions are purely objective. In fact, there's only one degree of freedom in a likelihood function, and that's the choice of hypothesis class. This choice is no more arbitrary than the choice of a "null hypothesis" in standard statistics, and indeed, it's significantly less arbitrary (you can pick a large class of hypotheses, rather than just one; and none of them needs to be singled out as subjectively "special").

This is in stark contrast with p-values, which require that you pick an "experimental design" in advance, or that you talk about what data you "could have seen" if the experiment turned out differently. Likelihood functions only depend on the hypothesis class that you're considering, and the data that you actually saw. (This is one of the reasons why likelihood functions would solve p-hacking.)

Likelihood functions are often used by Bayesian statisticians, and Bayesian statisticians do indeed use [4vr subjective probabilities], which has led some people to believe that reporting likelihood functions would somehow allow hated subjectivity to seep into the hallowed halls of science.

However, it's the [1rm priors] that are subjective in Bayesian statistics, not likelihood functions. In fact, according to the [1lz laws of probability theory], likelihood functions are precisely that-which-is-left-over when you factor out all subjective beliefs from an observation of evidence. In other words, probability theory tells us that likelihoods are the best summary there is for capturing the objective evidence that a piece of data provides (assuming your goal is to help make people's beliefs more accurate).

### How would reporting likelihoods solve p-hacking?

P-values depend on what experiment the experimenter says they had in mind. For example, if the data is HHHHHT and the experimenter says "I was planning to flip it six times and count the number of Hs" then the p-value (for the fair coin hypothesis) is 0.11, which is not "significant." If instead the experimenter says "I was planing to flip it until I got a T" then the p-value is 0.03, which _is_ "significant." Experimenters can ([http://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108 and do!]) misuse or abuse this degree of freedom to make their results appear more significant than they actually are. This is known as "p-hacking."

In fact, when running complicated experiments, this can (and does!) happen to honest well-meaning researchers. Some experimenters are dishonest, and many others simply lack the time and patience to understand the subtleties of good experimental design. We don't need to put that burden on experimenters. We don't need to use statistical tools that depend on which experiment the experimenter had in mind. We can instead report the likelihood that each hypothesis assigned to the actual data.

Likelihood functions don't have this "experiment" degree of freedom. They don't care what experiment you thought you were doing. They only care about the data you actually saw. To use likelihood functions correctly, all you have to do is look at stuff and then not lie about what you saw. Given the set of hypotheses you want to report likelihoods for, the likelihood function is completely determined by the data.

### But what if the experimenter tries to game the rules by choosing how much data to collect?

That's a problem if you're reporting p-values, but it's not a problem if you're reporting likelihood functions.

Let's say there's a coin that you think is fair, that I think might be biased 55% towards heads. If you're right, then every toss is going to (in [-4b5]) provide more evidence for "fair" than "biased." But sometimes (rarely), even if the coin is fair, you will flip it and it will generate a sequence that supports the "bias" hypothesis more than the "fair" hypothesis.

How often will this happen? It depends on how exactly you ask the question. If you can flip the coin at most 300 times, then there's about a 1.4% chance that at some point the sequence generated will support the hypothesis "the coin is biased 55% towards heads" 20x more than it supports the hypothesis "the coin is fair." (You can verify this yourself, and tweak the parameters, using [https://gist.github.com/Soares/941bdb13233fd0838f1882d148c9ac14 this code].)

This is an objective fact about coin tosses. If you look at a sequence of Hs and Ts generated by a fair coin, then some tiny fraction of the time, after some number $n$ of flips, it will support the "biased 55% towards heads" hypothesis 20x more than it supports the "fair" hypothesis. This is true no matter how or why you decided to look at those $n$ coin flips. It's true if you were always planning to look at $n$ coin flips since the day you were born. It's true if each coin flip costs $1 to look at, so you decided to only look until the evidence supported one hypothesis at least 20x better than the other. It's true if you have a heavy personal desire to see the coin come up biased, and were planning to keep flipping until the evidence supports "bias" 20x more than it supports "fair". It doesn't _matter_ why you looked at the sequence of Hs and Ts. The amount by which it supports "biased" vs "fair" is objective. If the coin really is fair, then the more you flip it the more the evidence will push towards "fair." It will only support "bias" a small unlucky fraction of the time, and that fraction is completely independent from your thoughts and intentions .

Likelihoods are objective. They don't depend on your state of mind.

P-values, on the other hand, run into some difficulties. A p-value is about a single hypothesis (such as "fair") in isolation. If the coin is fair, then [fair_coin_equally_likely all sequences of coin tosses are equally likely], so you need something more than the data in order to decide whether the data is "significant evidence" about fairness one way or the other. Which means you have to choose a "reference class" of ways the coin "could have come up." Which means you need to tell us which experiment you "intended" to run. And down the rabbit hole we go.

The p-value you report depends on how many coin tosses you say you were going to look at. If you lie about where you intended to stop, the p-value breaks. If you're out in the field collecting data, and the data just subconsciously begins to feel overwhelming, and so you stop collecting evidence (or if the data just subconsciously feels insufficient and so you collect more) then the p-value breaks. How badly to p-values break? If you can toss the coin at most 300 times, then by choosing when to stop looking, you can get a p < 0.05 significant result _21% of the time,_ and that's assuming you are required to look at at least 30 flips. If you're allowed to use small sample sizes, the number is more like 25%. You can verify this yourself, and tweak the parameters, using [https://gist.github.com/Soares/4955bb9268129476262b28e32b8ec979 this code].

It's no wonder that p-values are so often misused! To use p-values correctly, an experimenter has to meticulously report their intentions about the experimental design before collecting data, and then has to hold utterly unfaltering to that experiment design as the data comes in (even if it becomes clear that their experimental design was naive, and that there were crucial considerations that they failed to take into account). Using p-values correctly requires good intentions, constant vigilance, and inflexibility.

Contrast this with likelihood functions. Likelihood functions don't depend on your intentions. If you start collecting data until it looks overwhelming and then stop, that's great. If you start collecting data and it looks underwhelming so you keep collecting more, that's great too. Every new piece of data you do collect will support the true hypothesis more than any other hypothesis, in expectation — that's the whole point of collecting data. Likelihood functions don't depend upon your state of mind.

### What if the experimenter uses some other technique to bias the result?

They can't. Or, at least, it's a theorem of [-1bv] that they can't. This law is known as [-conservation_expected_evidence conservation of expected evidence], and it says that for any hypothesis $H$ and any piece of evidence $e$, $\mathbb P(H) = \mathbb P(H \mid e) \mathbb P(e) + \mathbb P(H \mid \lnot e) \mathbb P(\lnot e),$ where $\mathbb P$ stands for my personal subjective probabilities.

Imagine that I'm going to take your likelihood function $\mathcal L$ and blindly combine it with my personal beliefs using [1lz Bayes' rule]. The question is, can you use $\mathcal L$ to manipulate my beliefs? The answer is clearly "yes" if you're willing to lie about what data you saw. But what if you're honestly reporting all the data you _actually_ saw? Then can you manipulate my beliefs, perhaps by being strategic about what data you look at and how long you look at it?

Clearly, the answer to that question is "sort of." If you have a fair coin, and you want to convince me it's biased, and you toss it 10 times, and it (by sheer luck) comes up HHHHHHHHHH, then that's a lot of evidence in favor of it being biased. But you can't use the "hope the coin comes up heads 10 times in a row by sheer luck" strategy to _reliably_ bias my beliefs; and if you try just flipping the coin 10 times and hoping to get lucky, then on average, you're going to produce data that convinces me that the coin is fair. The real question is, can you bias my beliefs _in expectation?_

If the answer is "yes," then there will be times when I should ignore $\mathcal L$ even if you honestly reported what you saw. If the answer is "no," then there will be no such times — for every $e$ that would shift my beliefs heavily towards $H$ (such that you could say "Aha! How naive! If you look at this data and see it is $e$, then you will believe $H$, just as I intended"), there is an equal and opposite chance of alternative data which would push my beliefs _away_ from $H.$ So, can you set up a data collection mechanism that pushes me towards $H$ in expectation?

And the answer to that question is _no,_ and this is a trivial theorem of probability theory. No matter what subjective belief state $\mathbb P$ I use, if you honestly report the objective likelihood $\mathcal L$ of the data you actually saw, and I update $\mathbb P$ by [1lz multiplying it by $\mathcal L$], there is no way (according to $\mathbb P$) for you to bias my probability of $H$ on average — no matter how strategically you decide which data to look at or how long to look. For more on this theorem and its implications, see [conservation_expected_evidence Conservation of Expected Evidence].

There's a difference between metrics that can't be exploited in theory and metrics that can't be exploited in practice, and if a malicious experimenter really wanted to abuse likelihood functions, they could probably find some clever method. (At the least, they can always lie and make things up.) However, p-values aren't even provably inexploitable — they're so easy to exploit that sometimes well-meaning honest researchers exploit them _by accident_, and these exploits are already commonplace and harmful. When building better metrics, starting with metrics that are provably inexploitable is a good start.

### What if you pick the wrong hypothesis class?

If you don't report likelihoods for the hypotheses that someone cares about, then that person won't find your likelihood function very helpful. The same problem exists when you report p-values (what if you pick the wrong null and alternative hypotheses?). Likelihood functions make the problem a little better, by making it easy to report how well the data supports a wide variety of hypotheses (instead of just ~2), but at the end of the day, there's no substitute for the raw data.

Likelihoods are a summary of the data you saw. They're a useful summary, especially if you report likelihoods for a broad set of plausible hypotheses. They're a much better summary than many other alternatives, such as p-values. But they're still a summary, and there's just no substitute for the raw data.

### How does reporting likelihoods help prevent publication bias?

When you're reporting p-values, there's a stark difference between p-values that favor the null hypothesis (which are deemed "insignificant") and p-values that reject the null hypothesis (which are deemed "significant"). This "significance" occurs at arbitrary thresholds (e.g. p < 0.05), and significance is counted only in one direction (to be significant, you must reject the null hypothesis). Both these features contribute to publication bias: Journals only want to accept experiments that claim "significance" and reject the null hypothesis.

When you're reporting [56s likelihood functions], a 20 : 1 [1rq ratio] is a 20 : 1 ratio is a 20 : 1 ratio. It doesn't matter if your likelihood function is peaked near "the coin is fair" or whether it's peaked near "the coin is biased 82% towards heads." If the ratio between the likelihood of one hypothesis and the likelihood of another hypothesis is 20 : 1 then the data provides the same strength of evidence either way. Likelihood functions don't single out one "null" hypothesis and incentivize people to only report data that pushes away from that null hypothesis; they just talk about the relationship between the data and _all_ the interesting hypotheses.

Furthermore, there's no arbitrary significance threshold for likelihood functions. If you didn't have a ton of data, your likelihood function will be pretty spread out, but it won't be useless. If you find $5 : 1$ odds in favor of $H_1$ over $H_2$, and I independently find $6 : 1$ odds in favor of $H_1$ over $H_2$, and our friend independently finds $3 : 1$ odds in favor of $H_1$ over $H_2,$ then our studies as a whole constitute evidence that favors $H_1$ over $H_2$ by a factor of $90 : 1$ — hardly insignificant! With likelihood ratios (and no arbitrary "significance" cutoffs) progress can be made in small steps.

Of course, this wouldn't solve the problem of publication bias in full, not by a long shot. There would still be incentives to report cool and interesting results, and the scientific community might still ask for results to pass some sort of "significance" threshold before accepting them for publication. However, reporting likelihoods would be a good start.

### How does reporting likelihoods help address vanishing effect sizes?

In a field where an effect does not actually exist, we will often observe an initial study that finds a very large effect, followed by a number of attempts at replication that find smaller and smaller and smaller effects (until someone postulates that the effect doesn't exist, and does a meta-analysis to look for p-hacking and publication bias). This is known as the [https://en.wikipedia.org/wiki/Decline_effect decline effect]; see also [http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ _The control group is out of control_].

The decline effect is possible in part because p-values look only at whether the evidence says we should "accept" or "reject" a special null hypothesis, without any consideration for what that evidence says about the alternative hypotheses. Let's say we have three studies, all of which reject the null hypothesis "the coin is fair." The first study rejects the null hypothesis with a 95% confidence interval of [0.7, 0.9] bias in favor of heads, but it was a small study and some of the experimenters were a bit sloppy. The second study is a bit bigger and a bit better organized, and rejects the null hypothesis with a 95% confidence interval of [0.53, 0.62]. The third study is high-powered, long-running, and rejects the null hypothesis with a 95% confidence interval of [0.503, 0.511]. It's easy to say "look, three separate studies rejected the null hypothesis!"

But if you look at the likelihood functions, you'll see that _something very fishy is going on_ — none of the studies actually agree with each other! The effect sizes are incompatible. Likelihood functions make this phenomenon easy to detect, because they tell you how much the data supports _all_ the relevant hypotheses (not just the null hypothesis). If you combine the three likelihood functions, you'll see that _none_ of the confidence intervals fare very well. Likelihood functions make it obvious when different studies contradict each other directly, which makes it much harder to summarize contradictory data down to "three studies rejected the null hypothesis".

### What if I want to reject the null hypothesis without needing to have any particular alternative in mind?

Maybe you don't want to report likelihoods for a large hypothesis class, because you are pretty sure you can't generate a hypothesis class that contains the correct hypothesis. "I don't want to have to make up a bunch of alternatives," you protest, "I just want to show that the null hypothesis is _wrong,_ in isolation."

Fortunately for you, that's possible using likelihood functions! The tool you're looking for is the notion of [227 strict confusion]. A hypothesis $H$ will tell you how low its likelihood is supposed to get, and if its likelihood goes a lot lower than that value, then you can be pretty confident that you've got the wrong hypothesis.

For example, let's say that your one and only hypothesis is $H_{0.9}$ = "the coin is biased 90% towards heads." Now let's say you flip the coin twenty times, and you see the sequence THTTHTTTHTTTTHTTTTTH. The [log_likelihood log-likelihood] that $H_{0.9}$ _expected_ to get on a sequence of 20 coin tosses was about -9.37 [evidence_bit bits],%%note: According to $H_{0.9},$ each coin toss carries $0.9 \log_2(0.9) + 0.1 \log_2(0.1) \approx -0.469$ bits of evidence, so after 20 coin tosses, $H_{0.9}$ expects about $20 \cdot 0.469 \approx 9.37$ bits of [bayes_surprise surprise]. For more on why log likelihood is a convenient tool for measuring "evidence" and "surprise," see [1zh Bayes' rule: log odds form].%% for a likelihood score of about $2^{-9.37} \approx$ $1.5 \cdot 10^{-3},$ on average. The likelihood that $H_{0.9}$ actually gets on that sequence is -50.59 bits, for a likelihood score of about $5.9 \cdot 10^{-16},$ which is _thirteen orders of magnitude less likely than expected._ You don't need to be clever enough to come up with an alternative hypothesis that explains the data in order to know that $H_{0.9}$ is not the right hypothesis for you.

In fact, likelihood functions make it easy to show that _lots_ of different hypotheses are strictly confused — you don't need to have a good hypothesis in your hypothesis class in order for reporting likelihood functions to be a useful service.

### How does reporting likelihoods make it easier to combine multiple studies?

Want to combine two studies that reported likelihood functions? Easy! Just multiply the likelihood functions together. If the first study reported 10 : 1 odds in favor of "fair coin" over "biased 55% towards heads," and the second study reported 12 : 1 odds in favor of "fair coin" over "biased 55% towards heads," then the combined studies support the "fair coin" hypothesis over the "biased 55% towards heads" hypothesis at a likelihood ratio of 120 : 1.

Is it really that easy? Yes! That's one of the benefits of using a representation of evidence supported by a large edifice of [-1bv] — they're trivially easy to compose. You have to ensure that the studies are independent first, because otherwise you'll double-count the data. (If the combined likelihood ratios get really extreme, you should be suspicious about whether they were actually independent.) This isn't exactly a new problem in experimental science; we can just add it to the list of reasons why replication studies had better be independent of the original study. Also, you can only multiply the likelihood functions together on places where they're both defined: If one study doesn't report the likelihood for a hypothesis that you care about, you might need access to the raw data in order to extend their likelihood function. But if the studies are independent and both report likelihood functions for the relevant hypotheses, then all you need to do is multiply.

(Don't try this with p-values. A p < 0.05 study and a p < 0.01 study don't combine into anything remotely like a p < 0.0005 study.)

### How does reporting likelihoods make it easier to conduct meta-analyses?

When studies report p-values, performing a meta-analysis is a complicated procedure that requires dozens of parameters to be finely tuned, and (lo and behold) bias somehow seeps in, and meta-analyses often find whatever the analyzer set out to find. When studies report likelihood functions, performing a meta-analysis is trivial and doesn't depend on you to tune a dozen parameters. Just multiply all the likelihood functions together.

If you want to be extra virtuous, you can check for anomalies, such as one likelihood function that's tightly peaked in a place that disagrees with all the other peaks. You can also check for [227 strict confusion], to get a sense for how likely it is that the correct hypothesis is contained within the hypothesis class that you considered. But mostly, all you've got to do is multiply the likelihood functions together.

### How does reporting likelihood functions make it easier to detect fishy studies?

With likelihood functions, it's much easier to find the studies that don't match up with each other — look for the likelihood function that has its peak in a different place than all the other peaks. That study deserves scrutiny: either those experimenters had something special going on in the background of their experiment, or something strange happened in their data collection and reporting process.

Furthermore, likelihoods combined with the notion of [227 strict confusion] make it easy to notice when something has gone seriously wrong. As per the above answers, you can combine multiple studies by multiplying their likelihood functions together. What happens if the likelihood function is super small everywhere? That means that either (a) some of the data is fishy, or (b) you haven't considered the right hypothesis yet.

When you _have_ considered the right hypothesis, it will have decently high likelihood under _all_ the data. There's only one real world underlying all our data, after all — it's not like different experimenters are measuring different underlying universes. If you multiply all the likelihood functions together and _all_ the hypotheses turn out looking wildly unlikely, then you've got some work to do — you haven't yet considered the right hypothesis.

When reporting p-values, contradictory studies feel like the norm. Nobody even _tries_ to make all the studies fit together, as if they were all measuring the same world. With likelihood functions, we could actually aspire towards a world where scientific studies on the same topic are _all_ combined. A world where people try to find hypotheses that fit _all_ the data at once, and where a single study's data being out of place (and making all the hypotheses currently under consideration become [-227]) is a big glaring "look over here!" signal. A world where it feels like studies are _supposed_ to fit together, where if scientists haven't been able to find a hypothesis that explains all the raw data, then they know they have their work cut out for them.

Whatever the right hypothesis is, it will almost surely not be strictly confused under the actual data. Of course, when you come up with a completely new hypothesis (such as "the coin most of us have been using is fair but study #317 accidentally used a different coin") you're going to need access to the raw data of some of the previous studies in order to extend their likelihood functions and see how well they do on this new hypothesis. As always, there's just no substitute for raw data.

### Why would this make statistics easier to do and understand?

p < 0.05 does not mean "the null hypothesis is less than 5% likely" (though that's what young students of statistics often _want_ it to mean). What the null hypothesis means is "given a particular experimental design (e.g., toss the coin 100 times and count the heads) and the data (e.g., the sequence of 100 coin flips), if the null hypothesis _were_ true, then data that matches my chosen statistic (e.g., the number of heads) would only occur 5% of the time, if we repeated this experiment over and over and over."

Why the complexity? Statistics is designed to keep subjective beliefs out of the hallowed halls of science. Your science paper shouldn't be able to conclude "and, therefore, I personally believe that the coin is very likely to be biased, and I'd bet on that at 20 : 1 odds." Still, much of this complexity is unnecessary. Likelihood functions achieve the same goal of objectivity, but without all the complexity.

[51n $\mathcal L_e(H)$] $< 0.05$ _also_ doesn't mean "$H$ is less than 5% likely", it means "H assigned less than 0.05 probability to $e$ happening." The student still needs to learn to keep "probability of $e$ given $H$" and "probability of $H$ given $e$" distinctly separate in their heads. However, likelihood functions do have a _simpler_ interpretation: $\mathcal L_e(H)$ is the probability of the actual data $e$ occurring if $H$ were in fact true. No need to talk about experimental design, no need to choose a summary statistic, no need to talk about what "would have happened." Just look at how much probability each hypothesis assigned to the actual data; that's your likelihood function.

If you're going to report p-values, you need to be meticulous in considering the complexities and subtleties of experiment design, on pain of creating p-values that are broken in non-obvious ways (thereby contributing to the [https://en.wikipedia.org/wiki/Replication_crisis replication crisis]). When reading results, you need to take the experimenter's intentions into account. None of this is necessary with likelihoods.

To understand $\mathcal L_e(H),$ all you need to know is how likely $e$ was according to $H.$ Done.

### Isn't this just one additional possible tool in the toolbox? Why switch entirely away from p-values?

This may all sound too good to be true. Can one simple change really solve that many problems in modern science?

First of all, you can be assured that reporting likelihoods instead of p-values would not "solve" all the problems above, and it would surely not solve all problems with modern experimental science. Open access to raw data, preregistration of studies, a culture that rewards replication, and many other ideas are also crucial ingredients to a scientific community that zeroes in on truth.

However, reporting likelihoods would help solve lots of different problems in modern experimental science. This may come as a surprise. Aren't likelihood functions just one more statistical technique, just another tool for the toolbox? Why should we think that one single tool can solve that many problems?

The reason lies in [-1bv]. According to the axioms of probability theory, there is only one good way to account for evidence when updating your beliefs, and that way is via likelihood functions. Any other method is subject to inconsistencies and pathologies, as per the [probability_coherence_theorems coherence theorems of probability theory].

If you're manipulating equations like $2 + 2 = 4,$ and you're using methods that may or may not let you throw in an extra 3 on the right hand side (depending on the arithmetician's state of mind), then it's no surprise that you'll occasionally get yourself into trouble and deduce that $2 + 2 = 7.$ The laws of arithmetic show that there is only one correct set of tools for manipulating equations if you want to avoid inconsistency.

Similarly, the laws of probability theory show that there is only one correct set of tools for manipulating _uncertainty_ if you want to avoid inconsistency. According to [1lz those rules], the right way to represent evidence is through likelihood functions.

These laws (and a solid understanding of them) are younger than the experimental science community, and the statistical tools of that community predate a modern understanding of probability theory. Thus, it makes a lot of sense that the existing literature uses different tools. However, now that humanity _does_ possess a solid understanding of probability theory, it should come as no surprise that many diverse pathologies in statistics can be cleaned up by switching to a policy of reporting likelihoods instead of p-values.

### If it's so great why aren't we doing it already?

[1bv Probability theory] (and a solid understanding of all that it implies) is younger than the experimental science community, and the statistical tools of that community predate a modern understanding of probability theory. In particular, modern statistical tools were designed in an attempt to keep subjective reasoning out of the hallowed halls of science. You shouldn't be able to publish a scientific paper which concludes "and therefore, I personally believe that this coin is biased towards heads, and would bet on that at 20 : 1 odds." Those aren't the foundations upon which science can be built.

Likelihood functions are strongly associated with Bayesian statistics, and Bayesian statistical tools tend to manipulate subjective probabilities. Thus, it wasn't entirely clear how to use tools such as likelihood functions without letting subjectivity bleed into science.

Nowadays, we have a better understanding of how to separate out subjective probabilities from objective claims, and it's known that likelihood functions don't carry any subjective baggage with them. In fact, they carry _less_ subjective baggage than p-values do: A likelihood function depends only on the data that you _actually saw,_ whereas p-values depend on your experimental design and your intentions.

There are good historical reasons why the existing scientific community is using p-values, but now that humanity _does_ possess a solid theoretical understanding of probability theory (and how to factor subjective probabilities out from objective claims), it's no surprise that a wide array of diverse problems in modern statistics can be cleaned up by reporting likelihoods instead of p-values.

### Has this ever been tried?

No. Not yet. To our knowledge, most scientists haven't even considered this proposal — and for good reason! There are a lot of big fish to fry when it comes to addressing the [https://en.wikipedia.org/wiki/Replication_crisis replication crisis], [https://en.wikipedia.org/wiki/Data_dredging p-hacking], [http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ the problem of vanishing effect sizes], [https://en.wikipedia.org/wiki/Publication_bias publication bias], and other problems facing science today.  The scientific community at large is huge, decentralized, and has a lot of inertia. Most activists who are trying to shift it already have their hands full advocating for very important policies such as open access journals and pre-registration of trials. So it makes sense that nobody's advocating hard for reporting likelihoods instead of p-values — yet.

Nevertheless, there are good reasons to believe that reporting likelihoods instead of p-values would help solve many of the issues in modern experimental science.


Likelihood notation

The likelihood of a piece of evidence $e$ according to a hypothesis $H,$ known as "the likelihood of $e$ given $H$", is often written either $\mathcal L_e(H)$ or $\mathcal L(H \mid e).$ The latter notation is confusing, because then $\mathcal L(H \mid e) = \mathbb P(e \mid H).$ Many students of statistics find it hard enough to keep the difference between $\mathbb P(H \mid e)$ and $\mathbb P(e \mid H)$ straight in their heads if we _don't_ occasionally swap the order of the arguments when talking about similar functions, so on Arbital, we much prefer the notation $\mathcal L_e(H) = \mathbb P(e \mid H).$

[fixme: Make this a child of 'likelihood' when 'likelihood' exists.]


Blue oysters

You're collecting exotic oysters in Nantucket, and there are two different bays you could harvest oysters from. In both bays, 11% of the oysters contain valuable pearls and 89% are empty. In the first bay, 4% of the pearl-containing oysters are blue, and 8% of the non-pearl-containing oysters are blue. In the second bay, 13% of the pearl-containing oysters are blue, and 26% of the non-pearl-containing oysters are blue. You created a special device that helps you find blue oysters. Would you rather harvest blue oysters from the first bay or the second bay?

You're encouraged to try to solve this problem yourself, and to refrain from looking at the answer. The answer can be found in [1zh].


Example problem

Tag for pages that provide an example problem referenced by a number of other pages.

The summary of the page should contain the entire text of the problem (so that the problem can be referenced by greenlink). The body of the page should either contain the answer (e.g., hidden behind an 'Answer' button), or should contain a link to the page that contains the answer. The body of the page should not explain the solution to the problem, but may link to the page that does.


Shift towards the hypothesis of least surprise

The [1zh log-odds form of Bayes' rule] says that strength of belief and strength of evidence can both be measured in [3y2 bits]. These evidence-bits can also be used to measure a quantity called "Bayesian surprise", which yields [fixme: One final, if this is the last thing in the path]yet another intuition for understanding Bayes' rule.

Roughly speaking, we can measure how surprised a hypothesis $H_i$ was by the evidence $e$ by measuring how much probability it put on $e.$ If $H_i$ put 100% of its probability mass on $e$, then $e$ is completely unsurprising (to $H_i$). If $H_i$ put 0% of its probability mass on $e$, then $e$ is _as surprising as possible._ Any measure of $\mathbb P(e \mid H_i),$ the probability $H_i$ assigned to $e$, that obeys this property, is worthy of the label "surprise." Bayesian surprise is $-\!\log(\mathbb P(e \mid H_i)),$ which is a quantity that obeys these intuitive constraints and has some other interesting features.

Consider again the [-54v] problem. Consider the hypotheses $H$ and $\lnot H$, which say "the oyster will contain a pearl" and "no it won't", respectively. To keep the numbers easy, let's say we draw an oyster from a third bay, where $\frac{1}{8}$ of pearl-carrying oysters are blue and $\frac{1}{4}$ of empty oysters are blue.

Imagine what happens when the oyster is blue. $H$ predicted blueness with $\frac{1}{8}$ of its probability mass, while $\lnot H$ predicted blueness with $\frac{1}{4}$ of its probability mass. Thus, $\lnot H$ did better than $H,$ and goes up in probability. Previously, we've been combining both $\mathbb P(e \mid H)$ and $\mathbb P(e \mid \lnot H)$ into unified likelihood ratios, like $\left(\frac{1}{8} : \frac{1}{4}\right)$ $=$ $(1 : 2),$ which says that the 'blue' observation carries 1 bit of evidence $H.$ However, we can also take the logs first, and combine second.

Because $H$ assigned only an eighth of its probability mass to the 'blue' observation, and because [1y6 Bayesian update works by eliminating incorrect probability mass], we have to adjust our belief in $H$ by $\log_2\left(\frac{1}{8}\right) = -3$ bits away from $H.$ (Each negative bit means "throw away half of $H$'s probability mass," and we have to do that 3 times in order to remove the probability that $H$ failed to assign to $e$.)

Similarly, because $\lnot H$ assigned only a quarter of its probability mass to the 'blue' observation, we have to adjust our belief in $H$ by $\log_2\left(\frac{1}{4}\right) = -2$ bits away from $\lnot H.$

Thus, when the 'blue' observation comes in, we move our belief (measured in bits) 3 notches away from $H$ and then two notches back towards $H.$ On net, our belief shifts 1 notch away from $H$.

![hypotheses emitting surprise](https://i.imgur.com/ZXGB8x0.png)

_$H$ assigned 1/8th of its probability mass to blueness, so it emits $-\!\log_2\left(\frac{1}{8}\right)=3$ bits of surprise pushing away from $H$. $\lnot H$ assigned 1/4th of its probability mass to blueness, so it emits $-\!\log_2\left(\frac{1}{4}\right)=2$ bits of surprise pushing away from $\lnot H$ (and towards $H$). Thus, belief in $H$ moves 1 bit towards $\lnot H$, on net._

If instead $H$ predicted blue with probability 4% (penalty $\log_2(0.04) \approx -4.64$) and $\lnot H$ predicted blue with probability 8% (penalty $\log_2(0.08) \approx -3.64$), then we would have shifted a bit over 4.6 notches towards $\lnot H$ and a bit over 3.6 notches back towards $H,$ but we would have shifted the same number of notches _on net._ This is why it's only the _relative_ difference between the number of bits docked from $H$ and the number of bits docked from $\lnot H$ that matters.

In general, given an observation $e$ and a hypothesis $H,$ the number of bits we need to dock from our belief in $H$ is $\log_2(\mathbb P(e \mid H)),$ that is, the log of the probability that $H$ assigned to $e.$ This quantity is never positive, because the logarithm of $x$ for $0 \le x \le 1$ is in the range $[-\infty, 0]$. If we negate it, we get a non-negative quantity that relates $H$ to $e$, which is 0 when $H$ was certain that $e$ was going to happen, and which is infinite when $H$ was certain that $e$ wasn't going to happen, and which is measured in the same units as evidence and belief. Thus, this quantity is often called "surprise," and intuitively, it measures how surprised the hypothesis $H$ was by $e$ (in bits).

There is some correlation between Bayesian surprise and the times when a human would feel surprised (at seeing something that they thought was unlikely), but, of course, the human emotion is quite different. (A human can feel surprised for other reasons than "my hypotheses failed to predict the data," and humans are also great at ignoring evidence instead of feeling surprised.)

Given this definition of Bayesian surprise, we can view Bayes' rule as saying that _surprise repels belief._ When you make an observation $e,$ each hypothesis emits repulsive "surprise" signals, which shift your hypothesis. Referring again to the image above, when $H$ predicts the observation you made with $\frac{1}{8}$ of its probability mass, and $\lnot H$ predicts it with $\frac{1}{4}$ of its probability mass, we can imagine $H$ emitting a surprise signal with a strength of 3 bits away from $H$ and $\lnot H$ emitting a surprise signal with a strength of 2 bits away from $\lnot H$. Both those signals push the belief in $H$ in different directions, and it ends up 1 bit closer to $\lnot H$ (which emitted the weaker surprise signal).

In other words, whenever you find yourself feeling surprised by something you saw, think of the _least surprising explanation_ for that evidence — and then award that hypothesis a few bits of belief.


Bayes' rule: Definition

Bayes' rule is the mathematics of [1rf probability theory] governing how to update your beliefs in the light of new evidence.

[toc:]

## [1y9 Notation]

In much of what follows, we'll use the following [1y9 notation]:

- Let the hypotheses being considered be $H_1$ and $H_2$.
- Let the evidence observed be $e_0.$
- Let $\mathbb P(H_i)$ denote the [1rm prior probability] of $H_i$ before observing the evidence.
- Let the [1rj conditional probability] $\mathbb P(e_0\mid H_i)$ denote the [1rq likelihood] of observing evidence $e_0$ assuming $H_i$ to be true.
- Let the [1rj conditional probability] $\mathbb P(H_i\mid e_0)$ denote the [1rp posterior probability] of $H_i$ after observing $e_0.$

## [1x5 Odds]/[1zm proportional] form

Bayes' rule in the [1x5 odds form] or [1zm proportional form] states:

$$\dfrac{\mathbb P(H_1)}{\mathbb P(H_2)} \times \dfrac{\mathbb P(e_0\mid H_1)}{\mathbb P(e_0\mid H_2)} = \dfrac{\mathbb P(H_1\mid e_0)}{\mathbb P(H_2\mid e_0)}$$

In other words, the [1rm prior] [1rb odds] times the [1rq likelihood ratio] yield the [1rp posterior] odds.  [1rk Normalizing] these odds will then yield the posterior probabilities.

In [1zm other other words]:  If you initially think $h_i$ is $\alpha$ times as probable as $h_k$, and then see evidence that you're $\beta$ times as likely to see if $h_i$ is true as if $h_k$ is true, you should update to thinking that $h_i$ is $\alpha \cdot \beta$ times as probable as $h_k.$

Suppose that Professor Plum and Miss Scarlet are two suspects in a murder, and that we start out thinking that Professor Plum is twice as likely to have committed the murder as Miss Scarlet ([1rm prior] [1rb odds] of 2 : 1).  We then discover that the victim was poisoned.  We think that Professor Plum is around one-fourth as likely to use poison as Miss Scarlet ([1rq likelihood ratio] of 1 : 4).  Then after observing the victim was poisoned, we should think Plum is around half as likely to have committed the murder as Scarlet: $2 \times \dfrac{1}{4} = \dfrac{1}{2}.$  This reflects [1rp posterior] odds of 1 : 2, or a posterior probability of 1/3, that Professor Plum did the deed.

## [1xr Proof]

The [1xr proof of Bayes' rule] is by the definition of [1rj conditional probability] $\mathbb P(X\wedge Y) = \mathbb P(X\mid Y) \cdot \mathbb P(Y):$

$$
\dfrac{\mathbb P(H_i)}{\mathbb P(H_j)} \times  \dfrac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)}
= \dfrac{\mathbb P(e \wedge H_i)}{\mathbb P(e \wedge H_j)}
= \dfrac{\mathbb P(e \wedge H_i) / \mathbb P(e)}{\mathbb P(e \wedge H_j) / \mathbb P(e)}
= \dfrac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)}
$$

## [1zh Log odds form]

The [1zh log odds form of Bayes' rule] states:

$$\log \left ( \dfrac
   {\mathbb P(H_i)}
   {\mathbb P(H_j)}
\right )
+
\log \left ( \dfrac
   {\mathbb P(e\mid H_i)}
   {\mathbb P(e\mid H_j)}
\right )
 =
\log \left ( \dfrac
   {\mathbb P(H_i\mid e)}
   {\mathbb P(H_j\mid e)}
\right )
$$

E.g.:  "A study of Chinese blood donors found that roughly 1 in 100,000 of them had HIV (as determined by a very reliable gold-standard test). The non-gold-standard test used for initial screening had a sensitivity of 99.7% and a specificity of 99.8%, meaning that it was 500 times as likely to return positive for infected as non-infected patients."  Then our prior belief is -5 orders of magnitude against HIV, and if we then observe a positive test result, this is evidence of strength +2.7 orders of magnitude for HIV.  Our posterior belief is -2.3 orders of magnitude, or odds of less than 1 to a 100, against HIV.

In log odds form, the same [22x strength of evidence] (log [1rq likelihood ratio]) always [1zh moves us the same additive distance] along a line representing strength of belief (also in log odds).  If we measured distance in probabilities, then the same 2 : 1 likelihood ratio might move us a different distance along the probability line depending on whether we started with prior 10% probability or 50% probability.

## Visualizations

Graphical of visualizing Bayes' rule include [1wy frequency diagrams, the waterfall visualization], the [1zm spotlight visualization], the [1zh magnet visualization], and the [1xr Venn diagram for the proof].

## Examples

Examples of Bayes' rule may be found [1wt here].

## [1zg Multiple hypotheses and updates]

The [1x5 odds form of Bayes' rule] works for odds ratios between more than two hypotheses, and applying multiple pieces of evidence.  Suppose there's a bathtub full of coins.  1/2 of the coins are "fair" and have a 50% probability of producing heads on each coinflip; 1/3 of the coins produce 25% heads; and 1/6 produce 75% heads.  You pull out a coin at random, flip it 3 times, and get the result HTH.  You may legitimately calculate:

$$\begin{array}{rll}
(1/2 : 1/3 : 1/6) \cong & (3 : 2 : 1) & \\
\times & (2 : 1 : 3) & \\
\times & (2 : 3 : 1) & \\
\times & (2 : 1 : 3) & \\
= & (24 : 6 : 9) & \cong (8 : 2 : 3)
\end{array}$$

Since multiple pieces of evidence may not be [conditional_independence conditionally independent] from one another, it is important to be aware of the [naive_bayes_assumption Naive Bayes assumption] and whether you are making it.

## [554 Probability form]

As a formula for a single probability $\mathbb P(H_i\mid e),$ Bayes' rule states:

$$\mathbb P(H_i\mid e) = \dfrac{\mathbb P(e\mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)}$$

## [1zj Functional form]

In [1zj functional form], Bayes' rule states:

$$\mathbb P(\mathbf{H}\mid e) \propto \mathbb P(e\mid \mathbf{H}) \cdot \mathbb P(\mathbf{H}).$$

The posterior probability function over hypotheses given the evidence, is *proportional* to the likelihood function from the evidence to those hypotheses, times the prior probability function over those hypotheses.

Since posterior probabilities over [1rd mutually exclusive and exhaustive] possibilities must sum to $1,$ [1rk normalizing] the product of the likelihood function and prior probability function will yield the exact posterior probability function.


Bayes' rule: Probability form

[summary:  The formulation of [1lz] you are most likely to see in textbooks says:

$$\mathbb P(H_i\mid e) = \dfrac{\mathbb P(e\mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)}$$

This follows from the definition of [-1rj] which states that $\mathbb P(X \mid Y) = \frac{\mathbb P(X \wedge Y)}{\mathbb P (Y)}$, and the [law_marginal_probability law of marginal probability] which says that $\mathbb P(Y) = \sum_k \mathbb P(Y \wedge X_k)$.

We can think of the corresponding advice as saying, "Think of how much each hypothesis in $H$ contributed to our expectation of seeing the evidence $e$, including both the [56v likelihood] of seeing $e$ if $H_k$ is true, and the [1rm prior] probability of $H_k$.  The [1rp posterior] of $H_i$ after seeing $e,$ is the amount $H_i$ contributed to our expectation of seeing $e,$ within the total expectation of seeing $e$ contributed by every hypothesis in $H.$]

The formulation of [1lz] you are most likely to see in textbooks runs as follows:

$$\mathbb P(H_i\mid e) = \dfrac{\mathbb P(e\mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)}$$

Where:

- $H_i$ is the hypothesis we're interested in.
- $e$ is the piece of evidence we observed.
- $\sum_k (\text {expression containing } k)$ [summation_notation means] "Add up, for every $k$, the sum of all the (expressions containing $k$)."
- $\mathbf H$ is a set of [1rd mutually exclusive and exhaustive] hypotheses that include $H_i$ as one of the possibilities, and the expression $H_k$ inside the sum ranges over all the possible hypotheses in $\mathbf H$.

As a quick example, let's say there's a bathtub full of potentially biased coins.

- Coin type 1 is fair, 50% heads / 50% tails.  40% of the coins in the bathtub are type 1.
- Coin type 2 produces 70% heads.  35% of the coins are type 2.
- Coin type 3 produces 20% heads.  25% of the coins are type 3.

We want to know the [1rp posterior] probability that a randomly drawn coin is of type 2, after flipping the coin once and seeing it produce heads once.

Let $H_1, H_2, H_3$ stand for the hypotheses that the coin is of types 1, 2, and 3 respectively.  Then using [1rj conditional probability notation], we want to know the probability $\mathbb P(H_2 \mid heads).$

The probability form of Bayes' theorem says:

$$\mathbb P(H_2 \mid heads) = \frac{\mathbb P(heads \mid H_2) \cdot \mathbb P(H_2)}{\sum_k \mathbb P(heads \mid H_k) \cdot \mathbb P(H_k)}$$

Expanding the sum:

$$\mathbb P(H_2 \mid heads) = \frac{\mathbb P(heads \mid H_2) \cdot \mathbb P(H_2)}{[\mathbb P(heads \mid H_1) \cdot \mathbb P(H_1)] + [\mathbb P(heads \mid H_2) \cdot \mathbb P(H_2)] + [\mathbb P(heads \mid H_3) \cdot \mathbb P(H_3)]}$$

Computing the actual quantities:

$$\mathbb P(H_2 \mid heads) = \frac{0.70 \cdot 0.35 }{[0.50 \cdot 0.40] + [0.70 \cdot 0.35] + [0.20 \cdot 0.25]} = \frac{0.245}{0.20 + 0.245 + 0.05} = 0.\overline{49}$$

This calculation was big and messy.  Which is fine, because the probability form of Bayes' theorem is okay for directly grinding through the numbers, but not so good for doing things in your head.

# Meaning

We can think of the advice of Bayes' theorem as saying:

"Think of how much each hypothesis in $H$ contributed to our expectation of seeing the evidence $e$, including both the [56v likelihood] of seeing $e$ if $H_k$ is true, and the [1rm prior] probability of $H_k$.  The [1rp posterior] of $H_i$ after seeing $e,$ is the amount $H_i$ contributed to our expectation of seeing $e,$ within the total expectation of seeing $e$ contributed by every hypothesis in $H.$"

Or to say it at somewhat greater length:

Imagine each hypothesis $H_1,H_2,H_3\ldots$ as an expert who has to distribute the probability of their predictions among all possible pieces of evidence.  We can imagine this more concretely by visualizing "probability" as a lump of clay.

The total amount of clay is one kilogram (probability $1$).  Each expert $H_k$ has been allocated a fraction $\mathbb P(H_k)$ of that kilogram.  For example, if $\mathbb P(H_4)=\frac{1}{5}$ then expert 4 has been allocated 200 grams of clay.

We're playing a game with the experts to determine which one is the best predictor.

Each time we're about to make an observation $E,$ each expert has to divide up all their clay among the possible outcomes $e_1, e_2, \ldots.$

After we observe that $E = e_j,$ we take away all the clay that wasn't put onto $e_j.$  And then our new belief in all the experts is the relative amount of clay that each expert has left.

So to know how much we now believe in expert $H_4$ after observing $e_3,$ say, we need to know two things:  First, the amount of clay that $H_4$ put onto $e_3,$ and second, the total amount of clay that all experts (including $H_4$) put onto $e_3.$

In turn, to know *that,* we need to know how much clay $H_4$ started with, and what fraction of its clay $H_4$ put onto $e_3.$  And similarly, to compute the total clay on $e_3,$ we need to know how much clay each expert $H_k$ started with, and what fraction of their clay $H_k$ put onto $e_3.$

So Bayes' theorem here would say:

$$\mathbb P(H_4 \mid e_3) = \frac{\mathbb P(e_3 \mid H_4) \cdot \mathbb P(H_4)}{\sum_k \mathbb P(e_3 \mid H_k) \cdot \mathbb P(H_k)}$$

What are the incentives of this game of clay?

On each round, the experts who gain the most are the experts who put the most clay on the observed $e_j,$ so if you know for certain that $e_3$ is about to be observed, your incentive is to put all your clay on $e_3.$

But putting *literally all* your clay on $e_3$ is risky; if $e_5$ is observed instead, you lose all your clay and are out of the game. Once an expert's amount of clay goes all the way to zero, there's no way for them to recover over any number of future rounds. That hypothesis is done, dead, and removed from the game.  ("Falsification," some people call that.)  If you're not certain that $e_5$ is literally impossible, you'd be wiser to put at least a *little* clay on $e_5$ instead.  That is to say: if your mind puts some probability on $e_5,$ you'd better put some clay there too!

([bayes_score As it happens], if at the end of the game we score each expert by the logarithm of the amount of clay they have left, then each expert is incentivized to place clay exactly proportionally to their honest probability on each successive round.)

It's an important part of the game that we make the experts put down their clay in advance.  If we let the experts put down their clay afterwards, they might be tempted to cheat by putting down all their clay on whichever $e_j$ had actually been observed.  But since we make the experts put down their clay in advance, they have to *divide up* their clay among the possible outcomes: to give more clay to $e_3,$ that clay has to be taken away from some other outcome, like $e_5.$  To put a very high probability on $e_3$ and gain a lot of relative credibility if $e_3$ is observed, an expert has to stick their neck out and risk losing a lot of credibility if some other outcome like $e_5$ happens instead.  *If* we force the experts to make advance predictions, that is!

We can also derive from this game that the question "does evidence $e_3$ support hypothesis $H_4$?" depends on how well $H_4$ predicted $e_3$ _compared to the competition._ It's not enough for $H_4$ to predict $e_3$ well if every other hypothesis also predicted $e_3$ well--your amazing new theory of physics gets no points for predicting that the sky is blue.  $H_k$ only goes up in probability when it predicts $e_j$ better than the alternatives.  And that means we have to ask what the alternative hypotheses predicted, even if we think those hypotheses are false.

If you get in a car accident, and don't want to relinquish the hypothesis that you're a great driver, then you can find all sorts of reasons ("the road was slippery! my car freaked out!") why $\mathbb P(e \mid GoodDriver)$ is not too low. But $\mathbb P(e \mid BadDriver)$ is also part of the update equation, and the "bad driver" hypothesis *better* predicts the evidence. Thus, your first impulse, when deciding how to update your beliefs in the face of a car accident, should not be "But my preferred hypothesis allows for this evidence!" It should instead be "Points to the 'bad driver' hypothesis for predicting this evidence better than the alternatives!"  (And remember, you're allowed to [update_beliefs_incrementally increase $\mathbb P(BadDriver)$ a little bit], while still thinking that it's less than 50% probable.)

# Proof

The proof of Bayes' theorem follows from the definition of [-1rj]:

$$\mathbb P(X \mid Y) = \frac{\mathbb P(X \wedge Y)}{\mathbb P (Y)}$$

And from the [law_marginal_probability law of marginal probability]:

$$\mathbb P(Y) = \sum_k \mathbb P(Y \wedge X_k)$$

Therefore:

$$
\mathbb P(H_i \mid e) = \frac{\mathbb P(H_i \wedge e)}{\mathbb P (e)}  \tag{defn. conditional prob.}
$$

$$
\mathbb P(H_i \mid e) = \frac{\mathbb P(e \wedge H_i)}{\sum_k \mathbb P (e \wedge H_k)} \tag {law of marginal prob.}
$$

$$
\mathbb P(H_i \mid e) = \frac{\mathbb P(e \mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P (e \mid H_k) \cdot \mathbb P(H_k)} \tag {defn. conditional prob.}
$$

QED.


Odds form to probability form

The odds form of Bayes' rule works for any two hypotheses $H_i$ and $H_j,$ and looks like this:

$$\frac{\mathbb P(H_i \mid e)}{\mathbb P(H_j \mid e)} = \frac{\mathbb P(H_i)}{\mathbb P(H_j)} \times \frac{\mathbb P(e \mid H_i)}{\mathbb P(e \mid H_j)} \tag{1}.$$

The probabilistic form of Bayes' rule requires a hypothesis set $H_1,H_2,H_3,\ldots$ that is [-1rd], and looks like this:

$$\mathbb P(H_i\mid e) = \frac{\mathbb P(e\mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)} \tag{2}.$$

We will now show that equation (2) follows from equation (1). Given a collection $H_1,H_2,H_3,\ldots$ of mutually exclusive and exhaustive hypotheses and a hypothesis $H_i$ from that collection, we can form another hypothesis $\lnot H_i$ consisting of all the hypotheses $H_1,H_2,H_3,\ldots$ _except_ $H_i.$ Then, using $\lnot H_i$ as $H_j$ and multiplying the fractions on the right-hand side of equation (1), we see that

$$\frac{\mathbb P(H_i \mid e)}{\mathbb P(\lnot H_i \mid e)} = \frac{\mathbb P(H_i) \cdot \mathbb P(e \mid H_i)}{\mathbb P(\lnot H_i)\cdot \mathbb P(e \mid \lnot H_i)}.$$

$\mathbb P(\lnot H_i)\cdot \mathbb P(e \mid \lnot H_i)$ is the prior probability of $\lnot H_i$ times the degree to which $\lnot H_i$ predicted $e.$ Because $\lnot H_i$ is made of a bunch of mutually exclusive hypotheses, this term can be calculated by summing $\mathbb P(H_k) \cdot \mathbb P(e \mid H_k)$ for every $H_k$ in the collection except $H_i.$ Performing that replacement, and swapping the order of multiplication, we get:

$$\frac{\mathbb P(H_i \mid e)}{\mathbb P(\lnot H_i \mid e)} = \frac{\mathbb P(e \mid H_i) \cdot \mathbb P(H_i)}{\sum_{k \neq i} \mathbb P(e \mid H_k) \cdot \mathbb P(H_k)}.$$

These are the posterior odds for $H_i$ versus $\lnot H_i.$ Because $H_i$ and $\lnot H_i$ are mutually exclusive and exhaustive, we can convert these odds into a probability for $H_i,$ by calculating numerator / (numerator + denominator), in the same way that $3 : 4$ odds become a 3 / (3 + 4) probability. When we do so, equation (2) drops out:

$$\mathbb P(H_i\mid e) = \frac{\mathbb P(e\mid H_i) \cdot \mathbb P(H_i)}{\sum_k \mathbb P(e\mid H_k) \cdot \mathbb P(H_k)}.$$

Thus, we see that the probabilistic formulation of Bayes' rule follows from the odds form, but is less general, in that it only works when the set of hypotheses being considered are mutually exclusive and exhaustive.

We also see that the probabilistic formulation converts the posterior odds into a posterior probability. When computing [bayes_rule_vector multiple updates in a row], you actually only need to perform this "normalization" step once at the very end of your calculations — which means that the odds form of Bayes' rule is also more efficient in practice.


Sparking widgets

10% of widgets are bad and 90% are good. 4% of good widgets emit sparks, and 12% of bad widgets emit sparks. If a widget is sparking, find an easy way to calculate the chance that it's a bad widget, keeping the operations as simple as possible so that it's easy to do the calculation entirely in your head.

You're encouraged to solve the problem yourself without looking at the answer. an answer can be found in [bayes_extraordinary_claims].


Sock-dresser search

You left your socks somewhere in your room.  You think there's a 4/5 chance that they're in your dresser, so you start looking through your dresser's 8 drawers.  After checking 6 drawers at random, you haven't found your socks yet.  What is the probability you will find your socks in the next drawer you check?

You're encouraged to solve the problem yourself before looking up the answer. The answer can be found in [1y6].


Ordinary claims require ordinary evidence

A corollary to [21v extraordinary claims require extraordinary evidence] is "ordinary claims require only ordinary evidence"%%note: Attributed to [Gwern Branwen](https://www.gwern.net/), who hasn't popularized this claim nearly as much as Carl Sagan popularized the extraordinary version.%%

This corollary is probably more important in practice, as, in our day-to-day lives, we encounter more ordinary claims than extraordinary claims, but may be more tempted to reject many of those ordinary claims by demanding extraordinary evidence (a bias known as "motivated skepticism").

# Example: A failing new employee

One contributed example:

> A few years back, a senior person at my workplace told me that a new employee wasn't getting his work done on time, and that she'd had to micromanage him to get any work out of him at all.  This was an unpleasant fact for a number of reasons; I'd liked the guy, and I'd advocated for hiring him to our Board of Directors just a few weeks earlier (which is why the senior manager was talking to me).  I could have demanded more evidence, I could have demanded that we give him more time to work out, I could have demanded a videotape and signed affidavits… but a new employee not working out, just *isn't that improbable*.  Could I have named the exact prior odds of an employee not working out, could I have said [1zm how much more likely] I was to hear that exact report of a long-term-bad employee than a long-term-good employee?  No, but 'somebody hires the wrong person' happens all the time, and I'd seen it happen before.  It wasn't an extraordinary claim, and I wasn't licensed to ask for extraordinary evidence.  To put numbers on it, I thought the proportion of bad to good employees was on the order of 1 : 4 at the time, and the likelihood ratio for the manager's report seemed more like 10 : 1.

Or to put it another way:  The rule is 'extraordinary claims require extraordinary evidence', not 'inconvenient but ordinary claims require extraordinary evidence'.

In everyday life, we consider many more ordinary claims than extraordinary claims — including many ordinary claims whose inconvenience might tempt us to dismiss their adequate but ordinary evidence. In those cases, it's important to remember that ordinary claims only require ordinary evidence.


Frequency diagrams: A first look at Bayes

Bayesian reasoning is about how to revise our beliefs in the light of evidence.

We'll start by considering one scenario in which the strength of the evidence has clear numbers attached.

(Don't worry if you don't know how to solve the following problem.  We'll see shortly how to solve it.)

Suppose you are a nurse screening a set of students for a sickness called Diseasitis.%note: Literally "inflammation of the disease".%

- You know, from past population studies, that around 20% of the students will have Diseasitis at this time of year.

You are testing for Diseasitis using a color-changing tongue depressor, which usually turns black if the student has Diseasitis.

- Among patients with Diseasitis, 90% turn the tongue depressor black.
- However, the tongue depressor is not perfect, and also turns black 30% of the time for healthy students.

One of your students comes into the office, takes the test, and turns the tongue depressor black.  What is the probability that they have Diseasitis?

(*If* you think you see how to do it, you can try to solve this problem before continuing.  To quickly see if you got your answer right, you can expand the "Answer" button below; the derivation will be given shortly.)

%hidden(Answer): The probability a student with a blackened tongue depressor has Diseasitis is 3/7, roughly 43%. %

This problem can be solved a hard way or a clever easy way.  We'll walk through the hard way first.

First, we imagine a population of 100 students, of whom 20 have Diseasitis and 80 don't.%%note:Multiple studies show that thinking about concrete numbers such as "20 out of 100 students" or "200 out of 1000 students" is more likely to produce correct spontaneous reasoning on these problems than thinking about percentages like "20% of students." E.g. "[Probabilistic reasoning in clinical medicine](https://faculty.washington.edu/jmiyamot/p548/eddydm%20prob%20reas%20i%20clin%20medicine.pdf)" by David M. Eddy (1982).%%

![prior frequency](https://i.imgur.com/gvcoqQN.png)

90% of sick students turn their tongue depressor black, and 30% of healthy students turn the tongue depressor black.  So we see black tongue depressors on 90% * 20 = 18 sick students, and 30% * 80 = 24 healthy students.

![posterior frequency](https://i.imgur.com/0XDbrYi.png)

What's the probability that a student with a black tongue depressor has Diseasitis? From the diagram, there are 18 sick students with black tongue depressors.  18 + 24 = 42 students in total turned their tongue depressors black.  Imagine reaching into a bag of all the students with black tongue depressors, and pulling out one of those students at random; what's the chance a student like that is sick?

![conditional probability](http://i.imgur.com/W6avfHO.png?1)

The final answer is that a patient with a black tongue depressor has an 18/42 = 3/7 = 43% probability of being sick.

Many medical students have at first found this answer counter-intuitive:  The test correctly detects Diseasitis 90% of the time! If the test comes back positive, why is it still less than 50% likely that the patient has Diseasitis? Well, the test also incorrectly "detects" Diseasitis 30% of the time in a healthy patient, and we start out with lots more healthy patients than sick patients.

The test does provide *some* evidence in favor of of the patient being sick.  The probability of a patient being sick goes from 20% before the test, to 43% after we see the tongue depressor turn black.  But this isn't conclusive, and we need to perform further tests, maybe more expensive ones.

*If* you feel like you understand this problem setup, consider trying to answer the following question before proceeding: What's the probability that a student who does *not* turn the tongue depressor black - a student with a negative test result - has Diseasitis?  Again, we start out with 20% sick and 80% healthy students, 70% of healthy students will get a negative test result, and only 10% of sick students will get a negative test result.

%%hidden(Answer):Imagine 20 sick students and 80 healthy students.  10% * 20 = 2 sick students have negative test results.  70% * 80 = 56 healthy students have negative test results.  Among the 2+56=58 total students with negative test results, 2 students are sick students with negative test results.  So 2/58 = 1/29 = 3.4% of students with negative test results have Diseasitis.%%

%if-before([1x1]): Now let's turn to a faster, easier way to solve the same problem.%
%!if-before([1x1]): For a more clever way to perform the same calculation, see [1x1].%


Definition

When substantial controversy exists about how to define a term, good epistemic policy is for both sides to adopt new, more specific terms whose definitions are not further disputed.  To whatever extent possible, definitions should not be phrased in a way that tries to pre-emptively settle an argument or 'bake in' one answer to a factual or policy disagreement.  See [A Human's Guide to Words](http://wiki.lesswrong.com/wiki/A_Human%27s_Guide_to_Words).


Frequency diagram

[summary:  Frequency diagrams, like [1wy waterfall diagrams], provide a way of visualizing [1lz Bayes' Rule].  For example, if 20% of the patients in the screening patient are sick (red) and 80% are healthy (blue); and 90% of the sick patients get positive test results; and 30% of the healthy patients get positive test results, we could visualize the probabilities as proportions of a large population, as in the following frequency diagram:

![Frequency diagram](https://i.imgur.com/0XDbrYi.png)

Then, when asking about the probability that a patient with a black tongue depressor is sick, we can divide the number of sick patients with black tongue depressors (18 in this case) by the number of total patients with black tongue depressors (18 + 24, in this case).]

Frequency diagrams, like [1wy waterfall diagrams], provide a way of visualizing [1lz Bayes' Rule].  For example, if 20% of the patients in the screening patient are sick (red) and 80% are healthy (blue); and 90% of the sick patients get positive test results; and 30% of the healthy patients get positive test results, we could visualize the probabilities as proportions of a large population, as in the following frequency diagram:

![Frequency diagram](https://i.imgur.com/0XDbrYi.png)

See [55z] for a walkthrough of the diagram.


Odds: Introduction

Lets say we have a bag containing twice as many blue marbles as red marbles. Then, if you reach in without looking and pick out a marble at random, the odds are 2 : 1 in favor of drawing a blue marble as opposed to a red one.

Odds express *relative* quantities. 2 : 1 odds are the same as 4 : 2 odds are the same as 600 : 300 odds. For example, if the bag contains 1 red marble and 2 blue marbles, or 2 red marbles and 4 blue marbles, then your chance of pulling out a red marble is the same in both cases:

![](https://i.imgur.com/IcsOXl0.png?0)

In other words, given odds of $(x : y)$ we can scale it by any positive number $\alpha$ to get equivalent odds of $(\alpha x : \alpha y).$

# Converting odds to probabilities

If there were also green marbles, the *relative odds* for red *versus* blue would still be (1 : 2), but the *probability* of drawing a red marble would be lower.

![](https://i.imgur.com/ooyn9Py.png?0)

If red, blue, and green are the only kinds of marbles in the bag, then we can turn odds of $(r : b : g)$ into probabilities $(p_r : p_b : p_g)$ that say the probability of drawing each kind of marble.  Because red, blue, and green are the only possibilities, $p_r + p_g + p_b$ must equal 1, so $(p_r : p_b : p_g)$ must be odds equivalent to $(r : b : g)$ but "normalized" such that it sums to one. For example, $(1 : 2 : 1)$ would normalize to $\frac{1}{4} : \frac{2}{4} : \frac{1}{4},$ which are the probabilities of drawing a red / blue / green marble (respectively) from the bag on the right above.

Note that if red and blue are not the only possibilities, then it doesn't make sense to convert the odds $(r : b)$ of red vs blue into a probability. For example, if there are 100 green marbles, one red marble, and two blue marbles, then the odds of red vs blue are 1 : 2, but the probability of drawing a red marble is much lower than 1/3! Odds can only be converted into probabilities if its terms are [-1rd].

Imagine a forest with some sick trees and some healthy trees, where the odds of a tree being sick (as opposed to heathy) are (2 : 3), and every tree is either sick or healthy (there are no in-between states). Then the probability of randomly picking a sick tree from among *all* trees is 2 / 5, because 2 out of every (2 + 3) trees is sick.

![](https://i.imgur.com/GVZnz2c.png?0)

In general, the operation we're doing here is taking relative odds like $(a : b : c \ldots)$ and dividing each term by the sum $(a + b + c \ldots)$ to produce $$\left(\frac{a}{a + b + c \ldots} : \frac{b}{a + b + c \ldots} : \frac{c}{a + b + c \ldots}\ldots\right)$$ Dividing each term by the sum of all terms gives us an equivalent set of odds (because each element is divided by the same amount) whose terms sum to 1.

This process of dividing a set of odds by the sum of its terms to get a set of probabilities that sum to 1 is called [1rk normalization].

# Converting probabilities to odds

Let's say we have two events R and B, which might be things like "I draw a red marble" and "I draw a blue marble." Say $\mathbb P(R) = \frac{1}{4}$ and $\mathbb P(B) = \frac{1}{2}.$ What are the odds of R vs B? $\mathbb P(R) : \mathbb P(B) = \left(\frac{1}{4} : \frac{1}{2}\right),$ of course.

Equivalently, we can take the odds $\left(\frac{\mathbb P(R)}{\mathbb P(B)} : 1\right)$, because $\frac{\mathbb P(R)}{\mathbb P(B)}$ is how many more times likely R is than B. In this example, $\frac{\mathbb P(R)}{\mathbb P(B)} = \frac{1}{2},$ because R is half as likely as B. Sometimes, the quantity $\frac{\mathbb P(R)}{\mathbb P(B)}$ is called the "odds ratio of R vs B," in which case it is understood that the odds for R vs B are $\left(\frac{\mathbb P(R)}{\mathbb P(B)} : 1\right).$

# Odds to ratios

When there are only two terms $x$ and $y$ in a set of odds, the odds can be written as a ratio $\frac{x}{y}.$ The odds *ratio* $\frac{x}{y}$ refers to the *odds* $(x : y),$ or, equivalently, $\left(\frac{x}{y} : 1\right).$


Odds: Refresher

Let's say that, in a certain forest, there are 2 sick trees for every 3 healthy trees. We can then say that the odds of a tree being sick (as opposed to healthy) are $(2 : 3).$

Odds express *relative* chances.  Saying "There's 2 sick trees for every 3 healthy trees" is the same as saying "There's 10 sick trees for every 15 healthy trees."  If the original odds are $(x : y)$ we can multiply by a positive number $\alpha$ and get a set of equivalent odds $(\alpha x : \alpha y).$

If there's 2 sick trees for every 3 healthy trees, and every tree is either sick or healthy, then the *probability* of randomly picking a sick tree from among *all* trees is 2/(2+3):

![Odds v probabilities](https://i.imgur.com/GVZnz2c.png?0)

If the set of possibilities $A, B, C$ are [1rd mutually exclusive and exhaustive], then the probabilities $\mathbb P(A) + \mathbb P(B) + \mathbb P(C)$ should sum to $1.$  If there's no further possibilities $d,$ we can convert the relative odds $(a : b : c)$ into the probabilities $(\frac{a}{a + b + c} : \frac{b}{a + b + c} : \frac{c}{a + b + c}).$  The process of dividing each term by the sum of terms, to turn a set of proportional odds into probabilities that sum to 1, is called [1rk normalization].

When there are only two terms $x$ and $y$ in the odds, they can be expressed as a single ratio $\frac{x}{y}.$  An odds ratio of $\frac{x}{y}$ refers to odds of $(x : y),$ or, equivalently, odds of $\left(\frac{x}{y} : 1\right).$ Odds of $(x : y)$ are sometimes called odds ratios, where it is understood that the actual ratio is $\frac{x}{y}.$


Conditional probability: Refresher

[summary: $\mathbb P(\text{left} \mid \text{right})$ is the probability that the thing on the left is true _assuming_ the thing on the right is true, and it's defined as $\frac{\mathbb P(\text{left} \land \text{right})}{\mathbb P(\text{right})}.$

Thus, $\mathbb P(yellow \mid banana)$ is the probability that a banana is yellow ("the probability of yellowness given banana"), while $\mathbb P(banana \mid yellow)$ is the probability that a yellow object is a banana ("the probability of banana, given yellowness").]

$\mathbb P(\text{left} \mid \text{right})$ is the probability that the thing on the left is true _assuming_ the thing on the right is true, and it's defined as $\frac{\mathbb P(\text{left} \land \text{right})}{\mathbb P(\text{right})}.$

Thus, $\mathbb P(yellow \mid banana)$ is the probability that a banana is yellow ("the probability of yellowness given banana"), while $\mathbb P(banana \mid yellow)$ is the probability that a yellow object is a banana ("the probability of banana, given yellowness").%%note: In general, $\mathbb P(v)$ is an abbreviation of $\mathbb P(V = v)$ for some variable $V$, which is assumed to be known from the context. For example, $\mathbb P(yellow)$ might stand for $\mathbb P({ColorOfNextObjectInBag}=yellow)$ where $ColorOfNextObjectInBag$ is a [random_variable variable] in our [-probability_distribution] $\mathbb P,$ and $yellow$ is one of the [random_variable_value values] that that variable can take on.%%

$\mathbb P(x \land y)$ is used to denote the probability of both $x$ and $y$ being true simultaneously (according to some [-probability_distribution] $\mathbb P$). $\mathbb P(x\mid y)$, pronounced "the conditional probability of x, given y", is defined to be the quantity

$$\frac{\mathbb P(x \wedge y)}{\mathbb P(y)}.$$

For example, in the [22s Diseasitis] problem, $\mathbb P({sick}\mid {positive})$ is the probability that a patient is sick _given_ a positive test result, and it's calculated by taking the 18% patients who are sick *and* have positive test results, and dividing by all 42% of the patients who got positive test results.  That is, $\mathbb P({sick}\mid {positive})$ $=$ $\frac{\mathbb P({sick} \land {positive})}{\mathbb P({positive})}.$

Using a [bayes_frequency_diagrams frequency diagram], we can visualize $\mathbb P(sick \mid positive)$ as the probability of drawing a $sick$ result from a bag of only those people in the population who got a $positive$ result.

![diseasitis frequency](https://i.imgur.com/0XDbrYi.png?0)

![bag of 18 and 24 patients](https://i.imgur.com/W6avfHO.png?0)

The "given" operator in $\mathbb P(x\mid y)$ tells us to assume that $y$ is true, to restrict our attention to only possible cases where $y$ is true, and then ask about the probability of $x$ *within* those cases.

Note that $\mathbb P(positive \mid sick)$ is _not_ the same as $\mathbb P(sick \mid positive).$ To find the probability that a patient has a positive result given that they're sick, we can visualize taking the 20 sick patients and putting them in a group, and then asking the probability that a randomly selected one will have a positive result, which will be $\frac{18}{20} = 0.9$ — so $\mathbb P(positive \mid sick) = 90\%,$ while $\mathbb P(sick \mid positive) \approx 43\%.$ Mixing up which one is which is an unfortunate source of of many practical errors when you're trying to do these calculations using only the formal notation, at least until you get used to it. Just remember that $\mathbb P(\text{left} \mid \text{right})$ is the probability of the thing on the left _given_ that the thing on the right is true.