Likelihood functions, p-values, and the replication crisis

[summary:
Or, __Why Switching From Reporting p-values to Reporting Likelihood Functions Might Help Fix the Replication Crisis:  A personal view by Eliezer Yudkowsky.__

Short version: Report [56t likelihoods], not p-values.]

__Or:  Switching From Reporting p-values to Reporting Likelihood Functions Might Help Fix the Replication Crisis:  A personal view by Eliezer Yudkowsky.__

_Disclaimers:_

- _This dialogue was written by a [1r8 Bayesian].  The voice of the Scientist in the dialogue below may fail to pass the [Ideological Turing Test](https://en.wikipedia.org/wiki/Ideological_Turing_Test) for frequentism, that is, it may fail to do justice to frequentist arguments and counterarguments._
- _It does not seem sociologically realistic, to the author, that the proposal below could be adopted by the scientific community at large within the next 10 years. It seemed worth writing down nevertheless._

_If you don't already know Bayes' rule, check out Arbital's [1zq Guide to Bayes' Rule] if confused._

----------

**Moderator:**  Hello, everyone.  I'm here today with the **Scientist,** a working experimentalist in... chemical psychology, or something; with the **Bayesian,** who's going to explain why, on their view, we can make progress on the replication crisis by replacing p-values with some sort of Bayesian thing--

**Undergrad:**  Sorry, can you repeat that?

**Moderator:**  And finally, the **Confused Undergrad** on my right.  **Bayesian,** would you care to start by explaining the rough idea?

**Bayesian:**  Well, the rough idea is something like this.  Suppose we flip a possibly-unfair coin six times, and observe the sequence HHHHHT.  Should we be suspicious that the coin is biased?

**Scientist:**  No.

**Bayesian:**  This isn't a literal coin.  Let's say we present a series of experimental subjects with two cookies on a plate, one with green sprinkles and one with red sprinkles.  The first five people took cookies with green sprinkles and the sixth person took a cookie with red sprinkles. %note: And they all saw separate plates, on a table in the waiting room marked "please take only one" so nobody knew what was being tested, and none of them saw the others' cookie choices.%  Do we think most people prefer green-sprinkled cookies or do we think it was just random?

**Undergrad:**  I think I would be *suspicious* that maybe people liked green sprinkles better.  Or at least that the sort of people who go to the university and get used as test subjects like green sprinkles better.  Yes, even if I just saw that happen in the first six cases.  But I'm guessing I'm going to get dumped-on for that.

**Scientist:**  I think I would be genuinely not-yet-suspicious.  There's just too much stuff that looks good after N=6 that doesn't pan out with N=60.

**Bayesian:**  I'd at least strongly suspect that people in the test population *don't* mostly prefer red sprinkles.  But the reason I introduced this example is as an oversimplified example of how current scientific statistics calculate so-called "p-values", and what a Bayesian sees as the central problem with that.

**Scientist:**  And we can't use a more realistic example with 30 subjects?

**Bayesian:**  That would not be nice to the Confused Undergrad.

**Undergrad:**  *Seconded.*

**Bayesian:**  So:  Heads, heads, heads, heads, heads, tails.  I ask: is this "statistically significant", as current conventional statisticians would have the phrase?

**Scientist:**  I reply: no.  On the null hypothesis that the coin is fair, or analogously that people have no strong preference between green and red sprinkles, we should expect to see a result as extreme as this in 14 out of 64 cases.

**Undergrad:**  Okay, just to make sure I have this straight:  That's because we're considering results like HHHTHH or TTTTTT to be equally or more extreme, and there are 14 total possible cases like that, and we flipped the coin 6 times which gives us $2^6 = 64$ possible results.  14/64 = 22%, which is not less than 5%, so this is not statistically significant at the $p<0.05$ level.

**Scientist:**  That's right.  However, I'd also like to observe as a matter of practice that even if you get HHHHHH on your first six flips, I don't advise stopping there and sending in a paper where you claim that the coin is biased towards heads.

**Bayesian:**  Because if you can decide to *stop* flipping the coin at a time of your choice, then we have to ask "How likely it is that you can find some place to stop flipping the coin where it looks like there's a significant number of heads?"  That's a whole different kettle of fish according to the p-value concept.

**Scientist:**  I was just thinking that N=6 is not a good number of experimental subjects when it comes to testing cookie preferences.  But yes, that too.

**Undergrad:**  Uh... why does it make a difference if I can decide when to stop flipping the coin?

**Bayesian:**  What an excellent question.

**Scientist:**  Well, this is where the concept of p-values is less straightforward than plugging the numbers into a statistics package and believing whatever the stats package says.  If you previously decided to flip exactly six coins, and then stop, regardless of what results you got, then you would get a result as extreme as "HHHHHH" or "TTTTTT" 2/64 of the time, or 3.1%, so p<0.05.  However, suppose that instead you are a bad fraudulent scientist, or maybe just an ignorant undergraduate who doesn't realize what they're doing wrong.  Instead of picking the number of flips in advance, you keep flipping the coin until the statistics software package tells you that you got a result that *would have been* statistically significant *if,* contrary to the actual facts of the case, you'd decided *in advance* to flip the coin exactly that many times.  But you didn't decide in advance to flip the coin that many times.  You decided it after checking the actual results.  Which you are not allowed to do.

**Undergrad:**  I've heard that before, but I'm not sure I understand on a gut level why it's bad for me to decide when I've collected enough data.

**Scientist:**  What we're trying to do here is set up a test that the null hypothesis cannot pass--to make sure that where there's no fire, there's unlikely to be smoke.  We want a complete experimental process which is unlikely to generate a "statistically significant" discovery if there's no real phenomenon being investigated.  If you flip the coin exactly six times, if you decide that in advance, then you are less than 5% likely to get a result as extreme as "six heads" or "six tails".  If you flip the coin repeatedly, and check *repeatedly* for a result that *would* have had p<0.05 if you'd decided in advance to flip the coin exactly that many times, your chance of getting a nod from the statistics package is *much greater* than 5%.  You're carrying out a process which is much more than 5% likely to yell "smoke" in the absence of fire.

**Bayesian:**  The way I like to explain the problem is like this:  Suppose you flip a coin six times and get HHHHHT.  If you, in the secret depths of your mind, in your inner heart that no other mortal knows, decided to flip the coin exactly six times and then stop, this result is not statistically significant; p=0.22.  If you decided in the secret depths of your heart, that you were going to *keep flipping the coin until it came up tails,* then the result HHHHHT *is* statistically significant with p=0.03, because the chance of a fair coin requiring you to wait six or more flips to get one tail is 1/32.

**Undergrad:**  What.

**Scientist:**  It's a bit of a parody, obviously--nobody would really decide to flip a coin until they got one tail, then stop--but the Bayesian is technically correct about how the rules for p-values work.  What we're asking is how rare our outcome is, within the class of outcomes we *could* have gotten.  The person who keeps flipping the coin until they get one tail has possible outcomes {T, HT, HHT, HHHT, HHHHT, HHHHHT, HHHHHHT...} and so on.  The class of outcomes where you get to the sixth round or later is the class of outcomes {HHHHHT, HHHHHHT...} and so on, a set of outcomes which collectively have a probability of 1/64 + 1/128 + 1/256... = 1/32.  Whereas if you flip the coin exactly six times, your class of possible outcomes is {TTTTTT, TTTTTH, TTTTHT, TTTTHH...}, a set of 64 possibilities within which the outcome HHHHHT is something we could lump in with {HHHHTH, HHHTHH, THTTTT...} and so on.  So although it's counterintuitive, if we really had decided to run the first experiment, HHHHHT would be a statistically significant result that a fair coin would be unlikely to give us.  And if we'd decided to run the second experiment, HHHHHT would not be a statistically significant result because fair coins sometimes do something *like* that.

**Bayesian:**  And it doesn't bother you that the meaning of your experiment depends on your private state of mind?

**Scientist:**  It's an honor system.  Just like the process doesn't work if you lie about which coinflips you actually saw, the process doesn't work--is not a fair test in which non-fires are unlikely to generate smoke--if you lie about *which experiment you performed.*  You must honestly say the rules you followed to flip the coin the way you did.  Unfortunately, since what you were thinking about the experiment is less clearly visible than the actual coinflips, people are much more likely to twist *how they selected* their number of experimental subjects, or how they selected which tests to run on the data, than they are to tell a blatant lie about what the data said.  That's p-hacking.  There are, unfortunately, much subtler and less obvious ways of generating smoke without fire than claiming post facto to have followed the rule of flipping the coin until it came up tails.  It's a serious problem, and underpins some large part of the great replication crisis, nobody's sure exactly how much.

**Undergrad:**  That... sorta makes sense, maybe?  I'm guessing this is one of those cases where I have to work through a lot of example problems before it becomes obvious.

**Bayesian:**  No.

**Undergrad:**  No?

**Bayesian:**  You were right the first time, Undergrad.  If what the experimentalist is *thinking* has no causal impact on the coin, then the experimentalist's thoughts cannot possibly make any difference to what the coin is saying to us about Nature.  My dear Undergrad, you are being taught weird, ad-hoc, overcomplicated rules that aren't even internally consistent--rules that theoretically output *different* wrong answers depending on your private state of mind!  And *that* is a problem that runs far deeper into the replication crisis than people misreporting their inner thoughts.

**Scientist:**  A bold claim to say the least.  But don't be coy; tell us what you think we should be doing instead?

**Bayesian:**  I analyze as follows:  The exact result HHHHHT has a 1/64 or roughly 1.6% probability of being produced by a fair coin flipped six times.  To simplify matters, suppose we for some reason were already pondering the hypothesis that the coin was biased to produce 5/6 heads--again, this is an unrealistic example we can de-simplify later.  Then this hypothetical biased coin would have a $(5/6)^5 \cdot (1/6)^1 \approx 6.7\%$ probability of producing HHHHHT.  So between our two hypotheses "The coin is fair" and "The coin is biased to produce 5/6ths heads", our exact experimental result is *4.3 times more likely* to be observed in the second case.  HHHHHT is also 0.01% likely to be produced if the coin is biased to produce 1/6th heads and 5/6ths tails, so we've already seen some quite strong evidence *against* that particular hypothesis, if anyone was considering it.  The exact experimental outcome HHHHHT is 146 times as likely to be produced by a fair coin as by a coin biased to produce 1/6th heads. %note: Recall the Bayesian's earlier thought that, after seeing five subjects select green cookies followed by one subject selecting a red cookie, we'd already picked up strong evidence *against* the proposition: "Subjects in this experimental population lopsidedly prefer red cookies over green cookies."%

**Undergrad:**  Well, I think I can follow the calculation you just did, but I'm not clear on what that calculation means.

**Bayesian:**  I'll try to explain the meaning shortly, but first, note this.  That calculation we just did has *no dependency whatsoever* on *why* you flipped the coin six times.  You could have stopped after six because you thought you'd seen enough coinflips.  You could have done an extra coinflip after the first five because Namagiri Thayar spoke to you in a dream.  The coin doesn't care.  The coin isn't affected.  It remains true that the exact result HHHHHT is 23% as likely to be produced by a fair coin as by a biased coin that comes up heads 5/6ths of the time.

**Scientist:**  I agree that this is an interesting property of the calculation that you just did.  And then what?

**Bayesian:**  You report the results in a journal.  Preferably including the raw data so that others can calculate the likelihoods for any other hypotheses of interest.  Say somebody else suddenly becomes interested in the hypothesis "The coin is biased to produce 9/10ths heads."  Seeing HHHHHT is 5.9% likely in that case, so 88% as likely than if the coin is biased to produce 5/6ths heads (making the data 6.7% likely), or 3.7 times as likely than if the coin is fair (making the data 1.6% likely).  But you shouldn't have to think of all possible hypotheses in advance.  Just report the raw data so that others can calculate whatever likelihoods they need.  Since this calculation deals with the *exact* results we got, rather than summarizing it into some class or set of supposedly similar results, it puts a greater emphasis on reporting your exact experimental data to others.

**Scientist:**  Reporting raw data seems an important leg of a good strategy for fighting the replication crisis, on this we agree.  I nonetheless don't understand what experimentalists are supposed to *do* with this "X is Q times as likely as Y" stuff.

**Undergrad:**  Seconded.

**Bayesian:**  Okay, so... this isn't trivial to describe without making you run through [1zq a whole introduction to Bayes' rule]--

**Undergrad:**  Great.  Just what I need, another weird complicated 4-credit course on statistics.

**Bayesian:**  It's literally [693 a 1-hour read if you're good at math]. It just isn't literally *trivial* to understand with no prior introduction.  Well, even with no introduction whatsoever, I may be able to fake it with statements that will *sound* like they might be reasonable--and the reasoning *is* valid, it just might not be obvious that it is.  Anyway.  It is a theorem of probability that the following is valid reasoning:

*(the Bayesian takes a breath)*

**Bayesian:**  Suppose that Professor Plum and Miss Scarlet are two suspects in a murder.  Based on their prior criminal convictions, we start out thinking that Professor Plum is twice as likely to have committed the murder as Miss Scarlet.  We then discover that the victim was poisoned.  We think that, assuming he committed a murder, Professor Plum would be 10% likely to use poison; assuming Miss Scarlet committed a murder, she would be 60% likely to use poison.  So Professor Plum is around *one-sixth as likely* to use poison as Miss Scarlet.  Then after observing the victim was poisoned, we should update to think Plum is around one-third as likely to have committed the murder as Scarlet: $2 \times \frac{1}{6} = \frac{1}{3}.$

**Undergrad:**  Just to check, what do you mean by saying that "Professor Plum is one-third as likely to have committed the murder as Miss Scarlet"?

**Bayesian:**  I mean that if these two people are our only suspects, we think Professor Plum has a 1/4 probability of having committed the murder and Miss Scarlet has a 3/4 probability of being guilty.  So Professor Plum's probability of guilt is one-third that of Miss Scarlet's.

**Scientist:**  Now *I'd* like to know what you mean by saying that Professor Plum had a 1/4 probability of committing the murder.  Either Plum committed the murder or he didn't; we can't observe the murder be committed multiple times and Professor Plum doing it 1/4th of the time.

**Bayesian:**  Are we going there?  I guess we're going there.  My good Scientist, I mean that if you offered me either side of an even-money bet on whether Plum committed the murder, I'd bet that he didn't do it.  But if you offered me a gamble that costs \$1 if Professor Plum is innocent and pays out \$5 if he's guilty, I'd cheerfully accept that gamble.  We only ran the 2012 US Presidential Election one time, but that doesn't mean that on November 7th you should've refused a \$10 bet that paid out \$1000 if Obama won.  In general when prediction markets and large liquid betting pools put 60% betting odds on somebody winning the presidency, that outcome tends to happen 60% of the time; they are *well-calibrated* for probabilities in that range.  If they were systematically uncalibrated--if in general things happened 80% of the time when prediction markets said 60%--you could use that fact to pump money out of prediction markets.  And your pumping out that money would adjust the prediction-market prices until they were well-calibrated.  If things to which prediction markets assign 70% probability happen around 7 times out of 10, why insist for reasons of ideological purity that the probability statement is meaningless?

**Undergrad:**  I admit, that *sounds* to me like it makes sense, if it's not just the illusion of understanding due to my failing to grasp some far deeper debate.

**Bayesian:**  There is indeed a [4y9 deeper debate], but what the deeper debate works out to is that your illusion of understanding is pretty much accurate as illusions go.

**Scientist:**  Yeah, I'm going to want to come back to that issue later.  What if there are two agents who both seem 'well-calibrated' as you put it, but one agent says 60% and the other agent says 70%?

**Bayesian:**  If I flip a coin and don't look at it, so that I don't know yet if it came up heads or tails, then my ignorance about the coin isn't a fact about the coin, it's a fact about me.  Ignorance exists in the mind, not in the environment.  A blank map does not correspond to a blank territory.  If you peek at the coin and I don't, it's perfectly reasonable for the two of us to occupy different states of uncertainty about the coin.  And given that I'm not absolutely certain, I can and should quantify my uncertainty using probabilities.  There's like [7ry 300 different theorems] showing that I'll get into trouble if my state of subjective uncertainty *cannot* be viewed as a coherent probability distribution.  You kinda pick up on the trend after just the fourth time you see a slightly different clever proof that any violation of the standard probability axioms will cause the graves to vomit forth their dead, the seas to turn red as blood, and the skies to rain down dominated strategies and combinations of bets that produce certain losses--

**Scientist:**  Sorry, I shouldn't have said anything just then.  Let's come back to this later?  I'd rather hear first what you think we should do with the likelihoods once we have them.

**Bayesian:**  On the laws of probability theory, those likelihood functions *are* the evidence.  They are the objects that send our prior odds of 2 : 1 for Plum vs. Scarlet to posterior odds of 1 : 3 for Plum vs. Scarlet.  For any two hypotheses you care to name, if you tell me the relative likelihoods of the data given those hypotheses, I know how to update my beliefs.  If you change your beliefs in any other fashion, the skies shall rain dominated strategies etcetera.  Bayes' theorem:  It's not just a statistical method, it's the LAW.

**Undergrad:**  I'm sorry, I still don't understand.  Let's say we do an experiment and find data that's 6 times as likely if Professor Plum killed Mr. Boddy than if Miss Scarlet did.  Do we arrest Professor Plum?

**Scientist:**  My guess is that you're supposed to make up a 'prior probability' that sounds vaguely plausible, like 'a priori, I think Professor Plum is 20% likely to have killed Mr. Boddy'.  Then you combine that with your 6 : 1 likelihood ratio to get 3 : 2 posterior odds that Plum killed Mr. Boddy.  So your paper reports that you've established a 60% posterior probability that Professor Plum is guilty, and the legal process does whatever it does with that.

**Bayesian:**  *No.*  Dear God, no!  Is that really what people think Bayesianism is?

**Scientist:**  It's not?  I did always hear that the strength of Bayesianism is that it gives us posterior probabilities, which p-values don't actually do, and that the big weakness was that it got there by making up prior probabilities more or less out of thin air, which means that nobody will ever be able to agree on what the posteriors are.

**Bayesian:**  Science papers should report *likelihoods.*  Or rather, they should report the raw data and helpfully calculate some likelihood functions on it.  Not posteriors, never posteriors.

**Undergrad:**  What's a posterior?  I'm trusting both of you to avoid the obvious joke here.

**Bayesian:**  A [1rp posterior probability] is when you say, "There's a 60% probability that Professor Plum killed Mr. Boddy."  Which, as the Scientist points out, is something you never get from p-values.  It's also something that, in my own opinion, should never be reported in an experimental paper, because it's not *the result of an experiment.*

**Undergrad:**  But... okay, Scientist, I'm asking you this one.  Suppose we see data statistically significant at p<0.01, something we're less than 1% probable to see if the null hypothesis "Professor Plum didn't kill Mr. Boddy" is true. Do we arrest him?

**Scientist:**  First of all, that's not a realistic null hypothesis.  A null hypothesis is something like "Nobody killed Mr. Boddy" or "All suspects are equally guilty."  But even if what you just said made sense, even if we could reject Professor Plum's innocence at p<0.01, you still can't say anything like, "It is 99% probable that Professor Plum is guilty."  That is just not what p-values mean.

**Undergrad:**  Then what *do* p-values mean?

**Scientist:**  They mean we saw data inside a preselected class of possible results, which class, as a whole, is less than 1% likely to be produced if the null hypothesis is true.  That's *all* that it means.  You can't go from there to "Professor Plum is 99% likely to be guilty," for reasons the Bayesian is probably better at explaining.  You can't go from there to anywhere that's someplace else.  What you heard is what there is.

**Undergrad:**  Now I'm doubly confused.  I don't understand what we're supposed to do with p-values *or* likelihood ratios.  What kind of experiment does it take to throw Professor Plum in prison?

**Scientist:**  Well, realistically, if you get a couple more experiments at different labs also saying p<0.01, Professor Plum *is* probably guilty.

**Bayesian:**  And the 'replication crisis' is that it turns out he's *not* guilty.

**Scientist:**  Pretty much.

**Undergrad:**  That's not exactly reassuring.

**Scientist:**  Experimental science is not for the weak of nerve.

**Undergrad:**  So... Bayesian, are you about to say similarly that once you get an extreme enough likelihood ratio, say, anything over 100 to 1, or something, you can probably take something as true?

**Bayesian:**  No, it's a bit more complicated than that.  Let's say I flip a coin 20 times and get HHHTHHHTHTHTTHHHTHHHTTHT.  Well, the hypothesis "This coin was rigged to produce exactly HHHTHHHTHTHTTHHHTHHHTTHT" has a likelihood advantage of roughly a million-to-one over the hypothesis "this is a fair coin".  On any reasonable system, unless you wrote down that single hypothesis in advance and handed it to me in an envelope and didn't write down any other hypotheses or hand out any other envelopes, we'd say the hypothesis "This coin was rigged to produce HHHTHHHTHTHTTHHHTHHHTTHT" has a complexity penalty of at *least* $2^{20} : 1$ because it takes 20 bits just to describe what the coin is rigged to do.  In other words, the penalty to prior plausibility more than cancels out a million-to-one likelihood advantage.  And that's just the start of the issues.  But, *with* that said, I think there's a pretty good chance you could do okay out of just winging it, once you understood in an intuitive and common-sense way how Bayes' rule worked.  If there's evidence pointing to Professor Plum with a likelihood of 1,000 : 1 over any other suspects you can think of, in a field that probably only contained six suspects to begin with, you can figure that the prior odds against Plum weren't much more extreme than 10 : 1 and that you can legitimately be at least 99% sure now.

**Scientist:**  But you say that this is *not* something you should report in the paper.

**Bayesian:**  That's right.  How can I put this... one of the great commandments of Bayesianism is that you ought to take into account *all* the relevant evidence you have available; you can't exclude some evidence from your calculation just because you don't like it.  Besides sounding like common sense, this is also a rule you have to follow to prevent your calculations from coming up with paradoxical results, and there are various particular problems where there's a seemingly crazy conclusion and the answer is, "Well, you *also* need to condition on blah blah blah."  My point being, how do I, as an experimentalist, know what *all the relevant evidence* is?  Who am I to calculate a posterior?  Maybe somebody else published a paper that includes more evidence, with more likelihoods to be taken into account, and I haven't heard about it yet, but somebody else has.  I just contribute my own data and its likelihood function--that's all!  It's not my place to claim that I've collected *all* the relevant evidence and can now calculate posterior odds, and even if I could, somebody else could publish another paper a week later and the posterior odds would change again.

**Undergrad:**  So, roughly your answer is, "An experimentalist just publishes the paper and calculates the likelihood thingies for that dataset, and then somebody outside has to figure out what to do with the likelihood thingies."

**Bayesian:**  Somebody outside has to set up priors--probably just reasonable-sounding ignorance priors, maximum entropy stuff or complexity-based penalties or whatever--then try to make sure they've collected all the evidence, apply the likelihood functions, check to see if the result [227 makes sense], etcetera.  And then they might have to revise that estimate if somebody publishes a new paper a week later--

**Undergrad:**  That sounds *awful.*

**Bayesian:**  It would be awful if we were doing meta-analyses of p-values.  Bayesian updates are a *hell* of a lot simpler!  Like, you literally just [1zg multiply] the old posterior by the new likelihood [1zj function] and normalize.  If experiment 1 has a likelihood ratio of 4 for hypothesis A over hypothesis B, and experiment 2 has a likelihood ratio of 9 for A over B, the two experiments together have a likelihood ratio of 36.

**Undergrad:**  And you can't do that with p-values, I mean, a p-value of 0.05 and a p-value of 0.01 don't multiply out to p<0.0005--

**Scientist:**  *No.*

**Bayesian:**  I should like to take this moment to call attention to my superior smile.

**Scientist:**  I am still worried about the part of this process where somebody gets to make up prior probabilities.

**Bayesian:**  Look, that just corresponds to the part of the process where somebody decides that, having seen 1 discovery and 2 replications with p<0.01, they are willing to buy the new pill or whatever.

**Scientist:**  So your reply there is, "It's subjective, but so is what you do when you make decisions based on having seen some experiments with p-values."  Hm.  I was going to say something like, "If I set up a rule that says I want data with p<0.001, there's no further objectivity beyond that," but I guess you'd say that my asking for p<0.001 instead of p<0.0001 corresponds to my pulling a prior out of my butt?

**Bayesian**:  Well, except that asking for a particular p-value is not actually as good as pulling a prior out of your butt.  One of the first of those 300 theorems proving *doom* if you violate probability axioms, was Abraham Wald's "complete class theorem" in 1947.  Wald set out to investigate all the possible *admissible strategies,* where a strategy is a way of acting differently based on whatever observations you make, and different actions get different payoffs in different possible worlds.  Wald termed an *admissible strategy* a strategy which was not dominated by some other strategy across all possible measures you could put on the possible worlds.  Wald found that the class of admissible strategies was simply the class that corresponded to having a probability distribution, doing Bayesian updating on observations, and maximizing expected payoff.

**Undergrad:**  Can you perhaps repeat that in slightly smaller words?

**Bayesian:**  If you want to do different things depending on what you observe, and get different payoffs depending on what the real facts are, *either* your strategy can be seen as having a probability distribution and doing Bayesian updating, *or* there's another strategy that does better given at least some possible measures on the worlds and never does worse.  So if you say anything as wild as "I'm waiting to see data with p<0.0001 to ban smoking," in principle there must be some way of saying something along the lines of, "I have a prior probability of 0.01% that smoking causes cancer, let's see those likelihood functions" which does at least as well or better no matter what anyone else would say as their own prior probabilities over the background facts.

**Scientist:** Huh.

**Bayesian:**  Indeed.  And that was when the Bayesian revolution very slowly started; it's sort of been gathering steam since then.  It's worth noting that Wald only proved his theorem a couple of decades after "p-values" were invented, which, from my perspective, helps explain how science got wedged into its peculiar current system.

**Scientist:**  So you think we should burn all p-values and switch to reporting all likelihood ratios all the time.

**Bayesian:**  In a word... yes.

**Scientist:**  I'm suspicious, in general, of one-size-fits-all solutions like that.  I suspect you--I hope this is not too horribly offensive--I suspect you of idealism.  In my experience, different people need different tools from the toolbox at different times, and it's not wise to throw out all the tools in your toolbox except one.

**Bayesian:**  Well, let's be clear where I am and amn't idealistic, then.  Likelihood functions cannot solve the entire replication crisis.  There are aspects of this that can't be solved by using better statistics.  Open access journals aren't something that hinge on p-values versus likelihood functions.  The broken system of peer commentary, presently in the form of peer review, is not something likelihood functions can solve.

**Scientist:**  But likelihood functions will solve everything else?

**Bayesian:**  No, but they'll at least *help* on a surprising amount.  Let me count the ways:

**Bayesian:**  One.  Likelihood functions don't distinguish between 'statistically significant' results and 'failed' replications.  There are no 'positive' and 'negative' results.  What used to be called the null hypothesis is now just another hypothesis, with nothing special about it.  If you flip a coin and get HHTHTTTHHH, you have not "failed to reject the null hypothesis with p<0.05" or "failed to replicate".  You have found experimental data that favors the fair-coin hypothesis over the 5/6ths-heads hypothesis with a likelihood ratio of 3.78 : 1.  This may help to fight the file-drawer effect--not entirely, because there is a mindset in the journals of 'positive' results and biased coins being more exciting than fair coins, and we need to tackle that mindset directly.  But the p-value system *encourages* that bad mindset.  That's why p-hacking even exists.  So switching to likelihoods won't fix everything right away, but it *sure will help.*

**Bayesian:**  Two.  The system of likelihoods makes the importance of raw data clearer and will encourage a system of publishing the raw data whenever possible, because Bayesian analyses center around the probability of the *exact* data we saw, given our various hypotheses.  The p-value system encourages you to think in terms of the data as being just one member of a class of 'equally extreme' results.  There's a mindset here of people hoarding their precious data, which is not purely a matter of statistics.  But the p-value system *encourages* that mindset by encouraging people to think of their result as part of some undistinguished class of 'equally or more extreme' values or whatever, and that its meaning is entirely contained in it being a 'positive' result that is 'statistically significant'.

**Bayesian:**  Three.  The probability-theoretic view, or Bayesian view, makes it clear that different effect sizes are different hypotheses, as they must be, because they assign different probabilities to the exact observations we see.  If one experiment finds a 'statistically significant' effect size of 0.4 and another experiment finds a 'statistically significant' effect size of 0.1 on whatever scale we're working in, the experiment *has not replicated* and we do not yet know what real state of affairs is generating our observations.  This directly fights and negates the 'amazing shrinking effect size' phenomenon that is part of the replication crisis.

**Bayesian:**  Four.  Working in likelihood functions makes it far easier to aggregate our data.  It even helps to [227 point up] when our data is being produced under inconsistent conditions or when the true hypothesis is not being considered, because in this case we will find likelihood functions that end up being nearly zero everywhere, or where the best available hypothesis is achieving a much lower likelihood on the combined data than that hypothesis [227 expects itself to achieve].  It is a stricter concept of replication that helps quickly point up when different experiments are being performed under different conditions and yielding results incompatible with a single consistent phenomenon.

**Bayesian:**  Five.  Likelihood functions are objective facts about the data which do not depend on your state of mind.  You cannot deceive somebody by reporting likelihood functions unless you are literally lying about the data or omitting data.  There's no equivalent of 'p-hacking'.

**Scientist:**  Okay, that last claim in particular strikes me as *very* suspicious.  What happens if I want to persuade you that a coin is biased towards heads, so I keep flipping it until I randomly get to a point where there's a predominance of heads, and then choose to stop?

**Bayesian:**  "Shrug," I say.  You can't mislead me by telling me what a real coin actually did.

**Scientist:**  I'm asking you what happens if I keep flipping the coin, checking the likelihood each time, until I see that the current statistics favor my pet theory, and then I stop.

**Bayesian:**  As a pure idealist seduced by the seductively pure idealism of probability theory, I say that so long as you present me with the true data, all I can and should do is update in the way Bayes' theorem says I should.

**Scientist:**  Seriously.

**Bayesian:**  I am serious.

**Scientist:**  So it doesn't bother you if I keep checking the likelihood ratio and continuing to flip the coin until I can convince you of anything I want.

**Bayesian:**  Go ahead and try it.

**Scientist:**  What I'm actually going to do is write a Python program which simulates flipping a fair coin *up to* 300 times, and I'm going to see how many times I can get a 20:1 likelihood ratio falsely indicating that the coin is biased to come up heads 55% of the time... why are you smiling?

**Bayesian:**  I wrote pretty much the same Python program when I was first converting to Bayesianism and finding out about likelihood ratios and feeling skeptical about the system maybe being abusable in some way, and then a friend of mine found out about likelihood ratios and *he* wrote [essentially the same program, also in Python](https://gist.github.com/Soares/941bdb13233fd0838f1882d148c9ac14).  And lo, he found that false evidence of 20:1 for the coin being 55% biased was found at least once, somewhere along the way... 1.4% of the time.  If you asked for more extreme likelihood ratios, the chances of finding them dropped off even faster.

**Scientist:**  Okay, that's not bad by the p-value way of looking at things.  But what if there's some more clever way of biasing it?

**Bayesian:**  When I was... I must have been five years old, or maybe even younger, and first learning about addition, one of the earliest childhood memories I have at all, is of adding 3 to 5 by counting 5, 6, 7 and believing that you could get different results from adding numbers depending on exactly how you did it.  Which is cute, yes, and also indicates a kind of exploring, of probing, that was no doubt important in my starting to understand addition.  But you still look back and find it humorous, because now you're a big grownup and you know you can't do that.  My writing Python programs to try to find clever ways to fool myself by repeatedly checking the likelihood ratios was the same, in the sense that after I matured a bit more as a Bayesian, I realized that the feat I'd written those programs to try to do was *obviously* impossible.  In the same way that trying to find a clever way to break apart the 3 into 2 and 1, and trying to add them separately to 5, and then trying to add the 1 and then the 2, in hopes you can get to 7 or 9 instead of 8, is just never ever going to work.  The results in arithmetic are *theorems,* and it doesn't matter in what clever order you switch things up, you are never going to get anything except 8 when you carry out an operation that is validly equivalent to adding 3 plus 5.  The theorems of probability theory are also theorems.  If your Python program had actually worked, it would have produced a contradiction in probability theory, and thereby a contradiction in Peano Arithmetic, which provides a model for probability theory carried out using rational numbers.  The thing you tried to do is *exactly* as hard as adding 3 and 5 using the standard arithmetic axioms and getting 7.

**Undergrad:**  Uh, why?

**Scientist:**  Seconded.

**Bayesian:**  Because letting $e$ denote the evidence, $H$ denote the hypothesis, $\neg$ denote the negation of a proposition, $\mathbb P(X)$ denote the probability of proposition $X$, and $\mathbb P(X \mid Y)$ denote the [-1rj] of $X$ assuming $Y$ to be true, it is a theorem of probability that $$\mathbb P(H) = \left(P(H \mid e) \cdot P(e)\right) + \left(P(H\mid \neg e) \cdot P(\neg e)\right).$$ Therefore likelihood functions can *never* be p-hacked by *any possible* clever setup without you outright lying, because you can't have any possible procedure that a Bayesian knows in advance will make them update in a predictable net direction.  For every update that we expect to be produced by a piece of evidence $e,$ there's an equal and opposite update that we expect to probably occur from seeing $\neg e.$

**Undergrad:**  What?

**Scientist:**  Seconded.

**Bayesian:**  Look... let me try to zoom out a bit, and yes, look at the ongoing replication crisis.  The Scientist proclaimed suspicion of grand new sweeping ideals.  Okay, but the shift to likelihood functions is the kind of thing that *ought* to be able to solve a lot of problems at once.  Let's say... I'm trying to think of a good analogy here.  Let's say there's a corporation which is having a big crisis because their accountants are using floating-point numbers, only there's three different parts of the firm using three different representations of floating-point numbers to do numerically unstable calculations.  Somebody starts with 1.0 and adds 0.0001 a thousand times and then subtracts 0.1 and gets 0.999999999999989.  Or you can go to the other side of the building and use a different floating-point represenatation and get a different result.  And nobody has any conception that there's anything wrong with this.  Suppose there are BIG errors in the floating-point numbers, they're using the floating-point-number equivalent of crude ideograms and Roman numerals, you can get big pragmatic differences depending on what representation you use.  And naturally, people 'division-hack' to get whatever financial results they want.  So all the spreadsheets are failing to replicate, and people are starting to worry the 'cognitive priming' subdivision has actually been bankrupt for 20 years.  And then one day you come in and you say, "Hey.  Everyone.  Suppose that instead of these competing floating-point representations, we use my new representation instead.  It can't be fooled the same way, which will solve a surprising number of your problems."

*(The **Bayesian** now imitates the **Scientist's** voice:)*  "I'm suspicious," says the Senior Auditor.  "I suspect you of idealism.  In my experience, people need to use different floating-point representations for different financial problems, and it's good to have a lot of different numerical representations of fractions in your toolbox."

**Bayesian:**  "Well," I reply, "it may sound idealistic, but in point of fact, this thing I'm about to show you is *the* representation of fractions, in which you *cannot* get different results depending on which way you add things or what order you do the operations in.  It might be slightly more computationally expensive, but it is now no longer 1920 like when you first adopted the old system, and seriously, you can afford the computing power in a very large fraction of cases where you're only working with 30,000,000 bank accounts or some trivial number like that.  Yes, if you want to do something like take square roots, it gets a bit more complicated, but very few of you are actually taking the square root of bank account balances.  For the vast majority of things you are trying to do on a day-to-day basis, this system is unhackable without actually misreporting the numbers."  And then I show them how to represent arbitrary-magnitude finite integers precisely, and how to represent a rational number as the ratio of two integers.  What we would, nowadays, consider to be a direct, precise, computational representation of *the* system of rational numbers.  The one unique axiomatized mathematical system of rational numbers, to which floating-point numbers are a mere approximation.  And if you're just working with 30,000,000 bank account balances and your crude approximate floating-point numbers are *in practice* blowing up and failing to replicate and being exploited by people to get whatever results they want, and it is no longer 1920 and you can afford real computers now, it is an obvious step to have all the accountants switch to using *the* rational numbers.  Just as Bayesian updates are *the* rational updates, in the unique mathematical axiomatized system of probabilities.  And that's why you can't p-hack them.

**Scientist:**  That is a rather... audacious claim.  And I confess, even if everything you said about the math were true, I would still be skeptical of the pragmatics.  The current system of scientific statistics is something that's grown up over time and matured.  Has this bright Bayesian way actually been tried?

**Bayesian:**  It hasn't been tried very much in science.  In machine learning, where, uh, not to put too fine a point on it, we can actually see where the models are breaking because our AI doesn't work, it's been ten years since I've read a paper that tries to go at things from a frequentist angle and I can't *ever* recall seeing an AI algorithm calculate the p-value of anything.  If you're doing anything principled at all from a probability-theoretic stance, it's probably Bayesian, and pretty much never frequentist.  If you're classifying data using n-hot encodings, your loss function is the cross-entropy, not... I'm not even sure *what* the equivalent of trying to use 1920s-style p-values in AI would be like.  I would frankly attribute this to people in machine learning having to use statistical tools that visibly succeed or fail; rather than needing to get published by going through a particular traditional ritual of p-value reporting, and failure to replicate not being all that bad for your career.

**Scientist:**  So you're actually more of a computer science guy than an experimentalist yourself.  Why does this not surprise me?  It's not impossible that some better statistical system than p-values could exist, but I'd advise you to respect the wisdom of experience.  The fact that we know what p-hacking is, and are currently fighting it, is because we've had time to see where the edges of the system have problems, and we're figuring out how to fight those problems.  This shiny new system will also have problems; you just have no idea what they'll be.  Perhaps they'll be worse.

**Bayesian:**  It's not impossible that the accountants would figure out new shenanigans to pull with rational numbers, especially if they were doing some things computationally intensive enough that they could no longer afford to use *the* rational numbers and had to use some approximation instead.  But I stand by my statement that if your financial spreadsheets are *right now* blowing up in a giant replication crisis in ways that seem clearly linked to using p-values, and the p-values are, frankly, bloody ad-hoc inconsistent nonsense, an obvious first step is to *try* using the rational updates instead.  Although, it's possible we don't disagree too much in practice.  I'd also pragmatically favor trying to roll things out one step at a time, like, maybe just switch over the psychological sciences and see how that goes.

**Scientist:**  How would you persuade them to do that?

**Bayesian:**  I have no goddamn idea. Honestly, I'm not expecting anyone to actually fix anything.  People will just go on using p-values until the end of the world, probably.  It's just one more Nice Thing We Can't Have.  But there's a *chance* the idea will catch on.  I was pleasantly surprised when open access caught on as quickly as it did.  I was pleasantly surprised when people, like, actually noticed the replication crisis and it became a big issue that people cared about.  Maybe I'll be pleasantly surprised again and people will actually take up the crusade to bury the p-value at a crossroads at midnight and put a stake through its heart.  If so, I'll have done my part by making an understanding of [1lz Bayes' rule] and likelihoods [1zq more accessible] to everyone.

**Scientist:**  Or it could turn out that people don't *like* likelihoods, and that part of the wisdom of experience is the lesson that p-values are a kind of thing that experimentalists actually find useful and easy to use.

**Bayesian:**  If the experience of learning traditional statistics traumatized them so heavily that the thought of needing to learn a new system sends them screaming into the night, then yes, change might need to be imposed from outside.  I'm hoping though that the Undergrad will read a [1zq short, cheerful introduction to Bayesian probability], compare this with his ominous heavy traditional statistics textbook, and come back going "Please let me use likelihoods please let me use likelihoods oh god please let me use likelihoods."

**Undergrad:**  I'll guess I'll look into it and see?

**Bayesian:**  Weigh your decision carefully, Undergrad.  Some changes in science depend upon students growing up familiar with multiple ideas and choosing the right one.  Max Planck said so in a famous aphorism, so it must be true.  Ergo, the entire ability of science to distinguish good and bad ideas within that class must rest upon the cognitive capacities of undergrads.

**Scientist:**  Oh, now that is just--

**Moderator:**  And we're out of time.  Thanks for joining us, everyone!


C-Class

C-Class pages have substantial content and will often be useful to readers, but they may have significant prose and style problems, may require cleanup, may not cover all significant areas of the topic, or may not explain the concept in a way the target audience will reliably understand.

Contrast:

- [3rk Start-Class] pages, which are sufficiently unfinished that they should be viewed by editors rather than readers.
- [4yd B-Class] pages, which are reasonably comprehensive, have no major stylistic issues, and explain the concept in a way which seems likely to be often enlightening to the target audience.

**[4yg Quality scale]**

* [4ym]
* [4gs]
* [5xq]
* [72]
* [3rk]
* [4y7]
* [4yd]
* [4yf]
* [4yl]


Interpretations of "probability"

[summary:  What does it *mean* to say that a fair coin has a 50% probability of landing heads? There are three common views:

- __The propensity interpretation:__ Some probabilities are just out there in the world. It's a brute fact about coins that they come up heads half the time; we'll call this the coin's physical "propensity towards heads." When we say the coin has a 50% probability of being heads, we're talking directly about this propensity.
- __The frequentist interpretation:__  When we say the coin has a 50% probability of being heads after this flip, we mean that there's a class of events similar to this coin flip, and across that class, coins come up heads about half the time. That is, the _frequency_ of the coin coming up heads is 50% inside the event class (which might be "all other times this particular coin has been tossed" or "all times that a similar coin has been tossed" etc).
- __The subjective interpretation:__ Uncertainty is in the mind, not the environment. If I flip a coin and slap it against my wrist, it's already landed either heads or tails. The fact that I don't know whether it landed heads or tails is a fact about me, not a fact about the coin. The claim "I think this coin is heads with probability 50%" is an _expression of my own ignorance,_ which means that I'd bet at 1 : 1 odds (or better) that the coin came up heads.

When the event in question is deterministic, the propensity view is commonly considered an example of the [-4yk]. The frequentist view is self-consistent, but says that one-off events (such as the probability of it raining where you are _tomorrow in particular_) cannot be assigned "probabilities", because there are no objective frequencies to draw upon. The subjectivist view extends the frequentist view, and allows you to also consider probabilities (interpreted as betting odds) on one-off events. [1bv] provides tools for reasoning about subjective probabilities in a rigorous, consistent way.]

What does it *mean* to say that a flipped coin has a 50% probability of landing heads?

Historically, there are two popular types of answers to this question, the "[frequentist_probability frequentist]" and "[4vr subjective]" (aka "[1r8 Bayesian]") answers, which give rise to  [4xx radically different approaches to experimental statistics]. There is also a third "[propensity propensity]" viewpoint which is largely discredited (assuming the coin is deterministic). Roughly, the three approaches answer the above question as follows:

- __The propensity interpretation:__ Some probabilities are just out there in the world. It's a brute fact about coins that they come up heads half the time. When we flip a coin, it has a fundamental *propensity* of 0.5 for the coin to show heads.  When we say the coin has a 50% probability of being heads, we're talking directly about this propensity.
- __The frequentist interpretation:__  When we say the coin has a 50% probability of being heads after this flip, we mean that there's a class of events similar to this coin flip, and across that class, coins come up heads about half the time. That is, the _frequency_ of the coin coming up heads is 50% inside the event class, which might be "all other times this particular coin has been tossed" or "all times that a similar coin has been tossed", and so on.
- __The subjective interpretation:__ Uncertainty is in the mind, not the environment. If I flip a coin and slap it against my wrist, it's already landed either heads or tails. The fact that I don't know whether it landed heads or tails is a fact about me, not a fact about the coin. The claim "I think this coin is heads with probability 50%" is an _expression of my own ignorance,_ and 50% probability means that I'd bet at 1 : 1 odds (or better) that the coin came up heads.

For a visualization of the differences between these three viewpoints, see [4yj]. For examples of the difference, see [4yn]. See also the [http://plato.stanford.edu/entries/probability-interpret/ Stanford Encyclopedia of Philosophy article on interpretations of probability].

The propensity view is perhaps the most intuitive view, as for many people, it just feels like the coin is intrinsically random. However, this view is difficult to reconcile with the idea that once we've flipped the coin, it has already landed heads or tails. If the event in question is decided [determinism deterministically], the propensity view can be seen as an instance of the [-4yk]: When we mentally consider the coin flip, it feels 50% likely to be heads, so we find it very easy to imagine a *world* in which the coin is _fundamentally_ 50%-heads-ish. But that feeling is actually a fact about _us,_ not a fact about the coin; and the coin has no physical 0.5-heads-propensity hidden in there somewhere — it's just a coin.

The other two interpretations are both self-consistent, and give rise to pragmatically different statistical techniques, and there has been much debate as to which is preferable. The subjective interpretation is more generally applicable, as it allows one to assign probabilities (interpreted as betting odds) to one-off events.

# Frequentism vs subjectivism

As an example of the difference between frequentism and subjectivism, consider the question: "What is the probability that Hillary Clinton will win the 2016 US presidential election?", as analyzed in the summer of 2016.

A stereotypical (straw) frequentist would say, "The 2016 presidential election only happens once.  We can't *observe* a frequency with which Clinton wins presidential elections.  So we can't do any statistics or assign any probabilities here."

A stereotypical subjectivist would say:  "Well, prediction markets tend to be pretty [well_calibrated well-calibrated] about this sort of thing, in the sense that when prediction markets assign 20% probability to an event, it happens around 1 time in 5. And the prediction markets are currently betting on Hillary at about 3 : 1 odds. Thus, I'm comfortable saying she has about a 75% chance of winning. If someone offered me 20 : 1 odds _against_ Clinton — they get \$1 if she loses, I get \$20 if she wins — then I'd take the bet. I suppose you could refuse to take that bet on the grounds that you Just Can't Talk About Probabilities of One-off Events, but then you'd be pointlessly passing up a really good bet."

A stereotypical (non-straw) frequentist would reply:  "I'd take that bet too, of course.  But my taking that bet *is not based on rigorous epistemology,* and we shouldn't allow that sort of thinking in experimental science and other important venues. You can do subjective reasoning about probabilities when making bets, but we should exclude subjective reasoning in our scientific journals, and that's what frequentist statistics is designed for. Your paper should not conclude "and therefore, having observed thus-and-such data about carbon dioxide levels, I'd personally bet at 9 : 1 odds that anthropogenic global warming is real," because you can't build scientific consensus on opinions."

...and then it starts getting complicated. The subjectivist responds "First of all, I agree you shouldn't put posterior odds into papers, and second of all, it's not like your method is truly objective — the choice of "similar events" is arbitrary, abusable, and has given rise to [https://en.wikipedia.org/wiki/Data_dredging p-hacking] and the [https://en.wikipedia.org/wiki/Replication_crisis replication crisis]." The frequentists say "well your choice of prior is even more subjective, and I'd like to see you do better in an environment where peer pressure pushes people to abuse statistics and exaggerate their results," and then [4xx down the rabbit hole we go].

The subjectivist interpretation of probability is common among artificial intelligence researchers (who often design computer systems that manipulate subjective probability distributions), wall street traders (who need to be able to make bets even in relatively unique situations), and common intuition (where people feel like they can say there's a 30% chance of rain tomorrow without worrying about the fact that tomorrow only happens once). Nevertheless, the frequentist interpretation is commonly taught in introductory statistics classes, and is the gold standard for most scientific journals.

A common frequentist stance is that it is virtuous to have a large toolbox of statistical tools at your disposal. Subjectivist tools have their place in that toolbox, but they don't deserve any particular primacy (and they aren't generally accepted when it comes time to publish in a scientific journal).

An aggressive subjectivist stance is that frequentists have invented some interesting tools, and many of them are useful, but that refusing to consider subjective probabilities is toxic. Frequentist statistics were invented in a (failed) attempt to keep subjectivity out of science in a time before humanity really understood the laws of probability theory. Now we have [1lz theorems] about how to manage subjective probabilities correctly, and how to factor personal beliefs out from the objective evidence provided by the data, and if you ignore these theorems you'll get in trouble. The frequentist interpretation is broken, and that's why science has p-hacking and a replication crisis even as all the wall-street traders and AI scientists use the Bayesian interpretation. This "let's compromise and agree that everyone's viewpoint is valid" thing is all well and good, but how much worse do things need to get before we say "oops" and start acknowledging the subjective probability interpretation across all fields of science?

The most common stance among scientists and researchers is much more agnostic, along the lines of "use whatever statistical techniques work best at the time, and use frequentist techniques when publishing in journals because that's what everyone's been doing for decades upon decades upon decades, and that's what everyone's expecting."

See also [frequentist_probability], [4vr], and [4xx].

# Which interpretation is most useful?

Probably the subjective interpretation, because it subsumes the propensity and frequentist interpretations as special cases, while being more flexible than both.

When the frequentist "similar event" class is clear, the subjectivist can take those frequencies (often called [base_rate base rates] in this context) into account. But unlike the frequentist, she can also [1lz combine those base rates with other evidence that she's seen], and assign probabilities to one-off events, and make money in prediction markets and/or stock markets (when she knows something that the market doesn't).

When the laws of physics actually do "contain uncertainty", such as when they say that there are multiple different observations you might make next with differing likelihoods (as the [schrodinger_equation Schrodinger equation] often will), a subjectivist can combine her propensity-style uncertainty with her personal uncertainty in order to generate her aggregate subjective probabilities. But unlike a propensity theorist, she's not forced to think that _all_ uncertainty is physical uncertainty: She can act like a propensity theorist with respect to Schrodinger-equation-induced uncertainty, while still believing that her uncertainty about a coin that has already been flipped and slapped against her wrist is in her head, rather than in the coin.

This fully general stance is consistent with the belief that frequentist tools are useful for answering frequentist questions: The fact that you can _personally_ assign probabilities to one-off events (and, e.g., evaluate how good a certain trade is on a prediction market or a stock market) does not mean that tools labeled "Bayesian" are always better than tools labeled "frequentist". Whatever interpretation of "probability" you use, you're encouraged to use whatever statistical tool works best for you at any given time, regardless of what "camp" the tool comes from. Don't let the fact that you think it's possible to assign probabilities to one-off events prevent you from using useful frequentist tools!


B-Class

A B-Class page is fairly comprehensive, with good prose and structure; but needs further work to reach the A-Class requirement for engaging, professional, intuitive explanations, checked by asking for detailed feedback from both the target audience and [5ft reviewers].

B-Class pages are:

1. Comprehensive, covering all major areas of the topic without significant omissions or inaccuracies.
2. Fairly well-written and in line with the [-16t].
3. Reasonably well-organized, with similar concepts grouped together and reasonable section ordering.
4. Aimed at an audience, with a level of technicality, jargon, and notation appropriate for them.
5. Good for people wishing to learn the topic (well-explained, using examples, links, and visual aids as appropriate).
6. Well-[1kl summarized], in ways appropriate for the most likely audiences.

This tag should only be added by [5ft reviewers]. To request a review ask on [4ph] or [5g3 propose a page for B-Class].

**[4yg Quality scale]**

* [4ym]
* [4gs]
* [5xq]
* [72]
* [3rk]
* [4y7]
* [4yd]
* [4yf]
* [4yl]


A-Class

A-Class pages are comprehensive, intuitive, engaging, well-organized, and, where it would be useful, include excellent supporting material (e.g. examples, images, and exercises). You would enjoy learning from it, and send it to your friends. To reach A-Class pages must receive detailed feedback from both the target audience to make sure it's a good experience to learn from and [5ft reviewers] with a strong understanding of the topic.

For higher level audiences asking on [4ph Slack] is a good way to get feedback, for [1r3] or [1r5] sharing with your friends (in real life of via social media) is a better bet. To get the reviewers attention ask in #reviews on Slack.

A-Class pages are:

1. Comprehensive and authoritative, covering all areas of the topic without notable omissions or inaccuracies.
2. Written to an engaging, enjoyable, professional standard.
3. Well-organized, with useful overall and local structure.
4. Aimed at an audience, with a level of technicality, jargon, and notation appropriate for them.
5. Intuitively explained, making use of examples, visual aids, requisites, and anything else which aids reader comprehension.
6. Very well-[1kl summarized], in ways appropriate for all likely audiences.

This tag should only be added by [5ft reviewers].

**[4yg Quality scale]**

* [4ym]
* [4gs]
* [72]
* [3rk]
* [4y7]
* [4yd]
* [4yf]
* [4yl]


Correspondence visualizations for different interpretations of "probability"

[summary: Let's say you have a model which says a particular coin is 70% likely to be heads. How should we assess that model?

- According to the [propensity propensity] interpretation, the coin has a fundamental intrinsic comes-up-headsness property, and the model is correct if that property is set to 0.7. (This theory is widely considered discredited, and is an example of the [-4yk]).
- According to the [frequentist_probability frequentist] interpretation, the model is saying that there are a whole bunch of different places where some similar model is saying the same thing as this one (i.e., "the coin is 70% heads"), and all models in that reference class are true if, in 70% of those different places, the coin is heads.
- According to the [4vr subjectivist] interpretation, the model is saying that the one dang coin is 70 dang percent likely to be heads, and the model is either 70% accurate (if the coin is in fact heads) or 30% accurate (if it's tails).

In other words, the propensity and frequency interpretations try to find ways to say that the model is definitively "true" or "false" (one by postulating that uncertainty is an ontologically basic part of the world, the other by identifying a collection of similar events), whereas the subjective interpretation extends the notion of "correctness" to allow for shades of gray.]

[4y9 Recall] that there are three common interpretations of what it means to say that a coin has a 50% probability of landing heads:

- __The propensity interpretation:__ Some probabilities are just out there in the world. It's a brute fact about coins that they come up heads half the time; we'll call this the coin's physical "propensity towards heads." When we say the coin has a 50% probability of being heads, we're talking directly about this propensity.
- __The frequentist interpretation:__  When we say the coin has a 50% probability of being heads after this flip, we mean that there's a class of events similar to this coin flip, and across that class, coins come up heads about half the time. That is, the _frequency_ of the coin coming up heads is 50% inside the event class (which might be "all other times this particular coin has been tossed" or "all times that a similar coin has been tossed" etc).
- __The subjective interpretation:__ Uncertainty is in the mind, not the environment. If I flip a coin and slap it against my wrist, it's already landed either heads or tails. The fact that I don't know whether it landed heads or tails is a fact about me, not a fact about the coin. The claim "I think this coin is heads with probability 50%" is an _expression of my own ignorance,_ which means that I'd bet at 1 : 1 odds (or better) that the coin came up heads.

One way to visualize the difference between these approaches is by visualizing what they say about when a model of the world should count as a good model. If a person's model of the world is definite, then it's easy enough to tell whether or not their model is good or bad: We just check what it says against the facts.  For example, if a person's model of the world says "the tree is 3m tall", then this model is [correspondence_theory_of_truth correct] if (and only if) the tree is 3 meters tall.

![ordinary truth](http://i.imgur.com/5YriFTj.jpg)

Definite claims in the model are called "true" when they correspond to reality, and "false" when they don't. If you want to navigate using a map, you had better ensure that the lines drawn on the map correspond to the territory.

But how do you draw a correspondence between a map and a territory when the map is probabilistic? If your model says that a biased coin has a 70% chance of coming up heads, what's the correspondence between your model and reality? If the coin is actually heads, was the model's claim true? 70% true? What would that mean?

![probability truth?](http://i.imgur.com/EjAto4b.jpg)

The advocate of __propensity__ theory says that it's just a brute fact about the world that the world contains ontologically basic uncertainty. A model which says the coin is 70% likely to land heads is true if and only the actual physical propensity of the coin is 0.7 in favor of heads.

![propensity correspondence](http://i.imgur.com/0vQamhR.jpg)

This interpretation is useful when the laws of physics _do_ say that there are multiple different observations you may make next (with different likelihoods), as is sometimes the case (e.g., in quantum physics). However, when the event is deterministic — e.g., when it's a coin that has been tossed and slapped down and is already either heads or tails — then this view is largely regarded as foolish, and an example of the [-4yk]: The coin is just a coin, and has no special internal structure (nor special physical status) that makes it _fundamentally_ contain a little 0.7 somewhere inside it. It's already either heads or tails, and while it may _feel_ like the coin is fundamentally uncertain, that's a feature of your brain, not a feature of the coin.

How, then, should we draw a correspondence between a probabilistic map and a deterministic territory (in which the coin is already definitely either heads or tails?)

A __frequentist__ draws a correspondence between a single probability-statement in the model, and multiple events in reality. If the map says "that coin over there is 70% likely to be heads", and the actual territory contains 10 places where 10 maps say something similar, and in 7 of those 10 cases the coin is heads, then a frequentist says that the claim is true.

![frequentist correspondence](https://i.imgur.com/RaePEL7.png)

Thus, the frequentist preserves black-and-white correspondence: The model is either right or wrong, the 70% claim is either true or false. When the map says "That coin is 30% likely to be tails," that (according to a frequentist) means "look at all the cases similar to this case where my map says the coin is 30% likely to be tails; across all those places in the territory, 3/10ths of them have a tails-coin in them." That claim is definitive, given the set of "similar cases."

By contrast, a __subjectivist__ generalizes the idea of "correctness" to allow for shades of gray. They say, "My uncertainty about the coin is a fact about _me,_ not a fact about the coin; I don't need to point to other 'similar cases' in order to express uncertainty about _this_ case. I know that the world right in front of me is either a heads-world or a tails-world, and I have a [-probability_distribution] puts 70% probability on heads." They then draw a correspondence between their probability distribution and the world in front of them, and declare that the more probability their model assigns to the correct answer, the better their model is.

![bayesian correspondence](http://i.imgur.com/OWczeTe.jpg)

If the world _is_ a heads-world, and the probabilistic map assigned 70% probability to "heads," then the subjectivist calls that map "70% accurate." If, across all cases where their map says something has 70% probability, the territory is actually that  way 7/10ths of the time, then the Bayesian calls the map "[-well_calibrated]". They then seek methods to make their maps more accurate, and better calibrated. They don't see a need to interpret probabilistic maps as making definitive claims; they're happy to interpret them as making estimations that can be graded on a sliding scale of accuracy.

## Debate

In short, the frequentist interpretation tries to find a way to say the model is definitively "true" or "false" (by identifying a collection of similar events), whereas the subjectivist interpretation extends the notion of "correctness" to allow for shades of gray.

Frequentists sometimes object to the subjectivist interpretation, saying that frequentist correspondence is the only type that has any hope of being truly objective. Under Bayesian correspondence, who can say whether the map should say 70% or 75%, given that the probabilistic claim is not objectively true or false either way? They claim that these subjective assessments of "partial accuracy" may be intuitively satisfying, but they have no place in science. Scientific reports ought to be restricted to frequentist statements, which are definitively either true or false, in order to increase the objectivity of science.

Subjectivists reply that the frequentist approach is hardly objective, as it depends entirely on the choice of "similar cases". In practice, people can (and do!) [https://en.wikipedia.org/wiki/Data_dredging abuse frequentist statistics] by choosing the class of similar cases that makes their result look as impressive as possible (a technique known as "p-hacking"). Furthermore, the manipulation of subjective probabilities is subject to the [1lz iron laws] of probability theory (which are the [ only way to avoid inconsistencies and pathologies] when managing your uncertainty about the world), so it's not like subjective probabilities are the wild west or something. Also, science has things to say about situations even when there isn't a huge class of objective frequencies we can observe, and science should let us collect and analyze evidence even then.

For more on this debate, see [4xx].


Mind projection fallacy

[summary:  One commits the mind projection fallacy when they postulate that features of how their model of the world works are actually features of the world.

Suppose you flip a coin, slap it against your wrist, and don't look at the result.  Does it make sense to say that the probability of the coin being heads is 50%?  How can this be true, when the coin has already landed, and is either definitely heads or definitely tails? One who says "the coin is fundamentally uncertain; it is a feature of the coin that it is always 50% likely to be heads" commits the mind projection fallacy. Uncertainty is in the mind, not in reality. It makes sense that brains have an internal measure of how uncertain they are about the world, but that uncertainty is not a fact about the coin, it's a fact about the uncertain person. The coin itself is not sure or unsure.]

The "mind projection fallacy" occurs when somebody expects an overly direct resemblance between the intuitive language of the mind, and the language of physical reality.

Consider the [map_territory map and territory] metaphor, in which the world is a like a territory and your mental model of the world is like a map of that territory. In this metaphor, the  mind projection fallacy is analogous to thinking that the territory can be folded up and put into your pocket.

As an archetypal example:  Suppose you flip a coin, slap it against your wrist, and don't yet look at it.  Does it make sense to say that the probability of the coin being heads is 50%?  How can this be true, when the coin itself is already either definitely heads or definitely tails?

One who says "the coin is fundamentally uncertain; it is a feature of the coin that it is always 50% likely to be heads" commits the mind projection fallacy. Uncertainty is in the mind, not in reality.  If you're ignorant about a coin, that's not a fact about the coin, it's a fact about you. It makes sense that your brain, the map, has an internal measure of how it's more or less sure of something.  But that doesn't mean the coin itself has to contain a corresponding quantity of increased or decreased sureness; it is just heads or tails.

The [-ontology] of a system is the elementary or basic components of that system. The ontology of your model of the world may include intuitive measures of uncertainty that it can use to represent the state of the coin, used as primitives like [float floating-point numbers] are primitive in computers. The mind projection fallacy occurs whenever someone reasons as if the territory, the physical universe and its laws, must have the same sort of ontology as the map, our models of reality.

See also:

- [4vr]
- The LessWrong sequence on [Reductionism](https://wiki.lesswrong.com/wiki/Reductionism_%28sequence%29), especially:
 - [How an algorithm feels from the inside](http://lesswrong.com/lw/no/how_an_algorithm_feels_from_inside/)
 - [The Mind Projection Fallacy](http://lesswrong.com/lw/oi/mind_projection_fallacy/)
 - [Probability is in the mind](http://lesswrong.com/lw/oj/probability_is_in_the_mind/)


Probability interpretations: Examples

[summary: Consider evaluating, in June of 2016, the question:  "What is the probability of Hillary Clinton winning the 2016 US presidential election?"

- On the **propensity** view, Hillary has some fundamental chance of winning the election.  To ask about the probability is to ask about this objective chance.
- On the **subjective** view, saying that Hillary has an 80% chance of winning the election summarizes our *knowledge about* the election, or, equivalently, our *state of uncertainty* given what we currently know.
- On the **frequentist** view, we cannot formally or rigorously say anything about the 2016 presidential election, because it only happens once.]

## Betting on one-time events

Consider evaluating, in June of 2016, the question:  "What is the probability of Hillary Clinton winning the 2016 US presidential election?"

On the **propensity** view, Hillary has some fundamental chance of winning the election.  To ask about the probability is to ask about this objective chance.  If we see a prediction market in which prices move after each new poll — so that it says 60% one day, and 80% a week later — then clearly the prediction market isn't giving us very strong information about this objective chance, since it doesn't seem very likely that Clinton's *real* chance of winning is swinging so rapidly.

On the **frequentist** view, we cannot formally or rigorously say anything about the 2016 presidential election, because it only happens once.  We can't *observe* a frequency with which Clinton wins presidential elections.  A frequentist might concede that they would cheerfully buy for \$1 a ticket that pays \$20 if Clinton wins, considering this a favorable bet in an *informal* sense, while insisting that this sort of reasoning isn't sufficiently rigorous, and therefore isn't suitable for being included in science journals.

On the **subjective** view, saying that Hillary has an 80% chance of winning the election summarizes our *knowledge about* the election or our *state of uncertainty* given what we currently know.  It makes sense for the prediction market prices to change in response to new polls, because our current state of knowledge is changing.

## A coin with an unknown bias

Suppose we have a coin, weighted so that it lands heads somewhere between 0% and 100% of the time, but we don't know the coin's actual bias.

The coin is then flipped three times where we can see it.  It comes up heads twice, and tails once:  HHT.

The coin is then flipped again, where nobody can see it yet.  An honest and trustworthy experimenter lets you spin a wheel-of-gambling-odds,%note:The reason for spinning the wheel-of-gambling-odds is to reduce the worry that the experimenter might know more about the coin than you, and be offering you a deliberately rigged bet.% and the wheel lands on (2 : 1).  The experimenter asks if you'd enter into a gamble where you win \$2 if the unseen coin flip is tails, and pay \$1 if the unseen coin flip is heads.

On a **propensity** view, the coin has some objective probability between 0 and 1 of being heads, but we just don't know what this probability is.  Seeing HHT tells us that the coin isn't all-heads or all-tails, but we're still just guessing — we don't really know the answer, and can't say whether the bet is a fair bet.

On a **frequentist** view, the coin would (if flipped repeatedly) produce some long-run frequency $f$ of heads that is between 0 and 1.  If we kept flipping the coin long enough, the actual proportion $p$ of observed heads is guaranteed to approach $f$ arbitrarily closely, eventually.  We can't say that the *next* coin flip is guaranteed to be H or T, but we can make an objectively true statement that $p$ will approach $f$ to within epsilon if we continue to flip the coin long enough.

To decide whether or not to take the bet, a frequentist might try to apply an [unbiased_estimator unbiased estimator] to the data we have so far.  An "unbiased estimator" is a rule for taking an observation and producing an estimate $e$ of $f$, such that the [4b5 expected value] of $e$ is $f$.  In other words, a frequentist wants a rule such that, if the hidden bias of the coin was in fact to yield 75% heads, and we repeat many times the operation of flipping the coin a few times and then asking a new frequentist to estimate the coin's bias using this rule, the *average* value of the estimated bias will be 0.75.  This is a property of the _estimation rule_ which is objective.  We can't hope for a rule that will always, in any particular case, yield the true $f$ from just a few coin flips; but we can have a rule which will provably have an *average* estimate of $f$, if the experiment is repeated many times.

In this case, a simple unbiased estimator is to guess that the coin's bias $f$ is equal to the observed proportion of heads, or 2/3. In other words, if we repeat this experiment many many times, and whenever we see $p$ heads in 3 tosses we guess that the coin's bias is $\frac{p}{3}$, then this rule definitely is an unbiased estimator. This estimator says that a bet of \$2 vs. $\1 is fair, meaning that it doesn't yield an expected profit, so we have no reason to take the bet.

On a **subjectivist** view, we start out personally unsure of where the bias $f$ lies within the interval [0, 1].  Unless we have any knowledge or suspicion leading us to think otherwise, the coin is just as likely to have a bias between 33% and 34%, as to have a bias between 66% and 67%; there's no reason to think it's more likely to be in one range or the other.

Each coin flip we see is then [22x evidence] about the value of $f,$ since a flip H happens with different probabilities depending on the different values of $f,$ and we update our beliefs about $f$ using [1zj Bayes' rule]. For example, H is twice as likely if $f=\frac{2}{3}$ than if $f=\frac{1}{3}$ so by [1zm Bayes's Rule] we should now think $f$ is twice as likely to lie near $\frac{2}{3}$ as it is to lie near $\frac{1}{3}$.

When we start with a uniform [219 prior], observe multiple flips of a coin with an unknown bias, see M heads and N tails, and then try to estimate the odds of the next flip coming up heads, the result is [21c Laplace's Rule of Succession] which estimates (M + 1) : (N + 1) for a probability of $\frac{M + 1}{M + N + 2}.$

In this case, after observing HHT, we estimate odds of 2 : 3 for tails vs. heads on the next flip.  This makes a gamble that wins \$2 on tails and loses \$1 on heads a profitable gamble in expectation, so we take the bet.

Our choice of a [219 uniform prior] over $f$ was a little dubious — it's the obvious way to express total ignorance about the bias of the coin, but obviousness isn't everything. (For example, maybe we actually believe that a fair coin is more likely than a coin biased 50.0000023% towards heads.) However, all the reasoning after the choice of prior was rigorous according to the laws of [1bv probability theory], which is the [probability_coherence_theorems only method of manipulating quantified uncertainty] that obeys obvious-seeming rules about how subjective uncertainty should behave.

## Probability that the 98,765th decimal digit of $\pi$ is $0$.

What is the probability that the 98,765th digit in the decimal expansion of $\pi$ is $0$?

The **propensity** and **frequentist** views regard as nonsense the notion that we could talk about the *probability* of a mathematical fact.  Either the 98,765th decimal digit of $\pi$ is $0$ or it's not.  If we're running *repeated* experiments with a random number generator, and looking at different digits of $\pi,$ then it might make sense to say that the random number generator has a 10% probability of picking numbers whose corresponding decimal digit of $\pi$ is $0$.  But if we're just picking a non-random number like 98,765, there's no sense in which we could say that the 98,765th digit of $\pi$ has a 10% propensity to be $0$, or that this digit is $0$ with 10% frequency in the long run.

The **subjectivist** considers probabilities to just refer to their own uncertainty.  So if a subjectivist has picked the number 98,765 without yet knowing the corresponding digit of $\pi,$ and hasn't made any observation that is known to them to be entangled with the 98,765th digit of $\pi,$ and they're pretty sure their friend hasn't yet looked up the 98,765th digit of $\pi$ either, and their friend offers a whimsical gamble that costs \$1 if the digit is non-zero and pays \$20 if the digit is zero, the Bayesian takes the bet.

Note that this demonstrates a difference between the subjectivist interpretation of "probability" and Bayesian probability theory. A perfect Bayesian reasoner that knows the rules of logic and the definition of $\pi$ must, by the axioms of probability theory, assign probability either 0 or 1 to the claim "the 98,765th digit of $\pi$ is a $0$" (depending on whether or not it is). This is one of the reasons why [bayes_intractable perfect Bayesian reasoning is intractable]. A subjectivist that is not a perfect Bayesian nevertheless claims that they are personally uncertain about the value of the 98,765th digit of $\pi.$ Formalizing the rules of subjective probabilities about mathematical facts (in the way that [-1bv] formalized the rules for manipulating subjective probabilities about empirical facts, such as which way a coin came up) is an open problem; this in known as the problem of [-logical_uncertainty].