Log-Linear Models

I've spent most of my research career trying to build big, complex nonparametric models; however, I've more recently delved into the realm of natural language processing, where how awesome your model looks on paper is irrelevant compared to how well it models your data. In the spirit of this new work (and to lay the groundwork for a later post on NLP), I'd like to go over a family of models that I think is often overlooked due to not being terribly sexy (or at least, I overlooked it for a good while). This family is the family of log-linear models, which are models of the form:

where  maps a data point to a feature vector; they are called log-linear because the log of the probability is a linear function of . We refer to  as the score of .
This model class might look fairly restricted at first, but the real magic comes in with the feature vector . In fact, every probabilistic model that is absolutely continuous with respect to Lebesgue measure can be represented as a log-linear model for sufficient choices of  and . This is actually trivially true, as we can just take  to be  and  to be .
You might object to this choice of , since it maps into  rather than , and feature vectors are typically discrete. However, we can do just as well by letting , where the th coordinate of  is the th digit in the binary representation of , then let  be the vector .
It is important to distinguish between the ability to represent an arbitrary model as log-linear and the ability to represent an arbitrary family of models as a log-linear family (that is, as the set of models we get if we fix a choice of features  and then vary ). When we don't know the correct model in advance and want to learn it, this latter consideration can be crucial. Below, I give two examples of model families and discuss how they fit (or do not fit) into the log-linear framework. Important caveat: in both of the models below, it is typically the case that at least some of the variables involved are unobserved. However, we will ignore this for now, and assume that, at least at training time, all of the variables are fully observed (in other words, we can see  and  in the hidden Markov model and we can see the full tree of productions in the probabilistic context free grammar).
Hidden Markov Models. A hidden Markov model, or HMM, is a model with latent (unobserved) variables  together with observed variables . The distribution for  depends only on , and the distribution for  depends only on  (in the sense that  is conditionally independent of  given ). We can thus summarize the information in an HMM with the distributions  and .
We can express a hidden Markov model as a log-linear model by defining two classes of features: (i) features  that count the number of  such that  and ; and (ii) features  that count the number of  such that  and . While this choice of features yields a model family capable of expressing an arbitrary hidden Markov model, it is also capable of learning models that are not hidden Markov models. In particular, we would like to think of  (the index of  corresponding to ) as , but there is no constraint that  for each , whereas we do necessarily have  for each . If  is fixed, we still do obtain an HMM for any setting of , although  will have no simple relationship with . Furthermore, the relationship depends on , and will therefore not work if we care about multiple Markov chains with different lengths.
Is the ability to express models that are not HMMs good or bad? It depends. If we know for certain that our data satisfy the HMM assumption, then expanding our model family to include models that violate that assumption can only end up hurting us. If the data do not satisfy the HMM assumption, then increasing the size of the model family may allow us to overcome what would otherwise be a model mis-specification. I personally would prefer to have as much control as possible about what assumptions I make, so I tend to see the over-expressivity of HMMs as a bug rather than a feature.
Probabilistic Context Free Grammars. A probabilistic context free grammar, or PCFG, is simply a context free grammar where we place a probability distribution over the production rules for each non-terminal. For those unfamiliar with context free grammars, a context free grammar is specified by:

A set  of non-terminal symbols, including a distinguished initial symbol .
A set  of terminal symbols.
For each , one or more production rules of the form , where  and .

For instance, a context free grammar for arithmetic expressions might have , , and the following production rules:

 for all

The language corresponding to a context free grammar is the set of all strings that can be obtained by starting from  and applying production rules until we only have terminal symbols. The language corresponding to the above grammar is, in fact, the set of well-formed arithmetic expressions, such as , , and .
As mentioned above, a probabilistic context free grammar simply places a distribution over the production rules for any given non-terminal symbol. By repeatedly sampling from these distributions until we are left with only terminal symbols, we obtain a probability distribution over the language of the grammar.
We can represent a PCFG as a log-linear model by using a feature  for each production rule . For instance, we have a feature that counts the number of times that the rule  gets applied, and another feature that counts the number of times that  gets applied. Such features yield a log-linear model family that contains all probabilistic context free grammars for a given (deterministic) context free grammar. However, it also contains additional models that do not correspond to PCFGs; this is because we run into the same problem as for HMMs, which is that the sum of  over production rules of a given non-terminal does not necessarily add up to . In fact, the problem is even worse here. For instance, suppose that  in the model above. Then the expression  gets a score of , and longer chains of s get even higher scores. In particular, there is an infinite sequence of expressions with increasing scores and therefore the model doesn't normalize (since the sum of the exponentiated scores of all possible productions is infinite).
So, log-linear models over-represent PCFGs in the same way as they over-represent HMMs, but the problems are even worse than before. Let's ignore these issues for now, and suppose that we want to learn PCFGs with an unknown underlying CFG. To be a bit more concrete, suppose that we have a large collection of possible production rules for each non-terminal , and we think that a small but unknown subset of those production rules should actually appear in the grammar. Then there is no way to encode this directly within the context of a log-linear model family, although we can encode such "sparsity constraints" using simple extensions to log-linear models (for instance, by adding a penalty for the number of non-zero entries in ). So, we have found another way in which the log-linear representation is not entirely adequate.
Conclusion. Based on the examples above, we have seen that log-linear models have difficulty placing constraints on latent variables. This showed up in two different ways: first, we are unable to constrain subsets of variables to add up to  (what I call "local normalization" constraints); second, we are unable to encode sparsity constraints within the model. In both of these cases, it is possible to extend the log-linear framework to address these sorts of constraints, although that is outside the scope of this post.
 Parameter Estimation for Log-Linear Models
I've explained what a log-linear model is, and partially characterized its representational power. I will now answer the practical question of how to estimate the parameters of a log-linear model (i.e., how to fit  based on observed data). Recall that a log-linear model places a distribution over a space  by choosing  and  and defining

More precisely (assuming  is a discrete space), we have

Given observations , which we assume to be independent given , our goal is to choose  maximizing , or, equivalently, . In equations, we want

We typically use gradient methods (such as gradient descent, stochastic gradient descent, or L-BFGS) to minimize the right-hand side of (1). If we compute the gradient of (1) then we get:

We can re-write (2) in the following more compact form:

In other words, the contribution of each training example  to the gradient is the extent to which the features values for  exceed their expected values conditioned on .
One important consideration for such gradient-based numerical optimizers is convexity. If the objective function we are trying to minimize is convex (or concave), then gradient methods are guaranteed to converge to the global optimum. If the objective function is non-convex, then a gradient-based approach (or any other type of local search) may converge to a local optimum that is very far from the global optimum. In order to assess convexity, we compute the Hessian (matrix of second derivatives) and check whether it is positive definite. (In this case, we actually care about concavity, so we want the Hessian to be negative definite.) We can compute the Hessian by differentiating (2), which gives us

Again, we can re-write this more compactly as

The term inside the parentheses of (5) is exactly the negative of the covariance matrix of  given , and is therefore necessarily negative definite, so the objective function we are trying to minimize is indeed concave, which, as noted before, implies that our gradient methods will always reach the global optimum.
 Regularization and Concavity
We may in practice wish to encode additional prior knowledge about  in our model, especially if the dimensionality of  is large relative to the amount of data we have. Can we do this and still maintain concavity? The answer in many cases is yes: since the -norm is convex for all , we can add an  penalty to the objective for any such  and still have a concave objective function.
 Conclusion
Log-linear models provide a universal representation for individual probability distributions, but not for arbitrary families of probability distributions (for instance, due to the inability to capture local normalization constraints or sparsity constraints). However, for the families they do express, parameter optimization can be performed efficiently due to a likelihood function that is log-concave in its parameters. Log-linear models also have tie-ins to many other beautiful areas of statistics, such as exponential families, which will be the subject of the next post.

Log-Linear Models


Beyond Bayesians and Frequentists

(This is available in pdf form here.)
If you are a newly initiated student into the field of machine learning, it won't be long before you start hearing the words "Bayesian" and "frequentist" thrown around. Many people around you probably have strong opinions on which is the "right" way to do statistics, and within a year you've probably developed your own strong opinions (which are suspiciously similar to those of the people around you, despite there being a much greater variance of opinion between different labs). In fact, now that the year is 2012 the majority of new graduate students are being raised as Bayesians (at least in the U.S.) with frequentists thought of as stodgy emeritus professors stuck in their ways.
If you are like me, the preceding set of facts will make you very uneasy. They will make you uneasy because simple pattern-matching — the strength of people's opinions, the reliability with which these opinions split along age boundaries and lab boundaries, and the ridicule that each side levels at the other camp – makes the "Bayesians vs. frequentists" debate look far more like politics than like scholarly discourse. Of course, that alone does not necessarily prove anything; these disconcerting similarities could just be coincidences that I happened to cherry-pick.
My next point, then, is that we are right to be uneasy, because such debate makes us less likely to evaluate the strengths and weaknesses of both approaches in good faith. This essay is a push against that — I summarize the justifications for Bayesian methods and where they fall short, show how frequentist approaches can fill in some of their shortcomings, and then present my personal (though probably woefully under-informed) guidelines for choosing which type of approach to use.
Before doing any of this, though, a bit of background is in order…
1. Background on Bayesians and Frequentists
 1.1. Three Levels of Argument
As Andrew Critch [6] insightfully points out, the Bayesians vs. frequentists debate is really three debates at once, centering around one or more of the following arguments:

Whether to interpret subjective beliefs as probabilities
Whether to interpret probabilities as subjective beliefs (as opposed to asymptotic frequencies)
Whether a Bayesian or frequentist algorithm is better suited to solving a particular problem.

Given my own research interests, I will add a fourth argument:
4. Whether Bayesian or frequentist techniques are better suited to engineering an artificial intelligence.
Andrew Gelman [9] has his own well-written essay on the subject, where he expands on these distinctions and presents his own more nuanced view.
Why are these arguments so commonly conflated? I'm not entirely sure; I would guess it is for historical reasons but I have so far been unable to find said historical reasons. Whatever the reasons, what this boils down to in the present day is that people often form opinions on 1. and 2., which then influence their answers to 3. and 4. This is not good, since 1. and 2. are philosophical in nature and difficult to resolve correctly, whereas 3. and 4. are often much easier to resolve and extremely important to resolve correctly in practice. Let me re-iterate: the Bayes vs. frequentist discussion should center on the practical employment of the two methods, or, if epistemology must be discussed, it should be clearly separated from the day-to-day practical decisions. Aside from the difficulties with correctly deciding epistemology, the relationship between generic epistemology and specific practices in cutting-edge statistical research is only via a long causal chain, and it should be completely unsurprising if Bayesian epistemology leads to the employment of frequentist tools or vice versa.
For this reason and for reasons of space, I will spend the remainder of the essay focusing on statistical algorithms rather than on interpretations of probability. For those who really want to discuss interpretations of probability, I will address that in a later essay.
 1.2. Recap of Bayesian Decision Theory
(What follows will be review for many.) In Bayesian decision theory, we assume that there is some underlying world state  and a likelihood function  over possible observations. (A likelihood function is just a conditional probability distribution where the parameter conditioned on can vary.) We also have a space  of possible actions and a utility function  that gives the utility of performing action  if the underlying world state is . We can incorporate notions like planning and value of information by defining  recursively in terms of an identical agent to ourselves who has seen one additional observation (or, if we are planning against an adversary, in terms of the adversary). For a more detailed overview of this material, see the tutorial by North [11].
What distinguishes the Bayesian approach in particular is one additional assumption, a prior distribution  over possible world states. To make a decision with respect to a given prior, we compute the posterior distribution  using Bayes' theorem, then take the action  that maximizes .
In practice,  can be quite difficult to compute, and so we often attempt to approximate it. Such attempts are known as approximate inference algorithms.
 1.3. Steel-manning Frequentists
There are many different ideas that fall under the broad umbrella of frequentist techniques. While it would be impossible to adequately summarize all of them even if I attempted to, there are three in particular that I would like to describe, and which I will call frequentist decision theory, frequentist guarantees, and frequentist analysis tools.
Frequentist decision theory has a very similar setup to Bayesian decision theory, with a few key differences. These are discussed in detail and contrasted with Bayesian decision theory in [10], although we summarize the differences here. There is still a likelihood function  and a utility function . However, we do not assume the existence of a prior on , and instead choose the decision rule  that maximizes

In other words, we ask for a worst case guarantee rather than an average case guarantee. As an example of how these would differ, imagine a scenario where we have no data to observe, an unknown , and we choose an action . Furthermore,  for all ,  if , and  if  and . Then a frequentist will always choose  because any other action gets  utility in the worst case; a Bayesian, on the other hand, will happily choose any non-zero value of  since such an action gains  utility in expectation. (I am purposely ignoring more complex ideas like mixed strategies for the purpose of illustration.).
Note that the frequentist optimization problem is more complicated than in the Bayesian case, since the value of (1) depends on the joint behavior of , whereas with Bayes we can optimize  for each set of observations separately.
As a result of this more complex optimization problem, it is often not actually possible to maximize (1), so many frequentist techniques instead develop tools to lower-bound (1) for a given decision procedure, and then try to construct a decision procedure that is reasonably close to the optimum. Support vector machines [2], which try to pick separating hyperplanes that minimize generalization error, are one example of this where the algorithm is explicitly trying to maximize worst-case utility. Another example of a frequentist decision procedure is L1-regularized least squares for sparse recovery [3], where the procedure itself does not look like it is explicitly maximizing any utility function, but a separate analysis shows that it is close to the optimal procedure anyways.
The second sort of frequentist approach to statistics is what I call a frequentist guarantee. A frequentist guarantee on an algorithm is a guarantee that, with high probability with respect to how the data was generated, the output of the algorithm will satisfy a given property. The most familiar example of this is any algorithm that generates a frequentist confidence interval: to generate a 95% frequentist confidence interval for a parameter  is to run an algorithm that outputs an interval, such that with probability at least 95%  lies within the interval. An important fact about most such algorithms is that the size of the interval only grows logarithmically with the amount of confidence we require, so getting a 99.9999% confidence interval is only slightly harder than getting a 95% confidence interval (and we should probably be asking for the former whenever possible).
If we use such algorithms to test hypotheses or to test discrete properties of , then we can obtain algorithms that take in probabilistically generated data and produce an output that with high probability depends only on how the data was generated, not on the specific random samples that were given. For instance, we can create an algorithm that takes in samples from two distributions, and is guaranteed to output 1 whenever they are the same, 0 whenever they differ by at least  in total variational distance, and could have arbitrary output if they are different but the total variational distance is less than . This is an amazing property — it takes in random input and produces an essentially deterministic answer.
Finally, a third type of frequentist approach seeks to construct analysis tools for understanding the behavior of random variables. Metric entropy, the Chernoff and Azuma-Hoeffding bounds [12], and Doob's optional stopping theorem are representative examples of this sort of approach. Arguably, everyone with the time to spare should master these techniques, since being able to analyze random variables is important no matter what approach to statistics you take. Indeed, frequentist analysis tools have no conflict at all with Bayesian methods — they simply provide techniques for understanding the behavior of the Bayesian model.
2. Bayes vs. Other Methods
 2.1. Justification for Bayes
We presented Bayesian decision theory above, but are there any reasons why we should actually use it? One commonly-given reason is that Bayesian statistics is merely the application of Bayes' Theorem, which, being a theorem, describes the only correct way to update beliefs in response to new evidence; anything else can only be justified to the extent that it provides a good approximation to Bayesian updating. This may be true, but Bayes' Theorem only applies if we already have a prior, and if we accept probability as the correct framework for expressing uncertain beliefs. We might want to avoid one or both of these assumptions. Bayes' theorem also doesn't explain why we care about expected utility as opposed to some other statistic of the distribution over utilities (although note that frequentist decision theory also tries to maximize expected utility).
One compelling answer to this is Cox's Theorem, which shows that any agent must implicitly be using a probability model to make decisions, or else they can be dutch-booked — meaning there is a series of bets that they would be willing to make that causes them to lose money with certainty. Another answer is the complete class theorem, which shows that any non-Bayesian decision procedure is strictly dominated by a Bayesian decision procedure — meaning that the Bayesian procedure performs at least as well as the non-Bayesian procedure in all cases with certainty. In other words, if you are doing anything non-Bayesian, then either it is secretly a Bayesian procedure or there is another procedure that does strictly better than it. Finally, the VNM Utility Theorem states that any agent with consistent preferences over distributions of outcomes must be implicitly maximizing the expected value of some scalar-valued function, which we can then use as our choice of utility function . These theorems, however, ignore the issue of computation — while the best decision procedure may be Bayesian, the best computationally-efficient decision procedure could easily be non-Bayesian.
Another justification for Bayes is that, in contrast to ad hoc frequentist techniques, it actually provides a general theory for constructing statistical algorithms, as well as for incorporating side information such as expert knowledge. Indeed, when trying to model complex and highly structured situations it is difficult to obtain any sort of frequentist guarantees (although analysis tools can still often be applied to gain intuition about parts of the model). A prior lets us write down the sorts of models that would allow us to capture structured situations (for instance, when trying to do language modeling or transfer learning). Non-Bayesian methods exist for these situations, but they are often ad hoc and in many cases ends up looking like an approximation to Bayes. One example of this is Kneser-Ney smoothing for n-gram models, an ad hoc algorithm that ended up being very similar to an approximate inference algorithm for the hierarchical Pitman-Yor process [15, 14, 17, 8]. This raises another important point against Bayes, which is that the proper Bayesian interpretation may be very mathematically complex. Pitman-Yor processes are on the cutting-edge of Bayesian nonparametric statistics, which is itself one of the more technical subfields of statistical machine learning, so it was probably much easier to come up with Kneser-Ney smoothing than to find the interpretation in terms of Pitman-Yor processes.
 2.2. When the Justifications Fail
The first and most common objection to Bayes is that a Bayesian method is only as good as its prior. While for simple models the performance of Bayes is relatively independent of the prior, such models can only capture data where frequentist techniques would also perform very well. For more complex (especially nonparametric) Bayesian models, the performance can depend strongly on the prior, and designing good priors is still an open problem. As one example I point to my own research on hierarchical nonparametric models, where the most straightforward attempts to build a hierarchical model lead to severe pathologies [13].
Even if a Bayesian model does have a good prior, it may be computationally intractable to perform posterior inference. For instance, structure learning in Bayesian networks is NP-hard [4], as is topic inference in the popular latent Dirichlet allocation model (and this continues to hold even if we only want to perform approximate inference). Similar stories probably hold for other common models, although a theoretical survey has yet to be made; suffice to say that in practice approximate inference remains a difficult and unsolved problem, with many models not even considered because of the apparent hopelessness of performing inference in them.
Because frequentist methods often come with an analysis of the specific algorithm being employed, they can sometimes overcome these computational issues. One example of this mentioned already is L1 regularized least squares [3]. The problem setup is that we have a linear regression task  where  and  are known,  is a noise vector, and  is believed to be sparse (typically  has many more rows than , so without the sparsity assumption  would be underdetermined). Let us suppose that  has  rows and  non-zero rows — then the number of possible sparsity patterns is  — large enough that a brute force consideration of all possible sparsity patterns is intractable. However, we can show that solving a certain semidefinite program will with high probability yield the appropriate sparsity pattern, after which recovering x reduces to a simple least squares problem. (A semidefinite program is a certain type of optimization problem that can be solved efficiently [16].)
Finally, Bayes has no good way of dealing with adversaries or with cases where the data was generated in a complicated way that could make it highly biased (for instance, as the output of an optimization procedure). A toy example of an adversary would be playing rock-paper-scissors — how should a Bayesian play such a game? The straightforward answer is to build up a model of the opponent based on their plays so far, and then to make the play that maximizes the expected score (probability of winning minus probability of losing). However, such a strategy fares poorly against any opponent with access to the model being used, as they can then just run the model themselves to predict the Bayesian's plays in advance, thereby winning every single time. In contrast, there is a frequentist strategy called the multiplicative weights update method that fairs well against an arbitrary opponent (even one with superior computational resources and access to our agent's source code). The multiplicative weights method does far more than winning at rock-paper-scissors — it is also a key component of the fastest algorithm for solving many important optimization problems (including the network flow algorithm), and it forms the theoretical basis for the widely used AdaBoost algorithm [1, 5, 7].
 2.3. When To Use Each Method
The essential difference between Bayesian and frequentist decision theory is that Bayes makes the additional assumption of a prior over , and optimizes for average-case performance rather than worst-case performance. It follows, then, that Bayes is the superior method whenever we can obtain a good prior and when good average-case performance is sufficient. However, if we have no way of obtaining a good prior, or when we need guaranteed performance, frequentist methods are the way to go. For instance, if we are trying to build a software package that should be widely deployable, we might want to use a frequentist method because users can be sure that the software will work as long as some number of easily-checkable assumptions are met.
A nice middle-ground between purely Bayesian and purely frequentist methods is to use a Bayesian model coupled with frequentist model-checking techniques; this gives us the freedom in modeling afforded by a prior but also gives us some degree of confidence that our model is correct. This approach is suggested by both Gelman [9] and Jordan [10].
3. Conclusion
When the assumptions of Bayes' Theorem hold, and when Bayesian updating can be performed computationally efficiently, then it is indeed tautological that Bayes is the optimal approach. Even when some of these assumptions fail, Bayes can still be a fruitful approach. However, by working under weaker (sometimes even adversarial) assumptions, frequentist approaches can perform well in very complicated domains even with fairly simple models; this is because, with fewer assumptions being made at the outset, less work has to be done to ensure that those assumptions are met.
From a research perspective, we should be far from satisfied with either approach — Bayesian methods make stronger assumptions than may be warranted, and frequentists methods provide little in the way of a coherent framework for constructing models, and ask for worst-case guarantees, which probably cannot be obtained in general. We should seek to develop a statistical modeling framework that, unlike Bayes, can deal with unknown priors, adversaries, and limited computational resources.
4. Acknowledgements
Thanks to Emma Pierson, Vladimir Slepnev, and Wei Dai for reading preliminary versions of this work and providing many helpful comments.
5. References
[1] Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta algorithm and applications. Working Paper, 2005.
[2] Christopher J.C. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2:121–167, 1998.
[3] Emmanuel J. Candes. Compressive sampling. In Proceedings of the International Congress of Mathematicians. European Mathematical Society, 2006.
[4] D.M. Chickering. Learning bayesian networks is NP-complete. LECTURE NOTES IN STATISTICS-NEW YORK-SPRINGER VERLAG-, pages 121–130, 1996.
[5] Paul Christiano, Jonathan A. Kelner, Aleksander Madry, Daniel Spielman, and Shang-Hua Teng. Electrical flows, laplacian systems, and faster approximation of maximum flow in undirected graphs. In Proceedings of the 43rd ACM Symposium on Theory of Computing, 2011.
[6] Andrew Critch. Frequentist vs. bayesian breakdown: Interpretation vs. inference. http://lesswrong.com/lw/7ck/frequentist_vs_bayesian_breakdown_interpretation/.
[7] Yoav Freund and Robert E. Schapire. A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence, 14(5):771–780, Sep. 1999.
[8] J. Gasthaus and Y.W. Teh. Improvements to the sequence memoizer. In Advances in Neural Information Processing Systems, 2011.
[9] Andrew Gelman. Induction and deduction in bayesian data analysis. RMM, 2:67–78, 2011.
[10] Michael I. Jordan. Are you a bayesian or a frequentist? Machine Learning Summer School 2009 (video lecture at http://videolectures.net/mlss09uk_jordan_bfway/).
[11] D. Warner North. A tutorial introduction to decision theory. IEEE Transactions on Systems Science and Cybernetics, SSC-4(3):200–210, Sep. 1968.
[12] Igal Sason. On refined versions of the Azuma-Hoeffding inequality with applications in information theory. CoRR, abs/<PHONE>, 2011.
[13] Jacob Steinhardt and Zoubin Ghahramani. Pathological properties of deep bayesian hierarchies. In NIPS Workshop on Bayesian Nonparametrics, 2011. Extended Abstract.
[14] Y.W. Teh. A bayesian interpretation of interpolated Kneser-Ney. Technical Report TRA2/06, School of Computing, NUS, 2006.
[15] Y.W. Teh. A hierarchical bayesian language model based on pitman-yor processes. Coling/ACL, 2006.
[16] Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM Review, 38(1):49–95, Mar. 1996.
[17] F.~Wood, C.~Archambeau, J.~Gasthaus, L.~James, and Y.W. Teh. A stochastic memoizer for sequence data. In Proceedings of the 26th International Conference on Machine Learning, pages 1129–1136, 2009.

Beyond Bayesians and Frequentists


Verifying Stability of Stochastic Systems

I just finished presenting my recent paper on stochastic verification at RSS 2011. There is a conference version online, with a journal article to come later. In this post I want to go over the problem statement and my solution.
Problem Statement
Abstractly, the goal is to be given some sort of description of a system, and of a goal for that system, and then verify that the system will reach that goal. The difference between our work and a lot (but not all) of the previous work is that we want to work with an explicit noise model for the system. So, for instance, I tell you that the system satisfies

where  represents the nominal dynamics of the system,  represents how noise enters the system, and $dw(t)$ is a standard Wiener process (the continuous-time version of Gaussian noise). I would like to, for instance, verify that  for some function  and some final time . For example, if  is one-dimensional then I could ask that , which is asking for  to be within a distance of  of the origin at time . For now, I will focus on time-invariant systems and stability conditions. This means that  and  are not functions of , and the condition we want to verify is that  for all . However, it is not too difficult to extend these ideas to the time-varying case, as I will show in the results at the end.
The tool we will use for our task is a supermartingale, which allows us to prove bounds on the probability that a system leaves a certain region.
Supermartingales
Let us suppose that I have a non-negative function  of my state  such that  for all  and . Here we define  as

Then, just by integrating, we can see that . By Markov's inequality, the probability that  is at most .
We can actually prove something stronger as follows: note that if we re-define our Markov process to stop evolving as soon as , then this only sets  to zero in certain places. Thus the probability that  for this new process is at most . Since the process stops as soon as , we obtain the stronger result that the probability that  for any  is at most . Finally, we only need the condition  to hold when . We thus obtain the following:
Theorem. Let  be a non-negative function such that  whenever . Then with probability at least ,  for all .
We call the condition  the supermartingale condition, and a function that satisfies the martingale condition is called a supermartingale. If we can construct supermartingales for our system, then we can bound the probability that trajectories of the system leave a given region.
NOTE: for most people, a supermartingale is something that satisfies the condition . However, this condition is often impossible to satisfy for systems we might care about. For instance, just consider exponential decay driven by Gaussian noise:

Once the system gets close enough to the origin, the exponential decay ceases to matter much and the system is basically just getting bounced around by the Gaussian noise. In particular, if the system is ever at the origin, it will get perturbed away again, so you cannot hope to find a non-constant function of  that is decreasing in expectation everywhere (just consider the global minimum of such a function: in all cases, there is a non-zero probability that the Gaussian noise will cause  to increase, but a zero probability that  will decrease because we are already at the global minimum this argument doesn't actually work, but I am pretty sure that my claim is true at least subject to sufficient technical conditions).
Applying the Martingale Theorem
Now that we have this theorem, we need some way to actually use it. First, let us try to get a more explicit version of the Martingale condition for the systems we are considering, which you will recall are of the form . Note that .
Then . A Wiener process satisfies  and , so only the nominal dynamics () affect the limit of the first-order term while only the noise () affects the limit of the second-order term (the third-order and higher terms in  all go to zero). We thus end up with the formula

It is not that difficult to construct a supermartingale, but most supermartingales that you construct will yield a pretty poor bound. To illustrate this, consider the system . This is the example in the image from the previous section. Now consider a quadratic function . The preceding formula tells us that . We thus have , which means that the probability of leaving the region  is at most . This is not particularly impressive: it says that we should expect  to grow roughly as , which is how quickly  would grow if it was a random walk with no stabilizing component at all.
One way to deal with this is to have a state-dependent bound $\mathbb{E}[\dot{V}] \leq c-kV$. This has been considered for instance by Pham, Tabareau, and Slotine (see Lemma 2 and Theorem 2), but I am not sure whether their results still work if the supermartingale condition only holds locally instead of globally; I haven't spent much time on this, so they could generalize quite trivially.
Another way to deal with this is to pick a more quickly-growing candidate supermartingale. For instance, we could pick . Then , which has a global maximum of  at $x = rac{\sqrt{3}}{2}$. This bounds then says that  grows at a rate of at most , which is better than before, but still much worse than reality.
We could keep improving on this bound by considering successively faster-growing polynomials. However, automating such a process becomes expensive once the degree of the polynomial gets large. Instead, let's consider a function like . Then , which has a maximum of 0.5 at x=0. Now our bound says that we should expect x to grow like , which is a much better growth rate (and roughly the true growth rate, at least in terms of the largest value of  over the time interval ).
This leads us to our overall strategy for finding good supermartingales. We will search across functions of the form  where  is a matrix (the  means "positive semidefinite", which roughly means that the graph of the function  looks like a bowl rather than a saddle/hyperbola). This begs two questions: how to upper-bound the global maximum of  for this family, and how to search efficiently over this family. The former is done by doing some careful work with inequalities, while the latter is done with semidefinite programming. I will explain both below.
Upper-bounding
In general, if , then . We would like to show that such a function is upper-bounded by a constant . To do this, move the exponential term to the right-hand-side to get the equivalent condition . Then we can lower-bound  by  and obtain the sufficient condition

It is still not immediately clear how to check such a condition, but somehow the fact that this new condition only involves polynomials (assuming that f and g are polynomials) seems like it should make computations more tractable. This is indeed the case. While checking if a polynomial is positive is NP-hard, checking whether it is a sum of squares of other polynomials can be done in polynomial time. While sum of squares is not the same as positive, it is a sufficient condition (since the square of a real number is always positive).
The way we check whether a polynomial p(x) is a sum of squares is to formulate it as the semidefinite program: , where  is a vector of monomials. The condition  is a set of affine constraints on the entries of , so that the above program is indeed semidefinite and can be solved efficiently.
Efficiently searching across all matrices S
We can extend on the sum-of-squares idea in the previous section to search over . Note that if  is a parameterized polynomial whose coefficient are affine in a set of decision variables, then the condition  is again a set of affine constraints on . This almost solves our problem for us, but not quite. The issue is the form of  in our case:

Do you see the problem? There are two places where the constraints do not appear linearly in the decision variables:  and  multiply each other in the first term, and  appears quadratically in the last term. While the first non-linearity is not so bad ( is a scalar so it is relatively cheap to search over  exhaustively), the second non-linearity is more serious. Fortunately, we can resolve the issue with Schur complements. The idea behind Schur complements is that, assuming , the condition  is equivalent to . In our case, this means that our condition is equivalent to the condition that

where  is the identity matrix. Now we have a condition that is linear in the decision variable , but it is no longer a polynomial condition, it is a condition that a matrix polynomial be positive semidefinite. Fortunately, we can reduce this to a purely polynomial condition by creating a set of dummy variables  and asking that

We can then do a line search over  and solve a semidefinite program to determine a feasible value of . If we care about remaining within a specific region, we can maximize  such that  implies that we stay in the region. Since our bound on the probability of leaving the grows roughly as , this is a pretty reasonable thing to maximize (we would actually want to maximize , but this is a bit more difficult to do).
Oftentimes, for instance if we are verifying stability around a trajectory, we would like  to be time-varying. In this case an exhaustive search is no longer feasible. Instead we alternate between searching over  and searching over . In the step where we search over , we maximize . In the step where we search over , we maximize the amount by which we could change  and still satisfy the constraints (the easiest way to do this is by first maximizing , then minimizing , then taking the average; the fact that semidefinite constraints are convex implies that this optimizes the margin on  for a fixed ).
A final note is that systems are often only stable locally, and so we only want to check the constraint  in a region where . We can do this by adding a Lagrange multiplier to our constraints. For instance, if we want to check that  whenever , it suffices to find a polynomial  such that  and . (You should convince yourself that this is true; the easiest proof is just by casework on the sign of .) This again introduces a non-linearity in the constraints, but if we fix  and  then the constraints are linear in  and , and vice-versa, so we can perform the same alternating maximization as before.
Results
Below is the most exciting result, it is for an airplane with a noisy camera trying to avoid obstacles. Using the verification methods above, we can show that with probability at least  that the plane trajectory will not leave the gray region:


Useful Math

I have spent the last several months doing applied math, culminating in a submission of a paper to a robotics conference (although culminating might be the wrong word, since I'm still working on the project).
Unfortunately the review process is double-blind so I can't talk about that specifically, but I'm more interested in going over the math I ended up using (not expositing on it, just making a list, more or less). This is meant to be a moderate amount of empirical evidence for which pieces of math are actually useful, and which aren't (of course, the lack of appearance on this list doesn't imply uselessness, but should be taken as Bayesian evidence against usefulness).
I'll start with the stuff that I actually used in the paper, then stuff that helped me formulate the ideas in the paper, then stuff that I've used in other work that hasn't yet come to fruition. These will be labelled I, II, and III below. Let me know if you think something should be in III that isn't [in other words, you think there's a piece of math that is useful but not listed here, preferably with the application you have in mind], or if you have better links to any of the topics below.
I. Ideas used directly
Optimization: semidefinite optimization, convex optimization, sum-of-squares programming, Schur complements, Lagrange multipliers, KKT conditions
Differential equations: Lyapunov functions, linear differential equations, Poincaré return map, exponential stability, Ito calculus
Linear algebra: matrix exponential, trace, determinant, Cholesky decomposition, plus general matrix manipulation and familiarity with eigenvalues and quadratic forms
Probability theory: Markov's inequality, linearity of expectation, martingales, multivariate normal distribution, stochastic processes (Wiener process, Poisson process, Markov process, Lévy process, stopped process)
Multivariable calculus: partial derivative, full derivative, gradient, Hessian, Matrix calculus, Taylor expansion
II. Indirectly helpful ideas
Inequalities: Jensen's inequality, testing critical points
Optimization: (non-convex) function minimization
III. Other useful ideas
Calculus: calculus of variations, extended binomial theorem
Function Approximation: variational approximation, neural networks
Graph Theory: random walks and relation to Markov Chains, Perron-Frobenius Theorem, combinatorial linear algebra, graphical models (Bayesian networks, Markov random fields, factor graphs)
Miscellaneous: Kullback-Leibler divergence, Riccati equation, homogeneity / dimensional analysis, AM-GM, induced maps (in a general algebraic sense, not just the homotopy sense; unfortunately I have no good link for this one)
Probability: Bayes' rule, Dirichlet process, Beta and Bernoulli processes, details balance and Markov Chain Monte Carlo
Spectral analysis: Fourier transform, windowing, aliasing, wavelets, Pontryagin duality
Linear algebra: change of basis, Schur complement, adjoints, kernels, injectivity/surjectivity/bijectivity of linear operators, natural transformations / characteristic subspaces
Topology: compactness, open/closed sets, dense sets, continuity, uniform continuity, connectedness, path-connectedness
Analysis: Lipschitz continuity, Lesbesgue measure, Haar measure, manifolds, algebraic manifolds
Optimization: quasiconvexity

Useful Math


Generalizing Across Categories

Humans are very good at correctly generalizing rules across categories (at least, compared to computers). In this post I will examine mechanisms that would allow us to do this in a reasonably rigorous manner. To this end I will present a probabilistic model such that conditional inference on that model leads to generalization across a category.
There are three questions along these lines that I hope to answer:

How does one generalize rules across categories?
How does one determine which rules should generalize across which categories?
How does one determine when to group objects into a category in the first place?

I suspect that the mechanisms for each of these is rather complex, but I am reasonably confident that the methods I present make up at least part of the actual answer. A good exercise is to come up with examples where these methods fail.
Generalizing Across Categories
For simplicity I'm going to consider just some sort of binary rule, such as the existence of an attribute. So as an example, let's suppose that we see a bunch of ducks, which look to varying degrees to be mallards. In addition to this, we notice that some of the ducks have three toes, and some have four toes. In this case the category is "mallards", and the attribute is "has three toes".
A category is going to be represented as a probabilistic relation for each potential member, of the form , where  is the category and  is the object in question. This essentially indicates the degree to which the object  belongs to the category . For instance, if the category is "birds", then pigeons, sparrows, and eagles all fit into the category very well, so we assign a high probability (close to 1) to pigeons, sparrows, and eagles belonging in the category "birds". On the other hand, flamingos, penguins, ostriches, and pterodactyls, while still being bird-like, don't necessarily fit into our archetypal notion of what it means to be a bird. So they get lower probabilities (I'd say around 0.8 for the first 3 and around 0.1 for pterodactyls, but that is all pretty subjective). Finally, dogs and cats get a near-zero probability of being birds, and a table, the plus operator, or Beethoven's Ninth Symphony would get even-closer-to-zero probabilities of being a bird.
In the example with ducks, the probability of being a mallard will probably be based on how many observed characteristics the duck in question has in common with mallards.
For now we will think of a rule or an attribute as an observable binary property about various objects. Let's let  be the set of all things that have this property. Now we assume that membership in  has some base probability of occurrence , together with some probability of occurrence within  of . We will assume that there are objects , , and , such that the  were all observed to lie in , the  were all observed to not lie in , and membership in  has been unobserved for the .
Furthermore,  has probability  of belonging to ,  has probability  of belonging to , and  has probability  of belonging to .
Again, going back to the ducks example, the  are the ducks observed to have three toes, the  are the ducks observed to have four toes, and the  are the ducks whose number of toes we have not yet observed. The , , and  are the aforementioned probabilities that each of these ducks is a mallard.
So to summarize, we have three classes of objects — those which certainly lie in P, those which certainly don't lie in P, and those for which we have no information about P. For each element in each of these classes, we have a measure of the extent to which it belongs in C.
Given this setup, how do we actually go about specifying a model explicitly? Basically, we say that if something lies in  then it has property  with probability , and otherwise it has it with probability . Thus we end up with

What this ends up meaning is that  will shift to accommodate the objects that (most likely) don't lie in , and  will shift to accommodate the objects that (most likely) do lie in . At the same time, if for instance most elements in  also lie in , then the elements that don't lie in  will have a relatively lower posterior probability of lying in  (compared to its prior probability). So this model not only has the advantage of generalizing across all of  based on the given observations, it also has the advantage of re-evaluating whether an object is likely to lie in a category based on whether it shares attributes with the other objects in that category.
Let's go back to the mallards example one more time. Suppose that after we make all of our observations, we notice that most of the ducks that we think are mallards also have three toes. Then  in this case (the probability that a mallard has three toes) will be close to 1. Furthermore, any duck that we observe to have four toes, we will think much less likely to be a mallard, even if it otherwise looks similar (although it wouldn't be impossible; it could be a genetic mutant, for instance). At the same time, if non-mallards seem to have either three or four toes with roughly equal frequency, then  will be close to 0.5.
There is one thing that I am dissatisfied with in the above model, though. As it stands,  measures the probability of an object not lying in  to have property , rather than the probability of a generic object to have property . This is mainly a problem because, later on, I would like to be able to talk about an object lying in multiple categories, and I don't have a good way of doing that yet.
An important thing to realize here is that satisfying a rule or having an attribute is just another way of indicating membership in a set. So we can think of both the category and the attribute as potential sets that an object could lie in; as before, we'll call the category  and we will call the set of objects having a given attribute . Then  is something like , whereas  (should be) , although as I've set it up now it's more like .
As a final note, this same setup can be applied to the case when there are multiple attributes under consideration. We can also apply it to the case when the objects are themselves categories, and so instead of having strict observations about each attribute, we have some sort of probability that the object will possess an attribute. In this latter case we just treat these probabilities as uncertain observations, as discussed previously.
When To Generalize
Another important question is which rules / attributes we should expect to generalize across a category, and which we should not. For instance, if we would expect "number of toes" to generalize across all animals in a given species, but not "age".
I think that for binary attributes we will always have to generalize, although our generalization could just be "occurs at the base rate in the population". However, for more complicated attributes (for instance, age, which has an entire continuum of possible values), we can think about whether to generalize based on whether our posterior distribution ends up very tightly concentrated or very spread out. Even if it ends up fairly spread out, there is still some degree of generalization — for instance, we expect that most insects have fairly short lifespans compared to mammals. You could say that this is actually a scientific fact (since we can measure lifespan), but even if you ran into a completely new species of insect, you would expect it to have a relatively short lifespan.
On the other hand, if we went to a college town, we might quickly infer that everyone there was in their 20s; if we went to a bridge club, we would quickly infer that everyone there was past retirement age.
What I would say is that we can always generalize across a category, but sometimes the posterior distribution we end up with is very spread out, and so our generalization isn't particularly strong. In some cases, the posterior distribution might not even be usefully different from the prior distribution; in this case, from a computational standpoint it doesn't make sense to keep track of it, and so it feels like an attribute shouldn't generalize across a category, while what we really mean is that we don't bother to keep track of that generalization.
An important thing to keep in mind on this topic is that everything that we do when we construct models is a computational heuristic. If we didn't care about computational complexity and only wanted to arrive at an answer, we would use Solomonoff induction, or some computable approximation to it. So whenever I talk about something in probabilistic modeling, I'm partially thinking about whether the method is computationally feasible at all, and what sort of heuristics might be necessary to actually implement something in practice.
To summarize:

attributes always generalize across a category
but the generalization might be so weak as to be indistinguishable from the prior distribution
it would be interesting to figure out how we decide which generalizations to keep track of, and which not to

Forming Categories
My final topic of consideration is when to actually form a category. Before going into this, I'd like to share some intuition with you. This is the intuition of explanatory "cheapness" or "expensiveness". The idea is that events that have low probability under a given hypothesis are "expensive", and events that have relatively high probability are "cheap". When a fixed event is expensive under a model, that is evidence against that model; when it is cheap under a model, that is evidence for the model.
The reason that forming clusters can be cheaper is that it allows us to explain many related events simultaneously, which is cheaper than explaining them all separately. Let's take an extreme example — suppose that 70% of all ducks have long beaks, and 70% have three toes (the remaining 30% in both cases have short beaks and four toes). Then the "cheapest" explanation would be to assign both long beaks and three toes a probability of 0.7, and the probability of our observations would be (supposing there were 100 ducks) . However, suppose that we also observe that the ducks with long beaks are exactly the ducks with three toes. Then we can instead postulate that there are two categories, "(long,3)" and "(short,4)", and that the probability of ending up in the first category is 0.7 while the probability of ending up in the second category is 0.3. This is a much cheaper explanation, as the probability of our observations in this scenario becomes . We have to balance this slightly with the fact that an explanation that involves creating an extra category is more expensive (because it has a lower prior probability / higher complexity) than one that doesn't, but the extra category can't possibly be  times more expensive, so we should still always favor it.
This also demonstrates that it will become progressively cheaper to form categories as we get large amounts of similar evidence, which corresponds with our intuitive notion of categories as similarity clusters.
So, now that I've actually given you this intuition, how do we actually go about forming categories? Each time we form a category, we will have some base probability of membership in that category, together with a probability of each member of that category possessing each attribute under consideration. If we simply pick prior distributions for each of these parameters, then they will naturally adopt posterior distributions based on the observed data, although these distributions might have multiple high-probability regions if multiple categories can be formed from the data. At the same time, each object will naturally end up with some probability of being in each category, based on how well that category explains its characteristics (as well as the base probability that objects should end up in that category). Putting these together, we see that clusters will naturally form to accommodate the data.
As an example of how we would have multiple high-probability regions, suppose that, as before, there are long-billed, three-toed ducks and short-billed, four-toed ducks. But we also notice that ducks with orange beaks have white feathers and ducks with yellow beaks have brown feathers. If we form a single category, then there are two good candidates, so that category is likely to be either the (long,3) category or the (orange,white) category (or their complements). If we form two categories, each category in isolation is still likely to be either the (long,3) or (orange,white) category, although if the first category is (long,3), then the second is very likely to be (orange,white), and vice versa. In other words, it would be silly to make both categories (long,3), or both categories (orange,white).
The only thing that is left to do is to specify some prior probability of forming a category. While there are various ways to do this, the most commonly used way is the Indian Buffet Process. I won't explain it in detail here, but I might explain it later.
Future Considerations
There are still some unresolved questions here. First of all, in reality something has the potential to be in many, many categories, and it is not entirely clear how to resolve such issues in the above framework. Secondly, keeping track of all of our observations about a given category can be quite difficult computationally (updating in the above scenarios requires performing computations on high-degree polynomials), so we need efficient algorithms for dealing with all of the data we get.
I'm not sure yet how to deal with the first issue, although I'll be thinking about it in the coming weeks. To deal with the second issue, I intend to use an approach based on streaming algorithms, which will hopefully make up a good final project for a class I'm taking this semester (Indyk and Rubinfeld's class on Sublinear Algorithms, if you know about MIT course offerings).

Generalizing Across Categories


Uncertain Observations

What happens when you are uncertain about observations you made? For instance, you remember something happening, but you don't remember who did it. Or you remember some fact you read on wikipedia, but you don't know whether it said that hydrogen or helium was used in some chemical process.
How do we take this information into account in the context of Bayes' rule? First, I'd like to note that there are different ways something could be uncertain. It could be that you observed X, but you don't remember if it was in state A or state B. Or it could be that you think you observed X in state A, but you aren't sure.
These are different because in the first case you don't know whether to concentrate probability mass towards A or B, whereas in the second case you don't know whether to concentrate probability mass at all.
Fortunately, both cases are pretty straightforward as long as you are careful about using Bayes' rule. However, today I am going to focus on the latter case. In fact, I will restrict my attention to the following problem:
You have a coin that has some probability  of coming up heads. You also know that all flips of this coin are independent. But you don't know what  is. However, you have observed this coin  times in the past. But for each observation, you aren't completely sure that this was the coin you were observing. In particular, you only assign a probability  to your th observation actually being about this coin. Given this, and the sequence of heads and tails you remember, what is your estimate of
To use Bayes' rule, let's first figure out what we need to condition on. In this case, we need to condition on remembering the sequence of coin flips that we remembered. So we are looking for
p( | we remember the given sequence of flips),
which is proportional to
p(we remember the given sequence of flips | )  p().
The only thing that the uncertain nature of our observations does is cause there to be multiple ways to eventually land in the set of universes where we remember the sequence of flips; in particular, for any observation we remember, it could have actually happened, or we could have incorrectly remembered it. Thus if , and we remember the th coin flip as being heads, then this could happen with probability  if we incorrectly remembered a coin flip of heads. In the remaining probability  case, it could happen with probability  by actually coming up heads. Therefore the probability of us remembering that the th flip was heads is .
A similar computation shows that the probability of us remembering that the th flip was tails is .
For convenience of notation, let's actually split up our remembered flips into those that were heads and those that were tails. The probability of the th remembered heads being real is , and the probability of the th remembered tails being real is . There are  heads and  tails. Then we get

Note that when we consider values of  close to , the term from the remembered tails becomes close to  raised to the power of the expected number of tails, whereas the term from the remembered heads becomes close to the probability that we incorrectly remembered each of the heads. A similar phenomenon will happen when  gets close to . This is an instance of a more general phenomenon whereby unlikely observations get "explained away" by whatever means possible.
A Caveat
Applying the above model in practice can be quite tricky. The reason is that your memories are intimately tied to all sorts of events that happen to you; in particular, your assessment of how likely you are to remember an event probably already takes into account how well that event fits into your existing model. So if you saw 100 heads, and then a tails, you would place more weight than normal on your recollection of the tails being incorrect, even though that is the job of the above model. In essence, you are conditioning on your data twice — once intuitively, and once as part of the model. This is bad because it assumes that you made each observation twice as many times as you actually did.
What is interesting, though, is that you can actually compute things like the probability that you incorrectly remembered an event, given the rest of the data, and it will be different from the prior probability. So in addition to a posterior estimate of , you get posterior estimates of the likelihood of each of your recollections. Just be careful not to take these posterior estimates and use them as if they were prior estimates (which, as explained above, is what we are likely to do intuitively).
There are other issues to using this in practice, as well. For instance, if you really want the coin to be fair, or unfair, or be biased in a certain direction, it is very easy to fool yourself into assigning skewed probability estimates towards each of your recollections, thus ending up with a biased answer at the end. It's not even difficult — if I take a fair coin, and underestimate my recollection of each tails by 20%, and overestimate my recollection of each heads by 20%, then all of a sudden I "have a coin" that is 50% more likely to come up heads than tails.
Fortunately, my intended application of this model will be in a less slippery domain (hopefully). The purpose is to finally answer the question I posed in the last post, which I'll repeat here for convenience:
Suppose that you have never played a sport before, and you play soccer, and enjoy it. Now suppose instead that you have never played a sport before, and play soccer, and hate it. In the first case, you will think yourself more likely to enjoy other sports in the future, relative to in the second case. Why is this?
Or if you disagree with the premises of the above scenario, simply "If X and Y belong to the same category C, why is it that in certain cases we think it more likely that Y will have attribute A upon observing that X has attribute A?"

In the interest of making my posts shorter, I will leave that until next time, but hopefully I'll get to it in the next week.

Uncertain Observations


Nobody Understands Probability

The goal of this post is to give an overview of Bayesian statistics as well as to correct errors about probability that even mathematically sophisticated people commonly make. Hopefully by the end of this post I will convince you that you don't actually understand probability theory as well as you think, and that probability itself is something worth thinking about.
I will try to make this post somewhat shorter than the previous posts. As a result, this will be only the first of several posts on probability. Even though this post will be shorter, I will summarize its organization below:

Bayes' theorem: the fundamentals of conditional probability
modeling your sources: how not to calculate conditional probabilities; the difference between "you are given X" and "you are given that you are given X"
how to build models: examples using toy problems
probabilities are statements about your beliefs (not the world)
re-evaluating a standard statistical test

I bolded the section on models because I think it is very important, so I hope that bolding it will make you more likely to read it.
Also, I should note that when I say that nobody understands probability, I don't mean it in the sense that most people are bad at combinatorics. Indeed, I expect that most of the readers of this blog are quite proficient at combinatorics, and that many of them even have sophisticated mathematical definitions of probability. Rather I would say that actually using probability theory in practice is non-trivial. This is partially because there are some subtleties (or at least, I have found myself tripped up by certain points, and did not realize this until much later). It is also because whenever you use probability theory in practice, you end up employing various heuristics, and it's not clear which ones are the "right" ones.
If you disagree with me, and think that everything about probability is trivial, then I challenge you to come up with a probability-theoretic explanation of the following phenomenon:
Suppose that you have never played a sport before, and you play soccer, and enjoy it. Now suppose instead that you have never played a sport before, and play soccer, and hate it. In the first case, you will think yourself more likely to enjoy other sports in the future, relative to in the second case. Why is this?
Or if you disagree with the premises of the above scenario, simply "If X and Y belong to the same category C, why is it that in certain cases we think it more likely that Y will have attribute A upon observing that X has attribute A?"
Bayes' Theorem
Bayes' theorem is a fundamental result about conditional probability. It says the following:

Here  and  are two events, and  means the probability of  conditioned on . In other words, if we already know that  occurred, what is the probability of ? The above theorem is quite easy to prove, using the fact that , and thus also equals , so that , which implies Bayes' theorem. So, why is it useful, and how do we use it?
One example is the following famous problem: A doctor has a test for a disease that is 99% accurate. In other words, it has a 1% chance of telling you that you have a disease even if you don't, and it has a 1% chance of telling you that you don't have a disease even if you do. Now suppose that the disease that this tests for is extremely rare, and only affects 1 in 1,000,000 people. If the doctor performs the test on you, and it comes up positive, how likely are you to have the disease?
The answer is close to , since it is roughly  times as likely for the test to come up positive due to an error in the test relative to you actually having the disease. To actually compute this with Bayes' rule, you can say
p(Disease | Test is positive) = p(Test is positive | Disease)p(Disease)/p(Test is positive),
which comes out to  which is quite close to .
In general, we can use Bayes' law to test hypotheses:
p(Hypothesis | Data) = p(Data | Hypothesis) p(Hypothesis) / p(Data)
Let's consider each of these terms separately:

p(Hypothesis | Data) — the weight we assign to a given hypothesis being correct under our observed data
p(Data | Hypothesis) — the likelihood of seeing the data we saw under our hypothesis; note that this should be quite easy to compute. If it isn't, then we haven't yet fully specified our hypothesis.
p(Hypothesis) — the prior weight we give to our hypothesis. This is subjective, but should intuitively be informed by the consideration that "simpler hypotheses are better".
p(Data) — how likely we are to see the data in the first place. This is quite hard to compute, as it involves considering all possible hypotheses, how likely each of those hypotheses is to be correct, and how likely the data is to occur under each hypothesis.

So, we have an expression for p(Hypothesis | Data), one of which is easy to compute, the other of which can be chosen subjectively, and the last of which is hard to compute. How do we get around the fact that p(Data) is hard to compute? Note that p(Data) is independent of which hypothesis we are testing, so Bayes' theorem actually gives us a very good way for comparing the relative merits of two hypotheses:
p(Hypothesis 1 | Data) / p(Hypothesis 2 | Data) = [p(Data | Hypothesis 1) / p(Data | Hypothesis 2)] * p(Hypothesis 1) / p(Hypothesis 2)
Let's consider the following toy example. There is a stream of digits going past us, too fast for us to tell what the numbers are. But we are allowed to push a button that will stop the stream and allow us to see a single number (whichever one is currently in front of us). We push this button three times, and see the numbers 3, 5, and 3. How many different numbers would we estimate are in the stream?
For simplicity, we will make the (somewhat unnatural) assumption that each number between 0 and 9 is selected to be in the stream with probability 0.5, and that each digit in the stream is chosen uniformly from the set of selected numbers. It is worth noting now that making this assumption, rather than some other assumption, will change our final answer.
Now under this assumption, what is the probability, say, of there being exactly 2 numbers (3 and 5) in the stream? By Bayes' theorem, we have

Here  means "is proportional to".
What about the probability of there being 3 numbers (3, 5, and some other number)? For any given other number, this would be

However, there are 8 possibilities for  above, all of which correspond to disjoint scenarios, so the probability of there being 3 numbers is proportional to . If we compare this to the probability of there being 2 numbers, we get
p(2 numbers | (3,5,3)) / p(3 numbers | (3,5,3)) = 3/8.
Even though we have only seen two numbers in our first three samples, we still think it is more likely that there are 3 numbers than 2, just because the prior likelihood of there being 3 numbers is so much higher. However, suppose that we made 6 draws, and they were 3,5,3,3,3,5. Then we would get
p(2 numbers | (3,5,3,3,5)) / p(3 numbers | (3,5,3,3,5)) = (1/2)^6 / [9 * (1/3)^6] = 81/64.
Now we find it more likely that there are only 2 numbers. This is what tends to happen in general with Bayes' rule — over time, more restrictive hypotheses become exponentially more likely than less restrictive hypotheses, provided that they correctly explain the data. Put another way, hypotheses that concentrate probability density towards the actual observed events will do best in the long run. This is a nice feature of Bayes' rule because it means that, even if the prior you choose is not perfect, you can still arrive at the "correct" hypothesis through enough observations (provided that the hypothesis is among the set of hypotheses you consider).
I will use Bayes' rule extensively through the rest of this post and the next few posts, so you should make sure that you understand it. If something is unclear, post a comment and I will try to explain in more detail.
Model Your Sources
An important distinction that I think most people don't think about is the difference between experiments you perform, and experiments you observe. To illustrate what I mean by this, I would point to the difference between biology and particle physics — where scientists set out to test a hypothesis by creating an experiment specifically designed to do so — and astrophysics and economics, where many "experiments" come from seeking out existing phenomena that can help evaluate a hypothesis.
To illustrate why one might need to be careful in the latter case, consider empirical estimates of average long-term GDP growth rate. How would one do this? Since it would be inefficient to wait around for the next 10 years and record the data of all currently existing countries, instead we go back and look at countries that kept records allowing us to compute GDP. But in this case we are only sampling from countries that kept such records, which implies a stable government as well as a reasonable degree of economics expertise within that government. So such a study almost certainly overestimates the actual average growth rate.
Or as another example, we can argue that a scientist is more likely to try to publish a paper if it doesn't agree with prevalent theories than if it does, so looking merely at the proportion of papers that lend support to or take support away from a theory (even if weighted by the convincingness of each paper) is probably not a good way to determine the validity of a theory.
So why are we safer in the case that we forcibly gather our own data? By gathering our own data, we understand much better (although still not perfectly) the way in which it was constructed, and so there is less room for confounding parameters. In general, we would like it to be the case that the likelihood of observing something that we want to observe does not depend on anything else that we care about — or at the very least, we would like it to depend in a well-defined way.
Let's consider an example. Suppose that a man comes up to you and says "I have two children. At least one of them is a boy." What is the probability that they are both boys?
The standard way to solve this is as follows: Assuming that male and female children are equally likely, there is a  chance of two girls or two boys, and a  chance of having one girl and one boy. Now by Bayes' theorem,
p(Two boys | At least one boy) = p(At least one boy | Two boys) * p(Two boys) / p(At least one boy) = 1 * (1/4) / (1/2+1/4) = 1/3.
So the answer should be 1/3 (if you did math contests in high school, this problem should look quite familiar). However, the answer is not, in fact, 1/3. Why is this? We were given that the man had at least one boy, and we just computed the probability that the man had at two boys given that he had at least one boy using Bayes' theorem. So what's up? Is Bayes' theorem wrong?
No, the answer comes from an unfortunate namespace collision in the word "given". The man "gave" us the information that he has at least one male child. By this we mean that he asserted the statement "I have at least one male child." Now our issue is when we confuse this with being "given" that the man has at least one male child, in the sense that we should restrict to the set of universes in which the man has at least one male child. This is a very different statement than the previous one. For instance, it rules out universes where the man has two girls, but is lying to us.
Even if we decide to ignore the possibility that the man is lying, we should note that most universes where the man has at least one son don't even involve him informing us of this fact, and so it may be the case that proportionally more universes where the man has two boys involve him telling us "I have at least one male child", relative to the proportion of such universes where the man has one boy and one girl. In this case the probability that he has two boys would end up being greater than 1/3.
The most accurate way to parse this scenario would be to say that we are given (restricted to the set of possible universes) that we are given (the man told us that) that the man has at least one male child. The correct way to apply Bayes' rule in this case is
p(X has two boys | X says he has >= 1 boy) = p(X says he has >= 1 boy | X has two boys) * p(X has two boys) / p(X says he has >=1 boy)
If we further assume that the man is not lying, and that male and female children are equally likely and uncorrelated, then we get
p(X has two boys | X says he has >= 1 boy) = [p(X says he has >= 1 boy | X has two boys) * 1/4]/[p(X says he has >= 1 boy | X has two boys) * 1/4 + p(X says he has >= 1 boy | X has one boy) * 1/2]
So if we think that the man is  times more likely to tell us that he has at least one boy when he has two boys, then
p(X has two boys | X says he has >= 1 boy) = .
Now this means that if we want to claim that the probability that the man has two boys is , what we are really claiming is that he is equally likely to inform us that he has at least one boy, in all situations where it is true, independent of the actual gender distribution of his children.
I would argue that this is quite unlikely, as if he has a boy and a girl, then he could equally well have told us that he has at least one girl, whereas he couldn't tell us that if he has only boys. So I would personally put  closer to 2, which yields an answer of . On the other hand, situations where someone walks up to me and tells me strange facts about the gender distribution of their children are, well, strange. So I would also have to take into account the likely psychology of such a person, which would end up changing my estimate of $lpha$.
The whole point here is that, because we were an observer receiving information, rather than an experimenter acquiring information, there are all sorts of confounding factors that are difficult to estimate, making it difficult to get a good probability estimate (more on that later). That doesn't mean that we should give up and blindly guess , though — it might feel like doing so gets away without making unwarranted assumptions, but it in fact implicitly makes the assumption that , which as discussed above is almost certainly unwarranted.
What it does mean, though, is that, as scientists, we should try to avoid situations like the one above where there are lots of confounding factors between what we care about and our observations. In particular, we should avoid uncertainties in the source of our information by collecting the information ourselves.
I should note that, even when we construct our own experiments, we should still model the source of our information. But doing so is often much easier. In fact, if we wanted to be particularly pedantic, we really need to restrict to the set of universes in which our personal consciousness receives a particular set of stimuli, but that set of stimuli has almost perfect correlation with photons hitting our eyes, which has almost perfect correlation with the set of objects in front of us, so going to such lengths is rarely necessary — we can usually stop at the level of our personal observations, as long as we remember where they come from.
How to Build Models
Now that I've told you that you need to model your information sources, you perhaps care about how to do said modeling. Actually, constructing probabilistic models is an extremely important skill, so even if you ignore the rest of this post, I recommend paying attention to this section.
This section will have the following examples:

determining if a coin is fair
finding clusters

Determining if a Coin is Fair
Suppose that you have occasion to observe a coin being flipped (or better yet, you flip it yourself). You do this several times and observe a particular sequence of heads and tails. If you see all heads or all tails, you will probably think the coin is unfair. If you see roughly half heads and half tails, you will probably think the coin is fair. But how do we quantify such a calculation? And what if there are noticeably many more heads than tails, but not so many as to make the coin obviously unfair?
We'll solve this problem by building up a model in parts. First, there is the thing we care about, namely whether the coin is fair or unfair. So we will construct a random variable X that can take the values Fair and Unfair. Then p(X = Fair) is the probability we assign to a generic coin being fair, and p(X = Unfair) is the probability we assign to a generic coin being unfair.
Now supposing the coin is fair, what do we expect? We expect each flip of the coin to be independent, and have a  probability of coming up heads. So if we let F1, F2, …, Fn be the flips of the coin, then p(Fi = Heads | X = Fair) = 0.5.
What if the coin is unfair? Let's go ahead and blindly assume that the flips will still be independent, and furthermore that each possible weight of the coin is equally likely (this is unrealistic, as weights near 0 or 1 are much more likely than weights near 0.5). Then we have to have an extra variable , the probability that the unfair coin comes up heads. So we have p(Unfair coin weight = ) = 1. Note that this is a probability density, not an actual probability (as opposed to p(Fi = Heads | X = Fair), which was a probability).
Continuing, if F1, F2, …, Fn are the flips of the coin, then p(Fi = Heads | X = Fair, Weight = ) = .
Now we've set up a model for this problem. How do we actually calculate a posterior probability of the coin being fair for a given sequence of heads and tails? (A posterior probability is just the technical term for the conditional probability of a hypothesis given a set of data; this is to distinguish it from the prior probability of the hypothesis before seeing any data.)
Well, we'll still just use Bayes' rule:
p(Fair | F1, …, Fn)  p(F1, …, Fn | Fair) p(Fair) =  p(Fair)
p(Unfair | F1, …, Fn)  p(F1, …, Fn | Unfair) p(Unfair) =  p(Unfair)
Here H is the number of heads and T is the number of tails. In this case we can fortunately actually compute the integral in question and see that it is equal to . So we get that
p(Fair | F1, …, Fn) / p(Unfair | F1, …, Fn) = p(Fair)/p(Unfair) * .
It is often useful to draw a diagram of our model to help keep track of it:

Now suppose that we, being specialists in determining if coins are fair, have been called in to study a large collection of coins. We get to one of the coins in the collection, flip it several times, and observe the following sequence of heads and tails:
HHHHTTTHHTTT
Since there are an equal number of heads and tails, our previous analysis will certainly conclude that the coin is fair, but its behavior does seem rather suspicious. In particular, different flips don't look like they are really independent, so perhaps our previous model is wrong. Maybe the right model is one where the next coin value is usually the same as the previous coin value, but flips with some probability. Now we have a new value of X, which we'll call Weird, and a parameter  (basically the same as ) that tells us how likely a weird coin is to have a given probability of switching. We'll again give  a uniform distribution over [0,1], so p(Switching probability of weird coin = ) = 1.
To predict the actual coin flips, we get p(F1 = Heads | X = Weird, Switching probability = ) = 1, p(F(i+1) = Heads | Fi = Heads, X = Weird, Switching probability = ) = , and p(F(i+1) = Heads | Fi = Tails, X = Weird, Switching probability = ) = . We can represent this all with the following graphical model:

Now we are ready to evaluate whether the coin we saw was a Weird coin or not.
p(X = Weird | HHHHTTTHHTTT)  p(HHHHTTTHHTTT | X = Weird) p(X = Weird) = $\int_{0}^{1} rac{1}{2}(1-\phi)^8 \phi^3 d\phi$ p(X = Weird)
Evaluating that integral gives . So p(X = Weird | Data) = p(X = Weird) / 3960, compared to p(X = Fair | Data), which is p(X = Fair) / 4096. In other words, positing a Weird coin only explains the data slightly better than positing a Fair coin, and since the vast majority of coins we encounter are fair, it is quite likely that this one is, as well.
Note: I'd like to draw your attention to a particular subtlety here. Note that I referred to, for instance, "Probability that an unfair coin weight is ", as opposed to "Probability that a coin weight is  given that it is unfair". This really is an important distinction, because the distribution over  really is the probability distribution over the weights of a generic unfair coin, and this distribution doesn't change based on whether our current coin happens to be fair or unfair. Of course, we can still condition on our coin being fair or unfair, but that won't change the probability distribution over  one bit.
Finding Clusters
Now let's suppose that we have a bunch of points (for simplicity, we'll say in two-dimensional Euclidean space). We would like to group the points into a collection of clusters. Let's also go ahead and assume that we know in advance that there are  clusters. How do we actually find those clusters?
We'll make the further heuristic assumption that clusters tend to arise from a "true" version of the cluster, and some Gaussian deviation from that true version. So in other words, if we let there be k means for our clusters, , and multivariate Gaussians about their means with covariance matrices , and finally assume that the probability that a point belongs to cluster i is , then the probability of a set of points  is

From this, once we pick probability distributions over the , , and , we can calculate the posterior probability of a given set of clusters as
p(, ,  | )  p() p() p()
This corresponds to the following graphical model:

Note that once we have a set of clusters, we can also determine the probability that a given point belongs to each cluster:
p( belongs to cluster )  .
You might notice, though, that in this case it is much less straightforward to actually find clusters with high posterior probability (as opposed to in the previous case, where it was quite easy to distinguish between Fair, Unfair, and Weird, and furthermore to figure out the most likely values of  and ). One reason why is that, in the previous case, we really only needed to make one-dimensional searches over  and  to figure out what the most likely values were. In this case, we need to search over all of the , , and  simultaneously, which gives us, essentially, a -dimensional search problem, which becomes exponentially hard quite quickly.
This brings us to an important point, which is that, even if we write down a model, searching over that model can be difficult. So in addition to the model, I will go over a good algorithm for finding the clusters from this model, known as the EM algorithm. For the version of the EM algorithm described below, I will assume that we have uniform priors over , , and  (in the last case, we have to do this by picking a set of un-normalized  uniformly over  and then normalizing). We'll ignore the problem that it is not clear how to define a uniform distribution over a non-compact space.
The way the EM algorithm works is that we start by initializing  , and  arbitrarily. Then, given these values, we compute the probability that each point belongs to each cluster. Once we have these probabilities, we re-compute the maximum-likelihood values of the  (as the expected mean of each cluster given how likely each point is to belong to it). Then we find the maximum-likelihood values of the  (as the expected covariance relative to the means we just found). Finally, we find the maximum-likelihood values of the  (as the expected portion of points that belong to each cluster). We then repeat this until converging on an answer.
For a visualization of how the EM algorithm actually works, and a more detailed description of the two steps, I recommend taking a look at Josh Tenenbaum's lecture notes starting at slide 38.

The Mind Projection Fallacy
This is perhaps a nitpicky point, but I have found that keeping it in mind has led me to better understanding what I am doing, or at least to ask interesting questions.
The point here is that people often intuitively think of probabilities as a fact about the world, when in reality probabilities are a fact about our model of the world. For instance, one might say that the probability of a child being male versus female is 0.5. And perhaps this is a good thing to say in a generic case. But we also have a much better model of gender, and we know that it is based on X and Y chromosomes. If we could look at a newly conceived ball of cells in a mother's womb, and read off the chromosomes, then we could say with near certainty whether the child would end up being male or female.
You could also argue that I can empirically measure the probability that a person is male or female, by counting up all the people ever, and looking at the proportion of males and females. But this runs into two issues — first of all, the portion of males will be slightly off of 0.5. So how do we justify just randomly rounding off to 0.5? Or do we not?
Second of all, you can do this all you want, but it doesn't give me any reason why I should take this information, and use it to form a conjecture about how likely the next person I meet is to be male or female. Once we do that, we are taking into account my model of the world.
Statistics
This final section seeks to look at a result from classical statistics and re-interpret it in a Bayesian framework.
In particular, I'd like to consider the following strategy for rejecting a hypothesis. In abstract terms, it says that, if we have a random variable Data' that consists of re-drawing our data assuming that our hypothesis is correct, then
p(Hypothesis) < p(p(Data' | Hypothesis) <= p(Data | Hypothesis))
In other words, suppose that the probability of drawing data less likely (under our hypothesis) than the data we actually saw is less than . Then the likelihood of our hypothesis is at most .
Or actually, this is not quite true. But it is true that there is an algorithm that will only reject correct hypotheses with probability , and this algorithm is to reject a hypothesis when p(p(Data' | Hypothesis) <= p(Data | Hypothesis)) < . I will leave the proof of this to you, as it is quite easy.
To illustrate this example, let's suppose (as in a previous section) that we have a coin and would like to determine whether it is fair. In the above method, we would flip it many times, and record the number H of heads. If there is less than an  chance of coming up with a less likely number of heads than H, then we can reject the hypothesis that the coin is fair with confidence . For instance, if there are 80 total flips, and H = 25, then we would calculatae
.
So this seems like a pretty good test, especially if we choose  to be extremely small (e.g.,  or so). The mere fact that we reject good hypotheses with probability less than  is not helpful. What we really want is to also reject bad hypotheses with a reasonably large probability. I think you can get around this by repeating the same experiment many times, though.
Of course, Bayesian statistics also can't ever say that a hypothesis is good, but when given two hypotheses it will always say which one is better. On the other hand, Bayesian statistics has the downside that it is extremely aggressive at making inferences. It will always output an answer, even if it really doesn't have enough data to arrive at that answer confidently.

Nobody Understands Probability