Beyond Bayesians and Frequentists

(This is available in pdf form [here](http://web.mit.edu/jsteinha/www/stats-essay.pdf).)

If you are a newly initiated student into the field of machine learning, it won’t be long before you start hearing the words “Bayesian” and “frequentist” thrown around. Many people around you probably have strong opinions on which is the “right” way to do statistics, and within a year you’ve probably developed your own strong opinions (which are suspiciously similar to those of the people around you, despite there being a much greater variance of opinion between different labs). In fact, now that the year is 2012 the majority of new graduate students are being raised as Bayesians (at least in the U.S.) with frequentists thought of as stodgy emeritus professors stuck in their ways.

If you are like me, the preceding set of facts will make you very uneasy. They will make you uneasy because simple pattern-matching — the strength of people’s opinions, the reliability with which these opinions split along age boundaries and lab boundaries, and the ridicule that each side levels at the other camp – makes the “Bayesians vs. frequentists” debate look far more like politics than like scholarly discourse. Of course, that alone does not necessarily prove anything; these disconcerting similarities could just be coincidences that I happened to cherry-pick.

My next point, then, is that we are right to be uneasy, because such debate makes us less likely to evaluate the strengths and weaknesses of both approaches in good faith. This essay is a push against that — I summarize the justifications for Bayesian methods and where they fall short, show how frequentist approaches can fill in some of their shortcomings, and then present my personal (though probably woefully under-informed) guidelines for choosing which type of approach to use.

Before doing any of this, though, a bit of background is in order…

**1. Background on Bayesians and Frequentists**

 **1.1. Three Levels of Argument**

As Andrew Critch [6] insightfully points out, the Bayesians vs. frequentists debate is really three debates at once, centering around one or more of the following arguments:

1. Whether to interpret subjective beliefs as probabilities
2. Whether to interpret probabilities as subjective beliefs (as opposed to asymptotic frequencies)
3. Whether a Bayesian or frequentist algorithm is better suited to solving a particular problem.

Given my own research interests, I will add a fourth argument:

4. Whether Bayesian or frequentist techniques are better suited to engineering an artificial intelligence.

Andrew Gelman [9] has his own well-written essay on the subject, where he expands on these distinctions and presents his own more nuanced view.

Why are these arguments so commonly conflated? I’m not entirely sure; I would guess it is for historical reasons but I have so far been unable to find said historical reasons. Whatever the reasons, what this boils down to in the present day is that people often form opinions on 1. and 2., which then influence their answers to 3. and 4. This is *not good*, since 1. and 2. are philosophical in nature and difficult to resolve correctly, whereas 3. and 4. are often much easier to resolve and extremely important to resolve correctly in practice. Let me re-iterate: *the Bayes vs. frequentist discussion should center on the practical employment of the two methods, or, if epistemology must be discussed, it should be clearly separated from the day-to-day practical decisions*. Aside from the difficulties with correctly deciding epistemology, the relationship between generic epistemology and specific practices in cutting-edge statistical research is only via a long causal chain, and it should be completely unsurprising if Bayesian epistemology leads to the employment of frequentist tools or vice versa.

For this reason and for reasons of space, I will spend the remainder of the essay focusing on *statistical algorithms* rather than on *interpretations of probability*. For those who really want to discuss interpretations of probability, I will address that in a later essay.

 **1.2. Recap of Bayesian Decision Theory**

(What follows will be review for many.) In Bayesian decision theory, we assume that there is some underlying world state ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002) and a *likelihood function* ![{p(X_1, \ldots, Xn \mid \theta)}](https://s0.wp.com/latex.php?latex=%7Bp%28X_1%2C+%5Cldots%2C+Xn+%5Cmid+%5Ctheta%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) over possible observations. (A *likelihood function* is just a conditional probability distribution where the parameter conditioned on can vary.) We also have a space ![{A}](https://s0.wp.com/latex.php?latex=%7BA%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) of possible actions and a utility function ![{U(\theta; a)}](https://s0.wp.com/latex.php?latex=%7BU%28%5Ctheta%3B+a%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) that gives the utility of performing action ![{a}](https://s0.wp.com/latex.php?latex=%7Ba%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) if the underlying world state is ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002). We can incorporate notions like planning and value of information by defining ![{U(\theta; a)}](https://s0.wp.com/latex.php?latex=%7BU%28%5Ctheta%3B+a%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) recursively in terms of an identical agent to ourselves who has seen one additional observation (or, if we are planning against an adversary, in terms of the adversary). For a more detailed overview of this material, see the tutorial by North [11].

What distinguishes the Bayesian approach in particular is one additional assumption, a *prior distribution* ![{p(\theta)}](https://s0.wp.com/latex.php?latex=%7Bp%28%5Ctheta%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) over possible world states. To make a decision with respect to a given prior, we compute the posterior distribution ![{p_{\mathrm{posterior}}(\theta \mid X_1, \ldots, X_n)}](https://s0.wp.com/latex.php?latex=%7Bp_%7B%5Cmathrm%7Bposterior%7D%7D%28%5Ctheta+%5Cmid+X_1%2C+%5Cldots%2C+X_n%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) using Bayes’ theorem, then take the action ![{a}](https://s0.wp.com/latex.php?latex=%7Ba%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) that maximizes ![{\mathbb{E}_{p_{\mathrm{posterior}}}[U(\theta; a)]}](https://s0.wp.com/latex.php?latex=%7B%5Cmathbb%7BE%7D_%7Bp_%7B%5Cmathrm%7Bposterior%7D%7D%7D%5BU%28%5Ctheta%3B+a%29%5D%7D&bg=f0f0f0&fg=000000&s=0&c=20201002).

In practice, ![{p_{\mathrm{posterior}}}](https://s0.wp.com/latex.php?latex=%7Bp_%7B%5Cmathrm%7Bposterior%7D%7D%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) can be quite difficult to compute, and so we often attempt to approximate it. Such attempts are known as *approximate inference algorithms*.

 **1.3. Steel-manning Frequentists**

There are many different ideas that fall under the broad umbrella of frequentist techniques. While it would be impossible to adequately summarize all of them even if I attempted to, there are three in particular that I would like to describe, and which I will call *frequentist decision theory*, *frequentist guarantees*, and *frequentist analysis tools*.

Frequentist decision theory has a very similar setup to Bayesian decision theory, with a few key differences. These are discussed in detail and contrasted with Bayesian decision theory in [10], although we summarize the differences here. There is still a likelihood function ![{p(X_1, \ldots, X_n | \theta)}](https://s0.wp.com/latex.php?latex=%7Bp%28X_1%2C+%5Cldots%2C+X_n+%7C+%5Ctheta%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) and a utility function ![{U(\theta; a)}](https://s0.wp.com/latex.php?latex=%7BU%28%5Ctheta%3B+a%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002). However, we do not assume the existence of a prior on ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002), and instead choose the decision rule ![{a(X_1, \ldots, X_n)}](https://s0.wp.com/latex.php?latex=%7Ba%28X_1%2C+%5Cldots%2C+X_n%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) that maximizes

![\displaystyle \min\limits_{\theta} \mathbb{E}[U(a(X_1,\ldots,X_n); \theta) \mid \theta]. \ \ \ \ \ (1)](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle+%5Cmin%5Climits_%7B%5Ctheta%7D+%5Cmathbb%7BE%7D%5BU%28a%28X_1%2C%5Cldots%2CX_n%29%3B+%5Ctheta%29+%5Cmid+%5Ctheta%5D.+%5C+%5C+%5C+%5C+%5C+%281%29&bg=f0f0f0&fg=000000&s=0&c=20201002)

In other words, we ask for a worst case guarantee rather than an average case guarantee. As an example of how these would differ, imagine a scenario where we have no data to observe, an unknown ![{\theta \in \{1,\ldots,N\}}](https://s0.wp.com/latex.php?latex=%7B%5Ctheta+%5Cin+%5C%7B1%2C%5Cldots%2CN%5C%7D%7D&bg=f0f0f0&fg=000000&s=0&c=20201002), and we choose an action ![{a \in \{0,\ldots,N\}}](https://s0.wp.com/latex.php?latex=%7Ba+%5Cin+%5C%7B0%2C%5Cldots%2CN%5C%7D%7D&bg=f0f0f0&fg=000000&s=0&c=20201002). Furthermore, ![{U(0; \theta) = 0}](https://s0.wp.com/latex.php?latex=%7BU%280%3B+%5Ctheta%29+%3D+0%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) for all ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002), ![{U(a; \theta) = -1}](https://s0.wp.com/latex.php?latex=%7BU%28a%3B+%5Ctheta%29+%3D+-1%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) if ![{a = \theta}](https://s0.wp.com/latex.php?latex=%7Ba+%3D+%5Ctheta%7D&bg=f0f0f0&fg=000000&s=0&c=20201002), and ![{U(a;\theta) = 1}](https://s0.wp.com/latex.php?latex=%7BU%28a%3B%5Ctheta%29+%3D+1%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) if ![{a \neq 0}](https://s0.wp.com/latex.php?latex=%7Ba+%5Cneq+0%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) and ![{a \neq \theta}](https://s0.wp.com/latex.php?latex=%7Ba+%5Cneq+%5Ctheta%7D&bg=f0f0f0&fg=000000&s=0&c=20201002). Then a frequentist will always choose ![{a = 0}](https://s0.wp.com/latex.php?latex=%7Ba+%3D+0%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) because any other action gets ![{-1}](https://s0.wp.com/latex.php?latex=%7B-1%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) utility in the worst case; a Bayesian, on the other hand, will happily choose any non-zero value of ![{a}](https://s0.wp.com/latex.php?latex=%7Ba%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) since such an action gains ![{\frac{N-2}{N}}](https://s0.wp.com/latex.php?latex=%7B%5Cfrac%7BN-2%7D%7BN%7D%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) utility in expectation. (I am purposely ignoring more complex ideas like mixed strategies for the purpose of illustration.).

Note that the frequentist optimization problem is more complicated than in the Bayesian case, since the value of ([1](https://jsteinhardt.wordpress.com/feed/?paged=3#eqn1)) depends on the joint behavior of ![{a(X_1,\ldots,X_n)}](https://s0.wp.com/latex.php?latex=%7Ba%28X_1%2C%5Cldots%2CX_n%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002), whereas with Bayes we can optimize ![{a(X_1,\ldots,X_n)}](https://s0.wp.com/latex.php?latex=%7Ba%28X_1%2C%5Cldots%2CX_n%29%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) for each set of observations separately.

As a result of this more complex optimization problem, it is often not actually possible to maximize ([1](https://jsteinhardt.wordpress.com/feed/?paged=3#eqn1)), so many frequentist techniques instead develop tools to lower-bound ([1](https://jsteinhardt.wordpress.com/feed/?paged=3#eqn1)) for a given decision procedure, and then try to construct a decision procedure that is reasonably close to the optimum. Support vector machines [2], which try to pick separating hyperplanes that minimize generalization error, are one example of this where the algorithm is explicitly trying to maximize worst-case utility. Another example of a frequentist decision procedure is L1-regularized least squares for sparse recovery [3], where the procedure itself does not look like it is explicitly maximizing any utility function, but a separate analysis shows that it is close to the optimal procedure anyways.

The second sort of frequentist approach to statistics is what I call a *frequentist guarantee*. A frequentist guarantee on an algorithm is a guarantee that, with high probability with respect to how the data was generated, the output of the algorithm will satisfy a given property. The most familiar example of this is any algorithm that generates a frequentist confidence interval: to generate a 95% frequentist confidence interval for a parameter ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002) is to run an algorithm that outputs an interval, such that with probability at least 95% ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002) lies within the interval. An important fact about most such algorithms is that the size of the interval only grows logarithmically with the amount of confidence we require, so getting a 99.9999% confidence interval is only slightly harder than getting a 95% confidence interval (and we should probably be asking for the former whenever possible).

If we use such algorithms to test hypotheses or to test discrete properties of ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002), then we can obtain algorithms that take in probabilistically generated data and produce an output that with high probability depends *only on how the data was generated*, not on the specific random samples that were given. For instance, we can create an algorithm that takes in samples from two distributions, and is guaranteed to output 1 whenever they are the same, 0 whenever they differ by at least ![{\epsilon}](https://s0.wp.com/latex.php?latex=%7B%5Cepsilon%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) in total variational distance, and could have arbitrary output if they are different but the total variational distance is less than ![{\epsilon}](https://s0.wp.com/latex.php?latex=%7B%5Cepsilon%7D&bg=f0f0f0&fg=000000&s=0&c=20201002). This is an amazing property — it takes in random input and produces an essentially deterministic answer.

Finally, a third type of frequentist approach seeks to construct *analysis tools* for understanding the behavior of random variables. Metric entropy, the Chernoff and Azuma-Hoeffding bounds [12], and Doob’s optional stopping theorem are representative examples of this sort of approach. Arguably, everyone with the time to spare should master these techniques, since being able to analyze random variables is important no matter what approach to statistics you take. Indeed, frequentist analysis tools have no conflict at all with Bayesian methods — they simply provide techniques for understanding the behavior of the Bayesian model.

**2. Bayes vs. Other Methods**

 **2.1. Justification for Bayes**

We presented Bayesian decision theory above, but are there any reasons why we should actually use it? One commonly-given reason is that Bayesian statistics is merely the application of Bayes’ Theorem, which, being a theorem, describes the only correct way to update beliefs in response to new evidence; anything else can only be justified to the extent that it provides a good approximation to Bayesian updating. This may be true, but Bayes’ Theorem only applies if we already have a prior, and if we accept probability as the correct framework for expressing uncertain beliefs. We might want to avoid one or both of these assumptions. Bayes’ theorem also doesn’t explain why we care about expected utility as opposed to some other statistic of the distribution over utilities (although note that frequentist decision theory also tries to maximize expected utility).

One compelling answer to this is **Cox’s Theorem**, which shows that any agent must implicitly be using a probability model to make decisions, or else they can be *dutch-booked* — meaning there is a series of bets that they would be willing to make that causes them to lose money with certainty. Another answer is the **complete class theorem**, which shows that any non-Bayesian decision procedure is *strictly dominated* by a Bayesian decision procedure — meaning that the Bayesian procedure performs at least as well as the non-Bayesian procedure in all cases with certainty. In other words, if you are doing anything non-Bayesian, then either it is secretly a Bayesian procedure or there is another procedure that does strictly better than it. Finally, the **VNM Utility Theorem** states that any agent with consistent preferences over distributions of outcomes must be implicitly maximizing the expected value of some scalar-valued function, which we can then use as our choice of utility function ![{U}](https://s0.wp.com/latex.php?latex=%7BU%7D&bg=f0f0f0&fg=000000&s=0&c=20201002). These theorems, however, ignore the issue of computation — while the best decision procedure may be Bayesian, the best computationally-efficient decision procedure could easily be non-Bayesian.

Another justification for Bayes is that, in contrast to ad hoc frequentist techniques, it actually provides a general theory for constructing statistical algorithms, as well as for incorporating side information such as expert knowledge. Indeed, when trying to model complex and highly structured situations it is difficult to obtain any sort of frequentist guarantees (although analysis tools can still often be applied to gain intuition about parts of the model). A prior lets us write down the sorts of models that would allow us to capture structured situations (for instance, when trying to do language modeling or transfer learning). Non-Bayesian methods exist for these situations, but they are often ad hoc and in many cases ends up looking like an approximation to Bayes. One example of this is Kneser-Ney smoothing for n-gram models, an ad hoc algorithm that ended up being very similar to an approximate inference algorithm for the hierarchical Pitman-Yor process [15, 14, 17, 8]. This raises another important point *against* Bayes, which is that the proper Bayesian interpretation may be very mathematically complex. Pitman-Yor processes are on the cutting-edge of Bayesian nonparametric statistics, which is itself one of the more technical subfields of statistical machine learning, so it was probably much easier to come up with Kneser-Ney smoothing than to find the interpretation in terms of Pitman-Yor processes.

 **2.2. When the Justifications Fail**

The first and most common objection to Bayes is that a Bayesian method is only as good as its prior. While for simple models the performance of Bayes is relatively independent of the prior, such models can only capture data where frequentist techniques would also perform very well. For more complex (especially nonparametric) Bayesian models, the performance can depend strongly on the prior, and designing good priors is still an open problem. As one example I point to my own research on hierarchical nonparametric models, where the most straightforward attempts to build a hierarchical model lead to severe pathologies [13].

Even if a Bayesian model does have a good prior, it may be computationally intractable to perform posterior inference. For instance, structure learning in Bayesian networks is NP-hard [4], as is topic inference in the popular latent Dirichlet allocation model (and this continues to hold even if we only want to perform approximate inference). Similar stories probably hold for other common models, although a theoretical survey has yet to be made; suffice to say that in practice approximate inference remains a difficult and unsolved problem, with many models not even considered because of the apparent hopelessness of performing inference in them.

Because frequentist methods often come with an analysis of the specific algorithm being employed, they can sometimes overcome these computational issues. One example of this mentioned already is L1 regularized least squares [3]. The problem setup is that we have a linear regression task ![{Ax = b+v}](https://s0.wp.com/latex.php?latex=%7BAx+%3D+b%2Bv%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) where ![{A}](https://s0.wp.com/latex.php?latex=%7BA%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) and ![{b}](https://s0.wp.com/latex.php?latex=%7Bb%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) are known, ![{v}](https://s0.wp.com/latex.php?latex=%7Bv%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) is a noise vector, and ![{x}](https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) is believed to be sparse (typically ![{x}](https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) has many more rows than ![{b}](https://s0.wp.com/latex.php?latex=%7Bb%7D&bg=f0f0f0&fg=000000&s=0&c=20201002), so without the sparsity assumption ![{x}](https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) would be underdetermined). Let us suppose that ![{x}](https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) has ![{n}](https://s0.wp.com/latex.php?latex=%7Bn%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) rows and ![{k}](https://s0.wp.com/latex.php?latex=%7Bk%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) non-zero rows — then the number of possible sparsity patterns is ![{\binom{n}{k}}](https://s0.wp.com/latex.php?latex=%7B%5Cbinom%7Bn%7D%7Bk%7D%7D&bg=f0f0f0&fg=000000&s=0&c=20201002) — large enough that a brute force consideration of all possible sparsity patterns is intractable. However, we can show that solving a certain semidefinite program will with high probability yield the appropriate sparsity pattern, after which recovering x reduces to a simple least squares problem. (A *semidefinite program* is a certain type of optimization problem that can be solved efficiently [16].)

Finally, Bayes has no good way of dealing with adversaries or with cases where the data was generated in a complicated way that could make it highly biased (for instance, as the output of an optimization procedure). A toy example of an adversary would be playing rock-paper-scissors — how should a Bayesian play such a game? The straightforward answer is to build up a model of the opponent based on their plays so far, and then to make the play that maximizes the expected score (probability of winning minus probability of losing). However, such a strategy fares poorly against any opponent with access to the model being used, as they can then just run the model themselves to predict the Bayesian’s plays in advance, thereby winning every single time. In contrast, there is a frequentist strategy called the **multiplicative weights update method** that fairs well against an arbitrary opponent (even one with superior computational resources and access to our agent’s source code). The multiplicative weights method does far more than winning at rock-paper-scissors — it is also a key component of the fastest algorithm for solving many important optimization problems (including the network flow algorithm), and it forms the theoretical basis for the widely used AdaBoost algorithm [1, 5, 7].

 **2.3. When To Use Each Method**

The essential difference between Bayesian and frequentist decision theory is that Bayes makes the additional assumption of a prior over ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=000000&s=0&c=20201002), and optimizes for average-case performance rather than worst-case performance. *It follows, then, that Bayes is the superior method whenever we can obtain a good prior and when good average-case performance is sufficient.* However, if we have no way of obtaining a good prior, or when we need guaranteed performance, frequentist methods are the way to go. For instance, if we are trying to build a software package that should be widely deployable, we might want to use a frequentist method because users can be sure that the software will work as long as some number of easily-checkable assumptions are met.

A nice middle-ground between purely Bayesian and purely frequentist methods is to use a Bayesian model coupled with frequentist model-checking techniques; this gives us the freedom in modeling afforded by a prior but also gives us some degree of confidence that our model is correct. This approach is suggested by both Gelman [9] and Jordan [10].

**3. Conclusion**

When the assumptions of Bayes’ Theorem hold, and when Bayesian updating can be performed computationally efficiently, then it is indeed tautological that Bayes is the optimal approach. Even when some of these assumptions fail, Bayes can still be a fruitful approach. However, by working under weaker (sometimes even adversarial) assumptions, frequentist approaches can perform well in very complicated domains even with fairly simple models; this is because, with fewer assumptions being made at the outset, less work has to be done to ensure that those assumptions are met.

From a research perspective, we should be far from satisfied with either approach — Bayesian methods make stronger assumptions than may be warranted, and frequentists methods provide little in the way of a coherent framework for constructing models, and ask for worst-case guarantees, which probably cannot be obtained in general. We should seek to develop a statistical modeling framework that, unlike Bayes, can deal with unknown priors, adversaries, and limited computational resources.

**4. Acknowledgements**

Thanks to Emma Pierson, Vladimir Slepnev, and Wei Dai for reading preliminary versions of this work and providing many helpful comments.

**5. References**

[1] Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta algorithm and applications. *Working Paper*, 2005.

[2] Christopher J.C. Burges. A tutorial on support vector machines for pattern recognition. *Data Mining and Knowledge Discovery*, 2:121–167, 1998.

[3] Emmanuel J. Candes. Compressive sampling. In *Proceedings of the International Congress of Mathematicians*. European Mathematical Society, 2006.

[4] D.M. Chickering. Learning bayesian networks is NP-complete. *LECTURE NOTES IN STATISTICS-NEW YORK-SPRINGER VERLAG-*, pages 121–130, 1996.

[5] Paul Christiano, Jonathan A. Kelner, Aleksander Madry, Daniel Spielman, and Shang-Hua Teng. Electrical flows, laplacian systems, and faster approximation of maximum flow in undirected graphs. In *Proceedings of the 43rd ACM Symposium on Theory of Computing*, 2011.

[6] Andrew Critch. Frequentist vs. bayesian breakdown: Interpretation vs. inference. http://lesswrong.com/lw/7ck/frequentist\_vs\_bayesian\_breakdown\_interpretation/.

[7] Yoav Freund and Robert E. Schapire. A short introduction to boosting. *Journal of Japanese Society for Artificial Intelligence*, 14(5):771–780, Sep. 1999.

[8] J. Gasthaus and Y.W. Teh. Improvements to the sequence memoizer. In *Advances in Neural Information Processing Systems*, 2011.

[9] Andrew Gelman. Induction and deduction in bayesian data analysis. *RMM*, 2:67–78, 2011.

[10] Michael I. Jordan. Are you a bayesian or a frequentist? Machine Learning Summer School 2009 (video lecture at <http://videolectures.net/mlss09uk_jordan_bfway/>).

[11] D. Warner North. A tutorial introduction to decision theory. *IEEE Transactions on Systems Science and Cybernetics*, SSC-4(3):200–210, Sep. 1968.

[12] Igal Sason. On refined versions of the Azuma-Hoeffding inequality with applications in information theory. *CoRR*, abs/1111.1977, 2011.

[13] Jacob Steinhardt and Zoubin Ghahramani. Pathological properties of deep bayesian hierarchies. In *NIPS Workshop on Bayesian Nonparametrics*, 2011. Extended Abstract.

[14] Y.W. Teh. A bayesian interpretation of interpolated Kneser-Ney. Technical Report TRA2/06, School of Computing, NUS, 2006.

[15] Y.W. Teh. A hierarchical bayesian language model based on pitman-yor processes. *Coling/ACL*, 2006.

[16] Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. *SIAM Review*, 38(1):49–95, Mar. 1996.

[17] F.~Wood, C.~Archambeau, J.~Gasthaus, L.~James, and Y.W. Teh. A stochastic memoizer for sequence data. In *Proceedings of the 26th International Conference on Machine Learning*, pages 1129–1136, 2009.

Beyond Bayesians and Frequentists


Verifying Stability of Stochastic Systems

I just finished presenting my recent paper on stochastic verification at RSS 2011. There is a [conference version online](http://www.roboticsproceedings.org/rss07/p41.html), with a journal article to come later. In this post I want to go over the problem statement and my solution.

**Problem Statement**

Abstractly, the goal is to be given some sort of description of a system, and of a goal for that system, and then verify that the system will reach that goal. The difference between our work and a lot (but not all) of the previous work is that we want to work with an explicit noise model for the system. So, for instance, I tell you that the system satisfies

![dx(t) = f(x) dt + g(x) dw(t),](https://s0.wp.com/latex.php?latex=dx%28t%29+%3D+f%28x%29+dt+%2B+g%28x%29+dw%28t%29%2C&bg=f0f0f0&fg=555555&s=0&c=20201002)

where ![f(x)](https://s0.wp.com/latex.php?latex=f%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) represents the nominal dynamics of the system, ![g(x)](https://s0.wp.com/latex.php?latex=g%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) represents how noise enters the system, and $dw(t)$ is a standard Wiener process (the continuous-time version of Gaussian noise). I would like to, for instance, verify that ![h(x(T)) < 0](https://s0.wp.com/latex.php?latex=h%28x%28T%29%29+%3C+0&bg=f0f0f0&fg=555555&s=0&c=20201002) for some function ![h](https://s0.wp.com/latex.php?latex=h&bg=f0f0f0&fg=555555&s=0&c=20201002) and some final time ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002). For example, if ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) is one-dimensional then I could ask that ![x(10)^2-1 < 0](https://s0.wp.com/latex.php?latex=x%2810%29%5E2-1+%3C+0&bg=f0f0f0&fg=555555&s=0&c=20201002), which is asking for ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) to be within a distance of ![1](https://s0.wp.com/latex.php?latex=1&bg=f0f0f0&fg=555555&s=0&c=20201002) of the origin at time ![10](https://s0.wp.com/latex.php?latex=10&bg=f0f0f0&fg=555555&s=0&c=20201002). For now, I will focus on time-invariant systems and stability conditions. This means that ![f](https://s0.wp.com/latex.php?latex=f&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![g](https://s0.wp.com/latex.php?latex=g&bg=f0f0f0&fg=555555&s=0&c=20201002) are not functions of ![t](https://s0.wp.com/latex.php?latex=t&bg=f0f0f0&fg=555555&s=0&c=20201002), and the condition we want to verify is that ![h(x(t)) < 0](https://s0.wp.com/latex.php?latex=h%28x%28t%29%29+%3C+0&bg=f0f0f0&fg=555555&s=0&c=20201002) for all ![t \in [0,T]](https://s0.wp.com/latex.php?latex=t+%5Cin+%5B0%2CT%5D&bg=f0f0f0&fg=555555&s=0&c=20201002). However, it is not too difficult to extend these ideas to the time-varying case, as I will show in the results at the end.

The tool we will use for our task is a *supermartingale*, which allows us to prove bounds on the probability that a system leaves a certain region.

**Supermartingales**

Let us suppose that I have a non-negative function ![V](https://s0.wp.com/latex.php?latex=V&bg=f0f0f0&fg=555555&s=0&c=20201002) of my state ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) such that ![\mathbb{E}[\dot{V}(x(t))] \leq c](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%28t%29%29%5D+%5Cleq+c&bg=f0f0f0&fg=555555&s=0&c=20201002) for all ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![t](https://s0.wp.com/latex.php?latex=t&bg=f0f0f0&fg=555555&s=0&c=20201002). Here we define ![\mathbb{E}[\dot{V}(x(t))]](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%28t%29%29%5D&bg=f0f0f0&fg=555555&s=0&c=20201002) as

![\lim\limits_{\Delta t \to 0^+} \frac{\mathbb{E}[V(x(t+\Delta t)) \mid x(t)]-V(x(t))}{\Delta t}.](https://s0.wp.com/latex.php?latex=%5Clim%5Climits_%7B%5CDelta+t+%5Cto+0%5E%2B%7D+%5Cfrac%7B%5Cmathbb%7BE%7D%5BV%28x%28t%2B%5CDelta+t%29%29+%5Cmid+x%28t%29%5D-V%28x%28t%29%29%7D%7B%5CDelta+t%7D.&bg=f0f0f0&fg=555555&s=0&c=20201002)

Then, just by integrating, we can see that ![\mathbb{E}[V(x(t)) \mid x(0)] \leq V(x(0))+ct](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5BV%28x%28t%29%29+%5Cmid+x%280%29%5D+%5Cleq+V%28x%280%29%29%2Bct&bg=f0f0f0&fg=555555&s=0&c=20201002). By Markov’s inequality, the probability that ![V(x(t)) \geq \rho](https://s0.wp.com/latex.php?latex=V%28x%28t%29%29+%5Cgeq+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) is at most ![\frac{V(x(0))+ct}{\rho}](https://s0.wp.com/latex.php?latex=%5Cfrac%7BV%28x%280%29%29%2Bct%7D%7B%5Crho%7D&bg=f0f0f0&fg=555555&s=0&c=20201002).

We can actually prove something stronger as follows: note that if we re-define our Markov process to stop evolving as soon as ![V(x(t)) = \rho](https://s0.wp.com/latex.php?latex=V%28x%28t%29%29+%3D+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002), then this only sets ![\mathbb{E}[\dot{V}]](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D&bg=f0f0f0&fg=555555&s=0&c=20201002) to zero in certain places. Thus the probability that ![V(x(t)) \geq \rho](https://s0.wp.com/latex.php?latex=V%28x%28t%29%29+%5Cgeq+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) for this new process is at most ![\frac{V(x(0))+\max(c,0)t}{\rho}](https://s0.wp.com/latex.php?latex=%5Cfrac%7BV%28x%280%29%29%2B%5Cmax%28c%2C0%29t%7D%7B%5Crho%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Since the process stops as soon as ![V(x) \geq \rho](https://s0.wp.com/latex.php?latex=V%28x%29+%5Cgeq+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002), we obtain the stronger result that the probability that ![V(x(s)) \geq \rho](https://s0.wp.com/latex.php?latex=V%28x%28s%29%29+%5Cgeq+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) for *any* ![s \in [0,t]](https://s0.wp.com/latex.php?latex=s+%5Cin+%5B0%2Ct%5D&bg=f0f0f0&fg=555555&s=0&c=20201002) is at most ![\frac{V(x(0))+\max(c,0)t}{\rho}](https://s0.wp.com/latex.php?latex=%5Cfrac%7BV%28x%280%29%29%2B%5Cmax%28c%2C0%29t%7D%7B%5Crho%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Finally, we only need the condition ![\mathbb{E}[\dot{V}] \leq c](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D+%5Cleq+c&bg=f0f0f0&fg=555555&s=0&c=20201002) to hold when ![V(x) < \rho](https://s0.wp.com/latex.php?latex=V%28x%29+%3C+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002). We thus obtain the following:

**Theorem.** Let ![V(x)](https://s0.wp.com/latex.php?latex=V%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) be a non-negative function such that ![\mathbb{E}[\dot{V}(x(t))] \leq c](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%28t%29%29%5D+%5Cleq+c&bg=f0f0f0&fg=555555&s=0&c=20201002) whenever ![V(x) < \rho](https://s0.wp.com/latex.php?latex=V%28x%29+%3C+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002). Then with probability at least ![1-\frac{V(x(0))+\max(c,0)T}{\rho}](https://s0.wp.com/latex.php?latex=1-%5Cfrac%7BV%28x%280%29%29%2B%5Cmax%28c%2C0%29T%7D%7B%5Crho%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), ![V(x(t)) < \rho](https://s0.wp.com/latex.php?latex=V%28x%28t%29%29+%3C+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) for all ![t \in [0,T)](https://s0.wp.com/latex.php?latex=t+%5Cin+%5B0%2CT%29&bg=f0f0f0&fg=555555&s=0&c=20201002).

We call the condition ![\mathbb{E}[\dot{V}] \leq c](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D+%5Cleq+c&bg=f0f0f0&fg=555555&s=0&c=20201002) the *supermartingale condition*, and a function that satisfies the martingale condition is called a *supermartingale*. If we can construct supermartingales for our system, then we can bound the probability that trajectories of the system leave a given region.

NOTE: for most people, a supermartingale is something that satisfies the condition ![\mathbb{E}[\dot{V}] \leq 0](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D+%5Cleq+0&bg=f0f0f0&fg=555555&s=0&c=20201002). However, this condition is often impossible to satisfy for systems we might care about. For instance, just consider exponential decay driven by Gaussian noise:

[![noise](https://i0.wp.com/s4.postimage.org/ep072ktgn/noise.png)](http://postimage.org/image/ikg7ci90/full/)

Once the system gets close enough to the origin, the exponential decay ceases to matter much and the system is basically just getting bounced around by the Gaussian noise. In particular, if the system is ever at the origin, it will get perturbed away again, so you cannot hope to find a non-constant function of ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) that is decreasing in expectation everywhere (~~just consider the global minimum of such a function: in all cases, there is a non-zero probability that the Gaussian noise will cause ![V(x)](https://s0.wp.com/latex.php?latex=V%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) to increase, but a zero probability that ![V(x)](https://s0.wp.com/latex.php?latex=V%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) will decrease because we are already at the global minimum~~ this argument doesn’t actually work, but I am pretty sure that my claim is true at least subject to sufficient technical conditions).

**Applying the Martingale Theorem**

Now that we have this theorem, we need some way to actually use it. First, let us try to get a more explicit version of the Martingale condition for the systems we are considering, which you will recall are of the form ![dx(t) = f(x) dt + g(x) dw(t)](https://s0.wp.com/latex.php?latex=dx%28t%29+%3D+f%28x%29+dt+%2B+g%28x%29+dw%28t%29&bg=f0f0f0&fg=555555&s=0&c=20201002). Note that ![V(x+\Delta x) = V(x) + \frac{\partial V}{\partial x} \Delta x + \frac{1}{2}Trace\left(\Delta x^T \frac{\partial^2 V}{\partial x^2}\Delta x\right)+O(\Delta x^3)](https://s0.wp.com/latex.php?latex=V%28x%2B%5CDelta+x%29+%3D+V%28x%29+%2B+%5Cfrac%7B%5Cpartial+V%7D%7B%5Cpartial+x%7D+%5CDelta+x+%2B+%5Cfrac%7B1%7D%7B2%7DTrace%5Cleft%28%5CDelta+x%5ET+%5Cfrac%7B%5Cpartial%5E2+V%7D%7B%5Cpartial+x%5E2%7D%5CDelta+x%5Cright%29%2BO%28%5CDelta+x%5E3%29&bg=f0f0f0&fg=555555&s=0&c=20201002).

Then ![\mathbb{E}[\dot{V}(x)] = \lim_{\Delta t \to 0^+} \frac{\frac{\partial V}{\partial x} \mathbb{E}[\Delta x]+\frac{1}{2}Trace\left(\mathbb{E}[\Delta x^T \frac{\partial^2 V}{\partial x^2}\Delta x\right)+O(\Delta x^3)}{\Delta t}](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%29%5D+%3D+%5Clim_%7B%5CDelta+t+%5Cto+0%5E%2B%7D+%5Cfrac%7B%5Cfrac%7B%5Cpartial+V%7D%7B%5Cpartial+x%7D+%5Cmathbb%7BE%7D%5B%5CDelta+x%5D%2B%5Cfrac%7B1%7D%7B2%7DTrace%5Cleft%28%5Cmathbb%7BE%7D%5B%5CDelta+x%5ET+%5Cfrac%7B%5Cpartial%5E2+V%7D%7B%5Cpartial+x%5E2%7D%5CDelta+x%5Cright%29%2BO%28%5CDelta+x%5E3%29%7D%7B%5CDelta+t%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). A Wiener process satisfies ![\mathbb{E}[dw(t)] = 0](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5Bdw%28t%29%5D+%3D+0&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![\mathbb{E}[dw(t)^2] = dt](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5Bdw%28t%29%5E2%5D+%3D+dt&bg=f0f0f0&fg=555555&s=0&c=20201002), so only the nominal dynamics (![f(x)](https://s0.wp.com/latex.php?latex=f%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002)) affect the limit of the first-order term while only the noise (![g(x)](https://s0.wp.com/latex.php?latex=g%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002)) affects the limit of the second-order term (the third-order and higher terms in ![\Delta x](https://s0.wp.com/latex.php?latex=%5CDelta+x&bg=f0f0f0&fg=555555&s=0&c=20201002) all go to zero). We thus end up with the formula

![\mathbb{E}[\dot{V}(x)] = \frac{\partial V}{\partial x}f(x)+\frac{1}{2}Trace\left(g(x)^T\frac{\partial^2 V}{\partial x^2}g(x)\right).](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%29%5D+%3D+%5Cfrac%7B%5Cpartial+V%7D%7B%5Cpartial+x%7Df%28x%29%2B%5Cfrac%7B1%7D%7B2%7DTrace%5Cleft%28g%28x%29%5ET%5Cfrac%7B%5Cpartial%5E2+V%7D%7B%5Cpartial+x%5E2%7Dg%28x%29%5Cright%29.&bg=f0f0f0&fg=555555&s=0&c=20201002)

It is not that difficult to construct a supermartingale, but most supermartingales that you construct will yield a pretty poor bound. To illustrate this, consider the system ![dx(t) = -x dt + dw(t)](https://s0.wp.com/latex.php?latex=dx%28t%29+%3D+-x+dt+%2B+dw%28t%29&bg=f0f0f0&fg=555555&s=0&c=20201002). This is the example in the image from the previous section. Now consider a quadratic function ![V(x) = x^2](https://s0.wp.com/latex.php?latex=V%28x%29+%3D+x%5E2&bg=f0f0f0&fg=555555&s=0&c=20201002). The preceding formula tells us that ![\mathbb{E}[\dot{V}(x)] = -2x^2+1](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%29%5D+%3D+-2x%5E2%2B1&bg=f0f0f0&fg=555555&s=0&c=20201002). We thus have ![\mathbb{E}[\dot{V}(x)] \leq 1](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%29%5D+%5Cleq+1&bg=f0f0f0&fg=555555&s=0&c=20201002), which means that the probability of leaving the region ![x^2 < \rho](https://s0.wp.com/latex.php?latex=x%5E2+%3C+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) is at most ![\frac{x(0)^2+T}{\rho}](https://s0.wp.com/latex.php?latex=%5Cfrac%7Bx%280%29%5E2%2BT%7D%7B%5Crho%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). This is not particularly impressive: it says that we should expect ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) to grow roughly as ![\sqrt{T}](https://s0.wp.com/latex.php?latex=%5Csqrt%7BT%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), which is how quickly ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) would grow if it was a random walk with no stabilizing component at all.

One way to deal with this is to have a state-dependent bound $\mathbb{E}[\dot{V}] \leq c-kV$. This has been considered for instance by [Pham, Tabareau, and Slotine](http://web.mit.edu/nsl/www/preprints/stochastic_contraction09.pdf) (see Lemma 2 and Theorem 2), but I am not sure whether their results still work if the supermartingale condition only holds locally instead of globally; I haven’t spent much time on this, so they could generalize quite trivially.

Another way to deal with this is to pick a more quickly-growing candidate supermartingale. For instance, we could pick ![V(x) = x^4](https://s0.wp.com/latex.php?latex=V%28x%29+%3D+x%5E4&bg=f0f0f0&fg=555555&s=0&c=20201002). Then ![\mathbb{E}[\dot{V}] = -4x^4+6x^2](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D+%3D+-4x%5E4%2B6x%5E2&bg=f0f0f0&fg=555555&s=0&c=20201002), which has a global maximum of ![\frac{9}{4}](https://s0.wp.com/latex.php?latex=%5Cfrac%7B9%7D%7B4%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) at $x = \frac{\sqrt{3}}{2}$. This bounds then says that ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) grows at a rate of at most ![T^{\frac{1}{4}}](https://s0.wp.com/latex.php?latex=T%5E%7B%5Cfrac%7B1%7D%7B4%7D%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), which is better than before, but still much worse than reality.

We could keep improving on this bound by considering successively faster-growing polynomials. However, automating such a process becomes expensive once the degree of the polynomial gets large. Instead, let’s consider a function like ![V(x) = e^{0.5x^2}](https://s0.wp.com/latex.php?latex=V%28x%29+%3D+e%5E%7B0.5x%5E2%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Then ![\mathbb{E}[\dot{V}] = e^{0.5x^2}(0.5-0.5x^2)](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D+%3D+e%5E%7B0.5x%5E2%7D%280.5-0.5x%5E2%29&bg=f0f0f0&fg=555555&s=0&c=20201002), which has a maximum of 0.5 at x=0. Now our bound says that we should expect x to grow like ![\sqrt{\log(T)}](https://s0.wp.com/latex.php?latex=%5Csqrt%7B%5Clog%28T%29%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), which is a much better growth rate (and roughly the true growth rate, at least in terms of the largest value of ![|x|](https://s0.wp.com/latex.php?latex=%7Cx%7C&bg=f0f0f0&fg=555555&s=0&c=20201002) over the time interval ![[0,T]](https://s0.wp.com/latex.php?latex=%5B0%2CT%5D&bg=f0f0f0&fg=555555&s=0&c=20201002)).

This leads us to our overall strategy for finding good supermartingales. We will search across functions of the form ![V(x) = e^{x^TSx}](https://s0.wp.com/latex.php?latex=V%28x%29+%3D+e%5E%7Bx%5ETSx%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) where ![S \succeq 0](https://s0.wp.com/latex.php?latex=S+%5Csucceq+0&bg=f0f0f0&fg=555555&s=0&c=20201002) is a matrix (the ![\succeq](https://s0.wp.com/latex.php?latex=%5Csucceq&bg=f0f0f0&fg=555555&s=0&c=20201002) means “positive semidefinite”, which roughly means that the graph of the function ![x^TSx](https://s0.wp.com/latex.php?latex=x%5ETSx&bg=f0f0f0&fg=555555&s=0&c=20201002) looks like a bowl rather than a saddle/hyperbola). This begs two questions: how to upper-bound the global maximum of ![\mathbb{E}[\dot{V}]](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D&bg=f0f0f0&fg=555555&s=0&c=20201002) for this family, and how to search efficiently over this family. The former is done by doing some careful work with inequalities, while the latter is done with semidefinite programming. I will explain both below.

**Upper-bounding ![\mathbb{E}[\dot{V}]](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D&bg=f0f0f0&fg=555555&s=0&c=20201002)**

In general, if ![V(x) = e^{x^TSx}](https://s0.wp.com/latex.php?latex=V%28x%29+%3D+e%5E%7Bx%5ETSx%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), then ![\mathbb{E}[\dot{V}(x)] = e^{x^TSx}\left(2x^TSf(x)+Trace(g(x)^TSg(x))+2x^TSg(x)g(x)^TSx\right)](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%28x%29%5D+%3D+e%5E%7Bx%5ETSx%7D%5Cleft%282x%5ETSf%28x%29%2BTrace%28g%28x%29%5ETSg%28x%29%29%2B2x%5ETSg%28x%29g%28x%29%5ETSx%5Cright%29&bg=f0f0f0&fg=555555&s=0&c=20201002). We would like to show that such a function is upper-bounded by a constant ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002). To do this, move the exponential term to the right-hand-side to get the equivalent condition ![2x^TSf(x)+Trace(g(x)^TSg(x))+2x^TSg(x)g(x)^TSx \leq ce^{-x^TSx}](https://s0.wp.com/latex.php?latex=2x%5ETSf%28x%29%2BTrace%28g%28x%29%5ETSg%28x%29%29%2B2x%5ETSg%28x%29g%28x%29%5ETSx+%5Cleq+ce%5E%7B-x%5ETSx%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Then we can lower-bound ![e^{-x^TSx}](https://s0.wp.com/latex.php?latex=e%5E%7B-x%5ETSx%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) by ![1-x^TSx](https://s0.wp.com/latex.php?latex=1-x%5ETSx&bg=f0f0f0&fg=555555&s=0&c=20201002) and obtain the sufficient condition

![c(1-x^TSx)-2x^TSf(x)-Trace(g(x)^TSg(x))-2x^TSg(x)g(x)^TSx \geq 0.](https://s0.wp.com/latex.php?latex=c%281-x%5ETSx%29-2x%5ETSf%28x%29-Trace%28g%28x%29%5ETSg%28x%29%29-2x%5ETSg%28x%29g%28x%29%5ETSx+%5Cgeq+0.&bg=f0f0f0&fg=555555&s=0&c=20201002)

It is still not immediately clear how to check such a condition, but somehow the fact that this new condition only involves polynomials (assuming that f and g are polynomials) seems like it should make computations more tractable. This is indeed the case. While checking if a polynomial is positive is NP-hard, checking whether it is a **sum of squares** of other polynomials can be done in polynomial time. While sum of squares is not the same as positive, it is a sufficient condition (since the square of a real number is always positive).

The way we check whether a polynomial p(x) is a sum of squares is to formulate it as the semidefinite program: ![p(x) = z^TMz, M \succeq 0](https://s0.wp.com/latex.php?latex=p%28x%29+%3D+z%5ETMz%2C+M+%5Csucceq+0&bg=f0f0f0&fg=555555&s=0&c=20201002), where ![z](https://s0.wp.com/latex.php?latex=z&bg=f0f0f0&fg=555555&s=0&c=20201002) is a vector of monomials. The condition ![p(x) = z^TMz](https://s0.wp.com/latex.php?latex=p%28x%29+%3D+z%5ETMz&bg=f0f0f0&fg=555555&s=0&c=20201002) is a set of affine constraints on the entries of ![M](https://s0.wp.com/latex.php?latex=M&bg=f0f0f0&fg=555555&s=0&c=20201002), so that the above program is indeed semidefinite and can be solved efficiently.

**Efficiently searching across all matrices S**

We can extend on the sum-of-squares idea in the previous section to search over ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002). Note that if ![p](https://s0.wp.com/latex.php?latex=p&bg=f0f0f0&fg=555555&s=0&c=20201002) is a parameterized polynomial whose coefficient are affine in a set of decision variables, then the condition ![p(x) = z^TMz](https://s0.wp.com/latex.php?latex=p%28x%29+%3D+z%5ETMz&bg=f0f0f0&fg=555555&s=0&c=20201002) is again a set of affine constraints on ![M](https://s0.wp.com/latex.php?latex=M&bg=f0f0f0&fg=555555&s=0&c=20201002). This almost solves our problem for us, but not quite. The issue is the form of ![p(x)](https://s0.wp.com/latex.php?latex=p%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) in our case:

![c(1-x^TSx)-2x^TSf(x)-Trace(g(x)^TSg(x))-2x^TSg(x)g(x)^TSx](https://s0.wp.com/latex.php?latex=c%281-x%5ETSx%29-2x%5ETSf%28x%29-Trace%28g%28x%29%5ETSg%28x%29%29-2x%5ETSg%28x%29g%28x%29%5ETSx&bg=f0f0f0&fg=555555&s=0&c=20201002)

Do you see the problem? There are two places where the constraints do not appear linearly in the decision variables: ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002) multiply each other in the first term, and ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002) appears quadratically in the last term. While the first non-linearity is not so bad (![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) is a scalar so it is relatively cheap to search over ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) exhaustively), the second non-linearity is more serious. Fortunately, we can resolve the issue with Schur complements. The idea behind Schur complements is that, assuming ![A \succeq 0](https://s0.wp.com/latex.php?latex=A+%5Csucceq+0&bg=f0f0f0&fg=555555&s=0&c=20201002), the condition ![B^TA^{-1}B \preceq C](https://s0.wp.com/latex.php?latex=B%5ETA%5E%7B-1%7DB+%5Cpreceq+C&bg=f0f0f0&fg=555555&s=0&c=20201002) is equivalent to ![\left[ \begin{array}{cc} A & B \\ B^T & C \end{array} \right] \succeq 0](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+A+%26+B+%5C%5C+B%5ET+%26+C+%5Cend%7Barray%7D+%5Cright%5D+%5Csucceq+0&bg=f0f0f0&fg=555555&s=0&c=20201002). In our case, this means that our condition is equivalent to the condition that

![\left[ \begin{array}{cc} 0.5I & g^TSx \\ x^TSg & c(1-x^TSx)-2x^TSf(x)-Trace(g(x)^TSg(x))\end{array} \right] \succeq 0](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+0.5I+%26+g%5ETSx+%5C%5C+x%5ETSg+%26+c%281-x%5ETSx%29-2x%5ETSf%28x%29-Trace%28g%28x%29%5ETSg%28x%29%29%5Cend%7Barray%7D+%5Cright%5D+%5Csucceq+0&bg=f0f0f0&fg=555555&s=0&c=20201002)

where ![I](https://s0.wp.com/latex.php?latex=I&bg=f0f0f0&fg=555555&s=0&c=20201002) is the identity matrix. Now we have a condition that is linear in the decision variable ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002), but it is no longer a polynomial condition, it is a condition that a matrix polynomial be positive semidefinite. Fortunately, we can reduce this to a purely polynomial condition by creating a set of dummy variables ![y](https://s0.wp.com/latex.php?latex=y&bg=f0f0f0&fg=555555&s=0&c=20201002) and asking that

![y^T\left[ \begin{array}{cc} 0.5I & g^TSx \\ x^TSg & c(1-x^TSx)-2x^TSf(x)-Trace(g(x)^TSg(x))\end{array} \right]y \geq 0](https://s0.wp.com/latex.php?latex=y%5ET%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+0.5I+%26+g%5ETSx+%5C%5C+x%5ETSg+%26+c%281-x%5ETSx%29-2x%5ETSf%28x%29-Trace%28g%28x%29%5ETSg%28x%29%29%5Cend%7Barray%7D+%5Cright%5Dy+%5Cgeq+0&bg=f0f0f0&fg=555555&s=0&c=20201002)

We can then do a line search over ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) and solve a semidefinite program to determine a feasible value of ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002). If we care about remaining within a specific region, we can maximize ![\rho](https://s0.wp.com/latex.php?latex=%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) such that ![x^TSx < \rho](https://s0.wp.com/latex.php?latex=x%5ETSx+%3C+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) implies that we stay in the region. Since our bound on the probability of leaving the grows roughly as ![ce^{-\rho}T](https://s0.wp.com/latex.php?latex=ce%5E%7B-%5Crho%7DT&bg=f0f0f0&fg=555555&s=0&c=20201002), this is a pretty reasonable thing to maximize (we would actually want to maximize ![\rho-\log(c)](https://s0.wp.com/latex.php?latex=%5Crho-%5Clog%28c%29&bg=f0f0f0&fg=555555&s=0&c=20201002), but this is a bit more difficult to do).

Oftentimes, for instance if we are verifying stability around a trajectory, we would like ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) to be time-varying. In this case an exhaustive search is no longer feasible. Instead we alternate between searching over ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002) and searching over ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002). In the step where we search over ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002), we maximize ![\rho](https://s0.wp.com/latex.php?latex=%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002). In the step where we search over ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002), we maximize the amount by which we could change ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) and still satisfy the constraints (the easiest way to do this is by first maximizing ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002), then minimizing ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002), then taking the average; the fact that semidefinite constraints are convex implies that this optimizes the margin on ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) for a fixed ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002)).

A final note is that systems are often only stable locally, and so we only want to check the constraint ![\mathbb{E}[\dot{V}] \leq c](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5B%5Cdot%7BV%7D%5D+%5Cleq+c&bg=f0f0f0&fg=555555&s=0&c=20201002) in a region where ![V(x) < \rho](https://s0.wp.com/latex.php?latex=V%28x%29+%3C+%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002). We can do this by adding a *Lagrange multiplier* to our constraints. For instance, if we want to check that ![p(x) \geq 0](https://s0.wp.com/latex.php?latex=p%28x%29+%5Cgeq+0&bg=f0f0f0&fg=555555&s=0&c=20201002) whenever ![(x) \leq 0](https://s0.wp.com/latex.php?latex=%28x%29+%5Cleq+0&bg=f0f0f0&fg=555555&s=0&c=20201002), it suffices to find a polynomial ![\lambda(x)](https://s0.wp.com/latex.php?latex=%5Clambda%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002) such that ![\lambda(x) \geq 0](https://s0.wp.com/latex.php?latex=%5Clambda%28x%29+%5Cgeq+0&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![p(x)+\lambda(x)s(x) \geq 0](https://s0.wp.com/latex.php?latex=p%28x%29%2B%5Clambda%28x%29s%28x%29+%5Cgeq+0&bg=f0f0f0&fg=555555&s=0&c=20201002). (You should convince yourself that this is true; the easiest proof is just by casework on the sign of ![s(x)](https://s0.wp.com/latex.php?latex=s%28x%29&bg=f0f0f0&fg=555555&s=0&c=20201002).) This again introduces a non-linearity in the constraints, but if we fix ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![\rho](https://s0.wp.com/latex.php?latex=%5Crho&bg=f0f0f0&fg=555555&s=0&c=20201002) then the constraints are linear in ![c](https://s0.wp.com/latex.php?latex=c&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![\lambda](https://s0.wp.com/latex.php?latex=%5Clambda&bg=f0f0f0&fg=555555&s=0&c=20201002), and vice-versa, so we can perform the same alternating maximization as before.

**Results**

Below is the most exciting result, it is for an airplane with a noisy camera trying to avoid obstacles. Using the verification methods above, we can show that with probability at least ![0.99](https://s0.wp.com/latex.php?latex=0.99&bg=f0f0f0&fg=555555&s=0&c=20201002) that the plane trajectory will not leave the gray region:

[![uav](https://i0.wp.com/s4.postimage.org/b5eml2ipp/uav.png)](http://postimage.org/image/iktfna78/full/)


Useful Math

I have spent the last several months doing applied math, culminating in a submission of a paper to a [robotics conference](http://www.roboticsconference.org/) (although culminating might be the wrong word, since I’m still working on the project).

Unfortunately the review process is double-blind so I can’t talk about that specifically, but I’m more interested in going over the math I ended up using (not expositing on it, just making a list, more or less). This is meant to be a moderate amount of empirical evidence for which pieces of math are actually useful, and which aren’t (of course, the lack of appearance on this list doesn’t imply uselessness, but should be taken as [Bayesian evidence against](http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_of_absence/) usefulness).

I’ll start with the stuff that I actually used in the paper, then stuff that helped me formulate the ideas in the paper, then stuff that I’ve used in other work that hasn’t yet come to fruition. These will be labelled I, II, and III below. Let me know if you think something should be in III that isn’t [in other words, you think there’s a piece of math that is useful but not listed here, preferably with the application you have in mind], or if you have better links to any of the topics below.

I. Ideas used directly

Optimization: [semidefinite optimization](http://en.wikipedia.org/wiki/Semidefinite_programming), [convex optimization](http://en.wikipedia.org/wiki/Convex_optimization), [sum-of-squares programming](http://www.cds.caltech.edu/sostools/), [Schur complements](http://en.wikipedia.org/wiki/Schur_complement#Schur_complement_condition_for_positive_definiteness), [Lagrange multipliers](http://en.wikipedia.org/wiki/Lagrange_multiplier), [KKT conditions](http://en.wikipedia.org/wiki/Karush–Kuhn–Tucker_conditions)

Differential equations: [Lyapunov functions](http://en.wikipedia.org/wiki/Lyapunov_function), [linear differential equations](http://en.wikipedia.org/wiki/Linear_differential_equation), [Poincaré return map](http://en.wikipedia.org/wiki/Poincaré_map), [exponential stability](http://en.wikipedia.org/wiki/Exponential_stability), [Ito calculus](http://en.wikipedia.org/wiki/Itō_calculus)

Linear algebra: [matrix exponential](http://en.wikipedia.org/wiki/Matrix_exponential), [trace](http://en.wikipedia.org/wiki/Trace_(linear_algebra)), [determinant](http://en.wikipedia.org/wiki/Determinant), [Cholesky decomposition](http://en.wikipedia.org/wiki/Cholesky_decomposition), plus general matrix manipulation and familiarity with [eigenvalues](http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors) and [quadratic forms](http://en.wikipedia.org/wiki/Quadratic_form)

Probability theory: [Markov’s inequality](http://en.wikipedia.org/wiki/Markov's_inequality), [linearity of expectation](http://en.wikipedia.org/wiki/Expected_value#Linearity), [martingales](http://en.wikipedia.org/wiki/Martingale_(probability_theory)), [multivariate normal distribution](http://en.wikipedia.org/wiki/Multivariate_normal_distribution), [stochastic processes](http://en.wikipedia.org/wiki/Stochastic_process) ([Wiener process](http://en.wikipedia.org/wiki/Wiener_process), [Poisson process](http://en.wikipedia.org/wiki/Poisson_process), [Markov process](http://en.wikipedia.org/wiki/Markov_process), [Lévy process](http://en.wikipedia.org/wiki/Lévy_process), [stopped process](http://en.wikipedia.org/wiki/Stopped_process))

Multivariable calculus: [partial derivative](http://en.wikipedia.org/wiki/Partial_derivative), [full derivative](http://en.wikipedia.org/wiki/Total_derivative), [gradient](http://en.wikipedia.org/wiki/Gradient), [Hessian](http://en.wikipedia.org/wiki/Hessian_matrix), [Matrix calculus](http://en.wikipedia.org/wiki/Matrix_calculus), [Taylor expansion](http://en.wikipedia.org/wiki/Taylor_series)

II. Indirectly helpful ideas

Inequalities: [Jensen’s inequality](http://en.wikipedia.org/wiki/Jensen's_inequality), [testing critical points](http://en.wikipedia.org/wiki/Differential_calculus#Applications_of_derivatives)

Optimization: [(non-convex) function minimization](http://en.wikipedia.org/wiki/Optimization_(mathematics))

III. Other useful ideas

Calculus: [calculus of variations](http://en.wikipedia.org/wiki/Calculus_of_variations), [extended binomial theorem](http://en.wikipedia.org/wiki/Binomial_theorem#Newton.27s_generalized_binomial_theorem)

Function Approximation: [variational approximation](http://people.csail.mit.edu/tommi/papers/Jaa-var-tutorial.ps), [neural networks](http://en.wikipedia.org/wiki/Artificial_neural_network)

Graph Theory: random walks and relation to Markov Chains, Perron-Frobenius Theorem, combinatorial linear algebra, graphical models (Bayesian networks, Markov random fields, factor graphs)

Miscellaneous: [Kullback-Leibler divergence](http://en.wikipedia.org/wiki/Kullback–Leibler_divergence), [Riccati equation](http://en.wikipedia.org/wiki/Algebraic_Riccati_equation), [homogeneity](http://en.wikipedia.org/wiki/Homogeneous_function) / [dimensional analysis](http://en.wikipedia.org/wiki/Dimensional_analysis), [AM-GM](http://en.wikipedia.org/wiki/Inequality_of_arithmetic_and_geometric_means), induced maps (in a general algebraic sense, not just the homotopy sense; unfortunately I have no good link for this one)

Probability: [Bayes’ rule](http://en.wikipedia.org/wiki/Bayes'_theorem), [Dirichlet process](http://en.wikipedia.org/wiki/Dirichlet_process), [Beta and Bernoulli processes](http://jmlr.csail.mit.edu/proceedings/papers/v2/thibaux07a/thibaux07a.pdf), details balance and Markov Chain Monte Carlo

Spectral analysis: [Fourier transform](http://en.wikipedia.org/wiki/Fourier_transform), [windowing](http://en.wikipedia.org/wiki/Window_function), [aliasing](http://en.wikipedia.org/wiki/Aliasing), [wavelets](http://en.wikipedia.org/wiki/Wavelet), [Pontryagin duality](http://en.wikipedia.org/wiki/Pontryagin_duality)

Linear algebra: [change of basis](http://en.wikipedia.org/wiki/Change_of_basis), [Schur complement](http://en.wikipedia.org/wiki/Schur_complement), [adjoints](http://en.wikipedia.org/wiki/Adjoint), [kernels](http://en.wikipedia.org/wiki/Kernel_(linear_operator)), [injectivity/surjectivity/bijectivity of linear operators](http://en.wikipedia.org/wiki/Linear_map), [natural](http://en.wikipedia.org/wiki/Coordinate-free) [transformations](http://en.wikipedia.org/wiki/Natural_transformation) / [characteristic subspaces](http://en.wikipedia.org/wiki/Characteristic_subgroup)

Topology: [compactness](http://en.wikipedia.org/wiki/Compact_space), [open](http://en.wikipedia.org/wiki/Open_set)/[closed](http://en.wikipedia.org/wiki/Closed_set) sets, [dense sets](http://en.wikipedia.org/wiki/Dense_set), [continuity](http://en.wikipedia.org/wiki/Continuity_(topology)#Continuous_functions_between_topological_spaces), [uniform continuity](http://en.wikipedia.org/wiki/Uniform_continuity), [connectedness](http://en.wikipedia.org/wiki/Connected_space), [path-connectedness](http://en.wikipedia.org/wiki/Connected_space#Path_connectedness)

Analysis: [Lipschitz continuity](http://en.wikipedia.org/wiki/Lipschitz_continuity), [Lesbesgue measure](http://en.wikipedia.org/wiki/Lebesgue_measure), [Haar measure](http://en.wikipedia.org/wiki/Haar_measure), [manifolds](http://en.wikipedia.org/wiki/Manifold), [algebraic manifolds](http://en.wikipedia.org/wiki/Algebraic_manifold)

Optimization: [quasiconvexity](http://en.wikipedia.org/wiki/Quasiconvex_function)

Useful Math


Generalizing Across Categories

Humans are very good at correctly generalizing rules across categories (at least, compared to computers). In this post I will examine mechanisms that would allow us to do this in a reasonably rigorous manner. To this end I will present a probabilistic model such that conditional inference on that model leads to generalization across a category.

There are three questions along these lines that I hope to answer:

* How does one generalize rules across categories?
* How does one determine which rules should generalize across which categories?
* How does one determine when to group objects into a category in the first place?

I suspect that the mechanisms for each of these is rather complex, but I am reasonably confident that the methods I present make up at least part of the actual answer. A good exercise is to come up with examples where these methods fail.

**Generalizing Across Categories**

For simplicity I’m going to consider just some sort of binary rule, such as the existence of an attribute. So as an example, let’s suppose that we see a bunch of ducks, which look to varying degrees to be [mallards](http://en.wikipedia.org/wiki/Mallard). In addition to this, we notice that some of the ducks have three toes, and some have four toes. In this case the category is “mallards”, and the attribute is “has three toes”.

A category is going to be represented as a probabilistic relation for each potential member, of the form ![p(x \in C)](https://s0.wp.com/latex.php?latex=p%28x+%5Cin+C%29&bg=f0f0f0&fg=555555&s=0&c=20201002), where ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) is the category and ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) is the object in question. This essentially indicates the degree to which the object ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) belongs to the category ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002). For instance, if the category is “birds”, then pigeons, sparrows, and eagles all fit into the category very well, so we assign a high probability (close to 1) to pigeons, sparrows, and eagles belonging in the category “birds”. On the other hand, flamingos, penguins, ostriches, and pterodactyls, while still being bird-like, don’t necessarily fit into our archetypal notion of what it means to be a bird. So they get lower probabilities (I’d say around 0.8 for the first 3 and around 0.1 for pterodactyls, but that is all pretty subjective). Finally, dogs and cats get a near-zero probability of being birds, and a table, the plus operator, or Beethoven’s Ninth Symphony would get even-closer-to-zero probabilities of being a bird.

In the example with ducks, the probability of being a mallard will probably be based on how many observed characteristics the duck in question has in common with mallards.

For now we will think of a rule or an attribute as an observable binary property about various objects. Let’s let ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002) be the set of all things that have this property. Now we assume that membership in ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002) has some base probability of occurrence ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002), together with some probability of occurrence within ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) of ![\theta_C](https://s0.wp.com/latex.php?latex=%5Ctheta_C&bg=f0f0f0&fg=555555&s=0&c=20201002). We will assume that there are objects ![x_i](https://s0.wp.com/latex.php?latex=x_i&bg=f0f0f0&fg=555555&s=0&c=20201002), ![y_j](https://s0.wp.com/latex.php?latex=y_j&bg=f0f0f0&fg=555555&s=0&c=20201002), and ![z_k](https://s0.wp.com/latex.php?latex=z_k&bg=f0f0f0&fg=555555&s=0&c=20201002), such that the ![x_i](https://s0.wp.com/latex.php?latex=x_i&bg=f0f0f0&fg=555555&s=0&c=20201002) were all observed to lie in ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002), the ![y_j](https://s0.wp.com/latex.php?latex=y_j&bg=f0f0f0&fg=555555&s=0&c=20201002) were all observed to not lie in ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002), and membership in ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002) has been unobserved for the ![z_k](https://s0.wp.com/latex.php?latex=z_k&bg=f0f0f0&fg=555555&s=0&c=20201002).

Furthermore, ![x_i](https://s0.wp.com/latex.php?latex=x_i&bg=f0f0f0&fg=555555&s=0&c=20201002) has probability ![p_i](https://s0.wp.com/latex.php?latex=p_i&bg=f0f0f0&fg=555555&s=0&c=20201002) of belonging to ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002), ![y_j](https://s0.wp.com/latex.php?latex=y_j&bg=f0f0f0&fg=555555&s=0&c=20201002) has probability ![q_j](https://s0.wp.com/latex.php?latex=q_j&bg=f0f0f0&fg=555555&s=0&c=20201002) of belonging to ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002), and ![z_k](https://s0.wp.com/latex.php?latex=z_k&bg=f0f0f0&fg=555555&s=0&c=20201002) has probability ![r_k](https://s0.wp.com/latex.php?latex=r_k&bg=f0f0f0&fg=555555&s=0&c=20201002) of belonging to ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002).

Again, going back to the ducks example, the ![x_i](https://s0.wp.com/latex.php?latex=x_i&bg=f0f0f0&fg=555555&s=0&c=20201002) are the ducks observed to have three toes, the ![y_j](https://s0.wp.com/latex.php?latex=y_j&bg=f0f0f0&fg=555555&s=0&c=20201002) are the ducks observed to have four toes, and the ![z_k](https://s0.wp.com/latex.php?latex=z_k&bg=f0f0f0&fg=555555&s=0&c=20201002) are the ducks whose number of toes we have not yet observed. The ![p_i](https://s0.wp.com/latex.php?latex=p_i&bg=f0f0f0&fg=555555&s=0&c=20201002), ![q_j](https://s0.wp.com/latex.php?latex=q_j&bg=f0f0f0&fg=555555&s=0&c=20201002), and ![r_k](https://s0.wp.com/latex.php?latex=r_k&bg=f0f0f0&fg=555555&s=0&c=20201002) are the aforementioned probabilities that each of these ducks is a mallard.

So to summarize, we have three classes of objects — those which certainly lie in P, those which certainly don’t lie in P, and those for which we have no information about P. For each element in each of these classes, we have a measure of the extent to which it belongs in C.

Given this setup, how do we actually go about specifying a model explicitly? Basically, we say that if something lies in ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) then it has property ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002) with probability ![\theta_C](https://s0.wp.com/latex.php?latex=%5Ctheta_C&bg=f0f0f0&fg=555555&s=0&c=20201002), and otherwise it has it with probability ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002). Thus we end up with

![p(\theta, \theta_C \ \mid \ p_i, q_j) \propto p(\theta)p(\theta_C) \left(\prod_{i} p_i\theta_C+(1-p_i)\theta\right)\left(\prod_{j} q_j(1-\theta_C)+(1-q_j)(1-\theta)\right)](https://s0.wp.com/latex.php?latex=p%28%5Ctheta%2C+%5Ctheta_C+%5C+%5Cmid+%5C+p_i%2C+q_j%29+%5Cpropto+p%28%5Ctheta%29p%28%5Ctheta_C%29+%5Cleft%28%5Cprod_%7Bi%7D+p_i%5Ctheta_C%2B%281-p_i%29%5Ctheta%5Cright%29%5Cleft%28%5Cprod_%7Bj%7D+q_j%281-%5Ctheta_C%29%2B%281-q_j%29%281-%5Ctheta%29%5Cright%29&bg=f0f0f0&fg=555555&s=0&c=20201002)

What this ends up meaning is that ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) will shift to accommodate the objects that (most likely) don’t lie in ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002), and ![\theta_C](https://s0.wp.com/latex.php?latex=%5Ctheta_C&bg=f0f0f0&fg=555555&s=0&c=20201002) will shift to accommodate the objects that (most likely) do lie in ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002). At the same time, if for instance most elements in ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) also lie in ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002), then the elements that don’t lie in ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002) will have a relatively lower posterior probability of lying in ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) (compared to its prior probability). So this model not only has the advantage of generalizing across all of ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) based on the given observations, it also has the advantage of re-evaluating whether an object is likely to lie in a category based on whether it shares attributes with the other objects in that category.

Let’s go back to the mallards example one more time. Suppose that after we make all of our observations, we notice that most of the ducks that we think are mallards also have three toes. Then ![\theta_C](https://s0.wp.com/latex.php?latex=%5Ctheta_C&bg=f0f0f0&fg=555555&s=0&c=20201002) in this case (the probability that a mallard has three toes) will be close to 1. Furthermore, any duck that we observe to have four toes, we will think much less likely to be a mallard, even if it otherwise looks similar (although it wouldn’t be impossible; it could be a genetic mutant, for instance). At the same time, if non-mallards seem to have either three or four toes with roughly equal frequency, then ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) will be close to 0.5.

There is one thing that I am dissatisfied with in the above model, though. As it stands, ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) measures the probability of an object *not* lying in ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) to have property ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002), rather than the probability of a generic object to have property ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002). This is mainly a problem because, later on, I would like to be able to talk about an object lying in multiple categories, and I don’t have a good way of doing that yet.

An important thing to realize here is that satisfying a rule or having an attribute is just another way of indicating membership in a set. So we can think of both the category and the attribute as potential sets that an object could lie in; as before, we’ll call the category ![C](https://s0.wp.com/latex.php?latex=C&bg=f0f0f0&fg=555555&s=0&c=20201002) and we will call the set of objects having a given attribute ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002). Then ![\theta_C](https://s0.wp.com/latex.php?latex=%5Ctheta_C&bg=f0f0f0&fg=555555&s=0&c=20201002) is something like ![p(x \in A \ \mid \ x \in C)](https://s0.wp.com/latex.php?latex=p%28x+%5Cin+A+%5C+%5Cmid+%5C+x+%5Cin+C%29&bg=f0f0f0&fg=555555&s=0&c=20201002), whereas ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) (should be) ![p(x \in A)](https://s0.wp.com/latex.php?latex=p%28x+%5Cin+A%29&bg=f0f0f0&fg=555555&s=0&c=20201002), although as I’ve set it up now it’s more like ![p(x \in A \ \mid \ x \not\in C)](https://s0.wp.com/latex.php?latex=p%28x+%5Cin+A+%5C+%5Cmid+%5C+x+%5Cnot%5Cin+C%29&bg=f0f0f0&fg=555555&s=0&c=20201002).

As a final note, this same setup can be applied to the case when there are multiple attributes under consideration. We can also apply it to the case when the objects are themselves categories, and so instead of having strict observations about each attribute, we have some sort of probability that the object will possess an attribute. In this latter case we just treat these probabilities as [uncertain observations](https://jsteinhardt.wordpress.com/2010/09/18/uncertain-observations/), as discussed previously.

**When To Generalize**

Another important question is which rules / attributes we should expect to generalize across a category, and which we should not. For instance, if we would expect “number of toes” to generalize across all animals in a given species, but not “age”.

I think that for binary attributes we will always have to generalize, although our generalization could just be “occurs at the base rate in the population”. However, for more complicated attributes (for instance, age, which has an entire continuum of possible values), we can think about whether to generalize based on whether our posterior distribution ends up very tightly concentrated or very spread out. Even if it ends up fairly spread out, there is still some degree of generalization — for instance, we expect that most insects have fairly short lifespans compared to mammals. You could say that this is actually a scientific fact (since we can measure lifespan), but even if you ran into a completely new species of insect, you would expect it to have a relatively short lifespan.

On the other hand, if we went to a college town, we might quickly infer that everyone there was in their 20s; if we went to a bridge club, we would quickly infer that everyone there was past retirement age.

What I would say is that we can always generalize across a category, but sometimes the posterior distribution we end up with is very spread out, and so our generalization isn’t particularly strong. In some cases, the posterior distribution might not even be usefully different from the prior distribution; in this case, from a computational standpoint it doesn’t make sense to keep track of it, and so it *feels like* an attribute shouldn’t generalize across a category, while what we really mean is that we *don’t bother* to keep track of that generalization.

An important thing to keep in mind on this topic is that *everything* that we do when we construct models is a computational heuristic. If we didn’t care about computational complexity and only wanted to arrive at an answer, we would use [Solomonoff](http://singinst.org/blog/2007/06/25/solomonoff-induction/) [induction](http://wiki.lesswrong.com/wiki/Solomonoff_induction), or some computable approximation to it. So whenever I talk about something in probabilistic modeling, I’m partially thinking about whether the method is computationally feasible at all, and what sort of heuristics might be necessary to actually implement something in practice.

To summarize:

* attributes always generalize across a category
* but the generalization might be so weak as to be indistinguishable from the prior distribution
* it would be interesting to figure out how we decide which generalizations to keep track of, and which not to

**Forming Categories**

My final topic of consideration is when to actually form a category. Before going into this, I’d like to share some intuition with you. This is the intuition of explanatory “cheapness” or “expensiveness”. The idea is that events that have low probability under a given hypothesis are “expensive”, and events that have relatively high probability are “cheap”. When a fixed event is expensive under a model, that is evidence against that model; when it is cheap under a model, that is evidence for the model.

The reason that forming clusters can be cheaper is that it allows us to explain many related events simultaneously, which is cheaper than explaining them all separately. Let’s take an extreme example — suppose that 70% of all ducks have long beaks, and 70% have three toes (the remaining 30% in both cases have short beaks and four toes). Then the “cheapest” explanation would be to assign both long beaks and three toes a probability of 0.7, and the probability of our observations would be (supposing there were 100 ducks) ![0.7^{140}0.3^{60} \approx 10^{-53}](https://s0.wp.com/latex.php?latex=0.7%5E%7B140%7D0.3%5E%7B60%7D+%5Capprox+10%5E%7B-53%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). However, suppose that we also observe that the ducks with long beaks are exactly the ducks with three toes. Then we can instead postulate that there are two categories, “(long,3)” and “(short,4)”, and that the probability of ending up in the first category is 0.7 while the probability of ending up in the second category is 0.3. This is a *much* cheaper explanation, as the probability of our observations in this scenario becomes ![0.7^{70}0.3^{30} \approx 10^{-27}](https://s0.wp.com/latex.php?latex=0.7%5E%7B70%7D0.3%5E%7B30%7D+%5Capprox+10%5E%7B-27%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). We have to balance this slightly with the fact that an explanation that involves creating an extra category is more expensive (because it has a lower prior probability / higher complexity) than one that doesn’t, but the extra category can’t possibly be ![10^{26}](https://s0.wp.com/latex.php?latex=10%5E%7B26%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) times more expensive, so we should still always favor it.

This also demonstrates that it will become progressively cheaper to form categories as we get large amounts of similar evidence, which corresponds with our intuitive notion of categories as similarity clusters.

So, now that I’ve actually given you this intuition, how do we actually go about forming categories? Each time we form a category, we will have some base probability of membership in that category, together with a probability of each member of that category possessing each attribute under consideration. If we simply pick prior distributions for each of these parameters, then they will naturally adopt posterior distributions based on the observed data, although these distributions might have multiple high-probability regions if multiple categories can be formed from the data. At the same time, each object will naturally end up with some probability of being in each category, based on how well that category explains its characteristics (as well as the base probability that objects should end up in that category). Putting these together, we see that clusters will naturally form to accommodate the data.

As an example of how we would have multiple high-probability regions, suppose that, as before, there are long-billed, three-toed ducks and short-billed, four-toed ducks. But we also notice that ducks with orange beaks have white feathers and ducks with yellow beaks have brown feathers. If we form a single category, then there are two good candidates, so that category is likely to be either the (long,3) category or the (orange,white) category (or their complements). If we form two categories, each category in isolation is still likely to be either the (long,3) or (orange,white) category, although if the first category is (long,3), then the second is very likely to be (orange,white), and vice versa. In other words, it would be silly to make both categories (long,3), or both categories (orange,white).

The only thing that is left to do is to specify some prior probability of forming a category. While there are various ways to do this, the most commonly used way is the [Indian Buffet Process](http://www.cs.princeton.edu/courses/archive/fall07/cos597C/scribe/20071119.pdf). I won’t explain it in detail here, but I might explain it later.

**Future Considerations**

There are still some unresolved questions here. First of all, in reality something has the potential to be in many, many categories, and it is not entirely clear how to resolve such issues in the above framework. Secondly, keeping track of all of our observations about a given category can be quite difficult computationally (updating in the above scenarios requires performing computations on high-degree polynomials), so we need efficient algorithms for dealing with all of the data we get.

I’m not sure yet how to deal with the first issue, although I’ll be thinking about it in the coming weeks. To deal with the second issue, I intend to use an approach based on streaming algorithms, which will hopefully make up a good final project for a class I’m taking this semester (Indyk and Rubinfeld’s class on Sublinear Algorithms, if you know about MIT course offerings).

Generalizing Across Categories


Uncertain Observations

What happens when you are uncertain about observations you made? For instance, you remember something happening, but you don’t remember who did it. Or you remember some fact you read on wikipedia, but you don’t know whether it said that hydrogen or helium was used in some chemical process.

How do we take this information into account in the context of Bayes’ rule? First, I’d like to note that there are different ways something could be uncertain. It could be that you observed X, but you don’t remember if it was in state A or state B. Or it could be that you think you observed X in state A, but you aren’t sure.

These are different because in the first case you don’t know whether to concentrate probability mass towards A or B, whereas in the second case you don’t know whether to concentrate probability mass at all.

Fortunately, both cases are pretty straightforward as long as you are careful about using Bayes’ rule. However, today I am going to focus on the latter case. In fact, I will restrict my attention to the following problem:

> You have a coin that has some probability ![\pi](https://s0.wp.com/latex.php?latex=%5Cpi&bg=f0f0f0&fg=555555&s=0&c=20201002) of coming up heads. You also know that all flips of this coin are independent. But you don’t know what ![\pi](https://s0.wp.com/latex.php?latex=%5Cpi&bg=f0f0f0&fg=555555&s=0&c=20201002) is. However, you have observed this coin ![n](https://s0.wp.com/latex.php?latex=n&bg=f0f0f0&fg=555555&s=0&c=20201002) times in the past. But for each observation, you aren’t completely sure that this was the coin you were observing. In particular, you only assign a probability ![r_i](https://s0.wp.com/latex.php?latex=r_i&bg=f0f0f0&fg=555555&s=0&c=20201002) to your ![i](https://s0.wp.com/latex.php?latex=i&bg=f0f0f0&fg=555555&s=0&c=20201002)th observation actually being about this coin. Given this, and the sequence of heads and tails you remember, what is your estimate of ![\pi?](https://s0.wp.com/latex.php?latex=%5Cpi%3F&bg=f0f0f0&fg=555555&s=0&c=20201002)
>
>

To use Bayes’ rule, let’s first figure out what we need to condition on. In this case, we need to condition on remembering the sequence of coin flips that we remembered. So we are looking for

p(![\pi = \theta](https://s0.wp.com/latex.php?latex=%5Cpi+%3D+%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) | we remember the given sequence of flips),

which is proportional to

p(we remember the given sequence of flips | ![\pi = \theta](https://s0.wp.com/latex.php?latex=%5Cpi+%3D+%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002)) ![\cdot](https://s0.wp.com/latex.php?latex=%5Ccdot&bg=f0f0f0&fg=555555&s=0&c=20201002) p(![\pi = \theta](https://s0.wp.com/latex.php?latex=%5Cpi+%3D+%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002)).

The only thing that the uncertain nature of our observations does is cause there to be multiple ways to eventually land in the set of universes where we remember the sequence of flips; in particular, for any observation we remember, it could have actually happened, or we could have incorrectly remembered it. Thus if ![\pi = \theta](https://s0.wp.com/latex.php?latex=%5Cpi+%3D+%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002), and we remember the ![i](https://s0.wp.com/latex.php?latex=i&bg=f0f0f0&fg=555555&s=0&c=20201002)th coin flip as being heads, then this could happen with probability ![1-r_i](https://s0.wp.com/latex.php?latex=1-r_i&bg=f0f0f0&fg=555555&s=0&c=20201002) if we incorrectly remembered a coin flip of heads. In the remaining probability ![r_i](https://s0.wp.com/latex.php?latex=r_i&bg=f0f0f0&fg=555555&s=0&c=20201002) case, it could happen with probability ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) by actually coming up heads. Therefore the probability of us remembering that the ![i](https://s0.wp.com/latex.php?latex=i&bg=f0f0f0&fg=555555&s=0&c=20201002)th flip was heads is ![(1-r_i)+r_i \theta](https://s0.wp.com/latex.php?latex=%281-r_i%29%2Br_i+%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002).

A similar computation shows that the probability of us remembering that the ![i](https://s0.wp.com/latex.php?latex=i&bg=f0f0f0&fg=555555&s=0&c=20201002)th flip was tails is ![(1-r_i)+r_i(1-\theta) = 1-r_i\theta](https://s0.wp.com/latex.php?latex=%281-r_i%29%2Br_i%281-%5Ctheta%29+%3D+1-r_i%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002).

For convenience of notation, let’s actually split up our remembered flips into those that were heads and those that were tails. The probability of the ![i](https://s0.wp.com/latex.php?latex=i&bg=f0f0f0&fg=555555&s=0&c=20201002)th remembered heads being real is ![h_i](https://s0.wp.com/latex.php?latex=h_i&bg=f0f0f0&fg=555555&s=0&c=20201002), and the probability of the ![j](https://s0.wp.com/latex.php?latex=j&bg=f0f0f0&fg=555555&s=0&c=20201002)th remembered tails being real is ![t_i](https://s0.wp.com/latex.php?latex=t_i&bg=f0f0f0&fg=555555&s=0&c=20201002). There are ![m](https://s0.wp.com/latex.php?latex=m&bg=f0f0f0&fg=555555&s=0&c=20201002) heads and ![n](https://s0.wp.com/latex.php?latex=n&bg=f0f0f0&fg=555555&s=0&c=20201002) tails. Then we get

![\displaystyle p(\pi = \theta | \mathrm{\ our \ memory}) \propto p(\pi = \theta) \cdot \left(\prod_{i=1}^m (1-h_i)+h_i\theta \right) \cdot \left(\prod_{i=1}^n 1-t_i\theta\right).](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle+p%28%5Cpi+%3D+%5Ctheta+%7C+%5Cmathrm%7B%5C+our+%5C+memory%7D%29+%5Cpropto+p%28%5Cpi+%3D+%5Ctheta%29+%5Ccdot+%5Cleft%28%5Cprod_%7Bi%3D1%7D%5Em+%281-h_i%29%2Bh_i%5Ctheta+%5Cright%29+%5Ccdot+%5Cleft%28%5Cprod_%7Bi%3D1%7D%5En+1-t_i%5Ctheta%5Cright%29.&bg=f0f0f0&fg=555555&s=0&c=20201002)

Note that when we consider values of ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) close to ![0](https://s0.wp.com/latex.php?latex=0&bg=f0f0f0&fg=555555&s=0&c=20201002), the term from the remembered tails becomes close to ![1-\theta](https://s0.wp.com/latex.php?latex=1-%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) raised to the power of the expected number of tails, whereas the term from the remembered heads becomes close to the probability that we incorrectly remembered each of the heads. A similar phenomenon will happen when ![\theta](https://s0.wp.com/latex.php?latex=%5Ctheta&bg=f0f0f0&fg=555555&s=0&c=20201002) gets close to ![1](https://s0.wp.com/latex.php?latex=1&bg=f0f0f0&fg=555555&s=0&c=20201002). This is an instance of a more general phenomenon whereby unlikely observations get “explained away” by whatever means possible.

**A Caveat**

Applying the above model in practice can be quite tricky. The reason is that your memories are intimately tied to all sorts of events that happen to you; in particular, your assessment of how likely you are to remember an event probably *already takes into account* how well that event fits into your existing model. So if you saw 100 heads, and then a tails, you would place *more weight than normal* on your recollection of the tails being incorrect, even though that is the job of the above model. In essence, you are conditioning on your data twice — once intuitively, and once as part of the model. This is bad because it assumes that you made each observation twice as many times as you actually did.

What is interesting, though, is that you can actually compute things like the probability that you incorrectly remembered an event, *given the rest of the data*, and it will be different from the prior probability. So in addition to a posterior estimate of ![\pi](https://s0.wp.com/latex.php?latex=%5Cpi&bg=f0f0f0&fg=555555&s=0&c=20201002), you get posterior estimates of the likelihood of each of your recollections. Just be careful not to take these posterior estimates and use them as if they were prior estimates (which, as explained above, is what we are likely to do intuitively).

There are other issues to using this in practice, as well. For instance, if you *really want* the coin to be fair, or unfair, or be biased in a certain direction, it is very easy to fool yourself into assigning skewed probability estimates towards each of your recollections, thus ending up with a biased answer at the end. It’s not even difficult — if I take a fair coin, and underestimate my recollection of each tails by 20%, and overestimate my recollection of each heads by 20%, then all of a sudden I “have a coin” that is 50% more likely to come up heads than tails.

Fortunately, my intended application of this model will be in a less slippery domain (hopefully). The purpose is to finally answer the question I posed in the last post, which I’ll repeat here for convenience:

> Suppose that you have never played a sport before, and you play soccer, and enjoy it. Now suppose instead that you have never played a sport before, and play soccer, and hate it. In the first case, you will think yourself more likely to enjoy other sports in the future, relative to in the second case. Why is this?
>
>
> Or if you disagree with the premises of the above scenario, simply “If X and Y belong to the same category C, why is it that *in certain cases* we think it more likely that Y will have attribute A upon observing that X has attribute A?”
>
>
>

In the interest of making my posts shorter, I will leave that until next time, but hopefully I’ll get to it in the next week.

Uncertain Observations