Akshay Agrawal commited on
Commit
8bc43c8
·
unverified ·
2 Parent(s): 1530e43 89edddf

Merge pull request #85 from marimo-team/haleshot/19_mle

Browse files
probability/19_maximum_likelihood_estimation.py ADDED
@@ -0,0 +1,1231 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # requires-python = ">=3.10"
3
+ # dependencies = [
4
+ # "marimo",
5
+ # "matplotlib==3.10.1",
6
+ # "scipy==1.15.2",
7
+ # "numpy==2.2.4",
8
+ # "polars==0.20.2",
9
+ # "plotly==5.18.0",
10
+ # ]
11
+ # ///
12
+
13
+ import marimo
14
+
15
+ __generated_with = "0.12.0"
16
+ app = marimo.App(width="medium", app_title="Maximum Likelihood Estimation")
17
+
18
+
19
+ @app.cell(hide_code=True)
20
+ def _(mo):
21
+ mo.md(
22
+ r"""
23
+ # Maximum Likelihood Estimation
24
+
25
+ _This notebook is a computational companion to ["Probability for Computer Scientists"](https://chrispiech.github.io/probabilityForComputerScientists/en/part5/mle/), by Stanford professor Chris Piech._
26
+
27
+ Maximum Likelihood Estimation (MLE) is a fundamental method in statistics for estimating parameters of a probability distribution. The central idea is elegantly simple: **choose the parameters that make the observed data most likely**.
28
+
29
+ In this notebook, we'll try to understand MLE, starting with the core concept of likelihood and how it differs from probability. We'll explore how to formulate MLE problems mathematically and then solve them for various common distributions. Along the way, I've included some interactive visualizations to help build your intuition about these concepts. You'll see how MLE applies to real-world scenarios like linear regression, and hopefully gain a deeper appreciation for why this technique is so widely used in statistics and machine learning. Think of MLE as detective work - we have some evidence (our data) and we're trying to figure out the most plausible explanation (our parameters) for what we've observed.
30
+ """
31
+ )
32
+ return
33
+
34
+
35
+ @app.cell(hide_code=True)
36
+ def _(mo):
37
+ mo.md(
38
+ r"""
39
+ ## Likelihood: The Core Concept
40
+
41
+ Before diving into MLE, we need to understand what "likelihood" means in a statistical context.
42
+
43
+ ### Data and Parameters
44
+
45
+ Suppose we have collected some data $X_1, X_2, \ldots, X_n$ that are independent and identically distributed (IID). We assume these data points come from a specific type of distribution (like Normal, Bernoulli, etc.) with unknown parameters $\theta$.
46
+
47
+ ### What is Likelihood?
48
+
49
+ Likelihood measures how probable our observed data is, given specific values of the parameters $\theta$.
50
+
51
+ /// note
52
+ **Probability vs. Likelihood**
53
+
54
+ - **Probability**: Given parameters $\theta$, what's the chance of observing data $X$?
55
+ - **Likelihood**: Given observed data $X$, how likely are different parameter values $\theta$?
56
+ ///
57
+
58
+ To simplify notation, we'll use $f(X=x|\Theta=\theta)$ to represent either the PMF or PDF of our data, conditioned on the parameters.
59
+ """
60
+ )
61
+ return
62
+
63
+
64
+ @app.cell(hide_code=True)
65
+ def _(mo):
66
+ mo.md(
67
+ r"""
68
+ ### The Likelihood Function
69
+
70
+ Since we assume our data points are independent, the likelihood of all our data is the product of the likelihoods of each individual data point:
71
+
72
+ $$L(\theta) = \prod_{i=1}^n f(X_i = x_i|\Theta = \theta)$$
73
+
74
+ This function $L(\theta)$ gives us the likelihood of observing our entire dataset for different parameter values $\theta$.
75
+
76
+ /// tip
77
+ **Key Insight**: Different parameter values produce different likelihoods for the same data. Better parameter values will make the observed data more likely.
78
+ ///
79
+ """
80
+ )
81
+ return
82
+
83
+
84
+ @app.cell(hide_code=True)
85
+ def _(mo):
86
+ mo.md(
87
+ r"""
88
+ ## Maximum Likelihood Estimation
89
+
90
+ The core idea of MLE is to find the parameter values $\hat{\theta}$ that maximize the likelihood function:
91
+
92
+ $$\hat{\theta} = \underset{\theta}{\operatorname{argmax}} \, L(\theta)$$
93
+
94
+ The notation $\hat{\theta}$ represents our best estimate of the true parameters based on the observed data.
95
+
96
+ ### Working with Log-Likelihood
97
+
98
+ In practice, we usually work with the **log-likelihood** instead of the likelihood directly. Since logarithm is a monotonically increasing function, the maximum of $L(\theta)$ occurs at the same value of $\theta$ as the maximum of $\log L(\theta)$.
99
+
100
+ Taking the logarithm transforms our product into a sum, which is much easier to work with:
101
+
102
+ $$LL(\theta) = \log L(\theta) = \log \prod_{i=1}^n f(X_i=x_i|\Theta = \theta) = \sum_{i=1}^n \log f(X_i = x_i|\Theta = \theta)$$
103
+
104
+ /// warning
105
+ Working with products of many small probabilities can lead to numerical underflow. Taking the logarithm converts these products to sums, which is numerically more stable.
106
+ ///
107
+ """
108
+ )
109
+ return
110
+
111
+
112
+ @app.cell(hide_code=True)
113
+ def _(mo):
114
+ mo.md(
115
+ r"""
116
+ ### Finding the Maximum
117
+
118
+ To find the values of $\theta$ that maximize the log-likelihood, we typically:
119
+
120
+ 1. Take the derivative of $LL(\theta)$ with respect to each parameter
121
+ 2. Set each derivative equal to zero
122
+ 3. Solve for the parameters
123
+
124
+ Let's see this approach in action with some common distributions.
125
+ """
126
+ )
127
+ return
128
+
129
+
130
+ @app.cell(hide_code=True)
131
+ def _(mo):
132
+ mo.md(
133
+ r"""
134
+ ## MLE for Bernoulli Distribution
135
+
136
+ Let's start with a simple example: estimating the parameter $p$ of a Bernoulli distribution.
137
+
138
+ ### The Model
139
+
140
+ A Bernoulli distribution has a single parameter $p$ which represents the probability of success (getting a value of 1). Its probability mass function (PMF) can be written as:
141
+
142
+ $$f(x|p) = p^x(1-p)^{1-x}, \quad x \in \{0, 1\}$$
143
+
144
+ This elegant formula works because:
145
+
146
+ - When $x = 1$: $f(1|p) = p^1(1-p)^0 = p$
147
+ - When $x = 0$: $f(0|p) = p^0(1-p)^1 = 1-p$
148
+
149
+ ### Deriving the MLE
150
+
151
+ Given $n$ independent Bernoulli trials $X_1, X_2, \ldots, X_n$, we want to find the value of $p$ that maximizes the likelihood of our observed data.
152
+
153
+ Step 1: Write the likelihood function
154
+ $$L(p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i}$$
155
+
156
+ Step 2: Take the logarithm to get the log-likelihood
157
+ $$\begin{align*}
158
+ LL(p) &= \sum_{i=1}^n \log(p^{x_i}(1-p)^{1-x_i}) \\
159
+ &= \sum_{i=1}^n \left[x_i \log(p) + (1-x_i)\log(1-p)\right] \\
160
+ &= \left(\sum_{i=1}^n x_i\right) \log(p) + \left(n - \sum_{i=1}^n x_i\right) \log(1-p) \\
161
+ &= Y\log(p) + (n-Y)\log(1-p)
162
+ \end{align*}$$
163
+
164
+ where $Y = \sum_{i=1}^n x_i$ is the total number of successes.
165
+
166
+ Step 3: Find the value of $p$ that maximizes $LL(p)$ by setting the derivative to zero
167
+ $$\begin{align*}
168
+ \frac{d\,LL(p)}{dp} &= \frac{Y}{p} - \frac{n-Y}{1-p} = 0 \\
169
+ \frac{Y}{p} &= \frac{n-Y}{1-p} \\
170
+ Y(1-p) &= p(n-Y) \\
171
+ Y - Yp &= pn - pY \\
172
+ Y &= pn \\
173
+ \hat{p} &= \frac{Y}{n} = \frac{\sum_{i=1}^n x_i}{n}
174
+ \end{align*}$$
175
+
176
+ /// tip
177
+ The MLE for the parameter $p$ in a Bernoulli distribution is simply the **sample mean** - the proportion of successes in our data!
178
+ ///
179
+ """
180
+ )
181
+ return
182
+
183
+
184
+ @app.cell(hide_code=True)
185
+ def _(controls):
186
+ controls.center()
187
+ return
188
+
189
+
190
+ @app.cell(hide_code=True)
191
+ def _(generate_button, mo, np, plt, sample_size_slider, true_p_slider):
192
+ # generate bernoulli samples when button is clicked
193
+ bernoulli_button_value = generate_button.value
194
+
195
+ # get parameter values
196
+ bernoulli_true_p = true_p_slider.value
197
+ bernoulli_n = sample_size_slider.value
198
+
199
+ # generate data
200
+ bernoulli_data = np.random.binomial(1, bernoulli_true_p, size=bernoulli_n)
201
+ bernoulli_Y = np.sum(bernoulli_data)
202
+ bernoulli_p_hat = bernoulli_Y / bernoulli_n
203
+
204
+ # create visualization
205
+ bernoulli_fig, (bernoulli_ax1, bernoulli_ax2) = plt.subplots(1, 2, figsize=(12, 5))
206
+
207
+ # plot data histogram
208
+ bernoulli_ax1.hist(bernoulli_data, bins=[-0.5, 0.5, 1.5], rwidth=0.8, color='lightblue')
209
+ bernoulli_ax1.set_xticks([0, 1])
210
+ bernoulli_ax1.set_xticklabels(['Failure (0)', 'Success (1)'])
211
+ bernoulli_ax1.set_title(f'Bernoulli Data: {bernoulli_n} samples')
212
+ bernoulli_ax1.set_ylabel('Count')
213
+ bernoulli_y_counts = [bernoulli_n - bernoulli_Y, bernoulli_Y]
214
+ for bernoulli_idx, bernoulli_count in enumerate(bernoulli_y_counts):
215
+ bernoulli_ax1.text(bernoulli_idx, bernoulli_count/2, f"{bernoulli_count}",
216
+ ha='center', va='center',
217
+ color='white' if bernoulli_idx == 0 else 'black',
218
+ fontweight='bold')
219
+
220
+ # calculate log-likelihood function
221
+ bernoulli_p_values = np.linspace(0.01, 0.99, 100)
222
+ bernoulli_ll_values = np.zeros_like(bernoulli_p_values)
223
+
224
+ for bernoulli_i, bernoulli_p in enumerate(bernoulli_p_values):
225
+ bernoulli_ll_values[bernoulli_i] = bernoulli_Y * np.log(bernoulli_p) + (bernoulli_n - bernoulli_Y) * np.log(1 - bernoulli_p)
226
+
227
+ # plot log-likelihood
228
+ bernoulli_ax2.plot(bernoulli_p_values, bernoulli_ll_values, 'b-', linewidth=2)
229
+ bernoulli_ax2.axvline(x=bernoulli_p_hat, color='r', linestyle='--', label=f'MLE: $\\hat{{p}} = {bernoulli_p_hat:.3f}$')
230
+ bernoulli_ax2.axvline(x=bernoulli_true_p, color='g', linestyle='--', label=f'True: $p = {bernoulli_true_p:.3f}$')
231
+ bernoulli_ax2.set_xlabel('$p$ (probability of success)')
232
+ bernoulli_ax2.set_ylabel('Log-Likelihood')
233
+ bernoulli_ax2.set_title('Log-Likelihood Function')
234
+ bernoulli_ax2.legend()
235
+
236
+ plt.tight_layout()
237
+ plt.gca()
238
+
239
+ # Create markdown to explain the results
240
+ bernoulli_explanation = mo.md(
241
+ f"""
242
+ ### Bernoulli MLE Results
243
+
244
+ **True parameter**: $p = {bernoulli_true_p:.3f}$
245
+ **Sample statistics**: {bernoulli_Y} successes out of {bernoulli_n} trials
246
+ **MLE estimate**: $\\hat{{p}} = \\frac{{{bernoulli_Y}}}{{{bernoulli_n}}} = {bernoulli_p_hat:.3f}$
247
+
248
+ The plot on the right shows the log-likelihood function $LL(p) = Y\\log(p) + (n-Y)\\log(1-p)$.
249
+ The red dashed line marks the maximum likelihood estimate $\\hat{{p}}$, and the green dashed line
250
+ shows the true parameter value.
251
+
252
+ /// note
253
+ Try increasing the sample size to see how the MLE estimate gets closer to the true parameter value!
254
+ ///
255
+ """
256
+ )
257
+
258
+ # Display plot and explanation together
259
+ mo.vstack([
260
+ bernoulli_fig,
261
+ bernoulli_explanation
262
+ ])
263
+ return (
264
+ bernoulli_Y,
265
+ bernoulli_ax1,
266
+ bernoulli_ax2,
267
+ bernoulli_button_value,
268
+ bernoulli_count,
269
+ bernoulli_data,
270
+ bernoulli_explanation,
271
+ bernoulli_fig,
272
+ bernoulli_i,
273
+ bernoulli_idx,
274
+ bernoulli_ll_values,
275
+ bernoulli_n,
276
+ bernoulli_p,
277
+ bernoulli_p_hat,
278
+ bernoulli_p_values,
279
+ bernoulli_true_p,
280
+ bernoulli_y_counts,
281
+ )
282
+
283
+
284
+ @app.cell(hide_code=True)
285
+ def _(mo):
286
+ mo.md(
287
+ r"""
288
+ ## MLE for Normal Distribution
289
+
290
+ Next, let's look at a more complex example: estimating the parameters $\mu$ and $\sigma^2$ of a Normal distribution.
291
+
292
+ ### The Model
293
+
294
+ A Normal (Gaussian) distribution has two parameters:
295
+ - $\mu$: the mean
296
+ - $\sigma^2$: the variance
297
+
298
+ Its probability density function (PDF) is:
299
+
300
+ $$f(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$$
301
+
302
+ ### Deriving the MLE
303
+
304
+ Given $n$ independent samples $X_1, X_2, \ldots, X_n$ from a Normal distribution, we want to find the values of $\mu$ and $\sigma^2$ that maximize the likelihood of our observed data.
305
+
306
+ Step 1: Write the likelihood function
307
+ $$L(\mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)$$
308
+
309
+ Step 2: Take the logarithm to get the log-likelihood
310
+ $$\begin{align*}
311
+ LL(\mu, \sigma^2) &= \log\prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right) \\
312
+ &= \sum_{i=1}^n \log\left[\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right)\right] \\
313
+ &= \sum_{i=1}^n \left[-\frac{1}{2}\log(2\pi\sigma^2) - \frac{(x_i - \mu)^2}{2\sigma^2}\right] \\
314
+ &= -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2
315
+ \end{align*}$$
316
+
317
+ Step 3: Find the values of $\mu$ and $\sigma^2$ that maximize $LL(\mu, \sigma^2)$ by setting the partial derivatives to zero.
318
+
319
+ For $\mu$:
320
+ $$\begin{align*}
321
+ \frac{\partial LL(\mu, \sigma^2)}{\partial \mu} &= \frac{1}{\sigma^2}\sum_{i=1}^n (x_i - \mu) = 0 \\
322
+ \sum_{i=1}^n (x_i - \mu) &= 0 \\
323
+ \sum_{i=1}^n x_i &= n\mu \\
324
+ \hat{\mu} &= \frac{1}{n}\sum_{i=1}^n x_i
325
+ \end{align*}$$
326
+
327
+ For $\sigma^2$:
328
+ $$\begin{align*}
329
+ \frac{\partial LL(\mu, \sigma^2)}{\partial \sigma^2} &= -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^n (x_i - \mu)^2 = 0 \\
330
+ \frac{n}{2\sigma^2} &= \frac{1}{2(\sigma^2)^2}\sum_{i=1}^n (x_i - \mu)^2 \\
331
+ n\sigma^2 &= \sum_{i=1}^n (x_i - \mu)^2 \\
332
+ \hat{\sigma}^2 &= \frac{1}{n}\sum_{i=1}^n (x_i - \hat{\mu})^2
333
+ \end{align*}$$
334
+
335
+ /// tip
336
+ The MLE for a Normal distribution gives us:
337
+
338
+ - $\hat{\mu}$ = sample mean
339
+ - $\hat{\sigma}^2$ = sample variance (using $n$ in the denominator, not $n-1$)
340
+ ///
341
+ """
342
+ )
343
+ return
344
+
345
+
346
+ @app.cell(hide_code=True)
347
+ def _(normal_controls):
348
+ normal_controls.center()
349
+ return
350
+
351
+
352
+ @app.cell(hide_code=True)
353
+ def _(
354
+ mo,
355
+ normal_generate_button,
356
+ normal_sample_size_slider,
357
+ np,
358
+ plt,
359
+ true_mu_slider,
360
+ true_sigma_slider,
361
+ ):
362
+ # generate normal samples when button is clicked
363
+ normal_button_value = normal_generate_button.value
364
+
365
+ # get parameter values
366
+ normal_true_mu = true_mu_slider.value
367
+ normal_true_sigma = true_sigma_slider.value
368
+ normal_true_var = normal_true_sigma**2
369
+ normal_n = normal_sample_size_slider.value
370
+
371
+ # generate random data
372
+ normal_data = np.random.normal(normal_true_mu, normal_true_sigma, size=normal_n)
373
+
374
+ # calculate mle estimates
375
+ normal_mu_hat = np.mean(normal_data)
376
+ normal_sigma2_hat = np.mean((normal_data - normal_mu_hat)**2) # mle variance using n
377
+ normal_sigma_hat = np.sqrt(normal_sigma2_hat)
378
+
379
+ # create visualization
380
+ normal_fig, (normal_ax1, normal_ax2) = plt.subplots(1, 2, figsize=(12, 5))
381
+
382
+ # plot histogram and density curves
383
+ normal_bins = np.linspace(min(normal_data) - 1, max(normal_data) + 1, 30)
384
+ normal_ax1.hist(normal_data, bins=normal_bins, density=True, alpha=0.6, color='lightblue', label='Data Histogram')
385
+
386
+ # plot range for density curves
387
+ normal_x = np.linspace(min(normal_data) - 2*normal_true_sigma, max(normal_data) + 2*normal_true_sigma, 1000)
388
+
389
+ # plot true and mle densities
390
+ normal_true_pdf = (1/(normal_true_sigma * np.sqrt(2*np.pi))) * np.exp(-0.5 * ((normal_x - normal_true_mu)/normal_true_sigma)**2)
391
+ normal_ax1.plot(normal_x, normal_true_pdf, 'g-', linewidth=2, label=f'True: N({normal_true_mu:.2f}, {normal_true_var:.2f})')
392
+
393
+ normal_mle_pdf = (1/(normal_sigma_hat * np.sqrt(2*np.pi))) * np.exp(-0.5 * ((normal_x - normal_mu_hat)/normal_sigma_hat)**2)
394
+ normal_ax1.plot(normal_x, normal_mle_pdf, 'r--', linewidth=2, label=f'MLE: N({normal_mu_hat:.2f}, {normal_sigma2_hat:.2f})')
395
+
396
+ normal_ax1.set_xlabel('x')
397
+ normal_ax1.set_ylabel('Density')
398
+ normal_ax1.set_title(f'Normal Distribution: {normal_n} samples')
399
+ normal_ax1.legend()
400
+
401
+ # create contour plot of log-likelihood
402
+ normal_mu_range = np.linspace(normal_mu_hat - 2, normal_mu_hat + 2, 100)
403
+ normal_sigma_range = np.linspace(max(0.1, normal_sigma_hat - 1), normal_sigma_hat + 1, 100)
404
+
405
+ normal_mu_grid, normal_sigma_grid = np.meshgrid(normal_mu_range, normal_sigma_range)
406
+ normal_ll_grid = np.zeros_like(normal_mu_grid)
407
+
408
+ # calculate log-likelihood for each grid point
409
+ for normal_i in range(normal_mu_grid.shape[0]):
410
+ for normal_j in range(normal_mu_grid.shape[1]):
411
+ normal_mu = normal_mu_grid[normal_i, normal_j]
412
+ normal_sigma = normal_sigma_grid[normal_i, normal_j]
413
+ normal_ll = -normal_n/2 * np.log(2*np.pi*normal_sigma**2) - np.sum((normal_data - normal_mu)**2)/(2*normal_sigma**2)
414
+ normal_ll_grid[normal_i, normal_j] = normal_ll
415
+
416
+ # plot log-likelihood contour
417
+ normal_contour = normal_ax2.contourf(normal_mu_grid, normal_sigma_grid, normal_ll_grid, levels=50, cmap='viridis')
418
+ normal_ax2.set_xlabel('μ (mean)')
419
+ normal_ax2.set_ylabel('σ (standard deviation)')
420
+ normal_ax2.set_title('Log-Likelihood Contour')
421
+
422
+ # mark mle and true params
423
+ normal_ax2.plot(normal_mu_hat, normal_sigma_hat, 'rx', markersize=10, label='MLE Estimate')
424
+ normal_ax2.plot(normal_true_mu, normal_true_sigma, 'g*', markersize=10, label='True Parameters')
425
+ normal_ax2.legend()
426
+
427
+ plt.colorbar(normal_contour, ax=normal_ax2, label='Log-Likelihood')
428
+ plt.tight_layout()
429
+ plt.gca()
430
+
431
+ # relevant markdown for the results
432
+ normal_explanation = mo.md(
433
+ f"""
434
+ ### Normal MLE Results
435
+
436
+ **True parameters**: $\mu = {normal_true_mu:.3f}$, $\sigma^2 = {normal_true_var:.3f}$
437
+ **MLE estimates**: $\hat{{\mu}} = {normal_mu_hat:.3f}$, $\hat{{\sigma}}^2 = {normal_sigma2_hat:.3f}$
438
+
439
+ The left plot shows the data histogram with the true Normal distribution (green) and the MLE-estimated distribution (red dashed).
440
+
441
+ The right plot shows the log-likelihood function as a contour map in the $(\mu, \sigma)$ parameter space. The maximum likelihood estimates are marked with a red X, while the true parameters are marked with a green star.
442
+
443
+ /// note
444
+ Notice how the log-likelihood contour is more stretched along the σ axis than the μ axis. This indicates that we typically estimate the mean with greater precision than the standard deviation.
445
+ ///
446
+
447
+ /// tip
448
+ Increase the sample size to see how the MLE estimates converge to the true parameter values!
449
+ ///
450
+ """
451
+ )
452
+
453
+ # plot and explanation together
454
+ mo.vstack([
455
+ normal_fig,
456
+ normal_explanation
457
+ ])
458
+ return (
459
+ normal_ax1,
460
+ normal_ax2,
461
+ normal_bins,
462
+ normal_button_value,
463
+ normal_contour,
464
+ normal_data,
465
+ normal_explanation,
466
+ normal_fig,
467
+ normal_i,
468
+ normal_j,
469
+ normal_ll,
470
+ normal_ll_grid,
471
+ normal_mle_pdf,
472
+ normal_mu,
473
+ normal_mu_grid,
474
+ normal_mu_hat,
475
+ normal_mu_range,
476
+ normal_n,
477
+ normal_sigma,
478
+ normal_sigma2_hat,
479
+ normal_sigma_grid,
480
+ normal_sigma_hat,
481
+ normal_sigma_range,
482
+ normal_true_mu,
483
+ normal_true_pdf,
484
+ normal_true_sigma,
485
+ normal_true_var,
486
+ normal_x,
487
+ )
488
+
489
+
490
+ @app.cell(hide_code=True)
491
+ def _(mo):
492
+ mo.md(
493
+ r"""
494
+ ## MLE for Linear Regression
495
+
496
+ Now let's look at a more practical example: using MLE to derive linear regression.
497
+
498
+ ### The Model
499
+
500
+ Consider a model where:
501
+ - We have pairs of observations $(X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)$
502
+ - The relationship between $X$ and $Y$ follows: $Y = \theta X + Z$
503
+ - $Z \sim N(0, \sigma^2)$ is random noise
504
+ - Our goal is to estimate the parameter $\theta$
505
+
506
+ This means that for a given $X_i$, the conditional distribution of $Y_i$ is:
507
+
508
+ $$Y_i | X_i \sim N(\theta X_i, \sigma^2)$$
509
+
510
+ ### Deriving the MLE
511
+
512
+ Step 1: Write the likelihood function for each data point $(X_i, Y_i)$
513
+ $$f(Y_i | X_i, \theta) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(Y_i - \theta X_i)^2}{2\sigma^2}\right)$$
514
+
515
+ Step 2: Write the likelihood for all data
516
+ $$\begin{align*}
517
+ L(\theta) &= \prod_{i=1}^n f(Y_i, X_i | \theta) \\
518
+ &= \prod_{i=1}^n f(Y_i | X_i, \theta) \cdot f(X_i)
519
+ \end{align*}$$
520
+
521
+ Since $f(X_i)$ doesn't depend on $\theta$, we can simplify:
522
+ $$L(\theta) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(Y_i - \theta X_i)^2}{2\sigma^2}\right) \cdot f(X_i)$$
523
+
524
+ Step 3: Take the logarithm to get the log-likelihood
525
+ $$\begin{align*}
526
+ LL(\theta) &= \log \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(Y_i - \theta X_i)^2}{2\sigma^2}\right) \cdot f(X_i) \\
527
+ &= \sum_{i=1}^n \log\left[\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(Y_i - \theta X_i)^2}{2\sigma^2}\right)\right] + \sum_{i=1}^n \log f(X_i) \\
528
+ &= -\frac{n}{2} \log(2\pi\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^n (Y_i - \theta X_i)^2 + \sum_{i=1}^n \log f(X_i)
529
+ \end{align*}$$
530
+
531
+ Step 4: Since we only care about maximizing with respect to $\theta$, we can drop terms that don't contain $\theta$:
532
+ $$\hat{\theta} = \underset{\theta}{\operatorname{argmax}} \left[ -\frac{1}{2\sigma^2} \sum_{i=1}^n (Y_i - \theta X_i)^2 \right]$$
533
+
534
+ This is equivalent to:
535
+ $$\hat{\theta} = \underset{\theta}{\operatorname{argmin}} \sum_{i=1}^n (Y_i - \theta X_i)^2$$
536
+
537
+ Step 5: Find the value of $\theta$ that minimizes the sum of squared errors by setting the derivative to zero:
538
+ $$\begin{align*}
539
+ \frac{d}{d\theta} \sum_{i=1}^n (Y_i - \theta X_i)^2 &= 0 \\
540
+ \sum_{i=1}^n -2X_i(Y_i - \theta X_i) &= 0 \\
541
+ \sum_{i=1}^n X_i Y_i - \theta X_i^2 &= 0 \\
542
+ \sum_{i=1}^n X_i Y_i &= \theta \sum_{i=1}^n X_i^2 \\
543
+ \hat{\theta} &= \frac{\sum_{i=1}^n X_i Y_i}{\sum_{i=1}^n X_i^2}
544
+ \end{align*}$$
545
+
546
+ /// tip
547
+ **Key Insight**: MLE for this simple linear model gives us the least squares estimator! This is an important connection between MLE and regression.
548
+ ///
549
+ """
550
+ )
551
+ return
552
+
553
+
554
+ @app.cell(hide_code=True)
555
+ def _(linear_controls):
556
+ linear_controls.center()
557
+ return
558
+
559
+
560
+ @app.cell(hide_code=True)
561
+ def _(
562
+ linear_generate_button,
563
+ linear_sample_size_slider,
564
+ mo,
565
+ noise_sigma_slider,
566
+ np,
567
+ plt,
568
+ true_theta_slider,
569
+ ):
570
+ # linear model data calc when button is clicked
571
+ linear_button_value = linear_generate_button.value
572
+
573
+ # get parameter values
574
+ linear_true_theta = true_theta_slider.value
575
+ linear_noise_sigma = noise_sigma_slider.value
576
+ linear_n = linear_sample_size_slider.value
577
+
578
+ # generate x data (uniformly between -3 and 3)
579
+ linear_X = np.random.uniform(-3, 3, size=linear_n)
580
+
581
+ # generate y data according to the model y = θx + z
582
+ linear_Z = np.random.normal(0, linear_noise_sigma, size=linear_n)
583
+ linear_Y = linear_true_theta * linear_X + linear_Z
584
+
585
+ # calculate mle estimate
586
+ linear_theta_hat = np.sum(linear_X * linear_Y) / np.sum(linear_X**2)
587
+
588
+ # calculate sse for different theta values
589
+ linear_theta_range = np.linspace(linear_true_theta - 1.5, linear_true_theta + 1.5, 100)
590
+ linear_sse_values = np.zeros_like(linear_theta_range)
591
+
592
+ for linear_i, linear_theta in enumerate(linear_theta_range):
593
+ linear_y_pred = linear_theta * linear_X
594
+ linear_sse_values[linear_i] = np.sum((linear_Y - linear_y_pred)**2)
595
+
596
+ # convert sse to log-likelihood (ignoring constant terms)
597
+ linear_ll_values = -linear_sse_values / (2 * linear_noise_sigma**2)
598
+
599
+ # create visualization
600
+ linear_fig, (linear_ax1, linear_ax2) = plt.subplots(1, 2, figsize=(12, 5))
601
+
602
+ # plot scatter plot with regression lines
603
+ linear_ax1.scatter(linear_X, linear_Y, color='blue', alpha=0.6, label='Data points')
604
+
605
+ # plot range for regression lines
606
+ linear_x_line = np.linspace(-3, 3, 100)
607
+
608
+ # plot true and mle regression lines
609
+ linear_ax1.plot(linear_x_line, linear_true_theta * linear_x_line, 'g-', linewidth=2, label=f'True: Y = {linear_true_theta:.2f}X')
610
+ linear_ax1.plot(linear_x_line, linear_theta_hat * linear_x_line, 'r--', linewidth=2, label=f'MLE: Y = {linear_theta_hat:.2f}X')
611
+
612
+ linear_ax1.set_xlabel('X')
613
+ linear_ax1.set_ylabel('Y')
614
+ linear_ax1.set_title(f'Linear Regression: {linear_n} data points')
615
+ linear_ax1.grid(True, alpha=0.3)
616
+ linear_ax1.legend()
617
+
618
+ # plot log-likelihood function
619
+ linear_ax2.plot(linear_theta_range, linear_ll_values, 'b-', linewidth=2)
620
+ linear_ax2.axvline(x=linear_theta_hat, color='r', linestyle='--', label=f'MLE: θ = {linear_theta_hat:.3f}')
621
+ linear_ax2.axvline(x=linear_true_theta, color='g', linestyle='--', label=f'True: θ = {linear_true_theta:.3f}')
622
+ linear_ax2.set_xlabel('θ (slope parameter)')
623
+ linear_ax2.set_ylabel('Log-Likelihood')
624
+ linear_ax2.set_title('Log-Likelihood Function')
625
+ linear_ax2.grid(True, alpha=0.3)
626
+ linear_ax2.legend()
627
+
628
+ plt.tight_layout()
629
+ plt.gca()
630
+
631
+ # relevant markdown to explain results
632
+ linear_explanation = mo.md(
633
+ f"""
634
+ ### Linear Regression MLE Results
635
+
636
+ **True parameter**: $\\theta = {linear_true_theta:.3f}$
637
+ **MLE estimate**: $\\hat{{\\theta}} = {linear_theta_hat:.3f}$
638
+
639
+ The left plot shows the scatter plot of data points with the true regression line (green) and the MLE-estimated regression line (red dashed).
640
+
641
+ The right plot shows the log-likelihood function for different values of $\\theta$. The maximum likelihood estimate is marked with a red dashed line, and the true parameter is marked with a green dashed line.
642
+
643
+ /// note
644
+ The MLE estimate $\\hat{{\\theta}} = \\frac{{\\sum_{{i=1}}^n X_i Y_i}}{{\\sum_{{i=1}}^n X_i^2}}$ minimizes the sum of squared errors between the predicted and actual Y values.
645
+ ///
646
+
647
+ /// tip
648
+ Try increasing the noise level to see how it affects the precision of the estimate!
649
+ ///
650
+ """
651
+ )
652
+
653
+ # show plot and explanation
654
+ mo.vstack([
655
+ linear_fig,
656
+ linear_explanation
657
+ ])
658
+ return (
659
+ linear_X,
660
+ linear_Y,
661
+ linear_Z,
662
+ linear_ax1,
663
+ linear_ax2,
664
+ linear_button_value,
665
+ linear_explanation,
666
+ linear_fig,
667
+ linear_i,
668
+ linear_ll_values,
669
+ linear_n,
670
+ linear_noise_sigma,
671
+ linear_sse_values,
672
+ linear_theta,
673
+ linear_theta_hat,
674
+ linear_theta_range,
675
+ linear_true_theta,
676
+ linear_x_line,
677
+ linear_y_pred,
678
+ )
679
+
680
+
681
+ @app.cell(hide_code=True)
682
+ def _(mo):
683
+ mo.md(
684
+ r"""
685
+ ## Interactive Concept: Density/Mass Functions vs. Likelihood
686
+
687
+ To better understand the distinction between likelihood and density/mass functions, let's create an interactive visualization. This concept is crucial for understanding why MLE works.
688
+ """
689
+ )
690
+ return
691
+
692
+
693
+ @app.cell(hide_code=True)
694
+ def _(concept_controls):
695
+ concept_controls.center()
696
+ return
697
+
698
+
699
+ @app.cell(hide_code=True)
700
+ def _(concept_dist_type, mo, np, perspective_selector, plt, stats):
701
+ # current distribution type
702
+ concept_dist_type_value = concept_dist_type.value
703
+
704
+ # view mode from dropdown
705
+ concept_view_mode = "likelihood" if perspective_selector.value == "Likelihood Perspective" else "probability"
706
+
707
+ # visualization based on distribution type
708
+ concept_fig, concept_ax = plt.subplots(figsize=(10, 6))
709
+
710
+ if concept_dist_type_value == "Normal":
711
+ if concept_view_mode == "probability":
712
+ # density function perspective: fixed params, varying data
713
+ concept_mu = 0 # fixed parameter
714
+ concept_sigma = 1 # fixed parameter
715
+
716
+ # generate x values for the pdf
717
+ concept_x = np.linspace(-4, 4, 1000)
718
+
719
+ # plot pdf
720
+ concept_pdf = stats.norm.pdf(concept_x, concept_mu, concept_sigma)
721
+ concept_ax.plot(concept_x, concept_pdf, 'b-', linewidth=2, label='PDF: N(0, 1)')
722
+
723
+ # highlight specific data values
724
+ concept_data_points = [-2, -1, 0, 1, 2]
725
+ concept_colors = ['#FF9999', '#FFCC99', '#99FF99', '#99CCFF', '#CC99FF']
726
+
727
+ for concept_i, concept_data in enumerate(concept_data_points):
728
+ concept_prob = stats.norm.pdf(concept_data, concept_mu, concept_sigma)
729
+ concept_ax.plot([concept_data, concept_data], [0, concept_prob], concept_colors[concept_i], linewidth=2)
730
+ concept_ax.scatter(concept_data, concept_prob, color=concept_colors[concept_i], s=50,
731
+ label=f'PDF at x={concept_data}: {concept_prob:.3f}')
732
+
733
+ concept_ax.set_xlabel('Data (x)')
734
+ concept_ax.set_ylabel('Probability Density')
735
+ concept_ax.set_title('Density Function Perspective: Fixed Parameters (μ=0, σ=1), Different Data Points')
736
+
737
+ else: # likelihood perspective
738
+ # likelihood perspective: fixed data, varying parameters
739
+ concept_data_point = 1.5 # fixed observed data
740
+
741
+ # different possible parameter values (means)
742
+ concept_mus = [-1, 0, 1, 2, 3]
743
+ concept_sigma = 1
744
+
745
+ # generate x values for multiple pdfs
746
+ concept_x = np.linspace(-4, 6, 1000)
747
+
748
+ concept_colors = ['#FF9999', '#FFCC99', '#99FF99', '#99CCFF', '#CC99FF']
749
+
750
+ for concept_i, concept_mu in enumerate(concept_mus):
751
+ concept_pdf = stats.norm.pdf(concept_x, concept_mu, concept_sigma)
752
+ concept_ax.plot(concept_x, concept_pdf, color=concept_colors[concept_i], linewidth=2, alpha=0.7,
753
+ label=f'N({concept_mu}, 1)')
754
+
755
+ # mark the likelihood of the data point for this param
756
+ concept_likelihood = stats.norm.pdf(concept_data_point, concept_mu, concept_sigma)
757
+ concept_ax.plot([concept_data_point, concept_data_point], [0, concept_likelihood], concept_colors[concept_i], linewidth=2)
758
+ concept_ax.scatter(concept_data_point, concept_likelihood, color=concept_colors[concept_i], s=50,
759
+ label=f'L(μ={concept_mu}|X=1.5) = {concept_likelihood:.3f}')
760
+
761
+ # add vertical line at the observed data point
762
+ concept_ax.axvline(x=concept_data_point, color='black', linestyle='--',
763
+ label=f'Observed data: X=1.5')
764
+
765
+ concept_ax.set_xlabel('Data (x)')
766
+ concept_ax.set_ylabel('Probability Density / Likelihood')
767
+ concept_ax.set_title('Likelihood Perspective: Fixed Data Point (X=1.5), Different Parameter Values')
768
+
769
+ elif concept_dist_type_value == "Bernoulli":
770
+ if concept_view_mode == "probability":
771
+ # probability perspective: fixed parameter, two possible data values
772
+ concept_p = 0.3 # fixed parameter
773
+
774
+ # bar chart for p(x=0) and p(x=1)
775
+ concept_ax.bar([0, 1], [1-concept_p, concept_p], width=0.4, color=['#99CCFF', '#FF9999'],
776
+ alpha=0.7, label=f'PMF: Bernoulli({concept_p})')
777
+
778
+ # text showing probabilities
779
+ concept_ax.text(0, (1-concept_p)/2, f'P(X=0|p={concept_p}) = {1-concept_p:.3f}', ha='center', va='center', fontweight='bold')
780
+ concept_ax.text(1, concept_p/2, f'P(X=1|p={concept_p}) = {concept_p:.3f}', ha='center', va='center', fontweight='bold')
781
+
782
+ concept_ax.set_xlabel('Data (x)')
783
+ concept_ax.set_ylabel('Probability')
784
+ concept_ax.set_xticks([0, 1])
785
+ concept_ax.set_xticklabels(['X=0', 'X=1'])
786
+ concept_ax.set_ylim(0, 1)
787
+ concept_ax.set_title('Probability Perspective: Fixed Parameter (p=0.3), Different Data Values')
788
+
789
+ else: # likelihood perspective
790
+ # likelihood perspective: fixed data, varying parameter
791
+ concept_data_point = 1 # fixed observed data (success)
792
+
793
+ # different possible parameter values
794
+ concept_p_values = np.linspace(0.01, 0.99, 100)
795
+
796
+ # calculate likelihood for each p value
797
+ if concept_data_point == 1:
798
+ # for x=1, likelihood is p
799
+ concept_likelihood = concept_p_values
800
+ concept_ax.plot(concept_p_values, concept_likelihood, 'b-', linewidth=2,
801
+ label=f'L(p|X=1) = p')
802
+
803
+ # highlight specific values
804
+ concept_highlight_ps = [0.2, 0.5, 0.8]
805
+ concept_colors = ['#FF9999', '#99FF99', '#99CCFF']
806
+
807
+ for concept_i, concept_p in enumerate(concept_highlight_ps):
808
+ concept_ax.plot([concept_p, concept_p], [0, concept_p], concept_colors[concept_i], linewidth=2)
809
+ concept_ax.scatter(concept_p, concept_p, color=concept_colors[concept_i], s=50,
810
+ label=f'L(p={concept_p}|X=1) = {concept_p:.3f}')
811
+
812
+ concept_ax.set_title('Likelihood Perspective: Fixed Data Point (X=1), Different Parameter Values')
813
+
814
+ else: # x=0
815
+ # for x = 0, likelihood is (1-p)
816
+ concept_likelihood = 1 - concept_p_values
817
+ concept_ax.plot(concept_p_values, concept_likelihood, 'r-', linewidth=2,
818
+ label=f'L(p|X=0) = (1-p)')
819
+
820
+ # highlight some specific values
821
+ concept_highlight_ps = [0.2, 0.5, 0.8]
822
+ concept_colors = ['#FF9999', '#99FF99', '#99CCFF']
823
+
824
+ for concept_i, concept_p in enumerate(concept_highlight_ps):
825
+ concept_ax.plot([concept_p, concept_p], [0, 1-concept_p], concept_colors[concept_i], linewidth=2)
826
+ concept_ax.scatter(concept_p, 1-concept_p, color=concept_colors[concept_i], s=50,
827
+ label=f'L(p={concept_p}|X=0) = {1-concept_p:.3f}')
828
+
829
+ concept_ax.set_title('Likelihood Perspective: Fixed Data Point (X=0), Different Parameter Values')
830
+
831
+ concept_ax.set_xlabel('Parameter (p)')
832
+ concept_ax.set_ylabel('Likelihood')
833
+ concept_ax.set_xlim(0, 1)
834
+ concept_ax.set_ylim(0, 1)
835
+
836
+ elif concept_dist_type_value == "Poisson":
837
+ if concept_view_mode == "probability":
838
+ # probability perspective: fixed parameter, different data values
839
+ concept_lam = 2.5 # fixed parameter
840
+
841
+ # pmf for different x values plot
842
+ concept_x_values = np.arange(0, 10)
843
+ concept_pmf_values = stats.poisson.pmf(concept_x_values, concept_lam)
844
+
845
+ concept_ax.bar(concept_x_values, concept_pmf_values, width=0.4, color='#99CCFF',
846
+ alpha=0.7, label=f'PMF: Poisson({concept_lam})')
847
+
848
+ # highlight a few specific values
849
+ concept_highlight_xs = [1, 2, 3, 4]
850
+ concept_colors = ['#FF9999', '#99FF99', '#FFCC99', '#CC99FF']
851
+
852
+ for concept_i, concept_x in enumerate(concept_highlight_xs):
853
+ concept_prob = stats.poisson.pmf(concept_x, concept_lam)
854
+ concept_ax.scatter(concept_x, concept_prob, color=concept_colors[concept_i], s=50,
855
+ label=f'P(X={concept_x}|λ={concept_lam}) = {concept_prob:.3f}')
856
+
857
+ concept_ax.set_xlabel('Data (x)')
858
+ concept_ax.set_ylabel('Probability')
859
+ concept_ax.set_xticks(concept_x_values)
860
+ concept_ax.set_title('Probability Perspective: Fixed Parameter (λ=2.5), Different Data Values')
861
+
862
+ else: # likelihood perspective
863
+ # likelihood perspective: fixed data, varying parameter
864
+ concept_data_point = 4 # fixed observed data
865
+
866
+ # different possible param values
867
+ concept_lambda_values = np.linspace(0.1, 8, 100)
868
+
869
+ # calc likelihood for each lambda value
870
+ concept_likelihood = stats.poisson.pmf(concept_data_point, concept_lambda_values)
871
+
872
+ concept_ax.plot(concept_lambda_values, concept_likelihood, 'b-', linewidth=2,
873
+ label=f'L(λ|X={concept_data_point})')
874
+
875
+ # highlight some specific values
876
+ concept_highlight_lambdas = [1, 2, 4, 6]
877
+ concept_colors = ['#FF9999', '#99FF99', '#99CCFF', '#FFCC99']
878
+
879
+ for concept_i, concept_lam in enumerate(concept_highlight_lambdas):
880
+ concept_like_val = stats.poisson.pmf(concept_data_point, concept_lam)
881
+ concept_ax.plot([concept_lam, concept_lam], [0, concept_like_val], concept_colors[concept_i], linewidth=2)
882
+ concept_ax.scatter(concept_lam, concept_like_val, color=concept_colors[concept_i], s=50,
883
+ label=f'L(λ={concept_lam}|X={concept_data_point}) = {concept_like_val:.3f}')
884
+
885
+ concept_ax.set_xlabel('Parameter (λ)')
886
+ concept_ax.set_ylabel('Likelihood')
887
+ concept_ax.set_title(f'Likelihood Perspective: Fixed Data Point (X={concept_data_point}), Different Parameter Values')
888
+
889
+ concept_ax.legend(loc='best', fontsize=9)
890
+ concept_ax.grid(True, alpha=0.3)
891
+ plt.tight_layout()
892
+ plt.gca()
893
+
894
+ # relevant explanation based on view mode
895
+ if concept_view_mode == "probability":
896
+ concept_explanation = mo.md(
897
+ f"""
898
+ ### Density/Mass Function Perspective
899
+
900
+ In the **density/mass function perspective**, the parameters of the distribution are **fixed and known**, and we evaluate the function at **different possible data values**.
901
+
902
+ For the {concept_dist_type_value} distribution, we've fixed the parameter{'s' if concept_dist_type_value == 'Normal' else ''} and shown the {'density' if concept_dist_type_value == 'Normal' else 'probability mass'} function evaluated at different data points.
903
+
904
+ This is the typical perspective when:
905
+
906
+ - We know the true parameters of a distribution
907
+ - We want to evaluate the {'density' if concept_dist_type_value == 'Normal' else 'probability mass'} at different observations
908
+ - We make predictions based on our model
909
+
910
+ **Mathematical notation**: $f(x | \theta)$
911
+ """
912
+ )
913
+ else: # likelihood perspective
914
+ concept_explanation = mo.md(
915
+ f"""
916
+ ### Likelihood Perspective
917
+
918
+ In the **likelihood perspective**, the observed data is **fixed and known**, and we calculate how likely different parameter values are to have generated that data.
919
+
920
+ For the {concept_dist_type_value} distribution, we've fixed the observed data point{'s' if concept_dist_type_value == 'Normal' else ''} and shown the likelihood of different parameter values.
921
+
922
+ This is the perspective used in MLE:
923
+
924
+ - We have observed data
925
+ - We don't know the true parameters
926
+ - We want to find parameters that best explain our observations
927
+
928
+ **Mathematical notation**: $L(\theta | X = x)$
929
+
930
+ /// tip
931
+ The value of $\\theta$ that maximizes this likelihood function is the MLE estimate $\\hat{{\\theta}}$!
932
+ ///
933
+ """
934
+ )
935
+
936
+ # Display plot and explanation together
937
+ mo.vstack([
938
+ concept_fig,
939
+ concept_explanation
940
+ ])
941
+ return (
942
+ concept_ax,
943
+ concept_colors,
944
+ concept_data,
945
+ concept_data_point,
946
+ concept_data_points,
947
+ concept_dist_type_value,
948
+ concept_explanation,
949
+ concept_fig,
950
+ concept_highlight_lambdas,
951
+ concept_highlight_ps,
952
+ concept_highlight_xs,
953
+ concept_i,
954
+ concept_lam,
955
+ concept_lambda_values,
956
+ concept_like_val,
957
+ concept_likelihood,
958
+ concept_mu,
959
+ concept_mus,
960
+ concept_p,
961
+ concept_p_values,
962
+ concept_pdf,
963
+ concept_pmf_values,
964
+ concept_prob,
965
+ concept_sigma,
966
+ concept_view_mode,
967
+ concept_x,
968
+ concept_x_values,
969
+ )
970
+
971
+
972
+ @app.cell(hide_code=True)
973
+ def _(mo):
974
+ mo.md(
975
+ r"""
976
+ ## 🤔 Test Your Understanding
977
+
978
+ Which of the following statements about Maximum Likelihood Estimation are correct? Click each statement to check your answer.
979
+
980
+ /// details | Probability and likelihood have different interpretations: probability measures the chance of data given parameters, while likelihood measures how likely parameters are given data.
981
+ ✅ **Correct!**
982
+
983
+ Probability measures how likely it is to observe particular data when we know the parameters. Likelihood measures how likely particular parameter values are, given observed data.
984
+
985
+ Mathematically, probability is $P(X=x|\theta)$ while likelihood is $L(\theta|X=x)$.
986
+ ///
987
+
988
+ /// details | We use log-likelihood instead of likelihood because it's mathematically simpler and numerically more stable.
989
+ ✅ **Correct!**
990
+
991
+ We work with log-likelihood for several reasons:
992
+ 1. It converts products into sums, which is easier to work with mathematically
993
+ 2. It avoids numerical underflow when multiplying many small probabilities
994
+ 3. Logarithm is a monotonically increasing function, so the maximum of the likelihood occurs at the same parameter values as the maximum of the log-likelihood
995
+ ///
996
+
997
+ /// details | For a Bernoulli distribution, the MLE for parameter p is the sample mean of the observations.
998
+ ✅ **Correct!**
999
+
1000
+ For a Bernoulli distribution with parameter $p$, given $n$ independent samples $X_1, X_2, \ldots, X_n$, the MLE estimator is:
1001
+
1002
+ $$\hat{p} = \frac{\sum_{i=1}^n X_i}{n}$$
1003
+
1004
+ This is simply the sample mean, or the proportion of successes (1s) in the data.
1005
+ ///
1006
+
1007
+ /// details | For a Normal distribution, MLE gives unbiased estimates for both mean and variance parameters.
1008
+ ❌ **Incorrect.**
1009
+
1010
+ While the MLE for the mean ($\hat{\mu} = \frac{1}{n}\sum_{i=1}^n X_i$) is unbiased, the MLE for variance:
1011
+
1012
+ $$\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \hat{\mu})^2$$
1013
+
1014
+ is a biased estimator. It uses $n$ in the denominator rather than $n-1$ used in the unbiased estimator.
1015
+ ///
1016
+
1017
+ /// details | MLE estimators are always unbiased regardless of the distribution.
1018
+ ❌ **Incorrect.**
1019
+
1020
+ MLE is not always unbiased, though it often is asymptotically unbiased (meaning the bias approaches zero as the sample size increases).
1021
+
1022
+ A notable example is the MLE estimator for the variance of a Normal distribution:
1023
+ $$\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \hat{\mu})^2$$
1024
+
1025
+ This estimator is biased, which is why we often use the unbiased estimator:
1026
+ $$s^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \hat{\mu})^2$$
1027
+
1028
+ Despite occasional bias, MLE estimators have many desirable properties, including consistency and asymptotic efficiency.
1029
+ ///
1030
+ """
1031
+ )
1032
+ return
1033
+
1034
+
1035
+ @app.cell(hide_code=True)
1036
+ def _(mo):
1037
+ mo.md(
1038
+ r"""
1039
+ ## Summary
1040
+
1041
+ Maximum Likelihood Estimation really is one of those elegant ideas that sits at the core of modern statistics. When you get down to it, MLE is just about finding the most plausible explanation for the data we've observed. It's like being a detective - you have some clues (your data), and you're trying to piece together the most likely story (your parameters) that explains them.
1042
+
1043
+ We've seen how this works with different distributions. For the Bernoulli, it simply gives us the sample proportion. For the Normal, it gives us the sample mean and a slightly biased estimate of variance. And for linear regression, it provides a mathematical justification for the least squares method that everyone learns in basic stats classes.
1044
+
1045
+ What makes MLE so useful in practice is that it tends to give us estimates with good properties. As you collect more data, the estimates generally get closer to the true values (consistency) and do so efficiently. That's why MLE is everywhere in statistics and machine learning - from simple regression models to complex neural networks.
1046
+
1047
+ The most important takeaway? Next time you're fitting a model to data, remember that you're not just following a recipe - you're finding the parameters that make your observed data most likely to have occurred. That's the essence of statistical inference.
1048
+ """
1049
+ )
1050
+ return
1051
+
1052
+
1053
+ @app.cell(hide_code=True)
1054
+ def _(mo):
1055
+ mo.md(
1056
+ r"""
1057
+ ## Further Reading
1058
+
1059
+ If you're curious to dive deeper into this topic, check out "Statistical Inference" by Casella and Berger - it's the classic text that many statisticians learned from. For a more machine learning angle, Bishop's "Pattern Recognition and Machine Learning" shows how MLE connects to more advanced topics like EM algorithms and Bayesian methods.
1060
+
1061
+ Beyond the basics we've covered, you might explore Bayesian estimation (which incorporates prior knowledge), Fisher Information (which tells us how precisely we can estimate parameters), or the EM algorithm (for when we have missing data or latent variables). Each of these builds on the foundation of likelihood that we've established here.
1062
+ """
1063
+ )
1064
+ return
1065
+
1066
+
1067
+ @app.cell(hide_code=True)
1068
+ def _(mo):
1069
+ mo.md(r"""## Appendix (helper functions and imports)""")
1070
+ return
1071
+
1072
+
1073
+ @app.cell
1074
+ def _():
1075
+ import marimo as mo
1076
+ return (mo,)
1077
+
1078
+
1079
+ @app.cell
1080
+ def _():
1081
+ import numpy as np
1082
+ import matplotlib.pyplot as plt
1083
+ from scipy import stats
1084
+ import plotly.graph_objects as go
1085
+ import polars as pl
1086
+ from matplotlib import cm
1087
+
1088
+ # Set a consistent random seed for reproducibility
1089
+ np.random.seed(42)
1090
+
1091
+ # Set a nice style for matplotlib
1092
+ plt.style.use('seaborn-v0_8-darkgrid')
1093
+ return cm, go, np, pl, plt, stats
1094
+
1095
+
1096
+ @app.cell(hide_code=True)
1097
+ def _(mo):
1098
+ # Create interactive elements
1099
+ true_p_slider = mo.ui.slider(
1100
+ start =0.01,
1101
+ stop =0.99,
1102
+ value=0.3,
1103
+ step=0.01,
1104
+ label="True probability (p)"
1105
+ )
1106
+
1107
+ sample_size_slider = mo.ui.slider(
1108
+ start =10,
1109
+ stop =1000,
1110
+ value=100,
1111
+ step=10,
1112
+ label="Sample size (n)"
1113
+ )
1114
+
1115
+ generate_button = mo.ui.button(label="Generate New Sample", kind="success")
1116
+
1117
+ controls = mo.vstack([
1118
+ mo.vstack([true_p_slider, sample_size_slider]),
1119
+ generate_button
1120
+ ], justify="space-between")
1121
+ return controls, generate_button, sample_size_slider, true_p_slider
1122
+
1123
+
1124
+ @app.cell(hide_code=True)
1125
+ def _(mo):
1126
+ # Create interactive elements for Normal distribution
1127
+ true_mu_slider = mo.ui.slider(
1128
+ start =-5,
1129
+ stop =5,
1130
+ value=0,
1131
+ step=0.1,
1132
+ label="True mean (μ)"
1133
+ )
1134
+
1135
+ true_sigma_slider = mo.ui.slider(
1136
+ start =0.5,
1137
+ stop =3,
1138
+ value=1,
1139
+ step=0.1,
1140
+ label="True standard deviation (σ)"
1141
+ )
1142
+
1143
+ normal_sample_size_slider = mo.ui.slider(
1144
+ start =10,
1145
+ stop =500,
1146
+ value=50,
1147
+ step=10,
1148
+ label="Sample size (n)"
1149
+ )
1150
+
1151
+ normal_generate_button = mo.ui.button(label="Generate New Sample", kind="warn")
1152
+
1153
+ normal_controls = mo.hstack([
1154
+ mo.vstack([true_mu_slider, true_sigma_slider, normal_sample_size_slider]),
1155
+ normal_generate_button
1156
+ ], justify="space-between")
1157
+ return (
1158
+ normal_controls,
1159
+ normal_generate_button,
1160
+ normal_sample_size_slider,
1161
+ true_mu_slider,
1162
+ true_sigma_slider,
1163
+ )
1164
+
1165
+
1166
+ @app.cell(hide_code=True)
1167
+ def _(mo):
1168
+ # Create interactive elements for linear regression
1169
+ true_theta_slider = mo.ui.slider(
1170
+ start =-2,
1171
+ stop =2,
1172
+ value=0.5,
1173
+ step=0.1,
1174
+ label="True slope (θ)"
1175
+ )
1176
+
1177
+ noise_sigma_slider = mo.ui.slider(
1178
+ start =0.1,
1179
+ stop =2,
1180
+ value=0.5,
1181
+ step=0.1,
1182
+ label="Noise level (σ)"
1183
+ )
1184
+
1185
+ linear_sample_size_slider = mo.ui.slider(
1186
+ start =10,
1187
+ stop =200,
1188
+ value=50,
1189
+ step=10,
1190
+ label="Sample size (n)"
1191
+ )
1192
+
1193
+ linear_generate_button = mo.ui.button(label="Generate New Sample", kind="warn")
1194
+
1195
+ linear_controls = mo.hstack([
1196
+ mo.vstack([true_theta_slider, noise_sigma_slider, linear_sample_size_slider]),
1197
+ linear_generate_button
1198
+ ], justify="space-between")
1199
+ return (
1200
+ linear_controls,
1201
+ linear_generate_button,
1202
+ linear_sample_size_slider,
1203
+ noise_sigma_slider,
1204
+ true_theta_slider,
1205
+ )
1206
+
1207
+
1208
+ @app.cell(hide_code=True)
1209
+ def _(mo):
1210
+ # Interactive elements for likelihood vs probability demo
1211
+ concept_dist_type = mo.ui.dropdown(
1212
+ options=["Normal", "Bernoulli", "Poisson"],
1213
+ value="Normal",
1214
+ label="Distribution"
1215
+ )
1216
+
1217
+ # Replace buttons with a simple dropdown selector
1218
+ perspective_selector = mo.ui.dropdown(
1219
+ options=["Probability Perspective", "Likelihood Perspective"],
1220
+ value="Probability Perspective",
1221
+ label="View"
1222
+ )
1223
+
1224
+ concept_controls = mo.vstack([
1225
+ mo.hstack([concept_dist_type, perspective_selector])
1226
+ ])
1227
+ return concept_controls, concept_dist_type, perspective_selector
1228
+
1229
+
1230
+ if __name__ == "__main__":
1231
+ app.run()