Bayesian inference

Summary

Bayesian inference is a method of statistical inference that treats an unknown parameter $\theta$ as a random variable. It uses Bayes' theorem to update a probability distribution for $\theta$ when data arrive. One starts from a prior distribution with density $\pi(\theta)$ and a model with likelihood $L(\theta \mid x) = f(x \mid \theta)$. The posterior distribution has the density $$\pi(\theta \mid x) = \frac{f(x \mid \theta)\, \pi(\theta)}{\int_\Theta f(x \mid \theta')\, \pi(\theta')\, d\theta'}.$$

Remarks

In short, the posterior is proportional to the likelihood times the prior. Example: with a $\operatorname{Beta}(a, b)$ prior for the success probability $p$, and $k$ successes in $n$ Bernoulli($p$) trials, the posterior is $\operatorname{Beta}(a + k, b + n - k)$ (see beta distribution). The prior is conjugate. The posterior gives point estimates, such as the posterior mean, and credible intervals. For new observations, the posterior predictive distribution is $f(\tilde{x} \mid x) = \int_\Theta f(\tilde{x} \mid \theta)\, \pi(\theta \mid x)\, d\theta$, if $\tilde{x}$ and $x$ are conditionally independent given $\theta$.

Explanations

L1 human

Bayesian inference starts with a belief about an unknown quantity, written as a probability distribution. Each new piece of data updates this belief with Bayes' theorem. The result, the posterior, combines the prior belief with the evidence.