Introduction to Bayesian Statistics

Bayesian inference updates uncertainty within a model

Bayesian inference combines a prior distribution over unknown parameters with the likelihood of observed data. The resulting posterior is proportional to their product:

$p(\theta\mid y)\propto p(y\mid\theta)p(\theta).$

The likelihood describes the data conditional on a parameter; it is not itself a probability distribution over that parameter. A prior expresses the model’s uncertainty before these data, and a posterior expresses it after updating. Neither eliminates the need to check the data-generating model.

Worked example: an unknown success probability

Suppose ten independent hypothetical attempts contain seven successes and three failures, with a common unknown probability p. Under a binomial likelihood and a Beta(a,b) prior, the posterior is:

$p\mid y\sim\operatorname{Beta}(a+7,b+3).$

Using a uniform Beta(1,1) prior gives Beta(8,4). Its mean is 8/(8 + 4) = 0.6667, compared with the sample proportion 7/10 = 0.7. The posterior mean also gives the posterior predictive success probability for one new trial under this exchangeable model.

A 95% equal-tailed credible interval uses the 2.5th and 97.5th percentiles of Beta(8,4): approximately [0.3903, 0.8907]. Conditional on the prior and likelihood, posterior probability that p lies in this interval is 95%. This is not a guarantee of frequentist coverage under all possible true probabilities.

Try a different prior and explain the difference

A Beta(5,5) prior gives a Beta(12,8) posterior with mean 0.60. It carries more concentration near 0.5 than the uniform prior, so ten observations shift it less. These prior parameters influence the result and should be justified rather than chosen to obtain a desired answer.

Calling a prior “uninformative” does not make it assumption-free; a prior that is uniform on one parameter scale need not be uniform after transforming that parameter. Compare reasonable priors, especially with small or sparse data, and report how much substantive conclusions change.

Credible intervals and confidence intervals answer different questions

A credible interval concerns posterior probability under the chosen Bayesian model. A confidence procedure concerns the frequency with which intervals contain a fixed parameter over repeated sampling under its assumptions. Similar numerical endpoints do not make their interpretations identical.

A posterior probability of exceeding a meaningful threshold can be more useful than a binary label. Decision making still needs a loss or utility: a high probability of a tiny benefit may be insufficient to justify an expensive intervention.

Check predictions, not just parameter estimates

Posterior predictive checks compare features of observed data with simulated data from the fitted model. They can reveal missing heterogeneity, dependence or unlikely tails. Good-looking predictions do not prove causal identification or validate every assumption.

The Bayes calculator on this site updates discrete event probabilities using supplied conditional probabilities. It does not fit this beta-binomial posterior or perform Bayesian regression.

Practice

With a Beta(2,3) prior and four successes in six trials, what is the posterior and its mean?

Show the explanation

Add four successes and two failures to get Beta(6,5). Its mean is 6/11 ≈ 0.5455. This update relies on the common-probability independent-trial likelihood.

Reference

The Book of Statistical Proofs derives the beta-binomial posterior. The calculations here use hypothetical data and specified priors.