Bayes' theorem as learning from evidence
Bayesian inference treats probability as a degree of belief and updates it as evidence arrives, exactly as Bayes' theorem prescribes.
Two meanings of probability
In the frequentist view used so far, probability is a long-run frequency, and parameters are fixed unknowns. In the Bayesian view, probability also describes uncertainty about anything, including parameters. A Bayesian can say 'there is a 70% probability the true proportion exceeds 0.5', which a frequentist confidence interval cannot.
Prior, likelihood, posterior
A Bayesian analysis starts with a prior distribution expressing beliefs before the data. The likelihood measures how probable the observed data are for each parameter value. Bayes' theorem combines them into the posterior distribution: updated beliefs after seeing the data. In short, posterior is proportional to likelihood times prior.
\text{posterior} \propto \text{likelihood} \times \text{prior}Worked example: the medical test again
With a 1% prevalence (prior), 95% sensitivity, and 5% false-positive rate, a positive test gives a posterior probability of disease of about 16%. If the patient tests positive again on an independent test, the 16% becomes the new prior, and the posterior rises to about 78%. Evidence accumulates naturally in the Bayesian framework.
Choosing priors
Priors can be informative, based on previous studies or expert knowledge, or weakly informative, spreading belief broadly. With a lot of data, the likelihood dominates and different reasonable priors lead to similar posteriors. With little data, the prior matters, so it should be stated and justified openly.
Sequential updating
Bayesian updating can proceed one observation at a time: today's posterior is tomorrow's prior. Spam filters update word probabilities with every email marked as spam; navigation systems update position estimates from each new sensor reading using closely related methods.
Odds form of Bayes' theorem
Bayes' theorem is elegant in odds: posterior odds equal prior odds times the likelihood ratio. If prior odds of disease are 1 to 99 and a positive test is 19 times more likely with the disease (0.95/0.05), the posterior odds are 19 to 99, a probability of about 16%.
\text{posterior odds} = \text{prior odds} \times \text{likelihood ratio}Where Bayesian thinking helps
Bayesian methods shine when prior information is valuable, when data arrive over time, when we need direct probability statements about hypotheses, and in complex models with many parameters, where modern computing methods such as Markov chain Monte Carlo make the calculations possible.
Common mistakes
Ignoring the prior or base rate. Choosing a prior after seeing the data to get a desired result. Forgetting that with strong priors and little data, conclusions largely reflect the prior. Confusing the likelihood with the posterior.
Key takeaways
Bayesian inference updates prior beliefs with data through the likelihood to form a posterior. Posterior is proportional to likelihood times prior, or in odds form, prior odds times likelihood ratio. Priors should be stated openly; with enough data, the data dominate.
Flashcard
Beliefs before seeing the data
Check your understanding
In Bayesian inference, the posterior is proportional to: