Posterior

The posterior is what a Bayesian analysis produces: a distribution over the parameter, given the data. Bayes' theorem in the form used for estimation drops the normalizing constant and reads

\[ p(\theta \mid D) \propto p(D \mid \theta)\, p(\theta) \]

The posterior combines the likelihood and prior into a distribution. It may be discrete, continuous, mixed or multivariate. Here the parameter is a single coin proportion and the posterior is a continuous density curve, from which several different summaries can be taken.

A curve, and the numbers taken from it

For a constant coin probability θ, the seven-head, three-tail likelihood is proportional to θ⁷(1 − θ)³. A Beta(a,b) prior contributes θ^(a−1)(1−θ)^(b−1), so multiplying gives Beta(a+7,b+3). A Beta(1,1) prior therefore gives a Beta(8,4) posterior. The normalizing constant turns this shape into a density with total area one; proportional notation does not eliminate that requirement. For this distribution, the mean is 0.6667, mode 0.7000 and median 0.6762. Their difference is a property of this example, not a rule that all asymmetric distributions must have different summaries. Reporting a posterior estimate should identify which summary is being used.

An equal-tail 95% credible interval of approximately [0.3903, 0.8907] leaves 2.5% posterior probability below and 2.5% above. Its interpretation is: given the model and the prior, there is a 95% probability that \(p\) lies in this range. A confidence interval makes a claim about the procedure over repeated samples, not about this interval. These are different probability statements. The credible interval depends on the chosen prior and likelihood; confidence-interval coverage also depends on the procedure and its sampling assumptions.

The interval width is about 0.5005, roughly half the possible parameter range. A mean of 0.67 by itself hides that uncertainty. For a continuous posterior, probabilities are areas over ranges, not the height of the density at a point. P(θ > 0.5 | D) is the area to the right of 0.5, about 0.8867 here. An exact parameter value has probability zero under this continuous distribution, even at its mode.

Concentration, and what it costs

Consider illustrative datasets with the observed proportion held at 70% while sample size grows. The mean converges on 0.70, the interval narrows from 0.5005 to 0.0254, and \(P(p > 0.5 \mid D)\) goes from 0.886719 toward one (values displayed as 1.000000 may still leave a nonzero tail). Once the sample is sufficiently informative in this regular interior-proportion example, a tenfold increase roughly divides interval width by √10. This is not a universal concentration rate for every Bayesian model.

Here the posterior mean changes little as the sample grows while uncertainty changes substantially. A point estimate and an interval therefore answer different questions. At \(n = 10\) the evidence that the coin favors heads is real but weak; at \(n = 200\), the posterior probability is very high under this model and prior, while the opposite tail still has probability about \(5.34\times10^{-9}\). The mean barely distinguishes the two cases.

Predicting a new observation

The posterior describes an unknown parameter; the posterior predictive distribution describes a future observation after averaging over that parameter uncertainty. In this coin model, the next-toss head probability is E[θ | D] = 2/3. For two future tosses, the probability both are heads is E[θ² | D] = 8 × 9 / (12 × 13) = 6/13, about 0.4615. Squaring the mean gives 4/9, about 0.4444, and misses shared uncertainty in θ. Future tosses are independent conditional on θ, but need not be independent after θ is averaged out.

Updating is associative

The same Beta(8,4) distribution is reached by adding all seven heads and three tails together or by updating the Beta counts after each toss. General sequential updating uses \(p(\theta\mid D_1,D_2)\propto p(D_2\mid\theta,D_1)p(\theta\mid D_1)\). In this iid coin model, the conditional likelihood factors reduce to the familiar per-toss terms.

Yesterday’s posterior can therefore serve as today’s prior when the next likelihood correctly conditions on the previous observations and the model remains the same. Permuting the tosses leaves this iid model unchanged. A time-series model may instead use order-dependent conditional likelihoods and still support sequential updating; order-invariance is not a requirement of Bayesian inference.

The Beta-binomial pair is conjugate, so the posterior stays in the Beta family and updating reduces to two counters. Outside conjugate families, analytic computation can be harder, but is not always impossible. Depending on dimension and structure, options include direct integration or quadrature, grids, Laplace approximations, sampling and variational methods.

References

• SciPy documentation, scipy.stats.beta (ppf, cdf, median). docs.scipy.org
• SciPy documentation, statistical functions overview. docs.scipy.org