Likelihood

The likelihood evaluates the observed data under a model and treats that value as a function of the parameter. It uses a probability mass for discrete data and a probability density for continuous data. For conditionally independent observations \(x_1,\dots,x_n\) given parameter \(\theta\), it has the form:

\[ L(\theta) = \prod_{i=1}^{n} f(x_i \mid \theta) \]

The observed data are held fixed while the parameter varies. For continuous data, the probability of an exact point is zero, but its density need not be zero or lie below one. Likelihood compares the mass or density assigned by different parameter values; it is not a probability distribution over those values.

Why it is not a probability for the parameter

Suppose ten conditionally independent tosses contain seven heads and three tails, with an unknown constant head probability θ. For one particular ordered sequence, L(θ) = θ⁷(1 − θ)³. At θ = 0.5 this is 0.0009766; at θ = 0.7 it is about 0.0022236. The ratio is about 2.277, so the observed sequence has that much greater probability under θ = 0.7 than under θ = 0.5. It does not say there is a 0.0022236 probability that θ equals 0.7.

Changing θ asks which parameter value better explains the same observations. It does not generate a distribution over θ. A likelihood need not integrate to one over parameter values, and multiplying it by a positive parameter-independent constant leaves likelihood ratios unchanged. Turning it into a posterior distribution requires a prior and normalization using Bayes’ theorem. Simply normalizing a likelihood over one chosen parameter coordinate implicitly makes a modeling choice, rather than producing a parameterization-free probability of truth.

The maximum in this binomial example is 0.70, the observed proportion 7/10. Maximum likelihood estimation chooses parameters that maximize the observed data’s probability mass or density under the specified model. For many models this optimization is numerical.

Constants cancel, and that matters

If the observation is only the count of seven heads, rather than one particular order, there are 120 such sequences. The binomial likelihood is therefore 120θ⁷(1 − θ)³. At θ = 0.7 it is about 0.266828. The ratio comparing 0.7 with 0.5 remains 2.2769 because 120 appears in both numerator and denominator.

The factor 120 is common to every value of the coin parameter, so it cancels in parameter likelihood ratios and does not change the MLE here. This permission is local to that comparison. When comparing models through their evidence or a Bayes factor, keep normalizing factors unless they are common to both models for the same observed data. A count and a particular ordered sequence are different data representations; their raw likelihoods should not be compared as if they described the same event.

Why everything is written in logs

In this coin example, multiplying probabilities below one produces very small likelihoods: for an ordered sequence with 700 heads and 300 tails, L(0.7) is about 5.07 × 10⁻²⁶⁶. Doubling those counts puts the direct product below ordinary double-precision range. Computing the log-likelihood directly avoids that product. Continuous densities can exceed one, so underflow is not the only numerical issue; log calculations also help manage a wide dynamic range.

Here the log likelihood is 7 log θ + 3 log(1 − θ). Differentiating gives 7/θ − 3/(1 − θ), which is zero at θ = 0.7; this is the interior maximum for these counts. Taking logs preserves the location of a positive likelihood’s maximum and turns products into sums. Binary cross-entropy corresponds to a Bernoulli negative log-likelihood; multiclass cross-entropy corresponds to a categorical one. Squared error corresponds to Gaussian negative log-likelihood when variance is fixed, up to constants and a positive scale.

One caution about the model behind it all. The likelihood is computed from an assumed family of distributions, so it measures fit within that family. A larger likelihood favors one parameter value over another for the same data and model comparison. Its absolute size is not a general goodness-of-fit score: it depends on sample size, data representation, and, for densities, measurement units. Check model adequacy through diagnostics and predictive evaluation rather than treating a maximum as proof of a good model. Dependence matters too: correlated observations require their joint likelihood or correctly conditioned factors, not an unsupported product of independent terms.

References

• SciPy documentation, scipy.integrate.quad. docs.scipy.org
• SciPy documentation, scipy.special.comb. docs.scipy.org