Probability Density

A probability density function describes how probability is spread across the values of a continuous variable. It is not probability. The probability of a range comes from integrating it:

\[ P(a \le X \le b) = \int_a^b f(x)\,dx \]

The density must be non-negative everywhere and integrate to 1 over the whole line. Those two conditions are the entire definition, and neither of them says \(f(x) \le 1\).

Three things that surprise people

import numpy as np
from scipy.stats import norm, uniform, beta

print(f"  a density is not a probability, and it is not capped at 1:")
for name, dist, at in (("Uniform(0, 0.1)", uniform(0, 0.1), 0.05),
                       ("Beta(5, 2)", beta(5, 2), 0.8),
                       ("Normal(0, 1)", norm(0, 1), 0.0),
                       ("Normal(0, 0.01)", norm(0, 0.01), 0.0)):
    print(f"    f({at}) under {name:<16} = {dist.pdf(at):>9.4f}"
          f"   total area {dist.cdf(np.inf) - dist.cdf(-np.inf):.4f}")
"""Density is probability per unit of x. Squeeze the same total probability
   into a narrower range and the density has to rise to compensate."""

print(f"\n  the probability of any single value is exactly zero:")
for w in (1e-2, 1e-4, 1e-8):
    print(f"    P(|X| <= {w:g}) under Normal(0, 1) = {norm.cdf(w) - norm.cdf(-w):.10f}")
print("    the limit is 0, which is why P(X = 0) = 0 while f(0) = 0.3989")

heights = norm(170, 10)
print(f"\n  density carries units, so it changes when you change them:")
print(f"    heights in cm: f(170) = {heights.pdf(170):.6f} per cm")
print(f"    the same heights in meters: f(1.70) = {norm(1.70, 0.10).pdf(1.70):.6f} per meter")
print(f"    ratio {norm(1.70, 0.10).pdf(1.70) / heights.pdf(170):.0f}, exactly the unit change")
print(f"    but the probability is the same: P(160 <= X <= 180) = "
      f"{heights.cdf(180) - heights.cdf(160):.6f} = "
      f"{norm(1.70, 0.10).cdf(1.80) - norm(1.70, 0.10).cdf(1.60):.6f}")
print("  comparing densities across different units is comparing nothing")
#   a density is not a probability, and it is not capped at 1:
#     f(0.05) under Uniform(0, 0.1)  =   10.0000   total area 1.0000
#     f(0.8) under Beta(5, 2)       =    2.4576   total area 1.0000
#     f(0.0) under Normal(0, 1)     =    0.3989   total area 1.0000
#     f(0.0) under Normal(0, 0.01)  =   39.8942   total area 1.0000
#
#   the probability of any single value is exactly zero:
#     P(|X| <= 0.01) under Normal(0, 1) = 0.0079787126
#     P(|X| <= 0.0001) under Normal(0, 1) = 0.0000797885
#     P(|X| <= 1e-08) under Normal(0, 1) = 0.0000000080
#     the limit is 0, which is why P(X = 0) = 0 while f(0) = 0.3989
#
#   density carries units, so it changes when you change them:
#     heights in cm: f(170) = 0.039894 per cm
#     the same heights in meters: f(1.70) = 3.989423 per meter
#     ratio 100, exactly the unit change
#     but the probability is the same: P(160 <= X <= 180) = 0.682689 = 0.682689
#   comparing densities across different units is comparing nothing

The first block is the one that matters most. A Uniform(0, 0.1) has density 10 at every point in its range, and a Normal with standard deviation 0.01 reaches 39.8942 at its center. Both integrate to exactly 1. Density is probability per unit of \(x\), so squeezing the same total into a narrower range forces the density up to compensate. A density of 39.89 is not a probability of 3,989%; it is a rate.

The second block is the statement that a continuous variable assigns zero probability to every single value. Shrink the window around 0 and \(P(|X| \le w)\) goes 0.0079787126, 0.0000797885, 0.0000000080 — heading to zero in proportion to the window. In the limit it is exactly 0, which is why \(P(X = 0) = 0\) while \(f(0) = 0.3989\). The two numbers are answers to different questions.

A practical consequence follows immediately: for a continuous variable \(P(X \le x)\) and \(P(X < x)\) are equal, and moving an endpoint from strict to non-strict changes nothing. For a discrete variable it changes the answer by exactly the mass at that point.

Density has units

The third block is the one that is almost never stated and quietly causes trouble. The same distribution of heights has \(f(170) = 0.039894\) per centimeter and \(f(1.70) = 3.989423\) per meter. The ratio is exactly 100, the unit change — while the probability of the interval is 0.682689 either way.

So a density value has no meaning without its unit, and comparing density values measured in different units compares nothing. This matters whenever densities are read off a plot with a rescaled axis, or when a likelihood is reported as a number: a likelihood built from densities inherits the same unit dependence, which is one more reason only ratios of it are interpretable.

It also explains a common confusion about kernel density plots. The height of the curve depends on the horizontal units and on the bandwidth; only areas under it are probabilities, and two plots with different axes cannot be compared by eye at a point.

Where it fits

The density is one of three equivalent ways to write a distribution down, and it is the one that only exists for continuous variables. The cumulative distribution function exists for both and is related to the density by differentiation: \(f(x) = F'(x)\) wherever \(F\) is differentiable.

That relationship is the practical route to most calculations. Software gives you \(F\) and its inverse; interval probabilities are differences of \(F\), and the density is what you need when the object of interest is the shape itself — plotting it, maximizing it, or multiplying it by a prior.

References

• SciPy documentation, scipy.stats.norm (pdf, cdf). docs.scipy.org
• SciPy documentation, scipy.stats.beta and scipy.stats.uniform. docs.scipy.org