Probability Distribution

A probability distribution is the complete assignment of probability to the outcomes a random variable can take. Not a summary of it — the whole thing. Everything you can ask about the variable is answered by it, and nothing that is not in it can be recovered from summaries.

That last point is the reason this term needs an entry of its own rather than a line in a glossary of moments.

Same mean, same variance, different world

import numpy as np
from scipy.stats import norm, uniform, laplace, t as student

shapes = {"Normal": norm(0, 1),
          "Uniform": uniform(-np.sqrt(3), 2 * np.sqrt(3)),
          "Laplace": laplace(0, 1 / np.sqrt(2)),
          "Student t, 5 df": student(5, scale=np.sqrt(3 / 5))}
print(f"{'distribution':>18}{'mean':>8}{'variance':>11}{'P(|X| > 2)':>13}{'P(|X| > 4)':>13}")
for name, d in shapes.items():
    tail2 = 2 * d.sf(2)
    tail4 = 2 * d.sf(4)
    print(f"{name:>18}{d.mean():>8.4f}{d.var():>11.4f}{tail2:>13.6f}{tail4:>13.8f}")
print("  same center, same spread. Past 4 the normal gives 0.00006334 and the t")
print("  gives 0.00357283 - fifty-six times more - while the uniform gives exactly")
print("  zero, because a bounded distribution simply has no values out there")
"""A distribution is the whole assignment of probability to outcomes. Two
   summaries agreeing says nothing about the rest of it."""

print(f"\n  what the shape decides, in one number a business would use:")
for name, d in shapes.items():
    print(f"    {name:>16}: the 99.9th percentile sits at {d.ppf(0.999):>7.3f}")
print("  a capacity plan built on mean and variance alone is choosing one of these")
print("  four rows without saying which")

print(f"\n  and the summaries need not exist at all:")
cauchy_like = student(1)
print(f"    Student t with 1 df has no mean and no variance - the defining integrals")
print(f"    do not converge, and SciPy reports {cauchy_like.mean()} and {cauchy_like.var()}")
print(f"    yet it is a perfectly good distribution: P(|X| <= 1) = "
      f"{cauchy_like.cdf(1) - cauchy_like.cdf(-1):.4f}, P(|X| <= 10) = "
      f"{cauchy_like.cdf(10) - cauchy_like.cdf(-10):.4f}")
#       distribution    mean   variance   P(|X| > 2)   P(|X| > 4)
#             Normal  0.0000     1.0000     0.045500   0.00006334
#            Uniform  0.0000     1.0000     0.000000   0.00000000
#            Laplace  0.0000     1.0000     0.059106   0.00349349
#    Student t, 5 df  0.0000     1.0000     0.049313   0.00357283
#   same center, same spread. Past 4 the normal gives 0.00006334 and the t
#   gives 0.00357283 - fifty-six times more - while the uniform gives exactly
#   zero, because a bounded distribution simply has no values out there
#
#   what the shape decides, in one number a business would use:
#               Normal: the 99.9th percentile sits at   3.090
#              Uniform: the 99.9th percentile sits at   1.729
#              Laplace: the 99.9th percentile sits at   4.394
#      Student t, 5 df: the 99.9th percentile sits at   4.565
#   a capacity plan built on mean and variance alone is choosing one of these
#   four rows without saying which
#
#   and the summaries need not exist at all:
#     Student t with 1 df has no mean and no variance - the defining integrals
#     do not converge, and SciPy reports inf and nan
#     yet it is a perfectly good distribution: P(|X| <= 1) = 0.5000, P(|X| <= 10) = 0.9365

Four distributions, all centered at 0 with variance exactly 1. Past 2 they already disagree — 0.0455, 0.0000, 0.0591, 0.0493. Past 4 the normal gives 0.00006334 and the Student t gives 0.00357283, fifty-six times more, while the uniform gives exactly zero because a bounded distribution has no values out there at all.

The second block turns that into a decision. The 99.9th percentile — the number a capacity plan or a risk limit is actually built from — is 1.729 under the uniform and 4.565 under the t. That is a factor of 2.6 in required headroom, with identical mean and variance on the input.

So “mean 0, standard deviation 1” does not describe a distribution. It describes a family of them that disagree about exactly the region most decisions care about. Anyone who reports a mean and a spread and lets a reader fill in the shape has chosen one of these rows silently.

Summaries can fail to exist

The last block is the stronger version of the same point. A Student t with one degree of freedom has no mean and no variance — the defining integrals do not converge, and SciPy returns inf and nan rather than numbers.

It is nevertheless a perfectly ordinary distribution: \(P(|X| \le 1) = 0.5000\) and \(P(|X| \le 10) = 0.9365\), both entirely well defined. The distribution is the primary object; the mean is a derived quantity that sometimes does not exist. Computing a sample mean from such data produces a number that does not converge to anything as the sample grows.

How a distribution is written down

Three representations carry the same information, and which one is available depends on the variable.

• For a discrete variable, a probability mass function lists \(P(X = x)\) for each value. The numbers are probabilities and they sum to 1.
• For a continuous variable, a probability density function gives probability per unit of \(x\), and probabilities come from integrating it. The values are not probabilities and are not bounded by 1.
• A cumulative distribution function gives \(P(X \le x)\) and works for both, which is why it is the definition that generalizes.

Named families — normal, binomial, Poisson, exponential — are parameterized shapes chosen because they arise from identifiable mechanisms or because they are tractable. Choosing one is an assumption about the mechanism, and the table above is what that assumption is worth: the difference between a factor of 2.6 in your capacity plan and not knowing.

References

• SciPy documentation, continuous distributions in scipy.stats. docs.scipy.org
• SciPy documentation, scipy.stats.t. docs.scipy.org