This section is about what to do when we don’t really have prior information but still want to do Bayesian inference. People often say they want to “let the data speak for themselves,” and that leads to ideas like noninformative, flat, vague, or reference priors. But it turns out this is trickier than it sounds.

1. Why noninformative priors?

When a prior has no population or scientific basis, it’s hard to justify a specific distribution. So people look for a prior that:

  • influences the posterior as little as possible, and
  • is defined by some general principle, not by subjective belief.

Closely related is the idea of a weakly informative prior: not totally flat, but just enough to keep estimates in a reasonable range (to “regularize” the posterior).

2. Proper vs. improper priors

A prior is proper if it integrates to 1:

$\int p(θ)\, dθ = 1.$

An improper prior doesn’t integrate to 1 (often it integrates to ∞), e.g. $p(θ) \propto 1 \quad \text{for } θ \in (-\infty, \infty)$

or

$p(σ^2) \propto \frac{1}{σ^2}, \quad σ^2 > 0.$

Even though these aren’t real probability distributions, sometimes the posterior is still proper once you multiply by the likelihood. That’s why people use them — as limits of very vague proper priors.

Example 1: normal mean with known variance

Model:

$y_i \mid θ \sim N(θ, σ^2), \quad σ^2 \text{ known}.$

If we start with a normal prior $θ \sim N(μ_0, τ_0^2)$

and let $τ_0^2 \to \infty$, the posterior becomes

$p(θ \mid y) \approx N\left(\bar y,\; \frac{σ^2}{n}\right),$

which is exactly what we’d get if we had used the “flat” prior

$p(θ) \propto 1.$

That flat prior is improper, but the posterior is proper as long as we have data.

Example 2: normal variance with known mean

Model: $y_i \mid σ^2 \sim N(θ,\; σ^2), \quad θ \text{ known}.$

If the prior on $σ^2$ is the scaled inverse-$\chi^2$, $σ^2 \sim \text{Inv-}\chi^2(ν_0, s_0^2),$

and we let $ν_0 \to 0$, the posterior is approximately

$σ^2 \mid y \approx \text{Inv-}\chi^2(n, v),$

which is what we would get from the improper prior

$p(σ^2) \propto \frac{1}{σ^2}.$

So in both cases: improper prior → algebra still works → posterior is fine. But you must always check that the posterior integrates.

3. Jeffreys’ invariance principle

One big problem with “flat” priors is: flat in which parameterization? Flat in θ is not flat in $\log θ$.

Jeffreys’ idea: a rule for choosing a noninformative prior should not depend on how you parameterize the problem. If you re-express the parameter as $\phi = h(θ),$

then the prior for $\phi$ implied by the prior on θ should match the prior you would have gotten by applying the rule directly to the model in terms of $\phi$.

Jeffreys showed that the prior

$p(θ) \propto \sqrt{J(θ)}$

has this invariance, where $J(θ)$ is the Fisher information:

$J(θ) = \mathbb{E}\left[ \left( \frac{d}{dθ} \log p(y \mid θ) \right)^2 \right] = – \mathbb{E}\left[ \frac{d^2}{dθ^2} \log p(y \mid θ) \right].$

This is called the Jeffreys prior.

It can be shown that if you change variables to

$\phi = h(θ)$, then $p(\phi) \propto \sqrt{J(\phi)}$

comes out consistent — so this prior is parameterization-invariant.

4. Example: binomial with noninformative priors

Model:

$y \sim \text{Binomial}(n, θ)$ $\log p(y \mid θ) = \text{const} + y \log θ + (n-y)\log(1-θ)$

Compute Fisher information:

$J(θ) = \frac{n}{θ(1-θ)}$

So Jeffreys’ prior is

$p(θ) \propto \sqrt{J(θ)} = \sqrt{\frac{n}{θ(1-θ)}} \propto θ^{-1/2}(1-θ)^{-1/2}$

which is

$θ \sim \text{Beta}\left(\frac{1}{2}, \frac{1}{2}\right).$

Compare:

  • Bayes–Laplace “flat” prior: $\text{Beta}(1,1)$
  • Jeffreys prior:
    • $\text{Beta}\left(\tfrac12,\tfrac12\right)$
  • prior uniform in the logit $\text{logit}(θ) = \log \frac{θ}{1-θ}$​: corresponds to $\text{Beta}(0,0)$, which is improper.

In practice, these are often very similar once you have non-tiny data — e.g. going from Beta(0,0) to Beta(1,1) is like adding one success and one failure.

But: with the improper Beta(0,0), if you observe $y=0$ or $y=n$, the posterior can be improper, so you have to be careful.

5. Pivotal quantities and the two “nice” cases

For two special classes of parameters, all roads more or less lead to the same noninformative prior:

  1. Location parameter (pure shift):
    If $y \mid θ: \quad p(y \mid θ) = f(y – θ),$ then a natural noninformative prior is $p(θ) \propto 1 \quad (\text{uniform on } (-\infty, \infty))$ because $y – θ$ is a pivotal quantity (its distribution doesn’t depend on θ).
  2. Scale parameter (pure multiplier):
    If $y \mid θ: \quad p\left(\frac{y}{θ} \mid θ\right) = g\left(\frac{y}{θ}\right),$ then a natural noninformative prior is $p(θ) \propto \frac{1}{θ}$​ or equivalently uniform in $\log θ$.
    That’s the same prior we saw for $σ^2$: $p(σ^2) \propto 1/σ^2$.

These are the two classic “reference” forms.

6. Problems with noninformative priors

The authors also warn about overusing these:

  1. If the likelihood dominates, it doesn’t matter which flat-ish prior you choose — so hunting for “the” reference prior is sometimes pointless.
  2. Flatness depends on parameterization. Uniform in θ is not uniform in $\log θ$. That was the weakness in Laplace’s “principle of insufficient reason.”
  3. Improper priors and model comparison don’t mix well. If you need to average over models, improper priors can break things (they show this later).

So: noninformative priors are useful, but only if you

  • check the posterior is proper,
  • check sensitivity,
  • and accept they’re approximations.

7. Where weakly informative priors fit

Right at the start they mention weakly informative priors: instead of pretending to know nothing, you encode “I know the parameter is probably in a reasonable range.” This often gives better-behaved posteriors than purely improper priors, especially in more complex or hierarchical models.

TL;DR

  • Sometimes we use improper priors like $p(θ)\propto 1$ or $p(σ^2)\propto 1/σ^2$ to approximate “no prior info.”
  • These can still give proper posteriors if the data are informative.
  • Jeffreys’ prior: $p(θ) \propto \sqrt{J(θ)}$​ is parameterization-invariant.
  • For binomial, Jeffreys’ prior is $\text{Beta}(\tfrac12, \tfrac12)$.
  • But noninformative priors are not a magic answer — you still have to check the posterior, and sometimes a weakly informative prior is better.