Parametric and Nonparametric Bootstrap Confidence Intervals

Bootstrap methods approximate sampling uncertainty by repeatedly generating new datasets and recalculating an estimator. They can estimate not only a standard error but also the shape of an estimator’s sampling distribution.

This is particularly helpful when that distribution is skewed or when a symmetric normal confidence interval is inappropriate.

The Bootstrap Framework

Suppose the unknown population parameter is\[ \theta \]

and the observed sample produces an estimate\[ \hat{\theta}. \]

A bootstrap procedure generates repeated estimates\[ \hat{\theta}^{*(1)}, \hat{\theta}^{*(2)}, \ldots, \hat{\theta}^{*(B)}. \]

Their empirical distribution is called the bootstrap distribution.

The bootstrap distribution is used to approximate the sampling distribution of \(\hat{\theta}\).

Nonparametric Bootstrap

Suppose the observed data are\[ X_1,X_2,\ldots,X_n. \]

The nonparametric bootstrap treats the empirical distribution of these observations as an approximation to the unknown population distribution.

Each observed value receives probability\[ \frac{1}{n}. \]

A bootstrap sample is obtained by drawing\[ X_1^*,X_2^*,\ldots,X_n^* \]

with replacement from the original data.

For each bootstrap sample, calculate\[ \hat{\theta}^* = T(X_1^*,\ldots,X_n^*), \]

where \(T\) is the statistic used to estimate \(\theta\).

Repeat this process \(B\) times to obtain the bootstrap distribution.

Why Sampling with Replacement Is Necessary

A bootstrap sample must have the same size as the original sample but should differ from it.

Sampling with replacement means that:

• Some observations appear more than once.
• Some observations do not appear.
• The frequencies vary across bootstrap samples.

If sampling were performed without replacement using all \(n\) observations, every bootstrap dataset would simply be a rearrangement of the original sample. Order-invariant statistics such as the mean or median would not vary.

Nonparametric Bootstrap Algorithm

1. Start with the observed data:\[ X_1,\ldots,X_n. \]
2. Draw \(n\) observations with replacement.
3. Calculate the statistic:\[ \hat{\theta}^{*(b)}. \]
4. Repeat for\[ b=1,\ldots,B. \]
5. Use the resulting estimates to calculate standard errors, bias estimates, or confidence intervals.

The bootstrap standard error is\[ \boxed{ \widehat{SE}_{\text{boot}} = SD( \hat{\theta}^{*(1)}, \ldots, \hat{\theta}^{*(B)} ) }. \]

Parametric Bootstrap

Sometimes we are willing to assume a specific probability model for the population.

For example,\[ X_i\overset{\text{iid}}{\sim}N(\mu,\sigma^2), \]

but \(\mu\) and \(\sigma\) are unknown.

Estimate them from the sample:\[ \hat{\mu}=\bar{X} \]

and\[ \hat{\sigma}=s \]

or another model-appropriate estimator.

The fitted model is then\[ N(\hat{\mu},\hat{\sigma}^2). \]

Parametric bootstrap samples are generated from this fitted distribution:\[ X_1^*,\ldots,X_n^* \overset{\text{iid}}{\sim} N(\hat{\mu},\hat{\sigma}^2). \]

The estimator is recalculated for each simulated sample.

Parametric Bootstrap Algorithm

1. Choose a parametric family \(F_\eta\).
2. Fit the model to obtain \(\hat{\eta}\).
3. Generate a dataset of size \(n\) from \(F_{\hat{\eta}}\).
4. Calculate \(\hat{\theta}^*\).
5. Repeat \(B\) times.
6. Analyze the bootstrap distribution.

The fitted probability model replaces the empirical distribution as the resampling source.

Nonparametric Versus Parametric Bootstrap

Feature	Nonparametric bootstrap	Parametric bootstrap
Resampling source	Observed empirical distribution	Fitted probability model
Distributional assumption	Relatively weak	Explicit parametric family
New values possible?	No; only observed values repeat	Yes
Potential strength	Flexible and model-light	Efficient if model is correct
Main vulnerability	Small sample may poorly represent population	Model misspecification
Typical use	General independent observations	Well-supported probability models

The parametric bootstrap may provide more stable estimates when its model is correct. If the model is wrong, it may give misleadingly precise results.

Independent Data Assumption

The ordinary nonparametric bootstrap assumes that the observations are independent and identically distributed:\[ X_1,\ldots,X_n\overset{\text{iid}}{\sim}F. \]

Resampling individual observations independently attempts to reproduce this data-generating process.

If observations are dependent, ordinary resampling destroys the dependence structure and usually produces incorrect standard errors.

Dependent Data

Dependence arises in:

• Time-series observations
• Repeated measurements
• Household data
• Students within schools
• Patients within hospitals
• Spatial data
• Matched pairs

Specialized methods include:

• Block bootstrap
• Moving-block bootstrap
• Stationary bootstrap
• Cluster bootstrap
• Paired bootstrap
• Hierarchical bootstrap

The bootstrap resampling unit should match the original unit of independence.

For example, if students are dependent within schools, resampling individual students as though they were independent is inappropriate. Entire schools may need to be resampled.

The Normal Confidence Interval

A common confidence interval has the form\[ \boxed{ \hat{\theta} \pm z^*_{1-\alpha/2} \widehat{SE}(\hat{\theta}) }. \]

Here,\[ z^*_{1-\alpha/2} = \Phi^{-1}(1-\alpha/2) \]

is a positive standard normal critical value.

For a 95% confidence interval,\[ \alpha=0.05, \]

so\[ 1-\frac{\alpha}{2} = 0.975. \]

Therefore,\[ z^*_{0.975} \approx 1.96. \]

The interval is\[ \hat{\theta} \pm 1.96\widehat{SE}(\hat{\theta}). \]

Writing \(z_{\alpha/2}\) can be ambiguous because some conventions use it for a positive upper-tail cutoff while others use it for a lower quantile. The notation\[ z^*_{1-\alpha/2} \]

makes the intended positive critical value explicit.

Bootstrap Normal Interval

If the bootstrap estimates provide a standard-error estimate, a normal-style bootstrap interval is\[ \boxed{ \hat{\theta} \pm z^*_{1-\alpha/2} \widehat{SE}_{\text{boot}} }. \]

This method uses the bootstrap only to estimate the standard error. It still assumes that the estimator is approximately normal and that bias is negligible.

If the bootstrap histogram is highly skewed, this symmetric interval may be unsuitable.

Estimating the Full Sampling Distribution

The bootstrap can estimate more than a single standard deviation.

The empirical distribution of\[ \hat{\theta}^{*(1)},\ldots,\hat{\theta}^{*(B)} \]

can reveal:

• Skewness
• Heavy tails
• Multiple modes
• Bias
• Boundary effects
• Asymmetric uncertainty

A histogram or density plot of the bootstrap estimates provides a visual approximation to the estimator’s sampling behavior.

Bootstrap Percentile Interval

Let\[ q_\gamma^* \]

denote the \(\gamma\)-quantile of the bootstrap estimates.

A \(100(1-\alpha)\%\) percentile interval is\[ \boxed{ \left( q_{\alpha/2}^*, q_{1-\alpha/2}^* \right) }. \]

For a 95% interval,\[ \boxed{ \left( q_{0.025}^*, q_{0.975}^* \right) }. \]

This takes the middle 95% of the bootstrap estimate distribution.

Percentile-Interval Example

Suppose 10,000 bootstrap estimates of a median are generated.

After sorting them, assume:\[ q_{0.025}^*=42.1 \]

and\[ q_{0.975}^*=55.8. \]

The percentile confidence interval is\[ \boxed{ (42.1,55.8) }. \]

The interval is asymmetric around the original estimate if the bootstrap distribution is skewed.

Advantages of the Percentile Interval

The percentile method is:

• Easy to calculate
• Easy to explain
• Able to produce asymmetric intervals
• Invariant under monotonic transformations

Suppose\[ \psi=g(\theta) \]

for a monotonic function \(g\). Transforming the endpoints of a percentile interval for \(\theta\) gives the corresponding percentile interval for \(\psi\).

This transformation property is valuable for positive, ratio, and log-scale parameters.

Limitations of the Percentile Interval

The percentile interval is not automatically accurate merely because it uses a nonnormal bootstrap distribution.

It can perform poorly when:

• The estimator is biased.
• The bootstrap distribution is centered incorrectly.
• Standard error changes substantially with \(\theta\).
• The statistic is nonsmooth.
• The sample is small.
• The parameter is near a boundary.
• The empirical distribution misses important population features.

The percentile method often works best for relatively smooth, low-bias estimators.

Bootstrap Distribution and Sampling Distribution

The bootstrap distribution of\[ \hat{\theta}^* \]

is centered near the observed estimate\[ \hat{\theta}. \]

The actual sampling distribution of\[ \hat{\theta} \]

is centered near the unknown parameter\[ \theta \]

when bias is small.

The bootstrap principle approximates\[ \hat{\theta}^*-\hat{\theta} \]

with\[ \hat{\theta}-\theta. \]

This connection motivates confidence intervals based on bootstrap estimation errors.

The Basic Bootstrap Interval

Define the bootstrap error as\[ \hat{\theta}^*-\hat{\theta}. \]

Let its lower and upper quantiles be\[ q^{\text{err}}_{\alpha/2} \]

and\[ q^{\text{err}}_{1-\alpha/2}. \]

If\[ P\left( q^{\text{err}}_{\alpha/2} \leq \hat{\theta}-\theta \leq q^{\text{err}}_{1-\alpha/2} \right) \approx 1-\alpha, \]

then solving for \(\theta\) gives\[ \hat{\theta} – q^{\text{err}}_{1-\alpha/2} \leq \theta \leq \hat{\theta} – q^{\text{err}}_{\alpha/2}. \]

Since\[ q^{\text{err}}_\gamma = q_\gamma^*-\hat{\theta}, \]

the interval becomes\[ \boxed{ \left( 2\hat{\theta}-q_{1-\alpha/2}^*, \, 2\hat{\theta}-q_{\alpha/2}^* \right) }. \]

This is called the basic bootstrap interval. It is also sometimes described as a reverse-percentile or pivotal-style interval.

Basic-Interval Example

Suppose\[ \hat{\theta}=50 \]

and the bootstrap quantiles are\[ q_{0.025}^*=42 \]

and\[ q_{0.975}^*=56. \]

The percentile interval is\[ (42,56). \]

The basic interval is\[ \left( 2(50)-56,\, 2(50)-42 \right), \]

so\[ \boxed{ (44,58) }. \]

The two methods reflect the bootstrap distribution differently.

Pivotal Quantities

A pivotal quantity is a function of the data and parameter whose distribution does not depend—or depends only weakly—on unknown parameters.

A familiar example is\[ \frac{ \bar{X}-\mu }{ S/\sqrt{n} }, \]

which has a Student’s t-distribution under independent normal sampling.

The unstudentized difference\[ \hat{\theta}-\theta \]

is not always a true pivot because its distribution may depend strongly on \(\theta\) or other unknown quantities.

Calling the basic bootstrap interval “pivotal” reflects its attempt to approximate the distribution of estimation error. A more fully pivotal bootstrap method generally uses studentization.

Bootstrap-t or Studentized Interval

Define the studentized statistic\[ T = \frac{ \hat{\theta}-\theta }{ \widehat{SE}(\hat{\theta}) }. \]

For bootstrap sample \(b\), compute\[ T^{*(b)} = \frac{ \hat{\theta}^{*(b)}-\hat{\theta} }{ \widehat{SE}^{*(b)} }. \]

Let\[ q_{\alpha/2}^{T^*} \]

and\[ q_{1-\alpha/2}^{T^*} \]

be quantiles of the bootstrap studentized statistics.

The bootstrap-t interval is\[ \boxed{ \left( \hat{\theta} – q_{1-\alpha/2}^{T^*} \widehat{SE}(\hat{\theta}), \, \hat{\theta} – q_{\alpha/2}^{T^*} \widehat{SE}(\hat{\theta}) \right) }. \]

Studentization can improve accuracy because the distribution of \(T\) may be less sensitive to unknown scale parameters.

The method may require a nested bootstrap if each bootstrap standard error is itself estimated by resampling, making it computationally expensive.

BCa Confidence Interval

The bias-corrected and accelerated interval adjusts the percentile levels for:

• Median bias
• Skewness
• Changes in standard error across parameter values

The interval is\[ \left( q^*_{\alpha_1}, q^*_{\alpha_2} \right), \]

where \(\alpha_1\) and \(\alpha_2\) are adjusted rather than simply 0.025 and 0.975.

BCa intervals often provide better second-order accuracy for smooth estimators, although they require additional calculations such as jackknife estimates.

Comparing Bootstrap Intervals

Method	Basic idea	Main limitation
Normal bootstrap	Estimate \(\hat{\theta}\pm z^*SE_{\text{boot}}\)	Assumes symmetry and small bias
Percentile	Use bootstrap estimate quantiles directly	Can inherit bias
Basic	Reverse bootstrap error quantiles	Not transformation invariant
Bootstrap-t	Use studentized bootstrap statistic	Computationally intensive
BCa	Adjust percentile levels for bias and acceleration	More complex

No interval method is uniformly best.

How Many Bootstrap Samples Are Needed?

The number of replications is denoted by \(B\).

Approximate guidance:

• Standard-error estimation: at least 1,000
• Basic exploration: several thousand
• Percentile endpoints: often 5,000–10,000
• Tail probabilities or high confidence levels: substantially more

The stability of the desired output should be checked rather than relying only on a fixed rule.

Quantile estimates require more replications than central averages because they depend on relatively few values in the tails.

Monte Carlo Error of Bootstrap Quantiles

Bootstrap results vary slightly with the random seed because \(B\) is finite.

Increasing \(B\) reduces this Monte Carlo error. It does not improve the quality of the original sample or correct an inappropriate bootstrap scheme.

A useful practice is to:

• Set and report a seed
• Use a sufficiently large \(B\)
• Repeat the calculation with a different seed
• Check whether the endpoints change materially

Parametric Bootstrap Example

Suppose data are modeled as\[ X_i\sim N(\mu,\sigma^2) \]

and the parameter is\[ \theta=\frac{\mu}{\sigma}. \]

The estimator is\[ \hat{\theta} = \frac{\bar{X}}{s}. \]

A parametric bootstrap would:

1. Estimate \(\mu\) with \(\bar{X}\).
2. Estimate \(\sigma\) with \(s\).
3. Generate samples from\[ N(\bar{X},s^2). \]
4. Recalculate\[ \hat{\theta}^* = \frac{\bar{X}^*}{s^*}. \]
5. Use the bootstrap estimates to calculate a standard error or confidence interval.

This makes full use of the normal-model assumption.

Nonparametric Bootstrap Example

For the same parameter, a nonparametric bootstrap would:

1. Resample the observed \(X_i\)-values with replacement.
2. Calculate\[ \hat{\theta}^* = \frac{\bar{X}^*}{s^*}. \]
3. Repeat many times.
4. Analyze the distribution of the resulting ratios.

This avoids assuming normality but depends more directly on how well the observed sample represents the population.

Confidence Intervals Must Match the Scientific Parameter

A computational method cannot repair an unclear estimand.

Before bootstrapping, define whether \(\theta\) is:

• A population mean
• A conditional mean
• A median
• A prediction
• A causal effect
• A finite-population quantity
• A superpopulation parameter

The resampling procedure must reproduce the sampling or assignment process relevant to that parameter.

Common Mistakes

Resampling Individual Observations from Dependent Data

This destroys dependence and usually understates uncertainty.

Assuming the Percentile Interval Is Automatically Superior

It accommodates asymmetry but may have poor coverage when bias is substantial.

Confusing the Basic and Percentile Intervals

They use the same bootstrap quantiles but place them differently.

Ignoring Parameter Boundaries

Some intervals can produce impossible values, such as negative variances or probabilities above 1.

Using Too Few Replications

Tail quantiles can be unstable when \(B\) is small.

Treating More Bootstrap Samples as More Real Data

Increasing \(B\) reduces computational error. It does not increase the original statistical sample size.

Key Takeaway

The nonparametric bootstrap draws samples with replacement from the observed empirical distribution. The parametric bootstrap generates samples from a fitted probability model.

Bootstrap replications can estimate the full sampling distribution of an estimator, allowing confidence intervals that are not restricted to symmetric normal approximations.

A percentile interval is\[ \boxed{ \left( q_{\alpha/2}^*, q_{1-\alpha/2}^* \right) }. \]

A basic bootstrap interval is\[ \boxed{ \left( 2\hat{\theta}-q_{1-\alpha/2}^*, \, 2\hat{\theta}-q_{\alpha/2}^* \right) }. \]

A studentized bootstrap interval uses the distribution of\[ \frac{ \hat{\theta}^*-\hat{\theta} }{ \widehat{SE}^* }. \]

Bootstrap methods are powerful but not assumption-free. The resampling scheme must preserve the independence, dependence, grouping, and model structure of the original data, and the selected confidence-interval method must be appropriate for the estimator’s behavior.