Bootstrapping - Ciallo Wiki

1 Bootstrapping Basics

1.1 Core idea: resampling with replacement

Bootstrapping estimates the uncertainty of a statistic by repeatedly creating new “pseudo-datasets” from the observed data. Each pseudo-dataset is formed by sampling observations with replacement from the original dataset, typically using the same sample size as the original. For every resample, the statistic of interest is recomputed, producing an empirical approximation to how the statistic would vary if new data were collected under similar conditions.

1.2 What bootstraps estimate (distribution, SE, CI, bias)

From the collection of bootstrap replicates, one can estimate:

Sampling distribution of a statistic (via the empirical distribution of bootstrap replicate values).
Standard error (SE) as the standard deviation of bootstrap replicates.
Confidence intervals (CI) using resampling-based interval constructions.
Bias by comparing the mean (or other functional) of bootstrap replicates to the statistic computed on the original data.

1.3 Relationship to the empirical distribution

The theoretical justification of the bootstrap can be understood in terms of the empirical distribution. The empirical distribution assigns probability mass to each observed data point. The bootstrap simulates repeated sampling from this empirical distribution rather than from an unknown true data-generating process. Under conditions where the empirical distribution is a good proxy, the bootstrap replicates approximate the real sampling variability of the statistic.

1.4 Choosing the bootstrap statistic and number of resamples

The bootstrap must track the quantity you care about. If the statistic is nonlinear, involves transformations, or depends on tuning steps, the resampling procedure should mirror the full computation pipeline. The number of resamples affects Monte Carlo error: too few replicates yield noisy SEs and unstable interval endpoints, while larger numbers improve stability at the cost of computation. Common practice uses hundreds to tens of thousands of resamples depending on the precision required and computational budget.

2 Setup and Assumptions

2.1 Data requirements and notation

Consider observed data \(X_1,\dots,X_n\). Let \(\hat{\theta} = s(X_1,\dots,X_n)\) denote a statistic computed from the sample, such as a mean, median, regression coefficient, or another estimator. A bootstrap resample \(X_1^*,\dots,X_n^*\) is generated by drawing \(n\) points with replacement from \(\{X_1,\dots,X_n\}\). The statistic computed on the resample is \(\hat{\theta}^* = s(X_1^*,\dots,X_n^*)\). Repeating this yields \(\hat{\theta}^{*(1)},\dots,\hat{\theta}^{*(B)}\).

2.2 Independent and identically distributed (i.i.d.) considerations

Many foundational bootstrap results assume observations are independent and identically distributed. When this holds approximately, resampling observations as exchangeable units can reproduce the relevant variability.

2.2.1 Practical checks for dependence

In practice, analysts look for evidence that observations are not interchangeable. Examples include:

temporal ordering (time dependence),
repeated measures on the same subject (within-cluster dependence),
spatial or network proximity (correlated sampling),
sampling schemes that induce unequal inclusion or dependence structures.

When such dependence is present, standard i.i.d. bootstrapping may not reflect uncertainty correctly.

2.3 Handling small samples and discreteness

With small \(n\), the empirical distribution is coarse: each observation may represent a large jump in probability mass. This can lead to:

limited variety in bootstrap samples,
discretized or “stepped” confidence intervals,
wider uncertainty that may be hard to quantify precisely.

Discreteness in the underlying variable also affects interval smoothness, sometimes motivating interval methods designed to behave better under skewness or nonregularity.

2.4 When bootstrap assumptions may fail

Bootstrap approximations may be inaccurate when:

the statistic is highly sensitive to rare observations,
the estimator’s limiting behavior is nonstandard (for example, certain boundary or nonregular parameters),
dependence is strong but not addressed by a dependent-data bootstrap variant,
the resampling scheme does not respect key aspects of data generation (such as censoring mechanisms or complex survey designs).

3 Bootstrap Algorithms and Variants

3.1 Nonparametric bootstrap

The nonparametric bootstrap uses only the observed data without imposing a parametric model. It resamples from the empirical distribution, making it broadly applicable.

3.1.1 Steps for a generic estimator

A generic nonparametric workflow:

Compute the statistic \(\hat{\theta}\) from the original dataset.
For \(b=1,\dots,B\):

draw a bootstrap sample \(X_1^*,\dots,X_n^*\) by sampling with replacement from the observed data,
compute \(\hat{\theta}^{*(b)}\) from that resample.

Use \(\{\hat{\theta}^{*(b)}\}_{b=1}^B\) to estimate SE, bias, or construct a CI.

3.2 Parametric bootstrap

The parametric bootstrap assumes a parametric model for the data (e.g., a distribution family) and resamples from the fitted model. This can improve performance when the model is reasonable and the statistic depends on distributional structure.

3.2.1 Model fitting and simulation workflow

A typical parametric workflow:

Fit a model to the observed data to obtain parameter estimates \(\hat{\psi}\).
For \(b=1,\dots,B\):

simulate a synthetic dataset from the model using \(\hat{\psi}\),
refit the model on the synthetic dataset (if required by the target statistic),
compute the statistic \(\hat{\theta}^{*(b)}\).

Derive SE/bias/CI from the simulated replicate distribution.

Parametric bootstrap CIs often require careful mirroring of estimation steps so that the resamples reflect the same procedure used on the real data.

3.3 Residual and wild bootstrap (overview-level)

Residual-based methods appear mainly in regression contexts. The key idea is to keep fitted structure (such as regression functions) while resampling or perturbing residuals to reproduce uncertainty.

3.3.1 Residual bootstrap for regression contexts

In a residual bootstrap, one often:

fits a regression model,
computes residuals,
resamples residuals and adds them back to fitted values to form new response vectors,
refits the regression model and recalculates the statistic.

This aims to preserve the relationship between predictors and fitted mean while accounting for randomness in the errors.

3.3.2 Wild bootstrap for heteroskedasticity

When the error variance changes with predictors (heteroskedasticity), a wild bootstrap perturbs residuals using random variables designed to handle uneven variance. It is intended to produce replicate residual patterns that align with changing error scales.

4 Confidence Intervals via Bootstrapping

4.1 Percentile interval

The percentile interval uses the empirical quantiles of the bootstrap replicates. If the bootstrap distribution of \(\hat{\theta}^*\) is \(\{\hat{\theta}^{*(b)}\}\), then a nominal \(100(1-\alpha)\%\) interval can be formed from the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the bootstrap replicates. This method is intuitive and often works reasonably, but it can be sensitive to bias and skewness.

4.2 Basic (reverse percentile) interval

The basic interval reverses the percentile endpoints around the original estimate. If the percentile endpoints are taken from the bootstrap quantiles, the basic construction typically reflects them to correct for bias in a rough sense. It can perform better than the percentile method when the estimator’s distribution is asymmetric.

4.3 Studentized (t) interval

The studentized interval standardizes bootstrap replicates using an estimate of their variability, often yielding a “t-like” statistic: \[ t^* = \frac{\hat{\theta}^* - \hat{\theta}}{\widehat{\text{SE}}^*} \] The interval is then constructed from quantiles of these studentized values. This method can improve coverage when the standard error varies with the bootstrap world.

4.4 Bias-corrected and accelerated (BCa) interval

The BCa method adjusts both for bias and for changes in variability with the true parameter value. It typically uses information from:

the proportion of bootstrap estimates below the observed \(\hat{\theta}\) (bias adjustment),
an acceleration term related to how the statistic behaves under resampling (often linked to influence or jackknife diagnostics).

BCa intervals are widely used when the bootstrap distribution is skewed or the bias is non-negligible.

4.5 Coverage, skewness, and interval behavior

Bootstrap CIs are evaluated by their coverage probability: the chance that the interval contains the true parameter. Interval behavior depends on:

skewness of the statistic’s sampling distribution,
bias of the estimator,
tail thickness and nonlinearity,
adequacy of the resampling scheme relative to the data-generating process.

Skewed distributions often motivate BCa or studentized approaches rather than raw percentile intervals.

4.6 Selecting methods when distributions are nonstandard

When \(\hat{\theta}\) is constrained (e.g., parameters that cannot be negative), or when the distribution is strongly nonnormal, some interval constructions can misbehave. Selection is often guided by:

diagnostic plots of bootstrap replicates,
known theoretical guidance for the statistic,
comparison of multiple interval types on pilot runs,
attention to whether the statistic exhibits boundary behavior or strong skewness.

5 Bias and Standard Error Estimation

5.1 Bootstrap standard error estimation

The bootstrap SE is usually computed as the standard deviation of the bootstrap replicate values: \[ \widehat{\text{SE}}_{\text{boot}} = \sqrt{\frac{1}{B-1}\sum_{b=1}^B(\hat{\theta}^{*(b)}-\bar{\theta}^*)^2} \] where \(\bar{\theta}^* = \frac{1}{B}\sum_{b=1}^B \hat{\theta}^{*(b)}\). This estimates how much \(\hat{\theta}\) would vary if resampling reflected the relevant uncertainty.

5.2 Estimating bias using bootstrap replicates

A common bootstrap bias estimate is: \[ \widehat{\text{Bias}}_{\text{boot}} = \bar{\theta}^* - \hat{\theta} \] This compares the average bootstrap replicate to the original estimate. While helpful, bias estimates can be noisy for small \(B\) or unstable when the statistic is highly nonlinear.

5.3 Bias-corrected estimates

Bias-corrected point estimates attempt to adjust \(\hat{\theta}\) using the bootstrap bias estimate, for example: \[ \hat{\theta}_{\text{bc}} = \hat{\theta} - \widehat{\text{Bias}}_{\text{boot}} \] This can improve accuracy for some estimators, though it may worsen performance for others. In many workflows, bias correction is more naturally incorporated into the CI method rather than only into the point estimate.

5.4 Accuracy considerations (order-of-magnitude intuition)

Bootstrap accuracy depends on how closely the bootstrap approximation matches the estimator’s true sampling distribution. Informally:

With larger samples, bootstrap approximations tend to improve.
For complex estimators, the effective approximation can be limited by how well resampling captures estimator behavior.
Monte Carlo error from finite \(B\) adds additional noise; increasing \(B\) reduces this effect.

6 Estimators and Targets

6.1 Mean, median, and other location statistics

Location parameters are among the most common targets. The mean is sensitive to outliers, whereas the median is more robust to extreme values. Both can be bootstrapped, but interval behavior may differ: medians often yield more skewness in the sampling distribution when data are asymmetric, which affects the choice of interval construction.

6.2 Regression coefficients (high-level)

For regression, bootstrap methods can be applied to coefficients, fitted values, and derived quantities (such as contrasts between groups). The appropriate bootstrap variant depends on whether errors are independent, identically distributed, heteroskedastic, or dependent due to clustering or time structure. High-level guidance is to ensure that the resampling scheme respects the model’s error structure.

6.3 Variance and scale parameters

Scale estimation—such as variance, standard deviation, or other spread metrics—often has skewed sampling distributions. As a result, percentile intervals may be less reliable, and bias or studentization adjustments can improve coverage. When scale parameters are strictly positive, interval constructions may need to account for boundary effects.

6.4 Functionals and complex estimators

Bootstrapping generalizes beyond simple statistics. It can be used for:

nonlinear functionals of the data (e.g., ratios, transforms),
statistics based on multiple steps (e.g., preprocessing or selection),
performance metrics or composite indices.

However, resampling must reproduce the same computation pipeline, including any tuning rules, to avoid mismatch between the bootstrap world and the estimation world.

6.5 Bootstrap for ranking and comparing groups (overview)

Comparisons often involve estimating a difference between group-level statistics or ranking multiple candidates. A bootstrap can provide uncertainty for:

pairwise differences,
aggregated performance metrics,
uncertainty-aware rankings based on resampled distributions.

In such settings, dependence among comparisons (e.g., when groups share data transformations) should be considered when interpreting joint behavior.

7 Diagnostics and Best Practices

7.1 Monitoring convergence with number of resamples

Because bootstrap intervals and SEs are Monte Carlo estimates, analysts should check whether results stabilize as \(B\) increases. Practical indicators include:

SE estimates changing minimally after doubling \(B\),
CI endpoints stabilizing across runs,
distribution summaries (quantiles, bias estimates) becoming consistent.

If stability is not achieved, increasing \(B\) or improving numerical procedures may be necessary.

7.2 Sensitivity to outliers and influential observations

Bootstrapping resamples observations with replacement, meaning an outlier can appear multiple times in a resample and strongly affect replicate values. Consequently, bootstrap results can be sensitive to influential points. Best practice includes:

inspecting the original data for leverage and outliers,
examining bootstrap replicate distributions for extreme tails,
considering robust alternatives when appropriate.

7.3 Stability across random seeds and resampling schemes

Different random seeds generate different bootstrap sequences. While the overall results should be similar, small-sample settings or highly nonlinear statistics may show noticeable variability. Analysts can use repeated bootstrap runs to assess stability and verify that conclusions do not hinge on one particular resampling realization.

7.4 Computational considerations and performance tips

Computational cost grows with both \(B\) and the complexity of computing \(\hat{\theta}\). Common performance strategies include:

vectorizing computations when possible,
parallelizing across resamples,
storing only the needed replicate summaries (e.g., quantiles) rather than full replicate arrays,
using efficient random number generation and reproducible seeding.

8 Extensions for Dependent Data (Conceptual)

8.1 Block bootstrap rationale

For dependent observations (such as time series), resampling individual points can break correlation structure. A block bootstrap instead resamples contiguous blocks, preserving within-block dependence while approximating dependence across blocks.

8.1.1 Choosing block length (trade-offs)

Block length controls the bias-variance trade-off:

short blocks may not preserve dependence long enough,
long blocks better represent dependence but reduce the number of effectively independent resampled units.

Selecting block size often involves heuristic rules or data-driven methods that balance these competing effects.

8.2 Moving block vs. stationary bootstrap (overview)

Two common schemes are:

Moving block bootstrap, which draws blocks from the set of all contiguous blocks of a fixed length and concatenates them to form bootstrap samples.
Stationary bootstrap, which uses random block lengths to better reflect stationary dependence patterns.

Both aim to reproduce correlation properties relevant to the statistic.

8.3 Time-series versus clustered data considerations

Time dependence is one form of non-i.i.d. structure. Another is clustering, where observations within a group are correlated. For clusters, analysts often resample at the cluster level (or use methods designed for multilevel dependence), rather than using time-series blocks that assume ordering and contiguous structure. The resampling unit should align with the dependence mechanism.

9 Common Pitfalls and Limitations

9.1 When resampling does not mimic the data-generating process

The bootstrap can fail when the empirical distribution and resampling mechanism do not replicate critical features of the data-generating process. Examples include:

strong dependence ignored by i.i.d. resampling,
model structure (link functions, constraints) not mirrored by the bootstrap calculation,
survey sampling weights or complex sampling designs ignored when resampling.

9.2 Overconfidence from undercoverage or poor interval choice

If intervals systematically miss the true value (undercoverage), conclusions may become overconfident. Overconfidence often arises from:

using a poor CI construction for a skewed or biased estimator,
too few bootstrap replicates causing unstable quantiles,
ignoring dependence or boundary behavior that changes uncertainty patterns.

9.3 Issues with boundary parameters and nonregular problems

When parameters lie on boundaries or when regularity conditions fail, standard bootstrap logic can produce misleading intervals and bias estimates. In such cases, specialized bootstrap variants or alternative inference strategies may be needed.

9.4 Ties, censoring, and missing data (general caution)

Certain data features complicate resampling:

ties in discrete outcomes can yield coarse distributions and quantile instability,
censoring requires resampling that respects the censoring mechanism,
missing data needs an appropriate strategy (e.g., resampling observed patterns or using models that account for missingness).

General caution is warranted whenever the observation process alters what is recorded.

10 Practical Example Workflows

10.1 Computing a bootstrap SE and CI for a statistic

A typical workflow:

Choose the statistic \(\hat{\theta}\).
Compute \(\hat{\theta}\) on the original data.
Run \(B\) bootstrap resamples, producing \(\hat{\theta}^{*(1)},\dots,\hat{\theta}^{*(B)}\).
Compute SE as the standard deviation of the replicates.
Construct a CI using an appropriate interval method (percentile, basic, studentized, or BCa).

10.1.1 Interpreting interval results

An interval summarizes uncertainty under the bootstrap procedure. If the interval is narrow, the statistic appears stable; if it is wide, variability is substantial. Overlapping intervals between groups do not automatically imply “no difference,” especially when intervals are built separately; careful comparison should use distributions of differences when possible.

10.2 Comparing two statistics or groups

To compare groups, analysts often target a difference \(\Delta = \theta_1 - \theta_2\) or a ratio. One approach is:

compute \(\hat{\Delta}\) from the original data,
bootstrap within each group (or according to the correct resampling unit),
compute \(\hat{\Delta}^*\) for each replicate,
derive SE and CI for \(\Delta\) from the distribution of \(\hat{\Delta}^*\).

10.2.1 Resampling-based difference estimation

Resampling-based estimation naturally captures uncertainty in both group statistics. If the resampling respects independence between groups (or models shared dependence when present), the resulting CI for the difference supports more direct interpretation than comparing separate marginal CIs.

10.3 Reporting bootstrap results in a research paper

Common reporting elements include:

the bootstrap type (nonparametric, parametric, studentized, BCa, etc.),
the number of resamples \(B\),
the statistic definition and any preprocessing steps,
assumptions about dependence handling,
the resulting SE and confidence interval endpoints,
any sensitivity checks (e.g., different \(B\), robustness to outliers).

11 Implementation Notes

11.1 Pseudocode for a generic bootstrap

Generic pseudocode:

Input data \(X\), statistic function \(s(\cdot)\), resamples \(B\).
Compute \(\hat{\theta} = s(X)\).
For \(b=1\) to \(B\):

draw indices \(I_1,\dots,I_n\) uniformly from \(\{1,\dots,n\}\),
form \(X^* = (X_{I_1},\dots,X_{I_n})\),
set \(\hat{\theta}^{*(b)} = s(X^*)\).

Compute SE and construct CI from \(\{\hat{\theta}^{*(b)}\}\).

11.2 Software approach (language-agnostic guidance)

Most statistical environments provide bootstrap utilities, but regardless of tooling, the core implementation should:

ensure the statistic is recomputed exactly as on the original data,
store enough bootstrap results to compute quantiles and summary metrics,
handle random seeds for reproducibility,
allow flexible interval methods (percentile, BCa, studentized) if needed.

11.3 Reproducibility: seeds, storage, and documentation

Reproducibility depends on:

recording the random seed (or seed schedule),
documenting the bootstrap variant and interval method,
storing intermediate configuration (e.g., block length for block bootstraps),
recording \(B\) and any data preprocessing steps used before bootstrapping.

11.4 Scaling up: parallelization and batching (overview)

For large datasets or expensive statistics, analysts often parallelize across resamples. Another strategy is batching: computing replicates in chunks to manage memory and allow checkpointing. Care should be taken that parallel execution remains consistent with the intended random-number generation strategy.

12 Summary and Further Reading

12.1 Quick checklist of when and how to bootstrap

Identify the statistic \(\hat{\theta}\) and ensure the bootstrap recomputes it identically.
Use nonparametric bootstrap for i.i.d. data when distributional assumptions are unclear.
Choose an interval method aligned with skewness/bias concerns (percentile vs. studentized vs. BCa).
For dependent data, use an appropriate dependent-data bootstrap (e.g., block/bootstrap-at-cluster level).
Check stability by varying \(B\) and inspecting bootstrap replicate behavior.

12.2 Recommended references and learning paths

Foundational knowledge is typically built by studying:

resampling-based inference texts covering bootstrap theory and practice,
regression-oriented bootstrap literature for residual and wild bootstrap,
dependent-data resampling references for block methods,
applied examples demonstrating interval selection and diagnostic checks.

A practical learning path is to start with nonparametric bootstrap for simple statistics, then move to studentized/BCa intervals, and finally address regression and dependence structures.