Bootstrap

From a course by Davy Paindaveine and Nathalie Vialaneix

Camille Mondon
camille.mondon@tse-fr.eu

Toulouse School of Economics

Last updated on February 10, 2025

Table of contents

  • 1 Introduction
  • 2 A motivating example
    • 2.1 Optimal portfolio
    • 2.2 Sample case
    • 2.3 How to estimate the accuracy of λ^opt?
    • 2.4 Sampling from the population: infeasible
    • 2.5 Sampling from the sample: the bootstrap
    • 2.6 A comparison between both samplings
  • 3 The general procedure
    • 3.1 The bootstrap (Efron 1979)
  • 4 About the implementation in R
    • 4.1 A toy illustration
    • 4.2 Obtaining a bootstrap sample
    • 4.3 Generating B=1000 bootstrap means
    • 4.4 Bootstrap estimates
    • 4.5 The boot function
  • References

1 Introduction

Let X1,…,Xn∼Pθ i.i.d. Let θ^=θ^(X1,…,Xn) be an estimator for θ.

One often wants to evaluate the variance Var[θ^] to quantify the uncertainty of θ^.

The bootstrap is a powerful, broadly applicable method:

  • to estimate the variance Var[θ^]
  • to estimate the bias E[θ^]−θ
  • to construct confidence intervals for θ
  • more generally, to estimate the distribution of θ^.

The method is nonparametric and can deal with small n.

2 A motivating example

James et al. (2021)

2.1 Optimal portfolio

  • Let Y and Z be the values of two random assets and consider the portfolio: Wλ=λY+(1−λ)Z,λ∈[0,1] allocating a proportion λ of your wealth to Y and a proportion 1−λ to Z.
  • A common, risk-averse, strategy is to minimize the risk Var[Wλ].
  • It can be shown that this risk is minimized at λopt=Var[Z]−Cov[Y,Z]Var[Y]+Var[Z]−2Cov[Y,Z]
  • But in practice, Var[Y], Var[Z] and Cov[Y,Z] are unknown.

2.2 Sample case

Now, if historical data X1=(Y1,Z1),…,Xn=(Yn,Zn) are available, then we can estimate λopt by λ^opt=Var[Y]^−Cov[Y,Z]^Var[Y]^+Var[Z]^−2Cov[Y,Z]^ where

  • Var[Y]^ is the sample variance of the Yi’s
  • Var[Z]^ is the sample variance of the Zi’s
  • Cov[Y,Z]^ is the sample covariance of the Yi’s and Zi’s.

2.3 How to estimate the accuracy of λ^opt?

  • … i.e., its standard deviation Std[λ^opt]?
  • Using the available sample, we observe λ^opt only once.
  • We need further samples leading to further observations of λ^opt.
Figure 1: Portfolio data. For this sample, λ^opt=0.283 (James et al. 2021).

2.4 Sampling from the population: infeasible

We generated 1000 samples from the population. The first three are:

Figure 2: λ^opt(1)=0.283, λ^opt(2)=0.357, λ^opt(3)=0.299 (James et al. 2021).
  • This allows us to compute: λ¯opt=11000∑i=11000λ^opt(i)
  • Then: Std[λ^opt]^=1999∑i=11000(λ^opt(i)−λ¯opt)2.

Here: Std[λ^opt]^≈0.077,λ¯opt≈0.331  (≈λopt=13=0.333)

Figure 3: Histogram and boxplot of the empirical distribution of the λ^opt(i) (James et al. 2021).

(This could also be used to estimate quantiles of λ^opt.)

2.5 Sampling from the sample: the bootstrap

  • It is important to realize that this cannot be done in practice. One cannot sample from the population Pθ since it is unknown.
  • However, one may sample instead from the empirical distribution Pn (i.e., the uniform distribution over (X1,…,Xn)), that is close to Pθ for large n.
  • This means that we sample with replacement from (X1,…,Xn), providing a first bootstrap sample (X1∗1,…,Xn∗1) which allows us to evaluate λ^opt∗(1).
  • Further generating bootstrap samples (X1∗b,…,Xn∗b), b=2,…,B=1000, one can compute Std[λ^opt]^∗=1B−1∑b=1B(λ^opt∗(b)−λ¯opt∗)2 with λ¯opt∗=11000∑b=1Bλ^opt∗(b)

This provides Std[λ^opt]^∗≈0.079

Figure 4: Histogram and boxplot of the bootstrap distribution of λ^opt (James et al. 2021).

(This could again be used to estimate quantiles of λ^opt.)

2.6 A comparison between both samplings

Results are close: Std[λ^opt]^≈0.077 and Std[λ^opt]^∗≈0.079.

Figure 5: Bootstrap distributions from portfolio data (James et al. 2021).

3 The general procedure

3.1 The bootstrap (Efron 1979)

  • Let X1,…,Xn be i.i.d ∼Pθ.
  • Let T=T(X1,…,Xn) be a statistic of interest.
  • The bootstrap allows us to approximate the distribution of T.

(X1∗1,…,Xn∗1)⇝T∗1=T(X1∗1,…,Xn∗1)⋮(X1∗b,…,Xn∗b)⇝T∗b=T(X1∗b,…,Xn∗b)⋮(X1∗B,…,Xn∗B)⇝T∗B=T(X1∗B,…,Xn∗B)

Figure 6: The framework of bootstrap.

  • Each bootstrap sample (X1∗b,…,Xn∗b) is obtained by sampling (uniformly) with replacement among the original sample (X1,…,Xn).

  • Under mild conditions, the empirical distribution of (T∗1,…,T∗B) provides a good approximation of the sampling distribution of T under Pθ.

  • Possible uses:

    • 1B−1∑b=1B(T∗b−T¯∗)2, with T¯∗=1B∑b=1BT∗b, estimates Var[T]
    • The sample α-quantile qα∗ of (T∗1,…,T∗B) estimates T’s α-quantile

  • Possible uses when T is an estimator of θ:

    • (1B∑b=1BT∗b)−T estimates the bias E[T]−θ of T
    • [qα/2∗,q1−(α/2)∗] is an approximate (1−α)-confidence interval for θ.
    • …

4 About the implementation in R

4.1 A toy illustration

  • Let X1,…,Xn (n=4) be i.i.d t-distributed with 6 degrees of freedom.
  • Let X¯=1n∑i=1nXi be the sample mean.
  • How to estimate the variance of X¯ through the bootstrap?
n <- 4
(X <- rt(n,df=6))
[1] -0.08058779  0.28044078  1.19011050 -1.25212790
Xbar <- mean(X)
Xbar
[1] 0.0344589

4.2 Obtaining a bootstrap sample

X
[1] -0.08058779  0.28044078  1.19011050 -1.25212790
d <- sample(1:n,n,replace=TRUE)
d
[1] 2 4 4 4
Xstar <- X[d]
Xstar
[1]  0.2804408 -1.2521279 -1.2521279 -1.2521279

4.3 Generating B=1000 bootstrap means

B <- 1000
Bootmeans <- vector(length = B)
for (b in (1:B)) {
  d <- sample(1:n, n, replace = TRUE)
  Bootmeans[b] <- mean(X[d])
}
Bootmeans[1:4]
[1]  0.2370868 -0.3486833  0.3521335  0.2370868

4.4 Bootstrap estimates

Bootstrap estimates of E[X¯] and Var[X¯] are then given by

mean(Bootmeans)
[1] 0.03679914
var(Bootmeans)
[1] 0.1789107

The practical sessions will explore how well such estimates behave.

4.5 The boot function

A better strategy is to use the boot function from

library(boot)

The boot function takes typically 3 arguments:

  • data: the original sample
  • statistic: a user-defined function with the statistic to bootstrap
    • 1st argument: a generic sample
    • 2nd argument: a vector of indices pointing to a subsample on which the statistic is to be evaluated
  • R: the number B of bootstrap samples to consider

If the statistic is the mean, then a suitable user-defined function is

boot.mean <- function(x,d) {
  mean(x[d])
}

The bootstrap estimate of Var[X¯] is then

res.boot <- boot(X,boot.mean,R=1000)
var(res.boot$t)
          [,1]
[1,] 0.1844024

References

Efron, Bradley. 1979. “Bootstrap Methods: Another Look at the Jackknife.” The Annals of Statistics 7 (1). https://doi.org/10.1214/aos/1176344552.
James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2021. An Introduction to Statistical Learning: With Applications in R. Springer Texts in Statistics. New York, NY: Springer US. https://doi.org/10.1007/978-1-0716-1418-1.
1 / 23
Bootstrap From a course by Davy Paindaveine and Nathalie Vialaneix Camille Mondon camille.mondon@tse-fr.eu Toulouse School of Economics Last updated on February 10, 2025

  1. Slides

  2. Tools

  3. Close
  • Bootstrap
  • Table of contents
  • 1 Introduction
  • 2 A motivating example
  • 2.1 Optimal portfolio
  • 2.2 Sample case
  • 2.3 How to estimate the accuracy of λ^opt?
  • 2.4 Sampling from the population: infeasible
  • Here: \[ \widehat{\text{Std}[\hat{\lambda}_\text{opt}]}...
  • 2.5 Sampling from the sample: the bootstrap
  • This provides \[...
  • 2.6 A comparison between both samplings
  • 3 The general procedure
  • 3.1 The bootstrap (Efron 1979)
  • Each bootstrap sample...
  • 4 About the implementation in R
  • 4.1 A toy illustration
  • 4.2 Obtaining a bootstrap sample
  • 4.3 Generating B=1000 bootstrap means
  • 4.4 Bootstrap estimates
  • 4.5 The boot function
  • If the statistic...
  • References
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • r Scroll View Mode
  • ? Keyboard Help