[Statistics] Bootstrapping

This post covers bootstrapping.

1. Definition

Bootstrapping is a statistical method used to estimate the sampling distribution of a statistic by generating multiple samples from a given dataset.

In simpler terms, it is a resampling technique that involves randomly drawing samples from a dataset with replacement.

2. Principle

The idea behind bootstrapping is that the sample data is a representation of the population data, and by resampling the sample data, we can generate multiple samples that can provide an estimate of the population parameters.

The process involves creating many simulated samples by randomly selecting observations from the original dataset.

Each simulated sample usually has the same size as the original sample, but in some cases, selecting smaller resamples can be more appropriate and useful.

The observations are drawn with replacement, which means that an observation can be selected more than once in a simulated sample.

3. Example

Suppose we have a dataset of 5 observations, {e1, e2, e3, e4, e5}, and we want to estimate the mean of the population (sample size of 5 is likely to be insufficient for performing bootstrapping, but for the sake of the illustration, I'm holding the sample size very small now).

We can use bootstrapping to generate 1000 simulated samples, each with 5 observations, and calculate the mean of each sample.

Some of the possible resamples are: {e1, e1, e2, e2, e5}, {e1, e2, e3, e3, e4}, {e1, e2, e3, e4, e5}, {e1, e1, e1, e1, e1}

Notice how the same element can be drawn multiple times within a sample and some elements are not even included in the resamples.

We can then use the distribution of these means to estimate the population mean and calculate the confidence interval.

4. Application

Bootstrapping is a powerful tool for making inferences about population parameters when 1) the sample size is smal, 2) the data is non-parametric, or 3) the distribution is unknown.

For example, if we have a sample of data that is not normally distributed, we can use bootstrapping to estimate the population parameters, such as the mean or the standard deviation.

The method can also be used for hypothesis testing, confidence intervals, and model validation.

For example, suppose we want to test if the mean of a sample is significantly different from a population mean.

We can use bootstrapping to generate a null distribution of means under the assumption that the sample mean is equal to the population mean.

If the observed sample mean falls outside the 95% confidence interval of the null distribution, we can reject the null hypothesis and conclude that the sample mean is significantly different from the population mean.

5. Advantages

One of the advantages of bootstrapping is that it does not require any assumptions about the underlying distribution of the data.

This makes it a useful tool for analyzing complex data that may not follow a normal distribution.

Additionally, bootstrapping can be used to quantify the uncertainty in the estimates, by providing confidence intervals and standard errors.

6. Limitations

However, bootstrapping does have some limitations.

The accuracy of the estimates depends on the size of the original sample, and bootstrapping may not work well for small samples.

Additionally, bootstrapping can be computationally intensive, especially for large datasets, and it may not always be practical to generate a large number of simulated samples.

7. Conclusion

In conclusion, bootstrapping is a powerful statistical method that allows us to estimate the sampling distribution of a statistic by generating multiple samples from a given dataset.

It is particularly useful when the data is non-parametric or the distribution is unknown.

Bootstrapping can be used for hypothesis testing, confidence intervals, and model validation.

It does not require any assumptions about the underlying distribution of the data and provides a way to quantify the uncertainty in the estimates.

However, bootstrapping may not work well for small samples and can be computationally intensive for large datasets.

* Etymology

(Photo by Simona Sergi on Unsplash)

The term "bootstrapping" originated from the phrase "pulling oneself up by one's bootstraps", which refers to achieving a seemingly impossible task by using one's own resources and ingenuity. The phrase was first used in a literary context in the 19th century, but it became a popular metaphor in the early 20th century.

In the context of statistics, bootstrapping was first introduced by Bradley Efron in the late 1970s. Efron, a statistician at Stanford University, developed the bootstrap method as a way to estimate the sampling distribution of a statistic without making any assumptions about the underlying population distribution. He named the method "bootstrapping" to reflect the idea of using a small sample to "pull oneself up" and estimate the properties of the larger population.

The use of the term "bootstrapping" in statistics has since become widespread and is now a common technique in statistical analysis. It has been used in various fields, including finance, biology, psychology, and engineering, to estimate parameters, validate models, and test hypotheses. The term "bootstrapping" has also been used metaphorically in other fields, such as computer science and linguistics, to refer to the process of starting with a small amount of data and gradually building up more knowledge or resources through self-improvement or self-learning.