[Statistics] Central Limit Theorem

This post covers the central limit theorem.

1. Introduction

In the realm of statistics, the Central Limit Theorem (CLT) holds a position of paramount importance. This fundamental concept provides a powerful tool for understanding the behavior of sample means and lays the foundation for numerous statistical analyses. In this blog post, we will explore the Central Limit Theorem, its definition, origin and history, key characteristics, applications, and limitations.

2. Definition

The Central Limit Theorem (CLT) is a mathematical principle that states that the distribution of the sample means will tend to approximate a normal distribution as the sample size ($n$) increases (approaches to the infinity) regardless of the shape of the original population distribution. In simpler terms, it implies that if you take repeated random samples from any population, the distribution of the sample means will approach a normal distribution.

The CLT stipulates that the average of a set of sample means is equivalent to the average of the population. Additionally, it posits that the variance of the sample means is equivalent to the variance of the population, divided by the sample size ($n$). It is often considered that CLT is applied when the sample size ($n$) is equal to or greater than 30.

3. Origin and History

The Central Limit Theorem has its roots in probability theory and statistics. Its earliest foundations can be traced back to the works of 18th-century mathematicians such as Abraham de Moivre and Pierre-Simon Laplace. However, it was not until the early 20th century, with the contributions of luminaries like Carl Friedrich Gauss and Sir Francis Galton, that the theorem began to take shape.

4. Key Characteristics

1) Sample Size Matters: The Central Limit Theorem emphasizes that as the sample size increases, the distribution of sample means becomes increasingly close to a normal distribution. The larger the sample, the more reliable and accurate the approximation.

2) Independence and Random Sampling: The theorem assumes that the samples are selected independently and randomly from the population. This condition is crucial for the theorem to hold.

3) No Assumption of Population Distribution: One of the remarkable aspects of the CLT is its indifference to the underlying population distribution. Whether the population follows a normal, uniform, or skewed distribution, the theorem still holds, allowing us to make inferences about the population mean based on sample means.

5. Applications

The Central Limit Theorem has profound implications for statistical inference and hypothesis testing. It serves as the cornerstone for various statistical techniques, including:

1) Confidence Intervals: The CLT enables us to construct confidence intervals around sample means, providing a range of plausible values for the population mean.

2) Hypothesis Testing: By assuming that the distribution of sample means approximates a normal distribution, the CLT allows us to perform hypothesis tests and make inferences about population parameters.

3) Estimation of Population Parameters: The theorem's applicability extends to estimating population parameters, such as means and proportions, based on sample statistics.

6. Limitations

While the Central Limit Theorem is an invaluable tool, it does have certain limitations:

1) Sample size requirements: The CLT requires large sample sizes for its application. The larger the sample size, the closer the distribution of the sample means will be to a normal distribution. Therefore, for small sample sizes, the CLT may not provide accurate estimates of the population mean and other parameters.

2) Assumption of independence: The CLT assumes that the samples drawn from the population are independent of each other. In reality, it may be difficult to ensure complete independence between samples, which can lead to biased estimates of the population parameters.

3) Limited applicability to non-normal populations: The CLT assumes that the population distribution is normal, but in practice, many populations have non-normal distributions. In such cases, the CLT may not provide accurate estimates of the population parameters, and alternative statistical methods may be required.

4) Sampling bias: If the sampling process is biased, the CLT may not provide accurate estimates of the population parameters. For example, if a survey is conducted only among a certain group of people, such as those with internet access, the sample may not be representative of the overall population, which can lead to biased estimates.

5) Sensitivity to outliers: The CLT can be sensitive to outliers in the sample data. Outliers can significantly affect the sample mean, which can in turn affect the accuracy of the estimates of the population mean and other parameters.

6) Requirement for multiple independent samples: In some situations, it may be difficult or impossible to obtain multiple independent samples due to practical, ethical, or other constraints. For example, in some medical studies, it may not be feasible to draw multiple samples from patients due to the invasive nature of the procedures involved. In other cases, the population may not be well-defined, making it difficult to draw multiple samples. In such situations, alternative statistical methods such as bootstrapping or Bayesian inference may be used to estimate the population parameters.

7. Misunderstandings

1) The distribution of a sample: There is a common misconception that each sample will conform to a normal distribution, provided that the sample size is equal to or greater than 30. However, this is an oversimplification of the CLT. While it is true that the distribution of sample means will converge to a normal distribution as the sample size ($n$) approaches infinity, this does not imply that each individual sample will exhibit normality. In fact, the distribution of individual samples may deviate significantly from normality, even if the sample size is quite large. Rather, as the sample size becomes larger, it only apporaches the original distribution of the population.

Suppose we roll a dice. The more we roll it, the more it gets closer to the uniform distribution where the probability of getting 1, 2, 3, 4, 5, and 6 is all equally 1/6. This is definitely not a normal distribution: it is a uniform distribution.

2) The meaning of 'n': It can be confusing whether '$n$' means the sample size or the number of samples. Even though it is true that we need a sufficient number of samples, n refers to the sample size, and this is what we should care about. The following website helps us run simulations of the CLT.

http://www.ltcconline.net/greenl/java/Statistics/clt/cltsimulation.html

First, click the 'Skewed right' button, and you will see several options ranging from '$n$ = 2' to '$n$ = 100' ($n$ = sample size).

Clicking the '$n$ = 2' button, we see the sampling distribution is still rather skewed right, similar to the distribution of the population.

Notice that this result comes from calculating 20,000 samples, which we can imagine it would be almost the same with infinite numbers of samples.

As we also check what '$n$ = 9', '$n$ = 25', '$n$ = 36', '$n$ = 100' each provides, we realize that the greater the n becomes, the more the distributions converge to a normal distribution. For your information, the sample size in fact becomes smaller as the n increases, in the simulation above. To summarize, regardless of the number of samples we extract, securing a substantially large sample size still matters according to the CLT.

8. Conclusion

The Central Limit Theorem stands as a pillar of statistical theory, providing a bridge between sample statistics and population parameters. Its broad applications have shaped the field of statistics, enabling us to make accurate inferences and draw meaningful conclusions from sample data. While it has its limitations, understanding and utilizing the Central Limit Theorem is essential for any researcher or practitioner in the field of statistics, unlocking a world of insights from the world of samples.

Read more about:
Bootstrapping