7 Central Limit Theorem

7.1 Theorem: Central Limit Theorem

Let \(X_1\), \(X_2\), \(\ldots\), \(X_n\) be independent and identically distributed random variables with \(E(X_i) = \mu\) and \(V(X_i) = \sigma^2\).

Define

\[U = \sqrt{n}\Big(\frac{\bar{X} - \mu}{\sigma}\Big)\]

where

\[ \bar{X} = \frac{\sum\limits_{i = 1}^{n} X_i}{n} \]

Then the distribution function of \(U\) converges to a standard normal distribution function as \(n \to \infty\).

Proof:

Recall the definition of the moment generating function 28.2.1 for a random variable \(X\) can be written

\[1 + tE(X) + \frac{t^2}{2!}E(X^2) + \frac{t^3}{3!} E(X^3) + \cdots \]

Let \(Z_i = \frac{X_i - \mu}{\sigma}\). We observe the following preliminary results:

Preliminary 1

\[\begin{aligned} E(Z_i) &= E\Big(\frac{X_i - \mu}{\sigma}\Big) \\ &= \frac{1}{\sigma} \cdot E(X_i - \mu) \\ &= \frac{1}{\sigma} \cdot \big(E(X_i) - \mu \big) \\ &= \frac{1}{\sigma} \cdot (\mu - \mu) \\ &= \frac{1}{\sigma} \cdot 0 \\ &= 0 \end{aligned}\]

Preliminary 2

\[\begin{aligned} V(Z_i) &= V\Big(\frac{X_i - \mu}{\sigma}\Big) \\ &= \frac{1}{\sigma^2} V(X_i - \mu) \\ &= \frac{1}{\sigma^2} \cdot \big(V(X_i) - V(\mu)\big)\\ &= \frac{1}{\sigma^2} \cdot \big(V(X_i) - 0\big) \\ &= \frac{1}{\sigma^2} \cdot V(X_i) \\ &= \frac{1}{\sigma^2} \cdot \sigma^2 \\ &= \frac{\sigma^2}{\sigma^2} \\ &= 1 \\ \Rightarrow E(Z^2) - E(Z)^2 &= 1 \\ \Rightarrow E(Z^2) &= 1 - E(Z)^2 \\ ^{[1]} &= 1 - 0 \\ &= 1 \end{aligned}\]

From Preliminary 1, \(E(Z) = 0\).

Preliminary 3

\[\begin{aligned} U &= \sqrt{n}\Big(\frac{\bar{X} - \mu}{\sigma}\Big) \\ &= \sqrt{n}\Big(\frac{\frac{\sum\limits_{i = 1}^{n} X_i} {n} - \mu} {\sigma}\Big) \\ &= \sqrt{n}\Big(\frac{\frac{\sum\limits_{i = 1}^n X_i}{n} - \frac{n \cdot \mu}{n}} {\sigma}\Big) \\ &= \sqrt{n}\Big(\frac{\frac{1}{n}\sum\limits_{i = 1}^n X_i - n \cdot \mu}{\sigma}\Big) \\ &= \frac{\sqrt{n}}{n} \Big(\frac{\sum\limits_{i = 1}^n X_i - n \cdot \mu}{\sigma}\Big) \\ &= \frac{1}{\sqrt{n}} \Big(\frac{\sum\limits_{i = 1}^n X_i - n \cdot \mu}{\sigma}\Big) \\ &= \frac{1}{\sqrt{n}} \Big(\frac{\sum\limits_{i = 1}^n X_i - n \cdot \mu} {\sigma}\Big) \\ &= \frac{1}{\sqrt{n}} \Big(\frac{\sum\limits_{i = 1}^n (X_i - \mu)}{\sigma}\Big) \\ &= \frac{1}{\sqrt{n}} \sum\limits_{i = 1}^n \frac{X_i - \mu}{\sigma} \\ &= \frac{1}{\sqrt{n}} \sum\limits_{i = 1}^n Z_i \end{aligned}\]

Note that, due to the independence of the \(X_i\) random variables, it follows that the \(Z_i\) random variables are also independent.

The moment generating function of \(U\) is derived as

\[\begin{aligned} M_U(t) &= M_Z\Big(t \sum\limits_{i = 1}^{n} \frac{Z_i}{\sqrt{n}}\Big) \\ &= M_Z\Big(\frac{t}{\sqrt{n}} \sum\limits_{i = 1}^n Z_i\Big) \\ &= E\Big(\exp\left\{\sum\limits_{i = 1}^n \frac{t}{\sqrt{n}} Z_i \right\} \Big) \\ &= E\Big(\prod\limits_{i = 1}^{n}\exp\left\{ \frac{t}{\sqrt{n}} Z_i \right\} \Big) \\ &= \prod\limits_{i = 1}^{n}E\Big(\exp\left\{ \frac{t}{\sqrt{n}} Z_i \right\} \Big) \\ &= \prod\limits_{i = 1}^{n}M_{Z_i}(t) \\ ^{[1]}&= \prod\limits_{i = 1}^{n}M_{Z}(t) \\ &= \big(M_{Z_i}(t)\big)^n \\ ^{[2]} &= \Big(1 + \frac{t}{\sqrt{n}} E(Z) + \frac{1}{2!} \cdot \big(\frac{t}{\sqrt{n}}\big)^2 E(Z^2) + \frac{1}{3!} \cdot \big(\frac{t}{\sqrt{n}}\big)^3 E(Z^3) + \cdots \Big)^n \\ ^{[3]} &= \Big(1 + \frac{t}{\sqrt{n}} \cdot 0 + \frac{t}{2! \cdot n} E(Z^2) + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big)^n \\ &= \Big(1 + 0 + \frac{t}{2! \cdot n} E(Z^2) + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big)^n \\ &= \Big(1 + \frac{t}{2! \cdot n} E(Z^2) + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big)^n \\ ^{[4]} &= \Big(1 + \frac{t}{2! \cdot n} \cdot 1 + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big)\\ &= \Big(1 + \frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big)^n \\ \Rightarrow \ln M_U(t) &= \ln\left[\Big(1 + \frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big)^n\right] \\ &= n \cdot \ln\Big(1 + \frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots \Big) \\ ^{[5]} &= n \cdot \ln (1 + k) \\ ^{[6]} &= n \cdot (k - \frac{k^2}{2} + \frac{k^3}{3} - \frac{k^4}{4}) \\ ^{[7]} &= n \cdot \Big( \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{1}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \Big) \\ &= n \cdot \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{n}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \\ &= \big(\frac{n\cdot t}{2! \cdot n} + \frac{n\cdot t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{n}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \\ &= \frac{n\cdot t}{2! \cdot n} + \big(\frac{n\cdot t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{n}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \\ &= \frac{t}{2!} + \big(\frac{n\cdot t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{n}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \\ &= \frac{t}{2} + \big(\frac{n\cdot t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{n}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \end{aligned}\]

  1. Each of the \(Z_i\) are identically distributed, so they each have the same moment generating function.
  2. Definition of the moment generating function @ref{moment-definition-mgf}
  3. From Preliminary 1, \(E(Z) = 0\).
  4. From Preliminary 2, \(E(Z^2) = 1\).
  5. Let \(k = \frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\)
  6. \(\ln(1 + x)\) may be rewritten with a series expansion: \(\ln(1 + x) = x - \frac{x^2}{2} + \frac{x^3}{3} - \frac{x^4}{4} + \cdots\)

To complete the proof, we take the limit as \(n \to \infty\).

\[ \lim\limits_{n \to \infty} \ln[M_Z(t)] = \lim\limits_{n \to \infty}\left[\frac{t}{2} + \big(\frac{n\cdot t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big) - \frac{n}{2} \big(\frac{t}{2! \cdot n} + \frac{t^3}{3! \cdot n^{\frac{3}{2}}} E(Z^3) + \cdots\big)^2 + \cdots \right] \] By noting that, with the exception of the \(\frac{t}{2}\) term, every term in the series involves \(n\) in the denominator. Additionally, in each of those terms, the \(n\) in the denominator is larger than the \(n\) in the numerator. Thus, as \(n \to \infty\), each of those terms approaches \(0\), yielding.

\[\begin{aligned} \lim\limits_{n \to \infty} \ln[M_Z(t)] &= \frac{t}{2} \\ \Rightarrow \lim\limits_{n \to \infty} \exp\left\{\ln[M_Z(t)]\right\} &= \exp\left\{\frac{t}{2}\right\}\\ \Rightarrow \lim\limits_{n \to \infty} M_Z(t) &= e^{\frac{t}{2}} \end{aligned}\]

This is the form of the moment generating function for a normal random variable with \(\mu = 0\) and \(\sigma^2 = 1\).

7.2 References

D Wackerly, W Mendenhall, R Scheaffer, Mathematical Statistics with Applications 6th ed., Duxbury Thomson Learning, 2002 pp 352 - 354.