Unbiased Estimates (DP IB Applications & Interpretation (AI): HL): Revision Note

Unbiased estimates

What is an unbiased estimator of a population parameter?

• An estimator is a random variable that is used to estimate a population parameter

  • An estimate is the value produced by the estimator when a sample is used

• An estimator is called unbiased if its expected value is equal to the population parameter

  • An estimate from an unbiased estimator is called an unbiased estimate

  • This means that the mean of the unbiased estimates from individual samples will get closer to the population parameter as more samples are taken

• The sample mean is an unbiased estimate for the population mean

  • $\overline{x}=\frac{\sum x}{n}$

• The sample variance is not an unbiased estimate for the population variance

  • $s_n^2=\frac{\sum {(x-\overline{x})}^2}{n}=\frac{\sum x^2}{n}-{\left(\overline{x}\right)}^2$ 

  • On average the sample variance will underestimate the population variance

  • Although as the sample size increases the sample variance will tend to get closer to the unbiased estimate

What are the formulae for unbiased estimates of the mean and variance of a population?

• A sample of n data values (x₁, x₂, ... etc) can be used to find unbiased estimates for the mean and variance of the population

• An unbiased estimate for the mean μ of a population can be calculated using

  •  $\overline{x}=\frac{\sum x}{n}$

• An unbiased estimate for the variance σ² of a population can be calculated using

  • $s_{n-1}^2=\frac{n}{n-1}{s}_n^2$

  • This is given in the exam formula booklet

  • This can also be written as $s_{n-1}^2=\frac{\sum {(x-\overline{x})}^2}{n-1}$

    • Notice that dividing by $n$ gives a biased estimate but dividing by $n-1$ gives an unbiased estimate

Is s_n-1 an unbiased estimate for the standard deviation?

• Unfortunately s_n-1 is not an unbiased estimate for the standard deviation of the population

• It is better to work with the unbiased variance rather than standard deviation

• There is not a formula for an unbiased estimate for the standard deviation that works for all populations

  • Therefore you will not be asked to find one in your exam

How do I show the sample mean is an unbiased estimate for the population mean?

• You do not need to learn this proof

  • It is simply here to help with your understanding

• Suppose the population of X has mean μ and variance σ²  

• Take a sample of n observations

  • X_1, X_2, ..., X_n

  • E(X_i) = μ

• Using the formula for a linear combination of n independent variables:

$\begin{array}{rcl}\mathrm{E}\left(\overline{X}\right)& =& \mathrm{E}\left(\frac{X_1+X_2+ \dots +X_n}{n}\right)\\ & =& \frac{\mathrm{E}\left(X_1\right)+\mathrm{E}\left(X_2\right)+ \dots +\mathrm{E}\left(X_n\right)}{n}\\ & =& \frac{\mu +\mu + \dots +\mu }{n}\\ & =& \frac{n\mu }{n}\\ & =& \mu \end{array}$

• As $\begin{array}{rcl}\mathrm{E}\left(\overline{X}\right)& =& \mu \end{array}$ this shows the formula will produce an unbiased estimate for the population mean

Why is there a divisor of n-1 in the unbiased estimate for the variance?

• You do not need to learn this proof

  • It is simply here to help with your understanding

• Suppose the population of X has mean μ and variance σ²  

• Take a sample of n observations

  • X_1, X_2, ..., X_n

  • E(X_i) = μ

  • Var(X_i) = σ²

• Using the formula for a linear combination of n independent variables:

$\begin{array}{rcl}\mathrm{Var}\left(\overline{X}\right)& =& \mathrm{Var}\left(\frac{X_1+X_2+ \dots +X_n}{n}\right)\\ & =& \frac{\mathrm{Var}\left(X_1\right)+\mathrm{Var}\left(X_2\right)+ \dots +\mathrm{Var}\left(X_n\right)}{n^2}\\ & =& \frac{{\sigma }^2+{\sigma }^2+ \dots +{\sigma }^2 }{n^2}\\ & =& \frac{n{\sigma }^2}{n^2}\\ & =& \frac{{\sigma }^2}{n}\end{array}$

• It can be shown that $\mathrm{E}\left({\overline{X}}^2\right)={\mu }^2+\frac{{\sigma }^2}{n}$

  • This comes from rearranging $\mathrm{Var}\left(\overline{X}\right)=\mathrm{E}\left({\overline{X}}^2\right)-{\left[\mathrm{E}\left(\overline{X}\right)\right]}^2$

• It can be shown that $\mathrm{E}\left(X^2\right)=\mathrm{E}\left({X_i}^2\right)={\mu }^2+{\sigma }^2$

  • This comes from rearranging $\mathrm{Var}\left(X\right)=\mathrm{E}\left(X^2\right)-{\left[\mathrm{E}\left(X\right)\right]}^2$

• Using the formula for a linear combination of n independent variables:

$\begin{array}{rcl}\mathrm{E}\left(S_n^2\right)& =& \mathrm{E}\left(\frac{\sum X_i^2}{n}-{\stackrel{-}{X}}^2\right)\\ & =& \frac{\sum \mathrm{E}\left(X_i^2\right) }{n}-\mathrm{E}\left({\overline{X}}^2\right)\\ & =& \frac{\sum \left({\mu }^2+{\sigma }^2\right)}{n}-\left({\mu }^2+\frac{{\sigma }^2}{n}\right)\\ & =& \frac{n\left({\mu }^2+{\sigma }^2\right)}{n}-\left({\mu }^2+\frac{{\sigma }^2}{n}\right)\\ & =& {\mu }^2+{\sigma }^2-\left({\mu }^2+\frac{{\sigma }^2}{n}\right)\\ & =& {\sigma }^2-\frac{{\sigma }^2}{n}\\ & =& \frac{n{\sigma }^2-{\sigma }^2}{n}\\ & =& \frac{n-1}{n}\sigma ²\end{array}$

• As $\mathrm{E}\left(S_n^2\right)\ne {\mathrm{\sigma }}^2$ this shows that the sample variance is not unbiased

  • You need to multiply by $\frac{n}{n-1}$

  • Then $\mathrm{E}\left(S_{n-1}^2\right)=\mathrm{E}\left(\frac{n}{n-1}{S}_n^2\right)=\frac{n}{n-1}\mathrm{E}\left(S_n^2\right)={\mathrm{\sigma }}^2$