Size & Power of Test (Edexcel A Level Further Maths: Further Statistics 1): Revision Note

Size

What is the size of a test?

• The size of a test is the probability of rejecting H₀ when it was in fact true

  • P(in critical region | H₀ is true) 

  • The situation being described is not a good outcome 

    • Something has been rejected when it was actually true!

  • A better test has a smaller size

    • You want to minimise this error happening

• Size is related to the significance level, α%

  • A better test has a smaller significance level (e.g. 1%)

  • For continuous distributions (e.g. normal)

    • Size = significance level, α

    • You can often write this down with no calculation

  • For discrete distributions (e.g. binomial, Poisson, geometric)

    • Size = actual significance level (≤α)

    • As close to α% as a discrete variable can get, whilst still being critical

How does size relate to Type I errors?

• The size is exactly the same as the probability of a Type I error

  • Both want to know the probability of rejecting H₀ when it was in fact true

Power

What is the power of a test?

• The power of a test is the probability of rejecting H₀ when it was false

  • P(in critical region | H₀ is false) 

  • The situation being described has a good outcome 

    • The null hypothesis was false and it rightly got rejected

  • A better test has a higher power

    • You want to maximise this happening

• In practice, you need to be given the actual population parameter to calculate the power

  • For example, H₀ assumed $p=\frac{1}{2}$ but actually $p=\frac{1}{3}$

    • This is more helpful than just saying $p\ne \frac{1}{2}$ 

  • Power is P(in the critical region | actual population parameter)

How does power relate to Type II errors?

• The power of a test is 1 - P(Type II error) 

  • Power is when H₀ is false and it gets rejected

    • That's a good outcome

  • A Type II error is when H₀ is false and it does not get rejected

    • That's a bad outcome

• You ideally want the power of a test to be greater than 0.5 

  • That way it's less likely to produce a Type II error

    • And more likely to reach the correct conclusion

Power functions

What is a power function?

• The power function is the power of a test written algebraically

  • In terms of $p$ or $\lambda$

  • For when you're not given the actual population parameter in the question

• In reality, it's very unlikely you'll know the actual population parameter anyway

  • Otherwise you wouldn't be doing a hypothesis test on it!

  • Power functions don't need this information

• The power function is P(in the critical region | population parameter is $p$)

  • Or, for Poisson 

    • P(in the critical region | population parameter is $\lambda$)

How do I find power functions?

• It's easier to show in an example

  • If the critical region is $X\le 2$ for a binomial hypothesis test with $n=50$ 

  • Then the power function is P(in the critical region | population parameter is $p$)

    • Let $X~\mathrm{B}\left(50,p\right)$

    • $\mathrm{P}\left(X\le 2|p\right)=\left(\begin{array}{c}50\\ 0\end{array}\right)p^0{\left(1-p\right)}^{50}+\left(\begin{array}{c}50\\ 1\end{array}\right)p^1{\left(1-p\right)}^{49}+\left(\begin{array}{c}50\\ 2\end{array}\right)p^2{\left(1-p\right)}^{48}$

  • Simplify

    • ${\left(1-p\right)}^{50}+50p{\left(1-p\right)}^{49}+1225p^2{\left(1-p\right)}^{48}$

  • Factorise and collect like terms

    • The power function is ${\left(1-p\right)}^{48}\left(1+48p+1176p^2\right)$

What can I do with power functions?

• You can plot them against $p$ (or $\lambda$) 

  • You can then see where the power is biggest

• You can input different values of $p$ (or $\lambda$) 

  • To compare two (or more) different hypothesis tests

    • The better test is the one with the higher power

  • To check if the power of a test is greater than 0.5

    • So that it's more likely to reach the correct conclusion

    • And less likely to produce a Type II error

    • Because Power = 1 - P(Type II error)