5.1 Multiple $t$-tests

Carrying out multiple two sample $t$-tests seems the obvious way to compare means across a number of groups. However there are two reasons why this may not be such a sensible idea:

1. 1.

   There are a lot of tests;

2. 2.

   Individual test errors multiply across tests, see discussion below.

Large number of tests. How many tests are required to carry out all pairwise mean comparisons for $m$ groups?

Each test involves two groups, so we require the number of ways of selecting two items out of $m$. This is exactly the mathematical definition of a combination,

$$\left(\genfrac{}{}{0pt}{}{m}{2}\right)=\frac{m!}{2!(m-2)!}=\frac{m!}{2(m-2)!},$$

since $2!=2$.

Consider the case when you have just three groups. This would require three tests. If you had seven groups, this would become 21 tests. If you had ten groups then it would be 45 tests. However, it is not so difficult to write some code to automate these tests. The larger issue relates to the overall possibility of making an error in one, or more, of the tests.

How can we make an error when carrying out a hypothesis test? Suppose that we are testing

$H_0:\mu ={\mu }_0$

vs.

$H_1:\mu \ne {\mu }_0$

at the 5% level. We reject the null hypothesis if the absolute value of the test statistic

$t={\displaystyle \frac{\overline{X}-{\mu }_0}{S/\sqrt{n}}}$

lies above the 97.5% quantile of the $t_{n-1}$-distribution. Consequently there is a 5% probability of rejecting $H_0$ even when it is true.

There is a second type of error, which is less interesting for our purposes. This kind of error occurs if $H_0$ is accepted when $H_1$ is in fact true.

The probability of a Type II error depends on the true value of the population parameter and can therefore only be calculated for speculated values of this parameter. For example, we might say ‘If the difference between the true mean and ${\mu }_0$ was $d$ for some $d>0$, what would be the probability of a Type II error when testing $H_0:\mu ={\mu }_0$ against $H_1:\mu >{\mu }_0$’. The probability of a Type II error is linked to the power of the test.

There will be more investigation of Type I and Type II errors in the coursework and homework sheets. However, for now, we focus on an extension to the Type I error in the context of multiple testing. In particular, we also define the family-wise error rate:

When comparing the pairwise means of multiple groups, we would hope that the family-wise error rate would be equal to the probability of a Type I error for a single test. Unfortunately this is not the case. The probability that we incorrectly reject $H_0$ at least once across $k$ independent tests is the same as one minus the probability that we incorrectly reject $H_0$ in none of these tests. By definition, the probability that we incorrectly reject $H_0$ on any given test is $\alpha$, then

	$\mathrm{FWER}$	$=1-\mathrm{Pr}[\text{none of the }k\text{ tests are incorrectly rejected}]$
		$=1-\mathrm{Pr}{[\text{individual test is not incorrectly rejected}]}^k$
		$=1-{(1-\alpha )}^k.$

If two independent tests are preformed at the 5% significance level, what is the FWER? How does this change if 10 tests are performed?

For two tests, the FWER is 0.0975. For 10 tests it is 0.401. That is the probability of incorrectly rejected $H_0$ in at least one of the ten tests is 40%. This is considerably higher that the 5% probability for a single test.

Whilst much research has, and is being, carried out into the issues surrounding multiple testing, in particular in health research and genomics, we shall now concentrate on a different approach to the comparison of the means of three or more groups.