10.2 Link to one-way ANOVA

Recall from Chapter 5 that a one-way ANOVA is a method for comparing the group means of three or more groups; an extension of the unpaired $t$-test.

It turns out that the one-way ANOVA is a special case of a simple linear model, in which the explanatory variable is a factor with three or more levels, where each level represents membership of one of the groups.

Suppose that the factor has $m$-levels, then the linear model for a one-way ANOVA can be written as

where $x_{i,j}$ is the indicator variable for the $j$-th level of the factor.

The purpose of an ANOVA is to test whether the mean response varies between different levels of the factor. This is equivalent to testing

vs.

In turn, this is equivalent to a model selection between

• 1

  $H_0$: Model 1, where $𝔼[Y_i]={\beta }_1$; and

• 2

  $H_1$: Model 2, where $𝔼[Y_i]={\beta }_1{x}_{i,1}+{\beta }_2{x}_{i,2}+\mathrm{\dots }+{\beta }_m{x}_{i,m}$.

Now, for model 1 states that all responses share a common population mean, our design matrix is simply a column of 1’s and ${\widehat{\beta }}_1=\overline{y}$, the overall sample mean. For model 2, the design matrix has $m$ columns, with

Therefore $X^{\prime }X$ is an $m\times m$ diagonal matrix, the diagonal entries of which correspond to the number of individuals in each of the groups,

$j=1,\mathrm{\dots },m$, and $X^{\prime }y$ is a vector of length $m$, with $j$-th entry being the sum of all the responses in group $j$. It follows that

i.e. the least squares estimate of the $j$-th regression coefficient is the observed mean of that group.

Calculating the sums of squares for the two models, we have

which, in ANOVA terminology, is what we referred to has the ‘total sum of squares’, and

which, in ANOVA terminology, is what we referred to as the within groups sum of squares.

Consequently, the $F$-ratio for model selection can be shown to be identical to the test statistic used for the one-way ANOVA: