Home page for accesible maths 11.1 Normality of residuals 11.3 Residuals vs. Explanatory variables

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

11.2 Residuals vs. Fitted values

A further implication of the assumptions made in defining the linear regression model is that the residuals $\epsilon_{i}$ are independent of the fitted values $\hat{\mu}_{i}$ . This can be proved as follows:

Recall the model assumption that

\displaystyle Y\sim\operatorname{MVN}_{n}(X\beta,\sigma^{2}I).

Then the fitted values and estimated residuals are defined as

\displaystyle{\color[rgb]{1,0,0}\hat{\mu}=X\hat{\beta}=X(X^{\prime}X)^{-1}X^{% \prime}Y=HY}

(11.1)

and

\displaystyle\hat{\epsilon}=Y-\hat{\mu}=Y-HY,

(11.2)

where we define

\displaystyle H=X(X^{\prime}X)^{-1}X^{\prime}.

Remark.

Since both $\hat{\mu}$ and $\hat{\epsilon}$ are both functions of the random variable $Y$ , they are themselves random variables. This means that they have sampling distributions. We focus on the joint behaviour of the two random variables.

To show that the fitted values and estimated residuals are independent, we show that the vectors $\hat{\mu}$ and $\hat{\epsilon}$ are orthogonal, i.e. that they have a product of zero.

By definition of $\hat{\mu}$ and $\hat{\epsilon}$ ,

	$\displaystyle\hat{\mu}^{\prime}\hat{\epsilon}$	$\displaystyle=(HY)^{\prime}(Y-HY)$
		$\displaystyle=Y^{\prime}H^{\prime}(Y-HY)$
		$\displaystyle=Y^{\prime}H^{\prime}Y-Y^{\prime}H^{\prime}HY$
		$\displaystyle=Y^{\prime}H^{\prime}Y-Y^{\prime}H^{\prime}Y,$

since $H^{\prime}H=H=H^{\prime}$ . So $\hat{\mu}^{\prime}\hat{\epsilon}=0$ .

This result uses the identities $H^{\prime}H=H$ and $H=H^{\prime}$ . Starting from the definition of $H$ can you prove these identities? Now you should be able to show, by combining these identities, that $H$ is idempotent, i.e. that $H=H^{2}$ .

Since $H$ is idempotent, when applied to its image, the image remains unchanged. In other words, $H$ maps $\hat{\mu}$ to itself. Can you show this? Mathematically, $H$ can also be thought of as a projection. The matrix $H$ is often referred to as the hat matrix, since it transforms the observations $y$ to the fitted values $\hat{\mu}$ .

A sensible diagnostic to check the model fit is to plot the residuals against the fitted values and check that these appear to be independent:

\displaystyle\left\{(\hat{\mu}_{i},\hat{\epsilon}_{i}):i=1,\ldots,n\right\}.

TheoremExample 11.2.1 Brain weights cont.

For the fitted brain weight regression model described in example (11.1.1), a plot of the residuals against the fitted values is shown in Figure 11.3. The code used in R to produce this plot is

⬇

> plot(L1$fitted.values,L1$residuals,xlab="Fitted",

ylab="Residuals")

> R <- lm(L1$residuals~L1$fitted.values)

> abline(a=R$coefficients[1],b=R$coefficients[2])

The horizontal line indicates the line of best fit through the scatter plot. The correlation between the fitted values and residuals is

⬇

> cor(L1$fitted.values,L1$residuals)

[1] -1.691625e-17

In this case, there is clearly no linear relationship between the residuals and fitted values and so, by this criterion, the model is a good fit.