Home page for accesible maths 11 Diagnostics 11.2 Residuals vs. Fitted values

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

11.1 Normality of residuals

One of the key underlying assumptions of the linear regression model is that the errors $\epsilon_{i}$ have a Normal distribution. In reality, we do not know the errors and these are replaced with their estimates, $\hat{\epsilon}_{i}$ . We compare these estimated residuals to their model distribution using graphical diagnostics:

PP (probability-probability) and QQ plots can be used to check whether or not a sample of data can be considered to be a sample from a statistical model (usually a probability distribution). In the case of the normal linear regression model, they can be used to check whether or not the estimated residuals are a sample from a $\operatorname{Normal}(0,\sigma^{2})$ distribution. PP plots show the same information as QQ plots, but on a different scale.

{mdframed}

The PP plot is most useful for checking that values around the average (the body) fit the proposed distribution. It compares the percentiles of the sample of data, predicted under the proposed model, to the percentiles obtained for a sample of the same size, predicted from the empirical distribution

{mdframed}

The QQ plot is most useful for checking whether the largest and smallest values (the tails) fit the proposed distribution. It compares the ordered sample of data to the quantiles obtained for a sample of the same size from the proposed model.

First define the standardised residuals to be

\displaystyle\hat{r}_{i}=\frac{y_{i}-\hat{\mu}_{i}}{\hat{\sigma}}.

From Math230, standardising by $\hat{\sigma}$ means that these should be a sample from a $\operatorname{Normal}(0,1)$ distribution.

Denote by $\hat{r}^{(i)}$ the ordered standardised residuals, so that $\hat{r}^{(1)}$ is the smallest residual, and $\hat{r}^{(n)}$ the largest. We compare the standardised residuals to the standard normal distribution, using

1

A PP plot,

$\displaystyle\left\{\Phi(\hat{r}^{(i)}),\frac{i}{n+1}\right\}$

for $i=1,\ldots,n$ . Here $\Phi$ is the standard normal cumulative distribution function.
2

A QQ plot,

$\displaystyle\left\{\hat{r}^{(i)},\Phi^{-1}\left(\frac{i}{n+1}\right)\right\}$

for $i=1,\ldots,n$ , where $\Phi^{-1}$ is the inverse of the standard normal cumulative distribution function.

If the standardised residuals follow a $\operatorname{Normal}(0,1)$ distribution perfectly, both plots lie on the line $y=x$ . Because of random variation, even if the model is a good fit, the points won’t lie exactly on this line.

Remark.

You have seen QQ plots before in Math104; in that setting, they were used to examine whether data could be considered to be a sample from a Normal distribution.

TheoremExample 11.1.1 Brain weights cont.

In example 10.1.1 we fitted the following linear regression model to try to explain variability in (log) brain weight ( $Y_{i}$ ) using (log) body weight ( $x_{i,1}$ ),

\displaystyle\mathbb{E}[\log Y_{i}]=2.15+0.759\log x_{i,1}

We use R to create PP and QQ plots for the standardised residuals. First we will refit the model in R to obtain the required residuals,

⬇

> L1 <- lm(log(sleep$BrainWt)~log(sleep$BodyWt))

Next we need the residual variance,

⬇

> sigmasq <- sum(L1$resid^2)/56

and we can use this to get the standardised residuals:

⬇

> stdresid <- L1$residuals/sqrt(sigmasq)

R does not have an inbuilt function for creating a PP plot, but we can create one using the function qqplot,

⬇

> qqplot(c(1:58)/59,pnorm(stdresid),

xlab="Theoretical probabilities",ylab="Sample probabilities")

> abline(a=0,b=1)

Since we are comparing the standardised residuals to the standard Normal distribution, we can use the function qqnorm for the QQ plot,

⬇

> qqnorm(stdresid)

> abline(a=0,b=1)

The resulting plots are shown in Figure 11.1 and Figure 11.2. Both plots suggests that the standardised residuals do follow the standard Normal distribution closely.

Fig. 11.1: PP plot for standardised residuals from model for log Brain weight ( $g$ ). Straight lines show exact agreement between the residuals and a Normal(0,1) distribution.

Fig. 11.2: QQ plot for standardised residuals from model for log Brain weight ( $g$ ). Straight lines show exact agreement between the residuals and a Normal(0,1) distribution.

Remark.

In general, a QQ plot is more useful that a PP plot, as it tells us about the more ‘unusual’ values (i.e. the very high and very low residuals). It is the behaviour of these values which is most likely to highlight a lack of model fit.

Remark.

If the PP and QQ plots suggest that the residuals differ from the $\operatorname{Normal}(0,1)$ distribution in a systematic way, for example the points curve up (or down) and away from the 45 ${}^{\circ}$ line at either (or both) of the tails, it may be more appropriate to

1

Transform your response, e.g. use the log or square root functions, before fitting the model; or
2

Use a different residual distribution. This is discussed in Math333 Statistical Models.

Remark.

A lack of normality might also be due to the residuals having non-constant variance, referred to as heteroscedasticity. This can be assessed by plotting the residuals against the explanatory variables included in the model to see whether there is evidence of variability increasing, or decreasing, with the value of the explanatory variable.