6 Linear predictor and model formula 6.1 Elements of linear algebra 6.3 Factors for categorical variables

6.2 Model formulae for continuous variables

The linear predictor is a linear combination of the explanatory variables that is defined by $\boldsymbol{\eta}=\beta_{1}\mathbf{x}_{1}+\beta_{2}\mathbf{x}_{2}+\dots+\beta_% {p}\mathbf{x}_{p}$ . A more concise notation for the linear predictor suppresses reference to the coefficients of the combination, and write

\boldsymbol{\eta}\in\mathrm{span}(\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{% x}_{p}).

The specification $\eta=\alpha+\beta x$ , or in vector form $\boldsymbol{\eta}=\alpha\mathbf{1}+\beta\mathbf{x}$ , can be written as

\boldsymbol{\eta}\in\mathrm{span}(\mathbf{1},\mathbf{x}).

Now, it may be that the relationship between expected response $\mathbb{E}[Y]$ and explanatory variable $x$ is more complicated than this. A reasonable procedure might be to see if enlarging the model to include a quadratic term improves the fit, i.e.

\boldsymbol{\eta}\in\mathrm{span}(\mathbf{1},\mathbf{x},\mathbf{x}^{2})

where $\mathbf{x}^{2}={\mathbf{x}.\mathbf{x}}$ .

The notation for these subspaces can be streamlined by writing $X=\mathrm{span}(\mathbf{1},x)$ and $X^{2}=\mathrm{span}(\mathbf{1},x^{2})$ . (Notation care: $X$ is often a random variable, or a design matrix, as well.) The quadratic model above can be written as follows by the sum of two subspaces:

\boldsymbol{\eta}\in X+X^{2}.

In general if $X_{j}=\mathrm{span}(\mathbf{1},\mathbf{x}_{j})$ , then the model $\boldsymbol{\eta}=\beta_{0}\mathbf{1}+\beta_{1}\mathbf{x}_{1}+\dots+\beta_{p}% \mathbf{x}_{p}$ is equivalent to $\boldsymbol{\eta}\in X_{1}+X_{2}+\dots+X_{p}.$ The reason for requiring $\mathbf{1}\in X_{j}$ for each $j=1,\ldots,p$ is concerned with indicator variables and will emerge later. This notation highlights the view of linear models as the specification of a subspace to which the linear predictor belongs.

Standard models

Model	$\mathcal{M}$ (model formula)
Simple linear regression	$X$
Quadratic regression	$X+X^{2}$
Polynomial regression	$X+X^{2}+\dots+X^{k}$
Regression through the origin	$\mathrm{span}(\mathbf{x})$
Multiple regression	$X_{1}+X_{2}$
Multiple regression	$X_{1}+X_{2}+\dots+X_{p}$ .

Definition 6.2.1.

The degrees of freedom of model $\mathcal{M}$ is $df(\mathcal{M})=n-\dim(\mathcal{M})$ where $\dim(\mathcal{M})$ is the minimum number of vectors required to $\mathrm{span}(\mathcal{M})$ and $n$ is the number of observations.

Exercise 6.47
Consider the example of predicting timber volume from measured tree height and trunk radius. Define a linear predictor based on the volume of a cylinder.

If a tree is tall then it must have a wide trunk to support its height. Knowledge of one provides insight about the other, so the variables are likely to be correlated. How does this influence how to define the linear predictor?

Lattice diagrams provide a convenient format to summarise which models have been fitted. The diagram below gives all submodels for a linear predictor based on three variables.

Unnumbered Figure: Link

With an increasing number of variables these lattices rapidly become complex.