Home page for accesible maths 7 Linear regression - fitting 7.1.1 Examples

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

7.1 Estimation of regression coefficients $\beta$

We shall use the method of least squares to estimate the vector of regression coefficients $\beta$ . For a linear regression model, this approach to parameter estimation gives the same parameter estimates as the method of maximum likelihood, which will be discussed in weeks 7–10 of this course.

The basic idea of least squares estimation idea is to find the estimate $\hat{\beta}$ which minimises the sum of squares function

\displaystyle S(\beta)=\sum_{i=1}^{n}(y_{i}-\beta_{1}x_{i,1}-\ldots-\beta_{p}x% _{i,p})^{2}.

(7.1)

We can rewrite the linear regression model in terms of the residuals as

\displaystyle\epsilon_{i}=Y_{i}-\beta_{1}x_{i,1}-\ldots-\beta_{p}x_{i,p}

By replacing $Y_{i}$ with $y_{i}$ , $S(\beta)$ can be interpreted as the sum of squares of the observed residuals. In general, the sum of squares function $S(\beta)$ is a function of $p$ unknown parameters, $\beta_{1},\ldots,\beta_{p}$ . To find the parameter values which minimise the function, we calculate all $p$ first-order derivatives, set these derivatives equal to zero and solve simultaneously.

Using definition (7.1), the $j$ -th first-order derivative is

\displaystyle\frac{\delta S}{\delta\beta_{j}}=-2\sum_{i=1}^{n}x_{i,j}(y_{i}-% \beta_{1}x_{i,1}-\ldots-\beta_{p}x_{i,p}).

(7.2)

We could solve the resulting system of $p$ equations by hand, using e.g. substitution. Since this is time consuming we instead rewrite our equations using matrix notation. The $j$ -th first-order derivative corresponds to the $j$ -th element of the vector

\displaystyle-2X^{\prime}(y-X\beta).

Thus to find $\hat{\beta}$ we must solve the equation,

\displaystyle-2X^{\prime}(y-X\hat{\beta})=0.

Multiplying out the brackets gives

\displaystyle-2X^{\prime}y+2X^{\prime}X\hat{\beta}=0

which can be rearranged to

\displaystyle X^{\prime}X\hat{\beta}=X^{\prime}y.

Multiplying both sides by $(X^{\prime}X)^{-1}$ gives the least squares estimates for $\hat{\beta}$ ,

{mdframed}

\displaystyle\hat{\beta}=(X^{\prime}X)^{-1}X^{\prime}y.

This is one of the most important results of the course!

Remark 1.

In order for the least squares estimate (7.3) to exist, $(X^{\prime}X)^{-1}$ must exist. In other words, the $p\times p$ matrix $X^{\prime}X$ must be non-singular,

1

$X^{\prime}X$ is non-singular iff it has linearly independent columns;
2

This occurs iff $X$ has linearly independent columns;
3

Consequently, explanatory variables must be linearly independent;
4
This relates back to the discussion on factors in Section 6.4. Linear dependence occurs if
- 1
  
  An intercept term and the indicator variables for all levels of a factor are include in the model; since the columns representing the indicator variables sum to the column of 1’s.
- 2
  
  The indicator variables for all levels of two or more factors are included in a model; since the columns representing the indicator variables sum to the column of 1’s for each factor.
Consequently it is safest to include an intercept term and indicator variables for $p-1$ levels of each $p$ -level factor.

Remark 2.

If you want to bypass completely the summation notation used above, the sum of squares function (7.1) can be written as

\displaystyle S(\beta)=(y-X\beta)^{\prime}(y-X\beta)=y^{\prime}y-\beta^{\prime% }X^{\prime}y-y^{\prime}X\beta+\beta^{\prime}X^{\prime}X\beta.

(7.3)

1

Now $\beta^{\prime}X^{\prime}y=(y^{\prime}X\beta)^{\prime}$ and since both $\beta^{\prime}X^{\prime}y$ and $y^{\prime}X\beta$ are scalars (can you verify this?) we have that $\beta^{\prime}X^{\prime}y=y^{\prime}X\beta$ .
2

Hence,

$\displaystyle S(\beta)=y^{\prime}y-2\beta^{\prime}X^{\prime}y+\beta^{\prime}X^% {\prime}X\beta.$
3

Differentiating with respect to $\beta$ gives the vector of first-order derivatives

$\displaystyle-2X^{\prime}y+2X^{\prime}X\beta=-2X^{\prime}(y-X\beta)$

as before.

Remark 3.

To prove that $\hat{\beta}$ minimises the sum of squares function we must check that the matrix of second derivatives is positive definite at $\hat{\beta}$ .

1

This is the multi-dimensional analogue to checking that the second derivative is positive at the minimum of a function in one unknown.
2

Returning once more to summation notation,

$\displaystyle\frac{\delta^{2}S}{\delta\beta_{k}\beta_{j}}=2\sum_{i=1}^{n}x_{i,% j}x_{i,k}.$
3

This is the $(j,k)$ -th element of the matrix $X^{\prime}X$ . Thus the second derivative of $S(\beta)$ is $X^{\prime}X$ .
4

To prove that $X^{\prime}X$ is positive definite, we must show that $z^{\prime}X^{\prime}Xz>0$ for all non-zero vectors $z$ .
5

Since $z^{\prime}X^{\prime}Xz$ can be written as the product of a vector and its transpose, $(Xz)^{\prime}Xz$ , the result follows immediately.

7.1.1 Examples

Style control - access keys in brackets

7.1 Estimation of regression coefficients β

Remark 1.

Remark 2.

Remark 3.

7.1 Estimation of regression coefficients $\beta$