6.1 Elements of linear algebra

An indicator vector is one whose coordinates can only take the values 0 and 1. The unit basis vectors, $𝐞$’s, are examples and any vector $𝐱$ can be written as a linear combination of unit basis vectors.

For the previous example, the vectors ${𝐭}_1={(0,1,0,0,1,1)}^{\prime }$ and ${𝐭}_2={(1,0,1,0,0,1)}^{\prime }$ are both indicator vectors in six-dimensional space. The zero vector $\mathrm{𝟎}={(0,0,\mathrm{\dots },0)}^{\prime }$ and the one vector $\mathrm{𝟏}={(1,1,\mathrm{\dots },1)}^{\prime }$ are also both indicator vectors.

In six-dimensional space there are $2^6=64$ indicator vectors. An interesting subset are unit indicator vectors

$${𝐞}_1={(1,0,0,0,0,0)}^{\prime },{𝐞}_2={(0,1,0,0,0,0)}^{\prime },\mathrm{\dots },{𝐞}_6={(0,0,0,0,1)}^{\prime }.$$

Note that $𝐱={(x_1,x_2,\mathrm{\dots },x_6)}^{\prime }=x_1{𝐞}_1+x_2{𝐞}_2+\mathrm{\dots }+x_6{𝐞}_6$.

Pointwise multiplication (dot product)

If $𝐱$ and $𝐲$ are two vectors in $n$-dimensional space then we define the pointwise multiplication by:

$$𝐱.𝐲={(x_1{y}_1,x_2{y}_2,\mathrm{\dots },x_n{y}_n)}^{\prime }$$

so that $𝐱.𝐲$ is a vector in $n$-dimensional space constructed by multiplying the coordinates of $𝐱$ and $𝐲$ together.

Some useful identities of pointwise multiplication are:

$𝐱.𝐲=𝐲.𝐱,\mathrm{𝐱𝟏}=𝐱,\mathrm{and}\mathit{\hspace{1em}}𝐱(𝐲+𝐳)=\mathrm{𝐱𝐲}+\mathrm{𝐱𝐳}.$

If $𝐚$ and $𝐛$ are indicator vectors then $𝐚.𝐛$ is also an indicator vector that contains $1$ in position $i$ if and only if both $a_i=1$ and $b_i=1$. This condition is the main motivation for introducing pointwise multiplication.

Linear combinations

If ${𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p$ are $p$ vectors in $n$-dimensional space and ${\alpha }_1,{\alpha }_2,\mathrm{\dots },{\alpha }_p$ are $p$ scalars (real numbers) then:

$${\alpha }_1{𝐱}_1+{\alpha }_2{𝐱}_2+\mathrm{\cdots }+{\alpha }_p{𝐱}_p$$

is a linear combination of ${𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p$. For instance

$$2{𝐱}_1+{𝐱}_2-7{𝐱}_3,{𝐱}_1+{𝐱}_3,{\alpha }_1{𝐱}_1+{\alpha }_3{𝐱}_3$$

are linear combinations of ${𝐱}_1,{𝐱}_2,{𝐱}_3$. The linear predictor is a linear combination of the explanatory vectors.

Span

The set of all linear combinations of ${𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p$ is known as the span of ${𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p$, and written as $\mathrm{span}({𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p)$. Any vector in this set can be written as ${\alpha }_1{𝐱}_1+\mathrm{\dots }+{\alpha }_p{𝐱}_p$ for some ${\alpha }_1,{\alpha }_2,\mathrm{\dots },{\alpha }_p$.

For example, in three-dimensional space, the first two unit indicator vectors are ${𝐞}_1={(1,0,0)}^{\prime }$ and ${𝐞}_2={(0,1,0)}^{\prime }$. Any vector in the space defined by $\mathrm{span}({𝐞}_1,{𝐞}_2)$ can be written as ${({\alpha }_1,{\alpha }_2,0)}^{\prime }$, but, this space does not contain ${(0,0,1)}^{\prime }$.

The zero vector $\mathrm{𝟎}\in \mathrm{span}({𝐞}_1,{𝐞}_2)$. The one vector $\mathrm{𝟏}\in \mathrm{span}({𝐞}_1,{𝐞}_2,\mathrm{\dots },{𝐞}_6)$.

Let $S\subseteq V$ denote a subset of vectors in $n$-dimensional space and suppose that ${𝐬}_1$ and ${𝐬}_2$ are in $S$.

Then $S$ is a subspace if

• (i)

  ${𝐬}_1+{𝐬}_2$ is in $S$ and

• (ii)

  $\alpha {𝐬}_1$ is in $S$.

 
Exercise 6.43
Verify that $S=\mathrm{span}({𝐞}_1,{𝐞}_3)$ is a subspace of ${ℝ}^3$.

Geometrically, this defines a plane that passes through the origin at right angles to the second axis.

 

In a similar way, we can show that $\mathrm{span}({𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p)$ is a subspace of ${ℝ}^n$.

Dimension

The dimension of a subspace $S$, denoted by $dim(S)$, is the minimum number of vectors required to span the subspace. In three-dimensional space,

• •

  $dim\mathrm{span}({𝐞}_1,{𝐞}_3)=2$,

• •

  $dim\mathrm{span}({𝐞}_1,{𝐞}_2,{𝐞}_3)=3$

• •

  $dim\mathrm{span}(\mathrm{𝟏},{𝐞}_1,{𝐞}_2,{𝐞}_3)=3$.

The sum of two subspaces

If $S_1$ and $S_2$ are two subspaces then the sum $S=S_1+S_2$ is the set $S$ of vectors that can be written as $𝐬={𝐬}_1+{𝐬}_2$ where ${𝐬}_1\in S_1$ and ${𝐬}_2\in S_2$.

We can show that the sum of two subspaces is also a subspace itself, for instance:

• (i)

  If $S_1=\mathrm{span}({𝐞}_1,{𝐞}_2,{𝐞}_3)$ and $S_2=\mathrm{span}({𝐞}_2,{𝐞}_3,{𝐞}_4)$ are subspaces in six-dimensional space composed of the unit indicator vectors then $S_1+S_2=\mathrm{span}({𝐞}_1,{𝐞}_2,{𝐞}_3,{𝐞}_4)$.

  Note that $dim(S_1+S_2)\le dim{S}_1+dim{S}_2$.

• (ii)

  In general if $S_1=\mathrm{span}({𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p)$ and $S_2=\mathrm{span}({𝐳}_1,{𝐳}_2,\mathrm{\dots },{𝐳}_q)$ then $S_1+S_2=\mathrm{span}({𝐱}_1,\mathrm{\dots },{𝐱}_p,{𝐳}_1,\mathrm{\dots },{𝐳}_q)$.

The product of two subspaces

If $S_1$ and $S_2$ are two subspaces then the product $S=S_1.S_2$ is the span of the set of vectors, $s$, that can be written as the pointwise product $s={𝐬}_1.{𝐬}_2$ where ${𝐬}_1\in S_1$ and ${𝐬}_2\in S_2$. In particular, if $S_1=\mathrm{span}({𝐱}_1,{𝐱}_2,\mathrm{\dots },{𝐱}_p)$ and $S_2=\mathrm{span}({𝐳}_1,{𝐳}_2,\mathrm{\dots },{𝐳}_q)$, the product $S_1.S_2$ is

$$S_1.S_2=\mathrm{span}({𝐱}_1{𝐳}_1,{𝐱}_1{𝐳}_2,\mathrm{\dots },{𝐱}_1{𝐳}_q,{𝐱}_2{𝐳}_1,\mathrm{\dots },{𝐱}_2{𝐳}_q,\mathrm{\dots },{𝐱}_p{𝐳}_q).$$

 
Exercise 6.44
In three-dimensional space find $S_1.S_2$ when $S_1=\mathrm{span}({𝐞}_1,{𝐞}_2)$ and $S_2=\mathrm{span}({𝐞}_2,{𝐞}_3)$.

 

 
Exercise 6.45
In four-dimensional space, let ${𝐚}_1={(1,1,0,0)}^{\prime },{𝐚}_2={(0,0,1,1)}^{\prime },{𝐛}_1={(1,0,1,0)}^{\prime }$ and ${𝐛}_2={(0,1,0,1)}^{\prime }$. Put $A=\mathrm{span}({𝐚}_1,{𝐚}_2)$, $B=\mathrm{span}({𝐛}_1,{𝐛}_2)$. Find $A.B$.

 

 
Exercise 6.46
Note that the set of vectors $S$, for which $s={𝐬}_1.{𝐬}_2$ with ${𝐬}_1\in S_1$ and ${𝐬}_2\in S_2$ does not in general constitute a subspace.

Use the previous exercise to find a counter example.