Home page for accesible maths 7 Linear transformations 7.1 Covariance and Correlation 7.3 Expectations of Linear Transformations

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

7.2 Random Vectors and Matrices

A random vector $\boldsymbol{X}=(X_{1},\ldots,X_{n})^{\prime}$ has $n$ elements which are random variables with some joint distribution. For instance if $n=3$ , $\boldsymbol{X}=(X_{1},X_{2},X_{3})^{\prime}$ with a joint distribution $F_{X_{1},X_{2},X_{3}}$ . Matrix notation provides an excellent method of handling some parts of this information succinctly.

Definition.

When the elements of the vector $\boldsymbol{X}$ are the random variables $X_{i}$ for $i=1,2,\dots,n$ , the expectation, $\operatorname{\mathsf{E}}\left[{\boldsymbol{X}}\right]$ , is the $n\times 1$ vector with elements $\operatorname{\mathsf{E}}\left[{X_{i}}\right]$ and the variance, ${\operatorname{\mathsf{Var}}}\left[{\boldsymbol{X}}\right]$ , is the $n\times n$ matrix with elements ${\operatorname{\mathsf{Cov}}}\left[{X_{i},X_{j}}\right]$ .

Example 7.2.1.

The expectations of $X_{1}$ , $X_{2}$ , $X_{3}$ , are $0$ , $-1$ , $2$ , respectively. Write down $\operatorname{\mathsf{E}}\left[{\boldsymbol{X}}\right]$ .

Solution.

\displaystyle\operatorname{\mathsf{E}}\left[{\boldsymbol{X}}\right]={\color[% rgb]{0.76,0.01,0}\begin{bmatrix}{\color[rgb]{0.76,0.01,0}0}\\ {\color[rgb]{0.76,0.01,0}-1}\\ {\color[rgb]{0.76,0.01,0}2}\end{bmatrix}.}

Example 7.2.2.

The variance matrix of $\boldsymbol{X}$ is

\displaystyle{\operatorname{\mathsf{Var}}}\left[{\boldsymbol{X}}\right]=\begin% {bmatrix}{1}&{0}&{1}\\ {0}&{3}&{2}\\ {1}&{2}&{5}\end{bmatrix}.

(a)

What is the dimension of the vector $\boldsymbol{X}$ ?
(b)

What is the variance of $X_{3}$ ?
(c)

What is the covariance of $X_{2}$ and $X_{3}$ ?
(d)

What is the correlation between $X_{2}$ and $X_{3}$ ?
(e)

Which variables are uncorrelated?
(f)

Which variables are independent?

Solution.

(a)

$\dim(\boldsymbol{X})={\color[rgb]{0.76,0.01,0}3}$ as ${\operatorname{\mathsf{Var}}}\left[{\boldsymbol{X}}\right]$ is a ${\color[rgb]{0.76,0.01,0}3\times 3}$ matrix.
(b)

${\operatorname{\mathsf{Var}}}\left[{X_{3}}\right]={\color[rgb]{0.76,0.01,0}5}$ .
(c)

${\operatorname{\mathsf{Cov}}}\left[{X_{2},X_{3}}\right]={\color[rgb]{% 0.76,0.01,0}2}$ .
(d)

${\operatorname{\mathsf{Corr}}}\left[{X_{2},X_{3}}\right]={\color[rgb]{% 0.76,0.01,0}2/\sqrt{3\times 5}}$ .
(e)

Uncorrelated: ${\color[rgb]{0.76,0.01,0}X_{1},X_{2}.}$
(f)

We cannot answer this question from the information provided.

In summary the variance matrix is defined by

\displaystyle{\operatorname{\mathsf{Var}}}\left[{\boldsymbol{X}}\right]=\begin% {pmatrix}{\operatorname{\mathsf{Var}}}\left[{X_{1}}\right]&{\operatorname{% \mathsf{Cov}}}\left[{X_{1},X_{2}}\right]&\ldots&{\operatorname{\mathsf{Cov}}}% \left[{X_{1},X_{n}}\right]\\ {\operatorname{\mathsf{Cov}}}\left[{X_{2},X_{1}}\right]&{\operatorname{\mathsf% {Var}}}\left[{X_{2}}\right]&\ldots&{\operatorname{\mathsf{Cov}}}\left[{X_{2},X% _{n}}\right]\\ \vdots&\vdots&&\vdots\\ {\operatorname{\mathsf{Cov}}}\left[{X_{n},X_{1}}\right]&{\operatorname{\mathsf% {Cov}}}\left[{X_{2},X_{n}}\right]&\ldots&{\operatorname{\mathsf{Var}}}\left[{X% _{n}}\right]\end{pmatrix}.

Example 7.2.3.

Random variables $X$ and $Y$ have variances $\sigma_{X}^{2}$ and $\sigma_{Y}^{2}$ respectively and correlation $\rho_{XY}$ . Write down the variance matrix of the vector $(X,Y)^{\prime}$ and simplify if $\sigma_{X}=\sigma=\sigma_{Y}$ .

Solution.

	$\displaystyle{\operatorname{\mathsf{Var}}}\left[{\begin{bmatrix}X\\ Y\end{bmatrix}}\right]$	$\displaystyle={\color[rgb]{0.76,0.01,0}\begin{bmatrix}{\color[rgb]{0.76,0.01,0% }\sigma_{X}^{2}}&{\color[rgb]{0.76,0.01,0}\rho_{XY}\sigma_{X}\sigma_{Y}}\\ {\color[rgb]{0.76,0.01,0}\rho_{XY}\sigma_{X}\sigma_{Y}}&{\color[rgb]{% 0.76,0.01,0}\sigma_{Y}^{2}}\end{bmatrix}}$
		$\displaystyle={\color[rgb]{0.76,0.01,0}\sigma^{2}\begin{bmatrix}{\color[rgb]{% 0.76,0.01,0}1}&{\color[rgb]{0.76,0.01,0}\rho_{XY}}\\ {\color[rgb]{0.76,0.01,0}\rho_{XY}}&{\color[rgb]{0.76,0.01,0}1}\end{bmatrix}.}$

Notes

Variance matrices are sometimes called variance-covariance matrices.

1.

The expectation vector is simply the vector of expectations; the variance matrix has variances down the diagonal, covariances as off-diagonals.
2.

The variance matrix is always symmetric, because ${\operatorname{\mathsf{Cov}}}\left[{X_{i},X_{j}}\right]={\operatorname{\mathsf% {Cov}}}\left[{X_{j},X_{i}}\right]$ , and positive semi-definite (see Section 7.4 Example 7.4.1 for the reason).

Independent identically distributed (iid) random variables

We will often be interested in random variables $X_{1},X_{2},\ldots,X_{n}$ that are independent identically distributed, or iid for short. This condition requires that

(a)

each $X_{i}$ has the same distribution and
(b)

they are independent.

This course does not cover independence of multiple random variables in depth, but the definition is similar to the bivariate case:

\displaystyle F_{X_{1},X_{2},\ldots,X_{n}}(x_{1},x_{2},\ldots,x_{n})=F_{X_{1}}% (x_{1})F_{X_{2}}(x_{2})\ldots F_{X_{n}}(x_{n}).

This implies pairwise independence: every pair of random variables $X_{i}$ , $X_{j}$ for $i\neq j$ is independent. However independence is a stronger condition than this i.e. there are examples of pairwise independent random variables that do not satisfy the full independence definition.

For iid random variables with ${\operatorname{\mathsf{Var}}}\left[{X_{i}}\right]=\sigma^{2}$ for all $i$ ,

\displaystyle{\operatorname{\mathsf{Var}}}\left[{\boldsymbol{X}}\right]=\begin% {pmatrix}\sigma^{2}&0&\ldots&0\\ 0&\sigma^{2}&\ldots&0\\ \vdots&\vdots&&\vdots\\ 0&0&\ldots&\sigma^{2}\end{pmatrix}=\sigma^{2}\boldsymbol{I}.

Random matrices: a random matrix $W$ is a matrix with elements $W_{i,j}$ , $i=1,\dots,m$ , $j=1,\dots,n$ , each of which is a random variable.

Definition.

Let $W$ be an $m\times n$ random matrix. The expectation, $\operatorname{\mathsf{E}}\left[{W}\right]$ is the $m\times n$ matrix with elements $\operatorname{\mathsf{E}}\left[{W}\right]_{i,j}=\operatorname{\mathsf{E}}\left% [{W_{i,j}}\right]$ , $i=1,\dots,m$ , $j=1,\dots,n$ .

We now derive formulae for the expectations and variances of linear transformations.