Home page for accesible maths 7 Linear transformations 7 Linear transformations 7.2 Random Vectors and Matrices

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

7.1 Covariance and Correlation

Figure 7.1 (First Link, Second Link) and Figure 7.2 (First Link, Second Link) show samples from four different joint distributions. In all cases the variables have the same marginal distribution for both $X$ and $Y$ however the joint distributions have very different forms as they have different dependence structures. In this section we will try to characterise the dependence through a summary measure.

Figure 7.1: First Link, Second Link, Caption: Realisations from two different joint distributions of

X

and

Y

. The marginal distributions are the same in both cases.

Figure 7.2: First Link, Second Link, Caption: realisations from two more different joint distributions of

X

and

Y

. The marginal distributions are the same as in Figure Figure 7.1 (First Link, Second Link).

Throughout this section we use the notation

$\operatorname{\mathsf{E}}\left[{X}\right]=\mu_{X}$ ,
$\operatorname{\mathsf{E}}\left[{Y}\right]=\mu_{Y}$ ,
${\operatorname{\mathsf{StdDev}}}\left[{X}\right]=\sigma_{X}$ ,
${\operatorname{\mathsf{StdDev}}}\left[{Y}\right]=\sigma_{Y}$ .

The most common way of describing the relationship between two random variables is through the covariance or correlation. The covariance between $X$ and $Y$ is

	$\displaystyle{\operatorname{\mathsf{Cov}}}\left[{X,Y}\right]$	$\displaystyle=\operatorname{\mathsf{E}}\left[{(X-\mu_{X})(Y-\mu_{Y})}\right]$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{XY-\mu_{X}Y-X\mu_{Y}+\mu_{X}\mu_% {Y}}\right]$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{XY}\right]-\mu_{X}\operatorname{% \mathsf{E}}\left[{Y}\right]-\operatorname{\mathsf{E}}\left[{X}\right]\mu_{Y}+% \mu_{X}\mu_{Y}$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{XY}\right]-\operatorname{\mathsf% {E}}\left[{X}\right]\operatorname{\mathsf{E}}\left[{Y}\right]$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{XY}\right]-\mu_{X}\mu_{Y}.$

So, just as ${\operatorname{\mathsf{Var}}}\left[{X}\right]=\operatorname{\mathsf{E}}\left[{% (X-\mu_{X})^{2}}\right]=\operatorname{\mathsf{E}}\left[{X^{2}}\right]-% \operatorname{\mathsf{E}}\left[{X}\right]^{2}$ , there are two equivalent forms for ${\operatorname{\mathsf{Cov}}}\left[{X,Y}\right]$ . Indeed, ${\operatorname{\mathsf{Cov}}}\left[{X,X}\right]={\operatorname{\mathsf{Var}}}% \left[{X}\right]$ .

By symmetry, ${\operatorname{\mathsf{Cov}}}\left[{X,Y}\right]={\operatorname{\mathsf{Cov}}}% \left[{Y,X}\right]$ .

Moreover, we have the following, bilinearity properties:

1.

${\operatorname{\mathsf{Cov}}}\left[{aX,Y}\right]=a{\operatorname{\mathsf{Cov}}% }\left[{X,Y}\right]$ since both are $\operatorname{\mathsf{E}}\left[{aXY}\right]-\operatorname{\mathsf{E}}\left[{aX% }\right]\operatorname{\mathsf{E}}\left[{Y}\right]$ . By the symmetry property, ${\operatorname{\mathsf{Cov}}}\left[{X,cY}\right]=c{\operatorname{\mathsf{Cov}}% }\left[{X,Y}\right]$ .
2.

${\operatorname{\mathsf{Cov}}}\left[{W+X,Y}\right]={\operatorname{\mathsf{Cov}}% }\left[{W,Y}\right]+{\operatorname{\mathsf{Cov}}}\left[{X,Y}\right]$ since the left hand side is

$\displaystyle\operatorname{\mathsf{E}}\left[{(W+X)Y}\right]-\operatorname{% \mathsf{E}}\left[{W+X}\right]\operatorname{\mathsf{E}}\left[{Y}\right]=% \operatorname{\mathsf{E}}\left[{WY}\right]+\operatorname{\mathsf{E}}\left[{XY}% \right]-\operatorname{\mathsf{E}}\left[{W}\right]\operatorname{\mathsf{E}}% \left[{Y}\right]-\operatorname{\mathsf{E}}\left[{X}\right]\operatorname{% \mathsf{E}}\left[{Y}\right]$

which is the right hand side. Symmetry property $\Rightarrow{\operatorname{\mathsf{Cov}}}\left[{X,Y+Z}\right]={\operatorname{% \mathsf{Cov}}}\left[{X,Y}\right]+{\operatorname{\mathsf{Cov}}}\left[{X,Z}\right]$ .

For example, using property 1 twice, we have

\displaystyle{\operatorname{\mathsf{Cov}}}\left[{aX,cY}\right]=a{\operatorname% {\mathsf{Cov}}}\left[{X,cY}\right]=ac{\operatorname{\mathsf{Cov}}}\left[{X,Y}% \right].

Using property 2 twice and noticing that the covariance between a constant and a random variable is $0$

\displaystyle{\operatorname{\mathsf{Cov}}}\left[{X+b,Y+c}\right]={% \operatorname{\mathsf{Cov}}}\left[{X,Y}\right].

The covariance occurs in the variance of sums of random variables. Consider

	$\displaystyle{\operatorname{\mathsf{Var}}}\left[{X+Y}\right]$	$\displaystyle=\operatorname{\mathsf{E}}\left[{(X+Y)^{2}}\right]-\operatorname{% \mathsf{E}}\left[{X+Y}\right]^{2}$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{X^{2}+2XY+Y^{2}}\right]-\left(% \operatorname{\mathsf{E}}\left[{X}\right]+\operatorname{\mathsf{E}}\left[{Y}% \right]\right)^{2}$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{X^{2}}\right]+\operatorname{% \mathsf{E}}\left[{2XY}\right]+\operatorname{\mathsf{E}}\left[{Y^{2}}\right]-% \left(\operatorname{\mathsf{E}}\left[{X}\right]^{2}+2\operatorname{\mathsf{E}}% \left[{X}\right]\operatorname{\mathsf{E}}\left[{Y}\right]+\operatorname{% \mathsf{E}}\left[{Y}\right]^{2}\right)$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{X^{2}}\right]-\operatorname{% \mathsf{E}}\left[{X}\right]^{2}+\operatorname{\mathsf{E}}\left[{Y^{2}}\right]-% \operatorname{\mathsf{E}}\left[{Y}\right]^{2}+2\left(\operatorname{\mathsf{E}}% \left[{XY}\right]-\operatorname{\mathsf{E}}\left[{X}\right]\operatorname{% \mathsf{E}}\left[{Y}\right]\right)$
		$\displaystyle={\operatorname{\mathsf{Var}}}\left[{X}\right]+{\operatorname{% \mathsf{Var}}}\left[{Y}\right]+2{\operatorname{\mathsf{Cov}}}\left[{X,Y}\right].$

In particular, this means that

\displaystyle{\operatorname{\mathsf{Var}}}\left[{aX+bY}\right]={\operatorname{% \mathsf{Var}}}\left[{aX}\right]+{\operatorname{\mathsf{Var}}}\left[{bY}\right]% +2{\operatorname{\mathsf{Cov}}}\left[{aX,bY}\right]=a^{2}{\operatorname{% \mathsf{Var}}}\left[{X}\right]+b^{2}{\operatorname{\mathsf{Var}}}\left[{Y}% \right]+2ab{\operatorname{\mathsf{Cov}}}\left[{X,Y}\right].

(7.1)

Covariance has units = (units of $X$ ) $\times$ (units of $Y$ ) and changes if we change the scale of either $X$ or $Y$ we change the covariance between $X$ and $Y$ .

The correlation, often denoted by $\rho$ , is

\displaystyle\rho={\operatorname{\mathsf{Corr}}}\left[{X,Y}\right]=\frac{{% \operatorname{\mathsf{Cov}}}\left[{X,Y}\right]}{\sqrt{{\operatorname{\mathsf{% Var}}}\left[{X}\right]{\operatorname{\mathsf{Var}}}\left[{Y}\right]}}=\frac{{% \operatorname{\mathsf{Cov}}}\left[{X,Y}\right]}{\sigma_{X}\sigma_{Y}}.

Correlation has the benefit of being invariant to location and scale changes, which aids interpretation,

\displaystyle{\operatorname{\mathsf{Corr}}}\left[{aX+b,cY+d}\right]={% \operatorname{\mathsf{Corr}}}\left[{X,Y}\right],

provided $a>0$ , $c>0$ or $a<0,c<0$ . Quiz: Why not $a>0$ , $c<0$ ?

{\operatorname{\mathsf{Corr}}}\left[{aX,cY}\right]={\color[rgb]{0.76,0.01,0}% \frac{{\operatorname{\mathsf{Cov}}}\left[{aX,cY}\right]}{\sigma_{aX}\sigma_{cY% }}=\frac{ac{\operatorname{\mathsf{Cov}}}\left[{X,Y}\right]}{|a||c|\sigma_{X}% \sigma_{Y}}=\mbox{sign}(ac)\rho_{XY}.}

Indeed,

\displaystyle{\operatorname{\mathsf{Corr}}}\left[{aX+b,cY+d}\right]=% \operatorname{sgn}(ac){\operatorname{\mathsf{Corr}}}\left[{X,Y}\right].

Proposition 7.1.1.

$-1\leq\rho\leq 1$ .

Proof.

Without loss of generality suppose ${\operatorname{\mathsf{Var}}}\left[{X}\right]=1={\operatorname{\mathsf{Var}}}% \left[{Y}\right]$ (if not then just divide each by its standard deviation, since that does not change the correlation).

\displaystyle{\operatorname{\mathsf{Var}}}\left[{X+Y}\right]={\operatorname{% \mathsf{Var}}}\left[{X}\right]+{\operatorname{\mathsf{Var}}}\left[{Y}\right]+2% {\operatorname{\mathsf{Cov}}}\left[{X,Y}\right]=2+2{\operatorname{\mathsf{Corr% }}}\left[{X,Y}\right].

As the variance must be positive, $-1\leq{\operatorname{\mathsf{Corr}}}\left[{X,Y}\right]$ .

Now consider ${\operatorname{\mathsf{Var}}}\left[{X-Y}\right]$ to find ${\operatorname{\mathsf{Corr}}}\left[{X,Y}\right]\leq 1$ . ∎

The interpretation of the covariance/correlation between $X$ and $Y$ is that if one variable tends to increase when the other does then both the covariance and the correlation will be positive, and the stronger the association between $X$ and $Y$ the larger the value of the covariance and correlation, with $\rho=1$ corresponding to perfect positive linear association. If one variable tends to decrease when the other increases then both the covariance and the correlation will be negative, with $\rho=-1$ corresponding to perfect negative linear association. Figure 7.1 (First Link, Second Link) and Figure 7.2 (First Link, Second Link) show four joint distributions with different correlations.

When $X$ and $Y$ are independent we have $\operatorname{\mathsf{E}}\left[{XY}\right]=\operatorname{\mathsf{E}}\left[{X}% \right]\operatorname{\mathsf{E}}\left[{Y}\right]$ so the covariance and correlation are both $0$ , i.e. $\rho=0$ .

The converse, however, is not true: $\rho=0$ does not imply that $X$ and $Y$ are independent as Example 6.1.1 showed. With $X\sim N(0,1)$ and $Y=X^{2}-1$ , it was shown that $\operatorname{\mathsf{E}}\left[{XY}\right]-\operatorname{\mathsf{E}}\left[{X}% \right]\operatorname{\mathsf{E}}\left[{Y}\right]=0$ , i.e. $\rho={\operatorname{\mathsf{Cov}}}\left[{XY}\right]=0$ .

In general, we must be careful not to interpret too much into the value of these summary measures as both covariance and correlation measure linear association only and not association.