Home page for accesible maths 8 More than one random variable 8.1.1 Properties of

p_{X,Y}(x,y)

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

8.2 Independence

Independence is the simplest form for joint behaviour of two (or more) random variables. Informally, two random variables ${X}$ and ${Y}$ are independent if knowing the value of one of them gives no information about the value of the other.

The outcomes of, say, rolls of two separate dice are independent in exactly this sense: knowing that the red die showed a 4 does not give us any information about the score of the blue die, and, conversely, knowing that the score of the blue die was ${3}$ does not give any information about the red die.

Two random variables ${X}$ and ${Y}$ are independent if the events ${{\{X\in A\}}}$ and ${{\{Y\in B\}}}$ are independent for all sets ${A}$ and ${B}$ , i.e. $\displaystyle{\rm P}(X\in A,Y\in B)={\rm P}(X\in A){\rm P}(Y\in B)$ for all sets $A, B$ .

Theorem 8.4.

Two discrete random variables ${X}$ and ${Y}$ are independent if and only if

\displaystyle p_{XY}(x,y)=p_{X}(x)p_{Y}(y)

for all $x$ and $y$ .

Proof.

Let $X$ and $Y$ be independent, and let ${A=\{x\}}$ and ${B=\{y\}}$ . Then

$\displaystyle p_{XY}(x,y)$	$\displaystyle=$	$\displaystyle{\rm P}(X\in A,Y\in B)$
	$\displaystyle=$	$\displaystyle{\rm P}(X\in A){\rm P}(Y\in B)$
	$\displaystyle=$	$\displaystyle p_{X}(x)p_{Y}(y).$

Conversely, if the joint pmf factorises we get for arbitrary sets ${A}$ and ${B}$

$\displaystyle{\rm P}(X\in A,Y\in B)$	$\displaystyle=$	$\displaystyle\sum_{x\in A}\sum_{y\in B}p_{XY}(x,y)$
	$\displaystyle=$	$\displaystyle\sum_{x\in A}\sum_{y\in B}p_{X}(x)p_{Y}(y)$
	$\displaystyle=$	$\displaystyle\sum_{x\in A}p_{X}(x)\sum_{y\in B}p_{Y}(y)$
	$\displaystyle=$	$\displaystyle{\rm P}(X\in A){\rm P}(Y\in B).$

∎

If ${X}$ and ${Y}$ are discrete random variables, the conditional pmfs are

	$\displaystyle p_{X\mid Y}(x\mid y)$	$\displaystyle=$	$\displaystyle\frac{p_{X,Y}(x,y)}{p_{Y}(y)},$
	$\displaystyle p_{Y\mid X}(y\mid x)$	$\displaystyle=$	$\displaystyle\frac{p_{XY}(x,y)}{p_{X}(x)}.$

Thus $p_{X\mid Y}(x\mid y)={\rm P}(X=x,Y=y)/{\rm P}(Y=y)={\rm P}(X=x\mid Y=y).$

Exercise 8.5.

Show that if the discrete variables ${(X,Y)}$ are independent then for all ${x,y}$ :

\displaystyle p_{X\mid Y}(x\mid y)

\displaystyle=

\displaystyle p_{X}(x)

Solution.

$\displaystyle p_{X\mid Y}(x\mid y)$	$\displaystyle=$	$\displaystyle\frac{p_{X,Y}(x,y)}{p_{Y}(y)}$
	$\displaystyle=$	$\displaystyle\frac{p_{X}(x)p_{Y}(y)}{p_{Y}(y)}$
	$\displaystyle=$	$\displaystyle p_{X}(x)$

These results conform with intuition as, when ${X}$ and ${Y}$ are independent, knowing the value of ${X}$ should tell us nothing about ${Y}$ .

The converse is also true: if the conditional distribution of ${X}$ given ${{Y=y}}$ is independent of ${y}$ or, equivalently, the conditional distribution of ${Y}$ given ${{X=x}}$ is independent of ${x}$ , then ${X}$ and ${Y}$ are independent.

Example 8.6.

A fair coin is tossed. If it shows ${H}$ a fair die is thrown, if ${T}$ a biased die. The bias makes even numbers twice as probable as odd numbers. Find the joint pmf of ${X}$ , the toss of the coin, and ${Y}$ , the score on the die.

Solution.

Code ${T}$ and ${H}$ as ${0}$ and ${1}$ to make rvs. Marginal: ${{p_{X}(x)=}}{\color[rgb]{1,1,1}{1/2}}$ for ${{x=0,1}}$ .

Conditional:
${x=1}$ : ${p_{Y|X}(y|1)={\color[rgb]{1,1,1}{1/6}}}$ for ${y=1,2,\dots,6}$ .
${x=0}$ : ${p_{Y|X}(y|0)={\color[rgb]{1,1,1}{c}}}$ for ${y={\color[rgb]{1,1,1}{1,3,5}}}$ and ${p_{Y|X}(y|0)={\color[rgb]{1,1,1}{2c}}}$ for ${y={\color[rgb]{1,1,1}{2,4,6}}}$ . Hence ${c={\color[rgb]{1,1,1}{1/9}}}$ .

Using ${p_{XY}(x,y)}={\color[rgb]{1,1,1}{p_{Y|X}(y|x)\;p_{X}(x)}}$ delivers

	1	2	3	4	5	6
0	1/18	2/18	1/18	2/18	1/18	2/18
1	1/12	1/12	1/12	1/12	1/12	1/12

Example 8.7.

For the joint pmf in Example 8.3 obtain the conditional pmf of ${X}$ given ${Y=2}$ .

		${Y}$
		0	1	2	3
${X}$	1			2/60		16/60
	2			3/60		24/60
	3			6/60		20/60
				11/60

\displaystyle p_{X\mid Y}(x|2)

\displaystyle=

\displaystyle{\color[rgb]{1,1,1}{p_{XY}(x,2)/p_{Y}(2)}}

Thus $x=1$ w.p. $2/11$ , $x=2$ w.p. $3/11$ and $x=3$ w.p. $6/11$ .

We have seen that when $X$ and $Y$ are both discrete, they are independent if and only if their joint pmf can be factorised as a product of the marginal pmfs.

p_{XY}(x,y)=p_{X}(x)p_{Y}(y).

Our definition of independence holds also for continuous random variables, but there is no joint pmf for continuous random variables. It is beyond the scope of this module, but there can exist a joint probability density function $f_{X,Y}(x,y)$ . As with univariate random variables, results that hold in the discrete case with probability mass functions often hold in the continuous case with joint mass functions.

Theorem 8.8.

Two continuous random variables $X$ and $Y$ are independent if and only if

f_{XY}(x,y)=f_{X}(x)f_{Y}(y).

Proof.

Not given here. ∎

The result is needed for constructing likelihood-based estimates in statistics: often it is assumed that repeated experiments result in $n$ independent observations of a random variable, and the joint density function of the observations is the product of the marginal densities.

8.3 Weak law of large numbers

Recall from Exercise 5.13 that if an experiment is repeated $n$ times then, as $n$ gets large, the proportion of times an event $A$ occurs converges to ${\rm P}(A)$ . We will now prove a similar result concerning the average of several realisations of a random variable converging to the expected value. We start with a lemma which is proved in MATH230.

Lemma 8.9.

Let $X_{1},X_{2},\ldots,X_{n}$ be jointly distributed random variables with finite expectation and variance. Then

•

${\rm E}(X_{1}+X_{2}+\cdots+X_{n})={\rm E}(X_{1})+{\rm E}(X_{2})+\cdots+{\rm E}% (X_{n})$ , and
•

if $X_{1},X_{2},\ldots,X_{n}$ are independent then

${\rm Var}(X_{1}+X_{2}+\cdots+X_{n})={\rm Var}(X_{1})+{\rm Var}(X_{2})+\cdots+{% \rm Var}(X_{n}).$

Now suppose that $X_{1},X_{2},\ldots,X_{n}$ are independent copies of a random variable $X$ . For example, suppose we repeated an experiment $n$ times, and $X_{i}$ is the measured outcome on the $i$ th experiment. This setup means that for each $i$ we have

	$\displaystyle{\rm E}[X_{i}]$	$\displaystyle=$	$\displaystyle{\color[rgb]{1,1,1}{\rm E}[X]}$
	$\displaystyle{\rm Var}(X_{i})$	$\displaystyle=$	$\displaystyle{\color[rgb]{1,1,1}{\rm Var}(X)}$

If we want to report a value, scientists will usually measure it $n$ times and report the average measured value. Let $X_{i}$ be the measured value on the $i$ th experiment. The average measured value is

\bar{X}=\frac{1}{n}(X_{1}+X_{2}+\cdots+X_{n}).

Why do we do this?

Let’s consider the properties of $\bar{X}$ . For simplicity, write $\mu$ for ${\rm E}[X]$ and $\sigma^{2}$ for ${\rm Var}(X)$ .

$\displaystyle{\rm E}[\bar{X}]$	$\displaystyle=$	$\displaystyle{\rm E}\left[\frac{1}{n}(X_{1}+X_{2}+\cdots+X_{n})\right]$

	$\displaystyle=$	$\displaystyle\frac{1}{n}{\rm E}[X_{1}+X_{2}+\cdots+X_{n}]\qquad\mbox{by % linearity of ${\rm E}$}$

	$\displaystyle=$	$\displaystyle\frac{1}{n}\left\{{\rm E}[X_{1}]+{\rm E}[X_{2}]+\cdots+{\rm E}[X_% {n}]\right\}\qquad\mbox{by Lemma \ref{LemLinOfEVar}}$

	$\displaystyle=$	$\displaystyle\frac{1}{n}\left\{{\rm E}[X]+{\rm E}[X]+\cdots+{\rm E}[X]\right\}% \qquad\mbox{since}\quad{\rm E}[X_{i}]={\rm E}[X]$

	$\displaystyle=$	$\displaystyle\frac{1}{n}\{n\mu\}$

	$\displaystyle=$	$\displaystyle\mu$

So $\bar{X}$ has expectation the quantity we wish to report, the true expected value of $X$ . Of course, simply reporting the first measurement $X_{1}$ would also have this expected value.

Consider now the variance of $\bar{X}$ :

$\displaystyle{\rm Var}(\bar{X})$	$\displaystyle=$	$\displaystyle{\rm Var}\left(\frac{1}{n}(X_{1}+X_{2}+\cdots+X_{n})\right)$

	$\displaystyle=$	$\displaystyle\frac{1}{n^{2}}{\rm Var}(X_{1}+X_{2}+\cdots+X_{n})\qquad\mbox{by % the calculation on p\pageref{VarCX}}$

	$\displaystyle=$	$\displaystyle\frac{1}{n^{2}}\left\{{\rm Var}(X_{1})+{\rm Var}(X_{2})+\cdots+{% \rm Var}(X_{n})\right\}\qquad\mbox{by Lemma \ref{LemLinOfEVar}}$

	$\displaystyle=$	$\displaystyle\frac{1}{n^{2}}\{n\sigma^{2}\}$

	$\displaystyle=$	$\displaystyle\frac{\sigma^{2}}{n}$

The variance of our reported quantity, $\bar{X}$ , decreases as the number of measurements $n$ increases.

We can use Chebychev’s inequality (Section 4.6) to be more precise about this. Recall that for any random variable $R$ with expected value $\mu$ and standard deviation $s$

{\rm P}\left(|R-\mu|>cs\right)\leq\frac{1}{c^{2}},

for any $c>0$ .

[I am using $s$ for the standard deviation here, instead of $\sigma$ , to avoid confusion with the $\sigma^{2}$ already used for the variance of $X$ .]

Hence for the random variable $\bar{X}$ with expected value $\mu$ , variance $\sigma^{2}/n$ and hence standard deviation $\sigma/\sqrt{n}$ , we have

{\rm P}(|\bar{X}-\mu|>\frac{c\sigma}{\sqrt{n}})\leq\frac{1}{c^{2}}

By taking $k=c/\sqrt{n}$ , we can rearrange this expression to

{\rm P}(|\bar{X}-\mu|>k\sigma)\leq\frac{1}{k^{2}n}.

We see that as $n$ gets large, the probability that the sample average $\bar{X}$ is more than distance $k\sigma$ away from the expected value of the original random quantity $X$ decreases to 0.

Since $k$ is arbitrary, in some sense we can say that $\bar{X}$ converges to $\mu$ . This is called the weak law of large numbers. You will see various other forms of convergence of random variables in later courses.

One final thing to note: the standard deviation $\sigma$ is exactly the right quantity for determining the appropriate scale for measuring distance here: the events are of the type “random variable is more than $k$ standard deviations away from the mean”.