4 Markov chains 4.5 Classification of states 4.7 Ehrenfest model of diffusion and its invariant distribution

4.6 Calculating the asymptotic distribution

Lemma 4.6.1.

If $P$ has an asymptotic distribution $\pi$ , then $\pi$ is also its unique invariant distribution.

Proof.

First, let $t\rightarrow\infty$ in $\pi_{t+1}=\pi_{t}P$ which gives $\pi=\pi P$ , so that $\pi$ is also invariant. Secondly, suppose that $\pi^{*}$ is invariant, set $\pi_{1}=\pi^{*}$ , so that $\pi_{t}=\pi^{*}$ for all $t$ ; but we know that $\pi_{t}\rightarrow\pi$ . Thus $\pi^{*}$ must be the asymptotic distribution, $\pi$ . ∎

Remark.

This shows that, if it exists, the asymptotic distribution is the unique invariant distribution. The following theorem shows when a Markov chain has an asymptotic distribution.

Theorem 4.6.2.

A Markov chain, with a finite number of states, has an asymptotic distribution if and only if it is aperiodic and all persistent states are in the same communicating class.

Remark.

So all irreducible aperiodic Markov chains (with a finite number of states) have an asymptotic distribution.

Example 4.6.3.

In the following two examples with finite number of states, the conditions for the existence of the asymptotic distributions are not satisfied. Firstly,

P_{1}=\left(\begin{array}[]{cccc}0&0&\frac{1}{2}&\frac{1}{2}\\ 0&0&\frac{1}{2}&\frac{1}{2}\\ \frac{1}{2}&\frac{1}{2}&0&0\\ \frac{1}{2}&\frac{1}{2}&0&0\end{array}\right),\ P_{1}^{2}=\left(\begin{array}[% ]{cccc}\frac{1}{2}&\frac{1}{2}&0&0\\ \frac{1}{2}&\frac{1}{2}&0&0\\ 0&0&\frac{1}{2}&\frac{1}{2}\\ 0&0&\frac{1}{2}&\frac{1}{2}\end{array}\right).

Successive powers of $P_{1}$ alternate between these two forms with $P_{1}^{t}=P_{1}$ if $t$ is odd and $P_{1}^{t}=P_{1}^{2}$ if $t$ is even. Therefore, $P_{1}^{t}$ does not have a limit and so neither does $\pi_{t}=\pi_{0}P_{1}^{t}$ for arbitrary $\pi_{0}$ . Using a MC diagram, it is easy to see that this Markov chain is irreducible and periodic (with period 2).

The second example is:

P_{2}=\left(\begin{array}[]{cccc}\frac{1}{2}&\frac{1}{2}&0&0\\ \frac{1}{2}&\frac{1}{2}&0&0\\ 0&0&\frac{1}{2}&\frac{1}{2}\\ 0&0&\frac{1}{2}&\frac{1}{2}\end{array}\right)

In this case, it is easy to check that $P^{t}_{2}=P_{2}$ for all $t$ and hence $\lim_{t\to\infty}P_{2}^{t}$ exists. However, in the limit matrix not all rows are equal and hence the limiting distribution depends on the initial distribution and consequently, the asymptotic distribution does not exist. Easy to check that this chain is reducible, as it is impossible to go from states $\{1,2\}$ to $\{3,4\}$ , and vice-versa. In this case $\mathrm{P}(X_{t}=1|X_{0}=1)=1/2$ and $\mathrm{P}(X_{t}=1|X_{0}=3)=0$ for all $t$ .

Exercise 4.6.4.

For the following two Markov chains, find the transition matrix, compute the invariant distribution, and say whether the invariant distribution is the asymptotic distribution.

Consider a Markov chain defined by the position of an object on corners of (i) a square and (ii) a triangle. At each time point you toss a fair coin and move the object to the neighbouring corner anticlockwise (if the coin lands head up) or clockwise (otherwise).

(i)

P=\left(\begin{array}[]{cccc}0&\frac{1}{2}&0&\frac{1}{2}\\ \frac{1}{2}&0&\frac{1}{2}&0\\ 0&\frac{1}{2}&0&\frac{1}{2}\\ \frac{1}{2}&0&\frac{1}{2}&0\end{array}\right)

The invariant distribution is $\pi=(1/4,1/4,1/4,1/4)$ but is not the asymptotic distribution since the MC is periodic with period 2.

(ii)

P=\left(\begin{array}[]{ccc}0&\frac{1}{2}&\frac{1}{2}\\ \frac{1}{2}&0&\frac{1}{2}\\ \frac{1}{2}&\frac{1}{2}&0\end{array}\right)

The invariant distribution is $\pi=(1/3,1/3,1/3)$ and is the asymptotic distribution since the MC is aperiodic and irreducible.