4 Markov chains 4.4 Asymptotic and invariant distributions 4.6 Calculating the asymptotic distribution

4.5 Classification of states

Definition 4.5.1.

A state $i$ is persistent (also called recurrent) if

\mathrm{P}(X_{t}=i\mbox{ for some $t\geq 1$}|X_{0}=i)=1.

Otherwise a state is transient.

This definition states that a state, $i$ , is persistent if the chain, started from state $i$ , will always (eventually) return to that state, and transient if it does not necessarily return.

Remark.

(a)

An absorbing state $i$ (such that $\mathrm{P}(X_{t+1}=i,|X_{t}=i)=1$ ) is persistent by definition. Contrariwise, a state $i$ for which probability of returning to $i$ is zero (there is no arrow towards $i$ ) is transient.
(b)

Quite often, we show that a state is persistent by showing that

$\mathrm{P}(X_{t}\neq i,\mbox{for all $t\geq 1$}|X_{0}=i)=0.$

The following theorem gives us a way to calculate whether a state is transient or persistent.

Theorem 4.5.2.

A state $i$ is transient iff

\sum_{n=1}^{\infty}[P^{n}]_{ii}<\infty.

Proof.

For a state $i$ , let $V_{i}$ be the number of visits to $i$ . Then, using indicator functions,

V_{i}=\sum_{n=1}^{\infty}1\hskip-6.544134pt{1}_{\left\{{X_{n}=i}\right\}}

where $1\hskip-6.544134pt{1}_{\left\{{A}\right\}}$ is the indicator function, i.e. $1\hskip-6.544134pt{1}_{\left\{{A}\right\}}=1$ if $A$ is true, and $1\hskip-6.544134pt{1}_{\left\{{A}\right\}}=0$ otherwise. So

\mathrm{E}(V_{i}|X_{0}=i)=\sum_{n=1}^{\infty}\mathrm{P}(X_{n}=i|X_{0}=i)=\sum_% {n=1}^{\infty}[P^{n}]_{ii}.

Let

\gamma_{i}=\mathrm{P}(X_{t}=i\mbox{ for some $t\geq 1$}|X_{0}=i).

Then, conditional on $X_{0}=i$ , $V_{i}$ is a geometric random variable with parameter $1-\gamma_{i}$ (each return to $i$ corresponds to a failure, and successive trials are independent and identically distributed by the Markov property). Hence

\sum_{n=1}^{\infty}[P^{n}]_{ii}=\mathrm{E}(V_{i}|X_{0}=i)=\frac{\gamma_{i}}{1-% \gamma_{i}}

and the rhs is finite iff $i$ is transient. ∎

Recall the definition of communicating classes from Section 4.2. The following theorem shows that transience and persistence are class properties.

Theorem 4.5.3.

(i)

The states within a single communicating class are either all transient or all persistent.
(ii)

Every persistent class is closed.
(iii)

Every finite closed class is persistent. In particular, all states in a finite irreducible chain are persistent.

Proof.

(i)

Suppose that $i\leftrightarrow j$ and that $i$ is transient. Then there exist $n,m>0$ such that $[P^{n}]_{ij}>0$ and $[P^{m}]_{ji}>0$ . But then, for all $r$ , by matrix multiplication, and using the non-negativity of the matrix entries,

$[P^{n+r+m}]_{ii}=[P^{n}P^{r}P^{m}]_{ii}\geq[P^{n}]_{ij}[P^{r}]_{jj}[P^{m}]_{ji}.$

So

$[P^{r}]_{jj}\leq\frac{[P^{n+r+m}]_{ii}}{[P^{n}]_{ij}[P^{m}]_{ji}}$

and hence, by Theorem 4.5.2,

$\sum_{r=1}^{\infty}[P^{r}]_{jj}\leq\frac{1}{[P^{n}]_{ij}[P^{m}]_{ji}}\sum_{r=1% }^{\infty}[P^{n+r+m}]_{ii}\leq\frac{1}{[P^{n}]_{ij}[P^{m}]_{ji}}\sum_{s=1}^{% \infty}[P^{s}]_{ii}<\infty.$

Therefore, $j$ is transient by Theorem 4.5.2.
(ii)

Suppose that a class $C$ is not closed. Then there exist $i\in C$ , $j\notin C$ such that $P_{ij}>0$ . But then

$\displaystyle\mathrm{P}(X_{t}\neq i\mbox{ for all }t\geq 1|X_{0}=i)$ $\displaystyle\geq\mathrm{P}(X_{1}=j,X_{t}\neq i\mbox{ for all }t\geq 2|X_{0}=i)$

$\displaystyle=\mathrm{P}(X_{t}\neq i\mbox{ for all }t\geq 2|X_{1}=j,X_{0}=i)% \mathrm{P}(X_{1}=j|X_{0}=i)$

$\displaystyle=\mathrm{P}(X_{t}\neq i\mbox{ for all }t\geq 2|X_{1}=j)P_{ij}$

$\displaystyle>0,$

where the penultimate equality uses the Markov property and the last inequality follows since $j\not\rightarrow i$ as $j\notin C$ . Therefore $i$ , and hence $C$ is not persistent.

(iii)

Suppose that $C$ is a finite closed class and let $i\in C$ . Since $C$ is finite, there must exist some $j\in C$ for which

\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq 0|X_{0}=i)>0.

Since $j\rightarrow i$ , there exists some $s$ such that $\mathrm{P}(X_{s}=i|X_{0}=j)>0$ . But then

	$\displaystyle\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq 0\|X_{0}=j)$
	$\displaystyle\geq\mathrm{P}(X_{s}=i,X_{t}=j\mbox{ for infinitely many }t\geq s% \|X_{0}=j)$
	$\displaystyle=\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq s\|X_{s}=i,X_% {0}=j)\mathrm{P}(X_{s}=i\|X_{0}=j)$
	$\displaystyle=\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq s\|X_{s}=i)% \mathrm{P}(X_{s}=i\|X_{0}=j)$
	$\displaystyle=\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq 0\|X_{0}=i)% \mathrm{P}(X_{s}=i\|X_{0}=j)$
	$\displaystyle>0.$

But this means that $\mathrm{E}(V_{j}|X_{0}=j)=\infty$ , where $V_{j}$ is the number of visits to $j$ and by the proof of Theorem 4.5.2, $j$ and hence $C$ is persistent.

∎

It is generally not difficult to identify closed classes, so establishing transience or persistence for finite MCs is straightforward.

Exercise 4.5.4.

(i) For which of the Markov chains in 4.2.2 above, is there a transient state?

$P_{3}$ : state 1.

(ii) What are transient states of the Markov chain with transition matrix?

P=\left(\begin{array}[]{cccc}0.5&0&0.5&0\\ 0.25&0.5&0&0.25\\ 0.5&0&0.5&0\\ 0&0.25&0.25&0.5\end{array}\right).~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{% }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{% }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{% }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{% }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{% }~{}~{}~{}~{}

There are two classes $\{1,3\}$ and $\{2,4\}$ with $\{2,4\}$ being transient.

Definition 4.5.5.

The period of a state, $i$ , is defined as

d_{i}=\mbox{greatest common divider}\{t:\mathrm{P}(X_{t}=i|X_{0}=i)>0\}~{}.

A state, $i$ , has period $d$ if the Markov chain, started at $i$ , can only return to $i$ after a multiple of $d$ time-steps.

All states in the same communicating class of a Markov chain have the same period. In particular, all states of an irreducible Markov chain have the same period.

Definition 4.5.6.

A Markov chain is aperiodic if all states have period 1. (So an irreducible Markov chain is aperiodic if any state has period 1.)

Remark.

A periodic chain exhibits some form of deterministic behaviour, for example a Markov chain with period 2 will alternate between two sets of states, one of which it can be in at odd time points, and the other at even time points.

If an irreducible Markov chain has $P_{ii}>0$ for some $i$ then it is aperiodic.

$d_{i}=\mathrm{gcd}\{1,\dots\}=1$ .

Exercise 4.5.7.

Classify the Markov chains in 4.2.2 as either periodic or aperiodic. State the period of the periodic Markov chains.

$P_{1}$ has period 2; others are aperiodic.

$P_{2}$ and $P_{4}$ are aperiodic since each communicating class has a state which can return to itself in 1 step with positive probability. For $P_{3}$ , $\{1\}$ is aperiodic for the same reason. The class $\{2,3,4\}$ is aperiodic as it can return to state 2 in 2 steps or 3 steps and gcd(2,3)=1.

	$\displaystyle\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq 0\|X_{0}=j)$
	$\displaystyle\geq\mathrm{P}(X_{s}=i,X_{t}=j\mbox{ for infinitely many }t\geq s% \|X_{0}=j)$
	$\displaystyle=\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq s\|X_{s}=i,X_% {0}=j)\mathrm{P}(X_{s}=i\|X_{0}=j)$
	$\displaystyle=\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq s\|X_{s}=i)% \mathrm{P}(X_{s}=i\|X_{0}=j)$
	$\displaystyle=\mathrm{P}(X_{t}=j\mbox{ for infinitely many }t\geq 0\|X_{0}=i)% \mathrm{P}(X_{s}=i\|X_{0}=j)$
	$\displaystyle>0.$

	$\displaystyle\mathrm{P}(X_{t}\neq i\mbox{ for all }t\geq 1\|X_{0}=i)$	$\displaystyle\geq\mathrm{P}(X_{1}=j,X_{t}\neq i\mbox{ for all }t\geq 2\|X_{0}=i)$
		$\displaystyle=\mathrm{P}(X_{t}\neq i\mbox{ for all }t\geq 2\|X_{1}=j,X_{0}=i)% \mathrm{P}(X_{1}=j\|X_{0}=i)$
		$\displaystyle=\mathrm{P}(X_{t}\neq i\mbox{ for all }t\geq 2\|X_{1}=j)P_{ij}$
		$\displaystyle>0,$