4 Markov chains

4.2 Class structure

Method 4.2.1 (State transition diagrams of Markov chains).

To aid the understanding of the dynamics of a Markov chain it is useful to draw a diagram which indicates all the possible states of the chain, with arrows denoting the possible transitions. An arrow joins state i to state j if Pi⁢j>0; the precise values of the probabilities are irrelevant. If a state remains in itself with positive probability, then we simply draw a self-directed arrow.

Example 4.2.2.

Draw diagrams for the following Markov chains. Each chain can take the values {1,2,3,4}.

(i)

P1=(00.50.250.25100010001000).                                         

(ii)

P2=(0.75000.2500.50.5001001000).                                         

(iii)

P3=(0.50.20.20.1000.50.500.500.500.50.50).                                         

(iv)

P4=(0.750.250000.50.5000011000).                                         
Definition 4.2.3.

We say that i leads to j and write i→j if

P(Xt=j for some t≥0|X0=i)>0.

This is equivalent to saying that it is possible to get from i to j in some number of steps. We say that i communicates with j and write i↔j if both i→j and j→i.

Theorem 4.2.4.

The relation ↔ is an equivalence relation on the state space. It therefore partitions the state space into communicating classes.

Proof.

Note that, for disjoint sets i and j, i→j iff there exists a sequence of states i0,i1,…,in with i0=i and in=j for which Pi0,i1⁢Pi1,i2⁢…⁢Pin-1,in>0. From this it follows that i→j and j→k implies i→k. Also, i→i for any state i. Clearly i↔j implies j↔i. So ↔ is an equivalence relation and the equivalence classes partition the state space. ∎

Definition 4.2.5.

A communicating class C is closed if, for all i∈C, i→j implies that j∈C. A closed class is therefore one from which there is no escape. A state i is absorbing if {i} is a closed class.

Exercise 4.2.6.

Identify the communicating classes for each of the Markov chains in 4.2.2. Which are closed?

  • (i)

    {1,2,3,4}. Closed.

  • (ii)

    {1,4} and {2,3}. Both closed.

  • (iii)

    {1} and {2,3,4}. Only {2,3,4} is closed.

  • (iv)

    {1,2,3,4}. Closed.

Definition 4.2.7.

A Markov chain is irreducible if it has a single communicating class i.e. i↔j for all states i and j. Otherwise the chain is reducible.

Exercise 4.2.8.

Classify the the Markov chains in 4.2.2 as either irreducible or reducible.

P1,P4 are irreducible, P2 and P3 are reducible.