4 Markov chains 4.2 Class structure 4.4 Asymptotic and invariant distributions

4.3 Analysing Markov chains

Definition 4.3.1 (Multi-step transitions).

We call $P^{(m)}$ the matrix of $m$ -step transition probabilities:

\left(P^{(m)}\right)_{i,j}=\mathrm{P}(X_{t+m}=j|X_{t}=i).

Theorem 4.3.2.

P^{(m)}=P^{m},

that is the $m^{th}$ power of the matrix $P$ .

Proof.

For the case $m=2$ ; induction can be used for the general result.

\left(P^{(2)}\right)_{i,j}=\mathrm{P}(X_{t+2}=j|X_{t}=i)=\sum_{k}\mathrm{P}(X_% {t+2}=j,X_{t+1}=k|X_{t}=i).

Now in general, $\mathrm{P}(A,B|C)=\mathrm{P}(A|B,C)\mathrm{P}(B|C)$ , giving

\sum_{k}\mathrm{P}(X_{t+2}=j|X_{t+1}=k,X_{t}=i)\mathrm{P}(X_{t+1}=k|X_{t}=i).

By the Markov property this is

\sum_{k}\mathrm{P}(X_{t+2}=j|X_{t+1}=k)\mathrm{P}(X_{t+1}=k|X_{t}=i)=\sum_{k}P% _{i,k}P_{k,j}=P^{2}_{i,j}\ .

∎

Proposition 4.3.3 (The Chapman-Kolmogorov equation).

P^{(m+n)}\ =\ P^{(m)}\,P^{(n)}

\mathrm{P}(X_{t+m+n}=j|X_{t}=i)\,=\,\sum_{k}\mathrm{P}(X_{t+m+n}=j|X_{t+m}=k)% \mathrm{P}(X_{t+m}=k|X_{t}=i).

This is a relationship which we use just once; its analogue in continuous time is important. From Theorem 4.3.2, it follows from the associativity of matrix multiplication.

Theorem 4.3.4.

Let the row vector $\pi_{t}$ hold the pmf of $X_{t}$ , i.e.

\left(\pi_{t}\right)_{i}=\mathrm{P}(X_{t}=i)

\pi_{t}=\left(\mathrm{P}(X_{t}=1)\ \ \mathrm{P}(X_{t}=2)\ \ \cdots\right).

Then

\pi_{t+1}=\pi_{t}P

and more generally

\pi_{t+m}=\pi_{t}P^{(m)}=\pi_{t}P^{m}.

In particular, suppose that $X_{0}$ has initial distribution $\pi_{0}$ and that the chain is homogeneous in time, then

\pi_{1}=\pi_{0}P,\pi_{2}=\pi_{1}P=\pi_{0}P^{2},\ \pi_{3}=\pi_{2}P,\ \ldots\ % \pi_{t+1}=\pi_{t}P\,=\,\pi_{1}P^{t}=\pi_{0}P^{t+1}.

Remark.

Note that $\pi_{t}$ , the pmf of $X_{t}$ , depends on the initial distribution $\pi_{0}$ of $X_{0}$ . This is often determined by knowledge that $X_{0}$ is in some particular state, $k$ say. Then $\pi_{0}=(0,\ldots,0,1,0\ldots,0)$ where the $1$ is in the $k$ th position.

The point of the result is that $\pi_{t+1}$ , $\pi_{t+2}$ , …can be evaluated knowing only $\pi_{t}$ .

Example 4.3.5 (No claims bonus).

Let $X_{t}$ have four states representing either none, one year, two years or three years of no claims bonus on an automobile insurance of Mr X at the $t^{th}$ year. Let the transition probability matrix $P$ be

\left(\begin{array}[]{cccc}\frac{1}{3}&\frac{2}{3}&\ 0&0\\ \frac{1}{3}&0&\frac{2}{3}&0\\ \frac{1}{6}&\frac{1}{6}&0&\frac{2}{3}\\ 0&\frac{1}{6}&\frac{1}{6}&\frac{2}{3}\end{array}\right)

and take $\pi_{0}=(\,1,\,0,\,0,\,0\,)$ . Then

\pi_{1}^{T}=\left(\begin{array}[]{c}0.33\\ 0.67\\ 0\\ 0\end{array}\right)\,\pi_{2}^{T}=\left(\begin{array}[]{c}0.33\\ 0.22\\ 0.44\\ 0\end{array}\right)\,\pi_{3}^{T}=\left(\begin{array}[]{c}0.26\\ 0.30\\ 0.15\\ 0.29\end{array}\right)\,\pi_{10}^{T}=\left(\begin{array}[]{c}0.16\\ 0.21\\ 0.21\\ 0.42\end{array}\right)\,\pi_{20}^{T}=\left(\begin{array}[]{c}0.1579\\ 0.2105\\ 0.2105\\ 0.4210\end{array}\right).

So as $t$ increases, $\pi_{t}$ converges. In the following, we are interested to make predictions about the system behaviour of $\{X_{t}\}$ as $t$ goes larger.