4 Markov chains 4.7 Ehrenfest model of diffusion and its invariant distribution 4.9

N

-step formula for transition matrices - intro.

4.8 Expected hitting times

Definition 4.8.1.

If a Markov chain $X$ is started in state $i$ ( $X_{0}=i$ ) and reaches state $j$ for the first time at time $T_{ij}$ then $T_{ij}$ is called the hitting time for state $j$ from state $i$ . In mathematical notation

T_{ij}=\min\{T:X_{T}=j,T\geq 1|X_{0}=i\}.

When $i=j$ we obtain $T_{ii}$ , the return time for state $i$ .

Here we will be concerned with expected hitting times and expected return times:

\tau_{ij}=\mathbf{E}\left[{T_{ij}}\right].

Theorem 4.8.2.

For an $n$ state Markov chain with transition matrix $P$ ,

\tau_{ij}=1+\sum_{k=1,k\neq j}^{n}P_{ik}\tau_{kj}.

Proof.

For a given realisation of the Markov chain, let $T_{ij}$ be the actual hitting time for state $j$ from state $i$ . Then

T_{ij}=\left\{\begin{array}[]{clll}1&\mbox{ with probability }&P_{ij}&\\ 1+T^{\prime}_{kj}&\mbox{ with probability }&P_{ik}&(k\neq j)\end{array}\right.,

where $T^{\prime}_{kj}$ is the further time to reach state $j$ if the first transition is from $i$ to $k$ . By the Markov property, $T^{\prime}_{kj}$ has the same distribution as $T_{kj}$ . Taking expectations and using the Tower Law

$\displaystyle\tau_{ij}$	$\displaystyle=$	$\displaystyle\mathbf{E}\left[{T_{ij}}\right]$
	$\displaystyle=$	$\displaystyle P_{ij}\times 1+\sum_{k=1,k\neq j}^{n}P_{ik}\mathbf{E}\left[{(1+T% ^{\prime}_{kj})}\right]$
	$\displaystyle=$	$\displaystyle\sum_{k=1}^{n}P_{ik}+\sum_{k=1,k\neq j}^{n}P_{ik}\mathbf{E}\left[% {T_{kj}}\right]\quad\mbox{(by linearity of expectation)}$
	$\displaystyle=$	$\displaystyle 1+\sum_{k=1,k\neq j}^{n}P_{ik}\tau_{kj}.\quad\mbox{(since rows % sum to 1)}$

∎

Example 4.8.3.

A Markov chain has transition probability matrix

P=\left(\begin{array}[]{ccc}0.1&0.3&0.6\\ 0.3&0.5&0.2\\ 0.5&0.5&0.0\end{array}\right).

What is the expected return time for state $1$ and the expected hitting time for state $1$ from each of states $2$ and $3$ ? What is the expected time to hit state $1$ if the Markov chain has initial distribution $(1/2,1/10,2/5)$ ?

From Theorem 4.8.2

$\displaystyle\tau_{11}$	$\displaystyle=$	$\displaystyle 1+0.3\tau_{21}+0.6\tau_{31}$	(4.1)
$\displaystyle\tau_{21}$	$\displaystyle=$	$\displaystyle 1+0.5\tau_{21}+0.2\tau_{31}$	(4.2)
$\displaystyle\tau_{31}$	$\displaystyle=$	$\displaystyle 1+0.5\tau_{21}$	(4.3)

Substituting (4.3) into (4.2) gives

\tau_{21}=1+0.5\tau_{21}+0.2+0.1\tau_{21}\Rightarrow 0.4\tau_{21}=1.2% \Rightarrow\tau_{21}=3.0.

Substitution back in to (4.3) gives $\tau_{31}=2.5$ ; then substitution into the first equation provides $\tau_{11}=3.4$ .

NB: More generally (4.2) and (4.3) would be two simultaneous equations. The invariant distribution for $P$ is $\pi=(10/34,15/34,9/34)$ , so in this particular example: $\tau_{11}=1/\pi_{1}$ .

The expected hitting time for state $1$ from starting distribution $(1/2,1/10,2/5)$ is

(1/2)*\tau_{11}+(1/10)*\tau_{21}+(2/5)*\tau_{31}=17/10+3/10+1=3.

Example 4.8.4.

A Markov chain has transition kernel

P=\left(\begin{array}[]{ccc}0.6&0.4&0\\ 0.8&0.2&0\\ 0.5&0.3&0.2\end{array}\right)

What can be said about the return time for state $3$ and the hitting time for state $3$ from each of states $1$ and $2$ ?

It is impossible to get to state $3$ from either state $1$ or state $2$ . From state $3$ , either the return time is $1$ or the chain will never return to state $3$ . One way to view of this is that the hitting/return times are infinite. Also the classes are $\{1,2\}$ and $\{3\}$ .

The invariant distribution for $P$ is $\pi=(2/3,1/3,0)$ , so in this particular example: $\tau_{33}=1/\pi_{3}$ .

Remark.

Recall that for a finite irreducible Markov chain all states are persistent and it can be shown that the expected hitting times are finite.

Theorem 4.8.5.

If a Markov chain has invariant distribution $\pi$ then the expected return time to a persistent state $i$ is $\tau_{ii}=1/\pi_{i}$ .

Remark.

(a)

Compare with a geometric random variable for which the expected value of the first occurrence is the reciprocal of the success probability.
(b)

In an irreducible MC, a stationary distribution exists if and only if all states are persistent with finite $\tau_{ii}$ and in this case the invariant distribution is given by $\pi_{i}=1/\tau_{ii}$ . In particular in an irreducible MC with finite state space, the invariant distribution always exists whereas the asysmptotic distribution will exist if it is additionally aperiodic.

Heuristic of proof: (not a full rigorous proof) Let the Markov chain start in its stationary distribution $\pi$ and imagine letting the chain run for a very long time $T$ .

Let $t_{i}^{(A)}$ be the time until the chain first hits state $i$ and $t_{i}^{(B)}$ be the time from the last time the chain hits state $i$ until $T$ .

Suppose that the chain hits state $i$ $N_{i}$ times and let $T_{ii}^{(1)},\dots,T_{ii}^{(N_{i}-1)}$ be the return times to state $i$ .

Figure 4.1: Link, Caption: none provided

Now (see the picture) $T=t_{i}^{(A)}+\sum_{k=1}^{N_{i}-1}T_{ii}^{(k)}+t_{i}^{(B)}$ ; so taking expectations:

T=\mathbf{E}\left[{t_{i}^{(A)}+t_{i}^{(B)}}\right]+\mathbf{E}\left[{\sum_{k=1}% ^{N_{i}-1}T_{ii}^{(k)}}\right].

(4.4)

Because the chain is in its stationary distribution

\mathbf{E}\left[{N_{i}}\right]=\mathbf{E}\left[{\sum_{j=1}^{T}1\hskip-6.544134% pt{1}_{\left\{{X_{j}=i}\right\}}}\right]=\sum_{j=1}^{T}\mathbf{E}\left[{1% \hskip-6.544134pt{1}_{\left\{{X_{j}=i}\right\}}}\right]=\pi_{i}T;

here $1\hskip-6.544134pt{1}_{\left\{{A}\right\}}$ is the indicator function, i.e. $1\hskip-6.544134pt{1}_{\left\{{A}\right\}}=1$ if $A$ is true, and $1\hskip-6.544134pt{1}_{\left\{{A}\right\}}=0$ otherwise.
If we replace $N_{i}$ by $\pi_{i}T=\mathbf{E}\left[{N_{i}}\right]$ (this needs justification which is why this is not a formal proof) then by (4.4)

T=\mathbf{E}\left[{t_{i}^{(A)}+t_{i}^{(B)}}\right]+(\pi_{i}T-1)\tau_{ii}.

Dividing through by $T$ gives

1=(\pi_{i}-1/T)\tau_{ii}+\frac{1}{T}\mathbf{E}\left[{t_{i}^{(A)}+t_{i}^{(B)}}% \right].

By the Remark before this Theorem, $\mathbf{E}\left[{t_{i}^{(A)}+t_{i}^{(B)}}\right]$ is bounded (since $t_{i}^{(B)}<T_{ii}^{N_{i}}$ ), so letting $T\rightarrow\infty$ gives the required result. ∎