5 Continuous Markov chains 5.1 Transition probabilities 5.3 Asymptotic and invariant distributions

5.2 The rate matrix

5.2.1 Modelling a homogeneous cts MC by differentials

Assume that the probabilities vary smoothly with time so that the derivative of $P(t)$ exists at $t=0$ , i.e. that

\frac{d}{dt}P(t)_{\mbox{at\ }t=0}\ =P^{\prime}(0)=Q.

Remark.

(a)

$Q$ is a fixed matrix, not a function of $t$ , since it is evaluated at $t=0$ .
(b)

$\sum_{j}P(t)_{i\,j}=1\ \Rightarrow\ \sum_{j}\frac{d}{dt}P(t)_{i\,j}=0\ % \Rightarrow\ \sum_{j}Q_{i\,j}=0$ , i.e. the rows of $Q$ sum to zero.
(c)

$Q$ is called the transition probability rate matrix or infinitesimal generator.
(d)

For small $h$ ,

$\frac{P(h)-P(0)}{h}\approx Q\mbox{\ so\ }P(h)\approx P(0)+Qh=I+Qh.$

In particular $P(h)_{i\,i}\approx 1+Q_{i\,i}h$ so the diagonal elements of $Q$ must all be negative (or zero). We set $\lambda_{i}=-Q_{i\,i}\geq 0$ . This is the rate of transition out of state $i$ since $P(X(h)\neq i|X(0)=i)\approx\lambda_{i}h$ .

Further, for $j\neq i$ , $P(h)_{i\,j}\approx Q_{i\,j}h$ so that $Q_{i\,j}\geq 0$ , the rate of transition to state $j$ from state $i$ . Note that $\sum_{j\neq i}Q_{i\,j}=-Q_{i\,i}=\lambda_{i}$ . The outward and inward rates balance.

Interpretation

We shall see below that for the continuous time Markov process, the rate matrix is as important as the probability transition matrix of the discrete-time Markov chain.

As with the probability transition matrix, for two states $i\neq j$ , $Q_{ij}>0$ if and only if it is possible to move directly from state $i$ to state $j$ . (The rate matrix can be used to construct diagrams of the dynamics of the Markov chain as in 4.2.1.)

However, the entries in the rate matrix are not probabilities, so the off-diagonal elements can take values greater than 1. (This is similar to pdfs of continuous random variables.) The entries of the rate matrix relate to probabilities via

\mathrm{P}(X(t+h)=j|X(t)=i)\approx Q_{ij}h,\,\,i\neq j

\mathrm{P}(X(t+h)=i|X(t)=i)\approx 1+Q_{ii}h=1-\lambda_{i}h,

both for small $h$ .

The following two theorems help us to interpret the values of the rate matrix.

Theorem 5.2.1.

Let $T$ be the length of stay in state $j$ (say from time 0) before a transition to another state occurs. Then $T$ has an exponential pdf with mean $1/\lambda_{j}$ (the inverse of the rate of transition from state $j$ ).

Proof.

Let $S(t)=\mathrm{P}(T>t)$ , be the survivor function of $T$ . Then for small $h$

S(t+h)\begin{array}[t]{cl}=&\mathrm{P}(T>t+h)\\ =&\mathrm{P}(T>t+h,T>t)\\ =&\mathrm{P}(T>t+h|T>t)\mathrm{P}(T>t)\\ \approx&\mathrm{P}(X(t+h)=j|X(t)=j)S(t)\\ \approx&(1-\lambda_{j}h)S(t)\end{array}.

Thus

\frac{S(t+h)-S(t)}{h}\approx-\lambda_{j}S(t)\Rightarrow S^{\prime}(t)=-\lambda% _{j}S(t)\Rightarrow S(t)=S(0)\exp(-\lambda_{j}t).

But $S(0)=1$ giving $S(t)=\exp(-\lambda_{j}t)$ . The pdf is $-\frac{d}{dt}S(t)$ which is $\lambda_{j}\exp(-\lambda_{j}t)$ . ∎

Definition 5.2.2.

Define $J_{0}=0$ and set

J_{n+1}=\inf\{t\geq J_{n}:X(t)\neq X(J_{n})\}.

We call $J_{0}\leq J_{1}\leq\cdots$ the jump times of $X(t)$ . The above result tells us that if $X(J_{n})=j$ , then $J_{n+1}-J_{n}\sim\mathrm{Exp}(\lambda_{j})$ . By the Markov property it can be shown that this is independent of $J_{0},\dots,J_{n}$ . The difference $J_{n+1}-J_{n}$ is called the holding time at $j$ .

Theorem 5.2.3.

If the continous time Markov process is currently in state $i$ , then it moves to state $j$ next with probability

\frac{Q_{ij}}{\lambda_{i}}=-\frac{Q_{ij}}{Q_{ii}}.

Proof.

For small $h$ we have

\mathrm{P}(X(t+h)=j|X(t)=i)\approx Q_{ij}h,

\mathrm{P}(X(t+h)\neq i|X(t)=i)\approx\lambda_{i}h,

	$\displaystyle\mathrm{P}(X(t+h)=j\|X(t)=i,X(t+h)\neq i)$	$\displaystyle=$	$\displaystyle\frac{\mathrm{P}(X(t+h)=j,X(t+h)\neq i\|X(t)=i)}{\mathrm{P}(X(t+h)% \neq i\|X(t)=i)}$
		$\displaystyle\approx$	$\displaystyle\frac{Q_{ij}h}{\lambda_{i}h}=\frac{Q_{ij}}{\lambda_{i}},$

and the result follows because this does not depend on $t$ . ∎

Definition 5.2.4.

Define a discrete stochastic process by $Y_{n}=X_{J_{n}}$ for $n=0,1,\dots$ . The above result says that $Y_{n}$ is a discrete MC with transition matrix given by

P_{ij}=\frac{Q_{ij}}{\lambda_{i}}\quad(i\neq j);\quad P_{ii}=0.

We call $Y_{n}$ the jump chain of process $X(t)$ .

Exercise 5.2.5.

Consider a 3-state continuous time Markov process. The distributions of the time that the process remains in each state have means $1/2$ , $1$ and $2$ respectively. On leaving state 1 the process moves to state 2. On leaving state 2 the process is equally likely to go to either state 1 or state 3, and on leaving state 3 the process is twice as likely to go to state 1 than state 2. Write down the rate matrix noting that its entries need not be fraction and non-negative.

Q=\left(\begin{array}[]{ccc}-2&2&0\\ 1/2&-1&1/2\\ 1/3&1/6&-1/2\end{array}\right).