5 Continuous Markov chains 5 Continuous Markov chains 5.2 The rate matrix

5.1 Transition probabilities

Definition 5.1.1.

A continuous MC is a SP $X(t)$ defined in continous time with discrete state space and which satisfies the Markov property:

\mathrm{P}(X(t)=j|X(s_{1})=i_{1},X(s_{2})=i_{2},\ldots)\ =\ \mathrm{P}(X(t)=j|% X(s_{1})=i_{1})

for any $t>s_{1}>s_{2}>\ldots$ .

An example could be the number of telephone calls received by a call centre during $(0,t]$ .

Note on classification of states
The states of a continuous time Markov chain are classified in a similar way to the states of a discrete time Markov chain. The only difference is that, in continuous time, states cannot have a period (and continuous time Markov chains are always aperiodic).

Definition 5.1.2.

The probability transition functions (PTFs) of a cts MC are

P(s,t)_{i\,j}=\mathrm{P}(X(t)=j|X(s)=i)

for $t>s$ . These elements define a matrix $P(s,t)$ . Note that $P(0,0)=I$ , the identity matrix.

Definition 5.1.3.

The MC is homogeneous if $P(s,t)=P(t-s)$ , i.e. depends only on the length of the time interval. Note $P(0)=I$ .

Remark.

For the discrete MC we needed only to define the 1-step PTMs, because the multi-step transitions were constructed from a succession of single step transitions. For the cts MC there is no shortest step possible from which to build up the longer step transitions. Instead, we shall use differentials and integrate these to give finite step transitions.

In consequence, it is not usual to specify the PT functions $P(s,t)$ of a MC for all $t$ and $s$ , but if that were to be done, they must satisfy the following equations to ensure that there are no inconsistencies.

Theorem 5.1.4 (The Chapman–Kolmogorov equations).

The PT functions of a MC satisfy the matrix equations

P(s,t)=P(s,r)P(r,t)

for any $t>r>s$ .

Proof.

P(s,t)_{i\,j}\begin{array}[t]{cl}=&\mathrm{P}(X(t)=j|X(s)=i)\\ =&\sum_{k}\mathrm{P}(X(t)=j,X(r)=k|X(s)=i)\\ =&\sum_{k}\mathrm{P}(X(t)=j|X(r)=k,X(s)=i)\mathrm{P}(X(r)=k|X(s)=i)\\ =&\sum_{k}\mathrm{P}(X(t)=j|X(r)=k)\mathrm{P}(X(r)=k|X(s)=i)\\ =&\sum_{k}P(s,r)_{i\,k}P(r,t)_{k,j}\\ =&[P(s,r)P(r,t)]_{i\,j}\end{array}

which gives the required matrix product since it holds for all $i, j$ . ∎

Remark.

For a set of matrix valued functions $P(s,t)$ to be the PT functions of a MC, the only conditions are that they satisfy these equations and that the rows of the matrices are non-negative and sum to 1.

Corollary 5.1.5.

For a homogeneous MC

P(t-s)\ =\ P(r-s)P(t-r)

or, on setting $r-s=\alpha$ and $t-r=\beta$ ,

P(\alpha+\beta)=P(\alpha)P(\beta).

From now on, unless otherwise stated, we shall assume that the Markov chain is homogeneous.

Theorem 5.1.6.

Let the row vector $\pi(t)$ hold the pmf of $X(t)$ , i.e. $\pi(t)_{i}=\mathrm{P}(X(t)=i)$ . Then for any $s<t$

\pi(t)=\pi(s)P(t-s).

Proof.

An important point is that both $\pi(s)$ and $\pi(t)$ depend on some initial distribution, $\pi(0)$ : by the theorem of total probability,

\pi(t)_{i}=\mathrm{P}(X(t)=i)=\sum_{k}\mathrm{P}(X(0)=k)\mathrm{P}(X(t)=i|X(0)% =k)=\sum_{k}\pi(0)_{k}P(t)_{k,i},

so $\pi(t)=\pi(0)P(t)$ and similarly $\pi(s)=\pi(0)P(s)$ . The proof follows by multiplying the C-K equations by $\pi(0)$ to get

\pi(0)P(t)=\pi(0)P(s)P(t-s),

and then substituting the above expressions for $\pi(t)$ and $\pi(s)$ . ∎