STA 348Introduction to

Stochastic Processes

Lecture 19

1

CMC Transition Probability Function

CMC described by two sets of quantities: vi : exponential rate of leaving state i Pij : probability of going from state i to state j,

after leaving state i Based on these, want to find probability of

going from i to j after some time t, called the transition probability function Pij(t)

2

( ) ( ) | (0)

( ) | ( ) ,ijP t P X t j X i

P X t s j X s i s

CMC Transition Probability Function

3

(time)0

i

j

t

( )ijP tProbability of being in state jat time t, starting from i :

● What is ( ) ?ijjP t

Pure Birth Process Transition Probability Function

For pure birth process, transition probability function is straightforward to calculate: Birth rates λi=vi , death rates µi=0 → Pi,i+1=1 Let Ti be the iid Exp(λi) time it takes for process to

go from state i to i+1

4

0

i1i

2i

(time)

iT 1iT

Pure Birth Process Transition Probability Function

Starting from state i, process will be in some state ≤ j ( j ≥ i) at time t, only if there are less than j−i # transitions between time [0,t] Thus, , and from

this we can readily find Pij(t)

5

( ) | (0) jkk i

P X t j X i P T t

0

i

j

(time)

iTjT1iT

t

…

Example

Find the transition probability function for a Poisson process with rate λ

6

Chapman-KolmogorovEquations

For general CMC, need to solve a set of differential equations to find Pij(t)

Start with Chapman-Kolmogorov equations

Proof:

7

( ) ( ) ( ) , , & , 0ij ik kjkP t s P t P s i j s t

Instantaneous Transition Rates

Define quantities qij, called the instantaneous transition rates of the CMC, as They represent the rate at which the process

switches states over a small (~0) period of time For any instantaneous rates qij, we have

→ rates uniquely determine the CMC8

, ,ij i ijq v P i j

ij i ij ij jq v P v

ij i ijij

ij ij

q v PP

q v

Instantaneous Transition Rates

We can show (somewhat informally) that

9

0

1 ( )lim iiih

P h vh

0

( )lim ij

ijh

P hq

h

10

Kolmogorov’s Backward Equations

For all states i, j and times t ≥ 0, we have

Proof:

11

( ) ( ) ( )ij ij kj i ijk iP t q P t v P t

( )

( )where ijij

dP tP t

dt

Example

Find backward eqn’s of Birth & Death process

12

Example

Machine works for Exponential(λ) time until it breaks down, and it takes Exponential(µ) time to fix it. If machine is working at time 0, find probability it will be working at time 10.

13

Example (cont’d)

14

Example (cont’d)

15

Kolmogorov’s Forward Equations

For all states i, j and times t ≥ 0, we have

Note: Forward equations don’t hold for all CMC’s, but dohold for all Birth & Death and finite state-space CMC’s

Proof:

16

( ) ( ) ( )ij ik kj ij jk jP t P t q P t v

Example

Find Pij(t) for machine with Exp(λ) work time & Exp(µ) repair time using forward eqn’s

17

Example

Find forward eqn’s of Birth & Death process

18

Example

Find forward eqn’s for pure birth process,

and show that

19

1 , 10

( ) , 0

( ) ( ) , 1

i

j j

tii

tt sij j i j

P t e i

P t e e P s ds j i

Example (cont’d)

20