STOCHASTIC MODELS LECTURE 3

CONTINUOUS-TIME MARKOV PROCESSES

Nan ChenMSc Program in Financial EngineeringThe Chinese University of Hong Kong

(ShenZhen)Oct 14, 2015

Outline1. Introduction of Continuous-

Time Processes2. Limiting Probabilities

3.1 INTRODUCTION

A New Perspective on Poisson Process

• A Poisson process can be constructed as follows: – At each state it will stay for an exponential

time with mean – Then, it proceeds from to

Continuous-Time Markov Chains

• We may follow a similar way to construct a continuous-time Markov chains on a state space : – Each time when it enters a state , we select a

random sojourn time independently of its history;

– After a time it will exit the state . It will enter state with probability

(Remark: We have a discrete-time Markov chain embedded in the process. )

Transition Probability Function

• Let record the state of the above Markov chain over time. The following quantity

is known as the transition probability function of the process.

• Obviously, we have

Kolmogorov Backward/Forward Equations• Theorem (Kolmogorov Backward Equation) For states and times

where

Kolmogorov Backward/Forward Equations (Continued)• Theorem (Kolmogorov Forward Equation) For states and times

Example I: Two-State Chain

• Consider a machine that works for an exponential amount of time having mean before breaking down; and suppose that it takes an exponential amount of time having mean to repair the machine.

• If the machine is in working condition at time 0, then what is the probability that it will be working at time

Example I: Two-State Chain

• From the backward equations, we have

• From them, we can solve for

Computing Transition Probabilities

• For any pair of states and let

Then, we can rewrite the backward/forward equations as follows (backward) (forward)

Computing Transition Probabilities (Continued)• Introduce matrices by letting the

element in row , column of these matrices be, respectively,

The two equations can be written as (backward) (forward)• The solution should be

3.2 LIMITING PROBABILITIES

Limiting Probabilities

• In analogy with a basic result in discrete-time Markov chains, the probability that a continuous-time Markov chain will be in state at time often converges to a limiting value that is independent of the initial state.

• We intend to study

Limiting Probabilities (Continued)• To derive a set of equations for the , we

may let approach in the forward equation. Then,

In addition,

Limiting Probabilities (Continued)• It is easy to see that if the embedded Markov

chain has a limiting stationary distribution i.e.,

then

• The solution to the equations on the last slide should be

Example II: A Shoe Shine Shop• Consider a shoe shine establishment

consisting two chairs --- chair 1 and chair 2. A customer upon arrival goes initially to chair 1 where his shoes are cleaned and polish is applied. After this is done, the customer moves on to chair 2 where the polish is buffed.

• The service times at the two chairs are independent and exponentially distributed with respective rates and

Example II: A Shoe Shine Shop (Continued)• Suppose that potential customers arrive in

accordance with a Poisson process having rate , and that a potential customer will enter the system only if both chairs are empty.

• Let us define the state of the system as follows:– State 0: empty system– State 1: chair 1 is taken– State 2: chair 2 is taken

Example II: A Shoe Shine Shop (Continued)• What is the limiting probability for this

continuous-time Markov chain?

Homework Assignments

• Read Ross Chapter 6.1, 6.2, 6.4, 6.5, and 6.9.• Answer Questions:– Exercises 8 (Page 399, Ross)– Exercises 10, 13 (Page 400, Ross)– Exercises 14, 17 (Page 401, Ross)– Exercise 20 (Page 402, Ross)– (Optional, Extra Bonus) Exercise 48 (Page 407).