a brief review

A brief review

A continuous random variable X is said to have an

with parameter , 0, if its probability density

function is given by

0 ( )

0,

xe xf x

exponential

distribution

0

0

1 0 ( ) ( )

0, 0

1( ) ( )

xx

x

x

e xF x f y dy

x

E X xf x dx x e dx

The exponential distribution

A random variable is said to be memoryless if

( | ) ( ) for all , 0

( , ) ( )( )

( ) ( )

( ) ( ) ( )

P X s t X t P X s s t

P X s t X t P X s tP X s

P X t P X t

P X s t P X t P X s

The memoryless property

( )

If has the exponential distrbution, then

( ) ( ) ( )s t s t

X

P X s t e e e P X t P X s

Exponentially distributed random variables arememoryless.

The exponential distribution is the only distribution that has the memoryless property.

1 2

1 2 1 2

1 2

(min( , , ..., ) ) {( ), ( ), ..., ( )}

{( )} {( )} ... {( )}

...

n n

n

x x

P X X X x P X x X x X x

P X x P X x P X x

e e e

1 2( ... )

1 2

1 2

The distribution of the random variable (min( , , ..., )

1is exponentially distributed with mean .

...

N

N

x

x

n

n

e

P X X X

Suppose that X1, X2, ..., Xn are independent exponential random variables, with Xi having rate i, i=1, ..., n.What is P(min(X1, X2, ..., Xn )>x)?

The minimum of n exponentially distributed random variables

1

1 1

2 1 1 2

1 2 1 2 10

1 2 1 1 2 10 0

( )1 10 0

1

1 2

( ) ( | ) ( )

( | ) ( )

.

X

x x

x x x

P X X P X X X x f x dx

P X X X x e dx P x X e dx

e e dx e dx

Suppose that X1 and X2 are independent exponentially distributed random variables with rates 1 and 2.What is P(X1 < X2)?

Comparing two exponentially distributed random variables

The Poisson process

The counting process {N(t) t ≥ 0} is said to be a Poisson process having rate , > 0, if

(i) N(0) = 0.(ii) The process has independent increments.(iii) The number of events in any interval of length t is Poisson distributed with mean t. That is for all s, t ≥ 0

( ){ ( ) ( ) } , for 0,1,...

!

nt t

P N t s N t n e nn

Let Tn denote the inter-arrival time between the (n-1)th event and the nth event of a Poisson process, then the Tn (n=1, 2, ...) are independent, identically distributed exponential random variables having mean 1/.

The distribution of interarrival times for a Poisson process

Continuous Time Markov Chains (CTMC)

• A CTMC is a continuous time analog to a discrete time Markov chain

• A CTMC is defined with respect to a continuous time stochastic process {X(t): t ≥0}

• If X(t) = i the process is said to be in state i at time t

• {i: i=0, 1, 2, ...} is the state space

A stochastic process {X(t): t ≥0} is a continuous time Markov chain if for all s, t , u ≥0 and 0 ≤ u < s

P{X(t+s)=j|X(s)=i, X(u)=x(u), 0 ≤ u < s} = P{X(t+s)=j|X(s)=i}

A CTMC is said to have stationary transition probabilities if

P{X(t+s)=j|X(s)=i} is independent of s (and depends only on t).



Pij(t) = P{X(t+s)=j|X(s)=i}



Pij(t) = P{X(t+s)=j|X(s)=i}

Note: We shall always assume that the stationary property holds

• Ti: time the process spends in state i once it enters state i (the length of a visit to state i, a random variable).

Sojourn times


Example 1: X(0)=2, the first three transitions occur at t1 =3, t1 = 4.2 and t1 = 6.5 and X(3)=4, X(4.2) = 2 and X(6.5) =1.

Sojourn times


Example 1: X(0)=2, the first three transitions occur at t1 =3, t2 = 4.2 and t3 = 6.5 and X(3)=4, X(4.2) = 2 and X(6.5) =1.

The first sojourn time in state 4 = 4.2-3 = 1.2 The second sojourn time in state 2 = 6.5 – 4.2 = 1.3

Sojourn times

Example 2: Suppose the process has been in state 3 for 10 minutes, what is the probability that it will not leave state 3 in the next 5 minutes.

Example 2: Suppose the process has been in state 3 for 10 minutes, what is the probability that it will not leave state 3 in the next 5 minutes.

P(T3 > 15|T3> 10) = P(T3 > 5)

More generally,

P(Ti > s+t|Ti> s) = P(Ti > t)

Ti is memoryless and therefore has the exponential distribution

A CTMC is a stochastic process having the properties that each time it enters a state i

(i) the amount of time it spends in that state before making a transition into a different state is exponentially distributed with some mean 1/vi (or transition rate vi )

(ii) when the process leaves state i, it next enters state j with some probability Pij (Pii =0 and jPij =1, for all i)

An alternative definition of a CTMC

A CTMC is a stochastic process that moves from state to state according to a probability transition matrix (similar to a discrete time Markov chain) , but the amount of time it spends in each state is exponentially distributed.

To define a CTMC, we need to define a state space, a probability transition matrix, and a set of transition rates.

Example 1: Customers arrive to a store according to a Poisson process with rate . Let N(t) be the total number of customers that have arrived by time t.

Example 1: Customers arrive to a store according to a Poisson process with rate . Let N(t) be the total number of customers that have arrived by time t.

State space is {0, 1, 2, ...}; Ti is exponentially distributed with mean 1/ Pij =1 if j=i+1 and Pij = 0 otherwise

Example 2: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes the single agent at the counter to check-in a customer is exponentially distributed with mean 1/.


State space is {0, 1, 2, ...} P0,1 = 1; Pi,i+1 = /(+); Pi,i-1 = /(+) T0 =/v0 =; Ti = /(+) vi =+ for i =1, 2, ...


State space is {0, 1, 2, ...} P0,1 = 1; Pi,i+1 = /(+); Pi,i-1 = /(+) T0 =/v0 =; Ti = /(+) vi =+ for i =1, 2, ...

The above is an example of an M/M/1 queue.

The M/M/1 queue is an example of a birth and death process.

Example 3: Customers arrive to a service center according to a Poisson process with rate n when there are n customers in the system. Customers take an amount Tn that is exponentially distributed with mean 1/n when there are n customers in the system.

Birth and death process

Example 3: Customers arrive to a service center according to a Poisson process with rate n when there are n customers in the system. Customers take an amount Tn that is exponentially distributed with mean 1/n when there are n customers in the system.

State space is {0, 1, 2, ...} P0,1 = 1; Pn,n+1=n/(n+n); Pn,n-1=n/(n+n) T0 =/v0 =; Tn = /(n +n) vn =n +n for n =1, 2, ...

Birth and death process

State transition diagrams for a B&D process

10 2

3

• The Poisson process is a birth and death process with rate n= and n=.

• The M/M/1 queue is described by a birth and death process with rate n= and n=.

Example 4: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes one of the m agents at the counter to check-in a customer is exponentially distributed with mean 1/.


The system can be modeled as a birth and death process with transition rates n = ;n = n if 1 ≤ n < m n = m if n ≥ m


The system can be modeled as a birth and death process with transition rates n = ;n = n if 1 ≤ n < m n = m if n ≥ m

The above is an example of an M/M/m queue.

vi: rate with which the process leaves state i (once it enters state i)

qij: rate with which the process goes state j (once it enters state i) qij = Pijvi

(qij is also called the instantaneous transition rate from state i to j)

Transition rates




vij viPij = j qij

Pijqij/viqij/j qij

Transition rates




vij viPij = j qij

Pijqij/viqij/j qij

Specifying the instantaneous transition rates determines the parameters of the CTMC

Transition rates

State transition diagrams

10 2

q0,2

q2,1

q1,2q0,1

q2,0

q2,0

0

0

1 ( )lim

( )lim , for

iii

h

ijij

h

P hv

h

P hq i j

h

It can also be shown that

Properties

0 For all , 0, ( ) ( ) ( ).ij ik kjk

s t P t s P t P s

The Chapman-Kolmogrov equations

0 For all , 0, ( ) ( ) ( ).

( ) ( ( ) | (0) )

ij ik kjk

ij

s t P t s P t P s

P t s P X t s j X i

Proof :


0

0

For all , 0, ( ) ( ) ( ).

( ) ( ( ) | (0) )

= ( ( ) , ( ) | (0) )

ij ik kjk

ij

k

s t P t s P t P s

P t s P X t s j X i

P X t s j X t k X i

Proof :


0

0

0

For all , 0, ( ) ( ) ( ).

( ) ( ( ) | (0) )

= ( ( ) , ( ) | (0) )

= ( ( ) | ( ) , (0) ) (

ij ik kjk

ij

k

k

s t P t s P t P s

P t s P X t s j X i

P X t s j X t k X i

P X t s j X t k X i P

Proof :

( ) | (0) )

X t k X i


0

0

0

For all , 0, ( ) ( ) ( ).

( ) ( ( ) | (0) )

= ( ( ) , ( ) | (0) )

= ( ( ) | ( ) , (0) ) (

ij ik kjk

ij

k

k

s t P t s P t P s

P t s P X t s j X i

P X t s j X t k X i


Proof :

0

( ) | (0) )

= ( ( ) | ( ) ) ( ( ) | (0) )

k

X t k X i

P X t s j X t k P X t k X i


0

0

0

For all , 0, ( ) ( ) ( ).

( ) ( ( ) | (0) )

= ( ( ) , ( ) | (0) )

= ( ( ) | ( ) , (0) ) (

ij ik kjk

ij

k

k

s t P t s P t P s

P t s P X t s j X i

P X t s j X t k X i


Proof :

0

0

( ) | (0) )

= ( ( ) | ( ) ) ( ( ) | (0) )

= ( ) ( )

k

kj ikk

X t k X i

P X t s j X t k P X t k X i

P s P t


'

0

For all states , and time 0, ( ) ( ) ( )

( ) ( )= ( ) ( ) ( )

( ) ( ) [1 ( )] ( )

ij ik kj i ijk i

ij ij ik kj ijk

ik kj ii ijk i

i j t P t q P t v P t

P h t P t P h P t P t

P h P t P h P t

Proof :

Kolmogrov’s backward equations

'

0

0


( ) ( )= ( ) ( ) ( )

( ) ( ) [1 ( )] ( )

( ) ( )lim

ij ik kj i ijk i

ij ij ik kj ijk

ik kj ii ijk i

ij ij

h



P h P t P h P t

P h t P t

h

Proof :

0

( ) 1 ( )lim ( ) [ ] ( )ik ii

kj ijk ih

P h P hP t P t

h h


'

0

0


( ) ( )= ( ) ( ) ( )

( ) ( ) [1 ( )] ( )

( ) ( )lim

ij ik kj i ijk i

ij ij ik kj ijk

ik kj ii ijk i

ij ij

h



P h P t P h P t

P h t P t

h

Proof :

0

'

( ) 1 ( )lim ( ) [ ] ( )

( ) ( ) ( )

ik iikj ijk ih

ij ik kj i ijk i

P h P hP t P t

h h

P t q P t v P t


'Under suitable regularity conditions, ( ) ( ) ( )

for all , and time 0.

ij kj ik j ijk jP t q P t v P t

i j t

Kolmogrov’s forward equations

'

'

lim ( )

lim ( ) lim ( ) ( )

0

j ijt

ij kj ik j ijk jtt

kj k j jk j

kj k j j j j kj kk j k j

P P t

P t q P t v P t

q P v P

q P v P v P q P

Limiting probabilities

The limiting probabilities can be obtained by solving the following

system of equations:

0

1

kj k j jk j

jj

q P v P

P

rate at which the process leaves state

rate at which the process enters state

rate out of state rate into state

j j

kj kk j

j j kj kk j

v P j

q P j

v P q P

j j

The limiting probabilities Pj exist if

(a) all states of the Markov chain communicate (i.e., starting in state i, there is a positive probability of ever being in state j, for all i, j and

(b) the Markov is positive recurrent (i.e, starting in any state, the mean time to return to that state is finite).

When do the limiting probabilities exist?

The M/M/1 queue

10 2

3

0 1

1 2 0

State rate out of state rate into state

0

1 ( )

2

j j

P P

P P P

2 3 1

1 1

( )

1 ( ) n n n

P P P

n P P P

The M/M/1 queue

0 1

1 2

2 3

By adding to each equation the equation preceding it, we obtain

0

1

2

P P

P P

P P

n

11 n nP P

The M/M/1 queue

0

1 0

22 0

33 3

Solving in terms of yields

0 ( / )

1 ( / )

2 ( / )

1 ( /n

P

P P

P P

P P

n P

0)n P

The M/M/1 queue

0

0 0 1

0

1

Using the fact that 1,we obtain

( / ) 1

11

1 ( / )

/ (1 / )

Note that we must have / 1.

nn

n

n

n

n

n

n

P

P P

P

P

a brief review

Documents

state i

independent of s

time t

pti s tti s

discrete time markov

continuous time analog

interarrival time

pti t ti