Markov Processes-III

Presented by:

Outline

• Review of steady-state behavior• Probability of blocked phone calls• Calculating absorption probabilities• Calculating expected time to absorption

Preview

• Assume a single class of recurrent states, a-periodic:• Plus transient states, then

• Where Does not depend on the initial conditions

jijnn r )()lim(

j

jnijn xx )|()lim(

0

Preview

• Can be found as the unique solution to the balance equations

• Together with

m1

mjk

kjkj P 1,

j

j1

Example

1 2

7/5,7/221

5.0 5.0 8.0

2.0

Example

• Assume that process starts as state 1

)99()1,1(11111001 rPxx andP

prxx andP1211101100

)100()21(

The Phone Company Problem

• Calls originate as a poison process, rate Each call duration is exponentially distributed(parameter B lines available• Discrete time intervals of( small) length

The Phone Company Problem

ii

i

EquationsBalance

1

:

B

i

iiii

iii

000

!//1!/

i

Steady State Probability: Example#1

• Consider a Markov chain with given transition probabilities and with a single recurrent class which is a-periodic

• Assume that for the n-step transition probabilities are very close to steady state probabilities

• Find

500n

lJKJjkij PPrXXXX ilkJP

)1000(

)|,,(0200010011000

Steady State Probability: Example#1

• B) • Solution:• By using Bay’s Rule:

?/(10011000

jiP XX

jijiP

XXXXX jPjiPjiP

/

)(/),()/(10011001100010011000

Steady State Probability: Example#2

• An absent minded professor has two umbrellas that she uses when coming from home to office and back. If it rains and umbrella is available in her location, she takes it. If it is not raining, she always forget to take an umbrella. Suppose it rains with probability ‘P’ each time comes, independently of other times, what is the steady state probability that she wet during a rain?

Steady State Probability: Example#2

• Markov mode with following states:• State ‘i’ where i=0,1,2• ‘i’ umbrellas are available in current location• Transition probability matrix:

01

10

100

pp

pp

Steady State Probability: Example#2

• The chain has single recurrent class which is a-periodic

• So, steady state convergence theorem applies.

0 2 1 p1

p1 p

p1

Steady State Probability: Example#2

• Balance Equations are as below:

1

)1(

)1(

021

102

211

20

p

pp

p

Steady State Probability Example#2

• After Solving:

• So, the steady state probability that she gets wet is times the probability of rain=

)3(/1

)3(/1

)3(/)1(

1

1

0

p

p

pp

0p 0

Calculating Absorption Probabilities

• What is the probability that: process eventually settles in state 4, given that the initial state is i?

ai

3

2

5

4

1

1 12.0

2.0

3.0

4.0

5.0

6.0

8.0

Calculating Absorption Probabilities

SolutionuniqueiotherallFor

iFor

iFor

apa

aa

jj

iji

i

i

0,5

1,4

Expected time to absorption

• Find expected number of transitions until reaching the Absorbing state, given that the initial state is i?

3

2

4

1

1

2.0

5.0

4.0

5.0

6.0

8.0

i

Expected Time to Absorption

SolutionuniqueiotherallFor

ifor

jjiji

i

p

1:

40

Absorption ProbabilitiesExample#1

• Consider the Markov Chain

Mean First Passage and Recurrence Times

• Chain with one recurrent class;• Fix s recurrent• Mean first passage time from s to i

jjiji

s

m

ni

siallfor

tosolutionuniquetheare

isthatsuchnE

tptt

tttXXt

1

0

,.

]|}0[min{

21

0

Mean First Passage and Recurrence Times

• Mean recurrence time of s:

p

ssthatsuchnE

ajj js

ns

ttXXt

1

]|}1[min{0