Markov processes - I
• checkout counter example
• Markov process definition V
• n-step transition probabilities
• classification of states V
1
•••
• •
checkout counter example
1 , ...
• customer arrivals: Bernouilli (p)
• customer service times: geometric (q)
ers at time nXn : nu it -+ I .,.-f(1 --;)
•
o I 3
11',11'11," ----- /1 "7 ILl41':J' -f,"_e....
11',. ,1-, 1111' ---- - 11 -,. ILl4<f':J' """_e....
X~..::~ XI:'~ XL-=;.t;' 'X:. ::: 3 XCI =3-' =.e
9
___ =-t", ___ _ X~~.8 t<"~=.e*l-I
-/•
2
• •
discrete-time finite state Markov chains
,..,.. rn
~ ti'fi4U 0;. """ 'I
•
••
- belongs to - initial given or random
transition ~11 6::>
Pij = P (Xl = jI X o = i ) J ..o~.._S ;;;--- = P (Xn+l = j I X n = i ) •+f~=~
• Markov property/assumption: .J "given current state, the past doesn't matter"
Pij l'I>
P (Xn+l = j I X n = i )
= P (Xn+1 = j I X n = i, X n- l ,· · ·, XO)
• model specification: identify states. transitions, and transition probabilities
3
Time 0 Time n- I Time n n-step trlns~~qn.probabilities
Ir~(o)" o~;,y ~jCl)= f1j tf.l" ~. • state probabilities, given inj;ial state i:
,,' (n) = P(Xn = j I Xo = il ? r":;C..1.=1. 'J ''''5 ,~..s ~I ,I... 11'.,
,
Pkj= P(Xn-& j I "0= i) T .... /
Pmj
on
'" .tefi
• key recursion: ,';; (n) •
•
4
1
(.,(,.)= (i,("'-I).O.5+ flc.(1I-/)"O.eexample
r;~(",)= I-ro,(...)
o. 0.>0.5
2
0.2
riln)= P Xn j I XO=i)
r ll
r2 1
n = O n =l n =2 n = 100 n
I O.~ ? •
0 C.'; 0.'5 ? ~+-•
0 O.t. ~ ~V~ • •
I 10.8 Z 5/~ 5
generic convergence questions
• does Tij (n) converge to something?
I
0 .5 0 .5
? 3-n o dd : T22(n) = 0
n even: T22(n) = I I I
• does the limit depend on initial state?
Tll (n) = I0.4 T3 1(n) = 0
T2 1(n) = Yt0 .3 6
• I -
6
.
)--~ 4 '-
I
recurrent and transient states
• state i is recurrent if "starting from i, and from wherever you can go, there is a way of returning to i"
• if not recurrent, called transient c.\cAss
7
MIT OpenCourseWarehttps://ocw.mit.edu
Resource: Introduction to ProbabilityJohn Tsitsiklis and Patrick Jaillet
The following may not correspond to a p articular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource.
For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.