Markov processes - I

• checkout counter example

• Markov process definition V

• n-step transition probabilities

• classification of states V

1

•••

• •

checkout counter example

1 , ...

• customer arrivals: Bernouilli (p)

• customer service times: geometric (q)

ers at time nXn : nu it -+ I .,.-f(1 --;)

•

o I 3

11',11'11," ----- /1 "7 ILl41':J' -f,"_e....

11',. ,1-, 1111' ---- - 11 -,. ILl4<f':J' """_e....

X~..::~ XI:'~ XL-=;.t;' 'X:. ::: 3 XCI =3-' =.e

9

___ =-t", ___ _ X~~.8 t<"~=.e*l-I

-/•

2

• •

discrete-time finite state Markov chains

,..,.. rn

~ ti'fi4U 0;. """ 'I

•

••

- belongs to - initial given or random

transition ~11 6::>

Pij = P (Xl = jI X o = i ) J ..o~.._S ;;;--- = P (Xn+l = j I X n = i ) •+f~=~

• Markov property/assumption: .J "given current state, the past doesn't matter"

Pij l'I>

P (Xn+l = j I X n = i )

= P (Xn+1 = j I X n = i, X n- l ,· · ·, XO)

• model specification: identify states. transitions, and transition probabilities

3

Time 0 Time n- I Time n n-step trlns~~qn.probabilities

Ir~(o)" o~;,y ~jCl)= f1j tf.l" ~. • state probabilities, given inj;ial state i:

,,' (n) = P(Xn = j I Xo = il ? r":;C..1.=1. 'J ''''5 ,~..s ~I ,I... 11'.,

,

Pkj= P(Xn-& j I "0= i) T .... /

Pmj

on

'" .tefi

• key recursion: ,';; (n) •

•

4

1

(.,(,.)= (i,("'-I).O.5+ flc.(1I-/)"O.eexample

r;~(",)= I-ro,(...)

o. 0.>­0.5

2

0.2

riln)= P Xn j I XO=i)

r ll

r2 1

n = O n =l n =2 n = 100 n

I O.~ ? •

0 C.'; 0.'5 ? ~+-•

0 O.t. ~ ~V~ • •

I 10.8 Z 5/~ 5

generic convergence questions

• does Tij (n) converge to something?

I

0 .5 0 .5

? 3-n o dd : T22(n) = 0

n even: T22(n) = I I I

• does the limit depend on initial state?

Tll (n) = I0.4 T3 1(n) = 0

T2 1(n) = Yt0 .3 6

• I -

6

.

)--~ 4 '- ­

I

recurrent and transient states

• state i is recurrent if "starting from i, and from wherever you can go, there is a way of returning to i"

• if not recurrent, called transient c.\cAss

7

MIT OpenCourseWarehttps://ocw.mit.edu

Resource: Introduction to ProbabilityJohn Tsitsiklis and Patrick Jaillet

The following may not correspond to a p articular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource.

For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.