CS433Modeling and Simulation

Lecture 06 – Part 01

Discrete Markov Chains

Dr. Anis Koubâa12 Apr 2009

Al-Imam Mohammad Ibn Saud UniversityAl-Imam Mohammad Ibn Saud University

22

Goals for Today

Understand what is a Stochastic

Process

Understand the Markov property

Learn how to use Markov Chains

for modelling stochastic processes

33

The overall picture …

Markov Process Discrete Time Markov Chains

Homogeneous and non-homogeneous Markov chains

Transient and steady state Markov chains Continuous Time Markov Chains

Homogeneous and non-homogeneous Markov chains

Transient and steady state Markov chains

4

• Stochastic Process• Markov Property

Markov Process

5

5

What is “Discrete Time”?

0 1 2 3 4

time

Events occur at a specific points in time

6

6

What is “Stochastic Process”?

Day 1

Day

Day 2 Day 3 Day 4 Day 5 Day 6 Day 7

THU FRI SAT SUN MON TUE WED

X(dayi): Status of the weather observed each DAY

State Space = {SUNNY, RAINNY}

" "1X Sday " "3X Rday

" "2X Sday

" "5X Rday

" "4X Sday

" "7X Sday

" "6X Sday

" " or " " : RANDOM VARIABLE that varies with the DAYiX S Rday

IS A STOCHASTIC PROCESSiX day

7

7

Markov Processes

Stochastic Process X(t) is a random variable that varies with time.

A state of the process is a possible value of X(t) Markov Process

The future of a process does not depend on its past, only on its present

a Markov process is a stochastic (random) process in which the probability distribution of the current value is conditionally independent of the series of past value, a characteristic called the Markov property.

Markov property: the conditional probability distribution of future states of the process, given the present state and all past states, depends only upon the present state and not on any past states

Marko Chain: is a discrete-time stochastic process with

the Markov property

8

8

What is “Markov Property”?

Day 1

Day

Day 2 Day 3 Day 4 Day 5 Day 6 Day 7

THU FRI SAT SUN MON TUE WED

" "1X Sday " "3X Rday

" "2X Sday

" "5X Rday

" "4X Sday

NOW FUTURE EVENTSPAST EVENTS

Markov Property: The probability that it will be (FUTURE) SUNNY in DAY 6given that it is RAINNY in DAY 5 (NOW) is independent from PAST EVENTS

?Probability of “R” in DAY6 given all previous states Probability of “S” in DAY6 given all previous states

6 5 4 1

6 5

Pr " " | " ", " ",..., " "

Pr " " | " "

DAY DAY DAY DAY

DAY DAY

X S X R X S X S

X S X R

9

9

Notation

X(tk) or Xk = xk

The stochastic process at time tk or k

Discrete time tk or k Value of the stochasticprocess at instant tk or k

10

Discrete Time Markov Chains (DTMC)

Markov Chain

11

11

Markov Processes

Markov Process The future of a process does not depend on its past,

only on its present

1 0 01

11

Pr | ,...,

Pr |

k kk k

k kk k

X x X x X t xt t

X x X xt t

Since we are dealing with “chains”, X(ti) = Xi can take discrete values from a finite or a countable infinite set.

The possible values of Xi form a countable set S called the state space of the chain

For a Discrete-Time Markov Chain (DTMC), the notation is also simplified to

1 1 0 0 1 1Pr | ,..., Pr |k k k k k k k kX x X x X x X x X x

Where Xk is the value of the state at the kth step

12

12

General Model of a Markov Chain

S0 S1 S2

p01

p11

p12

p00p10

p21

p20

p22

Si State i

pijTransition Probability from State Si to State Sj

State Space 0, 1, 2S S S S

ior

Discrete Time (Slotted Time) 0 1 2, , ,...,

{0,1,2,..., }ktime t t t t

k

13

13

Example of a Markov ProcessA very simple weather model

SUNNY RAINY

pSR=0.3

pRR=0.4pSS=0.7

pRS=0.6

State Space ,S SUNNY RAINY

If today is Sunny, What is the probability that to have a SUNNY weather after 1 week?

If today is rainy, what is the probability to stay rainy for 3 days?

Problem: Determine the transition probabilities from Problem: Determine the transition probabilities from

one state to another after one state to another after nn events. events.

14

Five Minutes Break

You are free to discuss with your classmates about the previous slides, or to refresh a bit, or to ask questions.

15

Determine transition probabilities from one state to anothe after n events.

Chapman Kolmogorov Equation

16

16

Chapman-Kolmogorov Equations

We define the one-step transition probabilities at the instant k as 1Pr |ij k kp X j X ik

Necessary Condition: for all states i, instants k, and all feasible transitions from state i we have:

1 is all neighbor states to ijj i

p where i ik

xi

x1

xR

… xj

k u k+n

Pr |,ij k n kp X j X ik k n

We define the n-step transition probabilities from instant k to k+n as

Discrete time k+1

17

17

Chapman-Kolmogorov Equations

1

Pr |,

Pr | , Pr |

ij k n k

R

k n u k u kr

p X j X ik k n

X j X r X i X r X i

Using Law of Total Probability

xi

x1

xR

… xj

k u k+nDiscrete time k+1

18

18

1

, , , ,R

ij ir rjr

p p p k u k nk k n k u u k n

Chapman-Kolmogorov Equations

Pr | , Pr |k n u k k n uX j X r X i X j X r

Using the memoryless property of Markov chains

Therefore, we obtain the Chapman-Kolmogorov Equation

1

Pr |,

Pr | Pr |

ij k n k

R

k n u u kr

p X j X ik k n

X j X r X r X i

19

19

Chapman-Kolmogorov EquationsExample on the simple weather model

1, 3 1, 2 2, 3

+ 1, 2 2, 3

sunny rainy sunny sunny sunny rainy

sunny rainy rainy rainy

p p pday day day say day day

p pday day day day

What is the probability that the weather is rainy on day 3 knowing that it is sunny on day 1?

1, 3

0.7 0.3 0.3 0.4 0.21 0.12 0.33

sunny rainy ss sr sr rrp p p p pday day

+1, 3 1, 2 2, 3 1, 2 2, 3sunny rainy ss sr sr rrp p p p pday day day say day day day day day day

SUNNY RAINY

pSR=0.3

pRR=0.4pSS=0.7

pRS=0.6

20

Generalization Chapman-Kolmogorov Equations

Transition Matrix

21

21

Transition Matrix Simplify the transition probability representation

Define the n-step transition matrix as

We can re-write the Chapman-Kolmogorov Equation as follows:

Choose, u = k+n-1, then

,, ijp k k nk k n H

, , ,k k n k u u k n H H H

, , 1 1,

, 1 1

k k n k k n k n k n

k k n k n

H H H

H P

One step transition probability

Forward Chapman-Kolmogorov

22

22

Transition Matrix Simplify the transition probability representation

Choose, u = k+1, then

, , 1 1,

1,

k k n k k k k n

k k nk

H H H

P H

One step transition probability

Backward Chapman-Kolmogorov

23

23

Transition Matrix Example on the simple weather model

What is the probability that the weather is rainy on day 3 knowing that it is sunny on day 1?

SUNNY RAINY

pSR=0.3

pRR=0.4pSS=0.7

pRS=0.6

1, 3 1, 31, 3

1, 3 1, 3sunny sunny sunny rainy

rainy sunny rainy rainy

p pday day day dayday day

p pday day day day

H

24

24

Homogeneous Markov Chains Markov chains with time-homogeneous transition probabilities

The one-step transition probabilities are independent of time k.

1 or Pr |ij k kp X j X ik P P

Even though the one step transition is independent of k, this does not mean that the joint probability of Xk+1 and Xk is also independent of k. Observe that:

1 1Pr and Pr | Pr

Pr

k k k k k

ij k

X j X i X j X i X i

p X i

Time-homogeneous Markov chains (or, Markov chains with time-homogeneous transition probabilities) are processes where 1 1Pr | Pr |ij k k k kp X j X i X j X i

1Pr |ij k kp X j X i is said to be Stationary Transition Probability

25

Two Minutes Break

You are free to discuss with your classmates about the previous slides, or to refresh a bit, or to ask questions.

26

Example: Two Processors System

Consider a two processor computer system where, time is divided into time slots and that operates as follows: At most one job can arrive during any time slot and this can happen

with probability α. Jobs are served by whichever processor is available, and if both are

available then the job is given to processor 1. If both processors are busy, then the job is lost. When a processor is busy, it can complete the job with probability β

during any one time slot. If a job is submitted during a slot when both processors are busy but

at least one processor completes a job, then the job is accepted (departures occur before arrivals).

Q1. Describe the automaton that models this system (not included).

Q2. Describe the Markov Chain that describes this model.

27

Example: Automaton (not included) Let the number of jobs that are currently processed by the

system by the state, then the State Space is given by X= {0, 1, 2}.

Event set: a: job arrival, d: job departure

Feasible event set: If X=0, then Γ(X)= a If X= 1, 2, then Γ(Χ)= a, d.

State Transition Diagram

0 1 2

a

- / a,d

a

-d d / a,d,d

dd

-/a/ad

28

Example: Alternative Automaton(not included) Let (X1,X2) indicate whether processor 1 or 2 are busy, Xi= {0, 1}. Event set:

a: job arrival, di: job departure from processor i Feasible event set:

If X=(0,0), then Γ(X)= a If X=(0,1) then Γ(Χ)= a, d2. If X=(1,0) then Γ(Χ)= a, d1. If X=(0,1) then Γ(Χ)= a, d1, d2.

State Transition Diagram

a

- / a,d1

a

-

d2 d1

d1,d2

-/a/ad1/ad2

-

d1

a,d2

a,d1,d2

00

10

11

01

29

29

Example: Markov Chain

For the State Transition Diagram of the Markov Chain, each transition is simply marked with the transition probability

0 1 2

p01

p11

p12

p00p10

p21

p20

p22

00 1p 01p 02 0p

10 1p 11 1 1p 12 1p

220 1p 2

21 2 1 1p 2

22 21 1p

30

30

Example: Markov Chain

Suppose that α = 0.5 and β = 0.7, then,

0 1 2

p01

p11

p12

p00p10

p21

p20

p22

0.5 0.5 0

0.35 0.5 0.15

0.245 0.455 0.3ijp

P

31

How much time does it take for going from one state to another?

State Holding Time

32

32

State Holding Times

Suppose that at point k, the Markov Chain has transitioned into state Xk=i. An interesting question is how long it will stay at

state i. Let V(i) be the random variable that represents the number of

time slots that Xk=i.

We are interested on the quantity Pr{V(i) = n}

1 1

1

1 1

Pr Pr , ,..., |

Pr | ,...,

Pr ,..., |

k n k n k k

k n k n k

k n k k

V n X i X i X i X ii

X i X i X i

X i X i X i

1 1 2

2 1

Pr | Pr | ...,

Pr ,..., |

k n k n k n k n k

k n k k

X i X i X i X X i

X i X i X i

| | |P A B C P A B C P B C

33

33

State Holding Times

This is the Geometric Distribution with parameter Clearly, V(i) has the memoryless property

1

1 2

2 1

Pr Pr |

Pr | ...,

Pr ,..., |

k n k n

k n k n k

k n k k

V n X i X ii

X i X X i

X i X i X i

1Pr 1 nii iiV n p pi

1 2

2 3

3 1

1 Pr |

Pr | ,...,

Pr ,..., |

ii k n k n

k n k n k

k n k k

p X i X i

X i X i X i

X i X i X i

iip

34

34

State Probabilities

An interesting quantity we are usually interested in is the probability of finding the chain at various states, i.e., we define

Pri kX ik For all possible states, we define the vector

0 1, ...k k k π Using total probability we can write

1 1Pr | Pr

1

i k k kj

ij jj

X i X j X jk

p k k

In vector form, one can write

1k k k π π P 1k k π π POr, if homogeneous Markov Chain

35

35

State Probabilities Example

Suppose that

1 0 00 π

Find π(k) for k=1,2,…

with0.5 0.5 0

0.35 0.5 0.15

0.245 0.455 0.3

P

0.5 0.5 0

1 0 0 0.35 0.5 0.15 0.5 0.5 010.245 0.455 0.3

π

Transient behavior of the system In general, the transient behavior is obtained by

solving the difference equation 1k k π π P