stochastic processes and markov chains

7/29/2019 Stochastic Processes and Markov Chains

1/40

Exercise 2:Stochastic Processes and Markov Chains

Roman Dunaytsev

Department of Communications EngineeringTampere University of Technology

[email protected]

November 05, 2009


2/40

Outline

1 Random variables

2 Stochastic processes

3 Markov chainsTransition matrices and probability vectorsRegular matricesProbability to get from one state to another in a given number ofstepsProbability distribution after several stepsLong-term behavior

Roman Dunaytsev (TUT) TLT-2716, Exercise 2 November 05, 2009 2 / 40


3/40

Outline

1 Random variables





4/40

Random Variables

Suppose that to each point of a sample space we assign a real number

We then have a real-valued function defined on the sample spaceThis function is called a random variable

It is usually denoted by a capital letter such as X

Example: Suppose that a coin is tossed twice so that the samplespace is = {TT,TH,HT,HH}

Let X represent the number of heads that can come up

Thus, we have:

X(TT) = 0, X(TH) = 1, X(HT) = 1, X(HH) = 2



5/40

Random Variables (contd)

Example: In an experiment involving 2 rolls of a die, the followingare examples of random variables:

The sum of the 2 rollsThe number of 6s in the 2 rollsThe second roll raised to the 5th power

Example: In an experiment involving the transmission of a message,

the following are examples of random variables:The number of symbols received in errorThe number of retransmissions required to get an error-free copyThe time needed to transmit the message

Example: You ask people whether they approve of the presentgovernment. The sample space could be:

{disapprove, indifferent, approve}

To analyze your results, you could use X = {1, 0, 1} or X = {1, 2, 3}



6/40

Random Variables (contd)

A discrete random variable maps events to values of a countable

set (e.g., the integers)

A continuous random variable maps events to values of anuncountable set (e.g., the real numbers)

A function of a random variable defines another random variable



7/40

Outline

1 Random variables





8/40

Stochastic Processes

A stochastic process is a collection of random variables

{S

(t

),tT}, where

tis a parameter that runs over an

index set T

In general, we call t the time-parameter

Each S(t) takes values in some set E called the state space

Then S(t) is the state of the process at time (or step) t

E.g., S(t) may be:The number of incoming emails at time tThe balance of a bank account on day tThe number of heads shown by t flips of a coin

Since any stochastic process is simply a collection of randomvariables, the name random process is also used for them



9/40

Stochastic Processes (contd)

Stochastic processes are classified in a number of ways, such as by

the index set and by the state space

When the index set is countable, the stochastic process is said to be adiscrete-time stochastic process

I.e., T = {0, 1, 2, . . . } or T = {0,1,2, . . . }

When the index set is an interval of the real line, the stochasticprocess is said to be a continuous-time stochastic process

I.e., T = {t : t 0} or T = {t : < t< }

When the state space is countable, the stochastic process is said tobe a discrete-space stochastic process

When the state space is an interval of the real line, the stochasticprocess is said to be a continuous-space stochastic process



10/40

Stochastic Processes (contd)

Income of a self-employedperson at day t

tt

S S

SS

t t

Income of an employee attime t in the course of year

Air temperature at noonover t days

Air temperature at time t

DT & DS CT & DS

CT & CSDT & CS



11/40

Outline

1 Random variables





12/40

Markov Chains and Markov Processes

A Markov chain , named after the Russian mathematician Andrey

Markov, is a stochastic process for which the future behavior onlydepends on the present and not on the past

A stochastic process has the Markov property (aka memorylessnessor one-step memory) if the likelihood of a given future state, at any

given moment, depends only on its present state, and not on any paststates

A process with this property is called Markovian or a Markov process

The classic example of a Markov chain is a frog sitting in a pond fullof lily pads

Each pad represents a stateThe frog starts on one of the pads and then jumps from lily pad to lilypad with the appropriate transition probabilities



13/40

Markov Chains

A Markov chain as a frog jumping on a set of lily pads



14/40

Markov Chains (contd)

A finite Markov chain is a stochastic process with a finite number

of states in which the probability of being in a particular state at stepn + 1 depends only on the state occupied at step n

Let S= {S1,S2, . . . , Sm} be the possible states

Let p(n) = (p1, p2, . . . , pm) be the vector of probabilities of each

state at step nHere pi, i= 1, 2, . . . ,m, is the probability that the process is in stateSi at step n

For such a probability vector, p1 + p2 + + pm = 1

A Markov process at time n is fully defined by

pij = Pr{S(n + 1) = j|S(n) = i}

Where pij is the conditional probability of being in state j at stepn + 1 given that the process was in state i at step n



15/40


Let P be the transition matrix (aka the stochastic matrix)

Then P contains all the conditional probabilities of the Markov chain

P=

p11 . . . p1m...

. . ....

pm1 . . . pmm

A vector p = (p1, p2, . . . , pm) is called a probability vector if thecomponents are nonnegative and their sum is 1

The entries in a probability vector can represent the probabilities offinding a system in each of the states

A square matrix P is called a transition matrix if each of its rows isa probability vector


( )


16/40


Example: Which of the following vectors are probability vectors?

(0.75, 0,0.25, 0.5), (0.75, 0.5, 0, 0.25), (0.25, 0.25, 0.5)

Example:Which of the following matrices are matrices of transitionprobabilities?

1/3 0 2/33/4 1/2 1/4

1/3 1/3 1/3

1/4 3/41/3 1/3

0 1 0

1/2 1/6 1/3

1/3 2/3 0


M k Ch i ( d)


17/40


Example: A meteorologist studying the weather in a region decidesto classify each day as {sunny} and {cloudy}After analyzing several years of weather records, he finds:

The day after a sunny day is sunny 80% of the time,and cloudy 20% of the timeThe day after a cloudy day is sunny 60% of the time,and cloudy 40% of the time

Thus, the transition matrix and the transition diagram are as follows:

P=

pss pscpcs pcc

=

0.8 0.20.6 0.4

0.2

0.6

0.8 0.4

sunny cloudy


M k Ch i ( d)


18/40


Example from Ryan ODonnells lecture: Every day I wake up andtry to get a lot of work done

When I am working, I am easily distracted though

After each minute of work, I only keep working with probability 0.4,and with probability 0.6, I begin surfing the Web

After each minute of surfing the Web, I only return to working withprobability 0.1, and with probability 0.6 I continue surfing the Web

With probability 0.3 I feel kind of guilty, so I check my email, which issort of like working

After each minute of email-checking, I have probability 0.5 of coming

back to work

With probability 0.5 I continue checking my email


M k Ch i ( d)


19/40


Missing arrows indicate zero probability

P=

pww pws pwepsw pss psepew pes pee

=

0.4 0.6 00.1 0.6 0.3

0.5 0 0.5


M k Ch i ( td)


20/40


Example: 3 boys, A, B and C, are throwing a ball to each other

A always throws the ball to B and B always throws the ball to CC is just as likely to throw the ball to B as to A

Find the transition matrix of the Markov chain

This is a Markov chain since the person throwing the ball is notinfluenced by those who previously had the ball

P=

paa pab pacpba pbb pbcpca pcb pcc

=

0 1 00 0 1

0.5 0.5 0

The first row of the matrix corresponds to the fact that A alwaysthrows the ball to B

The second row corresponds to the fact that B always throws it to C

The last row corresponds to the fact that C throws the ball to A or Bwith equal probability, and does not throw it to himself


M k Ch i ( td)


21/40

Markov Chains (cont d)

Let us consider a particle which moves in a straight line in unit steps

Each step is one unit to the right with probability p or one unit to theleft with probability q

It moves until it reaches one of two extreme points which are calledboundary points

We consider the case of 5 states: S1, S2, S3, S4, S5States S1 and S5 are the boundary statesThen states S2, S3, and S4 are interior states

S1 S2 S3 S4 S5

S1 p11 . . . . . . . . . p15S2 . . . . . . . . . . . . . . .S3 . . . . . . . . . . . . . . .S4 . . . . . . . . . . . . . . .S5 p51 . . . . . . . . . p55




22/40


Example: Let us assume that once state S1 or S5 is reached, theparticle remains there

P=

1 0 0 0 0q 0 p 0 00 q 0 p 00 0 q 0 p

0 0 0 0 1

A state Si of a Markov chain is called absorbing if it is impossible toleave it (i.e., pii = 1)

A Markov chain is absorbing if it has at least one absorbing state, andif from every state it is possible to go to an absorbing state (notnecessarily in one step)

In an absorbing Markov chain, a state which is not absorbing is called

transientRoman Dunaytsev (TUT) TLT-2716, Exercise 2 November 05, 2009 22 / 40



23/40


Example: Assume that the particle is reflected when it reaches aboundary point and returns to the point from which it came

Thus, if it reaches S1, it goes on the next step back to S2

If it hits S5, it goes on the next step back to S4

Find the matrix of transition probabilities

P=

0 1 0 0 0q 0 p 0 00 q 0 p 00 0 q 0 p

0 0 0 1 0

Since q+ p= 1, then q= 1 p




24/40


Example: Assume that whenever the particle hits one of theboundary states, it goes directly to the center state S3

We may think of this as the process started at state S3 and repeatedeach time the boundary is reached


P=

0 0 1 0 0q 0 p 0 00 q 0 p 00 0 q 0 p

0 0 1 0 0

q= 1 p




25/40


Example: Assume that once a boundary state is reached, the particlestays at this state with probability 0.5 and moves to the otherboundary state with probability 0.5


P=

0.5 0 0 0 0.5q 0 p 0 00 q 0 p 00 0 q 0 p

0.5 0 0 0 0.5

q= 1 p




26/40


A matrix P is called a stochastic matrix, if it does not contain anynegative entries and the sum of each row of the matrix is equal to 1

The product of 2 stochastic matrices is again a stochastic matrix

Therefore, all powers of stochastic matrices Pn are stochasticmatrices

A stochastic matrix P is said to be regular if all elements of at leastone particular power ofP are positive and different from 0


Matrix Multiplication


27/40

Matrix Multiplication

In order for matrix multiplication to be defined, the dimensions of thematrices must satisfy (nm)(m p) = (n p)

The product C of 2 matrices A and B is defined as

c1,1 c1,2 c1,3

c2,1 c2,2 c2,3

c3,1 c3,2 c3,3

a1,1 a1,2

a2,1 a2,2

a3,1 a3,2

b1,1 b1,2 b1,3

b2,1 b2,2 b2,3

A

x

C

=

B

That is

c1,2 = a1,1b1,2 + a1,2b2,2

c3,3 = a3,1b1,3 + a3,2b2,3




28/40


Example: Let A be a stochastic matrix:

A =a1,1 a1,2a2,1 a2,2

=

0 10.5 0.5

Let us calculate A2:

A2 =a1,1 a1,2a2,1 a2,2

a1,1 a1,2a2,1 a2,2

Thus, we get:

A2 =a1

,

1a1,

1 + a1,

2a2,

1 a1,

1a1,

2 + a1,

2a2,

2a2,1a1,1 + a2,2a2,1 a2,1a1,2 + a2,2a2,2

=

0.5 0.50.25 0.75

Hence, A is regular




29/40


The entry pij in the transition matrix P of a Markov chain is theprobability that the system changes from state Si to state Sj in 1 step

P=

p11 . . . p1m

.... . .

...pm1 . . . pmm

What is the probability that the system changes from state Si to stateSj in exactly n steps?

As a rule, this probability is denoted as p(n)

ij

The probability of going from any state Si to another state Sj in afinite Markov chain with the transition matrix P in n steps is given bythe element (i,j) of the matrix Pn




30/40


Example: The Land of Oz is blessed by many things, but not bygood weather:

They never have 2 nice days in a rowIf they have a nice day, they are just as likely to have snow as rain thenext dayIf they have snow or rain, they have an even chance of having the samethe next day

If there is change from snow or rain, only half of the time is this achange to a nice day

With this information let us form a Markov chain

Let us denote as states the kinds of weather: {rain}, {nice}, and{snow}

From the above information we determine the transition matrix:

P=

prr prn prspnr pnn pnspsr psn pss

=

1/2 1/4 1/41/2 0 1/2

1/4 1/4 1/2




31/40


We consider the question of determining the probability that, giventhe chain is in state Si today, it will be in state Sj n days from now

(p(n)ij )

Example: Let us determine the probability that if it is rainy today

then the event that it is snowy 2 days from now (p(2)rs)

We have the transition matrix:prr prn prspnr pnn pnspsr psn pss

=

p1,1 p1,2 p1,3p2,1 p2,2 p2,3p3,1 p3,2 p3,3

=

1/2 1/4 1/41/2 0 1/2

1/4 1/4 1/2

Then

P2 =

p

(2)1,1 p

(2)1,2 p

(2)1,3

p(2)2,1 p

(2)2,2 p

(2)2,3

p(2)3,1 p

(2)3,2 p

(2)3,3

p(2)1,3 = p1,1p1,3 + p1,2p2,3 + p1,3p3,3 = 38




32/40

( )

Consider a Markov chain process with the transition matrix P

Let

p

(0)

= (p

(0)

1 , p

(0)

2 , . . . , p

(0)

m ) be the initial probability distributionAnd let p(n) = (p

(n)1 , p

(n)2 , . . . , p

(n)m ) be the probability distribution at

step n

Then the probability distribution at step n can be found as

p(n) = p(0)Pn

That is

p(1) = p(0)Pp(2) = p(1)P= p(0)P2

. . .

p(n) = p(n1)P= p(0)Pn




33/40

( )

Example: In the Land of Oz example, we have:

P=


=

0.5 0.25 0.250.5 0 0.5

0.25 0.25 0.5

Let the initial probability vector be p(0) = (1/3, 1/3, 1/3)

Hence, the probability distribution of the states after 3 days isp(3) = p(0)P3:

(1/3, 1/3, 1/3)

0.406 0.203 0.3910.406 0.188 0.406

0.391 0.203 0.406

= (0.401, 0.198, 0.401)




34/40

( )

Example: Every day, a man either catches a bus or drives his car towork

Suppose he never goes by bus 2 days in a row

But if he drives to work, then the next day he is just as likely to driveagain as he is to travel by bus

The state space of the system is {bus} and {car}

This stochastic process is a Markov chain since the outcome on anyday depends only on what happened the preceding day

P=pbb pbcpcb pcc

= 0 1

1/2 1/2




35/40

( )

Example: Suppose now that on the first day the man tosses a fair dieand drives to work if and only if a 6 appeared

Then the initial probability distribution is given by

p(0) = (5/6, 1/6), P=

pbb pbcpcb pcc

=

0 1

1/2 1/2

Find the probability distribution after 4 days

We have that p(4) = p(0)P4

Then

(5/6, 1/6)

3/8 5/85/16 11/16

= (35/96, 61/96)

Thus, the probability of traveling to work by bus is 35/96 and drivingto work is 61/96




36/40

( )

Let P be the transition matrix for a regular chain

Then, as n, the powers Pn approach a limiting matrix W,where all rows approach the same vector w

The vector w is a steady-state vector

A regular Markov chain has only one steady-state vector

The steady-state vector w givesthe long-term probability distribution of the states of the Markov

chainw = w P,

i

wi

= 1

Thus, if a Markov system is regular, then its long-term transitionmatrix is given by the square matrix whose rows are all the same andequal to the steady-state vector




37/40

Example from Ryan ODonnells lecture: After 1 minute:

P=

pww pws pwepsw pss psepew pes pee

=

0.4 0.6 00.1 0.6 0.3

0.5 0 0.5

After 10 minutes we get:

P10

0.2940 0.4413 0.26480.2942 0.4411 0.2648

0.2942 0.4413 0.2648

And after 1 hour (i.e., P60

) the result is almost the same0.294117647058823 0.441176470588235 0.2647058823529410.294117647068823 0.441176470588235 0.264705882352941

0.294117647068823 0.441176470588235 0.264705882352941




38/40

Example: After 1 day, the weather in the Land of Oz is given by

P=


= 0.5 0.25 0.250.5 0 0.5

0.25 0.25 0.5

After 1 week:

P7 =0.4 0.2 0.40.4 0.2 0.4

0.4 0.2 0.4

Let the initial probability vector p(0) be either (1/3, 1/3, 1/3) or(1/10, 8/10, 1/10)

Even then, after 1 week the result is the same

p(7) = p(0)P7 = (0.4, 0.2, 0.4)

Hence, in the long run, the starting state does not really matter




39/40

If the higher and higher powers ofP approach a fixed matrix P, werefer to P as the steady-state or long-term transition matrix

It is often possible to approximate P with great accuracy by simplycomputing a very large power ofP

Q: How large?A: You know it is large enough when the rows are all the same with the

accuracy you desire

Matrix Algebra Tool:

http://people.hofstra.edu/stefan waner/RealWorld/matrixalgebra/

fancymatrixalg2.html


http://people.hofstra.edu/stefan_waner/RealWorld/matrixalgebra/fancymatrixalg2.htmlhttp://people.hofstra.edu/stefan_waner/RealWorld/matrixalgebra/fancymatrixalg2.htmlhttp://people.hofstra.edu/stefan_waner/RealWorld/matrixalgebra/fancymatrixalg2.htmlhttp://people.hofstra.edu/stefan_waner/RealWorld/matrixalgebra/fancymatrixalg2.html


40/40

Example: For the Land of Oz, the transition matrix is

P=


= 0.5 0.25 0.250.5 0 0.5

0.25 0.25 0.5

The steady-state vector is given by w = (0.4, 0.2, 0.4), so

w P= (0.4, 0.2, 0.4)

0.5 0.25 0.250.5 0 0.5

0.25 0.25 0.5

= (0.4, 0.2, 0.4) = w

Therefore, the long-term transition matrix is

W = P =

0.4 0.2 0.40.4 0.2 0.4

0.4 0.2 0.4


stochastic processes and markov chains

Documents