Markov-opolyMarkov-opoly

Markov chains: an Applied Approach

By Daniel Huang and Mo Dwyer

What is a Markov Chain, Anyway?What is a Markov Chain, Anyway?

• It is a sequence of states where every It is a sequence of states where every future state is independent of the future state is independent of the preceding ones, except for the n-1 state.preceding ones, except for the n-1 state.

The ProcessThe Process

• Create a “transition matrix.” Create a “transition matrix.”

• This defines the probability of something This defines the probability of something being in any given location in the state being in any given location in the state you are interested in.you are interested in.

• Raise the transition matrix to the nth Raise the transition matrix to the nth power.power.

Confused? Have an example…Confused? Have an example…

The state: a car rental agency has three The state: a car rental agency has three locations in LA:locations in LA:

• Downtown location (A)Downtown location (A)

• East end location (B) East end location (B)

• West end location (C). West end location (C).

The agency's statistician has The agency's statistician has determined the following:determined the following:

Of the calls to the Downtown location, 30% are delivered Of the calls to the Downtown location, 30% are delivered in Downtown area, 30% are delivered in the East end, and in Downtown area, 30% are delivered in the East end, and 40% are delivered in the West end40% are delivered in the West end

Of the calls to the East end location, 40% are delivered in Of the calls to the East end location, 40% are delivered in Downtown area, 40% are delivered in the East end, and Downtown area, 40% are delivered in the East end, and 20% are delivered in the West end20% are delivered in the West end

Of the calls to the West end location, 50% are delivered in Of the calls to the West end location, 50% are delivered in Downtown area, 30% are delivered in the East end, and Downtown area, 30% are delivered in the East end, and 20% are delivered in the West end.20% are delivered in the West end.

The Transition Matrix:The Transition Matrix:

T= T=

TT77==

TT22==

Notice how it Notice how it starts to starts to cononverge!!

TTnn==

You now have the probability distribution You now have the probability distribution of the drivers at time n!of the drivers at time n!

So where will you be next?So where will you be next?

• Take the transition matrix and multiply Take the transition matrix and multiply by the current state.by the current state.

• That is: That is: XXO+1O+1=TX=TXOO

So where will you be next?So where will you be next?

• Take the transition matrix and multiply Take the transition matrix and multiply by the current state.by the current state.

• That is: That is: XXO+1O+1=TX=TXOO

• After time n approaching infinity, the After time n approaching infinity, the resulting vector is not dependent on Xoresulting vector is not dependent on Xo

P=TP=TnnXX

Our Project:Our Project:

Creating a Markov chain to predict Creating a Markov chain to predict monopoly moves. monopoly moves.

For MonopolyFor Monopoly

• First, find the probability of First, find the probability of rolling a certain number, and rolling a certain number, and thus landing on a certain square.thus landing on a certain square.

• Add the prob of Chance or Add the prob of Chance or Community Chest sending you Community Chest sending you somewhere.somewhere.

Thank goodness for Thank goodness for MATLABMATLAB

• You end up with a 40x40 matrix You end up with a 40x40 matrix because there are 40 squares to because there are 40 squares to land on. land on.

  Go Meridian CC Baltic Income RR Oriental

Go 0.0121 0.0086 0.0069 0.0052 0.0035 0.0034 0.0035

Meridian Ave 0 0 0 0 0 0 0

Community Chest 0.0244 0 0 0 0 0 0

Baltic Avenue 0.0556 0.0278 0 0 0 0 0

Income Tax 0.0833 0.0642 0.0347 0.0052 0.0035 0.0017 0

Reading Railroad 0.1319 0.0919 0.0625 0.033 0.0035 0.0017 0

Oriental Ave 0.1389 0.1111 0.0833 0.0556 0.0278 0 0

Chance 0.0626 0.0529 0.0416 0.0313 0.0205 0.0108 0

Vermont Ave 0.1389 0.1666 0.1389 0.1111 0.0833 0.0556 0.0278

Connecticut Ave 0.1111 0.1389 0.1666 0.1389 0.1111 0.0833 0.0556

Just Visiting 0.0833 0.1111 0.1389 0.1666 0.1389 0.1111 0.0833

St. Charles Pl 0.066 0.0919 0.1185 0.1441 0.1701 0.1406 0.1111

Electric Company 0.0382 0.0642 0.0902 0.1163 0.1425 0.1683 0.1389

States Ave 0 0.0278 0.0556 0.0833 0.1111 0.1389 0.1666

Virginia Ave 0 0 0.0278 0.0556 0.0833 0.1111 0.1389

Pennsylvania Railroad 0.0208 0.0172 0.0138 0.0382 0.0626 0.0867 0.1111

St. James Pl 0 0 0 0 0.0278 0.0556 0.0833

Et c. …

Alternatively:Alternatively:

• You can also use eigenvectors to solve You can also use eigenvectors to solve steady-states!steady-states!

Alternatively:Alternatively:

• You can also use eigenvectors to solve You can also use eigenvectors to solve steady-states!steady-states!

• Take Take λλ=1=1

• Tp=pTp=p

• p is the probability vector of p is the probability vector of dimension mx1. Its elements add to dimension mx1. Its elements add to equal 1.equal 1.

THE END!THE END!

Any questions?Any questions?