Markov ProcessesMarkov Processes

System Change Over TimeSystem Change Over Time

Data Mining and Forecast ManagementData Mining and Forecast ManagementMGMT E-5070MGMT E-5070

scene from the television series “Time Tunnel” (1970s)

Markov Process ModelsMarkov Process Models

Also known as Markov Chains.

Analyze how systems change over time.

Common applications include: consumer brand loyalty tendencies consumer brand-switching tendencies reliability analysis for equipment the aging / writeoff of accounts receivable the spoilage tendencies of perishable items over time

Andrei Andreyevich Markov( 1856 – 1922 )

Андрєй Aндрєєвич Марков

Ph.D , St. Petersburg University (1884)

Studied under Pafnuty Chebyshev.

Professor, St. Petersburg University (1886-1905) , but taught informally until 1922.

Early work in number theory, algebraic continual fractions, limits of integrals, and least squares method.

Launched the theory of stochastic processes ( Markov Chains ) : an all new branch of probability theory.

Markov Process ModelsMarkov Process Models

We will consider only We will consider only the simplest types the simplest types which are:which are:

discretediscrete finitefinite stationarystationary first-orderfirst-order

THE SCOPE OF STUDYTHE SCOPE OF STUDY

Markov Process ModelsMarkov Process Models

The states and transitionsThe states and transitions

are “discrete”are “discrete”

This means, for example,that market share among different stores can only change once per week or once per month, but not

second - to - second.

DISCRETE

Markov Process ModelsMarkov Process Models

The number of states is The number of states is “finite”“finite” Means, for example, that

an accounts receivablecan only age 3 months

before it is written off asa bad debt.

Once a bottle of winespoils, there are no

additional aging periods for it.

FINITE

Markov Process ModelsMarkov Process Models

Transitions dependTransitions dependonly on the only on the currentcurrentstate……….not on state……….not on

prior prior statesstatesThe chances of an

accounts receivablebeing paid off arehigher if it is one

month old as opposedto two months old.

STATIONARY

Markov Process ModelsMarkov Process Models

Transition probabilitiesTransition probabilitiesremain constant overremain constant over

timetimeFor example, thedefection rates ofcustomers from alocal supermarketto its competitorsmonth-by-monthwill always stay

the same.

FIRST-ORDER PROCESSES

Markov Process ModelsMarkov Process ModelsBASIC EXAMPLEBASIC EXAMPLE

TWO BARBERS IN A SMALL TOWN EACH HAVE A 50% MARKETSHARE. THEREFORE, THE VECTOR OF STATE PROBABILITIES

AT THIS TIME IS:

π ( 1 ) = ( 0.50.5 , 0.50.5 )

PERIOD NUMBER

ONE50% MARKET SHARE

FOR BARBER “AA”50% MARKET SHARE

FOR BARBER “BB”

Markov Process ModelsMarkov Process ModelsBASIC EXAMPLEBASIC EXAMPLE

P = P = [ ].90.90

.25.25

.10.10

.75.75

LOYALTYLOYALTYPROBABILITYPROBABILITYFOR BARBERFOR BARBER

““A” IN ANYA” IN ANYGIVEN PERIODGIVEN PERIOD

DEFECTIONDEFECTIONPROBABILITYPROBABILITYFOR BARBER FOR BARBER

““B” IN ANYB” IN ANYGIVEN PERIODGIVEN PERIOD

DEFECTIONDEFECTIONPROBABILITYPROBABILITYFOR BARBERFOR BARBER

““A” IN ANYA” IN ANYGIVEN PERIODGIVEN PERIOD

LOYALTYLOYALTYPROBABILITYPROBABILITYFOR BARBERFOR BARBER

““B” IN ANYB” IN ANYGIVEN PERIODGIVEN PERIOD

THE MATRIX OF TRANSITION PROBABILITIES PER MONTH ARE:

Markov Process ModelsMarkov Process ModelsBASIC EXAMPLEBASIC EXAMPLE

THE MARKET SHARE EACH BARBER HAS IN THE 2THE MARKET SHARE EACH BARBER HAS IN THE 2ndnd MONTH MONTHIS EQUAL TO THE PRODUCT OF THE IS EQUAL TO THE PRODUCT OF THE VECTOR OF STATEVECTOR OF STATE

PROBABILITIESPROBABILITIES IN PERIOD ( MONTH ) 1 AND THE MONTHLY IN PERIOD ( MONTH ) 1 AND THE MONTHLY MATRIX OF TRANSITION PROBABILITIESMATRIX OF TRANSITION PROBABILITIES::

ππ ( 2 ) = ( 2 ) = ππ ( 1 ) x P ( 1 ) x P

.90.90

.25.25

.10.10

.75.75

( ( 0.50.5 , , 0.50.5 ) ) = ( = ( 0.5750.575 , , 0.4250.425 ) )[ ].575.575 .425.425

BARBER ‘A’BARBER ‘A’ BARBER ‘B’BARBER ‘B’.45.45

.125.125

.05.05

.375.375BARBER ‘A’BARBER ‘A’ BARBER ‘B’BARBER ‘B’

Markov Process ModelsMarkov Process ModelsBASIC EXAMPLEBASIC EXAMPLE

THE MARKET SHARE EACH BARBER HAS IN THE 3THE MARKET SHARE EACH BARBER HAS IN THE 3rdrd MONTH MONTHIS EQUAL TO THE PRODUCT OF THE IS EQUAL TO THE PRODUCT OF THE VECTOR OF STATEVECTOR OF STATE

PROBABILITIESPROBABILITIES IN PERIOD (MONTH) 2 AND THE MONTHLY IN PERIOD (MONTH) 2 AND THE MONTHLY MATRIX OF TRANSITION PROBABILITIESMATRIX OF TRANSITION PROBABILITIES::

ππ ( 3 ) = ( 3 ) = ππ ( 2 ) x P ( 2 ) x P

.90.90

.25.25

.10.10

.75.75

( ( 0.5750.575 , , 0.4250.425 ) ) = ( = ( 0.620.62 , , 0.380.38 ) )[ ].62375.62375 .37625.37625

BARBER ‘A’BARBER ‘A’ BARBER ‘B’BARBER ‘B’.5175.5175

.10625.10625

.0575.0575

.31875.31875BARBER ‘A’BARBER ‘A’ BARBER ‘B’BARBER ‘B’

Markov Process ModelsMarkov Process ModelsBASIC EXAMPLEBASIC EXAMPLE

THE MARKET SHARE EACH BARBER HAS IN THE 4th MONTHTHE MARKET SHARE EACH BARBER HAS IN THE 4th MONTHIS EQUAL TO THE PRODUCT OF THE IS EQUAL TO THE PRODUCT OF THE VECTOR OF STATEVECTOR OF STATE

PROBABILITIESPROBABILITIES IN PERIOD (MONTH) 3 AND THE MONTHLY IN PERIOD (MONTH) 3 AND THE MONTHLY MATRIX OF TRANSITION PROBABILITIESMATRIX OF TRANSITION PROBABILITIES::

ππ ( 4 ) = ( 4 ) = ππ ( 3 ) x P ( 3 ) x P

.90.90

.25.25

.10.10

.75.75

( ( 0.620.62 , , 0.38 0.38 )) = ( = ( 0.653 0.653 , , 0.347 0.347 ))[ ].653.653 .347.347

BARBER ‘A’BARBER ‘A’ BARBER ‘B’BARBER ‘B’

.095.095

.062.062

.285.285

.558.558

BARBER ‘A’BARBER ‘A’ BARBER ‘B’BARBER ‘B’

Steady – State ProbabilitiesSteady – State Probabilities

Reached when the before and after state probabilities stay the same

forever, assuming no changes in thematrix of transition probabilities

Here, the eventual market share in the “barber” problem.

ALSO KNOWN AS THEALSO KNOWN AS THEEQUILIBRIUM OREQUILIBRIUM ORSTEADY-STATESTEADY-STATE

SOLUTIONSOLUTION

Equilibrium ConditionEquilibrium ConditionBARBER EXAMPLEBARBER EXAMPLE

The eventual market shares of the two barbers can be calculated directly from the matrix of transition probabilities.

Prior period vectors of state probabilities are not required.

The equations: .90 π1 + .25 π2 = π1

.10 π1 + .75 π2 = π2

.90 .10

.25 .75[ ]MATRIX OF TRANSITION

Equilibrium ConditionEquilibrium ConditionBARBER EXAMPLEBARBER EXAMPLE

Since π1 + π2 = 1.0 , π1 = 1.0 – π2

Therefore, “ 1.0 – π2 “ may be substituted for π1 in eitherof the two equations below:

.90 ( 1 – π2 ) + .25 π2 = 1 – π2

.10 ( 1 – π2 ) + .75 π2 = π2

or

Equilibrium ConditionEquilibrium ConditionBARBER EXAMPLEBARBER EXAMPLE

.90 ( 1 – π2 ) + .25 π2 = 1 – π2

.90 - .90 π2 + .25 π2 = 1 – π2

- .10 = - .35 π2

π2 = .2857

and π1 = ( 1 - .2857 ) = .7143

SUBSTITUTING IN THE 1st EQUATION, WE GET:

EVENTUALLY BARBER ‘A’ WILL HAVE 71% OF THE MARKET WHILE BARBER ‘B’ WILL HAVE THE REMAINING 29%

Equilibrium ConditionEquilibrium ConditionBARBER EXAMPLEBARBER EXAMPLE

.10 ( 1 – π2 ) + .75 π2 = π2

.10 - .10 π2 + .75 π2 = π2

.10 = .10 π2 - .75 π2 + 1.0 π2

.10 = .35 π2

π2 = .2857

π1 = ( 1 - .2857 ) = .7143

SUBSTITUTING IN THE 2nd EQUATION, WE GET:

Markov Processes with QM for WINDOWSMarkov Processes with QM for WINDOWS

We scroll to

“ Markov Analysis ”

We click on

“ New ”to solve a new problem

We specify thenumber of states.

Here, it is the market share

for the two barbers

The “ initial “ market shares(loyalty rates) are 50% / 50%

respectively.( Barber “A” is “1” )( Barber “B” is “2”)

The Matrix of Transitionis inserted to the right.

We desire to findthe market sharesover “4” periods

( months )

The “ End of Period 1 “ is actually the end of period “2”

The “ End of Period 2 “Is actually the end of period “3” , etc.

The market share ( loyalty rates ) after 4 months:

Barber A ( 1 ) - 66%Barber B ( 2 ) - 34%

The ‘Steady-State’or

‘Equilibrium’market shares

( loyalty rates ) are:

Barber ‘A’ ( 71% )Barber ‘B’ ( 29% )

Markov Processes UsingMarkov Processes Using

Template

Insert theMatrix of Transition

here

Matrix of Transition for the

next three periods

To obtain the Steady-State market shares,

( loyalty rates )

go to Tools, Solver

Steady State orEquilibirum

market shares( loyalty rates )

Gas Station ExampleGas Station ExampleMARKOV PROCESSESMARKOV PROCESSES

A town has three gas stations: A,B,C. The onlyfactor influencing the choice of station for the

next purchase is the prior purchase.

Each station is concerned about brand share.The town’s weekly gas sales are $10,000.00,and we assume each driver buys gas once

per week.

Gas Station ExampleGas Station ExampleMARKOV PROCESSESMARKOV PROCESSES

THE MARKET SHARES AT THIS PARTICULAR TIME AREAS FOLLOWS:

STATION ‘A’ - 30%STATION ‘B’ - 40%STATION ‘C’ - 30%

THEREFORE, THE VECTOR OF STATE PROBABILITIES IS:

π (1) = ( 0.3 , 0.4 , 0.3 )PERIOD NUMBER ONE

Gas Station ExampleGas Station Example

THE THE MATRIX OF TRANSITIONMATRIX OF TRANSITION PROBABILITIES PER WEEK ARE:PROBABILITIES PER WEEK ARE:

.90 .05 .05

.10 .80 .10

.20 .10 .70

P =

LOYALTY RATELOYALTY RATESTATION ‘A’STATION ‘A’

LOYALTY RATELOYALTY RATESTATION ‘B’STATION ‘B’

LOYALTY RATELOYALTY RATESTATION ‘C’STATION ‘C’

ALL OTHERS ARE DEFECTION RATESALL OTHERS ARE DEFECTION RATES

MARKOV PROCESSESMARKOV PROCESSES

Gas Station ExampleGas Station ExampleTHE MARKET SHARE THAT EACH GAS STATION HAS IN THETHE MARKET SHARE THAT EACH GAS STATION HAS IN THE

NEXT WEEK IS EQUAL TO THE PRODUCT OF THE VECTOR OFNEXT WEEK IS EQUAL TO THE PRODUCT OF THE VECTOR OFSTATESTATE PROBABILITIESPROBABILITIES IN PERIOD ( WEEK ) 1 AND THE WEEKLY IN PERIOD ( WEEK ) 1 AND THE WEEKLY

MATRIX OF TRANSITION PROBABILITIESMATRIX OF TRANSITION PROBABILITIES::

ππ ( 2 ) = ( 2 ) = ππ ( 1 ) x P ( 1 ) x P

P (A) = 0.30 A .90 .05 .05 .27 .015 .015P (A) = 0.30 A .90 .05 .05 .27 .015 .015 P (B) = 0.40 P (B) = 0.40 XX B .10 .80 .10 B .10 .80 .10 == .04 .32 .04 .04 .32 .04 P (C) = 0.30 C .20 .10 .70 .06 .03 .21P (C) = 0.30 C .20 .10 .70 .06 .03 .21 .37 .365 .265.37 .365 .265

A B CA B C

ππ ( 1 ) P ( 1 ) P

Gas Station ExampleGas Station Example THE MARKET SHARE THAT EACH GAS STATION HAS IN THETHE MARKET SHARE THAT EACH GAS STATION HAS IN THE 33rdrd WEEK IS EQUAL TO THE PRODUCT OF THE VECTOR OF WEEK IS EQUAL TO THE PRODUCT OF THE VECTOR OF

STATESTATE PROBABILITIESPROBABILITIES IN PERIOD ( WEEK ) 2 AND THE WEEKLY IN PERIOD ( WEEK ) 2 AND THE WEEKLY MATRIX OF TRANSITION PROBABILITIESMATRIX OF TRANSITION PROBABILITIES::

ππ ( 3 ) = ( 3 ) = ππ ( 2 ) x P ( 2 ) x P

P (A) = 0.37 A .90 .05 .05 .333 .018 .018P (A) = 0.37 A .90 .05 .05 .333 .018 .018 P (B) = 0.365 P (B) = 0.365 XX B .10 .80 .10 B .10 .80 .10 == .037 .292 .037 .037 .292 .037 P (C) = 0.265 C .20 .10 .70 .053 .027 .185P (C) = 0.265 C .20 .10 .70 .053 .027 .185 .423 .337 .24.423 .337 .24

A B CA B C

ππ ( 2 ) P ( 2 ) P

Gas Station ExampleGas Station ExampleTHE MARKET SHARE THAT EACH GAS STATION HAS IN THETHE MARKET SHARE THAT EACH GAS STATION HAS IN THE44thth WEEK IS EQUAL TO THE PRODUCT OF THE VECTOR OF WEEK IS EQUAL TO THE PRODUCT OF THE VECTOR OF

STATE PROBABILITIESSTATE PROBABILITIES IN PERIOD ( WEEK ) 3 AND THE WEEKLY IN PERIOD ( WEEK ) 3 AND THE WEEKLY MATRIX OF TRANSITION PROBABILITIESMATRIX OF TRANSITION PROBABILITIES::

ππ ( 4 ) = ( 4 ) = ππ ( 3 ) x P ( 3 ) x P

P (A) = 0.423 A .90 .05 .05 .381 .021 .021P (A) = 0.423 A .90 .05 .05 .381 .021 .021 P (B) = 0.337 P (B) = 0.337 XX B .10 .80 .10 B .10 .80 .10 = = .034 .269 .034 .034 .269 .034 P (C) = 0.240 C .20 .10 .70 .048 .024 .168P (C) = 0.240 C .20 .10 .70 .048 .024 .168 .463 .314 .223.463 .314 .223

A B CA B C

ππ ( 3 ) P ( 3 ) P

Steady–State ProbabilitiesSteady–State ProbabilitiesWE KNOW THAT WE HAVE ARRIVED AT THE EVENTUAL MARKETWE KNOW THAT WE HAVE ARRIVED AT THE EVENTUAL MARKETSHARES WHEN THE SHARES WHEN THE STARTING PROBABILITIESSTARTING PROBABILITIES ARE EQUAL TO ARE EQUAL TO

THE THE NEXT STATE PROBABILITIESNEXT STATE PROBABILITIES::

ππ ( Final State ) = ( Final State ) = ππ ( Starting ) x P ( Starting ) x P

P (A) = 0.P (A) = 0.589589 A .90 .05 .05 .530 .029 .029 A .90 .05 .05 .530 .029 .029 P (B) = 0.P (B) = 0.235235 XX B .10 .80 .10 B .10 .80 .10 == .024 .188 .024 .024 .188 .024 P (C) = 0.P (C) = 0.176176 C .20 .10 .70 .035 .018 .123 C .20 .10 .70 .035 .018 .123 ..589589 . .235235 . .176176

A B CA B Cππ ( starting ) P ( starting ) P

Gas Station ExampleGas Station ExampleCONCLUSIONCONCLUSION

Given that the total weekly gas sales are $10,000.00,Given that the total weekly gas sales are $10,000.00,the average weekly sales per station are:the average weekly sales per station are:

A : ( 0.589 ) ( 10,000 ) = $5,890.00A : ( 0.589 ) ( 10,000 ) = $5,890.00B : ( 0.235 ) ( 10,000 ) = $2,350.00B : ( 0.235 ) ( 10,000 ) = $2,350.00C : ( 0.176 ) ( 10,000 ) = $1,760.00C : ( 0.176 ) ( 10,000 ) = $1,760.00

Unless there is some change, the share of marketwill stay approximately 58.9% for station ‘A’,

23.5% for station ‘B’, and 17.6% for station ‘C’.

Calculation of Steady-State ProbabilitiesCalculation of Steady-State Probabilities

GAS STATION EXAMPLEGAS STATION EXAMPLE

INITIALINITIALSTATESTATE

PROBABILITIESPROBABILITIESTRANSITIONTRANSITION

PROBABILITIESPROBABILITIESNEW STATENEW STATE

PROBABILITIESPROBABILITIES

X1X1X2X2X3X3

AABBCC

A B CA B C

.90 .05 .05.90 .05 .05

.10 .80 .10.10 .80 .10

.20 .10 .70.20 .10 .70

A B CA B C

.90X1 .05X1 .05X1.90X1 .05X1 .05X1

.10X2 .80X2 .10X2.10X2 .80X2 .10X2

.20X3 .10X3 .70X3.20X3 .10X3 .70X3

P(A) = X1 P(B) = X2 P(C) = X3P(A) = X1 P(B) = X2 P(C) = X3

XX ==

Calculation of Steady-State Calculation of Steady-State ProbabilitiesProbabilities

GAS STATION EXAMPLEGAS STATION EXAMPLE

P(A) = .90XP(A) = .90X11 + .10X + .10X22 + .20X + .20X33 = 1X = 1X11

P(B) = .05XP(B) = .05X11 + .80X + .80X22 + .10X + .10X33 = 1X = 1X22

P(C) = .05XP(C) = .05X11 + .10X + .10X22 + .70X + .70X33 = 1X = 1X33

1X1X11 + 1X + 1X22 + 1X + 1X33 = 1 = 1

DEPENDENT EQUATIONDEPENDENT EQUATION

INDEPENDENT EQUATIONINDEPENDENT EQUATION

DEPENDENT EQUATIONDEPENDENT EQUATION

DEPENDENT EQUATIONDEPENDENT EQUATION

Equation ConversionEquation ConversionSET ALL 3 DEPENDENT EQUATIONS EQUAL TO ZERO:SET ALL 3 DEPENDENT EQUATIONS EQUAL TO ZERO:

P(A) = .9XP(A) = .9X11 + .1X + .1X22 + .2X + .2X33 = 1X = 1X11

.9X.9X11 – 1.0X – 1.0X11 + .1X + .1X22 + .2X + .2X33 = 0 = 0

-.1X-.1X11 + .1X + .1X22 + .2X + .2X33 = 0 = 0

P(B) = .05XP(B) = .05X11 + .8X + .8X22 + .1X + .1X33 = 1X = 1X22

.05X.05X11 + .8X + .8X22 – 1.0X – 1.0X22 + .1X + .1X33 = 0 = 0

.05X.05X11 -.2X -.2X22 + .1X + .1X33 = 0 = 0

P(C) = .05XP(C) = .05X11 + .1X + .1X22 + .7X + .7X33 = 1X = 1X33

.05X.05X11 + .1X + .1X22 + .7X + .7X33 – 1.0X – 1.0X33 = 0 = 0

.05X.05X11 + .1X + .1X22 -.3X -.3X33 = 0 = 0

Summary EquationsSummary Equations-.1X-.1X11 + .1X + .1X22 + .2X + .2X33 = 0 = 0

.05X.05X11 - .2X - .2X22 + .1X + .1X33 = 0 = 0

.05X.05X11 + .1X + .1X22 - .3X - .3X33 = 0 = 0

1X1X11 + 1X + 1X22 + 1X + 1X33 = 1 = 1

DEPENDENT EQUATIONSDEPENDENT EQUATIONS

INDEPENDENT EQUATIONINDEPENDENT EQUATION

The 3 dependent equations will not solve for theThe 3 dependent equations will not solve for thevalues of Xvalues of X11, X, X22, and X, and X33..

Therefore, we add the independent equation!Therefore, we add the independent equation!

The SolutionThe SolutionTO ELIMINATE X1 AMONG THE DEPENDENT EQUATIONS:

.05 X.05 X11 - .2X - .2X22 + .1X + .1X33 = 0 = 0 .05 X.05 X11 + .1X + .1X22 - .3X - .3X33 = 0 = 0

- .3X- .3X22 + .4X + .4X33 = 0 = 0

.05 X.05 X11 - .2X - .2X22 + .1X + .1X33 = 0 = 0

““.05”.05” ( 1.0 X ( 1.0 X11 + 1.0 X + 1.0 X22 + 1.0 X + 1.0 X33 = 1 ) = 1 )

.05 X05 X11 + .05 X + .05 X22 + .05 X + .05 X33 = .05 = .05

- .25 X.25 X22 + .05 X + .05 X33 = - .05 = - .05

The SolutionThe Solution

WITH JUST X2 AND X3 LEFT TO SOLVE, WE WILL SOLVE FORVARIABLE X2 FIRST:

- .3X- .3X22 + .4X + .4X33 = 0 = 0

““8”8” ( - .25X ( - .25X22 + .05X + .05X33 = - .05 ) = - .05 )

-2.0X2.0X22 + .4X + .4X33 = - .40 = - .40

1.7 X1.7 X22 = .40 = .40

XX22 = = .2352.2352

The SolutionThe SolutionWE SOLVE FOR XWE SOLVE FOR X33 BY SUBSTITUTING BY SUBSTITUTING “ X“ X22 = .2352 ” = .2352 ” INTO INTO

EITHER OF THE TWO DEPENDENT EQUATIONS:EITHER OF THE TWO DEPENDENT EQUATIONS:

-.3X.3X22 + .4X + .4X33 = 0 = 0-.3 (.2352) + .4X.3 (.2352) + .4X33 = 0 = 0

-.07056 + .4X.07056 + .4X33 = 0 = 0-.07056 = - .4X-.07056 = - .4X33

.1764.1764 = X = X33

WITH “XWITH “X22” and “X” and “X33” NOW KNOWN , WE EASILY SOLVE FOR” NOW KNOWN , WE EASILY SOLVE FOR““XX11” USING THE INDEPENDENT EQUATION:” USING THE INDEPENDENT EQUATION:

SINCE 1XSINCE 1X11 + 1X + 1X22 + 1X + 1X33 = 1 = 1

1X1X11 = 1 – 1X = 1 – 1X22 – 1X – 1X33

1X1X11 = 1 – ( = 1 – ( .2352.2352 ) – ( ) – ( .1764.1764 ) ) 1X1X11 = 1 – [ .4116 ] = 1 – [ .4116 ]

XX11 = = .5884.5884

Gas Station Market SharesGas Station Market SharesSTATE PROBABILITIESSTATE PROBABILITIES

Week P(A) P(B) P(C)

1 .30 .40 .30

2 .37 .365 .265

3 .423 .337 .240

4 .463 .314 .223

5 .492 .297 .211

6 .515 .283 .202

Gas Station MarketGas Station Market SharesSharesSTATE PROBABILITIESSTATE PROBABILITIES

Week P(A) P(B) P(C)

13 .577 .243 .180

14 .579 .242 .179

15 .581 .240 .179

16 .583 .239 .178

17 .584 .238 .178

18 .585 .237 .177

Gas Station Market SharesGas Station Market SharesSTATE PROBABILITIESSTATE PROBABILITIES

Week P(A) P(B) P(C)

24 .588 .236 .176

25 .588 .236 .176

26 .588 .236 .176

27 .588 .236 .176

28 .589 .235 .176

29 .589 .235 .176

Markov Processes with QM for WINDOWSMarkov Processes with QM for WINDOWS

We specify the number

of initial states.Here, it is themarket share

of the three ( 3 )gas stations

The “ Matrix of Transition “is inserted to the right of the

“ Initial States “

We arbitrarily selecttwelve ( 12 ) periods( weeks ) for analysis

The ‘ Steady - State ‘or

‘ Equilibrium ‘market shares ( loyalty rates ) are:

Station ‘A’ - 58.9 %Station ‘B’ - 23.5 %Station ‘C’ - 17.6 %

The market shares( loyalty rates )

for weeks2, 3, 4, and 5respectively

The market shares( loyalty rates )

for weeks6, 7, 8, and 9respectively

Themarket shares( loyalty rates )

for weeks 10, 11, 12, and 13

respectively

Markov Processes UsingMarkov Processes Using

Template

Changes to the transition matrixover 3 periods

Excel will not show the market shares for each gas station on a period-by-period basis !

By selecting Tools, Solver,the program will provide

the equilibrium (steady) statemarket shares

( loyalty probabilities )

Markov Process ModelsMarkov Process ModelsFOUR BASIC ASSUMPTIONS

I. The probability of an element changing from one state to another state remains constant from one period to another.

II. There are a limited number of states of nature.

Markov Process ModelsMarkov Process ModelsFOUR BASIC ASSUMPTIONS

III. If we know the present state and the matrix of transition, we can predict any future state.

IV. The parameters of the system do not change. That is, none of the states of nature are eliminated, and elements are not entering or leaving the system.

Absorbing StatesAbsorbing States

This is where the Markov model contains at least one absorbing or trapping state.

When the system reaches that absorbing state(s), it remains there forever !

Here, steady-state probabilities are no longer mean- ingful. Other measures of performance such as the mean number of transitions until absorption become important.

MARKOV PROCESSESMARKOV PROCESSES

Absorbing StatesAbsorbing StatesLIQUOR STORE EXAMPLELIQUOR STORE EXAMPLE

A liquor store owns 10,000 bottles of select wines. Each December, it permits its special customers to purchase wines from its collection. Some bottles that have not aged sufficiently are not offered and are kept for sale in future years. Currently, 3,000 bottles are not available for sale. Of the bottles available for sale in any year, some are not sold until future years. In addition, a bottle not sold may turn bad during the year and will not be available the next year.

REQUIREMENT:

How many of the 10,000 bottles will eventually be sold?

Absorbing StatesAbsorbing StatesLIQUOR STORE EXAMPLELIQUOR STORE EXAMPLE

THE FOLLOWING FOUR STATES ARE DEFINED:

1. AVAILABLE AND SOLD

2. TURNED BAD AND LOST

3. NOT SUFFICIENTLY AGED

4. AVAILABLE BUT NOT SOLD

Liquor Store ExampleLiquor Store ExampleTHE FOLLOWING THE FOLLOWING TRANSITION MATRIX TRANSITION MATRIX APPLIES:APPLIES:

FROMFROM

TOTO44332211

11

22

33

44

11 0 0 00 0 0

0 1 0 00 1 0 0

.1 .3 .4 .2.1 .3 .4 .2

.5 .2 0 .3.5 .2 0 .3

1. AVAILABLE AND SOLD , 2. TURNED BAD AND LOST , 3. NOT SUFFICIENTLY AGED , 4. AVAILABLE BUT NOT SOLD 1. AVAILABLE AND SOLD , 2. TURNED BAD AND LOST , 3. NOT SUFFICIENTLY AGED , 4. AVAILABLE BUT NOT SOLD

Absorption StateAbsorption State Problems ProblemsSUB - MATRIX SYMBOLSSUB - MATRIX SYMBOLS

• Transition Matrix “T” or “P”

• Fundamental Matrix “F” or “N”

• Transition Matrix for Absorption in the “R” or “A” Next Period •Transition Matrix for Movement Between “Q” or “B” Non-Absorption States • Identity Matrix “I”

The Identity MatrixThe Identity Matrix

State 1State 1Available Available

and Soldand Sold

State 2State 2Turned Bad Turned Bad

and Lostand Lost

State 1State 1Available Available

and Soldand Sold

State 2State 2Turned BadTurned Bad

and Lostand Lost

1 01 0

0 10 1““I”I”

( UPPER LEFT QUADRANT OF THE TRANSITION MATRIX )( UPPER LEFT QUADRANT OF THE TRANSITION MATRIX )

FROMFROM

TOTO

The “Q” or “B” MatrixThe “Q” or “B” Matrix

State 3State 3Not SufficientlyNot Sufficiently

AgedAged

State 4State 4Available but Available but

not Soldnot Sold

State 3State 3Not Sufficiently Not Sufficiently

AgedAged

State 4State 4Available but Available but

not Soldnot Sold

.4 .2.4 .2

0 .30 .3““Q”Q”

( BOTTOM RIGHT QUADRANT OF THE TRANSITION MATRIX )( BOTTOM RIGHT QUADRANT OF THE TRANSITION MATRIX )

FROMFROM

TOTO

The “R” or “A” MatrixThe “R” or “A” Matrix

State 1State 1Available Available

and Soldand Sold

State 2State 2Turned Bad Turned Bad

and Lostand Lost

State 3State 3Not Sufficiently Not Sufficiently

AgedAged

State 4State 4Available butAvailable but

not Soldnot Sold

.1 .3.1 .3

.5 .2.5 .2““R”R”

( BOTTOM LEFT QUADRANT OF THE TRANSITION MATRIX )( BOTTOM LEFT QUADRANT OF THE TRANSITION MATRIX )

FROMFROM

TOTO

Initial CalculationsInitial Calculations

1 0 .4 .2 1 0 .4 .2 .6 -.2 .6 -.2 0 1 0 .3 0 .70 1 0 .3 0 .7I - QI - Q

aa

bb

cc

dd

1 - .41 - .4 0 - .20 - .2

0 - 00 - 0 1 - .31 - .3

== ==--

LIQUOR STORE EXAMPLELIQUOR STORE EXAMPLE

Q or BMatrix

IdentityMatrix

The Fundamental MatrixThe Fundamental Matrix

F = [ I – Q ] =F = [ I – Q ] =-1 d/e – c/ed/e – c/e

-b/e a/e-b/e a/e==

..77/./.4242 . .22/./.4242

00 /. /.4242 . .66/./.4242

Where “Where “ee” = ( a x d ) – ( b x c ) = ( .6 x .7 ) – ( 0 x -.2 ) = [ .42 – 0] = ” = ( a x d ) – ( b x c ) = ( .6 x .7 ) – ( 0 x -.2 ) = [ .42 – 0] = .42.42

IdentityMatrix

“Q” or “B”

MatrixNegative Power

( matrix inversed )

The Fundamental MatrixThe Fundamental Matrix

F =F = 1.667 .4761.667 .476 0 1.4290 1.429

SUMSUM

2.1432.1431.4291.429

FOR “SFOR “S33” ” UNAVAILABLE UNAVAILABLE BOTTLES, THIS IS THE MEANBOTTLES, THIS IS THE MEANNUMBER OF YEARS UNTIL NUMBER OF YEARS UNTIL

DISPOSAL ( DISPOSAL ( 2.14 years2.14 years ) )

FOR “SFOR “S44” ” AVAILABLEAVAILABLEBOTTLES, THIS IS THE MEANBOTTLES, THIS IS THE MEAN

NUMBER OF YEARS UNTILNUMBER OF YEARS UNTILDISPOSAL ( DISPOSAL ( 1.429 years1.429 years ) )

THE NUMBER OF PERIODSTHE NUMBER OF PERIODS( ( HERE HERE YEARSYEARS ) THAT BOTTLES) THAT BOTTLES

WILL BE IN ANY OF THE WILL BE IN ANY OF THE NON - ABSORBING STATESNON - ABSORBING STATES

BEFORE ABSORPTION FINALLYBEFORE ABSORPTION FINALLYOCCURSOCCURS

The FR MatrixThe FR Matrix

FR = FR = 1.667 .4761.667 .476 0 1.4290 1.429 XX .1 .3.1 .3

.5 .2.5 .2

aa bb

cc dd

““FF” MATRIX” MATRIX ““RR” MATRIX” MATRIX

ee ff

gg hh

==.1667 + .238.1667 + .238 0 + .71450 + .7145

.5001 + .0952.5001 + .0952 0 + .28580 + .2858

ae + bgae + bg

ce + dgce + dg

af + bhaf + bh

cf + dhcf + dh

The FR MatrixThe FR Matrix

FR =FR = .4047 .5953.4047 .5953.7145 .2858.7145 .2858

SS33

SS44

SS11 S S22THE PROBABILITIES OFTHE PROBABILITIES OF

ABSORPTION GIVEN ANYABSORPTION GIVEN ANYSTARTING STATESTARTING STATE

40.47%40.47% chance that chance thatan inadequately-agedan inadequately-agedbottle will eventuallybottle will eventually

be soldbe sold

71.45%71.45% chance that chance thatan available, but notan available, but not

sold bottle will eventuallysold bottle will eventuallybe sold.be sold.

59.53%59.53% chance that chance thatan inadequately-agedan inadequately-agedbottle will eventuallybottle will eventuallyturn bad and be lostturn bad and be lost

28.58%28.58% chance that chance thatan available, but notan available, but not

sold bottle will eventuallysold bottle will eventuallyturn bad and be lost.turn bad and be lost.

The Final SolutionThe Final SolutionLIQUOR STORE EXAMPLELIQUOR STORE EXAMPLE

10,000 bottles of wine are currently owned by the liquor store

3,000 bottles of wine are not nownot now available for sale ( given )

7,000 bottles of wine are now availablenow available for sale ( inferred )

EVENTUAL ( EXPECTED ) SALES in bottles :

S3: .405.405 x 3,000 bottles = 1,215 bottles

S4: .714.714 x 7,000 bottles = 4,998 bottles

Σ = 6,2136,213 bottles

There are 4 states:

1. Available & sold2. Turned bad & lost3. Not sufficiently aged4. Available but not sold

Since we do not knowthe initial state

probabilities, we can arbitrarily

assign an equalprobability to each

event ( 25% )

The ‘ Matrix of Transition ‘is inserted to the right

of the initial stateprobabilities

The‘ FR ‘ Matrix

40.47% chance that an inadequately - aged bottle will eventually be sold

71.45% chance that an available but not yet sold bottle will eventually be sold

59.53% chance that an inadequately - aged bottle will turn bad and be lost

28.58% chance that an available but not yet sold bottle will eventually turn bad and be lost

Eventually,

52.98% of all bottles will be sold

&47.02% of all bottles

will be lost

The “ B “ or “ Q “ Matrix

The “ F “ or “ Fundamental Matrix “

The “ FA “ or “ FR “ Matrix

Markov Processes UsingMarkov Processes Using

Matrix of TransitionDevelopment

( period by period )

State 1Available + Sold

State 2Turned Bad + Lost

State 3Not Aged Enough

State 4Available, Not Sold

State 1Available + Sold

State 2Turned Bad + Lost

State 3Not Aged Enough

State 4Available, Not Sold

40.5% chance bottles not aged enough will be sold eventually

59.5% chance bottles not aged enough will be lost eventually

71.4% chance bottles available will be sold eventually

28.6% chance bottles available will be lost eventually

Markov ProcessesMarkov Processes

System Change Over TimeSystem Change Over Time