Hamid R. Rabiee

Stochastic Processes

Markov Chains

Absorption

1

Absorbing Markov Chain

• An absorbing state is one in which the

probability that the process remains in that

state once it enters the state is 1 (i.e., 𝑝𝑖𝑖 = 1).• A Markov chain is absorbing if it has at least

one absorbing state, and if from every state it is

possible to go to an absorbing state (not

necessarily in one step).

2Stochastic Processes

4 𝟏

𝟐

3 2 1 0 𝟏

𝟐

𝟏

𝟐

𝟏

𝟐𝟏

𝟐

𝟏

𝟐

𝟏

𝟏

States 0 and 4 are absorbing

The canonical form

• By separating transient (TR) and absorbing (ABS) states,

the transition matrix of any absorbing Markov chain can

be written as:

• And as time passes we can see that:

3Stochastic Processes

Absorption theorem

In an absorbing MC the probability that the process will

be absorbed is 1. (i.e. 𝑄𝑛 → 0 as 𝑛 → ∞).

Proof sketch:

By definition of an absorbing MC, There exist a path S

from any non-absorbing state 𝑠𝑗 to an absorbing state.

So there is a positive probability 𝑝𝑗 of taking this path

every time the process starts from 𝑠𝑗.

Therefore there exists p and m, such that the

probability of not absorbing after m steps is at most p.

After km steps the probability of not being absorbed is

at most 𝑝𝑘 , and as time goes to infinity this

probability approaches zero.

4Stochastic Processes

The Fundamental Matrix

5Stochastic Processes

Definition: For an absorbing Markov chain 𝑃, the following

matrix is called the fundamental matrix for 𝑃.

𝑁 = 𝐼 − 𝑄 −1

Theorem: For an absorbing MC

• the matrix 𝐼 − 𝑄 has an inverse 𝑁,

• and 𝑁 = 𝐼 + 𝑄 + 𝑄2 +⋯ .

• The 𝑖𝑗-entry 𝑛𝑖𝑗 of the Matrix 𝑁 is the expected number of

times the chain is in state 𝑠𝑗, given that it starts in state 𝑠𝑖.

Proof:

6Stochastic Processes

• 𝐼 − 𝑄 𝑥 = 0 ⇒ 𝑥 = 𝑄𝑥 ⇒ 𝑥 = 𝑄𝑛𝑥.

Since Qn → 0, we have 𝑄𝑛𝑥 → 0, so 𝑥 = 0.

Thus 𝑥 = 0 is the only point in the nullspace of 𝐼 − 𝑄,

therefore 𝐼 − 𝑄 −1 = 𝑁 exists.

• 𝐼 − 𝑄 𝐼 + 𝑄 + 𝑄2 +⋯+ 𝑄𝑛 = 𝐼 − 𝑄𝑛+1 ⇒

𝐼 + 𝑄 + 𝑄2 +⋯+ 𝑄𝑛 = 𝑁(𝐼 − 𝑄𝑛+1).

Letting 𝑛 tend to infinity we have:

𝑁 = 𝐼 + 𝑄 + 𝑄2 +⋯

Proof (cont’d):

7Stochastic Processes

• Consider two transient states 𝑖 and 𝑗, and suppose that 𝑆𝑖is the initial state.

• 𝑋(𝑘): a R.V. which equals 1 if the chain is in state 𝑠𝑗 after 𝑘

steps, and equals 0 otherwise. We have:

𝑃 𝑋 𝑘 = 1 = (𝑄𝑘)𝑖𝑗

• The expected number of times the chain is in state 𝑠𝑗 in

the first 𝑛 steps, given that it starts in state 𝑠𝑖 is:

𝐸 𝑋 0 + 𝑋 1 +⋯+ 𝑋 𝑛 = (𝑄0)𝑖𝑗 + (𝑄1)𝑖𝑗 +⋯+ (𝑄

𝑛)𝑖𝑗

• As 𝑛 goes to infinity we have:

𝐸 𝑋 0 + 𝑋 1 +⋯ = (𝑄0)𝑖𝑗 + (𝑄1)𝑖𝑗 +⋯ = 𝑁𝑖𝑗

Example:

8Stochastic Processes

• Consider the following Markov chain (1D random walk with 5

states):

• The transition matrix in canonical form is:

4 𝟏

𝟐

3 2 1 0 𝟏

𝟐

𝟏

𝟐

𝟏

𝟐𝟏

𝟐

𝟏

𝟐

𝟏

𝟏

;

Example (cont’d):

9Stochastic Processes

• If we start in state 2, then the expected number of times in

states 1, 2 and 3 before being absorbed are 1, 2 and 1.

Time to Absorption:

10Stochastic Processes

• Question: Given that the chain starts in state 𝑠𝑖, what is

the expected number of steps before the chain is absorbed?

• Reminder: Starting from 𝑠𝑖, the expected number of steps

the process will be in state 𝑠𝑗 before absorption is 𝑁𝑖𝑗 .

Therefor 𝑗𝑁𝑖𝑗 is the expected number of steps before

absorption.

• Theorem: Let 𝑡𝑖 be the expected number of steps before

the chain is absorbed, given that the chain starts in state

𝑠𝑖, and let t be the column vector whose 𝑖-th entry is 𝑡𝑖.Then 𝑡 = 𝑁𝑐 , where 𝑐 is a column vector all of whose

entries are 1.

Absorption Probabilities:

11Stochastic Processes

• Question: Given that the chain starts in the transient

state 𝑠𝑖, what is the probability that it will be absorbed in

the absorbing state 𝑠𝑗?

• Intuition: Starting from 𝑠𝑖 , the expected number the

process will be in state 𝑠𝑘 before absorption is 𝑁𝑖𝑘. Each

time, the probability to move to state 𝑠𝑗 is 𝑅𝑘𝑗 (𝑘𝑗 -th

element of matrix 𝑅 introduced in the canonical form).

Absorption Probabilities:

12Stochastic Processes

• Theorem: Let 𝐵𝑖𝑗 be the probability that an absorbing

chain will be absorbed in the absorbing state 𝑠𝑗 if it starts

in the transient state 𝑠𝑖. Let 𝐵 be the matrix with entries

𝑏𝑖𝑗. Then 𝐵 is a 𝑡-by-𝑟 matrix, and 𝐵 = 𝑁𝑅, where 𝑁 is the

fundamental matrix and 𝑅 is as in the canonical form.

• Proof:

𝑩𝒊𝒋 =

𝒏

𝒌

𝒒𝒊𝒌(𝒏)𝒓𝒌𝒋

=

𝒌

𝒏

𝒒𝒊𝒌(𝒏)𝒓𝒌𝒋 =

𝒌

𝒏𝒊𝒌𝒓𝒌𝒋

= 𝑵𝑹 𝒊𝒋

Example:

13Stochastic Processes

• In previous example (1D random walk with 5 states) we

found that:

• Hence

• The expected number of steps before absorption when the

process starts from states 1, 2, 3 is 3, 4 and 3

respectively.

Example (cont’d):

14Stochastic Processes

• From the canonical form:

• Hence

• Here the first row tells us that, starting from state 1, there is

probability 3/4 of absorption in state 0 and 1/4 of absorption in

state 4.

References

Grinstead C. M, and Snell J. L, Introduction to

probability, American Mathematical Society,

1997

15Stochastic Processes