the rg-factorizations in stochastic models

The RG-Factorizations The RG-Factorizations in Stochastic Modelsin Stochastic Models

Dr. Quan-Lin LiDr. Quan-Lin Li

Department of Industrial EngineeringDepartment of Industrial Engineering

Tsinghua UniversityTsinghua UniversityBeijing 100084, P.R. ChinaBeijing 100084, P.R. China

Why to needWhy to need the RG-factorizations the RG-factorizations

How to constructHow to construct the RG-factorizationsthe RG-factorizations

How to applyHow to apply the RG-factorizations the RG-factorizations

Promising issuesPromising issues in the future in the future

Outline of this talkOutline of this talk

Why to needWhy to need From 1996 to 2000, my research focuses on quasi-From 1996 to 2000, my research focuses on quasi-

stationary distributions of stochastic modelsstationary distributions of stochastic models

Our main problem is described as follows:

Our Question: How to compute?

Discrete

is a transition probability matrix Continuous

is an infinitesimal generator 1

P

PQ

Qe

Why to needWhy to need

When the size of the matrix is finite, this computation is similar to that for the stationaryprobability vectors of the finite-state Markovchains by using systems of linear equations

-classifica

P

tion of state; solving ; is the convengence radious

0 for all the other cases

Why to needWhy to need

When the size of the matrix is infinite, this computation will become different and difficult

Need to consider the existence Need to consider the uniqueness:

There are over one quasi-stationar

P

y distributions No available expression

No effective approach

Why to needWhy to need

Our work from 1996 to 1999 was to Our work from 1996 to 1999 was to develop thedevelop the LU-block-decompositionLU-block-decomposition for for Markov chains of M/G/1 type Markov chains of M/G/1 type and GI/M/1 typeand GI/M/1 type

Our method is different from Our method is different from that used by Bean, Latouche, that used by Bean, Latouche, Taylor etc.Taylor etc.

Why to needWhy to need

1 0

2 1 0

3 2 1 0

4 3 2 1 0

1 2 3 4

0 1 2 3

0 1 2

0 1

GI/M/1 type

M/G/1 type

B BB A A

P B A A AB A A A A

B B B BB A A A

P A A AA A

Why to needWhy to needThe LU-block-decomposition is

Our Question: Such a solution is OK?

Discrete

Continuous

Our computation is given by 0

Let . Then 0

Based on this, we can give a solution

I P L U

Q L U

L U

x L xU

Why to needWhy to need For a special Markov chain, we For a special Markov chain, we

obtained two differentobtained two different LU-block-LU-block-decompositiondecompositions, which lead to two s, which lead to two different expressions different expressions

One of them is correctOne of them is correct and is the and is the same as that in the literature; while same as that in the literature; while another is wronganother is wrong

Why?

Why to needWhy to need

We analyzed many real examples We analyzed many real examples and then found the main reasonsand then found the main reasons

These computations motivate us to These computations motivate us to

extend theextend the LU-block-decompositionLU-block-decomposition to the to the RG-factorizationRG-factorization

Why to needWhy to need

For an arbitary irreducible Markov chain, the RG-factorization is given by

Discrete

Continuous

Two different LU-decompositions ar

U D L

U D L

I P I R I I G

Q I R I G

e

, ;

,

Our computation is given by

0

U D L

U D L

U D L

L I R I U I G

L I R U I I G

I R I I G

Why to needWhy to need

For this computation

0

How to take the vector

A key observation:When is -positive recurrent with = ,

U D L

U

U D

I R I I G

I Rx

I R I

P

we should use

All the other cases, we should use

U

U D

x I R

x I R I

??

Our ComparisonsOur Comparisons Utility of the Utility of the RG-factorization RG-factorization is is

related to the classification of related to the classification of state by means of state by means of the diagonal the diagonal matrixmatrix, and keep effective , and keep effective computationscomputations

Better than Better than LU-decompositionLU-decomposition

How to constructHow to construct

0,0 0,1 0,

1,0 1,1 1,

,0 ,1 ,

Consider a discrete-time Markov chain

where or

Let 0, , and 1, , . Then

N

N

N N N N

c

c

c

P P PP P P

P

P P P

N N

E n E n N

E EP E T U

E V W

How to constructHow to construct

0

We can have two types of censored Markov chains:UL-type: To the level set

which leads to the UL-type RG-factorization

LU-type: To the level set

n k

k

c

n

E

P T U W V

E

P W

1

which yields the LU-type RG-factorization

V I T U

The UL-type RG-factorizationThe UL-type RG-factorization

0,0 0,1 0,

1,0 1,1 1,

,0 ,1 ,

,

1

, ,

1, ,

We write

We define

U-measure: , 0,

R-measure: , 0 ,

G-measure: , 0

n n nn

n n nn n

n n nn n n n

nn n n

ji j i j j

ii j i i j

P

n

R I i j

G I

.j i

The UL-type RG-factorizationThe UL-type RG-factorization

0,1 0,2 0,3

1,2 1,3

2,3

For an abitrary irreducible Markov chain , the UL-typeRG-factorization is given by ,

where0

0

0

U D L

U

P

I P I R I I G

R R RR R

RR

0 1 2 3

1,0

2,0 2,1

3,0 3,1 3,2

, , , ,

00

00

D

L

diag

GG G G

G G G

The UL-type RG-factorizationThe UL-type RG-factorization

0

0

0

Important Properties for Censoring Structure: is irreducible if is irreducible; is positive recurrent if is recurrent; is transient if is transient;

is transient for all 1.k

PP

P

k

Some special casesSome special cases

0,1

1,2 1,0

2,3 2,1

1,0

2,1

0,1

1

The QBD processes0 0

0 0 ,

0 0

The M/G/1 type0

0

0

The GI/M/1 type0

0

U L

L

U

RR G

R GR G

GG

G

RR

R

,2

2,30 R

Some special casesSome special cases

0,1 0,2 0,3

1 2

1

0

1,0

2,0 1

3,0 2 1

The GI/G/1 type0

0

0

, , , ,

00

00

U

D

L

R R RR R

RR

diag

GG G G

G G G

How to applyHow to apply

0

0 0

0

0

Let . Then 0.

Observating a non-zero nonnegative solution , 0,0,0, ,

where is the stationary probability vector of

Therefore, = ,0,0,0,

U D D

U D L

U

I R I I G

x I R x I I G

x x

x

x I R

1

0 01

0

yields

,

, 1.k

k i k ii

x

R k

RemarksRemarks

Computing the stationary probability vector of the Markov chain with a huge state space or an infinite state space is decomposited into two steps:

Step one: Computing the stat

P

0

,

ionary probability vector of the censored chain with a smaller state space

Step two: Computing the R-measure for 0

by using the above iterative relations.i jR i j

Finite statesSmaller

Finite statesHuge

Infinite states

The UL-type RG-factorization

A crucial advanceA crucial advance

The LU-type RG-factorizationThe LU-type RG-factorization

, , 1 , 2

1, 1, 1 1, 2

2, 2, 1 2, 2

,

1

, ,

We write

We define

U-measure: , 0,

R-measure: , 0 ,

G-measure:

n n nn n n n n n

n n nn n n n n n n

n n nn n n n n n

nn n n

ji j i j j

P

n

I j i

R

1, , , 0 .i

i j i i jI i j G

1,0

2,0 2,1

3,0 3,1 3,2

For an abitrary irreducible Markov chain , the LU-typeRG-factorization is given by ,

where0

0 0

0

L D U

L

P

I P I I I

R G

RR R R

R R R

0 1 2 3

0,1 0,2 0,3

1,2 1,3

2,3

, , , ,

00

0

D

U

diag

G G GG G

GG

The LU-type RG-factorizationThe LU-type RG-factorization

The LU-type RG-factorizationThe LU-type RG-factorization

1 1 1 1

1 11 1

2 3

2

is transient for all 0. The matrix or of size must be invertible,

An example,

1 1 12 2 2

1 1 1 2 2 2

k

U D L

U D L

kI P Q

I P I I I

Q I I

P

G R

G R

3

2 31 1 12 2 2

Some special casesSome special cases

0,1

1,0 1,2

2,1

0,1

1,2

1,0

2,1

The QBD processes0 0

0 0 ,

0 0

The M/G/1 type0

0

0

The GI/M/1 type0

0

0

U L

L

U

GR G

R GR

GG

G

RR

R

How to applyHow to applyThe LU-type RG-factorization is different from the UL-typecase. It may be used to deal with the first passage times and the sojourn times. In addition, we provide a better example:Consider a perturbed

| 0

| 0

Markov chain . Let and be the stationary probability vectors of and , respectively.

Then

which leads to

P PP P

Pd I P

d

dd

1 1 1U D LI I I G R

Comparison for UL- and LU-typeComparison for UL- and LU-type

Systems of linear equations 0or 0

xA

Ax

Systems of linear equations ( 0)or ( 0)

xA b b

Ax b b

UL-type RG-factorization

LU-type RG-factorization

Question:

Question: Systems of linear equations

Systems of linear equations

Our work on the RG-factorizationsOur work on the RG-factorizations

The RG-factorizations

The RG-factorizations

Quasi-stationary distributions

TheoryTailed analysis

Sensitive analysis

Markov Reward processes

Markov Decision processes

Evolutionary games

Applications

Networking safetyComputer networksProduction systemsReal-time management

Promising problems (1)Promising problems (1)

For the RG-factorizations:For the RG-factorizations:1. 1. It is interesting to consider the It is interesting to consider the dd-period for the -period for the

R-, U- and G-measures. For exampleR-, U- and G-measures. For example

(1)(1) A A = = AA0 0 + + AA1 1 + + AA2 2 is irreducible and is is irreducible and is dd-period, -period,

the two matrices the two matrices RR and and GG are are dd-period-period ??(2) (2) For a Markov chain of GI/G/1 type, what For a Markov chain of GI/G/1 type, what

happen to happen to ?? Such a work is useful for tailed analysisSuch a work is useful for tailed analysis

0 0

0 0

Discrete time

,

Continuous time

0, 0

k kk k

k k

k kk k

k k

R R A G A G

R A A G

Promising problems (2)Promising problems (2)

For the RG-factorizations:For the RG-factorizations:1.1. It is interesting to consider spectral analysis It is interesting to consider spectral analysis

for the R-, U- and G-measures. for the R-, U- and G-measures.

When When A A = = AA0 0 + + AA1 1 + + AA2 2 is irreducible and is is irreducible and is infinite size, how to analyze the spectral of infinite size, how to analyze the spectral of

the two matrices the two matrices RR and and G G ??2. 2. For a Markov chain of GI/G/1 type, what For a Markov chain of GI/G/1 type, what

happen to happen to ??

Promising problems (3)Promising problems (3)

, , , ,1

, , , ,1

, ,

To construct the RG-factorization, we have formedmany useful relations such as Winner-Holp equations

i j j i j i k k k jk j

i i j i j i k k k jk j

n n n n k

R I P R I G

I G P R I G

P R I

,1

Effective algorithms are necessary to compute the R-, U- and G-measures, and then compute performancemeasures of a stochastic models.

k k nk n

G

Promising problems (4)Promising problems (4)

0

Transient Performance:Continuous-time Markov chain: or

0 exp 0 exp

Continuous-time Markov reward process: ,

t

Q Q t

d dt t Q t t Q tdt dt

t Qt t Q u du

Q f X

0

or

, , , , ,

, , ,t

t

t

u i t

R

t f X du H t x P t x X i t x

t x t x R t x Qx

Train repairable networksTrain repairable networks

GM manufacturer

GM manufacturerPart Supplier

Part Supplier

Container Park

GM manufacturer

GM manufacturer

Container Park

Part Supplier

Real-time managementReal-time management

LocalOptimization

LocalOptimization

Global Optimization

Queueing

Networks

Information Theory

Thanks for youThanks for youand and

questions ?questions ?