math 456: mathematical modelingpeople.math.umass.edu/~nguillen/456/week8_countingvisits_etc.pdf ·...
TRANSCRIPT
Today
Today we will see:
1. Where are we? A quick review of last few classes.
2. Some comments on Problem Set 4.
3. Proof of the Strong Markov Property.
Next class:
1. Counting how often a state is visited.
2. Statement of the convergence theorem.
ReviewLast couple of classes
1. Transient and recurrent states.
2. Stopping times and exit distributions.
3. Calculus on graphs and the Laplacian.
4. The Strong Markov Property.
5. Probabilities of returns.
ReviewTransient and recurrent states
In a Markov chain, we said first (informally) that a state x isrecurrent if starting from x we are guaranteed to eventuallyreturn to x, and said to be transient if there is some chance ofnever returning to x.
In terms of the first visit time Tx, this is written as
x is said to be transient if Px[Tx <∞] < 1,
x is said to be recurrent if Px[Tx <∞] = 1.
(recall that Tx = min{n ≥ 0 | Xn = x}
)
ReviewStopping times and exit distributions
For a set A ⊂ S, the first arrival time TA is an example of astopping time.We were interested in its exit distribution
Gy(x) = Px[XTA = y],
this defined for y ∈ A and x ∈ S.
ReviewCalculus on graphs and the Laplacian
Going back to the exit distributions, we found that
∆Gy(x) = 0 if x 6∈ A,Gy(x) = 0 if x ∈ A, x 6= y,
Gy(y) = 1.
This system of equations is always sufficient to determine allthe exit probabilities Gy(x)!!
ReviewCalculus on graphs and the Laplacian
Consider a Markov chain with state space S and transitionprobability p(x, y).
Given f : S → R, we defined
∆f(x) =∑y
(f(y)− f(x))p(x, y).
ReviewThe Strong Markov Property
Theorem (Strong Markov Property)
Let Xn denote a Markov Chain. If T is a stopping time andYn := XT+n, then Yn is also a Markov chain. Moreover, thetransition probability for Yn is the same as the one for Xn.
ReviewProbabilities of return
Using the Strong Markov Property, we were able to computethe probability of returning to x at least k times,
Px[T (k)x <∞] = ρkxx
where
ρxy = Px[Ty <∞]
Today
Next
1. Some comments on Problem Set 4.
2. Proof of the Strong Markov Property.
3. Counting how often a state is visited.
The Strong Markov Property
Theorem (Strong Markov Property)
Let Xn denote a Markov Chain. If T is a stopping time andYn := XT+n, then Yn is also a Markov chain. Moreover, thetransition probability for Yn is the same as the one for Xn.
The Strong Markov Property
Theorem (Strong Markov Property)
Let Xn denote a Markov Chain. If T is a stopping time andYn := XT+n, then Yn is also a Markov chain. Moreover, thetransition probability for Yn is the same as the one for Xn.
What does this mean? Equivalently, it means that
P[Yn+1 = αn+1, Yn = αn, . . . Y1 = α1]
is equal to p(αn, αn+1)p(αn−1, αn) . . . p(α1, α2)P(Y1 = α1).
The Strong Markov Property
Proof
Let us analyze first P[XT+n = xn, . . . , XT+1 = x1, XT = x].
To do this, we first note that
P[XT+n = xn, . . . , XT+1 = x1, XT = x]
=
∞∑t=1
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]
The Strong Markov Property
Proof
Fix t ∈ N. Consider the sets
Ht = {α = (α1, . . . , αt) | X1 = α1, . . . , Xt = αt ⇒ T = t}Ht,x = {α ∈ Ht | αt = x}
In words: Ht is the possible trajectories for the chain, up totime t, for which the T = t; and Ht,x are those trajectories upto time t, with T = t, ending in x.
The Strong Markov Property
Proof
In particular, this means that
{XT = x, T = t} =⋃
α∈Ht,x
{X1 = α1, X2 = α2, . . . , Xt = αt}
The Strong Markov Property
Proof
Therefore, the probability
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]
is given by the sum of the probabilities
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = αt, . . . , X1 = α1]
The sum being over α ∈ Ht,x
The Strong Markov Property
Proof
Therefore, the probability
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]
is given by the sum of the probabilities
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = αt, . . . , X1 = α1]
for all α ∈ Ht,x
The Strong Markov Property
Proof
Now, we know that thanks to the Markov property,
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = αt, . . . , X1 = α1]
is equal to
p(x, x1)p(x1, x2) . . . p(xn−1, xn)P(Xt = αt, . . . X1 = α1)
The Strong Markov Property
Proof
Adding up the α’s we see that
P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]
is equal to
p(x, x1)p(x1, x2) . . . p(xn−1, xn)P[Xt = x, T = t]
The Strong Markov Property
Proof
. . . adding up in t, and dividing by P(XT = x) we see that
P[XT+n = xn, . . . , XT+1 = x1 | XT = x]
is equal to
p(x, x1)p(x1, x2) . . . p(xn−1, xn)
and it follows that XT+n is a Markov Chain with sametransition probability as Xn.
Last timeVisiting Times
Given a state y ∈ S, let
N(y) := #{n ≥ 1 | Xn = y}
That is, N(y) is the random variable given as the number oftimes n for which the Markov chain is at the state y. Inparticular, if y is visited infinitely many times, then N(y) =∞.
We are going to prove a few useful facts about N(y).
Last timeVisiting Times
First, note that
Px(N(y) ≥ 1) = 1− Px(N(y) = 0) = ρxy
Recall that ρxy = Px(Ty <∞), the probability of eventuallyvisiting y starting from x.
Let us computer the probabilities for the other values of N(y)!
Last timeVisiting Times
First, note that
Px(N(y) ≥ 1) = 1− Px(N(y) = 0) = ρxy
Recall that ρxy = Px(Ty <∞), the probability of eventuallyvisiting y starting from x.
Let us computer the probabilities for the other values of N(y)!
Last timeVisiting Times
First, note that
Px(N(y) ≥ 1) = 1− Px(N(y) = 0) = ρxy
Recall that ρxy = Px(Ty <∞), the probability of eventuallyvisiting y starting from x.
Let us computer the probabilities for the other values of N(y)!
Last timeVisiting Times
We showed that for any k ∈ N
Px[N(y) ≥ k] = Px[T (k)y <∞] = ρxyρ
k−1yy .
From here, one can compute the distribution of N(y)
Px[N(y) = k] = Px[N(y) ≥ k]− Px[N(y) = k + 1]
= ρxyρk−1yy − ρxyρkyy
= ρxyρk−1yy (1− ρyy)
Visiting Times
There is a simple formula for the expected number of visits
Lemma
Ex[N(y)] =ρxy
1− ρyy
Visiting Times
We use the following general (and useful!) fact about theexpectation of a real variable Y
E[Y ] =
∞∑k=1
P(Y ≥ k).
Why is this true?
Let χA denote the random variable which is equal to 1 if theeven happens, and 0 otherwise
χA :=
{1 if A happens0 otherwise
Visiting Times
We use the following general (and useful!) fact about theexpectation of a real variable Y
E[Y ] =
∞∑k=1
P(Y ≥ k).
Why is this true?Let χA denote the random variable which is equal to 1 if theeven happens, and 0 otherwise
χA :=
{1 if A happens0 otherwise
Visiting Times
If a random variable Y takes only the values k = 1, 2, . . ., wemay write it as
Y =
∞∑k=1
χ{Y≥k}
In which case, taking expectation on both sides
E[Y ] =
∞∑k=1
E[χ{Y≥k}]
But, it is clear that
E[χA] = P(A)
and we obtain the formula.
Visiting Times
Proof of the Lemma.
We use the formula for expectation we just obtained, it yields
Ex[N(y)] =
∞∑k=1
Px(N(y) ≥ k)
=
∞∑k=1
ρxyρk−1yy
Therefore, we have
Ex[N(y)] = ρxy
∞∑k=0
ρkyy
Visiting Times
Proof of the Lemma.
We use the formula for expectation we just obtained, it yields
Ex[N(y)] =
∞∑k=1
Px(N(y) ≥ k)
=
∞∑k=1
ρxyρk−1yy
Therefore, we have
Ex[N(y)] = ρxy
∞∑k=0
ρkyy
Visiting Times
Proof of the Lemma.
We use the formula for expectation we just obtained, it yields
Ex[N(y)] =
∞∑k=1
Px(N(y) ≥ k)
=
∞∑k=1
ρxyρk−1yy
Therefore, we have
Ex[N(y)] = ρxy
∞∑k=0
ρkyy
Visiting Times
Proof (continued).
Recall that if t ∈ (0, 1), then
1
1− t= 1 + t+ t2 + t3 + . . .
We obtain, using the above with t = ρyy,
Ex[N(y)] =ρxy
1− ρyyif ρyy < 1
Ex[N(y)] =∞ if ρyy = 1
Visiting Times
Proof (continued).
Recall that if t ∈ (0, 1), then
1
1− t= 1 + t+ t2 + t3 + . . .
We obtain, using the above with t = ρyy,
Ex[N(y)] =ρxy
1− ρyyif ρyy < 1
Ex[N(y)] =∞ if ρyy = 1
Visiting Times
A few takeaways from this formula: as long as x→ y, Ex[N(y)]gives us information about whether y is recurrent or not.
There is a completely different, formula for Ex[N(y)].
Visiting Times
Proof.
Notice that we may write,
N(y) =
∞∑n=1
χ{Xn=y}
But
Ex[χ{Xn=y}] = pn(x, y)
Therefore,
Ex[N(y)] =
∞∑n=1
pn(x, y)
Visiting Times
Combining these two formulas tell us the following.If y is a transient state, then for any state x
∞∑n=1
pn(x, y) <∞
and in particular, for a transient y the n-step transitionprobability from x to y goes to zero as n goes to infinity!
limn→∞
pn(x, y) = 0
Question:Does pn(x, y) have a limit as n→∞ for recurrent y?
Visiting Times
Combining these two formulas tell us the following.If y is a transient state, then for any state x
∞∑n=1
pn(x, y) <∞
and in particular, for a transient y the n-step transitionprobability from x to y goes to zero as n goes to infinity!
limn→∞
pn(x, y) = 0
Question:Does pn(x, y) have a limit as n→∞ for recurrent y?
Convergence Theorem
Theorem
Consider an irreducible, aperiodic chain, and let π(y) denote itsstationary distribution.
Then, for any y ∈ S, we have
limn→∞
pn(x, y) = π(y) ∀ x ∈ S.
This is great, but what does this aperiodicity refer to?!.
Convergence Theorem
Theorem
Consider an irreducible, aperiodic chain, and let π(y) denote itsstationary distribution.
Then, for any y ∈ S, we have
limn→∞
pn(x, y) = π(y) ∀ x ∈ S.
This is great, but what does this aperiodicity refer to?!.
Period of a state
Definition: Given a Markov Chain, the period of a state x isthe largest common divisor of the numbers in the set
Ix := {n ∈ N | pn(x, x) > 0}
Example: Remember the Ehrenfest chain? For any state inthis chain pn(x, x) > 0 if and only if n is even. Thus, theperiod of every state is 2.
Example: If x is such that p2(x, x) > 0 and p3(x, x) > 0 thenthe period of x is 1.
Period of a state
Definition: Given a Markov Chain, the period of a state x isthe largest common divisor of the numbers in the set
Ix := {n ∈ N | pn(x, x) > 0}
Example: Remember the Ehrenfest chain? For any state inthis chain pn(x, x) > 0 if and only if n is even. Thus, theperiod of every state is 2.
Example: If x is such that p2(x, x) > 0 and p3(x, x) > 0 thenthe period of x is 1.
Period of a stateProperties of the period
• The set Ix is closed under sums, that is, if n ∈ Ix and m ∈ Ix,then n+m ∈ Ix.
• If p(x, x) > 0, then x has period equal to 1.
• If x 7→ y and y 7→ x then x and y have the same period. Inparticular, in an irreducible chain, all states have the sameperiod.
• If x has period 1, then there is a number n0 such thatpn(x, x) > 0 for every number n ≥ n0.
Period of a stateProperties of the period
• The set Ix is closed under sums, that is, if n ∈ Ix and m ∈ Ix,then n+m ∈ Ix.
• If p(x, x) > 0, then x has period equal to 1.
• If x 7→ y and y 7→ x then x and y have the same period. Inparticular, in an irreducible chain, all states have the sameperiod.
• If x has period 1, then there is a number n0 such thatpn(x, x) > 0 for every number n ≥ n0.
Period of a stateProperties of the period
• The set Ix is closed under sums, that is, if n ∈ Ix and m ∈ Ix,then n+m ∈ Ix.
• If p(x, x) > 0, then x has period equal to 1.
• If x 7→ y and y 7→ x then x and y have the same period. Inparticular, in an irreducible chain, all states have the sameperiod.
• If x has period 1, then there is a number n0 such thatpn(x, x) > 0 for every number n ≥ n0.
Period of a stateProperties of the period
• The set Ix is closed under sums, that is, if n ∈ Ix and m ∈ Ix,then n+m ∈ Ix.
• If p(x, x) > 0, then x has period equal to 1.
• If x 7→ y and y 7→ x then x and y have the same period. Inparticular, in an irreducible chain, all states have the sameperiod.
• If x has period 1, then there is a number n0 such thatpn(x, x) > 0 for every number n ≥ n0.