.

.

. ..

.

.

Theory of Stochastic Processes13. Review

Tomonari Seisei@mist.i.u-tokyo.ac.jp

Department of Mathematical Informatics, University of Tokyo

July 13, 2017

http://www.stat.t.u-tokyo.ac.jp/~sei/lec.html

1 / 13

Handouts & Announcements

Handouts:

Slides (this one)

Solved problems

Announcements:

About the final exam (reminder)

The final exam is held on July 20 (Thu), 10:25–, 90 min.Contact me if you cannot attend it due to some unavoidable reasons.The exam is open-book and open-note. Write your answer in English.The exam will cover the whole material up to July 6. At least onequestion will be from the topics before the midterm exam.

2 / 13

Outline today

.

. .

1 Review of last week’s material

.

. .

2 Solved problems

3 / 13

Review: Ito calculus

Let Wt be the standard Brownian motion.An Ito process Xt is defined by

dXt = µtdt + σtdWt .

The conditional distribution given Ft = {Ws}s≤t is approximately

Xt+∆t |Ft ∼ N(µt∆t, σ2t ∆t).

Xt is a martingale if µt = 0.

Ito’s formula:

d{f (t, Xt)} =∂f

∂tdt +

∂f

∂xdXt +

1

2

∂2f

∂x2(dXt)

2.

4 / 13

Diffusion processes

A diffusion process is a solution to the stochastic differential equation

dXt = µ(t, Xt)dt + σ(t, Xt)dWt .

Xt is Markov.

The future behavior depends only on the present value.

5 / 13

Markov processes

We encountered many examples of Markov processes.

discrete time continuous time

discrete state

5 10 15 20 25

12

34

56

timest

ate

0.0 0.1 0.2 0.3 0.4

02

46

8

time

stat

e

Markov chain Poisson process

continuous state

5 10 15 20 25

−2

−1

01

time

stat

e

0 200 400 600 800 1000

−3.

0−

2.0

−1.

00.

0

timest

ate

AR(1) process diffusion process

6 / 13

The forward equation

Denote the transition density from Xs = x to Xt = y by p(t, y |s, x).The forward equation (Fokker-Planck equation) is

∂p

∂t= − ∂

∂y(µp) +

1

2

∂2

∂y2(σ2p).

.

Key exercise

.

.

.

. ..

.

.

If µ and σ are constant, then Xt ∼ N(µt, σ2t). Write down the transitiondensity p(t, y) = p(t, y |0, 0) and check that the forward equation holds.

Proof sketch: Ito’s formula implies

dE [f (Xt)] = E [f ′(Xt)µ + (1/2)f ′′(Xt)σ2]dt

for any function f . Since E [f (Xt)] =∫

f (y)p(t, y |s, x)dy , we have∫f (y)

∂p

∂tdy =

∫{f ′(y)µ + (1/2)f ′′(y)}pdy .

The result follows from the integral-by-parts formula.7 / 13

Langevin Monte Carlo

The diffusion process

dXt =1

2(log π∗(Xt))

′dXt + dWt

has the stationary distribution π(x) = π∗(x)/Z .

Indeed, p(t, y) = π(y) satisfies the forward equation:

−(

1

2(log π∗)

′π

)′+

1

2π′′ = −

(1

2π′

)′+

1

2π′′ = 0.

MCMC using the diffusion process is called Langevin Monte Carlo.

8 / 13

Metropolis-Adjusted Langevin algorithm (MALA)

In practice, discretize the equation as

Xt+∆t = Xt + µ(Xt)∆t + ∆Wt , ∆Wt ∼ N(0,√

∆t)

(Euler-Maruyama scheme), where µ(x) = (1/2)(log π∗)′(x).

The conditional density of Xt+∆t given Xt = x is

q(y |x) =1√

2π∆texp

(−(y − x − µ(x)∆t)2

2∆t

).

It is used as a proposal density for the Metropolis-Hastings algorithm.

Specifically, the algorithm is

.

.

.

1 Set an initial value x .

.

.

.

2 Generate a random number y according to q(y |x).

.

.

.

3 Compute the acceptance probability a = min(1, π∗(y)q(x|y)π∗(x)q(y |x) ).

.

.

.

4 Genrate U ∼ U(0, 1). If U ≤ a, then x ← y . Otherwise, x ← x .

.

.

.

5 Go to step 2.

9 / 13

Example 1

Let π∗(x) =1

{1 + (x − 2)2}3{1 + (x + 2)2}3.

Sample paths

∆t = 0.01 without MH ∆t = 1 with MH

t ∈ [0, 100]0 20 40 60 80 100

−4

−2

02

4

time

X

0 20 40 60 80 100

−4

−2

02

4

time

Xt ∈ [0, 10000]

0 2000 4000 6000 8000 10000

−4

−2

02

4

time

X

0 2000 4000 6000 8000 10000−

4−

20

24

time

X

10 / 13

Example 2

Let π∗(x) = φ(x)Φ(x), φ(x) = e−x2/2/√

2π and Φ(x) =∫ x−∞ φ(z)dz .

Comparison of Langevin MC (left) and random walk MC (right),where the random walk MC uses Xt+∆t = Xt + ∆Wt as the proposal.

0 200 600 1000

−1

01

23

Langevin MC

Index

X

0 200 600 1000

−1

01

23

random walk MC

Index

X

0 5 10 15 20 25 30

0.0

0.4

0.8

Lag

AC

F

0 5 10 15 20 25 30

0.0

0.4

0.8

Lag

AC

F

11 / 13

The underlined problems are solved. See the handout.

June 8 (Lec 8) Markov chain Monte Carlo

§6.5: Problem 1, 8*, 9.§6.14: Problem 1.§6.15: Problem 2.

June 15 (Lec 9) Stationary processes

Problem 1, 2, 3, 4, 5, 6.

June 22 (Lec 10) Martingales

§12.1: Problem 1, 2, 3, 4, 5, 6, 7*, 8, 9*.§12.2: Problem 1, 2.

June 29 (Lec 11) Queues

§8.4: Problem 1, 2, 3, 4*, 5*.§11.2: Problem 3, 7*.§11.3: Problem 1* .

July 6 (Lec 12) Diffusion processes

Problem 1, 2, 3, 4, 5, 6, 7, 8, 9.

12 / 13

Final remark

Before After

We have lost fresh flowers but obtained many leaves of knowledge!

Thanks

13 / 13