![Page 1: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/1.jpg)
Economics 2010c: Lectures 9-10Bellman Equation in Continuous Time
David Laibson
9/30/2014
![Page 2: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/2.jpg)
Outline Lectures 9-10:
9.1 Continuous-time Bellman Equation
9.2 Application: Merton’s Problem
9.3 Application: Stopping problem
9.4 Boundary condition: Value Matching
9.5 Boundary condition: Smooth Pasting
10.1 Stopping problem revisited
10.2 Solving second order ODE’s
10.3 Stopping problem resolved
![Page 3: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/3.jpg)
1 Continuous-time Bellman Equation
Let’s write out the most general version of our problem. Begin with equationof motion of the state variable:
= ( )+ ( )
Note that depends on choice of control .
Using Ito’s Lemma, derive continuous time Bellman Equation:
( ) = ( ∗ ) +
+
( ∗ ) +
1
2
2
2( ∗ )2
∗ = ( ) = optimal value of control variable
![Page 4: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/4.jpg)
• Note that value function is a second order partial differential equation(PDE).
• is the ‘dependent’ variable and and are the ‘independent’ variables.
• To solve this PDE need ‘boundary conditions,’ since many solutions exist.
![Page 5: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/5.jpg)
1.1 Terminal condition.
• Suppose problem ‘ends’ at date
• Then we know that
( ) = Ω( ) ∀
• Can solve the problem using techniques analogous to backwards induction
![Page 6: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/6.jpg)
1.2 Stationary ∞−horizon problem.
• Now value function doesn’t depend on
() = ( ∗) + ( ∗) 0 +1
2( ∗)2 00
• We have a second-order ordinary differential equation (ODE).
• In general a large class of functions are consistent with this ODE.
• To pin down a solution we need to know something about the economicsof the value function
• This will provide constraints that pick out a single solution.
![Page 7: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/7.jpg)
2 Application: Merton’s consumption problem
• Consumer has CRRA utility: () = 1−1−
• Consumer has two assets.
• Risk free: return
• Equity: return + and proportional variance 2
• Consumer invests asset share in equities and consumes at rate
= [( + )− ]+
so, ( ) = [( + )− ] and ( ) =
![Page 8: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/8.jpg)
• Bellman Equation and Ito’s Lemma:
( ) = max
()+( )
max
(()+
"
+
( ) +
1
2
2
2( )2
#
)
• For this problem, doesn’t depend directly on (why?):
() = max
(()+
"
[( + )− ] +
1
2
2
2()2
#
)
• Boundary condition: () = 1−1− So,
1−
1− = max
½()+
∙−[( + )− ]−
2−−1()2
¸¾
![Page 9: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/9.jpg)
• First-order conditions:
−− 2µ
2
¶−−1()2 = 0
0() = − = −
• Simplification implies: =
2
= −1
• Plugging this back into last equation on previous page implies:
−1 =
+
Ã1− 1
!Ã +
2
22
!
![Page 10: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/10.jpg)
• Interesting case: = 1.
• = So ' 005
• = 2
= 0061×(016)2
= 234!
• How many households place 2.34 times their wealth (including humancapital) in the stock market?
![Page 11: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/11.jpg)
3 Application: Stopping problem
• Every instant the firm decides whether to continue and get instantaneousflow payoff, ( ) or to stop and get termination payoff, Ω( )
• We’ll assume that ( ) is increasing in
• Continuous time value function is given by (limit as ∆→ 0):
( ) = maxn( )∆+ (1 + ∆)−1 (0 0)Ω( )
o
• Assume that
= ( )+ ( )
![Page 12: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/12.jpg)
• The solution to this problem is a stopping rule
if ∗() continueif ≤ ∗() stop
• Motivation: Irreversibly closing a production facility.
![Page 13: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/13.jpg)
• Distinguish continuation region and stopping region of state space.
• The stopping region is the set of points h i such that () ≤ ∗()
• In continuation region, use Ito’s Lemma to characterize the value function:
( ) = ( )+( )
= ( )+
"
+
( ) +
1
2
2
2( )2
#
• We need to solve this partial differential equation.
![Page 14: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/14.jpg)
4 Value Matching
Recall that Ω( ) is the termination payoff.
One set of boundary conditions is
lim→∗()
( ) = Ω(∗() )
for all boundary points, h∗() i.
This is referred to as a value matching condition (continuity of the value func-tion at the boundary).
![Page 15: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/15.jpg)
Heuristic proof. There exists a neighborhood (∗ ∗ + ) in which:
( ) = ( )∆+ (1 + ∆)−1 (0 0)
= ( )∆+ (1 + ∆)−1" (+
√∆ 0) +Ω(−
√∆ 0)
2
#
=1
2lim
↓∗() ( ) +
1
2Ω(∗() ) +
= lim↓∗()
( ) +1
2
"Ω(∗() )− lim
↓∗() ( )
#+
This implies that
lim↓∗()
( ) = lim↓∗()
( ) +1
2
"Ω(∗() )− lim
↓∗() ( )
#
which implies
lim↓∗()
( ) = Ω(∗() )
![Page 16: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/16.jpg)
5 Smooth Pasting
We could now apply backwards induction techniques if we knew the “freeboundary” ∗().
To pin down the free boundary, we need another boundary condition, which isderived from optimization and called smooth pasting:
(∗() ) = Ω(
∗() )
![Page 17: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/17.jpg)
Heuristic proof. Suppose slopes to left and right of stopping point are givenby and + where ≥ 0 This is a convex kink. If you stop now you getpayoff Ω If you wait another instant and then stop, you get payoff:
∆+ (1 + ∆)−1"Ω−
√∆
2+(+ )
√∆
2
#
So the net payoff of waiting is
∆+ (1 + ∆)−1"Ω−
√∆
2+(+ )
√∆
2
#−Ω
= ∆+ (1 + ∆)−1"Ω+
√∆
2
#− Ω
Multiply through by (1+∆) simplify and remove terms of at least order ∆
to find:
(1 + ∆)∆+
"Ω+
√∆
2
#− (1 + ∆)Ω =
√∆
2
![Page 18: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/18.jpg)
Intuition: At the boundary, the agent should be indifferent between continuationand stopping. If value functions don’t smooth paste at ∗(), then stopping at∗() can’t be optimal. Better to stop an instant later. If there is a (convex)kink at the boundary, then the gain from waiting is in
√∆ and the cost from
waiting is in ∆ So there can’t be a kink at the boundary. Hence:
(∗() ) = Ω(
∗() )
How would you rule out concave kinks?
![Page 19: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/19.jpg)
Outline Lectures 9-10:
9.1 Continuous-time Bellman Equation
9.2 Application: Merton’s Problem
9.3 Application: Stopping problem
9.4 Boundary condition: Value Matching
9.5 Boundary condition: Smooth Pasting
10.1 Stopping problem revisited
10.2 Solving second order ODE’s
10.3 Stopping problem resolved
![Page 20: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/20.jpg)
6 Stopping Problem Revisited
Assume that a continuous time stochastic process () is an Ito process,
= +
You might imagine that is the price of a commodity produced by this firm.While in operation, the firm has flow profit
() =
Assume that the firm can always costlessly (permanently) exit the industry andrealize termination payoff
Ω = 0
Intuitively, this firm will have a stationary stopping rule,
if ∗ continueif ≤ ∗ stop (exit)
![Page 21: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/21.jpg)
Let 0 = (+∆) and let represent the interest rate. So
() = maxn∆+ (1 + ∆)−1 (0) 0
oIn the stopping region,
() = 0
In the continuation region,
() = ∆+ (1 + ∆)−1 (0)
(1 + ∆) () = (1 + ∆)∆+ (0)
()∆ = (1 + ∆)∆+ (0)− ()
Multiply out and let ∆→ 0 Terms of order 2 = 0
() = +( ) (*)
![Page 22: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/22.jpg)
Now substitute in for ( ) using Ito’s Lemma:
( ) =
"
+
( ) +
1
2
2
2( )2
#
=
" 0 +
2
2 00
#
Substituting this expression into equation (*), we find
() = +
" 0 +
2
2 00
#
which is a second-order ordinary differential equation,
= + 0 +2
2 00
![Page 23: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/23.jpg)
What are our boundary conditions?
Value matching:
(∗) = 0
Smooth pasting:
0(∗) = 0
As → ∞ the option value of exiting goes to zero, so converges to thevalue associated with the policy of never exiting the industry. Hence,
lim→∞
()1
³+
´ = 1We’ll derive this equation later. We could also have written
lim→∞ 0() =
1
![Page 24: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/24.jpg)
7 Solving second order ODE’s
To solve the differential equations that come up in economics, it is helpful torecall a few general results from the theory of differential equations.
Consider a generic second order ordinary differential equation:
00() +() 0() +() () = ()
This equation is referred to as the “complete equation.” Note that (),(), and (), are given functions. We are trying to solve for withindependendent variable .
Now consider a “reduced equation” in which () is replaced by 0.
00() +() 0() +() () = 0
Solving this reduced differential equation will enable us to solve the completeequation. We begin by characterizing the solution of the reduced equation.
![Page 25: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/25.jpg)
Theorem 7.1 Any solution, (), of the reduced equation can be expressedas a linear combination of any two solutions of the reduced equation, 1 and2 that are linearly independent.
() = 11() + 22()
Note that two solutions are linearly independent if there do not exist constants1 and 2 such that
11() +22() = 0 for all
Theorem 7.2 The general solution of the complete equation is the sum of anyparticular solution of the complete equation and the general solution of thereduced equation.
![Page 26: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/26.jpg)
8 Stopping problem resolved
Recall the differential equation that characterizes the continuation region
= + 0 +2
2 00 (complete equation)
Consider the reduced equation,
0 = − + 0 +2
2 00 (reduced equation)
Our first challenge is to find solutions of this equation. Consider the class,
To confirm that this is in fact a solution, differentiate and plug in to find
0 = − + +2
22
![Page 27: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/27.jpg)
This implies that
0 = −+ +2
22
Apply quadratic formula,
=−±
q2 + 22
2
Let + represent the positive root and let − represent the negative root. Anysolution to the reduced equation can be expressed
++ + −
− (general solution to the reduced equation)
![Page 28: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/28.jpg)
The general solution to the complete equation can be expressed as the sumof a particular solution to the complete differential equation and the generalsolution to the reduced equation.
To find a particular solution, consider the payoff function of the policy “neverleave the industry.” The value of this policy is
Z ∞0
−()
Note that
[()] = ∙(0) +
Z
0()
¸=
∙(0) +
Z
0[+ ()]
¸=
∙(0) + +
Z
0()
¸= (0) +
![Page 29: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/29.jpg)
The value of the policy “never leave” is derived with integration by parts:
Z ∞0
−() =Z ∞0
− [(0) + ]
=(0)
+
"−1−
#∞0
−Z ∞0−1−
=1
Ã(0) +
!Hence, our (candidate) particular solution takes the form
() =1
Ã+
!
Confirm that this is a solution to the complete differential equation.
·"1
Ã+
!#= + · 1
+2
2· 0 X
![Page 30: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/30.jpg)
We now can draw all of these pieces together. First, we have our generalsolution to the reduced equation
++ + −
−
where the roots are given by
+ − =−±
q2 + 22
2
Second, we have our particular solution:
1
Ã+
!
So the general solution of the complete equation is
() =1
Ã+
!+ +
+ + −−
![Page 31: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/31.jpg)
Finally, we have boundary conditions.
Value matching:
(∗) = 0
Smooth pasting:
0(∗) = 0
![Page 32: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/32.jpg)
We know
lim→∞ 0() =
1
which implies + = 0 Value matching and smooth pasting imply
(∗) =1
Ã∗ +
!+ −
−∗ = 0 (1)
0(∗) =1
+ −−
−∗ = 0 (2)
Equation (1) implies
−−∗ = −1
Ã∗ +
!
![Page 33: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/33.jpg)
Plugging this expression into equation (2) implies
1
− −
Ã∗ +
!= 0
Hence,
∗ =1
−−
We have,
− =−−
q2 + 22
2
Hence,
∗ = − 2
+q2 + 22
−
0
![Page 34: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/34.jpg)
Some interesting special cases (see problem set):
∗(=0) = − √2
0
lim→∞∗ = −∞lim
→−∞∗ = 0
lim→0
∗ =
(− if ≥ 0
0 if 0
)lim→∞
∗ = −∞
lim→0
∗ =
(−∞ if ≥ 02
2 if 0
)lim→∞∗ = 0
![Page 35: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/35.jpg)
Illustrative calibration:
∗(=0) = − √2
= − √2× 02
= −5
Wait until the gets 5 standard deviations below the break-even threshold( = 0) before shutting down.
![Page 36: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/36.jpg)
Finally, now that we have our solution, it is easy to calculate the option valueof stopping:
() = ()− 1
Ã+
!Substituting in for () yields
() =
⎧⎨⎩ −− if ≥ ∗
−1³+
´if ∗
⎫⎬⎭So as →∞ the option value of stopping goes to zero.
![Page 37: Economics 2010c: Lectures 9-10 Bellman Equation in ... · 9/30/2014 · • Note that value function is a second order partial differential equation (PDE). • is the ‘dependent’](https://reader033.vdocuments.net/reader033/viewer/2022050406/5f83b91e1aba97480b7b6d6d/html5/thumbnails/37.jpg)
Outline Lectures 9-10:
9.1 Continuous-time Bellman Equation
9.2 Application: Merton’s Problem
9.3 Application: Stopping problem
9.4 Boundary condition: Value Matching
9.5 Boundary condition: Smooth Pasting
10.1 Stopping problem revisited
10.2 Solving second order ODE’s
10.3 Stopping problem resolved