topic 1: introduction to intertemporal optimizationyluo/teaching/6012_2019/lecture1b.pdffeasible....
TRANSCRIPT
Topic 1: Introduction to Intertemporal Optimization
Yulei Luo
FBE, HKU
September 20, 2019
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 1 / 59
Warm-Up: Intertemporal/Dynamic Optimization
In static optimization, the task is to �nd a single value for eachcontrol variable, such that the objective function will be maximized orminimized. In contrast, in a dynamic setting, time enters explicitlyand we encounter a dynamic optimization problem. In such aproblem, we need to �nd the optimal time path of control and stateduring an entire planning period.
Speci�cally, at any time t, we have to choose the value of somecontrol variable, c(t), which will then a¤ect the value of the statevariable, x(t + 1), via a state transition equation given the currentstate x(t).
Two leading methods to solve DO problems are: optimal control (OC)and dynamic programming (DP). They can be applied in deterministicor stochastic and discrete-time or continuous-time settings.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 2 / 59
The Setting of Optimal Control (OC)
In the dynamic discrete-time model, the objective should take theform of summation from t = 0 to t = T :
maxfc (t),x (t+1)g
T
∑t=0r (c (t) , x (t) , t) , (1)
subject to
x (t + 1)� x (t) = f (c (t) , x (t) , t) , (2)
x0 = x (0) , x (T + 1) � 0 , (3)
for any t � 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 3 / 59
(Conti.) As in the static optimization case, the Lagrangian for thisintertemporal (dynamic) problem is
L =T
∑t=0fr (ct , xt , t) + λt [f (ct , xt , t) + xt � xt+1]g+ λT+1xT+1,
(4)where λt is the shadow price (i.e., the Lagrange multiplier).
The FOCs for ct are:
∂L∂ct
= r 0c (ct , xt , t) + λt f 0c (ct , xt , t) = 0 for t = 0, 1, � � �,T . (5)
The FOCs for xt+1 are complicated because each xt appears in twoterms in the Lagrangian function.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 4 / 59
(Conti.) Consider some relevant terms in the above Lagrangianfunction:
T
∑t=0
λt (xt � xt+1) = (λ0x0 � λT xT+1) +T�1∑t=0
xt+1 (λt+1 � λt ) .
The FOCs for xt+1:
∂L∂xt+1
= r 0x (ct+1, xt+1, t + 1) + λt+1f 0x (ct+1, xt+1, t + 1)
+λt+1 � λt = 0,
for t = 0, � � �,T � 1.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 5 / 59
Theorem (The Maximum Principle)
The necessary FOCs for maximizing (1) subject to (2) are:(i) ∂Lt
∂ct= 0 for all t = 0, � � �,T .
(ii) xt+1 � xt = f (ct , xt , t) for all t (The state transition equation)(iii) r 0x (ct+1, xt+1, t + 1) + λt+1f 0x (ct+1, xt+1, t + 1) + λt+1 � λt = 0 forall t (The costate (λt ) equation)(iv) xT+1 � 0,λT+1 � 0, xT+1λT+1 = 0 (The transversality condition, orthe complementary-slackness condition)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 6 / 59
Example: Optimal Consumption-Saving Model
Suppose that a consumer has the utility function u (ct ) , where ct isconsumption at time t. The consumer�s utility is concave:
u0 > 0, u00 < 0, (6)
and u (ct ) satis�es the Inada conditions (They ensure thatconsumption will always be interior):
limc!0
u0 (c) = ∞ and limc!∞
u0 (c) = 0. (7)
The consumer is also endowed with a0 = a (0) , and has incomestream derived from holding the asset: yt = rat , where r is theinterest rate. The consumer uses the income to purchase c . Anyincome not consumed is added to the asset holdings as savings:
at+1 = (1+ r) at � ct . (8)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 7 / 59
The Model Setup
The consumer�s lifetime utility maximization problem is to
maxfct ,at+1g
T
∑t=0
βtu (ct ) (9)
subject to
at+1 = Rat � ct , t = 0, � � �,T , (10)
a0 = a (0) , aT+1 � 0, (11)
where β is the consumer�s rate of time preference (β � 1) andR = 1+ r is the gross interest rate.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 8 / 59
(Conti.) Since limc!0 u0 (c) = ∞, c (t) > 0 for all t � T . Further, toensure that a solution exists, assets are constrained to be nonnegativeafter the terminal period T (i.e., when the consumer dies), aT+1 � 0.Note that this nonnegativity constraint must bind, i.e., aT+1 = 0. Infact, this constraint serves two important purposes:
First, without this constraint aT+1 � 0, the consumer would want toset aT+1 = �∞ and die with outstanding debts. This is clearly notfeasible.Second, the fact that the constraint is binding in the optimal solutionguarantees that resources are not being thrown away after T .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 9 / 59
Using Optimal Control to Solve the Model
Setting up the Lagrangian:
L =T
∑t=0
βt fu(ct ) + λt [Rat � ct � at+1]g+ λT+1aT+1 (12)
The FOC for ct is ∂L∂ct= 0:
u0(ct )� λt = 0 for t = 0, 1, � � �,T . (13)
The FOC for at+1 is ∂L∂at+1
= 0:
�λt + βRλt+1 = 0 for t = 0, � � �,T � 1. (14)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 10 / 59
(Conti.) The terminal condition aT+1 � 0 implies that thetransversality condition (the complementary slackness condition) is
λT+1 � 0, aT+1 � 0, aT+1λT+1 = 0, (15)
which means that either the asset holdings (a) must be exhausted onthe terminal date, or the shadow price of capital (λt ) must be 0 onthe terminal date. Since u0 > 0, the marginal value of capital (λ)cannot be 0 and thus the capital stock should optimally be exhaustedby the terminal date T + 1, i.e., aT+1 = 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 11 / 59
(Conti.) Combining (13) with (14) gives
u0(ct ) = βRu0(ct+1), (16)
which is called the Euler equation characterizing optimal consumptiondynamics.
Suppose that the utility function has an isoelastic form
u(ct ) =c1�γt � 11� γ
, (17)
where γ � 0 (Note that when γ = 1, the function reduces to ln ct).We have
c�γt = βRc�γ
t+1 =)ct+1ct
= (βR)1/γ . (18)
Therefore, if βR > 1, the optimal consumption will rise over time; ifβR < 1, the optimal consumption will decline over time. WhenβR = 1, ct = ct+1.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 12 / 59
(Conti.) (18) implies that the elasticity of intertemporal substitution(EIS) is
d�ct+1ct
�/�ct+1ct
�dR/R
=d ln
�ct+1ct
�d lnR
=1γ. (19)
Hence, increasing γ�i.e., reducing 1
γ
�make the agent more unwilling
to postpone consumption (i.e., more unwilling to save). That�s whywe call this type of utility functions the isoelastic utility function.
The optimal problem is pinned down by a given initial condition (a0)and by a terminal condition (aT+1 = 0) . The sets of (T � 1) Eulerequations (18) and constraints (10) then determine the time path ofc as well as a.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 13 / 59
Obtaining the Intertemporal Budget Constraint
The original budget constraint (10) implies that
a1 = Ra0 � c0,a2 = Ra1 � c1,
� � � � �aT+1 = RaT � cT .
aT+1RT
+� cTRT
+ � � �+ c1R+ c0
�= Ra0 =)
T
∑t=0
ctR t
= Ra0, (20)
where we use the fact that aT+1 = 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 14 / 59
(Conti.) For simplicity, assume that γ = 1. Combining (18) with (20)gives
c0 +1R(βRc0) + � � �+
1RT
�βTRT c0
�= Ra0 =)
c0 =R
1+ β+ � � �+ βTa0. (21)
Hence, the time path of optimal consumption is
ct =R t+1βt
1+ β+ � � �+ βTa0 (22)
where t = 0, � � �,T . Given ct , it is straightforward to determine thetime path of at :
at+1 = Rat � ct = Rat �R t+1βt
1+ β+ � � �+ βTa0, (23)
where t = 0, � � �,T and aT+1 = 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 15 / 59
(Conti.) Consider an important special case, βR = 1. In this case,(18) implies that c0 = c1 = � � � = cT ,
c0
�1+ � � �+ 1
RT
�= Ra0 =)
ct =R � 1
1� R�(T+1)a0 . (24)
where t = 0, � � �,T .Next, the optimal path of at can be derived by solving the followingdi¤erence equation:
at+1 = Rat � ct = Rat �r
1� R�(T+1)a0 =)
at =
�1� 1
1� R�(T+1)
�a0R t +
11� R�(T+1)
a0, (25)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 16 / 59
Alternative Way to Solve Optimal Consumption
Instead of de�ning the Lagrange multiplier for each �ow budgetconstraint, we may de�ne a Lagrange multiplier for the intertemporalbudget constraint (that is, the lifetime budget constraint):
T
∑t=0
ctR t= Ra0,
where we have used the fact that aT+1 = 0. That is, we may writethe Lagrangian as follows
L =T
∑t=0
βtu(ct ) + λ
Ra0 �
T
∑t=0
ctR t
!,
where λ is the constant Lagrange multiplier for the lifetime budgetconstraint.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 17 / 59
(Conti.) The FOCs for an optimum are then
βtu0(ct ) = λ1R t, where t = 0, � � �,T . (26)
Since λ is a constant, the above FOCs implies that the Eulerequations are
u0(ct ) = βRu0(ct+1), where t = 0, � � �,T � 1,
which is identical to those Euler equations we derived above.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 18 / 59
The In�nite Horizon Case
There are some reasons to consider the in�nite horizon (T = ∞) case:
People don�t live forever, but they may care about their o¤spring (abequest motive).Assuming an in�nite horizon eliminates the terminal date complications(stop saving, consume their entire value, etc. There is no real worldcounterpart for these actions because the real economy continueforever).Many economic models with a long time horizon tend to show verysimilar results to IH models if the horizon is long enough.IH models are stationary in nature: the remaining time horizon doesn�tchange as we move forward in time.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 19 / 59
The Model Setup
The consumer�s lifetime utility maximization problem
maxc (t)
∞
∑t=0
βtu (ct ) (27)
subject to at+1 = Rat � ct , 8t � 0, given the initial asset holdingsa0 = a (0) .
In addition, we need to impose the following institutional assumption:no Ponzi game condition (nPg):
limt!∞
�at+1R t
�� 0, (28)
means that in present value terms, the agent cannot engage inborrowing and lending so that his terminal asset holdings are negative.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 20 / 59
(Conti.) (28) can help obtain an intertemporal budget constraint:
T
∑t=0
ctR t+aT+1RT
= Ra0.
Applying (28), we have
limT!∞
T
∑t=0
ctR t+aT+1RT
!= lim
T!∞
T
∑t=0
ctR t+ limT!∞
�aT+1RT
�=
∞
∑t=0
ctR t+ limT!∞
�aT+1RT
�=)
∞
∑t=0
ctR t� Ra0. (29)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 21 / 59
(Conti.) In an optimum, (29) must be binding:
∞
∑t=0
ctR t= Ra0 (30a)
because (28) is binding, i.e., limT!∞� at+1R t�= 0 (Otherwise, the
agent can consume the left amount and reach a higher utility level,which contradicts optimum).
Set up the Lagrangian:
L =∞
∑t=0
βtu (ct ) + λ
Ra0 �
∞
∑t=0
ctR t
!The FOCs w.r.t. ct for any t are
βtu0 (ct )� λ1R t= 0, 8t � 0, (31)
where λ is the Lagrangian multiplier associated with theintertemporal budget constraint.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 22 / 59
(Conti.) Note that the FOCs imply the following Euler equationlinking consumption in two consecutive periods
u0 (ct ) = βRu0 (ct+1) , 8t � 0. (32)
Suppose that the utility function is log, u (ct ) = log ct . (31) impliesthat ct = (βR)
t c0, 8t � 1. Substituting them into the (30a) gives
∞
∑t=0
(βR)t
R tc0 = Ra0 =)
∞
∑t=0
βt
!c0 = Ra0 =)
c0 = R (1� β) a0, (33)
and consumption in periods t � 1 can be recovered:
ct = (βR)t c0 = (βR)
t R (1� β) a0, 8t � 1. (34)
Similarly, we can determine the optimal time path of asset holdings:
at+1 = Rat � (βR)t R (1� β) a0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 23 / 59
Dynamic Programming (DP): Application to theConsumption-Savings Model
We use the same in�nite-horizon lifetime optimalconsumption-savings model to introduce DP and then provide moregeneral discussions on DP.Consider the maximization problem
Ut =∞
∑s=t
βs�tu (cs )
subject to as+1 = Ras � cs , 8s � t, given the initial asset holdingsat = a (t). In addition, impose:
limT!∞
�at+T+1R t+T
�� 0.
Of course, DP is also applicable with a �nite horizon.Assume that there is a function, called the value function, that givesus the maximal constrained value of Ut as a function of the initialwealth at . We write the value function as J (at ), and we assume it isdi¤erentiable.Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 24 / 59
(Conti.) DP rests on a recursive equation involving the valuefunction. That equation, called the Bellman equation, characterizesintertemporally optimizing plans:
A consumption plan optimal from the standpoint of t must maximizeUt+1 subject to the future wealth at+1 produced by today�sconsumption plan ct . If not, the consumer could raise utility bybehaving di¤erently after t.The foregoing property means that an optimizing consumer can behaveas if Ut = u (ct ) + βJ (at+1), where J (at+1) is the constrainedmaximal value of Ut+1.If this is so, ct maximizing lifetime utility is the one maximizingUt = u (ct ) + βJ (at+1) subject to the same constraint.
Formally, the Bellman equation can be written as
J (at ) = maxctfu (ct ) + βJ (at+1)g , (35)
s.t. at+1 = Rat � ct . (36)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 25 / 59
(conti.) (35) can be rewritten as:
J (at ) = maxctfu (ct ) + βJ (Rat � ct )g . (37)
The �rst-order condition is thus
u0 (ct )� βJ 0 (Rat � ct ) = 0. (38)
Note that (38) gives optimal consumption as an implicit function ofcurrent wealth, c = c (a). Substituting it into (37):
J (a) = u (c (a)) + βJ (Ra� c (a)) . (39)
Taking di¤erentiation w.r.t. a gives
J 0 (a) = u0 (c (a)) c 0 (a) + βJ 0 (Ra� c (a))�R � c 0 (a)
�, or
J 0 (a) = βRJ 0 (Ra� c (a)) , or J 0 (a) = Ru0 (c) , (40)
which is called the Envelop theorem.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 26 / 59
(conti.) Implication: For an optimizing consumer, an increment towealth on any date has the same e¤ect on lifetime utility regardless ofhow to put the wealth: consumption or saving.Combining (38) with (38), we obtain the usual consumption Eulerequation:
u0 (ct ) = βRu0 (ct+1) . (41)
Following the same procedure used above, we can easily solve foroptimal consumption path.Alternatively, we can solve for optimal consumption by using thevalue function. Given the same utility function:
u(ct ) =c1�γt
1� γ, (42)
we �rst guess the form of the value function and then verify that theguess was right. A natural guess is that the value function takes aparallel form as u(ct )
J (a) = Aa1�γ
1� γ. (43)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 27 / 59
(conti.) Substituting this conjectured solution intou0 (ct )� βJ 0 (Rat � ct ) = 0,
c�γt = βA (Rat � ct )�γ , or ct =
R
1+ (βA)1/γat . (44)
Substituting it into J (a) = u (c (a)) + βJ (Ra� c (a)) gives
Aa1�γ
1� γ=
c (a)1�γ
1� γ+ βA
(Ra� c (a))1�γ
1� γ, (45)
A =1
β�
β�1/γR1�1/γ � 1�γ , (46)
ct = (βR)1/γ�
β�1/γR1�1/γ � 1�at , (47)
which is consistent with that we obtained using optimal control.Finally, given A, we can obtain J (a).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 28 / 59
Another Application to the Consumption-Savings Model
Consider the following consumption-saving problem with a nonlinearsaving technology:
maxfct ,kt+1g
U0 =∞
∑t=0
βt ln (ct )
subject to
kt+1 = θkαt � ct ,
ct � 0, kt � 0.
given the initial asset holdings k0 = k (0).The Bellman equation can be written as:
V (k) = maxc ,eknln (c) + βV
�ek�o or V (k) = maxeknln�
θkα � ek�+ βV�ek�o .
Guess that: V (k) = E ln (k) + F . It turns out thatV (k) = α
1�αβ ln (k) + F , c (k) = (1� αβ) θkα, and ek (k) = αβθkα.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 29 / 59
Dynamic Programming (DP): General Setting
Choose an in�nite sequence of controls fctg∞t=0 to maximize
∞
∑t=0
βt r (xt , ct ) (48)
subject to xt+1 = g (xt , ct ), with x0 given.Assume that r (xt , ct ) is a concave function and the setf(xt+1, xt ) : xt+1 � g (xt , ct )g is convex and compact. DP seeks atime-invariant policy function c = h (x) mapping the state (x) intothe control (c), such that the sequence fctg∞
t=0 generated by iteratingthe h and g functions starting from x0 solves the original problem.To �nd h, we need to know the value function that is the optimalvalue of the original problem:
V (x0) = maxfctg∞
t=0
∞
∑t=0
βt r (xt , ct ) (49)
subject to the same constraint.Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 30 / 59
Note that if we knew V (�), c can be obtained by solving for each xthe problem
maxcfr (x , c) + βV (ex)g (50)
subject to ex = g (x , c) with x given and ex denotes the state nextperiod.Key point: Have exchanged the original problem of �nding fctg∞
t=0for the problem of �nding the optimal V (x) and h (x) that solves thecontinuum of maximum problems (one maximum problem for each x).The task is now to solve for (V (x) , h (x)) linked by the Bellmanequation:
V (x) = maxcfr (x , c) + βV (g (x , c))g . (51)
The maximizer of the right side of it is h (x) that satis�es
V (x) = r (x , h (x)) + βV (g (x , h (x))) , (52)
which is a functional equation to be solved for unknown functions:(V (x) , h (x)). Note that the function equation must hold for everyvalue of x within the permitted domain and the solutions to it arefunctions, V (x) and h (x).Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 31 / 59
Some Properties of the Bellman Equation
In the general setting, we encounter the same problems that we hadin the consumption-savings model:
We need to know the derivatives of the value function, V (x).The value function is unknown.
The envelop theorem of Benveniste and Scheinkman (1978) gives aset of su¢ cient conditions under which one can obtain the derivativeof the value function:
State x 2 X , where X is a convex set with a nonempty interior.Control c 2 C , where C is a convex set with a nonempty interior.The period objective function, r , is concave and di¤erentiable.The RHS of the state transition equation, g , is concave, di¤erentiable,and invertible in c .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 32 / 59
Under the above particular assumptions about r , g , x , and c , there arefour main �ndings:
1 The Bellman equation, (51), has a unique strictly concave solution.2 This solution can be approached in the limit as j ! ∞ by iterations on
Vj+1 (x) = maxcfr (x , c) + βVj (ex)g (53)
subject to ex = g (x , c) with x given, starting from any bounded andcontinuous initial V0.
3 There is a unique and time-invariant optimal policy of the formc = h (x), where h is chosen to maximize the right side of (51).
4 The limiting value function V is di¤erentiable.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 33 / 59
Optimal Conditions
Since V is di¤erentiable, the FOC for (51) is
rc (x , c) + βV 0 (g (x , c)) gc (x , c) = 0. (54)
If we also assume that h is di¤erentiable, di¤erentiation of (52) gives
V 0 (x) = rx (x , h (x)) + rc (x , h (x)) h0 (x) (55)
+βV 0 (g [x , h (x)])�gx (x , h (x)) + gc (x , h (x)) h0 (x)
�,
which can be reduced to
V 0 (x) = rx (x , h (x)) + βV 0 (g (x , h (x))) gx (x , h (x)) . (56)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 34 / 59
Solution Methods
There are three main types of solution methods for solving DP.
1 Guess and verify. It involves guessing and verifying a solution V tothe Bellman equation.
2 Value function iteration. It proceeds by constructing a sequence ofvalue functions and associated policy functions. The sequence iscreated by iterating on the following equation, starting from V0 = 0,and continuing until Vj has converged:
Vj+1 (x) = maxcfr (x , c) + βVj (ex)g (57)
subject to ex = g (x , c) with x given.3 Policy function iteration (or Howard�s improvement algorithm).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 35 / 59
Value Function Iteration
First, consider an initial guess for the value function V0 (x). (It doesnot matter very much and we usually guess it is 0.)Second, one can then calculate an updated value function V1 (x)using:
V1 (x) = maxcfr (x , c) + βV0 (g (x , c))g , (58)
and doing the maximization numerically over the domain of x . Thismaximization de�nes approximately the function V1 (x).Using this new function V1 (x), one can update again and get a newV2 (x):
V2 (x) = maxcfr (x , c) + βV1 (g (x , c))g , (59)
over the domain of x . Repeated this step until the sequence of theapproximate value functions converges to V (x). Note that in therepeated calculations of the value function, one is also calculatingrepeated approximations to the policy function, c = h (x).Note that the subscript (0, 1, 2, � � �) represents the iteration on thevalue function and does not have any link to time periods.Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 36 / 59
Application: Value Function Iteration
Essentially, the value function iteration is searching for a �xed point,at which point we have numerically solved the value function.
When applying to the consumption-saving problem, we can write theprocess as:
ΓV (k) = maxeknln�
θkα � ek�+ βV�ek�o , 8k, (60)
where Γ is an operator representing this process of iteration on thevalue function until ΓV (k) = V (k).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 37 / 59
Amount of Capital, k0 10 20 30 40 50 60 70 80 90 100
Valu
e Fu
nctio
n, V
15
10
5
0
5
10Value Function Iteration
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 38 / 59
Finite-horizon Dynamic Programming
The basic idea of �nite-horizon DP is to solve the optimizationproblem period by period � starting from the terminal period, takingthe previous periods�solutions as given, and then working backsequentially to the �rst period.Having optimized the terminal period (T ), this solution is substitutedinto the period T � 1 problem and then period T � 1 is optimized.Substituting these previous solutions, the solutions for T � 2, T � 3,� � �, 0 are obtained in sequence in the same way.Suppose that the problem is to maximize:
U (xt ) = r (xt , ct ) + βV (xt+1) , (61)
for t = 0, � � �,T , subject toxt+1 = f (xt , ct ) , (62)
with x0 given and xT+1 = x . Note that here we assume that thevalue function, V (xt+1), is time invariant and later we can see that itis correct.Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 39 / 59
Consider the problem for period T . First, we maximize
U (xT ) = r (xT , cT ) + βV (xT+1) , (63)
with respect to cT , subject to xT+1 = f (xT , cT ) and taking xT asgiven. Thus we maximize
U (xT ) = r (xT , cT ) + βV (f (xT , cT )) .
The FOC is
∂r (xT , cT )∂cT
+ β∂V (f (xT , cT ))
∂cT= 0,
which means that the solution for cT has the form:
cT = gT (xT ) . (64)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 40 / 59
(Conti.) Substituting this solution into U (xT ):
U (xT ) = r (xT , cT ) + βV (f (xT , cT ))
= r (f (xT�1, cT�1) , gT (f (xT�1, cT�1)))
+βV (f (f (xT�1, cT�1) , gT (f (xT�1, cT�1))))
, VT (xT�1, cT�1) .
Turning to period T � 1, we maximize
U (xT�1) = r (xT�1, cT�1) + βV (xT ) ,
w.r.t. cT�1, subject to xT = f (xT�1, cT�1) and taking xT�1 asgiven. The FOC is
∂r (xT�1, cT�1)∂cT�1
+ β∂VT (xT�1, cT�1)
∂cT�1= 0, (65)
from which we can write the solution for cT�1 ascT�1 = gT�1 (xT�1) .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 41 / 59
Substituting this into U (xT�1) gives
U (xT�1) = r (xT�1, gT�1 (xT�1)) + βV (f (xT�1, gT�1 (xT�1)))
, VT�1 (xT�2, cT�2) . (66)
We can proceed similarly in periods T � 2, � � �, 0.The general solution for period t has the FOC
∂r (xt , ct )∂ct
+ β∂V (f (xt , ct ))
∂ct= 0, (67)
and optimal control:ct = gt (xt ) . (68)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 42 / 59
Application to the Same Consumption-Saving Model
Consider the same consumption-saving problem as above:
V (a0) = maxfct ,at+1g
T
∑t=0
βtu (ct )
s.t.at+1 = Rat � ct , f (at , ct ) , t = 0, � � �,T ,
given a0 = a (0) and aT+1 = 0, where u (ct ) = ln (ct ).
We �rst consider the solution for period T . This requires to maximize
U (aT ) = u (cT ) + βV (aT+1) , (69)
with respect to cT , subject to aT+1 = f (aT , cT ), aT+1 = 0, andtaking aT as given. We thus have V (aT ) = u (cT ), i.e., nomaximization is required for the terminal period: cT = RaT , and
U (aT ) = u (RaT ) + βV (0) = ln (RaT ) + βV (0) . (70)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 43 / 59
(Conti.) The T � 1 problem is to maximize:
U (aT�1) = u (cT�1) + βV (aT )
= u (cT�1) + β [ln (RaT ) + βV (0)] (71)
with respect to cT�1, subject to aT = RaT�1 � cT�1 and takingaT�1 as given. The FOC is:
1cT�1
� β1
RaT�1 � cT�1= 0,
which implies that
cT�1 =R
1+ βaT�1 (72)
and
V (aT�1) = ln�
R1+ β
aT�1
�+ β
�ln�
βR2
1+ βaT�1
�+ βV (0)
�.
(73)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 44 / 59
(Conti.) Similarly, solving the T � 2 problem yields the followingoptimal solution:
cT�2 =R
1+ β+ β2aT�2. (74)
It is straightforward to show that the general solution for periods0, 1, � � �,T takes the following form:
ct =R
1+ β+ β2 + � � �+ βT�tat , (75)
which is identical to the corresponding solution we obtained usingoptimal control (i.e., the Lagrange multiplier method). As T ! ∞,we have
ct = (1� β)Rat . (76)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 45 / 59
From Discrete-time to Continuous-time
We may treat the continuous-time case as a limiting case of thediscrete-time case as the time interval (∆t) goes to 0. (Note that inthe discrete-time models, ∆t = 1.)Speci�cally, in discrete-time, the �ow constraint is
x (t + 1)� x (t) = f (c (t) , x (t) , t) , (77)
which can be rewritten as
x (t + ∆t)� x (t) = f (c (t) , x (t) , t)∆t (78)
if the time interval is ∆t.Dividing both sides by ∆t and letting it goes to 0 gives
x 0t = lim∆t!0
x (t + ∆t)� x (t)∆t
= f (c (t) , x (t) , t) (79)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 46 / 59
(Conti.) Further, note that when the time interval is ∆t, the objectivefunction can be written as
T /∆t
∑i=0
r (c (i∆t) , x (i∆t) , i∆t)∆t; (80)
as ∆t ! 0,
lim∆t!0
T /∆t
∑i=0
r (c (i∆t) , x (i∆t) , i∆t)∆t =Z T
0r (c (t) , x (t) , t) dt
(81)Hence, the continuous-time version of the dynamic optimizationproblem can be written as
maxc (t)
Z T
0r (c (t) , x (t) , t) dt,
s.t. x 0t =dx (t)dt
= f (c (t) , x (t) , t) ,
x0 = x (0) , and x (T ) � 0.Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 47 / 59
Solving the Model
The Lagrange function for the above intertemporal problem is
L =Z T
0
�r (ct , xt , t) + λt
�f (ct , xt , t)� x 0t
�dt
=Z T
0
�r (ct , xt , t) + λt f (ct , xt , t) + λ0txt
dt + λ0x0 � λT xT
where we use the fact that the rule of integration by parts implies
�Z T
0λtx 0tdt =
Z T
0λ0txtdt + λ0x0 � λT xT (82)
Now we can regard the integral as the sum and take �rst derivativesw.r.t. ct and xt :
rc (ct , xt , t) + λt fc (ct , xt , t) = 0 (83)
rx (ct , xt , t) + λt fx (ct , xt , t) + λ0t = 0 (84)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 48 / 59
Hamiltonian Function
Similarly, we can de�ne a new function H, a Hamiltonian function,
H = r (ct , xt , t) + λt [f (ct , xt , t)] . (85)
Hence, the FOC for ct , (83), can be rewritten as
∂Ht∂ct
= 0. (86)
Theorem (The Maximum Principle)
The necessary FOCs for maximizing (81) subject to (79) are:(i) Hc , ∂Ht
∂ct= 0 for all t 2 [0,T ] .
(ii) x 0t = f (ct , xt , t) for all t (The state transition equation)(iii) Hx + λ0t = 0
�rx (ct , xt , t) + λt fx (ct , xt , t) + λ0t = 0
�for all t (The
costate (λt ) equation)(iv) xT � 0,λT � 0, xT λT = 0 (The transversality condition, or thecomplementary-slackness condition)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 49 / 59
Continuous-time Version of the Consumption-Saving Model
Consider the following deterministic continuous-time optimalconsumption-saving problem:
maxct
Z T
0u (ct ) exp (�ρt) dt, (87)
subject toa0t = rat � ct , (88)
where ρ is called the discount rate, r is the instantaneous interestrate, given a0, and aT � 0 (preventing the consumer from dying withdebts).
Here we assume that the utility function u (ct ) satis�es the usualconditions (just like those discussed in the discrete-time case) andct > 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 50 / 59
The Hamiltonian for this problem is
H = u (ct ) exp (�ρt) + λt (rat � ct ) . (89)
Using the maximum principle,
Hc = 0 =) λt = u0 (ct ) exp (�ρt) , (90)
Ha + λ0t = 0 =) rλt + λ0t = 0, (91)
a0t = rat � ct . (92)
In addition, aT = 0, i.e., it is optimal for the consumer to leave nobequest after T . Note that if at optimum aT > 0, λT > 0, whichcontradicts (90).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 51 / 59
(Conti.) Using (91), we have
λ0tλt= �r , (93)
which can be easily solved for λt :
λt = λ0 exp (�rt) . (94)
Suppose that u (ct ) =c1�γt �11�γ . Then combining (90) with (94) gives
c�γt = c�γ
0 exp ((ρ� r) t) . (95)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 52 / 59
(Conti.) Di¤erentiating (91) gives
a00t = ra0t � c 0t .
Since
c 0t ,dctdt=u0 (ct )u00 (ct )
(ρ� r) = r � ρ
γct ,
we have
a00t = ra0t +ρ� r
γct
= ra0t +ρ� r
γ
�rat � a0t
�=
�r � ρ� r
γ
�a0t +
r (ρ� r)γ
at , (96)
which is a second-order di¤erential equation (SODE) in terms of at .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 53 / 59
(Conti.) For r 6= � 1γ (ρ� r), (i.e., no repeated roots for the
characteristic equation, λ2 ��r � ρ�r
γ
�λ� r (ρ�r )
γ = 0,) this SODEhas the general solution:
at = A exp (rt) + B exp��ρ� r
γt�, (97)
where A and B can be pinned down by using the initial and terminalconditions:
a0 = A+ B, (98)
aT = A exp (rT ) + B exp��ρ� r
γT�(= 0) . (99)
From (99), we have A = �B exp�h� 1
γ (r � ρ)� riT�.
Substituting it into (98):
B =
�1� exp
��� r � ρ
γ� r�T��a0,
A = � exp��� r � ρ
γ� r�T� �1� exp
��� r � ρ
γ� r�T��a0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 54 / 59
Optimal consumption:
ct , rat � a0t = r�A exp (rt) + B exp
��ρ� r
γt��
��Ar exp (rt)� ρ� r
γB exp
��ρ� r
γt��.
Consider a special case in which r = ρ:
c 0t =u0 (ct )u00 (ct )
(ρ� r) = 0,
i.e., ct = c , independent of time t. In this case, a0t = rat � ctbecomes a0t = rat � c , whose solution is:
at = exp (rt)�a0 �
cr(1� exp (�rt))
�. (100)
At T , we have aT = exp (rT )�a0 � c
r (1� exp (�rT ))�= 0:
c =r
1� exp (�rT )a0. (101)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 55 / 59
Continuous-time Bellman Equation (Optional)
Consider the same deterministic continuous-time consumption-savingproblem:
J (a0) = maxct
Z ∞
0u (ct ) exp (�ρt) dt (102)
subject to a0t = rat � ct , given a0, and aT � 0. Here J (a0) is de�nedas the value function.
(102) can be rewritten as the following recursive form:
J (a0) = maxc
�Z ∆t
0u (ct ) exp (�ρt) dt +
Z ∞
∆tu (ct ) exp (�ρt) dt
�= max
c0fu (c0)∆t + exp (�ρ∆t) J (a∆t )g (103)
for a small interval, ∆t.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 56 / 59
(Conti.) Here we use the facts:Z ∆t
0u (ct ) exp (�ρt) dt = u (c0)
Z ∆t
0exp (�ρt) dt = u (c0)∆t,Z ∞
∆tu (ct ) exp (�ρt) dt
= exp (�ρ∆t)Z ∞
0u (ct ) exp (�ρ (t � ∆t)) d (t � ∆t)
, exp (�ρ∆t) J (a∆t ) . (104)
Linearizing J (a∆t ) around a0:
J (a∆t ) �= J (a0) + J 0 (a0) (a∆t � a0) .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 57 / 59
(Conti.) Substituting it back into (103):
J (a0) = maxc0
�u (c0)∆t + (1� ρ∆t)
�J (a0) + J 0 (a0) (a∆t � a0)
�() ρ∆tJ (a0) = max
c0
�u (c0)∆t + J 0 (a0) (a∆t � a0)�ρ∆tJ 0 (a0) (a∆t � a0)
�,(105)
where we use the fact that exp (�ρ∆t) �= 1� ρ∆t.Dividing ∆t on both sides:
ρJ (a0) = maxc0
�u (c0) + J 0 (a0)
�lim∆t!0
a∆t�a0∆t
��ρJ 0 (a0) lim∆t!0 (a∆t � a0)
�= max
c0
�u (c0) + J 0 (a0) a00
= max
c0
�u (c0) + J 0 (a0) (ra0 � c0)
(106)
and taking derivative w.r.t. the control c0: u0 (c0) = J 0 (a0), whichimplies that c0 = h (a0).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 58 / 59
(Conti.) Substituting c0 = h (a0) into the Bellman equation (106):
ρJ (a0) = u (h (a0)) + J 0 (a0) [ra0 � h (a0)] . (107)
The envelop theorem implies that
ρJ 0 (a0) = u0 (h (a0)) h0 (a0) + J 00 (a0) [ra0 � h (a0)]+J 0 (a0)
�r � h0 (a0)
�= J 00 (a0) [ra0 � h (a0)] + J 0 (a0) r ,
which means that
(ρ� r) u0 (c0) = u00 (c0)dc0da0
[ra0 � h (a0)] ,
c 00c0
=(ρ� r) u0 (c0)c0u00 (c0)
, r � ρ
γ, (108)
where we use the fact that u00 (c0) dc0da0 = J00 (a0).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 20, 2019 59 / 59