an introduction to optimal control theory and hamilton ...briani/appunticorsobriani.pdf · an...
TRANSCRIPT
An introduction to optimal control theory and
Hamilton-Jacobi equations.
Ariela Briani,LMPT, Universite de Tours, France.
email: [email protected]
Contents
1 Introduction 2
2 Some optimal control problems. 2
3 The Dynamic Programming Principle 4
4 The Hamilton Jacobi equation and the viscosity solutions. 7
4.1 On the need for non-smooth solutions. . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2 The definition of viscosity solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.3 The value function is a solution of an Hamilton-Jacobi equation. . . . . . . . . . . . 12
5 An idea of the technique for the uniqueness results. 15
5.1 The uniqueness result for the infinite horizon, the finite horizon and the minimumtime problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
6 A stability result and a further remark on the origin of the definition. 21
7 A primer on Semi-Lagrangian schemes:the infinite horizon and the minimum time problem. 23
1
1 Introduction
The main aim of this notes is to give an idea of the complete method to solve optimal controlproblems via the solution of a partial differential equation. We will consider tree basic examples:the infinite horizon problem, the finite horizon problem and the minimum time problem. We will tryto complete the following program: define the value function and prove a Dynamic ProgrammingPrinciple, characterize this value function as the unique viscosity solution of an Hamilton Jacobiequation and give an idea of the numerical methods we can use to solve this equation. A sectionis devoted to the technique for the comparison and stability results for Hamilton Jacobi equations.Please note that in the bibliography we will only cite the main books on the subjects: please referto the bibliography therein for more informations.
2 Some optimal control problems.
We consider here a controlled system where the trajectories are solutions of the following ordinarydifferential equation:
y′(t) = f(y(t), α(t)) , t ∈ R+
y(0) = x(2.1)
here the function α is called the control: this is the way ”we can act on the system”.Our assumptions on the controls and the dynamics are :
(Ha) The set A ⊂ RM is compact and we consider the following set of admissible controlsA := α : R+ → A : α(·) is measurable.
(Hf) The function f : RN ×A→ RN is continuous, moreover there exist Lf ,Mf > 0 such that
|f(x, α)− f(y, α)| ≤ Lf |x− y| and |f(x, α)| ≤Mf ∀x, y ∈ RN , ∀α ∈ A.
Under these assumptions classical results on ordinary differential equations apply, therefore foreach control α ∈ A and each initial data x ∈ RN there exists a unique Lipschitz function solutionof (2.1). We will call this function a trajectory of the controlled system and we will denote it byyx(·, α) ∈ Lip(R+;RN ). Moreover, we have the classical following estimates
|yx(t, α)− yz(t, α)| ≤ eLfT |x− z| ∀α ∈ A , ∀t ∈ [0,+∞[ , ∀x, z ∈ RN (2.2)
|yx(t, α)− x| ≤Mf t ∀α ∈ A , ∀t ∈ [0,+∞[ , ∀x ∈ RN (2.3)
We consider now tree different optimal control problems, this meaning that the cost we want tominimize are different.
The infinite horizon problem.
Given an initial data x and a control a ∈ A our aim is to minimize the following cost:
J(x, α) =
∫ +∞
0l(yx(t, α), α(t)) e−λt dt (2.4)
where λ > 0 is a given discount factor and the running cost l fulfills the following assumption:
2
(Hl) The function l : RN × A → RN is continuous bounded and continuous, more precisely thereexist a modulus ωl(·) and a constant Ml > 0 such that
|l(x, α)− l(y, α)| ≤ ωl(|x− y|) and |l(x, α)| ≤Ml ∀x, y ∈ RN , ∀α ∈ A.
(Recall that a modulus is a non decreasing continuous function ω : R+ → R+ such thatω(0) = 0.)
Therefore we want to calculate the following value function V : RN → R,
V (x) := infα∈A
J(x, α) (2.5)
The finite horizon problem: Bolza and Mayer problems.
In this case a finite time t > 0 and a final cost g : RN → R are given, the cost to minimize is
J(x, t, α) =
∫ t
0l(yx(s, α), α(s)) e−λs ds+ g(yx(t, α)) . (2.6)
We will always assume here λ ≥ 0, assumption (Hl) on the running cost l and that
(Hg) The function g : RN → R is bounded and uniformly continuous on RN .
This general case is called the Bolza problem and the value function V : RN × R+ → R is givenby
V (x, t) := infα∈A
J(x, t, α). (2.7)
In the special case of the running cost identically zero and not discount factor (i.e. λ = 0) thisproblem is called a Mayer problem. We remark here that if λ = 0 and we can assume therunning cost Lipschitz with respect to the state variables uniformly in the control variables, we cantransform a Bolza problem into a Mayer problem. Indeed, we can add a scalar state variable yN+1
to the vector y, with dynamics y′N+1(t) = l(yN+1(t), α(t)), yN+1(0) = 0 and minimize the payoffyN+1(t, α) + g(yx(t, α)).
The minimum time problem.
The aim of this problem is to minimize the time for the system to reach a given target setT ⊂ RN. We assume
(HTarg) The set T ⊂ RN is closed with compact boundary ∂T .
Given an initial data x and a control α ∈ A we give the precise definition of the exit time from cTor the entry time in T as follows:
tx(α) :=
+∞ if t : yx(t, α) ∈ T = ∅mint : yx(t, α) ∈ T otherwise.
(2.8)
Therefore we look for the minimal time function T : RN → R given by
T (x) := infα∈A
tx(α). (2.9)
An important set to define here is the set of starting points from which the system can reach thetarget, this is called the Reachable set R and is defined as
R := x ∈ RN such that there exist α ∈ A and t ∈ R+ : yx(t, α) ∈ T . (2.10)
3
3 The Dynamic Programming Principle
The basic tool in this approach is the Dynamic Programming Principle. This principle express theintuitive idea that the minimum cost is achieved if one behaves as follows:(a) Let the system evolve for a small amount of time T choosing an arbitrary control α(·).(b) Pay the corresponding cost.(c) Pay what remains to pay after time T with the best possible control.(d) Minimize the sum of these two costs over all the possible controls.
The basic tool to prove this principle is the following semigroup property for the solutions ofsystems (2.1).
Lemma 3.1. Assume (Ha) and (Hf). For each initial data x ∈ RN and control α ∈ A, if yx(·, α)is the solution of (2.1), we have
yx(s+ t, α) = yz(t, α(·+ s)) with z := yx(s, α) . (3.1)
Proof. By the classical Cauchy-Lipschitz theorem for each initial data x ∈ RN and control α ∈A there exists a unique solution of (2.1). Moreover, since this solution has an integral formulationwe can easily derive (3.1) as follows
yx(s+ t, α) = x+
∫ s+t
0f(yx(u), α(u)) du = x+
∫ s
0f(yx(u), α(u)) du+
∫ s+t
sf(yx(u), α(u)) du =
= z+
∫ s+t
sf(yx(u), α(u))du = z+
∫ t
0f(yx(v+s), α(v+s))dv = z+
∫ t
0f(yz(v), α(v+s))dv = yz(t, α(·+s))
where we set u = v + s and we used that by uniqueness yx(v + s) = yz(v), ∀v ∈ (0, t).
Remark 3.2. Since this will be very useful in the following we remark here the following propertiesof the admissible controls:
(i) α(·) ∈ A , t > 0⇒ α(·+ t) ∈ A (3.2)
(ii) α1(·) ∈ A , α2(·) ∈ A , t > 0 and α(s) :=
α1(s) , if s ≤ tα2(s) , if s > t
⇒ α(·) ∈ A (3.3)
We are now ready to prove the Dynamic Programming Principle (DPP).
Theorem 3.3. DPP for the infinite horizon problem.Assume (Ha) and (Hf). For each initial data x ∈ RN and any time t > 0 we have
V (x) = infα∈A
∫ t
0l(yx(s, α), α(s)) e−λs ds+ e−λtV (yx(t, α))
. (3.4)
Proof. Fix x ∈ RN and t > 0 we first remark that, for each α ∈ A we have
J(x, α) =
∫ +∞
0l(yx(s, α), α(s))e−λsds =
∫ t
0l(yx(s, α), α(s))e−λsds+
∫ +∞
tl(yx(s, α), α(s))e−λsds =
4
=
∫ t
0l(yx(s, α), α(s)) e−λs ds+
∫ +∞
0l(yx(u+ t, α), α(u+ t))e−λ(u+t) du =
+
∫ t
0l(yx(s, α), α(s)) e−λs ds+ e−λt
∫ +∞
0l(yz(u, α(·+ t)), α(u+ t))e−λu du
where we set s = u+ t, z = yx(t, α) and we used (3.1).We recall here that the control α(·+t) ∈ A by (3.2), therefore by the definition of the value function
we have V (yz(t, α)) ≤∫ +∞
0l(yz(u, α(·+ t)), α(u+ t))e−λu du. Therefore for each α ∈ A, we have
J(x, α) ≥∫ t
0l(yx(s, α), α(s)) e−λs ds+ e−λtV (yz(t, α))
and taking the inf over the controls α ∈ A we obtain the first inequality:
V (x) ≥ infα∈A
∫ t
0l(yx(s, α), α(s)) e−λs ds+ e−λtV (yx(t, α))
.
Fix now α ∈ A we set z = yx(t, α). For each ε > 0 there exists a control αε ∈ A such that
V (z) + ε ≥ J(z, αε). We define α(s) :=
α(s) , if s ≤ tαε(s− t) , if s > t
since this is an admissible control
(see (3.3)) we can write
V (x) ≤ J(x, α) =
∫ +∞
0l(yx(s, α), α(s))e−λsds =
∫ t
0l(yx(s, α), α(s))e−λsds+
∫ +∞
tl(yx(s, α), α(s))e−λsds =
=
∫ t
0l(yx(s, α), α(s))e−λs ds+
∫ +∞
tl(yx(s, α), αε(s− t))e−λs ds =
=
∫ t
0l(yx(s, α), α(s))e−λs ds+
∫ +∞
0l(yx(u+ t, α), αε(u))e−λ(u+t)du
=
∫ t
0l(yx(s, α), α(s))e−λs ds+ e−λt
∫ +∞
0l(yz(u, α(·+ u)), αε(u))e−λudu =
=
∫ t
0l(yx(s, α), α(s))e−λs ds+ e−λt J(z, αε) ≤
∫ t
0l(yx(s, α), α(s))e−λs ds+ e−λt(V (z) + ε).
Summarizing we have obtained for a fixed arbitrary α ∈ A and ε > 0
V (x) ≤∫ t
0l(yx(s, α), α(s))e−λs ds+ e−λt(V (yx(t, α)) + ε).
Therefore, taking the limit as ε → 0 and the infimum over the controls α ∈ A, we can prove thereverse inequality and conclude the proof.
Remark 3.4. Note that with similar arguments as in the proof of the DPP Theorem 3.3 we canprove that the following function is non decreasing
t 7→∫ t
0l(yx(s, α), α(s)) e−λs ds+ e−λtV (yx(t, α)) , t ∈ [0,+∞[
Moreover it is constant if and only if the control α(·) is optimal for the initial position x.
5
Theorem 3.5. DPP for the Bolza problem.Assume (Ha) and (Hf). Given an initial data x ∈ RN and an horizon time t > 0, for each 0 < τ ≤ twe have
V (x, t) = infα∈A
∫ τ
0l(yx(s, α), α(s)) e−λs ds+ e−λτV (yx(τ, α), t− τ)
. (3.5)
Proof. The arguments are very similar to the ones of the proof Theorem 3.3 so we will omitthem.
Remark 3.6. Note that with similar arguments as in the proof of the DPP Theorem 3.5 we canprove that the following function is non decreasing
τ 7→∫ τ
0l(yx(s, α), α(s)) e−λs ds+ e−λτV (yx(τ, α), t− τ) τ ∈ [0, t]
Moreover it is constant if and only if the control α(·) is optimal for the initial position x and finaltime t.
Theorem 3.7. DPP for the Minimum time problem.Assume (Ha) and (Hf). Given an initial data x ∈ RN we have
T (x) = infα∈As+ T (yx(s, α)) , if 0 ≤ s ≤ T (x) < +∞. (3.6)
Proof. We first remark that thenks to the semigroup properties for the trajectories (3.1) inLemma 3.1 we have the following: if 0 ≤ s ≤ T (x) < +∞, for each α ∈ A
tx(α) = s+ tz(α(·+ s)) with z := yx(s, α) . (3.7)
Therefore, thanks to (3.2) we have tx(α) = s+ tz(α(·+s)) ≥ s+T (yx(s, α)) for each α ∈ A. Takingthe infimum over the admissible controls this yields to our first inequality:
T (x) ≥ infα∈As+ T (yx(s, α)) .
Fix now α ∈ A we set z = yx(t, α). For each ε > 0 there exists a control αε ∈ A such that
T (z) + ε ≥ tz(αε). We define α(s) :=
α(s) , if s ≤ tαε(s− t) , if s > t
since α is an admissible control (see
(3.3)), by (3.7) we can write
T (x) ≤ tx(α) = s+ tyx(s,α)(α(·+ s)) . (3.8)
Note that, by construction of α we have yx(s, α) = yx(s, α) = z and α(t + s) = αε(t), ∀t > 0.Therefore (3.8) reads T (x) ≤ s + tz(αε) ≤ T (z) + ε. Passing to the limit as ε → 0 and taking theinfimum over the admissible controls we obtain then the reverse inequality:
T (x) ≤ infα∈As+ T (yx(s, α))
and we conclude the proof.
Remark 3.8. Note that, with similar arguments as in the proof of the DPP Theorem 3.7, we canprove that the following function is non decreasing
s 7→ s+ T (yx(s, α)) s ∈ [0, tx(α)] if T (x) < +∞
Moreover it is constant if and only if the control α(·) is optimal for the initial position x.
6
4 The Hamilton Jacobi equation and the viscosity solutions.
The aim of this theory is to prove that the value function is the unique solution of a partial differen-tial equation in a sense that ensure uniqueness and stability properties. To set a general frameworkwe start by considering the general case of fully non linear second order partial differential equation.Let Ω be an open subset of RN, the general formulation is
F (x, u,Du,D2u) = 0 x ∈ Ω (4.1)
where the unknown is a function u : Ω → R, Du = (ux1 , ux2 , . . . , uxN ) is the gradient vector andD2u = (uxi,xj ) is the Hessian matrix. Thus, F : RN × R × RN × S(N) → R where S(N) is theset of real symmetric N ×N matrices. We will always assume the following that F is degenerateelliptic and proper. More precisely:
(HFe) The function F : RN ×R×RN × S(N)→ R is degenerate elliptic, i.e. for each X,Y ∈ S(N)we have
Y ≤ X =⇒ F (x, r, p,X) ≤ F (x, r, p, Y ) ∀(x, r, p) ∈ RN × R× RN
(recall that Y ≤ X means that ∀x ∈ RN we have Y x · x ≤ Xx · x).
(HFp) The function F : RN × R × RN × S(N) → R is degenerate elliptic and proper, i.e. for eachX,Y ∈ S(N) and r, s ∈ R we have
Y ≤ X , r ≤ s =⇒ F (x, r, p,X) ≤ F (x, s, p, Y ) ∀(x, p) ∈ RN × RN .
Note that the function F might be first order, i.e. F (x, r, p,X) = H(x, r, p). This is a (very)degenerate elliptic function and it is proper if it is non decreasing with respect to r.
4.1 On the need for non-smooth solutions.
The basic idea to derive a partial differential equation solved by the value function is that theHamilton Jacobi equation is a differential version of the Dynamic Programming Principle. Tounderstand this idea let us perform now this computation under the assumption that the valuefunction is regular. We will take the infinite horizon problem and assume that V ∈ C1(RN). Takeany constant admissible control α(·) = α ∈ A, by the Dynamic programming Principle (Theorem3.3), we have
V (x) ≤∫ t
0l(yx(s, α), α) e−λs ds+ e−λtV (yx(t, α)).
Thus
V (x)− e−λtV (x) + e−λtV (x)− e−λtV (yx(t, α))−∫ t
0l(yx(s, α), α) e−λs ds ≤ 0. (4.2)
Since the trajectory yx(·, α) is a solution of (2.1), if we denote by DV the gradient vector of thevalue function V , we have
V (yx(t, α))−V (x) =
∫ t
0
d
dsV (yx(s, α))ds =
∫ t
0DV (yx(s, α))·y′x(s, α)ds =
∫ t
0DV (yx(s, α))·f(yx(s, α), α)ds
(4.3)
7
therefore, also dividing by t > 0, (4.2) gives us
(1− e−λt
t)V (x) +
1
t
∫ t
0
(−e−λtDV (yx(s, α)) · f(yx(s, α), α)− l(yx(s, α), α) e−λs
)ds ≤ 0 .
Letting t→ 0 thanks to the regularity assumptions on f and l we have for each α ∈ A
λV (x)−DV (x) · f(x, α)− l(x, α) ≤ 0
therefore, taking the supremum over the constant controls α ∈ A we get
λV (x) + supα∈A−DV (x) · f(x, α)− l(x, α) ≤ 0.
With similar calculation we can obtain the reverse inequality and finally prove that if the valuefunction V ∈ C1(RN) then it is the solution of the Hamilton-Jacobi equation
λV (x) +H(x,DV ) = 0 x ∈ RN ,
where the Hamiltonian H : RN × RN → R is defined by
H(x, p) := supα∈A−p · f(x, α)− l(x, α).
Unfortunately even in the case of smooth dynamics and cost the assumption V ∈ C1(RN ) is toorestrictive as we will show in the following example.
Example 4.1. Let us consider N = 1 , A = −1; 1, f(x, α) = α and l(x, α) = l(x) with l(x) aneven smooth function such that l(x) = 0 for |x| > R, maxx∈R l(x) = l(0) > 0 and xl′(x) < 0 for|x| < R. Fix x ∈ R since A = −1; 1 we have only two possible choices for our control: α = 1 thatgives the trajectory y(t) = t+ x and α = −1 that gives the trajectory y(t) = −t+ x, therefore
V (x) = min
∫ +∞
0l(x+ t)e−λtdt ;
∫ +∞
0l(x− t)e−λtdt
Since the cost is even we have
V (x) = min
eλx∫ +∞
xl(u)e−λudu ; e−λx
∫ +∞
−xl(u)e−λudu
.
Recalling that maxx∈R l(x) = l(0) > 0 the idea is that to minimize our cost we have ”to avoid thepassage from zero in the integral” thus
V (x) =
∫ +∞
0l(x+ t)e−λtdt if x > 0∫ +∞
0l(t)e−λtdt =
∫ +∞
0l(−t)e−λt if x = 0∫ +∞
0l(x− t)e−λtdt if x < 0
(4.4)
A calculation shows that
V ′+(0) =
∫ +∞
0l′(t)e−λt dt and V ′−(0) =
∫ +∞
0l′(−t)e−λt dt
8
Since l′(−x) = −l′(x) then V ′+(0) 6= V ′−(0) thus even if the running cost and the dynamics areregular the value function V is not differentiable at x = 0.We complete this example by computing the Hamiltonian
H(x, p) = supα∈−1;1
−p · α− l(x) = maxp,−p − l(x) = |p| − l(x)
Therefore the Hamilton-Jacobi equation for this example is
λV (x) + |V ′(x)| − l(x) = 0 x ∈ RN
and has not classical meaning for x = 0 if V is not differentiable at this point.
To overcame the problem of defining a solution not regular for the equation (4.1) Kruzkowintroduced in the 60 the idea of generalized solutions, i.e. solutions which satisfies the equationalmost everywhere. This is a powerful idea and a lot of result have been obtained under differentset of hypotheses (for a complete description see [8] and references therein). However we can easilybuilt equations in the framework of optimal control problems where we have a lack of uniquenessand stability for the generalized solution.
Example 4.2. Let us consider the optimization problem of minimizing the distance from theboundary of an open convex subset of RN in the easy case of Ω = (0, 1). The ”cost” we want tominimize is then the distance function. Easy computations show that the value function
V (x) := infy∈∂Ω
|y − x| = min | − x| ; |1− x| =1
2−∣∣∣12− x∣∣∣
fulfills |V ′(x)| = 1 for almost every x ∈ (0, 1) and the boundary conditions V (0) = V (1) = 0.However, it easy to see that if we consider the general equation
|u′(x)| = 1 x ∈]0, 1[ u(0) = u(1) = 0
V is a generalized solution (i.e. fulfills this equation almost everywhere), but we can built infinitelymany other generalized solutions by alternating segments with slope 1 with segments with slope−1. Moreover, we can easily construct a sequence of generalized solution that converge to the null
function which is not a generalized solution. Indeed take un(0) = 0, u′n(x) = 1 if x ∈]2k
2n,2k + 1
2n[
and u′n(x) = −1 if x ∈]2k + 1
2n + 2,2k + 1
2n[ for k between 0 and 2n−1 − 1.
The conclusion is than that from an application point of view the definition of generalizedsolution is not useful: if our aim is to characterize the value function as a unique solution of apartial differential equation a new definition has to be introduced.
4.2 The definition of viscosity solution
The definition of viscosity solution can be easily introduced starting from a different formulationof the Maximum principle for classical solution of second order elliptic equations.
9
Theorem 4.3. Maximum principle Let F : RN × R × RN × S(N) → R be degenerate ellipticand proper. A function u ∈ C2(Ω) is a classical solution of
F (x, u,Du,D2u) = 0 x ∈ Ω (4.5)
if and only if
a) if φ ∈ C2(Ω) and x0 is a local maximum point of u− φ we have
F (x0, u(x0), Dφ(x0), D2φ(x0)) ≤ 0 ,
b) if φ ∈ C2(Ω) and x0 is a local minimum point of u− φ we have
F (x0, u(x0), Dφ(x0), D2φ(x0)) ≥ 0 .
Proof. If u ∈ C2(Ω) is a classical solution of (4.5), φ ∈ C2(Ω) and x0 is a local maximum ofu− φ then Du(x0) = Dφ(x0) and D2u(x0) ≤ D2φ(x0). Since F is elliptic we have
0 = F (x0, u(x0), Du(x0), D2u(x0)) ≥ F (x0, u(x0), Dφ(x0), D2φ(x0))
which is property a). One obtains b) in the same way. Conversely, if u is C2(Ω) we can take φ = uin a) and b) so that each x0 is both a local maximum and a local minimum of u − φ. Thus bothinequalities hold for any point x0 We can conclude than that F (x0, u(x0), Dφ(x0), D2φ(x0)) = 0,thus u is a classical solution of (4.5).
Observe that in a) and b) there is no need to ask regularity for the solution u: the derivationis performed on φ!! In fact the basic idea in the definition of viscosity solution is to use a) and b)to define the solution in order to avoid the use of the derivatives of u.
We are finally ready to give the definition of viscosity solution introduced first introduced in1981 by M.G. Crandall and P.L.Lions. The first two papers in which appeared are [3] and [5]. Wewill give here the definition for a continuous function u specifying also the definitions of viscositysubsolutions and viscosity supersolutions.
Definition 4.4. Viscosity solution. A function u ∈ C(Ω) is a viscosity solution of
F (x, u,Du,D2u) = 0 x ∈ Ω (4.6)
if and only if
a) if φ ∈ C2(Ω) and x0 is a local maximum point of u− φ we have
F (x0, u(x0), Dφ(x0), D2φ(x0)) ≤ 0 , (4.7)
b) if φ ∈ C2(Ω) and x0 is a local minimum point of u− φ we have
F (x0, u(x0), Dφ(x0), D2φ(x0)) ≥ 0 . (4.8)
If u verifies only (4.7) then we say that is a viscosity subsolution of (4.6) and if u verifies only(4.8) then we say that is a viscosity supersolution of (4.6).
10
To give an equivalent definition of viscosity solution that looks more like the calculus of agradient we recall the definition of sub and super differential for set for a continuous function u.Fix x0 ∈ Ω we define the superdifferential of u at point x0 as
D+u(x0) :=
p : lim sup
x→x0 , x∈Ω
u(x)− u(x0)− p · (x− x0)
|x− x0|≤ 0
(4.9)
and the subdifferential of u at point x0 as
D−u(x0) :=
p : lim inf
x→x0 , x∈Ω
u(x)− u(x0)− p · (x− x0)
|x− x0|≥ 0
(4.10)
In order to give an equivalent definition for the first order case we state the following Lemma.
Lemma 4.5. Let u ∈ C(RN ) , we have
(i) A vector p ∈ D+u(x0) if and only if there exists a function φ ∈ C1(Ω) such that Dφ(x0) = pand u− φ has a local maximum at point x0.
(ii) A vector p ∈ D−u(x0) if and only if there exists a function φ ∈ C1(Ω) such that Dφ(x0) = pand u− φ has a local mimimum at point x0.
Proof. Since D−u(x0) = −D+(−u)(x0) we will detail here only the proof of (i).If p ∈ D+u(x0) there exists a δ > 0 such that
u(x) ≤ u(x0) + p · (x− x0) + σ(|x− x0|)|x− x0| ∀x ∈ B(0, δ)
where σ : [0,+∞[→ [0,+∞[ is a continuous increasing function such that σ(0) = 0. If we define
ρ(r) :=
∫ r
0σ(t) dt and set φ(x) := u(x0) + p · (x− x0) + ρ(2|x− x0|) we can easily check that u− φ
has a maximum point in x0 and Dφ(x0) = p (note that ρ(2r) ≥ rσ(r), ∀r ≥ 0.)Conversely if x0 is a maximum point of u− φ and p = Dφ(x0) we have
lim supx→x0 , x∈Ω
u(x)− u(x0)− p · (x− x0)
|x− x0|≤ lim sup
x→x0 , x∈Ω
φ(x)− φ(x0)−Dφ(x0) · (x− x0)
|x− x0|= 0
therefore p ∈ D+u(x0).
Recall that if u is differentiable at point x then D−u(x) = D+u(x) = Du(x). With this remarkand the Maximum Principle (Theorem 4.3) in hand, we easily see that any C1-viscosity solution isa classical solution.
It follows directly that the following is an equivalent definition of viscosity solution for the firstorder equation in Ω ⊂ RN
H(x, u,Du) = 0 x ∈ Ω. (4.11)
Definition 4.6. Viscosity solution II.A function u ∈ C(Ω) is a viscosity subsolution of (4.11) if and only if
H(x0, u(x0), p) ≤ 0 , ∀p ∈ D+u(x0) . (4.12)
11
A function u ∈ C(Ω) is a viscosity supersolution of (4.11) if and only if
H(x0, u(x0), p) ≥ 0 , ∀p ∈ D−u(x0) . (4.13)
If u verifies both (4.12) and (4.13) then we say that is a viscosity solution of (4.11).
The main point is that with this definition we select among all the generalized solution the”correct one” form the optimal control theory point of view. This can be easily shown is the case of
Example 4.2. Indeed, consider V (x) = 12 −
∣∣∣12 − x∣∣∣. The only point in which it is not differentiable
is x = 12 . We have D+V (1
2) = [−1; 1] and D−V (12) = ∅ therefore if we define H(x, u, p) = |p| − 1 it
is easy to verify that |p| − 1 ≥ 0, ∀p ∈ D−V (12) and |p| − 1 ≤ 0, ∀p ∈ D+V (1
2).
Remark 4.7. We want to underline here that when dealing with viscosity solution one has to bereally careful in the choice of the signs. Indeed, note that if we consider the equation |u′| = 1 onecould think that choosing H(x, u, p) = |p| − 1 or H(x, u, p) = 1 − |p|, is equivalent: this is not thecase! Indeed, following Definition (4.6) one can see that the viscosity solution of 1 − |Du| = 0 is
given by −V (x) = −12 +
∣∣∣12 − x∣∣∣ and not V (x).
4.3 The value function is a solution of an Hamilton-Jacobi equation.
Let us prove now that the value functions of the optimal control problems we consider are viscositysoution of an Hamilton Jacobi equation. Note that from a EDP point of view this is an existenceresult.
Theorem 4.8. The Hamilton-Jacobi equation for the infinite horizon problem.Assume (Ha), (Hf) and (Hl). The value function defined in (2.5) is a continuous viscosity solutionof
λV (x) +H(x,DV ) = 0 x ∈ RN , (4.14)
where the Hamiltonian H : RN × RN → R is given by
H(x, p) := supα∈A−p · f(x, α)− l(x, α). (4.15)
Proof. We split the proof in three steps.
V is a bounded continuous function.We first remark that by definition of the cost function we have |J(x, α)| ≤ Ml
λ , for all α ∈ A,
therefore |V (x)| ≤ Mlλ , for any x ∈ RN .
To prove the continuity let us fix x, z ∈ RN , for all ε > 0 there exists αε ∈ A such that
v(z) + ε ≥∫ +∞
0l(yz(t, αε), αε(t))e
−λt dt
Therefore
v(x)− v(z) ≤∫ +∞
0l(yx(t, αε), αε(t))− l(yz(t, αε), αε(t))e−λt dt+ ε.
12
The idea is now to split this integral in two parts by choosing a time T such that 2Mλ e−λT ≤ ε, in
order to write
v(x)− v(z) ≤∫ T
0l(yx(t, αε), αε(t))− l(yz(t, αε), αε(t))e−λt dt+
2M
λe−λT + ε
≤∫ T
0ωl(|yx(t, αε)− yz(t, αε)|)e−λt dt+ 2ε ≤
∫ T
0ωl(e
LlT |x− z|)e−λt dt+ 2ε
where we used assupltion (Hl) and the classical inequality (2.2) (|yx(t, αε)−yz(t, αε)| ≤ eLlT |x−z|).Since the roles of x and z are arbitrary this yelds to
|v(x)− v(z)| ≤∫ T
0ωl(e
LlT |x− z|)e−λt dt+ 2ε = ωl(eLlT |x− z|)
∫ T
0e−λt dt+ 2ε
which gives the uniform continuity of the value function V .
V is a subsolution. Let φ ∈ C1(RN ) and x be a maximum point of V −φ such that V (x)−φ(x) =0. Take any constant admissible control α(·) = α ∈ A, by the Dynamic programming Principle(Theorem 3.3), we have
V (x) ≤∫ t
0l(yx(s, α), α) e−λs ds+ e−λtV (yx(t, α)).
Since x is a zero-maximum point we have 0 = V (x) − φ(x) ≥ V (yx(t, α)) − φ(yx(t, α)), thereforee−λtV (yx(t, α)) ≤ e−λtφ(yx(t, α)), V (x) = φ(x) and the above inequality reads
φ(x) ≤∫ t
0l(yx(s, α), α) e−λs ds+ e−λtφ(yx(t, α)). (4.16)
The main point here is that φ ∈ C1(RN ) therefore we can exploit the same calculation as in (4.3)without assuming that V ∈ C1(RN ). More precisely we have
φ(x)− e−λtφ(x) + e−λtφ(x)− e−λtφ(yx(t, α))−∫ t
0l(yx(s, α), α) e−λs ds ≤ 0. (4.17)
Since the trajectory yx(·, α) is a solution of (2.1), we have
φ(yx(t, α))−φ(x) =
∫ t
0
d
dsφ(yx(s, α))ds =
∫ t
0Dφ(yx(s, α))·y′x(s, α)ds =
∫ t
0Dφ(yx(s, α))·f(yx(s, α), α)ds
(4.18)therefore, also dividing by t > 0, (4.17) gives us
(1− e−λt
t)φ(x) +
1
t
∫ t
0
(−e−λtDφ(yx(s, α)) · f(yx(s, α), α)− l(yx(s, α), α) e−λs
)ds ≤ 0 .
Letting t→ 0 thanks to the regularity assumptions on f and l we have for each α ∈ A
λφ(x)−Dφ(x) · f(x, α)− l(x, α) ≤ 0
13
therefore, taking the supremum over the constant controls α ∈ A (recalling that φ(x) = V (x)) weget
λV (x) + supα∈A−Dφ(x) · f(x, α)− l(x, α) ≤ 0.
that is V is a subsolution of equation (4.14).
V is a supersolution. Let φ ∈ C1(RN ) and x be a local minimum point of V −φ, i.e V (z)−φ(z) ≤V (x) − φ(x), ∀z ∈ B(0, R). By the dynamic programming principle Theorem 3.4, for every ε > 0and t there exists a control αε ∈ A such that
V (x) + εt ≥∫ t
0l(yx(s, αε), αε(s)) e
−λs ds+ e−λtV (yx(t, αε))
therefore, by choosing t small enougt we have
φ(x)−φ(yx(t, αε)) ≥ V (x)−V (yx(t, αε)) ≥∫ t
0l(yx(s, αε), αε(s))e
−λsds−εt+(e−λt−1)V (yx(t, αε)).
(4.19)By assumption (Hl) and estimate (2.3) we note that |l(yx(s, αε), αε(s)) − l(x, αε)| ≤ ωl(Mfs)therefore∫ t
0l(yx(s, αε), αε(s))e
−λsds =
∫ t
0l(x, αε(s))e
−λsds+o(t) =
∫ t
0l(x, αε(s))ds+
∫ t
0l(x, αε(s))(e
−λs−1)ds+o(t)
Moreover,
φ(x)− φ(yx(t, αε)) = −∫ t
0
d
dsφ(yx(s, αε)) ds = −
∫ t
0Dφ(yx(s, αε)) · f(yx(s, αε), αε(s))ds =
= −∫ t
0Dφ(x) · f(x, αε(s))ds+ o(t)
thus (4.19) become∫ t
0−Dφ(x)·f(x, αε(s))−l(x, αε(s))ds ≥ −o(t)+
∫ t
0l(x, αε(s))(e
−λs−1)ds+o(t)−εt+(e−λt−1)V (yx(t, αε))
Since for any time s ∈ (0, t) we have
supα∈A−Dφ(x) · f(x, α)− l(x, α) ≥ −Dφ(x) · f(x, αε(s))− l(x, αε(s))
we get
supα∈A−Dφ(x)·f(x, α)−l(x, α)t−(e−λt−1)V (yx(t, αε)) ≥ −o(t)+
∫ t
0l(x, αε(s))(e
−λs−1)ds+o(t)−εt
dividing by t > 0 and letting t→ 0 by the continuity of the value function this yelds to
supα∈A−Dφ(x) · f(x, α)− l(x, α)+ λV (x) ≥ −ε
for ε→ 0 we finally have that V is a supersolution and complete the proof.
14
Theorem 4.9. The Hamilton-Jacobi equation for the finite horizon problem.Assume (Ha), (Hf) and (Hl). The value function defined in (2.7) is a continuous viscosity solutionof
∂V
∂t(x, t) + λV (x, t) +H(x,DxV ) = 0 x ∈ RN , (4.20)
where the Hamiltonian H : RN × RN → R is given by
H(x, p) := supα∈A−p · f(x, α)− l(x, α). (4.21)
Proof. This proof follows the same ideas of the proof of Theorem (4.8). For a detailed referencesee for instance Proposition 3.1 page 148 and Proposition 3.5 page 150 in [1].
Theorem 4.10. The Hamilton-Jacobi equation for the minimum time problem.Assume (Ha), (Hf) and (Hl). The value function defined in (2.9) is a viscosity solution of
H(x,DT ) = 1 in R \ T (4.22)
where the Hamiltonian H : RN × RN → R is given by
H(x, p) := supα∈A−p · f(x, α). (4.23)
Moreover, T is a continuous function in R, bounded below and satisfying:
T (x) = 0 on ∂T and T (x)→ +∞ as x→ x0 ∈ ∂R. (4.24)
Proof. See, for instance, Proposition 2.3 page 240 of [1].
5 An idea of the technique for the uniqueness results.
The main aim of this section is ti give the basic ideas to prove a comparison result for viscositysolutions from a PDE point of view. We consider then a bounded open subset Ω ⊂ RN and thefollowing first order equation
H(x, u,Du) = 0 x ∈ Ω (5.1)
coupled withe the boundary condition
u = g on ∂Ω. (5.2)
where g : ∂Ω→ R is a continuous function. We assume that H : RN ×R×RN → R is a continuousfunction fulfilling
(HH1) For each R > 0, there exists a strictly positive constant γR > 0 such that
γR(u− v) ≤ H(x, u, p)−H(x, v, p) for any x ∈ Ω , −R ≤ v ≤ u ≤ R , p ∈ RN
15
(HH2) For each R > 0 there exists a function mR : R+ → R+ such that mR(t)→ 0 when t→ 0 and
|H(x, u, p)−H(y, u, p)| ≤ mR(|x− y|(1 + |p|)) for any x ∈ Ω , −R ≤ u ≤ R , p ∈ RN .
We first give the definition of viscosity solution for this equation with boundary conditions.
Definition 5.1. Viscosity solutions.
a) u is a viscosity subsolution of (5.1)-(5.2) in Ω if it is upper semicontinuous, for each φ ∈C1(Ω) and local maximum point x0 ∈ Ω of u− φ we have
H(x0, u(x0), Dφ(x0)) ≤ 0 (5.3)
and satisfies u ≤ g on ∂Ω.
b) u is a viscosity supersolution of (5.1)-(5.2) in Ω if it is lower semicontinuous, for eachφ ∈ C1(Ω) and local minimum point x0 ∈ Ω of u− φ we have
H(x0, u(x0), Dφ(x0)) ≥ 0 (5.4)
and satisfies u ≥ g on ∂Ω.
u is a solution if it is both a viscosity sub and supersolution. (Note that then is a continuousfunction).
For the sake of simplicity in the following sub, super and solutions will be always understoodin the viscosity sense. In order to have a uniqueness result we will prove the so called comparisonresult, more precisely
(CR) Let u be a subsolution of (5.1)-(5.2) and v be a supersolution of (5.1)-(5.2), we want to provethat u(x) ≤ v(x) in all Ω.
Note that once the comparison is proved the uniqueness follows easily. Indeed, suppose by con-tradiction that u1 and u2 are both viscosity solutions of (5.1)− (5.2) since they are both sub andsuper solution by comparison we have u1 ≤ u2 and u1 ≥ u2, therefore u1 = u2.
To explain the idea of the proof of the comparison principle let us suppose first that u and vare regular: u, v ∈ C1(R). The proof goes like this:Let x0 be an interior maximum point of u(x)− v(x) and suppose that u, v are smooth at point x0.Since u and v are respectively a sub and a supersolution we have
H(x0, u(x0), Du(x0)) ≤ 0 and H(x0, v(x0), Dv(x0)) ≥ 0
therefore (recalling that Du(x0)−Dv(x0) = 0), we have
H(x0, u(x0), Du(x0))−H(x0, v(x0), Du(x0)) ≤ 0
16
If, by the assumption one has on H, this implies u(x0)− v(x0) ≤ 0 the proof is concluded. In fact,since x0 is a maximum this gives u(x)− v(x) ≤ u(x0)− v(x0) ≤ 0 for all x ∈ Ω.
Note that in the case of the Hamiltonian for the infinte horizon problem for example the con-clusion easily follows. Indeed, in that case H(x, u, p) = λu+H(x, p) thereforeH(x0, u(x0), Du(x0))−H(x0, v(x0), Du(x0)) = λ(u(x0)− v(x0)).
Unfortuntaly real life is much more complicate and, as we have seen before, the assumption uand v smooth it is not reasonable. To overcame this lack of regularity the idea is to double thevariable. More precisely, we chose a test function φ : Ω×Ω→ R for which u(x)− v(y)−φ(x, y) hasa maximum at (x0, y0) thus x0 is a maximum for x → u(x) − (v(y0) + φ(x, y0)) (and the functionv(y0)+φ(x, y0) is now regular in the x variable), and y0 is a minimum for y → v(y)−(u(x0)−φ(x0, y))(and the function u(x0) − φ(x0, y) is now regular in the y variable). Thus we can apply now thedefinition of viscosity subsolution and viscosity supersolution respectively, and we get
H(x0, u(x0), Dxφ(x0, y0))−H(y0, v(y0),−Dyφ(x0, y0)) ≤ 0
The main idea to conclude is to choose the test function φ is such a way that Dxφ(x0, y0) =−Dyv(x0, y0) and exploit the hypotheses on H.
We prove now the result for the general equation (5.1)-(5.2) under the assumption that the suband super solutions u and v are continuous functions in Ω.
Theorem 5.2. Let Ω be an open bounded subset of RN . Assume (HH1) and (HH2). Let u andv be continuous functions in Ω. If u is a subsolution of (5.1)-(5.2) and v is a supersolution of(5.1)-(5.2), we have
u(x) ≤ v(x) for all x ∈ Ω.
Proof. Since u and v are continuous in a compact set the function u− v has a maximum pointin Ω, more precisely there exists a point x0 ∈ Ω such that
maxx∈Ω
(u(x)− v(x)) = u(x0)− v(x0) := M
Our aim is then to prove that M ≤ 0. There are two possibilities.If x0 ∈ Ω we have by definition of sub and super solution u(x0) ≤ g(x0) ≤ v(x0) therefore M ≤ 0and the proof is achieved.Let then suppose that x0 ∈ Ω. We assume by contradiction that M > 0.Following the idea of doubling the variable, for each ε > 0 we consider the following test function
ψε(x, y) := u(x)− v(y)− |x− y|2
ε2
and we denote by (xε, yε) the maximum point of ψε(x, y) in Ω× Ω. We set Mε := ψε(xε, yε).
Note that this is a good candidate since the penalizing term |x−y|2ε2
force the x ”near” y when ε tends
to zero and then we can expect that Mε is ”near” M . Moreover Dx( |x−y|2
ε2) = −Dy(
|x−y|2ε2
).More precisely we have the following (for the proof see below).
17
Lemma 5.3. For ε small enough (xε, yε) ∈ Ωε, moreover1) Mε →M as ε tends to 0.2) u(xε)− v(yε)→ 0 as ε tends to 0 and
|xε − yε|2
ε2→ 0 as ε→ 0
Since (xε, yε) is a maximum for ψε(x, y) we have :
-xε ∈ Ω is a maximum for x 7→ u(x)− (v(yε) +|x− yε|2
ε2) with v(yε) +
|x− yε|2
ε2∈ C1(RN ) and u a
subsolution, therefore H(xε, u(xε),(xε−yε)ε2
) ≤ 0.
-yε is a minimum for y 7→ v(y) − (u(xε) −|xε − y|2
ε2) with u(xε) −
|xε − y|2
2ε2∈ C1(RN ) and v a
supersolution, therefore H(yε, v(yε),(xε−yε)ε2
) ≥ 0.
Thus, setting pε :=(xε − yε)
ε2we have finally obtained H(xε, u(xε), pε)−H(yε, v(yε), pε) ≤ 0.
By adding and subtracting the term H(xε, v(xε), pε) this yelds to
H(xε, u(xε), pε)−H(xε, v(xε), pε) ≤ H(xε, v(xε), pε)−H(yε, v(yε), pε)
therefore by applying assumptions (HH1) and (HH2) we have
γR (u(xε)− v(xε)) ≤ mR(|xε − yε|(1 + |pε|)) = mR(|xε − yε|+|xε − yε|2
ε2)
Recalling that Mε = u(xε)− v(xε)− |xε−yε|2
ε2we finally have
γRMε ≤ mR(|xε − yε|+|xε − yε|2
ε2)− γR
|xε − yε|2
ε2
therefore letting ε → 0 by Lemma 5.3 we obtain the contradiction γR M ≤ 0 and the proof iscompleted.
Proof of Lemma 5.3. Since (xε, yε) is a max for ψε(x, y) we have u(x)−v(y)− |x− y|2
ε2≤Mε
for all x, y ∈ Ω, therefore by choosing y = x we have u(x)− v(x) ≤ Mε for all x ∈ Ω, that impliesM ≤Mε. Similarly, since u and v are bounded functions we have
u(x)− v(x) ≤Mε = u(xε)− v(yε)−|xε − yε|2
ε2≤ 2R− |xε − yε|
2
ε2
therefore, passing to the sup over x ∈ Ω and recalling that M > 0 we have the following estimatefor the penalizing term
0 < M ≤ 2R− |xε − yε|2
ε2=⇒ |xε − yε|2
ε2≤ 2R =⇒ |xε − yε| ≤
√2R ε. (5.5)
Let mv be a modulus of continuity for the continuous function v, by the positivity of the penalizingterm we have
Mε ≤ u(xε)− v(yε)−|xε − yε|2
ε2≤ u(xε)− v(xε) +mv(|xε − yε|) ≤ u(xε)− v(xε) +mv(
√2R ε)
18
thenM ≤Mε ≤M +mv(
√2R ε) =⇒ Mε →M when ε→ 0 .
Moreover
Mε = u(xε)− v(yε)−|xε − yε|2
ε2≤ u(xε)− v(yε) ≤M +mv(
√2R ε)
gives u(xε)− v(yε)→ 0 as ε tends to zero. Therefore since Mε = u(xε)− v(yε)− |xε−yε|2
ε2and both
Mε and u(xε)− v(yε) tend to zero as ε→ 0 we can conclude that|xε − yε|2
ε2→ 0 as ε→ 0 and the
proof is completed.
5.1 The uniqueness result for the infinite horizon, the finite horizon and theminimum time problem.
In order to discuss the uniqueness result for the optimal control problem we considered we firstprove the properties of the Hamiltonian
H(x, p) := supα∈A−p · f(x, α)− l(x, α). (5.6)
Lemma 5.4. Assume (Ha), (Hf) and (Hl). We have
|H(x, p)−H(y, p)| ≤ Lf |p||x− y|+ ωl(|x− y|) ∀x, y ∈ RN , ∀p ∈ RN (5.7)
and|H(x, p)−H(x, q)| ≤Mf |q − p| ∀x ∈ RN , ∀p, q ∈ RN (5.8)
Proof. Fix x, p ∈ RN by definition of the Hamiltonian, for each ε > 0 there exists αε ∈ A suchthat H(x, p)− ε ≤ −f(x, αε) · p− l(x, αε) moreover H(y, p) ≥ −f(y, αε) · p− l(y, αε), therefore
H(x, p)−H(y, p) ≤ −f(x, αε) · p− l(x, αε) + f(y, αε) · p+ l(y, αε) + ε ≤ Lf |p||x− y|+ ωl(|x− y|)
thanks to assumptions (Hf) and (Hl). Changing the roles of x and y we easily obtain (5.7). Theproof of (5.8) follows the same arguments.
We have the following results.
Theorem 5.5. Uniqueness for the infinite horizon problem.Assume (Ha), (Hf) and (Hl). The value function defined in (2.5) is the unique continuous viscositysolution of
λV (x) +H(x,DV ) = 0 x ∈ RN , (5.9)
where the Hamiltonian H : RN × RN → R is given by
H(x, p) := supα∈A−p · f(x, α)− l(x, α). (5.10)
19
Proof. Let us first observe that is we set H(x, u, p) := λu+H(x, p) it is immediate to see thatassumptions (HH1) and (HH2) in the comparison result Theorem 5.2 are fulfilled. Indeed λ > 0implies (HH1) and (HH2) follows directly from (5.8). However, since Theorem 5.2 is proved in abounded domain we can not apply it directly. The proof goes in a similar why but with a differentchoice of the test function. We need to add a term that penalize the ’infinity” in order to be surethat the maximum is attained. A possible choice is
ψε,δ(x, y) := u(x)− v(y)− |x− y|2
ε2− δ(|x|2 + |y|2)
For more details in this case and a compete proof one can see Theorem 2.11 in [2] or Theorem 2.12in [1]
Theorem 5.6. Uniqueness for the finite horizon problem.Assume (Ha), (Hf) and (Hl) and λ ≥ 0. The value function V defined in (2.7) is the uniquecontinuous viscosity solution of
∂V
∂t(x, t) + λV (x, t) +H(x,DxV ) = 0 x ∈ RN , (5.11)
with initial condition V (x, 0) = g(x) , ∀x ∈ RN . The Hamiltonian H : RN × RN → R is given by
H(x, p) := supα∈A−p · f(x, α)− l(x, α). (5.12)
Proof. We remark here that if λ > 0 this case in treated exactly as in the infinite horizonproblem, if λ = 0 is the term with the time derivative that ensures us the right estimate toconclude in the comparison result. For a detailed reference see for instance Theorem 3.7 page 152in [1].
Theorem 5.7. Uniqueness for the minimum time problem.Assume (Ha), (Hf) and (Hl). The value function defined T in (2.9) is the unique function contin-uous in R, bounded below, viscosity solution of
H(x,DT ) = 1 in R \ T (5.13)
T (x) = 0 on ∂T and T (x)→ +∞ as x→ x0 ∈ ∂R. (5.14)
where the Hamiltonian H : RN × RN → R is given by H(x, p) := supα∈A−p · f(x, α).
Proof. See, for instance, Theorem 2.6 page 241 of [1].
20
6 A stability result and a further remark on the origin of thedefinition.
We will state here a stability result for a second order equation in an open subset Ω ⊆ RN . For amore general discussion on different stability results we refer to [2].
Theorem 6.1. Stability result.For each ε > 0 assume that Fε : Ω×R×RN × S(N)→ R is a continuous function fulfilling (HFe)and (HFp). Let uε ∈ C(Ω) be a subsolution (resp. supersolution) of
Fε(x, uε, Duε, D2uε) = 0 x ∈ Ω
If the sequences (uε)ε and (Fε)ε converge uniformly on compact set to a fonction u and F respec-tively, then u is a subsolution (resp. supersolution) of
F (x, u,Du,D2u) = 0 x ∈ Ω
Proof. Let us only prove the subsolution case, the proof for supersolution being completelysimilar. Let φ ∈ C2(Ω) and x0 is a local maximum point of u− φ we aim to prove that
F (x0, u(x0), Dφ(x0), D2φ(x0)) ≤ 0 ,
By replacing for example φ by φ + |x − x0|4 we first remark that it is not restrictive to supposethat this maximum is strict. Therefore we can apply the following lemma at the sequence uε − φ
Lemma 6.2. Consider a sequence (vε)ε such that ∀ε > 0, vε ∈ C(Ω) and vε → v in C(Ω) (i.e.uniformly on compact subset). If x0 is a strict local maximum point of v then there exists a sequence(xε)ε of local maximum points of vε such that xε → x0.
Let then xε be a maximum point of uε − φ, since uε is a subsolution we have
Fε(xε, uε(xε), Dφ(xε), D2φ(xε)) ≤ 0
moreover by the regularity of φ we get Dφ(xε) → Dφ(x0) and D2φ(xε) → D2φ(x0). Since Fεconverges uniformily on compact set this implies for ε→ 0,
F (x0, u(x0), Dφ(x0), D2φ(x0)) ≤ 0 ,
and the proof is achieved.
In order to give a further remark on the origin of the definition we consider now the ”stochas-tic version” of the infinite horizon optimal control problem. The state equation is the followingstochastic differential equation
dy = f(y(t), α(t))dt+√
2εdw(t)y(0) = x
(6.1)
21
where ε > 0 and w is the N -dimensional standard Brownian motion. The value function will behere
vε(x) = infα∈A
Ex
(∫ +∞
0l(yεx(t, α), α(t)) e−λt dt
)where Ex denotes the expectation and the infimum is taken on the class of progressive measurablefunctions α with values on A. Under suitable conditions on the data that we will not detail herevε happens to be a smooth (at least C2) solution of the following second order equation
−ε∆vε(x) + λvε(x) + supα∈A−Dvε(x) · f(x, α)− l(x, α) = 0 x ∈ RN .
A natural question arises: if ε→ 0 does vε tends to a function v solution of the limit equation
λv(x) + supα∈A−Dv(x) · f(x, α)− l(x, α) = 0 x ∈ RN .
The question is not so easy because the regularizing effect of the term ε∆vε(x) vanishes as ε → 0but the answer is yes and the right sense in which understand the solution of the limit equation isof course the viscosity one. This is very important because can be thought as a way to define weaksolution of the limit equation, and is actually the motivation for the terminology viscosity solutionused in the original paper of M.G. Crandall and P.L.Lions [3].We conclude this section roughly giving the idea of the proof (for subsolution). Assume vε ∈ C2(RN )and vε converging to a continuous function v locally uniformly as ε tends to 0. Let φ ∈ C2(RN )and x0 be a local maximum point of v−φ. By uniform convergence vε−φ attains a local maximumat some point xε0 and xε0 → x0 as ε → 0. So by elementary calculus, D(vε − φ)(xε0) = 0 and−∆(vε − φ)(xε0) ≥ 0. Moreover, vε is a classical solution, thus
0 = −ε∆vε(xε0) + λvε(xε0) + supα∈A−Dvε(xε0) · f(xε0, α)− l(xε0, α) ≥
≥ −ε∆vε(xε0) + λvε(xε0)−Dvε(xε0) · f(xε0, α)− l(xε0, α) ≥
≥ −ε∆φ(xε0) + λφ(xε0)−Dφ(xε0) · f(xε0, α)− l(xε0, α) .
Assuming enough regularity on the data f and l we can let ε→ 0 in the last inequality and obtain,for each α ∈ A,
0∆φ(x0) + λφ(x0)−Dφ(x0) · f(x0, α)− l(x0, α) ≤ 0
thusλφ(x0) + sup
α∈A−Dφ(x0) · f(x0, α)− l(x0, α) ≤ 0
that is, by definition, v is a viscosity subsolution.
22
7 A primer on Semi-Lagrangian schemes:the infinite horizon and the minimum time problem.
The reference for all this section is the book of Falcone and Ferretti [6]. (In particular Chapter 8.)I particularly thank M. Falcone for the courtesy of giving me the images at the end of this section.The basic idea of the Semi-Lagrangian scheme is to solve the problem by using the DynamicProgramming Principle.We start by approximating the state equation (2.1) for example by an explicit Euler scheme:
ym+1 = ym + ∆t f(ym, am) m ∈ Ny0 = x
(7.1)
where ∆t is given and am ∈ A is the discrete control. We denote by α∆t the whole sequence (am)mand by ym(x, α∆t) the solution of (7.1).As a second step we construct a discrete version of the cost functional. For the infinite horizonproblem we can, for example, approximate the cost (2.4) by a rectangle quadrature
J∆tx (α∆t) = ∆t
+∞∑m=0
l(ym, am) e−λtm (7.2)
where tm = m∆t. Therefore the definition of the discrete value function follows naturally
V ∆t(x) := infα∆t
J∆tx (α∆t) . (7.3)
With a completely similar proof we can derive the discrete version of the DPP: for all x ∈ RN andn ∈ N we have
V ∆t(x) = infα∆t
∆t
n−1∑m=0
l(ym, am) e−λtm + e−λtn V ∆t(yn(x, α∆t))
.
By choosing n = 1 we derive the easiest Semi-Lagrangian scheme:
V ∆t(x) = mina∈A
∆t l(x, a) + e−λ∆t V ∆t(y(x+ f(x, a) ∆t)
where sometimes e−λ∆t is approximate by the 1st order Taylor expansion 1 − λ∆t. To completethe discretization we have now to add a spatial discretization, for example one can replace v∆t bya polynomial interpolation I[V ] and obtain a complete fully discrete scheme:
Vj = mina∈A
l(xj , a) ∆t+ e−λ∆t I[V ](xj + f(xj , a) ∆t)
:= S(∆, V ) .
This scheme can be written in an iterative way as follows: V (k+1) = S(∆, V (k)). As explained inSection 8.4.1 in [6] on can prove that:
1. S(∆, ·) is monotone.
2. S(∆, ·) is a contraction in l∞.
23
3. The scheme is l∞ stable.
4. The scheme converge with the following error estimate |vj − v(xj)| ≤ C1∆t+ C2∆x∆t .
The minimal time problem.In order to solve the minimum time problem numerically we introduce the Kruzkow transformation.This is very useful also from a theoretical point of view because it allows to avoid the infiniteboundary conditions in Theorem 4.10. We set:
v(x) :=
1 if T (x) = +∞1− e−T (x) if T (x) < +∞
This transformation is very useful becouse v(x) is the value function of the infinite horizon controlproblem with running cost l(x, α) = 1 and λ = 1. Indeed set
Jx(α) :=
∫ tx(α)
0e−s ds
it is easy to see that v(x) = infα∈A Jx(α), therefore we can characterize v as the unique solution ofthe following Hamilton-Jacobi equation
λv(x) + supα∈A−Dv(v) · f(x, α)− 1. = 0 x ∈ RN \ T ,
with boundary condition v(x) = 0 for any x ∈ T . Let us remark that 0 ≤ v(x) ≤ 1, and that oncewe have computed v we can reconstruct the minimum time function T and the reachable set asfollows
T (x) = − ln(1− v(x)) R := x ∈ RN : v(x) < 1.
By this transformation we can apply the numerical scheme for the infinite horizon problem toconstruct a Semi-Lagrangian scheme for the minimum time problem. This is done in Section 8.8.4in [6] and applied to solve the following example.
Example 7.1. This example is treated in section 8.4.9 of [6]. We are in R2, the target is the pointT := (−1,−1) the admissible controls take values in B(0, 1), the dynamic is f(x, a) = c(x)a wherethe velocity c is given by
c(x1, x2) :=
0 if (x1, x2) ∈ ([0, 0.5]× [−2, 1.5]) ∪ ([1, 1.5]× [−1.5, 2])1 elsewhere
Note that the two region where c vanish are two obstacle.In the following images are represented the domain of the equation (fig.1) , the minimum timefunction (fig. 2) and the optimal trajectory (fig .3 ) obtained by a Semi-Lagrangian scheme.
24
Figure 1: The domain of the equation
2
1
0
1
2
2
1
0
1
20
2
4
6
8
10
12
Figure 2: The minimum time function
25
1.5 1 0.5 0 0.5 1 1.5 2
1.5
1
0.5
0
0.5
1
1.5
2
griglia= 80x80 FMM SL
Figure 3: The optimal trayectory
References
[1] M. Bardi, I. Capuzzo Dolcetta, Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, Systems & Control: Foundations & Applications, Birkhauser Boston Inc.,Boston, MA, 1997.
[2] G. Barles, Solutions de viscosite des equations de Hamilton-Jacobi, Springer-Verlag, Paris, 1994.
[3] M.G. Crandall, P.L.Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer.Math.Soc. 277, (1983), 1-42.
[4] M. G. Crandall, H. Ishii, P. L. Lions, User’s guide to viscosity solutions of second order partialdifferential equations, Bull. Amer. Math. Soc. (N.S.) 27 (1992) 1-67.
[5] M.G. Crandall - L.C. Evans - P.L.lions, Some properties of viscosity solutions of Hamilton-Jacobiequations, Trans. Amer. Math. Soc. 282, (1984), 487-502.
[6] M. Falcone - R. Ferretti Semi-Lagrangian Approximation Schemes for Linear and Hamilton-Jacobi Equations, SIAM (2014).
[7] W.H. Fleming, H.M. Soner, Controlled Markov Processes and Viscosity Solutions, Applicationsof Mathematics, Springer-Verlag, New York, 1993.
[8] Lions P.L. (1982) Generalized Solutions of Hamilton-Jacobi Equations, Research Notes in Math-ematics 69, Pitman, Boston.
26