Opt

imal

Con

trol

201

9/20

20

Lecture 1

Optimal Control WS 2019/2020

Prof. Dr.-Ing. Rolf Findeisen

Laboratory for Systems Theory and Automatic Control

Opt

imal

Con

trol

201

9/20

20

Team

Lecturer

Prof. Rolf Findeisen

Dr. Navid Noroozi

Assistants

Hoang Hai Nguyen

Hannes Rewald

Mohamed Ibrahim

Process Control and ModelingAchim Kienle

Institute for Automation Otto-von-Guericke-University Magdeburg

3

• 65 employees, approx. 45 Ph.D. students• Close connection to• 7 research groups/chaired positions (at the University)

Autonomous Automation Systems Steffi Knorn

Integrated AutomationChristian Diedrich

Process AutomationUlrich Jumar

Measurement TechnologyUlrike Steinmann

Control of Distributed Parameter SystemsStefan Palis

Systems Theory and ControlRolf Findeisen

Research Activities• 6 Postdocs/scientific associates• 21 Phd students• Acad. coop. : MIT, EPFL, ETH, UC Berkley, Imperial Coll., DLR, ...• Industrial coop. : Airbus, Baker Hughes, Bosch, IAV, Siemens, Volkswagen,...

Fields of Applications• Autonomous Systems, Autonomous Driving• Energy Systems (Smart Grids, Batteries, Wind Energy…)• Robotics, Aerospace, UAV• Chemical Processes, Biotechnologies, Biopharmaceuticals

Theoretical Basis• Optimal and predictive control (MPC)• Machine Learning and Artificial Intelligence• Uncertainty Description and Quantification

Focal Research Areas• Fusing Machine Learning and Control with Guarantees• Distributed Systems, Modularization and Scalability, Cooperative Systems• Cyber Physical Systems, Network Controlled Systems• Fusion of Planning and Control• Identification, Verification, Validation• Embedded optimization

Goal: Development of theoretical sound control methods for save, flexible cooperative autonomous systems

Opt

imal

Con

trol

201

9/20

20

Objectives of the lecture

Objectives• Introduce main concepts and theory behind optimal control• Overview of numerical solution approaches• Overview of embedded optimization/model pred. control

Optimal control:Find an input/feedback for a dynamical system such that a performance measure is optimized satisfying constraints

?

?

Lecture is method and theory oriented!

Opt

imal

Con

trol

201

9/20

20

Organization

Format

• 2 SWS lecture, 1 SWS exercise• lectures and exercises are given in English

Lectures and exercises are handled flexible

• XXX XXX XXX• Wednesday, 17:00-19:00 G03-315• Thursday, 11:00-13:00 G02-109

Announcements/downloadshttp://ifatwww.et.uni-magdeburg.de/syst/education/courses/oc/

Exercises

• 4 exercises (classroom and computer) + 2 general Q&A sessions • exercise sheets are available approx. 1 week before exercise

Consultation hours

• make an appointment via emailNext forseen lectures/exercises

30.10, 17:00-18:00, 25.10, 1.11, 8.11

Opt

imal

Con

trol

201

9/20

20

Recommended literature

Optimal Control[1] R. Bellman. Dynamic Programming. Princeton University Press, Princeton, New Jersey, 1957. [2] L.D. Berkovitz. Optimal Control Theory. Springer-Verlag, New York, 1974. [3] D.P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific Press. 2nd edition, 2000. [4] L.M. Hocking. Optimal Control. An Introduction to the Theory with Applications. Oxford Applied Mathematics and Computing Science Series. Oxford University Press, Oxford, 1991. [5] J.L. Troutmann. Variational Calculus and Optimal Control. Undergraduate Texts in Mathematics. Springer, 1991.

Optimization[6] S. Boyd, L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004. [7] J. Nocedal, S. Wright. Numerical Optimization. Springer, 2006.

Model Predictive Control[8] J.B. Rawlings, D.Q. Mayne. Model Predictive Control: Theory and Design, 2009. [9] E.F. Camacho, C. Bordons. Model Predictive Control, Springer, 1995.

Opt

imal

Con

trol

201

9/20

20

Structure

1. IntroductionWhat is optimal control?Examples

2. Static Optimization

3. Basic Setup of optimal control problemsCost function, constraintsExistence of solutions

4. Analytic approaches to optimal controlDynamic ProgrammingProntryagin minimum principle

5. Numerical approaches to optimal controlDirect and indirect methodsConvex optimizationModel predictive control

6. Embedded model predictive controlEmbedded optimizationSolution of QPs for MPC on embedded platformsCode generation and implementation aspects

0 0 0 0 0 0

1 1 1 1

-1 -1 -1 -1

Stage 0 Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

“Time“

“S tate“Initial-node

Terminal-node

5 4 1

5 1 5

2 3 41 3

44 1

11

2

2 2

2

233

3

3

CostSource: NASA

Opt

imal

Con

trol

201

9/20

20

Introduction to optimal control

•Optimal control?•Examples•Mathematical setup•Remarks on existence of solutions

Opt

imal

Con

trol

201

9/20

20

Optimal Control

Goal: „To optimize the operation of a dynamical system.“

„optimize“ could be minimization or maximization• Minimize cost/energy• Maximize profit• Minimize tracking error• …

Opt

imal

Con

trol

201

9/20

20

Optimal Control

Given a dynamical system , determine open-loop input or feedback , such that we are• minimizing performance objective• satisfying constraints: input constraints

state constraints

Key components: 1. model of the system2. constraints3. performance objective/functional

?

?

Opt

imal

Con

trol

201

9/20

20

Example (rocket car)

Objective: minimize fuel expense while bringing the car from to in a finite time ► there can be many solutions, or none, depending on constraints.

Mathematical:

s.t.:

no friction (designed with optimization methods)

Remarks: • is a function, not a single value; • is a functional of (in general of )

►we have to look at the differential equation (it is a problem!)

Opt

imal

Con

trol

201

9/20

20

Example:

Difference to static optimization

Static optimization:

s.t.

• No dynamical system• Finite dimensional

• u is not a function/no trajectory• F is not a functional

Opt

imal

Con

trol

201

9/20

20

More examples

Example (rocket car 2): bring the car in minimum time to the origin

Example (rocket car 3): minimize least square cost for fixed

is free

► Linear Quadratic Regulator (LQR)

Opt

imal

Con

trol

201

9/20

20

More examples: discrete time

Example (managing spending/savings): maximize consumption over n years

income per yearconsumed money

System model:

Objective:

Constraint:

given

Remarks:• discrete problem• end time fixed• is free

System model

Constraints

Objective

Opt

imal

Con

trol

201

9/20

20

More examples: traveling salesman (1800s, 1930s)

Traveling salesman problem: visit all cities/shortest path

► Shortest path problem• Very classical problem• Multi-stage optimization problem ► Bellman, Dynamic Programming• Finite dimensional• Still a lot of ongoing research

0 0 0 0 0 0

1 1 1 1

-1 -1 -1 -1

Stage 0 Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

“Time“

“S tate“Initial-node

Terminal-node

5 4 1

5 1 5

2 3 41 3

44 1

11

2

2 2

2

233

3

3

Cost

Opt

imal

Con

trol

201

9/20

20

More examples: Goddard‘s Rocket Problem (1919)

How to use the throttle to send the rocket as high as as possible?

System model

Constraints Objective

Height (h) Mass (h)

Velocity (v) Thrust (u)

www.mcs.anl.gov

Opt

imal

Con

trol

201

9/20

20

More examples: penicillin production

System model

Constraints Objective

where

How to use the input to a reactor to produce as much penicillin as possible?

Opt

imal

Con

trol

201

9/20

20

Control in the era of communication

Technology advancements in many fields1. cheap communication

• high speed and reliability• affordable wireless communication

2. affordable computation & memory• computational power even in the smallest device• possibility to store and process (large) amounts of data

3. new sensors and actuators4. reliable batteries & improved energy efficiency

• How to control large networks of interconnected systems?• Smart energy networks• Smart buildings• Smart factories

• Optimal control and embedded optimization will play a key role

Opt

imal

Con

trol

201

9/20

20

Internet of things - Industry 4.0: The future?

Internet of things:• Even the smallest device will have the possibility

of computing control commands and communicating information

ECU

Embedded system

Si

Technology advancements open new possibilities for the future of computing

Industry 4.0:The fourth industrial revolution1. First revolution: mechanization steam engine (1800s)2. Second revolution: electrification electricity (1900s)3. Third revolution: digitalization computer (1960s)4. Fourth revolotion: computarization sens. Inf. & comp. (Today?)

Six design principles of Industry 4.0:1. Interoperatibility2. Virtualization3. Decentralization4. Real-time Capability5. Service Orientation6. Modularity

Opt

imal

Con

trol

201

9/20

20

Advantages and challenges optimal control

Advantages

• Systematic way to design controller/find optimal input► objective, model, constraints

• Consider constraints• Nonlinear systems• Nature behaves “optimal” (►Fermat‘s principle)

Drawbacks/challenges• Finding a solution is challenging

• Often not possible in an analytic way (only linear case trivial)• Numerical solution is necessary

• Existence? uniqueness?

Fields of applications:economics, aeronautics, mechanical engineering, chemical engineering, biology, medicine, electrical engineering, information technology, …

Opt

imal

Con

trol

201

9/20

20

Static Optimization

Opt

imal

Con

trol

201

9/20

20

Why static optimization for optimal control?

Direct approaches transform optimal control problems into static optimization problemsHow to solve static optimization problems?

Necessary and sufficient conditions for optimality• Unconstrained optimization• Constrained optimization

• Equality constraints• Inequality constraints

Usually iterative algorithms are based on derivative information

Opt

imal

Con

trol

201

9/20

20

Definition (Optimality):A feasible optimal point is optimal if

Optimal points are denoted by:

Static optimization

Static optimization: Find min (max) of a scalar objective function subject to constraints.

When satisfies (1) and (2), it is called a feasible point.

maximum?

Opt

imal

Con

trol

201

9/20

20

Example: ball and spring

Goal: find point of rest ►minimize potential energy of spring and ball

Opt

imal

Con

trol

201

9/20

20

Local / global minima: convexity

Finding a global minima or maxima is in general hard.

Opt

imal

Con

trol

201

9/20

20

Convexity of functions and sets

Definition (convex function): A function is called convex, if

Definition (convex set): A set is called convex, if for all

Opt

imal

Con

trol

201

9/20

20

Local / global minima: convexity

Theorem (convex problem):If is convex and if the feasible set is convex , then the optimization problem is convex and any local minimizer of is a global one

The feasible set is convex if:• Inequality constraints are convex• Equality constraints are affine

Opt

imal

Con

trol

201

9/20

20

• The intersection of an arbitrary number of convex sets is a convex set:

• The empty set is convex because it satisfies the definition of convexity.• The sub-level sets of a convex function are convex

• If are convex functions, then is a convex function for all

• A quadratic function is convex if and only if is positive semi definite.

• A quadratic function is strictly convex if and only if is positive definite.

Operations preserving convexity

Details, see e.g.Boyd, S. P. & Vandenberghe, L. Convex Optimization, University Press, Cambridge, 2004

Opt

imal

Con

trol

201

9/20

20

Example: constrained LQR

constrained LQR (linear quadratic regulator)

• constraints are affine• cost function convex iff

►usually convex problem

Opt

imal

Con

trol

201

9/20

20

We consider the problem

is twice continuously differentiable• is the gradient of • is the Hessian of

Unconstrained (multidimensional) static optimization

Theorem (necessary conditions for a local minimum)If is a local minimizer of then:• First order condition

►If is convex, this condition is also sufficient for a (global) minimum• Second order condition

Theorem (sufficient condition for a local minimum)If the following conditions are satisfied:

Then is a local minimizer of ( is locally convex around )

Opt

imal

Con

trol

201

9/20

20

Example: unconstrained ball and spring

Note: is convex.

Opt

imal

Con

trol

201

9/20

20

For difficult problems, finding an analytic solution is not possibleGeneral idea: find a sequence , which converges to the optimal vector

Iterative/numerical solution methods

Most methods are based on the following algorithm:• is a search direction and is the step length

1. Find a descent direction (the cost is decreasing)2. Pick step length (typically: line search)3. Set

Direct search: Search by evaluating Indirect search: Search by using derivative information

Opt

imal

Con

trol

201

9/20

20

Steepest descent approach/ gradient method

Based on linear approximation:Choose steepest descent direction:

Linear convergence:

Remarks:• Simple to calculate• Gradient required• Only linear convergence

Opt

imal

Con

trol

201

9/20

20

Quadratic convergence:

Newton-Raphson-method

Based on quadratic approximation:

Minimize with respect to

Remarks: • Gradient and Hessian required • Need to calculate the inverse• Much faster

Opt

imal

Con

trol

201

9/20

20

Constrained static optimization: Lagrangian

Problem setup:

Definition (Generalized Lagrangian)

Opt

imal

Con

trol

201

9/20

20

Duality

Primal Problem Dual Problem

Dual variables: Dual function:

Properties of the dual problem:• The optimal value of the dual problem is always a lower bound of the optimal value of the primal problem• The dual problem is always concave, even when the primal is not convex

Opt

imal

Con

trol

201

9/20

20

Duality gap

The difference between the optimal value of the primal problem and the optimal value of the dual problem is called duality gap:

For a convex problem the duality gap is 0In that case it is said that strong duality holds.• (Constraint qualification should also hold)

Duality gap

Opt

imal

Con

trol

201

9/20

20

Remark• It is often assumed that the constraints satisfy certain regularity conditions

at the optimum. These conditions are called constraint qualification (CQ)• The most common CQ condition is the linear independence constraint

qualification (LICQ)• The constraint gradients are linearly independent

Equality constrained static optimization

problem setup:

Does not satisfy LICQ

Opt

imal

Con

trol

201

9/20

20

Necessary conditions of optimality

Definition (Lagrangian):

Theorem (necessary conditions using the Lagrangian)If is a local minimum of subject to and the gradientsare linearly independent, then there exists a Lagrange multiplier vector s.t.• first order condition

• second order condition

Remark:If: , then the first and second order conditions are also sufficient

Opt

imal

Con

trol

201

9/20

20

Example: ball and springCooking recipe:

1. Build Lagrangian2. Calculate3. Solve for4. If necessary, check second

order conditions

Opt

imal

Con

trol

201

9/20

20

Inequality constrained problems

Problem setup:

Definition (Generalized Lagrangian)

Opt

imal

Con

trol

201

9/20

20

First order necessary conditions: KKT conditionsTheorem (Karush Kuhn Tucker conditions)Let be a regular point (i.e. and are linearly

independent respectively) and a local minimum. Then there exists and s.t.

The conditions are also sufficient for convex problems with strong duality

Opt

imal

Con

trol

201

9/20

20

Example: ball and spring

KKT conditions yield the equilibrium of forces

two inequalities

Opt

imal

Con

trol

201

9/20

20

Lagrange multiplier/shadow price

What happens if one relaxes a constraint?

more general way:

Therefore, Lagrange multipliers are known as shadow prices.

Opt

imal

Con

trol

201

9/20

20

Iterative solution methods for constrained optimization

Constrained optimization problem• Transform the problem directly to an unconstrained problem

► Solve unconstrained problem• Penalty method• Barrier method• Method of augmented Lagrangian

• Apply approximation methods to KKT conditions• SQP method (Newton‘s method applied to KKT conditions) .

Opt

imal

Con

trol

201

9/20

20

Idea: Approximate constrained problem

with an additional quadratic penalizing term

Quadratic penalty method

Example:

yields

Opt

imal

Con

trol

201

9/20

20

Quadratic penalty method (continued)

Algorithm:1. Choose an initial penalty factor (small) and an initial guess for 2. Solve

3. Increase and use as initial guess for and go back to 2.

Remarks:• Use algorithms to solve unconstrained problems• For equality constrained problems is often as differentiable as • In the case of inequality constaints the Hessian is not differentiable• Ill conditioned for big • For finite the method may yield infeasible solutions

Opt

imal

Con

trol

201

9/20

20

Logarithmic barrierIdea: approximate constrained problem using an extended cost function

Remarks:• Intermediate minimizers are feasible• The extended cost is at least as

smooth as the constraint function• A feasible inital guess is required• Optima at borders are hard to find• Problem of ill conditioning with growing

iterations steps • Add quadratic penalty for equality

constraints

Algorithm: solve for decreasing values of

Opt

imal

Con

trol

201

9/20

20

Idea: apply Newton‘s method to the KKT conditions of

the KKT conditions are

use Newton‘s method to find a solution

SQP methods (sequential quadratic programming)

First two terms of Taylor series of the KKT conditions around

solve for in each iteration step

Opt

imal

Con

trol

201

9/20

20

SQP methods (continued)

Alternative point of viewsolve at all iterates a quadratic approximation of the original problem with linearized equality constraints

Possible to extended to inequality constraints.

Remarks:• If inequality constraints, need to select active and inactive constraints• additional line search step is possible • need for efficient QP solvers• huge storage demand for• SQP is widely used for nonlinear problems

Opt

imal

Con

trol

201

9/20

20

Summary

• Field of static optimization well developed• Convexity plays key role• Lagrange multipliers to deal with constraints• Many good numerical/iterative solution approaches

• Transform to unconstrained problem• Penalty methods• Barrier methods

• SQP (sequential quadratic programming)• Finding a global maxima/minima challenging