global optimization with matlab products(draft)_mathworks

8/11/2019 Global Optimization With MATLAB Products(Draft)_MathWorks

1/67

1

2011 The MathWorks, Inc.

Global Optimization with MATLAB

Products

Account Manager

Application Engineer


2/67

2

Agenda

Introduction to Global Optimization

Survey of Solvers with Examples

MultiStart

Global Search

Pattern Search

Simulated Annealing

Genetic Algorithm / Multiobjective

Genetic Algorithm

Additional Resources

Question & Answer

-2

0

2

-3-2

-10

123

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Local minima

Global minima


3/67

3

OptimizationFinding answers to problems automatical ly

Objectives

Achieved?

NO

Optimal

Design

YESModel or

Prototype

Modify Design

Parameters

Initial

Design

Parameters

OPTIMIZATION PROCESS

Finding better (optimal) designs

Faster design evaluations

Useful for trade-off analysis (N dimensions) Non-intuitive designs may be found

Optimization benefits include:Design process can be performed:

Antenna Design Using Genetic Algorithmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htm

Manually

(trial-and-error or iteratively)

Automatically

(using optimizationtechniques)
http://ic.arc.nasa.gov/projects/esg/research/antenna.htmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htm


4/67

40 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

t

c

c=b1

e-b

4t+b

2e

-b5

t+b

3e

-b6

t

Example Global Optimization Problems

Why does fminconhave a hard time finding the

function minimum?

Why didnt nonlinear regression find a good fit?

Why didnt fminuncfind the maximum volumetric

efficiency?

0 5 10

-10

-5

0

5

10

x

Starting at 10

0 5 10

-10

-5

0

5

10

x

Starting at 8

0 5 10

-10

-5

0

5

10

x

Starting at 6

xsin(x)+xcos(2x)

0 5 10

-10

-5

0

5

10

x

Starting at 3

0 5 10

-10

-5

0

5

10

x

Starting at 1

0 5 10

-10

-5

0

5

10

x

Starting at 0

xsin(x)+xcos(2x)

Revolutions Per Minute, RPM

ManifoldPressureRatio

Peak VE Value = 0.96144

Start

End

1000 2000 3000 4000 5000 60000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


5/67

5

Global Optimization

Goal:

Want to find the lowest/largest value of

the nonlinear function that has many local

minima/maxima

Problem:

Traditional solvers often return one of the

local minima (not the global)

Solution:

A solver that locates globally optimal

solutionsRastrigins Function


6/67

6

Global Optimization ToolboxFor problems that

contain multiple maxima/minima or are non-smooth

Optimization T

oolbox

Global Optimization

Toolbox

Faster/fewer function eval

uations

Larger problems (higher d

imensions)

Finds local minima/maxim

a

Finds global minima/maxi

ma (most of the time)

Better on

non-smooth

stochastic

discontinuous

undefined gradients

Custom data types

(in GA and SA solvers)


7/677

MULTISTART


8/678

What is MultiStart?

Run a local solver from

each set of start points

Option to filter starting

points based feasibility

Supports parallelcomputing


9/6711

GLOBAL SEARCH


10/6712

What is GlobalSearch?

Multistart heuristic algorithm

Calls fminconfrom multiple

start points to try and find a

global minimum

Filters/removes non-promising

start points


11/6713

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewSchematic Problem

Peaks function

Three minima

Green, z = -0.065

Red, z= -3.05Blue, z = -6.55

x

y


12/6714

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 0Run from speci f ied x0

x

y


13/6715

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 1Generate stage 1 start po ints via Scatter Search

3

6

0

00

4

0

-2

x

y


14/6716

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 1Find stage 1 start poin t with lowest penalty value

3

6

0

00

4

0

-2

x

y


15/6717

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 1Run from best stage 1 poin t

x

y


16/6718

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Generate stage 2 start poin ts using Scatter Search

x

y


17/67

19

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Analyse each stage 2 po int in turn .

x

y


18/67

20

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Dont run points that are in basins of existing

min imum

x

y


19/67

21

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Analyse each stage 2 po int in turn .

x

y


20/67

22

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Dont run points whose penalty value exceeds

threshold

6

Current penalty

threshold value : 4

x

y


21/67

23

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Analyse each stage 2 po int in tu rn

x

y


22/67

24

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Run s tart point i f i t sat isf ies distance & meri t cr i ter ia

Current penalty

threshold value : 4

-3

x

y


23/67

25

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch OverviewStage 2Expand basin of attract ion i f m inimum already found

Current penalty threshold value : 2

-0.1

x

y Basins can overlap


24/67

28

SIMULATED ANNEALING


25/67

29

What is Simulated Annealing?

A probabilistic metaheuristic approach based upon the

physical process of annealing in metallurgy.

Controlled cooling of a metal allows atoms to realign from arandom higher energy state to an ordered crystalline

(globally) lower energy state

By analogy, simulated annealing replaces a current solution

by randomly choosing a nearby solution

A nearby solution is determined by the solution

temperature


26/67


27/67

31

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration 1Randomly generate a new point according to pro babi l i ty dis t r ibut ion and

current temperature

3

x

y 0.9

Possible New Points:

Standard Normal N(0,1) * Temperature

Temperature = 1


28/67

32

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration 1If low er, accept the po int , i f high er, accept b ased upon acceptance

probabi l i ty

3

x

y 0.9

Temperature = 1

11.0

1

1/)9.03(

Taccept

e

P


29/67

33

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


current temperature

3

x

y 0.9

Temperature = 1

0.3


30/67

34

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


current temp erature, accept new po int i f lower value

3

x

y 0.9

Temperature = 1

0.3


31/67

35

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration 2Lower temperature accordin g to temp erature schedule

3

x

y 0.9

Temperature = 1

0.3


32/67

36

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration 2Lower temperature acco rding to temperature schedule and generate new

poin t

3

x

y 0.9

Temperature = 0.75

0.3

-1.2


33/67

37

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration N-1After several i terat ions , the search radiu s becomes small and w e narrow

in on a local solut ion

3

x

y 0.9

Temperature = 0.1

0.3

-1.2

-3


34/67

38

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration NReset temperature and s tart the pro cess again (reannel ing )

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2


35/67


36/67


37/67

41

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration N+1Reduce temperature and continue

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3


38/67

42

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration N+1Reduce temperature and continue

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3


39/67

43

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing OverviewIteration Reduce temperature and continue

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

-6.5


40/67

47

PATTERN SEARCH(DIRECT SEARCH)


41/67

48

What is a Pattern Search?

An approach that uses apatternof search directions around

the existing points, the mesh

Polls the mesh for a better solution and moves to that point

Expands/contracts the mesh around the current point when a

solution is not found

Does not rely on gradient information


42/67

49

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search OverviewIteration 1Run from speci f ied x0

x

y

3


43/67

50

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search OverviewIteration 1Apply pattern vector, pol l new po ints for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)


44/67

51

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search OverviewIteration 2Increase mesh size and repeat

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Complete Poll


45/67

52

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search OverviewIteration 3Mesh expansion: increase mesh size and repeat

x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8


46/67

53

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search OverviewIteration 4Refine mesh: decrease mesh size and repeat

x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8


47/67

54

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search OverviewIteration NContinue expansion/contraction until convergence

x

y

31.6

0.4

4.6

2.8

-4

0.3-2.8

-6.5


48/67

57

GENETIC ALGORITHM


49/67

58

What is a Genetic Algorithm?

Genetic Algorithms use concepts from evolutionary biologyto

find exact or approximate solutions to optimization problems

Start with an initial generation of candidate solutions that are

tested against the objective function

Subsequent generations evolve from the 1st

through selection, crossoverand mutation

The individualthat best minimizes the given

objective is returned as the ideal solution


50/67

59

How Evolution WorksBinary Case

Selection

Retainthe best performing bit strings from one generation to the

next. Favor these for reproduction

parent1 = [ 1 0 1 0 0 1 1 0 0 0]

parent2 = [ 1 0 0 1 0 0 1 0 1 0]

Crossover

parent1 = [ 1 0 1 0 0 1 1 0 0 0]

parent2 = [ 1 0 0 1 0 0 1 0 1 0]

child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

parent = [ 1 0 1 0 0 1 1 0 0 0 ]

child = [ 0 1 0 1 0 1 0 0 0 1]


51/67

60

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic AlgorithmIteration 1Evaluate ini t ial populat ion

x

y


52/67

61

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic AlgorithmIteration 1Select a few good solut ion s for reproduct ion

x

y


53/67

62

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic AlgorithmIteration 2Generate new popu lat ion and evaluate

x

y


54/67

63

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y


55/67

64

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic AlgorithmIteration 3Generate new popu lat ion and evaluate

x

y


56/67

65

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y


57/67

66

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic AlgorithmIteration NCont inue process unt i l stopp ing c r i ter ia are met

x

y

Solution found


58/67

69

Comparison of Solver (Default) Performance

x

y

rf([x,y])

-6 -4 -2 0 2 4 6

-6

-4

-2

0

2

4

6

FuncValue

Min

StartPt

x

y

rf([x,y])

-60 -40 -20 0 20 40 60

-60

-40

-20

0

20

40

60

FuncValue

Min

StartPt

x

y

rf([x,y])

-40 -20 0 20 40 60

-40

-20

0

20

40

60 FuncValue

Min

StartPt


59/67


60/67

71

Additional Resources

Upcoming Webinars

Speeding Up Optimization with Parallel Computing

(August 2010)

On-demand Webinars

Genetic Algorithm in Financial Applications

Tips & Tricks: Getting Started with Optimization

Introduction to Optimization


61/67

72

Contact Information

North America

Phone: 508-647-7000

E-mail: [email protected]

Outside North America

Contact your local MathWorks office or reseller:

www.mathworks.com/contact


62/67

73

Questions?


63/67

74

MATLAB Provides the Foundation for

Optimization

The leading environment fortechnical computing

Customizable Numeric computation

Data analysis and visualization

The de factoindustry-standard,high-level programming languagefor algorithm development

Toolboxes for statistics, optimization,symbolic math, signal and imageprocessing, and other areas

Foundation of theMathWorks product family


64/67

75

Optimization ToolboxSolve standard and large-scale optimization problems

Graphical user interface and commandline functions for: Linear and nonlinear programming

Quadratic programming

Nonlinear least squares and nonlinearequations

Multi-objective optimization

Binary integer programming

Additional Capabilities:

Parallel computing support in selectedsolvers

Customizable algorithm options Choose between standard and large-

scale algorithms

Output diagnostics


65/67

76

Global Optimization ToolboxSolve multiple maxima, multiple minima, and nonsmooth optimization problems

Graphical user interface and commandline functions for: Global Search solver

Multistart solver

Genetic algorithm solver

Single objective Multiobjective with Pareto front

Direct search solver

Simulated annealing solver

Useful for problems not easily addressedwith Optimization Toolbox: Discontinuous Highly nonlinear

Stochastic

Discrete or custom data types

Undefined derivatives

Multiple maxima/minima


66/67


67/67

MATLAB Optimization Products and Example

Applications

MATLAB Statistics Toolbox

Curve Fitting Toolbox

Optimization Toolbox Genetic Algorithm and

Direct Search Toolbox

Solving Equations

f(x) = 0

Root finding

Systems of equations

Real roots finding (1D):

Linear Systems:

F(x) = 0

Ax-b = 0 (i.e. Ax=b)

>> x = A \ b

Real roots finding (N-D):

Nonlinear Systems

F(X) = 0

Noisy, discontinuous root

finding (N-D)

Noisy, discontinuous systems:

F(X) = 0

Curve/Modeling

Fitting

curve fitting

parameter estimation

(model fitting)

Basic (linear) curve fitting Advanced (nonlinear) curve

fitting

Model Fitting (least squares)

Constrained curve fitting

Parameter Estimation

Noisy, Discontinuous

parameter estimation

Trade-Off Studies

Maximization

Minimization

Goal seeking

Multiobjective

Unconstrained nonlinear

minimization

Constrained nonlinear

minimization

Noisy, Discontinuous, ill-

defined mimization

-6 - 4 - 2 0 2 4 6-100

-50

0

50

100

x

x3-2 x-5

0 0.5 1 1.5 2 2.50.5

0.6

0.7

0.8

0.9

global optimization with matlab products(draft)_mathworks

Documents