optimization problems 虞台文 大同大學資工所 智慧型多媒體研究室. content...
TRANSCRIPT
Optimization Problems
虞台文大同大學資工所智慧型多媒體研究室
ContentIntroductionDefinitionsLocal and Global OptimaConvex Sets and FunctionsConvex Programming
Problems
Optimization Problems
Introduction
大同大學資工所智慧型多媒體研究室
General Nonlinear Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
Local Minima vs. Global Minima
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
local minimum
global minimum
Convex Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
f (x)
gi (x)
hj (x)
convex
concave
linear
Local optimality Global optimality
Linear Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
f (x)
gi (x)
hj (x)
linear
linear
linear
Local optimality Global optimality
a special case of convex programming problems
Linear Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
f (x)
gi (x)
hj (x)
linear
linear
linear
Local optimality Global optimality
Integer Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx Z
objective function
constraints
f (x)
gi (x)
hj (x)
linear
linear
linear
The Hierarchy of Optimization Problems
NonlinearPrograms
ConvexPrograms
LinearPrograms
(Polynomial) IntegerPrograms(NP-Hard)
Flowand
Matching
Optimization Problems
General Nonlinear Programming Problems
Convex Programming Problems
Linear Programming Problems
Integer Linear Programming Problems
Optimization Techniques
General Nonlinear Programming Problems
Convex Programming Problems
Linear Programming Problems
Integer Linear Programming Problems
ContinuousVariables
DiscreteVariables
ContinuousOptimization
CombinatorialOptimization
Optimization Problems
Definitions
大同大學資工所智慧型多媒體研究室
Optimization Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
( )f xminimize
Optimization Problems
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
Define the set of feasible points
F
Minimize cost c: FR1
Definition:Instance of an Optimization Problem
(F, c) F: the domain of feasible points
c: F R1 cost function
Goal: To find f F such that
c( f ) c(g) for all gF.
A global optimum
Definition:Optimization Problem
A set of instances of an optimization problem, e.g.– Traveling Salesman Problem (TSP)– Minimal Spanning Tree (MST)– Shortest Path (SP)– Linear Programming (LP)
Traveling Salesman Problem (TSP)
Traveling Salesman Problem (TSP)
Instance of the TSP – Given n cities and an n n distance matrix [dij], t
he problem is to find a Hamiltonian cycle with minimal total length.
on F n all cyclic permutations objects
( )1
n
j jj
c d
1 2 3 4 5 6 7 8
2 5 3 6 1 8 4 7
e.g.,
Minimal Spanning Tree (MST)
Minimal Spanning Tree (MST)
Instance of the MST – Given an integer n > 0 and an n n symmetric distance m
atrix [dij], the problem is to find a spanning tree on n vertices that has minimum total length of its edge.
( , ) {1,2, , }VF E V n all spanning trees with
( , )
: ( , ) iji j E
c V E d
Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
1
2
n
c
cc
c
11 12 1
21 22 2
1 2
n
n
m m mn
a a a
a a aA
a a a
1
2
m
b
bb
b
1
2
n
x
xx
x
minimize
Subject to
c x
Ax b0x
Linear Programming (LP)
, , 0nx x R AF x b x
:c x c x
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize
Subject to
c x
Ax b0x
Example:Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
1 2 34 2 3x x x
1 2 3
1 2 3
2
, , 0
x x x
x x x
4 2 3c
1 1 1A 2b
minimize
Subject to
Example:Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
1 2 34 2 3x x x
1 2 3
1 2 3
2
, , 0
x x x
x x x
minimize
Subject to
x1
x2
x3
v1
v2
v3
c(v1) = 8
c(v2) = 4
c(v3) = 6
The optimum
The optimal point is at one of the vertices.
Example:Minimal Spanning Tree (3 Nodes)
1 2 34 2 3x x x
1 2 3 2x x x
minimize
Subject to
c1=4
c3=3
c2=2
1 2 3, , {0,1}x x x
x1{0, 1}
x2{0, 1}
x3{0, 1}
Integer Programming
x1
x2
x3
Example:Minimal Spanning Tree (3 Nodes)
1 2 34 2 3x x x
1 2 3 2x x x
minimize
Subject to
c1=4
c3=3
c2=2
x1{0, 1}
x2{0, 1}
x3{0, 1}
Linear Programming
x1
x2
x3
1 2 3, , 0x x x 1 2 3, , 1x x x
Some integer programs can be transformed into linear programs.
Optimization Problems
Local and Global Optima
大同大學資工所智慧型多媒體研究室
Neighborhoods
Given an optimization problem with instance
(F, c),
a neighborhood is a mapping
defined for each instance.
: 2FN F
For combinatorial optimization, the choice of N is critical.
TSP (2-Change)
f F gN2(f )
2 ( ) N f g g F g and can be obtained as above
TSP (k-Change)
( )
.k
g F gN f g
k f
and can be obtained
by changing edges of
MST
f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.
( ) N f g g F g and can be obtained as above
LP
minimize
Subject to
c x
Ax b0x
( ) , 0, N x y Ay b y y x and
Local Optima
Given(F, c)
N
an instance of an optimization problem
neighborhood
f F is called locally optimum with respect to N (or simply
locally optimum whenever N is understood by context) if
c(f ) c(g) for all gN(f ).
0 1 F
c
small
Local Optima
F = [0, 1] R1
( ) , 0, N f x x F y x f and
C
B
A Local minimum
Local minimum
Global minimum
Decent Algorithm
f = initial feasible solution
While Improve(f ) do
f = any element in Improve(f )
return f
Improve( ) ( ) ( ) ( )f s s N f c s c f and
Decent algorithm is usually stuck at a
local minimum unless the neighborhood N
is exact.
Exactness of Neighborhood
Neighborhood N is said to be exact if it makes
Local minimum Global Minimum
Exactness of Neighborhood
0 1 F
c
F = [0, 1] R1
( ) , 0, N f x x F y x f and
C
B
A Local minimum
Local minimum
Global minimum
N is exact if 1.
TSP
N2: not exact
Nn: exact
f F gN2(f )
MST N is exact
f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.
( ) N f g g F g and can be obtained as above
Optimization Problems
Convex Sets and Functions
大同大學資工所智慧型多媒體研究室
Convex Combination
x, y Rn
0 1 z = x +(1)y
A convex combination of x, y.
A strict convex combination of x, y if 0, 1.
Convex Sets
S Rn
z = x +(1)y
is convex if it contains all convex combinations of pairs x, y S.
convex nonconvex
0 1
Convex Sets
S Rn
z = x +(1)y
is convex if it contains all convex combinations of pairs x, y S.
n = 1
S is convex iff S is an interval.
0 1
Convex Sets
Fact: The intersection of any number of convex sets is convex.
c
Convex Functions
x yx +(1)y
c(x)
c(y)c(x) + (1)c(y)
c(x +(1)y)
S Rn a convex set
c:S R a convex function if
c(x +(1)y) c(x) + (1)c(y), 0 1
Every linear function is convex.
LemmaS
c(x)
t
a convex set
a convex function on S
a real number
( ) ,tS c x x Stx
is convex.
Pf) Let x, y St x +(1)y S
c(x +(1)y) c(x) + (1)c(y)
t + (1)t
= t
x +(1)y St
Level Contours
c = 1
c = 2
c = 3
c = 4
c = 5
Concave Functions
S Rn a convex set
c:S R a concave function if
c is a convex
Every linear function is concave as well as convex.
Optimization Problems
Convex Programming Problems
大同大學資工所智慧型多媒體研究室
Theorem
(F, c) an instance of optimization problem
a convex set
a convex function
Define ( )N x y y F x y and
( )N x is exact for every > 0.
• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point.
Theorem
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
Pf)
xF
( )N x
yNext, we now want to show that c(y) c(x).
• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point. <<1 such that• Since c is convex, we have
• Therefore,
Theorem
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
Pf)
xF
( )N x
yz
(1 ) ( ).x y xz z N and
( ) ( (1 ) )c c x yz
( ) (1 ) ( )c x c y
( ) ( )( )
1
zc c xc y
( ) ( )
1
c x c x
( )c x
( ) ( )zc c x
Convex Programming Problems
(F, c)
Defined by ( ) 0, 1, ,ig x i m
: nig R R
Convex function
an instance of optimization problem
Important property:
Local minimum Global Minimum
Concave functions
Convexity of Feasible Set
(F, c)
Defined by ( ) 0, 1, ,ig x i m
: nig R R
Convex function
an instance of optimization problem
Important property:
Local minimum Global Minimum
Concave functions
( ) : ig x convex
( ) : ig x concave
( ) 0 : ig x convex
( ) 0 : ig x convex
: iF convex
1
: m
ii
F F
convex
Convex Programming Problems
(F, c)
Defined by ( ) 0, 1, ,ig x i m
: nig R R
Convex function
an instance of optimization problem
Important property:
Local minimum Global Minimum
Concave functionsConvex
Theorem
In a convex programming problem, every
point locally optimal with respect to the
Euclidean distance neighborhood N is also
global optimal.