convex sets - university of michiganmepelman/teaching/ioe611/handouts/sets.pdfane set: contains the...
TRANSCRIPT
Convex sets
I a�ne and convex sets
I some important examples
I operations that preserve convexity
I separating and supporting hyperplanes
I generalized inequalities
I dual cones and generalized inequalities
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–1
A�ne set
a�ne combination of x1 and x2: any point x of the form
x = ✓x1 + (1� ✓)x2 where ✓ 2 R
Affine set
line through x1, x2: all points
x = θx1 + (1 − θ)x2 (θ ∈ R)PSfrag replacements
x1
x2
θ = 1.2θ = 1
θ = 0.6
θ = 0θ = −0.2
affine set: contains the line through any two distinct points in the set
example: solution set of linear equations {x | Ax = b}
(conversely, every affine set can be expressed as solution set of system oflinear equations)
Convex sets 2–2
line through x1, x2: all a�ne combinations of x1 and x2
a�ne set: contains the line through any two distinct points in theset (i.e., a set that’s closed under a�ne combinations)
example: solution set of linear equations {x | Ax = b}(conversely, every a�ne set can be expressed as solution set ofsystem of linear equations)
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–2
Convex setconvex combination of x1 and x2: any point x of the form
x = ✓x1 + (1� ✓)x2 where 0 ✓ 1
line segment between x1 and x2: all points
x = ✓x1 + (1� ✓)x2
with 0 ✓ 1convex set: contains line segment between any two points in theset
x1, x2 2 C , 0 ✓ 1 =) ✓x1 + (1� ✓)x2 2 C
examples (one convex, two nonconvex sets)
Convex set
line segment between x1 and x2: all points
x = θx1 + (1 − θ)x2
with 0 ≤ θ ≤ 1
convex set: contains line segment between any two points in the set
x1, x2 ∈ C, 0 ≤ θ ≤ 1 =⇒ θx1 + (1 − θ)x2 ∈ C
examples (one convex, two nonconvex sets)
Convex sets 2–3
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–3
Convex combination and convex hull
convex combination of x1,. . . , xk
: any point x of the form
x = ✓1x1 + ✓2x2 + · · ·+ ✓k
x
k
with ✓1 + · · ·+ ✓k
= 1, ✓i
� 0
convex hull conv S : set of all convex combinations of points in S
Convex combination and convex hull
convex combination of x1,. . . , xk: any point x of the form
x = θ1x1 + θ2x2 + · · · + θkxk
with θ1 + · · · + θk = 1, θi ≥ 0
convex hull conv S: set of all convex combinations of points in S
Convex sets 2–4IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–4
Convex conecone: a set C such that for any x 2 C and ✓ � 0, ✓x 2 C
conic (nonnegative) combination of x1 and x2: any point of theform
x = ✓1x1 + ✓2x2 where ✓1, ✓2 � 0
Convex cone
conic (nonnegative) combination of x1 and x2: any point of the form
x = θ1x1 + θ2x2
with θ1 ≥ 0, θ2 ≥ 0
PSfrag replacements
0
x1
x2
convex cone: set that contains all conic combinations of points in the set
Convex sets 2–5
convex cone: a cone that is convex, i.e., set that contains allconic combinations
{✓1x1 + ✓2x2 + · · ·+ ✓k
x
k
| ✓i
� 0}
of points in the setIOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–5
Examples of a�ne/convex sets and convex cones
Before we get to interesting examples, some trivial ones:
I An empty set ;, a singleton {x0} ⇢ R
n, the whole space R
n
are a�ne and convex sets in R
n
I Any line is a�ne. If it passes through 0, it is also a convexcone
I A line segment is convex, but not a�ne (unless a singleton)
I A ray (set {x0 + ✓v | ✓ � 0}) is convex but not a�ne; ifx0 = 0, it is a convex cone
I A subspace (a subset of Rn closed under sums and scalarmultiplications) is a�ne, convex, and is a convex cone
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–6
Hyperplanes and halfspaceshyperplane: set of the form {x | aT x = b} (a 6= 0)
Hyperplanes and halfspaces
hyperplane: set of the form {x | aTx = b} (a ̸= 0)
PSfrag replacements a
x
aTx = b
x0
halfspace: set of the form {x | aTx ≤ b} (a ̸= 0)
PSfrag replacements
a
aTx ≥ b
aTx ≤ b
x0
• a is the normal vector
• hyperplanes are affine and convex; halfspaces are convex
Convex sets 2–6
halfspace: set of the form {x | aT x b} (a 6= 0)
Hyperplanes and halfspaces
hyperplane: set of the form {x | aTx = b} (a ̸= 0)
PSfrag replacements a
x
aTx = b
x0
halfspace: set of the form {x | aTx ≤ b} (a ̸= 0)
PSfrag replacements
a
aTx ≥ b
aTx ≤ b
x0
• a is the normal vector
• hyperplanes are affine and convex; halfspaces are convex
Convex sets 2–6
Ia is the normal vector of the hyperplane/halfspace
I hyperplanes are a�ne and convex; halfspaces are convex
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–7
Polyhedrasolution set of finitely many linear inequalities and equalities
Ax � b, Cx = d
(A 2 R
m⇥n, C 2 R
p⇥n, � is componentwise inequality)
Polyhedra
solution set of finitely many linear inequalities and equalities
Ax ≼ b, Cx = d
(A ∈ Rm×n, C ∈ Rp×n, ≼ is componentwise inequality)
PSfrag replacements
a1 a2
a3
a4
a5
P
polyhedron is intersection of finite number of halfspaces and hyperplanes
Convex sets 2–9
polyhedron is intersection of finite number of halfspaces andhyperplanes
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–8
Positive semidefinite cone
notation:
IS
n is set of symmetric n ⇥ n matrices
IS
n
+ = {X 2 S
n | X ⌫ 0}: positive semidefinite n ⇥ n matrices
X 2 S
n
+ () z
T
Xz � 0 for all z
IS
n
++ = {X 2 S
n | X � 0}: positive definite n ⇥ n matrices
S
n
+ is a convex cone
example:
x y
y z
�2 S
2+
Positive semidefinite cone
notation:
• Sn is set of symmetric n × n matrices
• Sn+ = {X ∈ Sn | X ≽ 0}: positive semidefinite n × n matrices
X ∈ Sn+ ⇐⇒ zTXz ≥ 0 for all z
Sn+ is a convex cone
• Sn++ = {X ∈ Sn | X ≻ 0}: positive definite n × n matrices
example:
!x yy z
"∈ S2
+
PSfrag replacements
xyz
0
0.5
1
−1
0
10
0.5
1
Convex sets 2–10IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–9
Euclidean balls and ellipsoids(Euclidean) ball with center x
c
and radius r :
B(xc
, r) = {x | kx � x
c
k2 r} = {xc
+ ru | kuk2 1}
ellipsoid: set of the form
{x | (x � x
c
)TP�1(x � x
c
) 1}
with P 2 S
n
++ (i.e., P symmetric positive definite)
Euclidean balls and ellipsoids
(Euclidean) ball with center xc and radius r:
B(xc, r) = {x | ∥x − xc∥2 ≤ r} = {xc + ru | ∥u∥2 ≤ 1}
ellipsoid: set of the form
{x | (x − xc)TP−1(x − xc) ≤ 1}
with P ∈ Sn++ (i.e., P symmetric positive definite)
PSfrag replacementsxc
other representation: {xc + Au | ∥u∥2 ≤ 1} with A square and nonsingular
Convex sets 2–7
other representation: {xc
+ Au | kuk2 1} with A square andnonsingular
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–10
Norm balls and norm cones
norm: a function k · k that satisfies
I kxk � 0; kxk = 0 if and only if x = 0
I ktxk = |t| · kxk for t 2 R
I kx + yk kxk+ kyknotation: k · k is a general (unspecified) norm; k · ksymb is aparticular normnorm ball with center x
c
and radius r : {x | kx � x
c
k r}
norm cone:
{(x , t) | kxk t} ⇢ R
n+1
Euclidean norm cone is calledsecond-order cone
Norm balls and norm cones
norm: a function ∥ · ∥ that satisfies
• ∥x∥ ≥ 0; ∥x∥ = 0 if and only if x = 0
• ∥tx∥ = |t| ∥x∥ for t ∈ R
• ∥x + y∥ ≤ ∥x∥ + ∥y∥
notation: ∥ · ∥ is general (unspecified) norm; ∥ · ∥symb is particular norm
norm ball with center xc and radius r: {x | ∥x − xc∥ ≤ r}
norm cone: {(x, t) | ∥x∥ ≤ t}
Euclidean norm cone is called second-order cone
PSfrag replacements
x1x2
t
−10
1
−1
0
10
0.5
1
norm balls and cones are convex
Convex sets 2–8
norm balls and cones are convex
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–11
Operations that preserve convexity
practical methods for establishing convexity of a set C
1. apply definition
x1, x2 2 C , 0 ✓ 1 =) ✓x1 + (1� ✓)x2 2 C
2. show that C is obtained from simple convex sets(hyperplanes, halfspaces, norm balls, . . . ) by operations thatpreserve convexity
I Cartesian product (S1 ⇥ S2 = {(x1, x2) | x1 2 S1, x2 2 S2}; ifS1 and S2 are convex, so is S1 ⇥ S2)
I intersectionI a�ne functionsI perspective functionI linear-fractional functions
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–12
Intersectionthe intersection of (any number of) convex sets is convex
example:
S = {x 2 R
m | |px
(t)| 1 for |t| ⇡/3}
where p
x
(t) = x1 cos t + x2 cos 2t + · · ·+ x
m
cosmt
for m = 2:
Intersection
the intersection of (any number of) convex sets is convex
example:S = {x ∈ Rm | |p(t)| ≤ 1 for |t| ≤ π/3}
where p(t) = x1 cos t + x2 cos 2t + · · · + xm cosmt
for m = 2:
PSfrag replacements
0 π/3 2π/3 π
−1
0
1
t
p(t
)
PSfrag replacements
x1
x2 S
−2 −1 0 1 2−2
−1
0
1
2
Convex sets 2–12S
t
= {x 2 R
m | �1 (cos t, . . . , cosmt)T x 1}; S = \|t|⇡/3St .
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–13
A�ne functionsuppose f : Rn ! R
m is a�ne (f (x) = Ax + b with A 2 R
m⇥n,b 2 R
m)I the image of a convex set under f is convex
S ✓ R
n convex =) f (S) = {f (x) | x 2 S} ✓ R
m convex
Iexamples: scaling, translation, projection, sum of sets
I the inverse image f
�1(C ):
f
�1(C ) = {x 2 R
n | f (x) 2 C}
the inverse image of a convex set under f is convex
C ✓ R
m convex =) f
�1(C ) convex
Iexample: solution set of linear matrix inequality{x | x1A1 + · · ·+ x
m
A
m
� B} (with A
i
,B 2 S
p)I
example: hyperbolic cone {x | xTPx (cT x)2, cT x � 0}(with P 2 S
n
+)IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–14
Perspective and linear-fractional function
perspective function P : Rn+1 ! R
n:
P(x , t) = x/t, domP = {(x , t) | t > 0}
images and inverse images of convex sets under perspective areconvex
linear-fractional function f : Rn ! R
m:
f (x) =Ax + b
c
T
x + d
, dom f = {x | cT x + d > 0}
images and inverse images of convex sets under linear-fractionalfunctions are convex
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–15
example of a linear-fractional function
f (x) =1
x1 + x2 + 1x : R2 ! R
2
example of a linear-fractional function
f(x) =1
x1 + x2 + 1x
PSfrag replacements
x1
x2 C
−1 0 1−1
0
1
PSfrag replacements
x1
x2
f(C)
−1 0 1−1
0
1
Convex sets 2–15
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–16
Separating hyperplane theorem
if C and D are disjoint convex sets, then there exists a 6= 0, b suchthat
a
T
x b for x 2 C , a
T
x � b for x 2 D
Separating hyperplane theorem
if C and D are disjoint convex sets, then there exists a ̸= 0, b such that
aTx ≤ b for x ∈ C, aTx ≥ b for x ∈ D
PSfrag replacementsD
C
a
aTx ≥ b aTx ≤ b
the hyperplane {x | aTx = b} separates C and D
strict separation requires additional assumptions (e.g., C is closed, D is asingleton)
Convex sets 2–19
the hyperplane {x | aT x = b} separates C and D
strict separation (aT x < b for x 2 C , a
T
x > b for x 2 D)requires additional assumptions (e.g., C is closed, D is a singleton;see example 2.20)
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–17
Supporting hyperplane theorem
supporting hyperplane to set C at boundary point x0:
{x | aT x = a
T
x0}
where a 6= 0 and a
T
x a
T
x0 for all x 2 C
Supporting hyperplane theorem
supporting hyperplane to set C at boundary point x0:
{x | aTx = aTx0}
where a ̸= 0 and aTx ≤ aTx0 for all x ∈ C
PSfrag replacements C
a
x0
supporting hyperplane theorem: if C is convex, then there exists asupporting hyperplane at every boundary point of C
Convex sets 2–20
supporting hyperplane theorem: if C is convex, then thereexists a supporting hyperplane at every boundary point of C
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–18
A theorem of alternative for linear systems
Theorem Given A 2 R
m⇥n and b 2 R
m, exactly one of
(i) Ax < b and (ii) AT� = 0, b
T� 0, � � 0, � 6= 0
has a solution.Proof
I Easy to see (i) and (ii) can’t both have solutions.
I If (i) does not have a solution, C = {b � Ax | x 2 R
n} andD = {y 2 R
m
++} are convex disjoint sets and can be separated.
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–19
Generalized inequalities
a convex cone K ✓ R
n is a proper cone if
IK is closed (contains its boundary)
IK is solid (has nonempty interior)
IK is pointed (contains no line)
examples
I nonnegative orthantK = R
n
+ = {x 2 R
n | xi
� 0, i = 1, . . . , n}I positive semidefinite cone K = S
n
+
I nonnegative polynomials of degree n � 1 on [0, 1]:
K = {x 2 R
n | x1+x2t+x3t2+ · · ·+x
n
t
n�1 � 0 for t 2 [0, 1]}
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–20
Generalized inequalitiesa proper cone K can be used to define a generalized inequality:
x �K
y () y�x 2 K , x �K
y () y�x 2 intK
examples
I componentwise inequality (K = R
n
+)
x �R
n
+y () x
i
y
i
, i = 1, . . . , n
I matrix inequality (K = S
n
+)
X �S
n
+Y () Y � X positive semidefinite
these two types are so common that we drop the subscript in �K
properties: many properties of �K
are similar to on R, e.g.,
x �K
y , u �K
v =) x + u �K
y + v
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–21
Minimum and minimal elements�
K
is not in general a linear ordering: we can have x 6�K
y andy 6�
K
x
x 2 S is the minimum element of S with respect to �K
if
y 2 S =) x �K
y (equivalently, S ✓ x + K ).
x 2 S is a minimal element of S with respect to �K
if
y 2 S , y �K
x =) y = x (equivalently, (x�K )\S = {x}).
example (K = R
2+)
x1 is the minimum element of S1x2 is a minimal element of S2
Minimum and minimal elements
≼K is not in general a linear ordering : we can have x ̸≼K y and y ̸≼K x
x ∈ S is the minimum element of S with respect to ≼K if
y ∈ S =⇒ x ≼K y
x ∈ S is a minimal element of S with respect to ≼K if
y ∈ S, y ≼K x =⇒ y = x
example (K = R2+)
x1 is the minimum element of S1
x2 is a minimal element of S2
PSfrag replacements
x1
x2S1
S2
Convex sets 2–18IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–22
optimal production frontier
I di↵erent production methods use di↵erent amounts ofresources x 2 R
n
I production set P : resource vectors x for all possibleproduction methods
I e�cient (Pareto optimal) methods correspond to resourcevectors x that are minimal w.r.t. Rn
+
example (n = 2)x1, x2, x3 are e�cient; x4, x5 are not
optimal production frontier
• different production methods use different amounts of resources x ∈ Rn
• production set P : resource vectors x for all possible production methods
• efficient (Pareto optimal) methods correspond to resource vectors xthat are minimal w.r.t. Rn
+
example (n = 2)
x1, x2, x3 are efficient; x4, x5 are not
PSfrag replacements
x4x2
x1
x5
x3λ
P
labor
fuel
Convex sets 2–23IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–23
Dual cones and generalized inequalities
dual cone of a cone K :
K
⇤ = {y | hx , yi � 0 for all x 2 K}
examples
IK = R
n
+: K⇤ = R
n
+ (for x , y 2 R
n, hx , yi = y
T
x)
IK = S
n
+: K⇤ = S
n
+ (for X ,Y 2 S
n,hX ,Y i =
Pi ,j Xij
Y
ij
= tr(XY ))
IK = {(x , t) | kxk2 t}: K ⇤ = {(y , s) | kyk2 s}
IK = {(x , t) | kxk1 t}: K ⇤ = {(y , s) | kyk1 s}
first three examples are self-dual cones
dual cones of proper cones are proper, hence define generalizedinequalities:
y ⌫K
⇤ 0 () y
T
x � 0 for all x ⌫K
0
IOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–24
Minimum and minimal elements via dual inequalities
minimum element w.r.t. �K
x is minimum element of S i↵ forall � �
K
⇤ 0, x is the uniqueminimizer of �T
z over S
Minimum and minimal elements via dual inequalities
minimum element w.r.t. ≼K
x is minimum element of S iff for allλ ≻K∗ 0, x is the unique minimizerof λTz over S
PSfrag replacements x
S
minimal element w.r.t. ≽K
• if x minimizes λTz over S for some λ ≻K∗ 0, then x is minimal
PSfrag replacements
Sx1
x2
λ1
λ2
• if x is a minimal element of a convex set S, then there exists a nonzeroλ ≽K∗ 0 such that x minimizes λTz over S
Convex sets 2–22
minimal element w.r.t. �K
I if x minimizes �T
z over S for some � �K
⇤ 0, then x isminimal
Minimum and minimal elements via dual inequalities
minimum element w.r.t. ≼K
x is minimum element of S iff for allλ ≻K∗ 0, x is the unique minimizerof λTz over S
PSfrag replacements x
S
minimal element w.r.t. ≽K
• if x minimizes λTz over S for some λ ≻K∗ 0, then x is minimal
PSfrag replacements
Sx1
x2
λ1
λ2
• if x is a minimal element of a convex set S, then there exists a nonzeroλ ≽K∗ 0 such that x minimizes λTz over S
Convex sets 2–22
I if x is a minimal element of a convex set S , then there existsa nonzero � ⌫
K
⇤ 0 such that x minimizes �T
z over SIOE 611: Nonlinear Programming, Fall 2017 2. Convex sets Page 2–25