cse 203b: convexoptimization week 3 discusssession
TRANSCRIPT
Contents
โข Convex functions (Ref. Chap.3)โข Definitionโข First order conditionโข Second order conditionโข Epigraphโข Operationsthatpreserveconvexity
โข Conjugatefunctions
2
Definition of Convex Functionsโข A function ๐: ๐ $ โ ๐ is convex if dom๐ is a convex set and if for all๐ฅ, ๐ฆ โ dom๐ and 0 โค ๐ โค 1
๐ ๐๐ฅ + 1 โ ๐ ๐ฆ โค ๐๐ ๐ฅ + 1โ ๐ ๐(๐ฆ)โข Review the proof in class: necessary and sufficiency
โข Strict convexity:๐ ๐๐ฅ + 1โ ๐ ๐ฆ < ๐๐ ๐ฅ + 1 โ ๐ ๐ ๐ฆ , ๐ฅ โ ๐ฆ, 0 < ๐ < 1โข Concave functions:โ๐ is convex
3
sometimes calledJensenโs inequality
(๐๐ฅ + 1 โ ๐ ๐ฆ, ๐ฝ๐(๐) + ๐ โ๐ฝ ๐(๐))
Example: convexity of functions with definition
4
i. Necessary: suppose๐ is convex, then ๐ satisfies the basic inequality๐ ๐ฅ + ๐ ๐ฆ โ ๐ฅ โค ๐ ๐ฅ + ๐(๐ ๐ฆ โ ๐ ๐ฅ )
for0 โค ๐ โค 1. Integratingboth sides from0 to 1 on ๐
> ๐ ๐ฅ + ๐ ๐ฆ โ ๐ฅ ๐๐@
Aโค > ๐ ๐ฅ + ๐ ๐ ๐ฆ โ ๐ ๐ฅ ๐๐
@
A=๐ ๐ฅ + ๐(๐ฆ)
2ii. Sufficiency: suppose๐ is not convex, then there exists0 โค ๐D โค 1 and
๐ ๐ฅ + ๐D ๐ฆ โ ๐ฅ > ๐ ๐ฅ + ๐D(๐ ๐ฆ โ ๐ ๐ฅ )
there exist๐ผ,๐ฝ โ [0,1] so that in the interval [๐ผ, ๐ฝ] the above
inequality alwaysholds,which contradicts the given inequality.๐D
๐ผ๐ฝ
Restriction of a convex function to a lineโข A function ๐: ๐ $ โ ๐ is convex if and only if the function๐: ๐ โ ๐ ,
๐ ๐ก = ๐ ๐ฅ + ๐ก๐ฃ , dom๐ = ๐ก ๐ฅ + ๐ก๐ฃ โ dom๐}
is a convexon its domain for โ๐ฅ โ dom๐, ๐ฃ โ ๐ $.
โข The property can be useful to check the convexity of a function
Example: Prove ๐ ๐ = log det ๐ , dom๐ = ๐VV$ is concave.
5
Restriction of a convex function to a lineExample: Prove ๐ ๐ = log det ๐ , dom๐ = ๐VV$ is concave.
Consider an arbitrary line๐ = ๐ + ๐ก๐, where ๐ โ ๐VV$ ,๐ โ ๐$. Define ๐ ๐ก =๐ ๐ + ๐ก๐ and restrict ๐ to the interval values of ๐ก for ๐ + ๐ก๐ โป 0.We have
๐ ๐ก = logdet(๐ + ๐ก๐) = logdet(๐@Z ๐ผ + ๐ก๐\
@Z๐๐\
@Z ๐@/Z)
= ^log 1 + ๐ก๐_
$
_`@
+ logdet ๐
where ๐_ are the eigenvalues of ๐\ab๐๐\
ab. So we have
๐c ๐ก = ^๐_
1 + ๐ก๐_
$
_`@
, ๐cc ๐ก = โ^๐_Z
1 + ๐ก๐_ Z
$
_`@
โค 0
๐ ๐ is concave. For more practice, see Exercise 3.18.6
ร The property of eigenvaluesdet๐ด = โ ๐_$
_`@
First-order Conditionโข Suppose ๐ is differentiable (๐๐๐๐ is open and ๐ป๐ exists at โ๐ฅ โ ๐๐๐๐),then ๐ is convex iff dom๐ is convex and for all ๐ฅ, ๐ฆ โ ๐๐๐๐
๐ ๐ฆ โฅ ๐ ๐ฅ + ๐ป๐ ๐ฅ j(๐ฆโ ๐ฅ)โข Review the proof in class: necessary and sufficiency
โข Strict convexity:๐ ๐ฆ > ๐ ๐ฅ + ๐ป๐ ๐ฅ j ๐ฆโ ๐ฅ , ๐ฅ โ ๐ฆ
โข Concave functions: ๐ ๐ฆ โค ๐ ๐ฅ + ๐ป๐ ๐ฅ j(๐ฆ โ ๐ฅ)
7
a global underestimator
Proofoffirst-ordercondition:chap3.1.3withthepropertyofrestricting๐ toaline.
Second Order Conditionโข Suppose ๐ is twice differentiable (๐๐๐๐ is open and its Hessianexists at โ๐ฅ โ ๐๐๐๐), then ๐ is convex iff dom๐ is convexand for all๐ฅ, ๐ฆ โ ๐๐๐๐
๐ปZ๐ ๐ฅ โฝ 0 (positive semidefinite)
โข Review the proof in class: necessary and sufficiency
โข Strict convexity:๐ปZ๐ ๐ฅ โป 0โข Concave functions: ๐ปZ๐ ๐ฅ โผ 0
8
Example of Convex Functionsโข Quadratic over linear function
๐ ๐ฅ, ๐ฆ =๐ฅZ
๐ฆ , for๐ฆ > 0
Its gradient ๐ป๐ ๐ฅ =opoqopor
=
Zqr\qb
rb
Hessian ๐ปZ๐ ๐ฅ =obpoqb
obpoqor
obporoq
obporb
= Zrs
๐ฆZ โ๐ฅ๐ฆโ๐ฅ๐ฆ ๐ฅZ
โฝ 0 โ convex
Positive semidefinite? ๐ฆโ๐ฅ
๐ฆโ๐ฅ
u, for any ๐ข โ ๐ Z, ๐ขj ๐ฃ๐ฃj ๐ข = ๐ฃj๐ข j ๐ฃj๐ข =
๐ฃj๐ข ZZ โฅ 0.
9Moreexamplesseechap.3.1.5
Epigraphโข ๐ผ-sublevel set of ๐: ๐ $ โ ๐
๐ถx = ๐ฅ โ ๐๐๐๐ ๐ ๐ฅ โค ๐ผ}sublevel sets of a convex function are convex for any value of ๐ผ.โข Epigraph of ๐: ๐ $ โ ๐ is defined as
๐๐ฉ๐ข๐ = (๐ฅ, ๐ก) ๐ฅ โ ๐๐๐๐, ๐ ๐ฅ โค ๐ก} โ ๐น๐V๐
โข A function is convex iff its epigraph is a convex set. 10
Relation between convex sets and convex functionsโข A function is convex iff its epigraph is a convex set.
โข Consider a convex function ๐ and ๐ฅ, ๐ฆ โ ๐๐๐๐๐ก โฅ ๐ ๐ฆ โฅ ๐ ๐ฅ + ๐ป๐ ๐ฅ j(๐ฆ โ ๐ฅ)
โข The hyperplane supports epi ๐ at (๐ฅ, ๐ ๐ฅ ), for any
๐ฆ, ๐ก โ ๐๐ฉ๐ข๐ โ๐ป๐ ๐ฅ j ๐ฆโ ๐ฅ + ๐ ๐ฅ โ ๐ก โค 0
โ ๐ป๐ ๐ฅโ1
j ๐ฆ๐ก โ ๐ฅ
๐ ๐ฅ โค 0
11
Supporting hyperplane, derivedfrom first order condition
First order condition for convexityepi ๐๐ก
๐ฅ
Operations that preserve convexityPractical methods for establishing convexity of a functionโข Verify definition ( often simplified by restricting to a line)โข For twice differentiable functions, show ๐ปZ๐(๐ฅ) โฝ 0โข Show that ๐ isobtained from simple convex functions by operationsthat preserve convexity (Ref. Chap. 3.2)โข Nonnegativeweighted sumโข Compositionwith affine functionโข Pointwisemaximum and supremumโข Compositionโข Minimizationโข Perspective
12
Pointwise Supremumโข If for each ๐ฆ โ ๐, ๐(๐ฅ, ๐ฆ): ๐ $ โ ๐ is convex in ๐ฅ, then function
๐(๐ฅ) = suprโ๏ฟฝ
๐(๐ฅ, ๐ฆ)
is convex in ๐ฅ.โข Epigraph of ๐ ๐ฅ is the intersection of epigraphs with ๐ and set ๐
๐๐ฉ๐ข๐ =โฉrโ๏ฟฝ ๐๐ฉ๐ข๐(๏ฟฝ, ๐ฆ)knowing ๐๐ฉ๐ข๐(๏ฟฝ, ๐ฆ) is a convex set (๐ is a convex function in ๐ฅ and regard
๐ฆ as a const), so ๐๐ฉ๐ข๐ is convex.
โข An interesting method to establish convexity of a function: the pointwisesupremum of a family of affine functions. (ref. Chap 3.2.4)
13
Contents
โข Convex functions (Ref. Chap.3)โข Definitionโข First order conditionโข Second order conditionโข Epigraphโข Operationsthatpreserveconvexity
โข Conjugatefunctions
14
Conjugate Functionโข Given function ๐: ๐ $ โ ๐ , the conjugate function
๐โ ๐ฆ = supqโ๏ฟฝ๏ฟฝ๏ฟฝp
๐ฆj๐ฅ โ ๐(๐ฅ)
โข The dom ๐โ consists ๐ฆ โ ๐ $ for which supqโ๏ฟฝ๏ฟฝ๏ฟฝp
๐ฆj๐ฅ โ ๐(๐ฅ) is bounded
Theorem: ๐โ(๐ฆ) is convex even ๐ ๐ฅ is not convex.15
(Proof with pointwise supremum)๐๐ป๐ โ ๐(๐) is affine function in ๐
Supportinghyperplane: if๐ is convex in that domain
(๐ฅ๏ฟฝ , ๐(๐ฅ๏ฟฝ)) where๐ป๐j ๐ฅ๏ฟฝ = ๐ฆ if ๐ is differentiable
(0,โ๐โ(0))
Slope ๐ฆ = โ1
Slope ๐ฆ = 0
Examples of Conjugatesโข Derive the conjugates of ๐: ๐ โ ๐
๐โ ๐ฆ = supqโ๏ฟฝ๏ฟฝ๏ฟฝp
๐ฆ๐ฅ โ ๐(๐ฅ)
16
Affine
Normquadratic
See moreexamplesinchap3.3.1
Properties of Conjugates
17
(1)๐ ๐ฅ + ๐โ ๐ฆ โฅ ๐ฅj๐ฆFenchelโsinequality.Thus,intheaboveexample
๐ฅj๐ฆ โค @Z๐ฅj๐๐ฅ + @
Z๐ฆj๐\@๐ฆ, โ๐ฅ, ๐ฆ โ ๐ $,๐ โ ๐VV$
(2)๐โโ = ๐,iff isconvex&f isclosed(i.e.epif isaclosedset)(3)Iff isconvex&differentiable,๐๐๐๐ = ๐ $
Formax๐ฆj๐ฅ โ ๐(๐ฅ),wehave๐ฆ = ๐ป๐(๐ฅโ)Thus,๐โ ๐ฆ = ๐ฅโj๐ป๐ ๐ฅโ โ ๐ ๐ฅโ , ๐ฆ = ๐ป๐(๐ฅโ)