dagm 2011 tutorial on convex optimization for computer ... · tugraz graz university of technology...
TRANSCRIPT
![Page 1: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/1.jpg)
DAGM 2011 Tutorial onConvex Optimization for Computer Vision
Part 1: Convexity and Convex Optimization
Daniel CremersComputer Vision Group
Technical University of Munich
Graz University of Technology
Thomas PockInstitute for Computer Graphics and Vision
Graz University of Technology
Frankfurt, August 30, 2011
![Page 2: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/2.jpg)
tugrazGraz University of Technology
Overview
1 Introduction
2 Basics of convex analysis
3 Convex optimization
4 A class of convex problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
2 / 40
![Page 3: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/3.jpg)
tugrazGraz University of Technology
Overview
1 Introduction
2 Basics of convex analysis
3 Convex optimization
4 A class of convex problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
3 / 40
![Page 4: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/4.jpg)
tugrazGraz University of Technology
Computer vision deals with inverse problems
Projection of the 3D world onto the 2D image plane
Determine unknown model parameters based on observed data
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 40
![Page 5: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/5.jpg)
tugrazGraz University of Technology
Computer vision deals with inverse problems
Projection of the 3D world onto the 2D image plane
Determine unknown model parameters based on observed data
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 40
![Page 6: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/6.jpg)
tugrazGraz University of Technology
Computer vision is highly ambiguous
What you see ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
5 / 40
![Page 7: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/7.jpg)
tugrazGraz University of Technology
Computer vision is highly ambiguous
What you see ... is maybe not what it is!
[Fukuda’s Underground Piano Illusion]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
5 / 40
![Page 8: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/8.jpg)
tugrazGraz University of Technology
Energy minimization methods
It is in general not possible to solve inverse problems directlyAdd some smoothness assumption to the unknown solutionLeads to the energy minimization approach
minuE(u) = R(u) +D(u, f ) ,
where f is the input data and u is the unknown solutionEnergy functional is designed such that low-energy states reflect the physicalproperties of the problemMinimizer provides the best (in the sense of the model) solution to the problem
Different philosophies:Discrete MRF setting: Images are represented as graphs G(V, E), consisting of a nodeset V, and an edge set E. Each node v ∈ V can take a label from a discrete label setU ⊂ Z, i.e. u(v) ∈ UContinuous Variational setting: Images are considered as continuous functionsu : Ω→ R, where Ω ⊂ Rn is the image domain.
Link to statistics: In a Bayesian setting, the energy relates to the posteriorprobability via
p(u|f ) =1
Zexp(−E(u))
Computing the minimizer of E(u) is equivalent to MAP estimation on p(u|f )
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 40
![Page 9: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/9.jpg)
tugrazGraz University of Technology
Energy minimization methods
It is in general not possible to solve inverse problems directlyAdd some smoothness assumption to the unknown solutionLeads to the energy minimization approach
minuE(u) = R(u) +D(u, f ) ,
where f is the input data and u is the unknown solutionEnergy functional is designed such that low-energy states reflect the physicalproperties of the problemMinimizer provides the best (in the sense of the model) solution to the problem
Different philosophies:Discrete MRF setting: Images are represented as graphs G(V, E), consisting of a nodeset V, and an edge set E. Each node v ∈ V can take a label from a discrete label setU ⊂ Z, i.e. u(v) ∈ UContinuous Variational setting: Images are considered as continuous functionsu : Ω→ R, where Ω ⊂ Rn is the image domain.
Link to statistics: In a Bayesian setting, the energy relates to the posteriorprobability via
p(u|f ) =1
Zexp(−E(u))
Computing the minimizer of E(u) is equivalent to MAP estimation on p(u|f )
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 40
![Page 10: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/10.jpg)
tugrazGraz University of Technology
Energy minimization methods
It is in general not possible to solve inverse problems directlyAdd some smoothness assumption to the unknown solutionLeads to the energy minimization approach
minuE(u) = R(u) +D(u, f ) ,
where f is the input data and u is the unknown solutionEnergy functional is designed such that low-energy states reflect the physicalproperties of the problemMinimizer provides the best (in the sense of the model) solution to the problem
Different philosophies:Discrete MRF setting: Images are represented as graphs G(V, E), consisting of a nodeset V, and an edge set E. Each node v ∈ V can take a label from a discrete label setU ⊂ Z, i.e. u(v) ∈ UContinuous Variational setting: Images are considered as continuous functionsu : Ω→ R, where Ω ⊂ Rn is the image domain.
Link to statistics: In a Bayesian setting, the energy relates to the posteriorprobability via
p(u|f ) =1
Zexp(−E(u))
Computing the minimizer of E(u) is equivalent to MAP estimation on p(u|f )
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 40
![Page 11: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/11.jpg)
tugrazGraz University of Technology
Example: Total variation based image restoration
Image model: f = k ∗ u + n, blur kernel k =Variational model: [Rudin, Osher, Fatemi ’92]
minu
∫Ω|Du|+
λ
2‖k ∗ u − f ‖2
2
(a) Degraded image f
(b) Reconstructed image u
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
7 / 40
![Page 12: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/12.jpg)
tugrazGraz University of Technology
Example: Total variation based image restoration
Image model: f = k ∗ u + n, blur kernel k =Variational model: [Rudin, Osher, Fatemi ’92]
minu
∫Ω|Du|+
λ
2‖k ∗ u − f ‖2
2
(a) Degraded image f (b) Reconstructed image u
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
7 / 40
![Page 13: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/13.jpg)
tugrazGraz University of Technology
Optimization problems are unsolvable
Consider the following general mathematical optimization problem:
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
x ∈ S ,
where f0(x)...fm(x) are real-valued functions, x = (x1, ...xn)T ∈ Rn is a n-dimensionalreal-valued vector, and S is a subset of Rn
How to solve this problem?
Naive: “Download a commercial package ...”
Reality: “Finding a solution is far from being trivial!”
Example: Minimize a function of 10 variables over the unit box
Can take more than 30 million years to find an approximate solution!
“Optimization problems are unsolvable”
[Nesterov ’04]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 40
![Page 14: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/14.jpg)
tugrazGraz University of Technology
Optimization problems are unsolvable
Consider the following general mathematical optimization problem:
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
x ∈ S ,
where f0(x)...fm(x) are real-valued functions, x = (x1, ...xn)T ∈ Rn is a n-dimensionalreal-valued vector, and S is a subset of Rn
How to solve this problem?
Naive: “Download a commercial package ...”
Reality: “Finding a solution is far from being trivial!”
Example: Minimize a function of 10 variables over the unit box
Can take more than 30 million years to find an approximate solution!
“Optimization problems are unsolvable”
[Nesterov ’04]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 40
![Page 15: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/15.jpg)
tugrazGraz University of Technology
Optimization problems are unsolvable
Consider the following general mathematical optimization problem:
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
x ∈ S ,
where f0(x)...fm(x) are real-valued functions, x = (x1, ...xn)T ∈ Rn is a n-dimensionalreal-valued vector, and S is a subset of Rn
How to solve this problem?
Naive: “Download a commercial package ...”
Reality: “Finding a solution is far from being trivial!”
Example: Minimize a function of 10 variables over the unit box
Can take more than 30 million years to find an approximate solution!
“Optimization problems are unsolvable”
[Nesterov ’04]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 40
![Page 16: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/16.jpg)
tugrazGraz University of Technology
Optimization problems are unsolvable
Consider the following general mathematical optimization problem:
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
x ∈ S ,
where f0(x)...fm(x) are real-valued functions, x = (x1, ...xn)T ∈ Rn is a n-dimensionalreal-valued vector, and S is a subset of Rn
How to solve this problem?
Naive: “Download a commercial package ...”
Reality: “Finding a solution is far from being trivial!”
Example: Minimize a function of 10 variables over the unit box
Can take more than 30 million years to find an approximate solution!
“Optimization problems are unsolvable”
[Nesterov ’04]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 40
![Page 17: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/17.jpg)
tugrazGraz University of Technology
Optimization problems are unsolvable
Consider the following general mathematical optimization problem:
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
x ∈ S ,
where f0(x)...fm(x) are real-valued functions, x = (x1, ...xn)T ∈ Rn is a n-dimensionalreal-valued vector, and S is a subset of Rn
How to solve this problem?
Naive: “Download a commercial package ...”
Reality: “Finding a solution is far from being trivial!”
Example: Minimize a function of 10 variables over the unit box
Can take more than 30 million years to find an approximate solution!
“Optimization problems are unsolvable”
[Nesterov ’04]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 40
![Page 18: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/18.jpg)
tugrazGraz University of Technology
Convex versus non-convex
Non-convex problemsOften give more accurate modelsIn general no chance to find the global minimizerResult strongly depends on the initializationDilemma: Wrong model or wrong algorithm?
Convex problemsConvex models often inferiorAny local minimizer is a global minimizerResult is independent of the initializationNote: Convex does not mean easy!
Current research: Bridging the gap between convex and non-convex optimizationConvex approximations of non-convex modelsNew modelsAlgorithmsBounds
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
9 / 40
![Page 19: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/19.jpg)
tugrazGraz University of Technology
Convex versus non-convex
Non-convex problemsOften give more accurate modelsIn general no chance to find the global minimizerResult strongly depends on the initializationDilemma: Wrong model or wrong algorithm?
Convex problemsConvex models often inferiorAny local minimizer is a global minimizerResult is independent of the initializationNote: Convex does not mean easy!
Current research: Bridging the gap between convex and non-convex optimizationConvex approximations of non-convex modelsNew modelsAlgorithmsBounds
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
9 / 40
![Page 20: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/20.jpg)
tugrazGraz University of Technology
Convex versus non-convex
Non-convex problemsOften give more accurate modelsIn general no chance to find the global minimizerResult strongly depends on the initializationDilemma: Wrong model or wrong algorithm?
Convex problemsConvex models often inferiorAny local minimizer is a global minimizerResult is independent of the initializationNote: Convex does not mean easy!
Current research: Bridging the gap between convex and non-convex optimizationConvex approximations of non-convex modelsNew modelsAlgorithmsBounds
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
9 / 40
![Page 21: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/21.jpg)
tugrazGraz University of Technology
Overview
1 Introduction
2 Basics of convex analysis
3 Convex optimization
4 A class of convex problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
10 / 40
![Page 22: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/22.jpg)
tugrazGraz University of Technology
Literatur on convex analysis and optimization
Convex optimization, [Boyd,Vandenberghe ’04]
Introductory lectures on convex optimization, [Nesterov ’04]
Nonlinear programming, [Bertsekas ’99]
Variational analysis, [Rockafellar, Wets ’88]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
11 / 40
![Page 23: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/23.jpg)
tugrazGraz University of Technology
Convex sets
Consider two points x1, x2 ∈ Rn
The line segment between these two points is given by the points
x = θx1 + (1− θ)x2, θ ∈ [0, 1]
A set C is said to be convex, if it contains all line segments between any twopoints in the set
x1, x2 ∈ C ⇒ θx1 + (1− θ)x2 ∈ C , ∀θ ∈ [0, 1]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 40
![Page 24: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/24.jpg)
tugrazGraz University of Technology
Convex sets
Consider two points x1, x2 ∈ Rn
The line segment between these two points is given by the points
x = θx1 + (1− θ)x2, θ ∈ [0, 1]
A set C is said to be convex, if it contains all line segments between any twopoints in the set
x1, x2 ∈ C ⇒ θx1 + (1− θ)x2 ∈ C , ∀θ ∈ [0, 1]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 40
![Page 25: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/25.jpg)
tugrazGraz University of Technology
Convex combination and convex hull
Convex combination of a set of points x1, ..., xk, xi ∈ Rn is given by
x =k∑
i=1
θixi , θi ≥ 0,k∑
i=1
θi = 1
The convex hull of a set of points x1, ..., xk is the set of all convex combinations
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
13 / 40
![Page 26: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/26.jpg)
tugrazGraz University of Technology
A few important convex sets
Hyperplane: x | aT x = b
Halfspace: x | aT x ≤ b
Polyhedra: Interesection of finitely many hyperplanes and halfspaces
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
14 / 40
![Page 27: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/27.jpg)
tugrazGraz University of Technology
A few important convex sets
Hyperplane: x | aT x = b
Halfspace: x | aT x ≤ b
Polyhedra: Interesection of finitely many hyperplanes and halfspaces
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
14 / 40
![Page 28: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/28.jpg)
tugrazGraz University of Technology
A few important convex sets
Hyperplane: x | aT x = b
Halfspace: x | aT x ≤ b
Polyhedra: Interesection of finitely many hyperplanes and halfspaces
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
14 / 40
![Page 29: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/29.jpg)
tugrazGraz University of Technology
A few important convex sets
Norm Ball: x | ‖x − xc‖ ≤ r, where ‖·‖ is any norm
Ellipsoid: x | (x − xc )P−1(x − xc ) ≤ 1, P symmetric and positive definite
Norm cone: (x , t) | ‖x‖ ≤ t
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
15 / 40
![Page 30: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/30.jpg)
tugrazGraz University of Technology
A few important convex sets
Norm Ball: x | ‖x − xc‖ ≤ r, where ‖·‖ is any norm
Ellipsoid: x | (x − xc )P−1(x − xc ) ≤ 1, P symmetric and positive definite
Norm cone: (x , t) | ‖x‖ ≤ t
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
15 / 40
![Page 31: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/31.jpg)
tugrazGraz University of Technology
A few important convex sets
Norm Ball: x | ‖x − xc‖ ≤ r, where ‖·‖ is any norm
Ellipsoid: x | (x − xc )P−1(x − xc ) ≤ 1, P symmetric and positive definite
Norm cone: (x , t) | ‖x‖ ≤ t
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
15 / 40
![Page 32: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/32.jpg)
tugrazGraz University of Technology
How to show convexity of a set?
Check definition
x1, x2 ∈ C ⇒ θx1 + (1− θ)x2 ∈ C , ∀θ ∈ [0, 1]
Show that the set is obtained from simple sets by operations that preserveconvexity
IntersectionAffine transformation (scaling, translation, ...)...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
16 / 40
![Page 33: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/33.jpg)
tugrazGraz University of Technology
Convex functions
Definition: A function f : Rn → R is convex, if dom f is a convex set and
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y), ∀x , y ∈ dom f , θ ∈ [0, 1]
f is said to be concave, if −f is convex
f is strictly convex, if
f (θx + (1− θ)y) < θf (x) + (1− θ)f (y), ∀x , y ∈ dom f , x 6= y , θ ∈ (0, 1)
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
17 / 40
![Page 34: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/34.jpg)
tugrazGraz University of Technology
Examples of convex functions
Examples on RLinear: ax + b, a, b ∈ RExponential: eax , a ∈ R
Powers: xα, for x ≥ 0, α ≥ 1 or α ≤ 0
Powers of absolute values: |x |α, α ≥ 0
Negative entropy: x log x , for x ≥ 0
Examples on Rn
Affine: aT x + b, a, b ∈ Rn
Norms: ‖x‖p =(∑n
i=1 |xi |p) 1
p , p ≥ 1
Examples on Rm×n
Affine: tr(ATX ) + b =∑m
i=1
∑nj=1 AijXij + b, A ∈ Rm×n, b ∈ R
Spectral norm (max non-singular value): ‖X‖2
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
18 / 40
![Page 35: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/35.jpg)
tugrazGraz University of Technology
Examples of convex functions
Examples on RLinear: ax + b, a, b ∈ RExponential: eax , a ∈ R
Powers: xα, for x ≥ 0, α ≥ 1 or α ≤ 0
Powers of absolute values: |x |α, α ≥ 0
Negative entropy: x log x , for x ≥ 0
Examples on Rn
Affine: aT x + b, a, b ∈ Rn
Norms: ‖x‖p =(∑n
i=1 |xi |p) 1
p , p ≥ 1
Examples on Rm×n
Affine: tr(ATX ) + b =∑m
i=1
∑nj=1 AijXij + b, A ∈ Rm×n, b ∈ R
Spectral norm (max non-singular value): ‖X‖2
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
18 / 40
![Page 36: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/36.jpg)
tugrazGraz University of Technology
Examples of convex functions
Examples on RLinear: ax + b, a, b ∈ RExponential: eax , a ∈ R
Powers: xα, for x ≥ 0, α ≥ 1 or α ≤ 0
Powers of absolute values: |x |α, α ≥ 0
Negative entropy: x log x , for x ≥ 0
Examples on Rn
Affine: aT x + b, a, b ∈ Rn
Norms: ‖x‖p =(∑n
i=1 |xi |p) 1
p , p ≥ 1
Examples on Rm×n
Affine: tr(ATX ) + b =∑m
i=1
∑nj=1 AijXij + b, A ∈ Rm×n, b ∈ R
Spectral norm (max non-singular value): ‖X‖2
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
18 / 40
![Page 37: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/37.jpg)
tugrazGraz University of Technology
Sufficient conditions of convexity
First-order condition: A differentiable function f with convex domain is convex iff
f (y) ≥ f (x) +∇f (x)T (y − x), ∀x , y ∈ dom f
Second-order condition: A twice differentiable function f with convex domain isconvex iff
∇2f (x) 0, ∀x ∈ dom f
It is strictly convex if the assertion becomes ∇2f (x) 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
19 / 40
![Page 38: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/38.jpg)
tugrazGraz University of Technology
Sufficient conditions of convexity
First-order condition: A differentiable function f with convex domain is convex iff
f (y) ≥ f (x) +∇f (x)T (y − x), ∀x , y ∈ dom f
Second-order condition: A twice differentiable function f with convex domain isconvex iff
∇2f (x) 0, ∀x ∈ dom f
It is strictly convex if the assertion becomes ∇2f (x) 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
19 / 40
![Page 39: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/39.jpg)
tugrazGraz University of Technology
Example
Quadratic over linear function
f (x , y) =x2
y
Computing the gradient and Hessian
∇f (x , y) = (2x
y,−
x2
y2), ∇2f (x , y) =
2
y3(y ,−x)T (y ,−x)
Second-order sufficient condition: ∇2f (x , y) 0 for y ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
20 / 40
![Page 40: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/40.jpg)
tugrazGraz University of Technology
Example
Quadratic over linear function
f (x , y) =x2
y
Computing the gradient and Hessian
∇f (x , y) = (2x
y,−
x2
y2), ∇2f (x , y) =
2
y3(y ,−x)T (y ,−x)
Second-order sufficient condition: ∇2f (x , y) 0 for y ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
20 / 40
![Page 41: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/41.jpg)
tugrazGraz University of Technology
Example
Quadratic over linear function
f (x , y) =x2
y
Computing the gradient and Hessian
∇f (x , y) = (2x
y,−
x2
y2), ∇2f (x , y) =
2
y3(y ,−x)T (y ,−x)
Second-order sufficient condition: ∇2f (x , y) 0 for y ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
20 / 40
![Page 42: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/42.jpg)
tugrazGraz University of Technology
Example
Quadratic over linear function
f (x , y) =x2
y
Computing the gradient and Hessian
∇f (x , y) = (2x
y,−
x2
y2), ∇2f (x , y) =
2
y3(y ,−x)T (y ,−x)
Second-order sufficient condition: ∇2f (x , y) 0 for y ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
20 / 40
![Page 43: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/43.jpg)
tugrazGraz University of Technology
How to verify convexity?
Verify the definition of convex functions
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y), ∀x , y ∈ dom f , θ ∈ [0, 1]
Sometimes simpler to consider the 1D case
Check for ∇2f 0
∇2f 0 iff vT (∇2f )v ≥ 0, ∀v 6= 0
Show that f is obtained from simple convex functions by operations that preserveconvexity
Non-negative weihgted sum: f =∑
i fi , is convex if αi > 0, fi are convex
Composition with an affine function: f (aT x + b) is convex if f is convexPointwise maximum: f (x) = maxf1(x), ..., fn(x) is convex if f1...fn are convex
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
21 / 40
![Page 44: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/44.jpg)
tugrazGraz University of Technology
How to verify convexity?
Verify the definition of convex functions
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y), ∀x , y ∈ dom f , θ ∈ [0, 1]
Sometimes simpler to consider the 1D case
Check for ∇2f 0
∇2f 0 iff vT (∇2f )v ≥ 0, ∀v 6= 0
Show that f is obtained from simple convex functions by operations that preserveconvexity
Non-negative weihgted sum: f =∑
i fi , is convex if αi > 0, fi are convex
Composition with an affine function: f (aT x + b) is convex if f is convexPointwise maximum: f (x) = maxf1(x), ..., fn(x) is convex if f1...fn are convex
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
21 / 40
![Page 45: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/45.jpg)
tugrazGraz University of Technology
How to verify convexity?
Verify the definition of convex functions
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y), ∀x , y ∈ dom f , θ ∈ [0, 1]
Sometimes simpler to consider the 1D case
Check for ∇2f 0
∇2f 0 iff vT (∇2f )v ≥ 0, ∀v 6= 0
Show that f is obtained from simple convex functions by operations that preserveconvexity
Non-negative weihgted sum: f =∑
i fi , is convex if αi > 0, fi are convex
Composition with an affine function: f (aT x + b) is convex if f is convexPointwise maximum: f (x) = maxf1(x), ..., fn(x) is convex if f1...fn are convex
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
21 / 40
![Page 46: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/46.jpg)
tugrazGraz University of Technology
The convex conjugate
The convex conjugate f ∗(y) of a function f (x) is defined as
f ∗(y) = supx∈dom f
〈x , y〉 − f (x)
f ∗(y) is a convx function (pointwise supremum over linear functions)
The biconjugate function f ∗∗(x) is the largest convex l.s.c. function below f (x)
If f (x) is a convex, l.s.c. function, f ∗∗(x) = f (x)
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
22 / 40
![Page 47: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/47.jpg)
tugrazGraz University of Technology
Examples
f (x) = |x |:
f ∗(y) = supx〈x , y〉 − |x | =
0 if |y | ≤ 1∞ else
f (x) = 12xTQx , Q, posoitive definite
f ∗(y) = supx〈x , y〉 −
1
2xTQx =
1
2yTQ−1y
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
23 / 40
![Page 48: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/48.jpg)
tugrazGraz University of Technology
Duality
Fenchel’s duality theorem
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
24 / 40
![Page 49: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/49.jpg)
tugrazGraz University of Technology
Overview
1 Introduction
2 Basics of convex analysis
3 Convex optimization
4 A class of convex problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
25 / 40
![Page 50: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/50.jpg)
tugrazGraz University of Technology
Convex optimization
A general convex optimization problem is defined as
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
Ax = b
where f0(x)...fp(x) are real-valued convex functions, x = (x1, ...xn)T ∈ Rn is am-dimensional real-valued vector, Ax = b are affine equality constraints
Convex optimization problems are considered as solvable!
“The great watershed in optimization isn’t between linearity and non-linearity,but convexity and non-convexity” [Rockafellar ’93]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
26 / 40
![Page 51: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/51.jpg)
tugrazGraz University of Technology
Convex optimization
A general convex optimization problem is defined as
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
Ax = b
where f0(x)...fp(x) are real-valued convex functions, x = (x1, ...xn)T ∈ Rn is am-dimensional real-valued vector, Ax = b are affine equality constraints
Convex optimization problems are considered as solvable!
“The great watershed in optimization isn’t between linearity and non-linearity,but convexity and non-convexity” [Rockafellar ’93]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
26 / 40
![Page 52: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/52.jpg)
tugrazGraz University of Technology
Convex optimization
A general convex optimization problem is defined as
min f0(x)s.t. fi (x) ≤ 0 , i = 1 . . .m
Ax = b
where f0(x)...fp(x) are real-valued convex functions, x = (x1, ...xn)T ∈ Rn is am-dimensional real-valued vector, Ax = b are affine equality constraints
Convex optimization problems are considered as solvable!
“The great watershed in optimization isn’t between linearity and non-linearity,but convexity and non-convexity” [Rockafellar ’93]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
26 / 40
![Page 53: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/53.jpg)
tugrazGraz University of Technology
Black box convex optimization
Generic iterative algorithm for convex optimization:
1 Pick any initial vector x0 ∈ Rn, set k = 0
2 Compute search direction dk ∈ Rn
3 Choose step size τk such that f (xk + τkdk ) < f (xk )
4 Set xk+1 = xk + τkdk , k = k + 1
5 Stop if converged, else goto 2
Different methods to determine the search direction dk
steepest descend
conjugate gradients
Newton, quasi-Newton
Working-horse for the lazy: Limited memory BFGS quasi-Newton method[Nocedal ’80]
Black box methods do not exploit the structure of the problem and hence are oftenless effective
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
27 / 40
![Page 54: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/54.jpg)
tugrazGraz University of Technology
Black box convex optimization
Generic iterative algorithm for convex optimization:
1 Pick any initial vector x0 ∈ Rn, set k = 0
2 Compute search direction dk ∈ Rn
3 Choose step size τk such that f (xk + τkdk ) < f (xk )
4 Set xk+1 = xk + τkdk , k = k + 1
5 Stop if converged, else goto 2
Different methods to determine the search direction dk
steepest descend
conjugate gradients
Newton, quasi-Newton
Working-horse for the lazy: Limited memory BFGS quasi-Newton method[Nocedal ’80]
Black box methods do not exploit the structure of the problem and hence are oftenless effective
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
27 / 40
![Page 55: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/55.jpg)
tugrazGraz University of Technology
Black box convex optimization
Generic iterative algorithm for convex optimization:
1 Pick any initial vector x0 ∈ Rn, set k = 0
2 Compute search direction dk ∈ Rn
3 Choose step size τk such that f (xk + τkdk ) < f (xk )
4 Set xk+1 = xk + τkdk , k = k + 1
5 Stop if converged, else goto 2
Different methods to determine the search direction dk
steepest descend
conjugate gradients
Newton, quasi-Newton
Working-horse for the lazy: Limited memory BFGS quasi-Newton method[Nocedal ’80]
Black box methods do not exploit the structure of the problem and hence are oftenless effective
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
27 / 40
![Page 56: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/56.jpg)
tugrazGraz University of Technology
Structured convex optimization
There are a number of efficient solvers available for important types of structuredconvex optimization problems [Boyd, Vandenberghe ’04]:
Least squares problems:minx‖Ax − b‖2
2
Quadratic program (QP) and linear program (LP) (Q = 0):
min 12xTQx + cT x , Q 0
s.t. Ax = b, l ≤ x ≤ u
Second order cone program (SOCP)
min f T xs.t. ‖Aix + bi‖2 ≤ cTi x + di , i = 1...m, Fx = g
Many problems can be formulated in these frameworks
Fast methods available (simplex, interior point, ...)
Sometimes ineffective for large-scale problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
28 / 40
![Page 57: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/57.jpg)
tugrazGraz University of Technology
Structured convex optimization
There are a number of efficient solvers available for important types of structuredconvex optimization problems [Boyd, Vandenberghe ’04]:
Least squares problems:minx‖Ax − b‖2
2
Quadratic program (QP) and linear program (LP) (Q = 0):
min 12xTQx + cT x , Q 0
s.t. Ax = b, l ≤ x ≤ u
Second order cone program (SOCP)
min f T xs.t. ‖Aix + bi‖2 ≤ cTi x + di , i = 1...m, Fx = g
Many problems can be formulated in these frameworks
Fast methods available (simplex, interior point, ...)
Sometimes ineffective for large-scale problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
28 / 40
![Page 58: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/58.jpg)
tugrazGraz University of Technology
Structured convex optimization
There are a number of efficient solvers available for important types of structuredconvex optimization problems [Boyd, Vandenberghe ’04]:
Least squares problems:minx‖Ax − b‖2
2
Quadratic program (QP) and linear program (LP) (Q = 0):
min 12xTQx + cT x , Q 0
s.t. Ax = b, l ≤ x ≤ u
Second order cone program (SOCP)
min f T xs.t. ‖Aix + bi‖2 ≤ cTi x + di , i = 1...m, Fx = g
Many problems can be formulated in these frameworks
Fast methods available (simplex, interior point, ...)
Sometimes ineffective for large-scale problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
28 / 40
![Page 59: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/59.jpg)
tugrazGraz University of Technology
Structured convex optimization
There are a number of efficient solvers available for important types of structuredconvex optimization problems [Boyd, Vandenberghe ’04]:
Least squares problems:minx‖Ax − b‖2
2
Quadratic program (QP) and linear program (LP) (Q = 0):
min 12xTQx + cT x , Q 0
s.t. Ax = b, l ≤ x ≤ u
Second order cone program (SOCP)
min f T xs.t. ‖Aix + bi‖2 ≤ cTi x + di , i = 1...m, Fx = g
Many problems can be formulated in these frameworks
Fast methods available (simplex, interior point, ...)
Sometimes ineffective for large-scale problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
28 / 40
![Page 60: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/60.jpg)
tugrazGraz University of Technology
Structured convex optimization
There are a number of efficient solvers available for important types of structuredconvex optimization problems [Boyd, Vandenberghe ’04]:
Least squares problems:minx‖Ax − b‖2
2
Quadratic program (QP) and linear program (LP) (Q = 0):
min 12xTQx + cT x , Q 0
s.t. Ax = b, l ≤ x ≤ u
Second order cone program (SOCP)
min f T xs.t. ‖Aix + bi‖2 ≤ cTi x + di , i = 1...m, Fx = g
Many problems can be formulated in these frameworks
Fast methods available (simplex, interior point, ...)
Sometimes ineffective for large-scale problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
28 / 40
![Page 61: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/61.jpg)
tugrazGraz University of Technology
Overview
1 Introduction
2 Basics of convex analysis
3 Convex optimization
4 A class of convex problems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
29 / 40
![Page 62: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/62.jpg)
tugrazGraz University of Technology
A class of problems
Let us consider the following class of structured convex optimization problems
minx∈X
F (Kx) + G(x) ,
K : X → Y is a linear and continuous operator from a Hilbert space X to aHilbert space Y .
F : Y → R ∪ ∞, G : X → R ∪ ∞ are “simple” convex, proper, l.s.c.functions, and hence have an easy to compute prox operator:
proxG (z) = (I + ∂G)−1(z) = arg minx
‖x − z‖2
2+ G(x)
It turns out that many standard problems can be cast in this framework.
There exists a vast literature of numerical algorithms to solve this class ofproblems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
30 / 40
![Page 63: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/63.jpg)
tugrazGraz University of Technology
A class of problems
Let us consider the following class of structured convex optimization problems
minx∈X
F (Kx) + G(x) ,
K : X → Y is a linear and continuous operator from a Hilbert space X to aHilbert space Y .
F : Y → R ∪ ∞, G : X → R ∪ ∞ are “simple” convex, proper, l.s.c.functions, and hence have an easy to compute prox operator:
proxG (z) = (I + ∂G)−1(z) = arg minx
‖x − z‖2
2+ G(x)
It turns out that many standard problems can be cast in this framework.
There exists a vast literature of numerical algorithms to solve this class ofproblems
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
30 / 40
![Page 64: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/64.jpg)
tugrazGraz University of Technology
Examples
Image restoration: The ROF model
minu‖∇u‖1 +
λ
2‖k ∗ u − f ‖2
2 ,
Compressed sensing: Basis pursuit problem (LASSO)
minx‖x‖1 +
λ
2‖Ax − b‖2
2
Machine learning: Linear support vector machine
minw,b
λ
2‖w‖2
2 +n∑
i=1
max (0, 1− yi (〈w , xi 〉 + b))
General linear programming problems
minx〈c, x〉 , s.t.
Ax = bx ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
31 / 40
![Page 65: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/65.jpg)
tugrazGraz University of Technology
Examples
Image restoration: The ROF model
minu‖∇u‖1 +
λ
2‖k ∗ u − f ‖2
2 ,
Compressed sensing: Basis pursuit problem (LASSO)
minx‖x‖1 +
λ
2‖Ax − b‖2
2
Machine learning: Linear support vector machine
minw,b
λ
2‖w‖2
2 +n∑
i=1
max (0, 1− yi (〈w , xi 〉 + b))
General linear programming problems
minx〈c, x〉 , s.t.
Ax = bx ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
31 / 40
![Page 66: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/66.jpg)
tugrazGraz University of Technology
Examples
Image restoration: The ROF model
minu‖∇u‖1 +
λ
2‖k ∗ u − f ‖2
2 ,
Compressed sensing: Basis pursuit problem (LASSO)
minx‖x‖1 +
λ
2‖Ax − b‖2
2
Machine learning: Linear support vector machine
minw,b
λ
2‖w‖2
2 +n∑
i=1
max (0, 1− yi (〈w , xi 〉 + b))
General linear programming problems
minx〈c, x〉 , s.t.
Ax = bx ≥ 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
31 / 40
![Page 67: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/67.jpg)
tugrazGraz University of Technology
Primal, dual, primal-dual
The real power of convex optimization comes through duality
Recall the convex conjugate:
F∗(y) = supx∈X〈x , y〉 − F (x) , F∗∗(x) = sup
y∈Y〈x , y〉 − F∗(y)
If F (x) convex, l.s.c. than F∗∗(x) = F (x)
minx∈X
F (Kx) + G(x) (Primal)
minx∈X
maxy∈Y〈Kx , y〉+ G(x)− F∗(y) (Primal-Dual)
maxy∈Y− (F∗(y) + G∗(−K∗y)) (Dual)
Allows to compute the duality gap, which is a measure of optimality
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
32 / 40
![Page 68: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/68.jpg)
tugrazGraz University of Technology
Primal, dual, primal-dual
The real power of convex optimization comes through duality
Recall the convex conjugate:
F∗(y) = supx∈X〈x , y〉 − F (x) , F∗∗(x) = sup
y∈Y〈x , y〉 − F∗(y)
If F (x) convex, l.s.c. than F∗∗(x) = F (x)
minx∈X
F (Kx) + G(x) (Primal)
minx∈X
maxy∈Y〈Kx , y〉+ G(x)− F∗(y) (Primal-Dual)
maxy∈Y− (F∗(y) + G∗(−K∗y)) (Dual)
Allows to compute the duality gap, which is a measure of optimality
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
32 / 40
![Page 69: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/69.jpg)
tugrazGraz University of Technology
Primal, dual, primal-dual
The real power of convex optimization comes through duality
Recall the convex conjugate:
F∗(y) = supx∈X〈x , y〉 − F (x) , F∗∗(x) = sup
y∈Y〈x , y〉 − F∗(y)
If F (x) convex, l.s.c. than F∗∗(x) = F (x)
minx∈X
F (Kx) + G(x) (Primal)
minx∈X
maxy∈Y〈Kx , y〉+ G(x)− F∗(y) (Primal-Dual)
maxy∈Y− (F∗(y) + G∗(−K∗y)) (Dual)
Allows to compute the duality gap, which is a measure of optimality
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
32 / 40
![Page 70: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/70.jpg)
tugrazGraz University of Technology
Primal, dual, primal-dual
The real power of convex optimization comes through duality
Recall the convex conjugate:
F∗(y) = supx∈X〈x , y〉 − F (x) , F∗∗(x) = sup
y∈Y〈x , y〉 − F∗(y)
If F (x) convex, l.s.c. than F∗∗(x) = F (x)
minx∈X
F (Kx) + G(x) (Primal)
minx∈X
maxy∈Y〈Kx , y〉+ G(x)− F∗(y) (Primal-Dual)
maxy∈Y− (F∗(y) + G∗(−K∗y)) (Dual)
Allows to compute the duality gap, which is a measure of optimality
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
32 / 40
![Page 71: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/71.jpg)
tugrazGraz University of Technology
Optimality conditions
We focus on the primal-dual formulation:
minx∈X
maxy∈Y〈Kx , y〉+ G(x)− F∗(y)
We assume, there exists a saddle-point (x , y) ∈ X × Y which satisfies theEuler-Lagrange equations
Kx − ∂F∗(y) 3 0
K∗y + ∂G(x) 3 0
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
33 / 40
![Page 72: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/72.jpg)
tugrazGraz University of Technology
Standard first order approaches
1. Classical Arrow-Hurwicz method [Arrow, Hurwicz, Uzawa ’58]yn+1 = (I + τ∂F∗)−1(yn + τKxn)
xn+1 = (I + τ∂G)−1(xn − τK∗yn+1)
Alternating forward-backward step in the dual variable and the primal variable
Convergence under quite restrictive assumptions on τ
For some problems very fast, e.g. ROF problem using adaptive time steps [Zhu,Chan, ’08]
2. Proximal point method [Martinet ’70], [Rockafellar ’76] for the search of zeros ofan operator, i.e. 0 ∈ T (x)
xn+1 = (I + τnT )−1(xn)
Very simple iteration
In our framework T (x) = K∗∂F (Kx) + ∂G(x)
Unfortunately, in most interesting cases, (I + τnT )−1 is hard to evaluate
Hence, the practical interest is limited
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
34 / 40
![Page 73: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/73.jpg)
tugrazGraz University of Technology
Standard first order approaches3. Douglas-Rachford splitting (DRS) [Mercier, Lions ’79]
Special case of the proximal point method if the operator T is the sum of twooperators, i.e. T = A + B
wn+1 = (I + τA)−1(2xn − wn) + wn − xn
xn+1 = (I + τB)−1(wn+1)
Only needs to evaluate the resolvent operator with respect to A and B
Let A = K∗∂F (K) and B = ∂G , the DRS algorithm becomeswn+1 = arg min
vF (Kv) +
1
2τ‖v − (2xn − wn)‖2 + wn − xn
xn+1 = arg minx
G(x) +1
2τ
∥∥x − wn+1∥∥2
Equivalent to the alternating direction method of multipliers (ADMM) [Eckstein,Bertsekas ’89]
Equivalent to the split-Bregman iteration [Goldstein, Osher ’09]
Equivalent to an alternating minimization on the augmented Lagrangianformulation
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
35 / 40
![Page 74: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/74.jpg)
tugrazGraz University of Technology
A simple primal-dual algorithm
Proposed in a series of papers: [Pock, Cremers, Bischof, Chambolle, ’09], [Chambolle,Pock, ’10], [Pock, Chambolle, ’11]
Initialization: Choose s.p.d. T, Σ, θ ∈ [0, 1], (x0, y0) ∈ X × Y .
Iterations (n ≥ 0): Update xn, yn as follows:xn+1 = (I + T∂G)−1(xn − TK∗yn)
yn+1 = (I + Σ∂F∗)−1(yn + ΣK(xn+1 + θ(xn+1 − xn)))
Alternates gradient descend in x and gradient ascend in y
Linear extrapolation of iterates of x in the y step
T, Σ can be seen as preconditioning matrices
Can be derived from a pre-conditioned DRS splitting algorithm
Can be seen as a relaxed Arrow-Hurwicz scheme
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
36 / 40
![Page 75: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/75.jpg)
tugrazGraz University of Technology
Relations to the proximal-point algorithm
The iterations of PD can be written as the variational inequality[He,Yuan ’10]⟨(
x − xn+1
y − yn+1
),
(∂G(xn+1) + K∗yn+1
∂F∗(yn+1)− Kxn+1
)+ M
(xn+1 − xn
yn+1 − yn
)⟩≥ 0 ,
M =
[T−1 −K∗−θK Σ−1
]
This is exactly the proximal-point algorithm, but with a norm in M.
Convergence is ensured if M is symmetric and positive definite
This is the case if θ = 1 and the assertion ‖Σ12 KT
12 ‖2 < 1 is fulfilled.
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
37 / 40
![Page 76: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/76.jpg)
tugrazGraz University of Technology
Relations to the proximal-point algorithm
The iterations of PD can be written as the variational inequality[He,Yuan ’10]⟨(
x − xn+1
y − yn+1
),
(∂G(xn+1) + K∗yn+1
∂F∗(yn+1)− Kxn+1
)+ M
(xn+1 − xn
yn+1 − yn
)⟩≥ 0 ,
M =
[T−1 −K∗−θK Σ−1
]
This is exactly the proximal-point algorithm, but with a norm in M.
Convergence is ensured if M is symmetric and positive definite
This is the case if θ = 1 and the assertion ‖Σ12 KT
12 ‖2 < 1 is fulfilled.
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
37 / 40
![Page 77: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/77.jpg)
tugrazGraz University of Technology
Relations to the proximal-point algorithm
The iterations of PD can be written as the variational inequality[He,Yuan ’10]⟨(
x − xn+1
y − yn+1
),
(∂G(xn+1) + K∗yn+1
∂F∗(yn+1)− Kxn+1
)+ M
(xn+1 − xn
yn+1 − yn
)⟩≥ 0 ,
M =
[T−1 −K∗−θK Σ−1
]
This is exactly the proximal-point algorithm, but with a norm in M.
Convergence is ensured if M is symmetric and positive definite
This is the case if θ = 1 and the assertion ‖Σ12 KT
12 ‖2 < 1 is fulfilled.
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
37 / 40
![Page 78: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/78.jpg)
tugrazGraz University of Technology
Relations to the proximal-point algorithm
The iterations of PD can be written as the variational inequality[He,Yuan ’10]⟨(
x − xn+1
y − yn+1
),
(∂G(xn+1) + K∗yn+1
∂F∗(yn+1)− Kxn+1
)+ M
(xn+1 − xn
yn+1 − yn
)⟩≥ 0 ,
M =
[T−1 −K∗−θK Σ−1
]
This is exactly the proximal-point algorithm, but with a norm in M.
Convergence is ensured if M is symmetric and positive definite
This is the case if θ = 1 and the assertion ‖Σ12 KT
12 ‖2 < 1 is fulfilled.
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
37 / 40
![Page 79: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/79.jpg)
tugrazGraz University of Technology
A family of diagonal preconditioners
It is important to choose T, Σ such that the prox-operators are still easy tocompute
Restrict the preconditioning matrices to diagonal matrices
It turns out that: Let K ∈ Rm×n, T = diag(τ ) and Σ = diag(σ) such that
τj =1∑m
i=1 |Ki,j |2−α, σi =
1∑nj=1 |Ki,j |α
then for any α ∈ [0, 2]
‖Σ12 KT
12 ‖2 ≤ 1 .
Gives an automatic problem-dependent choice of the primal and dual steps
Allows to apply the algorithm in a plug-and-play fashion
For α = 0, equivalent to the alternating step method [Eckstein, Bertsekas, ’90]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
38 / 40
![Page 80: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/80.jpg)
tugrazGraz University of Technology
Convergence rates
The algorithm gives different convergence rates on different problem classes[Chambolle, Pock, ’10]
F∗ and G nonsmooth: O(1/N)
F∗ or G uniformly convex: O(1/N2)
F∗ and G uniformly convex: O(ωN), ω < 1
Coincides with so far best known rates of first-order methods
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
39 / 40
![Page 81: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/81.jpg)
tugrazGraz University of Technology
Convergence rates
The algorithm gives different convergence rates on different problem classes[Chambolle, Pock, ’10]
F∗ and G nonsmooth: O(1/N)
F∗ or G uniformly convex: O(1/N2)
F∗ and G uniformly convex: O(ωN), ω < 1
Coincides with so far best known rates of first-order methods
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
39 / 40
![Page 82: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/82.jpg)
tugrazGraz University of Technology
Convergence rates
The algorithm gives different convergence rates on different problem classes[Chambolle, Pock, ’10]
F∗ and G nonsmooth: O(1/N)
F∗ or G uniformly convex: O(1/N2)
F∗ and G uniformly convex: O(ωN), ω < 1
Coincides with so far best known rates of first-order methods
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
39 / 40
![Page 83: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/83.jpg)
tugrazGraz University of Technology
Convergence rates
The algorithm gives different convergence rates on different problem classes[Chambolle, Pock, ’10]
F∗ and G nonsmooth: O(1/N)
F∗ or G uniformly convex: O(1/N2)
F∗ and G uniformly convex: O(ωN), ω < 1
Coincides with so far best known rates of first-order methods
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
39 / 40
![Page 84: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/84.jpg)
tugrazGraz University of Technology
Parallel computing?
The algorithm basically computes matrix-vector productsThe matrices are usually very sparse
One simple processor for each unknown
Each processor has a small amount of local memory
According to the structure of K , each processor can inter-change data with itsneighboring processors
Recent GPUs already go into this direction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
40 / 40
![Page 85: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/85.jpg)
tugrazGraz University of Technology
Parallel computing?
The algorithm basically computes matrix-vector productsThe matrices are usually very sparse
One simple processor for each unknown
Each processor has a small amount of local memory
According to the structure of K , each processor can inter-change data with itsneighboring processors
Recent GPUs already go into this direction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
40 / 40
![Page 86: DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class](https://reader033.vdocuments.net/reader033/viewer/2022042303/5ece373fad894a6137715a6a/html5/thumbnails/86.jpg)
tugrazGraz University of Technology
Parallel computing?
The algorithm basically computes matrix-vector productsThe matrices are usually very sparse
One simple processor for each unknown
Each processor has a small amount of local memory
According to the structure of K , each processor can inter-change data with itsneighboring processors
Recent GPUs already go into this direction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
40 / 40