Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization and Image Processing

Erin E. Tripp

Syracuse University

November 2018

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Table of Contents

1 Overview

2 Image Denoising

3 Foreground Detection

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Image Denoising

We receive a blurry, noisy image:

and we want to recover the original.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Image Denoising

For example, even a visually imperceptible amount of noise cancause a neural net to misclassify the image with highprobability. 1

1From Explaining and Harnessing Adversarial Examples by Goodfellow, Shlens, and Szegedy.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Foreground Detection

We want to separate moving objects from the nonmovingbackground in video data:

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization

How are these optimization problems?

First, we develop a mathematical model for our problem.

This consists of:

a function C (x) that measures the quantity of interest(variations in pixel values, similarity to input, etc)a constraint set S which restricts the acceptable answers(matrices satisfying certain inequalities, etc.)

So the problem becomes

arg minx∈S

C (x)

We can then use a variety of algorithms for findingminimizers of functions.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization

How are these optimization problems?

First, we develop a mathematical model for our problem.

This consists of:

a function C (x) that measures the quantity of interest(variations in pixel values, similarity to input, etc)a constraint set S which restricts the acceptable answers(matrices satisfying certain inequalities, etc.)

So the problem becomes

arg minx∈S

C (x)

We can then use a variety of algorithms for findingminimizers of functions.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization

How are these optimization problems?

First, we develop a mathematical model for our problem.

This consists of:

a function C (x) that measures the quantity of interest(variations in pixel values, similarity to input, etc)

a constraint set S which restricts the acceptable answers(matrices satisfying certain inequalities, etc.)

So the problem becomes

arg minx∈S

C (x)

We can then use a variety of algorithms for findingminimizers of functions.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization

How are these optimization problems?

First, we develop a mathematical model for our problem.

This consists of:

a function C (x) that measures the quantity of interest(variations in pixel values, similarity to input, etc)a constraint set S which restricts the acceptable answers(matrices satisfying certain inequalities, etc.)

So the problem becomes

arg minx∈S

C (x)

We can then use a variety of algorithms for findingminimizers of functions.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization

How are these optimization problems?

First, we develop a mathematical model for our problem.

This consists of:

a function C (x) that measures the quantity of interest(variations in pixel values, similarity to input, etc)a constraint set S which restricts the acceptable answers(matrices satisfying certain inequalities, etc.)

So the problem becomes

arg minx∈S

C (x)

We can then use a variety of algorithms for findingminimizers of functions.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Optimization

How are these optimization problems?

First, we develop a mathematical model for our problem.

This consists of:

a function C (x) that measures the quantity of interest(variations in pixel values, similarity to input, etc)a constraint set S which restricts the acceptable answers(matrices satisfying certain inequalities, etc.)

So the problem becomes

arg minx∈S

C (x)

We can then use a variety of algorithms for findingminimizers of functions.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing image data

How do we turn an image into a mathematical object?

The image becomes a matrix, and the entries of the matrixare the pixel values, ranging from 0 (black) to 255 (white).For example, 0 10 83

25 103 15011 60 98

represents a 3× 3 pixel grayscale image.

The cameraman example is 256× 256 (65536) pixels.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing image data

How do we turn an image into a mathematical object?

The image becomes a matrix, and the entries of the matrixare the pixel values, ranging from 0 (black) to 255 (white).For example, 0 10 83

25 103 15011 60 98

represents a 3× 3 pixel grayscale image.

The cameraman example is 256× 256 (65536) pixels.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing image data

How do we turn an image into a mathematical object?

The image becomes a matrix, and the entries of the matrixare the pixel values, ranging from 0 (black) to 255 (white).For example, 0 10 83

25 103 15011 60 98

represents a 3× 3 pixel grayscale image.

The cameraman example is 256× 256 (65536) pixels.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing video data

For video footage, we want to deal with several frames(individual images) at once.

Each frame is entered as a matrix then transformed into acolumn:

0 10 8325 103 15011 60 98

−→

0251110

1036083

15088

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing video data

For video footage, we want to deal with several frames(individual images) at once.

Each frame is entered as a matrix then transformed into acolumn:

0 10 8325 103 15011 60 98

−→

0251110

1036083

15088

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing video data

0 10 8325 103 15011 60 98

20 100 01 0 270 12 201

17 0 55234 60 0

0 106 111

frame 1 frame 2 frame 3

Multiple frames are then concatenated:

0 20 1725 1 23411 0 010 100 0

103 0 6060 12 10683 0 55

150 27 098 201 111

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing video data

In our video example, each frame is 120× 160 (19200)pixels. This becomes a 19200× 1 column.

If we use 200 frames, our video matrix will be a19200× 200 matrix!

Note: More frames = more accurate foregroundseparation, but also more computations.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing video data

In our video example, each frame is 120× 160 (19200)pixels. This becomes a 19200× 1 column.

If we use 200 frames, our video matrix will be a19200× 200 matrix!

Note: More frames = more accurate foregroundseparation, but also more computations.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Processing video data

In our video example, each frame is 120× 160 (19200)pixels. This becomes a 19200× 1 column.

If we use 200 frames, our video matrix will be a19200× 200 matrix!

Note: More frames = more accurate foregroundseparation, but also more computations.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Table of Contents

1 Overview

2 Image Denoising

3 Foreground Detection

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How do we represent image denoising as an optimizationproblem?

arg min f (Y ) +1

2λ‖X − Z‖2F

subject to Y = DX

Z is the noisy input imageD computes the difference between adjacent pixelsf penalizes small differences in the data

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How do we represent image denoising as an optimizationproblem?

arg min f (Y ) +1

2λ‖X − Z‖2F

subject to Y = DX

Z is the noisy input imageD computes the difference between adjacent pixelsf penalizes small differences in the data

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How do we represent image denoising as an optimizationproblem?

arg min f (Y ) +1

2λ‖X − Z‖2F

subject to Y = DX

Z is the noisy input image

D computes the difference between adjacent pixelsf penalizes small differences in the data

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How do we represent image denoising as an optimizationproblem?

arg min f (Y ) +1

2λ‖X − Z‖2F

subject to Y = DX

Z is the noisy input imageD computes the difference between adjacent pixels

f penalizes small differences in the data

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How do we represent image denoising as an optimizationproblem?

arg min f (Y ) +1

2λ‖X − Z‖2F

subject to Y = DX

Z is the noisy input imageD computes the difference between adjacent pixelsf penalizes small differences in the data

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Algorithm

We use the Alternating Direction Method of Multipliers.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Algorithm

We use the Alternating Direction Method of Multipliers.

First write the augmented Lagrangian Lη(X ,Y ,P) for thisproblem:

f (Y ) +1

2λ‖X − Z‖2F − 〈P,DX − Y 〉+

η

2‖DX − Y ‖2F

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Algorithm

We use the Alternating Direction Method of Multipliers.

First write the augmented Lagrangian Lη(X ,Y ,P) for thisproblem:

f (Y ) +1

2λ‖X − Z‖2F − 〈P,DX − Y 〉+

η

2‖DX − Y ‖2F

ADMM:

Input X0,Y0

For k = 1, . . . ,N

Yk+1 = arg minY

Lη(Xk ,Y ,Pk)

Xk+1 = arg minX

Lη(X ,Yk+1,Pk)

Pk+1 = Pk − η(DXk+1 − Yk+1)

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Algorithm

We use the Alternating Direction Method of Multipliers.

First write the augmented Lagrangian Lη(X ,Y ,P) for thisproblem:

f (Y ) +1

2λ‖X − Z‖2F − 〈P,DX − Y 〉+

η

2‖DX − Y ‖2F

ADMM:

Input X0,Y0

For k = 1, . . . ,N

Yk+1 = arg minY

Lη(Xk ,Y ,Pk)

Xk+1 = arg minX

Lη(X ,Yk+1,Pk)

Pk+1 = Pk − η(DXk+1 − Yk+1)

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Results

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Table of Contents

1 Overview

2 Image Denoising

3 Foreground Detection

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How can we model foreground detection as an optimizationproblem?

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How can we model foreground detection as an optimizationproblem?Let Y be the video matrix. We assume Y can be decomposedas Y = M + S , where

M is a low rank matrix (the stable background)

S is a sparse matrix (the moving foreground)

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How can we model foreground detection as an optimizationproblem?Let Y be the video matrix. We assume Y can be decomposedas Y = M + S , where

M is a low rank matrix (the stable background)

S is a sparse matrix (the moving foreground)

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

How can we model foreground detection as an optimizationproblem?Let Y be the video matrix. We assume Y can be decomposedas Y = M + S , where

M is a low rank matrix (the stable background)

S is a sparse matrix (the moving foreground)

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

Since we don’t know M and S we want to find approximationsM̃ and S̃ .

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

Since we don’t know M and S we want to find approximationsM̃ and S̃ .

Then the problem can be rephrased as, “How close is ourapproximation to the actual solution?”

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Modeling the Problem

Since we don’t know M and S we want to find approximationsM̃ and S̃ .

Then the problem can be rephrased as, “How close is ourapproximation to the actual solution?”

Mathematically, we want to minimize

‖M̃ + S̃ − Y ‖2F

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Sketch of Algorithm

Projected gradient descent:

Create an initial estimate S0 of S and set M0 = Y − S0.

Factor M0 = U0V>0

Perform gradient descent on the function

‖UV> + S − Y ‖2F

with respect to U and V respectively, projecting each stepback onto the constraint sets. Update S accordingly.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Sketch of Algorithm

Projected gradient descent:

Create an initial estimate S0 of S and set M0 = Y − S0.

Factor M0 = U0V>0

Perform gradient descent on the function

‖UV> + S − Y ‖2F

with respect to U and V respectively, projecting each stepback onto the constraint sets. Update S accordingly.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Sketch of Algorithm

Projected gradient descent:

Create an initial estimate S0 of S and set M0 = Y − S0.

Factor M0 = U0V>0

Perform gradient descent on the function

‖UV> + S − Y ‖2F

with respect to U and V respectively, projecting each stepback onto the constraint sets. Update S accordingly.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Sketch of Algorithm

Projected gradient descent:

Create an initial estimate S0 of S and set M0 = Y − S0.

Factor M0 = U0V>0

Perform gradient descent on the function

‖UV> + S − Y ‖2F

with respect to U and V respectively, projecting each stepback onto the constraint sets. Update S accordingly.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

Results

The original frame; the foreground; and the background.

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection

References

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein,Distributed Optimization and Statistical Learning via theAlternating Direction Method of Multipliers, Foundations andTrends in Machine Learning, Vol. 3, No. 1 (2010) 1–122.

X. Yi, D. Park, Y. Chen, and C. Caramanis, Fast Algorithms forRobust PCA via Gradient Descent, Preprint, arXiv:1605.07784v1 [cs.IT] (2016).

Optimizationand ImageProcessing

Erin E. Tripp

Overview

ImageDenoising

ForegroundDetection Thank you!