model correction using a nuclear norm constraint · model correction using a nuclear norm...

Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work

Model Correction using a NuclearNorm Constraint

Ning Hao, Lior Horesh, Misha Kilmer

Ragon InstituteIBM Thomas J. Watson Research Center

Tufts UniversityThanks: NSF:CIF:SMALL 1319653, NSF-DMS 0914957

Householder Symposium XIX, June 2014

Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer


MotivationWe create mathematical models to simulate the operation of areal world or system.

Not surprisingly, the models will contain some error:

• simplifications for computational tractability

• linearization, numerical error

• boundary conditions or geometry








“All models are wrong . . . but some are useful “George E. P. Box, 1987








“All models are wrong . . . but some are useful “George E. P. Box, 1987

Thinking outside the Box, some models can be made lesswrong / more useful. This is the subject of today’s talk.



Problem DefinitionLet F : Rn → Rm be “true” operator taking input x ∈ Rn toobservable space. Let d ∈ Rm be an observation obtained as

d = F(x) + ε

where ε represents measurement noise.

Goal: Formulate and solve a model correction optimizationproblem to recover an F that is only partially specified.In this talk, assume A is known and

F(x) = A(x) +B(x).

Model correction problem is to recover B. Need additionalconstraints on B to obtain a well-posed optimization problem.



Motivating Test ProblemDiscrete ill-posed problem (e.g. deblurring, Xray-CT)

Ax = d+ ε.

If operator not known exactly model should be replaced by

(A+B)x = d+ ε.

While B is unknown, and in general x is unknown, we assumewe can sample input space and have access to correspondingoutput realizations for each of the samples.



Sample Average ApproximationSAA[1]1: Expectation wrt the input space & the measurementnoise approximated by a sample ave. est. of a random sample.

B̂ = argminB

1

nxnε

nx,nε∑j=1,j=1

D ((A(xi) +B(xi))− dij)

subject to constraint on B, nx is the number of input draws,nε is number of data realizations for each input realization.

Problem (A(x) = Ax,B(x) = Bx,D(y) = ‖y‖22)

B̂ = argminB

1

nxnε

nx,nε∑i=1,j=1

‖(A+B)xi − di,j‖22,

subject to a structural constraint on B .1[1]Shapiro, Dentcheva, Ruszczynski, editors. “Lecture Notes on

Stochastic Programming: Modeling and Theory.” SIAM, 2009.Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer


Restrictions on Operator StructureMany constraints on B possible. Choose rank-based:

Problem

B̂ =argminB‖(A+B)X −D‖2F

s.t. rank(B) ≤ δ

2

Relax the constraint:

Problem

B̂ =argminB||(A+B)X −D||2F

s.t. ‖B‖∗ ≤δ

2



Optimization Problem with Nuclear NormRegularization

Several options for solving the optimization problem[1]2

• Interior Point Methods

• Alternating Direction Method of Multipliers (ADMM)

• Projected Sub-Gradient Method

• More recent work by Jaggi and Sulovsky: recast theoptimization problem over positive semidefinite matriceswith unit trace and then apply Hazan’s algorithm.

2[1]Hao, Horesh, & K., “Nuclear norm optimization and its applicationto observation model specification,” in Compressed Sensing and SparseFiltering, 2014



Hazan’s Algorithm1

Hazan’s Algorithm deals with problems of the form:

minZ∈S

f(Z)

where f is convex and S is the set of all symmetric positivesemidefinite d× d matrices with unit trace.

Each iteration involves the calculation of a single approximateeigenvector of a matrix of size of d× d.

1Sparse approximation solutions to semidefinite programs by Hazan,E.2008



AlgorithmFrom Jaggi and Sulovsky “A Simple Algorithm for NuclearNorm Regularized Problem” 2010, for any non-zero matrixB ∈ Rn×m and δ ∈ R:

‖B‖∗ ≤δ

2⇐⇒ ∃ symmetric M,Ns.t.(M BBT N

)� 0

andTr(M) + Tr(N) = δ.



Steps

Let Z =

(M BBT N

).

• Define f̂(Z) = f(B) = ‖(A+B)X −D‖2F ,

• Want to solveminZf̂(Z)

s.t. Z ∈ S(m+n)×(m+n), Z � 0,Tr(Z) = δ

• Scale all matrix entries by 1δ

• Apply Hazan’s algorithm, then unscale



AlgorithmInput: f , v0 ∈ R(m+n)×1 with ||v0|| = 1; scaled matricesInitialize: Z1 = v0v

T0

for k = 1 until convergence doExtract Bk = Zk(1 : m,m+1:m+n)Compute ∇fk := ((A+Bk)X −D)XT

We have ∇f̂k :=(

0 ∇fk∇fTk 0

)Compute vk := eigs(−∇f̂k, 1,′ LA′)Line search for step length akUpdate Zk+1 = Zk + ak(vkv

Tk − Zk)

end forReturn B̂ = δ · Z(1 :m,m+1:m+n)



Numerical Examples

• Mimic the semi-blind deconvolution problem, only anapproximation to blurring operator is known a-priori.

• 100 MR images from the Auckland MRI Research groupdatabase. 3

• Pre-process the images by cropping the watermark andresizing the cropped images to the size of 55× 55.

• Randomly choose 80 images (sampled integers from1:100 uniformly) to serve as the training set and 20 ofremainder as the test set.

3http://atlas.scmr.org/download.htmlModel Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer


Example 1

• Let T (bw, σ) be symmetric, doubly block Toeplitz matrixT that models blurring of an image by a Gaussian pointspread function; bw, σ control (block) bandwidth and blur.

• A = T (4, 4)

• B defined from singular triples of T (3, 2) numbered 110to 120.

• Compute D = (A+B)X +N where N ’s columns havezero mean, white Gaussian noise such that the noise tosignal ratio for all measured data is .5 percent.

• Run 120 iterations. Compare the best TSVD regularizedsolutions using the SVDs of A,A+B,A+ B̂.



True Blurred Training Images



AX vs (A +B)X



Rel. Errors, TSVD recons, 20 from test set



Example 1, Test Slice 16



Rel. Errors after 400 iterations



Enforcing Different StructureWhat if B(x) 6= Bx, but rather B(x) = K[B]x?

Example [Kamm & Nagy, 1998] Let v be of length n, n odd,

toep(v) creates an n× n banded Toeplitz matrix with v(n+12)

as the diagonal entry:

toep

abc

=

b a 0c b a0 c b



Additional Structure from B

Given n× n B, make a BTTB matrix from blocks toepB(:, i)in a similar manner.

For B with 3 columns:

K[B] =

toep(B(:, 2)) toep(B(:, 1)) 0toep(B(:, 3)) toep(B(:, 2)) toep(B(:, 1))

0 toep(B(:, 3)) toep(B(:, 2))

.



Convexity, Constraint RevisitedIt is possible to show thatK[B]x = K[reshape(x, n, n)]vec(B).

So,∑‖Axij +B(xij)− dij‖2F

still convex in entries of B

Significance of the (ideal) rank-based constraint on B:

B =k∑i=1

σiuivTi ⇒ K[B] =

k∑i=1

toep(√σivi)⊗ toep(

√σiui),

Algorithm now cheap - matrix of partials is now n× n,matvecs with FFTs.



Example 2

• A = T (10, 4)

• B = T (7, 3.1) + T (4, 2.2) + T (5, 3.5)

• Same testing setup as before (random selection of 80images) and testing on the remainder, Gaussian 1

2percent

noise added (once per test image).



Relative Squared Convergence Error



Difference in Data Space: |BXtrain|



Difference in Data Space: |(B − B̂)Xtrain|



Errors on 20 Test Data

‖AX−(A+B)X‖F‖(A+B)X‖F

vs. ‖(B̂−B)X‖F‖(A+B)X‖F



One Extension to TensorsB is order > 2 tensor from which we define K[B](x).

B ∈ Rn×n×n, B =∑r

i=1 ui ◦ vi ◦ wi ⇒ K[B] to a sum of3-way Kronecker products of structured matrices

• Nagy & K., “Kronecker Product Approximation forThree-Dimensional Imaging Applications, IEEE TIP 2006.

• Rezghi & Elden, “A Kronecker Product Approximation ofthe Blurring Operator in the Three Dimensional ImageRestoration Problem,” SIMAX to appear.

But what do we mean by ‖B‖∗?









More thinking outside the Box....









Use ‖B‖TNN definition [Hao, 2014] based on tSVD [K. &Martin, 2011].



Current & Future WorkOther examples (PDEs) (thanks: Raya Horesh, IBM Watson)



Current & Future Work

• Tensors

• Other examples (PDES)

• Other choices for D• Investigate other possible constraints on the operator

(nonlinear correction).


model correction using a nuclear norm constraint · model correction using a nuclear norm...

Documents