muri meeting july 2002 gert lanckriet ( [email protected] ) [email protected] l. el...

30
MURI Meeting July 2002 Gert Lanckriet ([email protected]. edu ) L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett U.C. Berkeley Convex Optimization in Machine Learning

Post on 20-Dec-2015

217 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MURI MeetingJuly 2002

Gert Lanckriet ([email protected])L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P.

BartlettU.C. Berkeley

Convex Optimization in Machine Learning

Page 2: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Convex Optimization in Machine Learning

Page 3: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

QPLP

QCQP

SDPSOCP

Advanced Convex Optimization in Machine Learning

Page 4: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Advanced Convex Optimization in Machine Learning

Page 5: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Linear Programming (LP)

Page 6: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Second Order Cone Programming (SOCP)

Page 7: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Semi-Definite Programming

Page 8: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Advanced Convex Optimization in Machine Learning

Page 9: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: Problem Sketch (1)

aT z = b : decision hyperplane

Page 10: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: Problem Sketch (2)

Page 11: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: Problem Sketch (3)

Probability of misclassification…

… for worst-case class-conditional density…

… should be minimized !

Page 12: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: Main Result (5)

Page 13: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: GeometricInterpretation

Page 14: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Robustness to Estimation Errors: Robust MPM (R-MPM)

Page 15: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Robust MPM (R-MPM)

Page 16: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Robust MPM (R-MPM)

Page 17: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: Convex Optimization to solve the problem

Linear Classifier

Nonlinear Classifier Kernelizing

Convex Optimization:Second OrderCone Program (SOCP)

) competitive with Quadratic Program

(QP) SVMs

LemmLemmaa

Page 18: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

MPM: Empirical results=1– and TSA (test-set accuracy) of the MPM, compared to BPB (best performance in Breiman's report (Arcing classifiers, 1996)) and SVMs. (averages for 50 random partitions into 90% training and 10% test sets)

• Comparable with existing literature, SVMs• =1- is indeed smaller than the test-set accuracy in all cases (consistent with as worst-case bound on probability of misclassification)• Kernelizing leads to more powerfull decision boundaries (linear

decision boundary < nonlinear decision boundary (Gaussian kernel))

Page 19: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Advanced Convex Optimization in Machine Learning

Page 20: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

The idea (1)

Machine learning

Kernel-based machine learning

Page 21: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

The idea (2)

Page 22: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

training set (labelled)

test set (unlabelled)

The idea (4)

Page 23: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Hard margin SVM classifiers (3)

Page 24: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Hard margin SVM classifiers (4)

Page 25: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

SDP !

Hard margin SVM classifiers (5)

Page 26: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

training set (labelled)

test set (unlabelled)

Learning the kernel matrix !

Hard margin SVM classifiers (7)

Page 27: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

?

Hard margin SVM classifiers (8)

Page 28: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Hard margin SVM classifiers (11)

Learning Kernel Matrix with SDP !

Page 29: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

Empirical results hard margin SVMs

Page 30: MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett

See also