MURI MeetingJuly 2002
Gert Lanckriet ([email protected])L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P.
BartlettU.C. Berkeley
Convex Optimization in Machine Learning
Convex Optimization in Machine Learning
QPLP
QCQP
SDPSOCP
Advanced Convex Optimization in Machine Learning
Advanced Convex Optimization in Machine Learning
Linear Programming (LP)
Second Order Cone Programming (SOCP)
Semi-Definite Programming
Advanced Convex Optimization in Machine Learning
MPM: Problem Sketch (1)
aT z = b : decision hyperplane
MPM: Problem Sketch (2)
MPM: Problem Sketch (3)
Probability of misclassification…
… for worst-case class-conditional density…
… should be minimized !
MPM: Main Result (5)
MPM: GeometricInterpretation
Robustness to Estimation Errors: Robust MPM (R-MPM)
Robust MPM (R-MPM)
Robust MPM (R-MPM)
MPM: Convex Optimization to solve the problem
Linear Classifier
Nonlinear Classifier Kernelizing
Convex Optimization:Second OrderCone Program (SOCP)
) competitive with Quadratic Program
(QP) SVMs
LemmLemmaa
MPM: Empirical results=1– and TSA (test-set accuracy) of the MPM, compared to BPB (best performance in Breiman's report (Arcing classifiers, 1996)) and SVMs. (averages for 50 random partitions into 90% training and 10% test sets)
• Comparable with existing literature, SVMs• =1- is indeed smaller than the test-set accuracy in all cases (consistent with as worst-case bound on probability of misclassification)• Kernelizing leads to more powerfull decision boundaries (linear
decision boundary < nonlinear decision boundary (Gaussian kernel))
Advanced Convex Optimization in Machine Learning
The idea (1)
Machine learning
Kernel-based machine learning
The idea (2)
training set (labelled)
test set (unlabelled)
The idea (4)
Hard margin SVM classifiers (3)
Hard margin SVM classifiers (4)
SDP !
Hard margin SVM classifiers (5)
training set (labelled)
test set (unlabelled)
Learning the kernel matrix !
Hard margin SVM classifiers (7)
?
Hard margin SVM classifiers (8)
Hard margin SVM classifiers (11)
Learning Kernel Matrix with SDP !
Empirical results hard margin SVMs
See also
