SUPPORT VECTOR MACHINE IN MATLAB

Weicheng Guo

PROBLEM DEFINITION

2

We classify red points (in a circle with radius of 1) as 1 and green points (in a ring with radius from 1 to 2) as -1 in this figure. We want to find the hyperplane separating these two classes. Our decision boundary will have equation:

𝑤𝑤𝑇𝑇x+b=0

FINDING THE DECISION BOUNDARY

3

Let {x1,…, xn} be the training data set and let yi∈{1,-1} be the class label of xi

The decision boundary should classify all points correctly with

The decision boundary can be found by solving the following constrained optimization problem

𝑦𝑦𝑖𝑖 𝑤𝑤𝑇𝑇𝑥𝑥𝑖𝑖 + 𝑏𝑏 ≥ 1 − 𝜉𝜉𝑖𝑖

min  12‖𝑤𝑤‖2 + 𝐶𝐶�𝜉𝜉𝑖𝑖

𝑛𝑛

𝑖𝑖=1𝑠𝑠𝑠𝑠𝑏𝑏𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠    𝑠𝑠𝑡𝑡    𝑦𝑦𝑖𝑖 𝑤𝑤𝑇𝑇𝑥𝑥𝑖𝑖 + 𝑏𝑏 ≥ 1 − 𝜉𝜉𝑖𝑖

REFORMULATING AS A LAGRANGIAN

4

We can introduce Lagrange multipliers to represent the condition and thus have the following formulation:

j1 1 1

1

1max x x2

subject to 0 , 0

n n nT

i i j i j ii i j

n

i i ii

y y

C y

α αα

α α

= = =

=

−

≤ ≤ =

∑ ∑∑

∑

KERNEL TRICK

5

Since it is a nonlinear classification in this case, we can use kernel functions that operates on the lower dimension vectors xi and xj to produce a value equivalent to the dot product of the higher-dimensional vectors. Therefore, the data can be linearly separated in that space and the formulation is transformed to:

1 1 1

1

1max ( , )2

subject to 0 , 0

n n n

i i j i j i ji i j

n

i i ii

y y K x x

C y

α αα

α α

= = =

=

−

≤ ≤ =

∑ ∑∑

∑

KERNEL FUNCTIONS

6

Linear kernel

Polynomial kernel with γ, d and r

Radial basis function kernel with γ

Sigmoid kernel with γ and r

𝐾𝐾(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗) = 𝑥𝑥𝑖𝑖𝑇𝑇𝑥𝑥𝑗𝑗

𝐾𝐾(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗) = 𝛾𝛾𝑥𝑥𝑖𝑖𝑇𝑇𝑥𝑥𝑗𝑗 + r 𝑑𝑑

𝐾𝐾 𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗 = exp(−𝛾𝛾||𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑗𝑗||2)

𝐾𝐾 𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗 = tanh(𝛾𝛾𝑥𝑥𝑖𝑖𝑇𝑇𝑥𝑥𝑗𝑗 + r)

LIBSVM -- A LIBRARY FOR SUPPORT VECTOR

7

LIBSVM[1] is an open source machine learning library developed at the National Taiwan University and written in C++ though with a C API. LIBSVM implements the SMO algorithm for kernelized support vector machines (SVMs), supporting classification and regression. [1] https://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html

Differences between LIBSVM and SVM function in Matlab

LIBSVM SVM function in Matlab

Classification or regression Both Classification

Classification type Two-class and multi-class Two-class

SVM formulations C-SVC, nu-SVC, one-class SVM, epsilon-SVR, nu-SVR C-SVC

LIBSVM -- A LIBRARY FOR SUPPORT VECTOR

8

PROCEDURE 1. Transform data to the format of an SVM package

2. Conduct simple scaling on the data

3. Consider the RBF kernel K(𝑥𝑥𝑖𝑖,𝑥𝑥𝑗𝑗) = exp(−γ| 𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑗𝑗 |2)

4. Find the best parameter C and γ

5. Use the best parameter C and γ to train the whole training data

6. Test

TRAINING RESULT

9

GAMMA=0.1

TRAINING RESULT

10

GAMMA=3

TRAINING RESULT

11

GAMMA=50

TESTING RESULT

12

GAMMA=3

TESTING RESULT

13

GAMMA=3

CONCLUSION

14

SVM is a new method of machine learning based on statistics theory. In contrast to ‘black box’ learning approaches (artificial neural network), SVM is supported by certain mathematical models.

The training of SVM is relatively easy. Unlike in neural network, SVM can get global optimum and the training time does not depend on dimensionality of feature space any more by using the kernel trick.

The key to training SVM is to select a kernel function and its parameters, which cannot be conducted by a principled manner.