1

Mixing Linear SVMs for Nonlinear ClassificationZhouyu Fu, Antonio Robles-Kelly, Senior Member, IEEE and Jun Zhou, Member, IEEE

Abstract—In this paper, we address the problem of combininglinear Support Vector Machines (SVMs) for classification oflarge-scale non-linear data sets. The motivation is to exploitboth the efficiency of linear SVMs in learning and predictionand the power of non-linear SVMs in classification. To thisend, we develop a linear SVM mixture model that exploits adivide-and-conquer strategy by partitioning the feature spaceinto sub-regions of linearly separable data-points and learninga linear SVM for each of these regions. We do this implicitlyby deriving a generative model over the joint data and labeldistributions. Consequently, we can impose priors on the mixingcoefficients and do implicit model selection in a top-down mannerduring the parameter estimation process. This guarantees thesparsity of the learned model. Experimental results show that theproposed method can achieve the efficiency of linear SVMs in theprediction phase while still providing a classification performancecomparable to non-linear SVMs.

Index Terms—EM Algorithm, Classification, Mixture of Ex-perts, Model Selection, Support Vector Machines

I. INTRODUCTION

Classification is a fundamental problem in pattern analysis.It involves learning a function that separates data points fromdifferent classes. The support vector machine (SVM) classifier,which aims at recovering a maximal margin separating hyper-plane in the feature space, is a powerful tool for classificationand has demonstrated state-of-the-art performance in manyproblems [33]. SVMs can operate either explicitly in the inputspace leading to the linear SVM or implicitly in the featurespace via the kernel mapping giving rise to the kernel SVM.Linear SVMs are simple to train and use, as they only involveinner product operations with the input data. However, theycan be quite restricted in discriminative power and can notbe applied to non-linear data. This limits their applicationto many real-world problems where the data distributions areinherently non-linear. The non-linear SVM, on the other hand,can handle linearly inseparable data but is not as efficientas the linear classifier. Its complexity is multiplied with thenumber of support vectors. This is unfavourable for predictiontasks on large-scale data sets.

Hence, it is desirable to have a classifier model with both theefficiency of linear SVMs and the power of non-linear SVMs.Here, we develop a mixture model so as to combine linearSVMs for the classification of non-linear data. The proposedmixture of linear SVMs (MLSVM) is based on a divide-and-conquer strategy that partitions the input space into hyper-

Z. Fu is with the Gippsland School of Information Technology, MonashUniversity, Churchill, Victoria 3842, Australia.

A. Robles-Kelly and J. Zhou are with the National ICT Australia (NICTA)and the College and Engineering and Computer Science at the AustralianNational University, Canberra, ACT 2601, Australia.

NICTA is funded by the Australian Government as represented by theDepartment of Broadband, Communications and the Digital Economy and theAustralian Research Council through the ICT Centre of Excellence program.

spherical regions. Data in each region are linearly separableand can be handled by a linear SVM. We do this implicitlyby deriving a probabilistic model over the joint data andlabel distributions. The joint distribution model enables us toexplicitly incorporate mixture coefficients into the likelihoodfunction. Consequently, we can perform structure learningand parameter learning simultaneously by imposing a properprior on the mixture coefficients based on the minimummessage length (MML) criterion [31]. Parameter update is thenachieved by the EM algorithm [9]. For model selection, westart with an over-complete model and automatically prunethe nodes corresponding to the vanishing mixture coefficientsduring the parameter optimisation process. Hence, structurelearning is implicitly incorporated into the optimisation pro-cess and performed in a top-down manner. This is the majoradvantage of the proposed model as compared to alternativeprobabilistic models for classification, like the mixture ofexperts model by Jacobs and Jordan [20]. Moreover, unlikethe mixture of experts, which uses linear gating networkswith soft-max activation functions, we have adopted gatingnetworks with non-linear radial basis functions (RBF). Thisprovides more flexibility in the partitioning of the featurespace. It leads to a smooth partition over the feature spacethat prevents abrupt change across region boundaries.

The proposed method is quite general and can be usedin combination with other choices of linear classifiers withminor modification in the M-step of the EM algorithm aslong as they have proper probabilistic interpretations. Notethat, in this paper, we focus on the case of large samplesize with moderate feature dimension, since we are mainlyinterested in the problem of efficiency in prediction for non-linear data. High dimensional features are more likely to belinearly separable and, therefore, linear classifiers are usuallysufficient for them.

The remainder of the paper is organised as follows. SectionII provides the background on the SVM classifier model.Section III describes the details of the proposed classifiermodel, including the model formulation, parameter estimationand model selection. Experimental results are presented inSection V, followed by conclusions in Section VI.

II. THE SVM CLASSIFIER MODEL

In this paper, we focus our attention on binary classi-fication problems. Multiclass classification can be handledby converting the problem at hand into a number of binaryclassification ones through classifier fusion methods [18].Given the labelled binary data set (X,Y ) = (xi, yi)|i =1, . . . , N, yi ∈ −1, 1, the linear SVM classifier recovers anoptimal separating hyperplane wTx+b = 0 which maximisesthe margin of the classifier. This can be formulated as the

2

following constrained optimisation problem [33], [17]

minw

||w||2

2+ C

∑i

`(w;xi, yi) (1)

The first term on the right-hand-side is the regularisationterm on the classifier weights, which is related to the classifiermargin through the inverse distance between the marginalhyperplanes wTx = 1 and wTx = −1. Without loss ofgenerality, we have subsumed the bias term b in the aboveformulation by appending each instance with an additionaldimension xTi = [xTi , 1] and wT = [wT , b]. The second termon the right-hand-side is related to the classification error,where `(w;xi, yi) = max (1− yiwTxi, 0) is the Hinge lossthat upper bounds on the empirical error of the classifier.The parameter C controls the relative importance of theregularisation term and the error term.

SVMs can also be trained by solving the Lagrangian dualof Equation 1, which results:

maxα

∑i

αi −1

2

∑i,j

yiαixTi xiyjαj (2)

s.t. 0 ≤ α ≤ C and∑i

yiαi = 0 ∀i

The classifier for linear SVM is then represented by

f (x) = wTx + b (3)

=∑αi>0

αiyi〈xi,x〉+ b (4)

where w is the classifier weight vector defined by the dualvariables. The weight vector can be computed explicitly byw =

∑αi>0 αiyixi and used for prediction.

The advantage of solving the dual form is that only innerproducts between data-points are needed. Consequently, wecan derive the non-linear SVM by implicitly mapping theinput data x into the feature space and training the SVMfor the mapped features φ(x). This is achieved by the kerneltrick, where the implicit feature space is induced by the kernelfunction governed by the inner product for feature space mapsK(xi,xj) = φ(xi)

Tφ(xj). Non-linear SVMs can then betrained by replacing the inner products in Equation 2 withthe corresponding kernel K(xi,xj). The resulting classifierfor the non-linear SVM is then represented by

f (x) =∑αi>0

αiyiK(xi,x) + b (5)

where the α’s are the Lagrangian multipliers. Conceptually, theonly difference between the non-linear SVM and their linearcounterparts is in the use of kernel function instead of the innerproduct in Equation 4. Computationally, linear SVMs can bedirectly evaluated by using Equation 3, which is much moreefficient than non-linear SVMs for purposes of prediction.

Note that only those instances with positive value of αi,called support vectors, will contribute to classification. Geo-metrically, these correspond to the points lying on or outsidethe marginal hyperplanes which incur nonzero hinge losses,i.e. yif (xi) < 1 with f (x). Different kernels can be usedfor non-linear SVMs. The most common ones include Radial

Basis Functions (RBF), polynomial and the sigmoidal kernels.In this paper, we focus on the RBF kernel given below, which,in practice, is the most widely used for non-linear SVMs.

K(xi,xj) = exp (−τ ||xi − xj ||2)

where τ is the scale parameter that controls the decay rate ofthe distance.

Despite the effectiveness in classification of non-linear data,the non-linear SVM has much higher computational com-plexity for prediction than its linear counterpart. Accordingto Equation 5, it requires |V|d summations and |V| expo-nential operations to compute the kernel function values forevaluating the classifier output at a single testing instance,where d is the data dimension and |V| is the cardinality ofthe set of support vectors. Depending on the training data setsize and the degree of non-linearity, prediction can be verytime consuming. To cope with a highly non-linear decisionboundary, a great number of support vectors may be generatedin the training process, which leads to higher computationalcomplexity in testing time. Also, a larger training data set willresult in greater number of support vectors in training than asmaller data set does. This also adds up to the complexity ofprediction. Hence non-linear SVM is inefficient for large-scaleprediction tasks where there are thousands or millions of datapoints to classify in the testing set.

Linear SVMs, on the other hand, do not have such problem.They can perform prediction with d summations via Equation3 and the testing time is independent of the number of supportvectors. This renders them a much more efficient alternativethan non-linear SVMs. Moreover, there are very efficientalgorithms for training linear SVMs which can run in lineartime [21], [6]. This is impossible for non-linear SVMs, as theevaluation of the full kernel matrix is already quadratic in timecomplexity. Nevertheless, the performance of linear SVMs isusually suboptimal as compared to non-linear ones since theycan not handle linearly inseparable data. Hence it is desirableto develop an SVM model with both the efficiency of the linearSVM at the prediction stage and the classification accuracy ofthe non-linear SVM classifier.

III. MIXTURE OF LINEAR SVM MODEL

A. Model Formulation

In this section, we present our Mixture of Linear SVMsmodel (MLSVM). Without loss of generality, we illustrate ourmodel with a two-layer mixture model as shown in Figure 1(a).Generalisation to hierarchical mixtures is straightforward andwill be discussed in Section III-E. The model in Figure 1(a)consists of two parts. The hidden layer is composed of thegating network that produces a soft-partition of the input spaceby generating a data-dependent weight distribution. Each nodein the hidden layer is connected to a linear SVM classifier inthe input layer, which is responsible for the classification ofdata points residing in the corresponding partition.

The proposed mixture model captures the posterior proba-

3

Input Layer

Mixture 1

LSVM 1 LSVM K

Mixture K

Hidden Layer

(a)

ξ

z

y

β

γ

x

N

(b)

Fig. 1. MLSVM model. Left: model topology. Right: graphical modelrepresentation.

bilistic distributions of the labels Y given input data X by

P (Y |X,Θ) =∏i

P (yi|xi,Θ) (6)

=∏i

∑zi

ξziP (zi|xi, β)P (yi|zi,xi, γ)

where i indexes correspond to the data samples and Θ =ξ, β, γ are the parameters of the underlying model.

In the expression above, we have assumed that the traininginstances/labels are independent and identically distributed andzi is the hidden variable introduced for the ith instance thattakes values in 1, . . . , k, where k is the total number ofmixtures. The weight of the ith instance is determined by bothξzi and P (zi|xi, β). The data independent term ξzi specifiesthe prior probability of the mixture component indexed byzi, whereas the data dependent term P (zi|xi, β) specifies theprobability model for the gating function and is related to theimportance of the ith instance with respect to the zith mixturecomponent. P (yi|zi,xi, γ) is governed by the classificationprocess for the zith component and specifies the posteriorprobability of the ith instance output by the correspondinglinear SVM model. The parameters for the gating functionsare given by β and γ, respectively. We elaborate further ontheir specific parametric forms later in the paper.

Our MLSVM model can also be visualised by a graphicalmodel as shown in Figure 1(b). In the figure, x and y are thetarget variables over which the probability model is defined,and z is the hidden variable. The box containing x, y and zindicates i.i.d. copies of variables, where N is the number ofcopies. The arrows indicate dependencies between the vari-ables, where the variable pointed by an arrow is conditionallydependent on the variable where the arrow originates from.

The mixture model is generated from a multinomial distri-bution with parameter ξ = ξ1, . . . , ξk for k-mixture. Theposterior probability P (z|x, β) is specified by the followingequation

P (z = j|x, β) =exp

(−τ‖x− vj‖2

)∑j exp (−τ‖x− vj‖2)

(7)

The parameter β = v1, . . . ,vk specifies the centroid ofeach component. The probability for each centroid is deter-mined by the Euclidean distance between the instance andthe centroid location. The closer the distance, the larger the

probability. The assignment of instances to the componentsis probabilistic in nature, where τ is a scale parameter thatcontrols the softness of the assignment. The smaller the valueof τ , the “softer” the assignment. In the extreme case whenτ = 0, the gating function produces equal weights for allcomponents. On the other hand, when τ approaches infinity,the gating function returns 1 for the closest component and0 for the others. The probability P (y|z,x, γ) is governed bythe classifier model with parameter γ conditional on x andz, where γ = w1, . . . ,wk and wj is the parameter forthe jth linear SVM. We can clearly see the correspondencebetween the graphical model and the probability distributionin Equation 6.

It is worth noting that the proposed MLSVM model bearssome resemblance with the classical mixture of expert modelproposed by Jacobs and Jordan [20]. Nonetheless, there aretwo main differences. Firstly, our model is a mixture ofweighted experts and more general in nature. This is sincethe mixture of experts model can be treated as a special case

of ours with the mixing coefficients ξj set to1

k. The explicit

use of mixing coefficients enables us to perform implicit modelselection via enforcing a sparseness prior over ξ as we will dis-cuss in the following section. For the mixture of expert model,however, there is no way to control the model structure byenforcing constraints on the parameters and we have to resortto heuristics for explicit model selection [35], [14]. Secondly,we have used a non-linear RBF gating function in Equation 7,whereas the mixture of expert model adopted a linear functionfor the gating network. Hence our model is more flexible ingenerating the partitions over the data space than the mixtureof expert model. Note that an alternative mixture of expertmodel was proposed in [37] which also makes use of the RBFgating function by modelling the joint distributions of the dataand labels in a generative manner. Hence, it is not directlycomparable with discriminative models for classification suchas the one proposed in this paper.

Following the maximum likelihood estimation (MLE) prin-ciple, the parameter estimation of our classification modelcan be achieved by maximising the following log-likelihoodfunction

L(Θ) =∑i

logP (yi|xi,Θ) +∑j

Ω(wj) (8)

=∑i

log∑j

ξjP (zi = j|xi,vj)P (yi|xi,wj) +∑j

Ω(wj)

where Ω(wj) = log (P (wj)) specifies the log-prior term overthe SVM parameters for purpose of regularisation and P (zi =j|xi,vj) is specified in Equation 7.

In order to incorporate linear SVMs into the above MLEframework, we have to reformulate it from a probabilisticviewpoint. To this end, we view the constrained quadraticoptimisation problem in Equation 1 as that corresponding tothe negative log-likelihood of the energy function whose firstterm on the right-hand side is related to the prior Ω(w) andthe second term corresponds to the conditional probability

4

P (y|x,w) related to classification error. These are given by

Ω(w) = −ζ||w||2 (9)P (yi|xi,w) = e−`(w;xi,yi) (10)

Here we have omitted the normalisation factor for the con-ditional probability P (yi|xi,w), which leads to an approx-imation of the probability measure. This is mainly for theconvenience of numerical optimisation which enables us touse the existing fast linear SVM solvers [6] during parameterestimation. This simplification is particularly valid in thelarge margin case where the probability of misclassificationis usually very small. More importantly, from an optimisationpoint of view, we can still apply the EM algorithm to optimisethe likelihood function in Equation 8, which is guaranteed toincrease in each EM iteration regardless of whether or notP (yi|xi,wj) is a proper probability measure over yi as wewill discuss in the next section. The posterior probability isunity , i.e. zero loss, for instance xi if and only if it has amargin greater or equal to unity, i.e. yi(wTxi + b) ≥ 1.

B. The EM Algorithm

Now, we turn our attention to the use of the EM algorithmfor solving the mixture of linear SVM model presented in theprevious section. The E-step updates the posterior probabilitiesof assigning each instance to the component classifiers. LetΘ(t) = θ(t)j j=1,...,k = ξ(t)j ,v

(t)j ,w

(t)j |j = 1, . . . , k be the

parameters at the current iteration, the probability of assigningthe ith instance to the jth classifier at iteration t is

q(t)i,j =

ξ(t)j P (zi = j|xi,v(t)

j )P (yi|xi,w(t)j )∑

j ξ(t)j P (zi = j|xi,v(t)

j )P (yi|xi,w(t)j )

(11)

where P (zi = j|xi,v(t)j ) and P (yi|xi,w(t)

j ) are posteriorprobabilities associated with the gating networks and localexperts at iteration t given by Equations 7 and 10, respectively.

The M-step involves simultaneously updating the parame-ters for the gating functions and SVM classifiers. This involvessolving two independent optimisation problems. For the gatingfunctions, the values of the mixing coefficients ξj at iterationt+ 1 can be obtained in closed form

ξ(t+1)j =

∑i q

(t)i,j

N(12)

Estimation of the centroid parameters for the gating functionsis equivalent to unconstrained optimisation. Specifically, forthe jth mixture component, the centroid vj is updated bysolving the following optimisation problem

v(t)j = arg min

vj

f(vj) (13)

= arg minvj

−∑i,j

q(t)i,j logP (zi = j|xi,vj)

where P (zi = j|xi,vj) is defined in Equation 7. The aboveproblem is very similar to parameter estimation for logisticregression and can be tackled by widely available numericalroutines for unconstrained optimisation [28]. Here, we adopt alimited memory BFGS (L-BFGS) method suited for large scale

optimisation [27] to minimise the cost function in Equation 13.The L-BFGS method only requires the gradient informationof the cost function for the past few iterations, which can becalculated by the equation below

∇vjf(vj) = 2β

∑i

(q(t)i,j − P (zi = j|xi,v(t)

j ))(v(t)j − xi)

(14)Furthermore, it is worth noting that we do not have to solve theoptimisation problem associated with the parameters v

(t)j in

Equation 12 for each iteration. We can initialise the L-BFGSmethod for each iteration with the values v

(t−1)j , since the

parameters change slowly across iterations. This can greatlyreduce the time for parameter estimation and speed up thetraining process.

Parameter estimation for the linear SVMs reduces to updat-ing the classifiers for reweighted samples where the weightsare specified by the posterior probabilities computed in theE-step. Specifically, for the jth linear classifier, we solve thefollowing classification problem

max

∑i

q(t)i,j logP (yi|xi, θ(t)j ) + logP (θ

(t)j )

= max

−∑i

q(t)i,j `(w

(t)j ;xi, yi)− ζ||w(t)

j ||2

(15)

This is exactly the same problem as the training of linearSVMs in Equation 1 with sample weights given by q

(t)i,j

and C =1

2ζ. Again, for every iteration, we can train the

weighted linear SVMs in incremental manner via updatingthe results from the previous iteration. Since we have useda dual coordinate descent method for solving linear SVMs,we have retained the dual variables as well as status variablesfor each linear SVM in each iteration for their use by thecorresponding linear SVM for the next iteration. This can bedone via trivial adaption of the original training algorithm andcan greatly reduce the time spent for linear SVM training toa great extent.

Each EM iteration increases the log-likelihood given byEquation 8. This argument can be easily established by makinguse of the auxiliary function parameterised with respect to θ(t)

given by

Q(Θ(t); Θ(t−1)) =∑i,j

q(t)i,j log ξjP (zi = j|xi,v(t)

j )P (yi|xi,w(t)j )

−∑i,j

q(t−1)i,j log q

(t−1)i,j +

∑j

Ω(w(t−1)j ) (16)

which is the lower bound of L(Θ(t)) since

L(Θ(t))−Q(Θ(t); Θ(t−1)) = q(t−1)i,j log

q(t−1)i,j

q(t)i,j

(17)

The gap is non-negative and varnishes if and only if θ(t) =θ(t−1). Hence the log-likelihood increases with the followingrelation

g(Θ(t)) ≥ Q(Θ(t); Θ(t−1)) ≥ Q(Θ(t−1),Θ(t−1)) = g(Θ(t−1))

5

The second inequality is true due to the maximisation step.Therefore, by repeating the EM steps in an interleaved manner,we can obtain a convergent solution of the original maximumlikelihood estimation problem.

C. Model Selection

Model selection is a challenging problem for learning mix-ture models, which involves estimating the number of mixturecomponents. The dominant approach for doing so relies onoptimising certain criteria via a search over a range of numberof components k ∈ [kmin, kmax] such that

k = arg mink

− logP (X|Θk) +R(k)

where we have written Θk so as to denote the estimate forthe candidate value Θ when the number of components isset to k and R(k) is a function that penalises large valuesof k. The first term on the right-hand side is related toMLE, whereas the second term is related to model complexity.Examples of criteria for complexity that have been used formixtures include Bayesian inference criterion (BIC), minimumdescription length (MDL), minimum message length (MML)and so on [31].

One major disadvantage of the above deterministic ap-proach is that parameter estimation and model evaluationare performed independently, and one may have to do anexhaustive search over a large interval of k and run separateparameter estimation process for each of them to obtain anoptimal estimate. To sidestep this difficulty, we adopt analternative approach that couples parameter estimation andmodel selection in a unified framework. We start with anover-complete model with a large number of components andimplicitly select the components in a top-down manner. Thisis achieved by enforcing priors on the mixing coefficientsξj , where the components with vanishing coefficients areannihilated automatically in the training process. Following[11], we adopt the Dirichlet-type prior for ξj which is givenby

P (ξ)∞ exp

−ν k∑j=1

log ξj

(18)

where ν is a trade-off parameter for the balance between thedata and the complexity terms. Incorporating it into the log-likelihood function in Equation 8 leads to the following overalllog-likelihood

Γ(Θ) = L(Θ) + logP (ξ) = L(Θ)− ν∑j

log ξj (19)

The advantages of using the above Dirichlet-type prior aretwo-fold. Firstly, it enforces sparseness over ξ and hencepenalises complex models with many components. Secondly,it is the conjugate prior of multinomial random variables.Therefore ξ can be computed in the following closed form

ξj =max (0,

∑i q

(t)i,j − ν)∑

j max (0,∑i q

(t)i,j − ν)

(20)

Input: Labelled sample xi, yi|i = 1, . . . , N and kmax

Output: Parameter Θ = ξj ,vj ,wj |j = 1, . . . , k

• Initialise Θ for k = kmax.• E-step: calculate qi,j via Equation 11.• M-step:

– update ξj’s via Equation 20.– update vj via Equation 13 for all j such that ξj 6= 0.– update classifiers wj’s for all j such that ξj 6= 0 by

re-training linear SVM in Equation 15.• Repeat the above two steps until convergence.

Fig. 2. Algorithm for our Mixture of Linear SVMs

It is worth noting that here the only difference from the EMalgorithm developed in the previous section is in the estimationof ξj ∈ ξ.

D. Algorithm and Implementation

We now present the step-sequence of the MLSVM algorithmin Figure 2. We start with a relatively large kmax and applycomponent elimination in the parameter estimation process.K-means clustering is used to initialise EM. Specifically, wedivide all training instances into k clusters and apply a linearSVM to instances from each cluster. The cluster centroidsare initial values of v’s for the gating function in Equation7 and initial SVM parameters yielded by the linear SVMapplied to each cluster. In the case that any cluster onlycontains instances from a single class, one can initialise thecorresponding SVM weight to ed+1 for clusters containingonly positive instances and −ed+1 for those clusters comprisedof negative instances, where ed+1 is a d+1-dimensional vectorwhose entry indexed d+1 is unity and 0 for all others. In eachM step, the SVM classifiers are updated from the weights fromthe previous iteration. This is much faster than training fromrandom initialisation since, in practice, the classifiers do notchange abruptly between subsequent iterations.

There are two free parameters to be tuned in our model,the regularisation parameter C for the linear SVM, and theregularisation parameter ν for the prior term over ξ. Thesetwo parameters control the complexity of the linear SVM aswell as the overall mixture model and can be chosen via crossvalidation.

In the testing stage, a novel testing data-point is classifiedby the weighted average over the classifier mixture as givenby the following discriminant function∑ξj 6=0

ξjP (zi = j|x,vj)(emax(0,1−wT

j x) − emax(0,1+wTj x))

(21)Prediction is made by the sign of the discriminant function.Despite the complicated form of the function, it is straightfor-ward to see that it has a low computational complexity whichis linear in the number of mixture components with non-zerocoefficients.

6

E. Extension to a Hierarchical Mixture

Similar to the extension from mixture of experts to thehierarchical mixture model (HME) [22], it is natural to extendMLSVM to the hierarchical mixture model (HMLSVM). For athree-layer network with two hidden layers of gating functions,we have the following probability model,

P (Y |X,Θ) =∏n

∑i

∑j

[ξi%jP (rn = j|xn,ui) (22)

P (zn = i|xn,vi,j)P (yn|xn,wi,j)

]where rn and zn denote the hidden variables for the selectionof the gating units at the first and second hidden layers. In theequation above, ξi and %j are the data independent weightsfor the ith gating unit in the first layer and the jth gatingunit in the second layer, which is connected to the ith unitin the first layer. The probabilities P (rn = j|xn,ui) andP (zn = i|xn,vi,j) specify the data dependent posteriors forthe corresponding gating units.

Note that, for the hierarchical mixture above, we can stillapply the EM algorithm to estimate the model parameters withslight modifications in the estimation of mixing coefficients.Model selection can also be achieved by imposing the sameprior on the mixing coefficients, where a component is anni-hilated if its corresponding ξi or %j are negligible. In practice,the single-layer mixture model, which acts like a RBF kernelSVM, is capable of handling most classification problemsarising in practical applications. On the other hand, the hier-archical model, despite its elegant theoretical underpinnings,is prone to over fitting because it is difficult to regularise themodel. Hence, in this paper, we focus on the use of a single-layer MLSVM.

IV. DISCUSSION

The motivation of the work presented here is quite differentfrom that elsewhere in the literature. Competing approaches[24], [7], [23] employ a divide-and-conquer strategy similarin spirit to the mixture of expert framework [20] in thepartitioning of the input space into disjoint regions. Once thespace has been partitioned, a local non-linear SVM classifieris trained on samples falling in each region. Again, thismay greatly reduce the training time but does not create theefficiency for prediction due to the usage of non-linear SVMsas local experts. A straightforward remedy might be to replacethe non-linear SVMs with their linear counterparts.

Note that, in general there is no guarantee that the trainingdata in each region are linearly separable. Moreover, asmentioned in Section III-C, model selection is a hard problemfor the HME. Previous methods for model selection appliedto the mixture of experts [20] and its hierarchical extension[22] has been addressed by several authors [35], [32], [30],[14]. However, all of these approaches are focused on theuse of heuristic bottom-up procedures that start with a smallnetwork and grow it adaptively for learning the topology ofthe network. This process can hardly be automated and, thus,in practice, a fixed number of components is usually usedin mixture of expert learning. The work presented in this

paper is most similar in nature to the linear SVM mixturemodel proposed in [15], which is directly based on the mixtureof experts model [20] for linear SVMs. To the best of ourknowledge, the problem of locally linear learning for fastprediction with automatic model selection has not yet beeninvestigated.

Along a different line of research, approximation basedmethods have been widely used for non-linear SVMs toaccelerate the process of training them for large-scale data.Such methods are based on approximating the kernel matrixwith the outer product of linear vectors spanned by a set ofbasis functions. The bases can be obtained by using low-rankdecomposition methods like incomplete Cholesky factorisation[12], [1] or the Nystrom approximation [36]. Alternatively,they can also be chosen randomly following certain principles[34], [29]. Despite effective in speeding up the training pro-cess, kernel approximation methods come at an expense forthe efficiency in testing. This is due to the fact that the linearfeatures are obtained on top of the kernel matrix. Hence, thefull kernel matrix needs to be computed in the testing phase,which breaks the sparsity of support vectors and further slowsdown the prediction process.

Another natural choice for combining linear classifiers fornon-linear classification is ensemble methods such as bagging[4], boosting [13], and random forests [5]. These methods arequite effective in learning combinations of weak classifiers forclassification of data with complicated distributions, yet theydo not enforce sparseness on the hypothesis space. Therefore,when linear classifiers are used as the base learner, ensemblemethods tend to use a greater number of them for difficultcases compared to our approach and thus are likely to increasethe time spent on prediction. Moreover, ensemble methods arebest combined with unstable base learners, such as decisiontrees, to fully exploit different random natures of the weaklearners for combination. For stable classifier models likeSVMs and artificial neural networks, the training process isquite robust to data perturbation. Thus, the classifiers obtainedat different stages are likely to be highly correlated. Thiswould not only increase the number of iterations required forconvergence, but more importantly, would negatively affect theperformance of the combination scheme with such classifiersused as weak learners.

V. EXPERIMENTS

In this section, we demonstrate the performance of theproposed algorithm on both synthetic and real-world data.Two real-world applications are considered, namely opticalcharacter recognition (OCR) and hyperspectral image clas-sification. We compare our MLSVM algorithm with linearSVMs (LSVM), non-linear SVMs with the RBF kernel, andboosted linear SVMs. The boosting algorithm used here isAdaBoost [13]. Unless otherwise stated, in our framework wehave set the initial number of linear SVMs to 10. The finalnumber of linear models, however, might be less than 10 due tothe implicit model selection ability of the proposed MLSVMalgorithm in Figure 2. We empirically set the parameter τ inEquation 7 to unity. Note that τ may also be treated as an

7

independent parameter whose value can be chosen via crossvalidation. In practice, we have experimented with differentsettings of τ and found that its value does not have muchimpact on the classification performance. We have chosen theother parameters for MLSVM and the alternatives via 5-foldcross validation. LibSVM [10] and LibLinear [6] were usedfor training kernel and the linear SVMs, respectively. Forboosted linear SVMs, we have used the optimal regularisationparameter obtained via cross validation and fixed the numberof boosting iterations to 100.

Figure 3 shows the toy binary data-set used in our syntheticdata experiment, where 200 positive instances are shown inred circles which occupy the circular region in the middlesurrounded by 200 negative instances in green circles. Theinstances are chosen by subsampling a large sample of in-stances generated by uniform distribution within the squareregion. Clearly, the instances from these two classes are notlinearly separable. The top row, from left to right, shows thedecision boundary recovered by kernel SVMs, test errors ofboosted SVMs over sample iterations, and the log-likelihoodvalues of MLSVM as given by Equation 8. We have fixed thenumber of components to 4 for the MLSVM. For the non-linear SVM, the decision boundary was plotted in black. Thedecision boundaries with margin values of 1 corresponding tothe two classes. The support vectors are shown in solid circles.Even for this simple example, the SVM with RBF kernelyielded 71 support vectors, which is not particularly efficientfor large-scale prediction tasks. For the boosted SVMs, we cansee that the training error reduces slowly and becomes stableonly after a large number of iterations. This has confirmedour previous assertion that boosting requires many iterations toconverge. For MLSVM, on the other hand, we can see that thelog-likelihood value increases with every iteration and requiresonly a few iterations for convergence.

This is further justified by the plots in Figure 3. In themiddle row of the figure, from left to right, we show the resultsof MLSVM at the initial, 5th and final iterations. In each plot,the digit in each circle indicates the index of the mixturecomponent for the training instances. The shaded regionsindicate negative classes, whereas positive classes are in white.It can be seen that the classification model learned by the 5thiteration is already in good accordance to that yielded by thefinal iteration. This is in contrast with the boosted linear SVMmodel, which is constantly evolving. This can be seen from theresults recovered by the boosting model at the first, tenth, andfinal iterations as shown in the bottom row of the figure. Sinceboosting is a greedy strategy for classifier combination, it usesa much larger number of weak classifiers, most of which aimat correcting errors incurred in previous iterations. This hasthe counterproductive effect of cancelling the effects of oneanother. More importantly, unlike the MLSVM, each boostingiteration introduces a new classifier into the ensemble model.Hence, boosting has to combine a lot more linear classifiersthan the proposed algorithm to yield comparable performance.Even for this simple data-set, boosting employs approximately100 linear SVMs to achieve a comparable performance to thatof MLSVM, which only used 4 classifiers.

To further demonstrate the power of MLSVM for the

recovery of complex decision boundaries, we performed clas-sification on a second synthetic data set comprised by twospirals intertwined with each other, as shown in Figure 4.Each spiral has five turns and consists of 500 instances. Theseare generated by subsampling a larger sample of instancesuniformly distributed around the spiral curve given by thepolar function r = 1.3θ(−1.3θ) whose independent variable isθ ∈ [0.5π, 5.5π]. This is a hard classification problem due tothe high non-linearity of the data set. We have trained ourMLSVM classifier with 20 linear classifiers. The top row,from left to right, shows the results obtained by MLSVM inthe initial, tenth and final iterations. In each plot, the digit ineach circle indicates the index of the mixture component forthe training instances. The shaded regions indicate negativeclasses, whereas positive classes are in white. It can beseen that MLSVM can model rather complicated decisionboundaries through mixing the linear classifiers.

Furthermore, the decision boundary of MLSVM is refinedover iterations, from initially correctly classifying 95.7% of thedata to 99.7% in the final iteration. This is further justified bythe increased log-likelihood over iterations as shown by therightmost plot in the bottom row of the figure. The first twoplots in the bottom row show the decision boundaries producedby RBF kernel based non-linear and linear SVMs respectively.The linear decision boundary returned by the linear SVMonly achieves a random classification rate of 50%. This alsoinvalidates the use of the boosted SVM model, as boostingrequires at least better than random classification performanceat each iteration to proceed. The non-linear SVM achieves100% accuracy with a smoother decision boundary than thatof MLSVM. However, this is at the cost of generating over 750support vectors in the training process so as to approximatesuch complicated decision boundary. This accounts for overthree quarters of the training set and about 20 times ofthe complexity of our MLSVM, which only uses 20 linearclassifiers 1. For the two-dimensional spiral data set, it takes70 milliseconds for the non-linear SVM prediction and 3.5milliseconds for our MLSVM. The experiments were effectedon an Intel 3.0GHz Core 2 Duo desktop with 4GB RAM.Training MLSVM is more expensive than SVM, which takes5 seconds, as compared to only 0.5 seconds for the linear SVMwith a fully optimised LibSVM routine. However, trainingcan be done offline for most pattern classification problemswhereas prediction must be performed online. Hence onlineprediction efficiency is often more important especially forlarge-scale problems, which is the main focus of our MLSVMmethod.

Now we turn our attention to real-world data. We firstdemonstrate the performance of our algorithm for OpticalCharacter Recognition (OCR) on the MNIST database [26].The database contains 60000 training images and 10000testing images of handwritten digits from 0 to 9. We havedivided them into two classes, one for the odd numbers andanother one for the even digits. The resolution of each imageis 28×28, which leads to a 784-dimensional feature vector by

1Note that the complexity of MLSVM scales with twice the number oflinear classifiers, including gate function and classifier evaluations.

8

Fig. 3. MLSVM vs. boosting on synthetic two-class data. Top row, from left to right: decision boundary of non-linear SVM, test error of boosted SVMs,and log-likelihood of our algorithm as a function of iteration number; Middle row: results of our algorithm at the initial, fifth and final iterations; Bottomrow: results for the boosted SVMs at the first, tenth and final iterations.

Fig. 4. MLSVM vs. SVM on the synthetic spiral data set. Top row: results for our MLSVM at the initial, tenth and final iterations; Bottom row, from leftto right: decision boundaries of non-linear and linear SVMs, and log-likelihood of MLSVM as a function of iteration number.

9

concatenating the pixel values column-wise. We have trainedthe algorithms under comparison over subsets of different sizesdrawn randomly from the training set and applied them tothe testing set. We have repeated this procedure 10 times andrecovered quantitative results regarding classification perfor-mance.

Table I shows the classification rates on the test set interms of accuracy for MLSVM and the alternatives. It canbe seen clearly that the non-linear SVM outperforms theother methods. It delivers the highest classification accuracyfollowed by our MLSVM model. The MLSVM is comparablein performance with non-linear SVM on small sample sizesand is better than linear SVMs regardless of the size of trainingdata set. The performance of boosted linear SVMs is, however,unsatisfactory for the OCR task. For small training samplesizes, it achieves the same performance as the linear SVM.This may be due to the fact that the randomly selected subsetof the training data is most likely linearly separable, whichleads to the early stopping of the boosting procedure in thefirst iterations. This is confirmed by the number of iterationsused for boosting reported in Table II. For larger training sizes,the performance of boosting slightly decreases with increasingnumber of iterations, as reported in the last four rows of TableI. This suggests that boosted SVMs are not the model of choicefor bridging between the complexity of non-linear SVMs andthe performance of linear classifiers. On the contrary, boostingappears to have a higher complexity for increasing number ofiterations and lower performance as compared to linear SVMs.Our MLSVM model, on the other hand, clearly outperformslinear SVMs for OCR and is much more efficient than thenon-linear SVM. It is worth noting that although the trainingdata are likely to be linearly separable for small sample sizes,their distributions are inherently non-linear. Hence, non-linearSVMs still achieve higher classification accuracy than theirlinear counterparts. Our MLSVM can implicitly exploit thenon-linear nature of the data set and deliver better performancethan the linear SVM for small sample sizes.

We also compare the complexity of the methods understudy. Table II shows the complexity for boosted SVMs andMLSVM as measured in terms of number of linear classi-fiers used. For the non-linear SVM, we list the number ofsupport vectors as a measure of complexity. Notice that, fordifferent training data, SVMs may produce different numberof support vectors. Our proposed MLSVM method may alsoselect different number of linear classifiers via the trainingalgorithm in Figure 2 with implicit model selection. For eachtraining data size, we have listed the average, minimum andmaximum number of linear classifiers/support vectors usedover all the training rounds. From the table, note that thatthe complexity of the non-linear SVM grows with respect toincreasing training data sizes. This is as the number of supportvectors obtained in the training process is proportional to thesize of training set. The proposed MLSVM model, on the otherhand, is not affected by the size of the training data set. As canbe seen from the table, it is much more efficient than the non-linear SVM. Further, it is orders of magnitude faster for largetraining data sizes. For instance, it takes the non-linear SVM27, 46 and 195 seconds, in average, for the prediction of 10000

testing instances given training sets of sizes 5000, 10000 andall available data. On the other hand, the linear SVM takes, inaverage, only 10 milliseconds and MLSVM 0.7 seconds for thesame prediction task regardless of the sizes of the training dataset. Nevertheless, the MLSVM takes an hour for training usingthe full training data set, as compared to 1 and 2 minutes forthe non-linear and linear SVMs respectively 2. Again, as weargued before, this is not essential as training can be performedoffline. The behaviour of boosted SVMs is quite unexpected.For small training sizes, the randomly drawn training data setis likely to be linearly separable, so boosting stops at the firstiteration with zero training error. For larger training sets, itrequires a large number of iterations to converge. Since eachboosting iteration introduces a new classifier in the ensemblemodel, this greatly increases the complexity of the model.

We repeated the experiments above on the forest covertypedata set. This is a data set presented in [3] derived from dataoriginally obtained from US Geological Survey (USGS) databy the Colorado State University 3. The data set contains581012 instances from 7 cover types. Each instance has54 feature attributes capturing cartographic information. Thepurpose is to predict various cover types from the cartographicvariables. We converted this to a binary classification problemby predicting whether each instance belongs to cover type2, the largest cover type, containing 283301 instances, i.e.approximately half the data set. As for the OCR experiments,we compare the performances of our MLSVM algorithmand the alternatives using different training data set sizes.Training instances were randomly drawn from the full dataset and the rest were used for prediction. The procedure wasrepeated 10 times. The average accuracy rates along withstandard deviations are shown in Table III. We show the timecomplexities in Table IV.

The trend on the forest cover prediction data is similarto that for the OCR. The non-linear SVM achieves the bestperformance, followed by the MLSVM. Both of them performbetter than the Linear SVM. On the other hand, the non-linearSVM has the highest complexity for prediction. It is muchmore costly than the MLSVM and linear SVM. Moreover,the MLSVM achieves the optimal trade-off in predictionperformance and complexity. Given sufficient training data,the cost of MLSVM is almost independent of the trainingdata size. This is in contrast with the non-linear SVM, whosecomplexity scales with the number of training data used. Thisis further justified by the average testing speed. The non-linearSVM takes 40, 167 and 281 seconds for prediction giventraining set sizes of 1000, 5000 and 10000, respectively. OurMLSVM method takes 2 seconds and the linear SVM takes50 milliseconds for prediction regardless of the sizes of thetraining data set. Boosted linear SVMs are inferior to the othermethods being studied and do not achieve any improvementover linear SVM.

2We found the timing results for training quite unusual, with the non-linearSVM outperforming linear SVMs in training speed. This also greatly affectsthe speed of our MLSVM method, as it calls linear SVM solvers frequentlyfor training. This is counterintuitive, since LibLinear, the solver we used forlinear SVM training, should scale linearly with training sample sizes [6].

3The data set can be found at http://kdd.ics.uci.edu/databases/covertype/

10

Training Size LSVM (%) SVM (%) Bst-SVM (%) MLSVM (%)100 80.62± 2.33 84.70± 1.21 80.62± 2.33 82.82± 1.97500 82.71± 0.80 91.71± 0.96 82.71± 0.80 90.12± 1.43

1000 84.71± 0.53 94.59± 0.21 83.67± 0.24 92.82± 0.265000 88.47± 0.28 97.30± 0.16 87.40± 0.24 93.91± 0.3310000 89.60± 0.13 98.01± 0.09 88.98± 0.21 94.89± 0.23

All 90.28 98.49 89.72 95.20

TABLE IOCR PERFORMANCE. COMPARISON OF LINEAR SVM (LSVM), NON-LINEAR SVM (SVM), BOOSTED SVMS (BST-SVM) AND THE PROPOSED

MLSVM.

Training Size SVM Bst-SVM MLSVMmean min max mean min max mean min max

100 79.2 69 86 1 1 1 5.2 2 8500 263.2 236 288 1 1 1 5 2 7

1000 428.8 414 446 100 100 100 8.8 8 105000 1178.4 1157 1202 100 100 100 8.6 8 1010000 1817.0 1800 1833 61.8 24 100 8.6 7 10

All 3564 100 10

TABLE IIOCR COMPLEXITY. NUMBER OF SUPPORT VECTORS USED BY THE NON-LINEAR SVM (SVM) AND NUMBER OF CLASSIFIERS EMPLOYED BY BOOSTING

(BST-SVM) AND THE PROPOSED MLSVM.

Training Size LSVM (%) SVM (%) Bst-SVM (%) MLSVM (%)100 64.79± 3.52 66.19± 2.47 65.70± 2.86 65.98± 2.79500 73.34± 1.10 73.05± 0.93 73.30± 1.50 73.62± 0.89

1000 74.78± 0.47 75.22± 0.43 74.76± 0.51 74.89± 0.365000 75.97± 0.20 78.92± 0.29 76.06± 0.25 77.12± 0.1510000 76.19± 0.12 79.98± 0.16 76.13± 0.12 77.34± 0.18

TABLE IIIPERFORMANCE COMPARISON OF THE LINEAR SVM (LSVM), NON-LINEAR SVM (SVM), BOOSTED SVMS (BST-SVM) AND THE PROPOSED MLSVM

ALGORITHM ON THE FOREST COVER-TYPE DATA SET.

Training Size SVM Bst-SVM MLSVMmean min max mean min max mean min max

100 74.2 68 80 61.9 13 100 3.4 1 10500 323.9 286 356 33.9 10 62 1.4 1 3

1000 604.9 571 635 61.0 20 100 2.1 1 65000 2643.4 2568 2743 60.5 29 100 4.5 1 1010000 5028.6 4934 5121 53.3 19 100 4.2 1 9

TABLE IVCOMPLEXITY COMPARISON FOR THE FOREST COVER-TYPE DATA. NUMBER OF SUPPORT VECTORS USED BY THE NON-LINEAR SVM (SVM) AND

NUMBER OF CLASSIFIERS EMPLOYED BY THE BOOSTING ALGORITHM (BST-SVM) AND OUR PROPOSED MLSVM METHOD.

We now turn our attention to the application of our MLSVMalgorithm to pixel-level skin classification in hyperspectralimages. Here, we aim at exploiting the information-rich rep-resentation of the scene in hyperspectral imaging. To this end,we have used a set of 696 × 520 images of human subjectscaptured in-house against a cluttered background making useof a hyperspectral camera sensor. Each pixel in the imagehas 33 spectral channels covering the narrow band spectralreflectance in the visible range from 400 to 720nm in steps of10nm.

Example pseudo-colour images of some of the subjectsunder study are shown in the top row of Figure 5. For purposesof training, we have selected patches enclosing the skin areasfrom a few images. The spectral feature vectors were thenobtained using the photometric invariant preprocessing methodin [16]. Since boosted SVMs do not produce satisfactory

results in terms of either classification accuracy or efficiency ofprediction, we have focused our comparison on the remainingthree algorithms. Some example skin classification results areshown in the two bottom rows of Figure 5. These correspond,from top-to-bottom, to the results yielded by the MLSVM,the non-linear SVM with RBF kernel and the linear SVM,respectively. We can see that the skin maps recovered by theMLSVM are quite similar to those yielded by the non-linearSVM, whereas the skin maps computed using the linear SVMare less accurate than those corresponding to the other twomethods. Furthermore, from the figure, we can see that the skinmaps returned by the linear SVM contain more false positivesthan those yielded by the MLSVM. This is particularly evidenton the small regions in the background wallpaper, which havebeen falsely classified as skin.

In Table V, we show a more quantitative analysis of the

11

MLSVM SVM LSVMAccuracy 95.20± 0.30 95.45± 0.22 91.34± 0.15Time (s) 6.55± 0.25 76.25± 1.30 1.65± 0.05

TABLE VAVERAGE CROSS VALIDATION ERROR AND TESTING TIME FOR SKIN

CLASSIFICATION.

performance for the alternatives under study. This includesthe average testing accuracy as well as the average testingtime in seconds per image. The average testing accuracy ismeasured by splitting the training data into two sets, onefor training and the other one for testing. This is done byevenly splitting the data set and averaging the results over 5different random partitions. From the table, we again noticethat the MLSVM is much more efficient while achieving aperformance comparable to that yielded by the non-linearSVM. The linear SVM, on the other hand, is quite efficientfor testing but has a lower accuracy rate as compared to thenon-linear SVM and our MLSVM approach.

VI. CONCLUSIONS

In this paper we have proposed a mixture of linear SVMs(MLSVM). This is a novel method to combine linear classifiersfor the classification of non-linear data. We have addressedthe problem by casting it into a probabilistic setting soas to implicitly perform model selection. This is done byusing the EM algorithm on a generative model that permitsthe use of priors on the mixing coefficients. Our MLSVMmethod provides a means to efficient non-linear classificationby presenting a top-down parameter estimation process. Themethod is quite general in nature and is particularly well suitedfor large-scale prediction tasks. We have done experimentson synthetic and real-world data and provided comparison toboosting, linear and non-linear SVMs.

There are a number of possible directions for future re-search. One potential application of the proposed frameworkis feature combination, where each example in the data setis represented by multiple feature vectors extracted from dif-ferent sources. The standard approach to solving this problemwith SVM is multiple kernel learning (MKL) [25], [19], whereindividual kernel matrices computed from different featuresare linearly combined. MKL is then formulated as a jointoptimisation problem over the kernel combination weights andthe SVM classifier. Unfortunately, testing can be computa-tionally expensive for large data sets with many input featuretypes [19]. The method proposed here can learn a mixture oflinear SVMs. This can be done separately for the individualfeatures so as to combine the output scores. Another optionis to combine feature types directly making use of the jointfeature vectors. The mixture model can capture the nonlinearinformation in each individual feature while still maintainingefficiency in prediction. Another possible extension is the useof error generalisation bounds for parameter selection [2], [8].This is especially effective for small sample size problems andprovides a more accurate estimate of the prediction error thanthat given by cross validation.

VII. ACKNOWLEDGMENT

The work presented here was done while Z. Fu was withthe Australian National University, Canberra, Australia.

REFERENCES

[1] F. Bach and M. I. Jordan, “Predictive low-rank decomposition for kernelmethods,” in Intl. Conf. on Machine Learning, 2005.

[2] P. Bartlett, S. Boucheron, and G. Lugosi, “Model selection and errorestimation,” Machine Learning, vol. 48, pp. 85–113, 2001.

[3] J. A. Blackard and D. J. Dean, “Comparative accuracies of artificial neu-ral networks and discriminant analysis in predicting forest cover typesfrom cartographic variables,” Computers and Electronics in Agriculture,vol. 24, no. 3, pp. 131–151, 2000.

[4] L. Breiman, “Bagging predictors,” Machine Learning, vol. 26, no. 2,1996.

[5] ——, “Random forests,” Machine Learning, vol. 45, no. 1, 2001.[6] K.-W. C. C-J. Hsieh, C.-J. Lin, S. Keerthi, and S. sundararajan, “A dual

coordinate descent method for large-scale linear SVM,” in Intl. Conf.on Machine Learning, 2008.

[7] R. Collobert, S. Bengio, and Y. Bengio, “A parallel mixture of SVMsfor very large scale problems,” in Advances in Neural InformationProcessing Systems, 2002.

[8] S. Decherchi, S. Ridella, R. Zunino, P. Gastaldo, and D. Anguita, “Usingunsupervised analysis to constrain generalization bounds for supportvector machines,” IEEE Trans. on Neural Networks, vol. 21, no. 3, pp.424–438, 2010.

[9] A. Dempster, N. Laird, and D. Rubin, “Maximum-likehood from incom-plete data via the EM algorithm,” J. Royal Statistical Society B, vol. 39,pp. 1–38, 1977.

[10] R.-E. Fan, P.-H. Chen, and C.-J. Lin, “Working set selection usingthe second order information for training SVM,” Journal of MachineLearning Research, vol. 6, pp. 1889–1918, 2005.

[11] M. Figueiredo and A. K. Jain, “Unsupervised learning of finite mixturemodels,” IEEE Trans. on Pattern Analysis and Machine Intelligence,vol. 24, no. 3, pp. 381–396, 2002.

[12] S. Fine and K. Scheinberg, “Efficient SVM training using low-rankkernel representations,” Journal of Machine Learning Research, vol. 2,pp. 243–264, 2001.

[13] Y. Freund and R. E. Schapire, “A decision-theoretic generalization ofon-line learning and an application to boosting,” Journal of Computerand System Sciences, vol. 55, 1997.

[14] J. Fritsch, M. Finke, and A. Waibel, “Adaptively growing hierarchicalmixtures of experts,” in Advances in Neural Information ProcessingSystems, 1997.

[15] Z. Fu and A. Robles-Kelly, “On linear SVM mixtures for nonlinearclassification,” in Intl. Workshop on Statistical Pattern Recognition,2008.

[16] Z. Fu, A. Robles-Kelly, R. Tan, and T. Caelli, “Invariant object materialidentification via discriminant learning on absorption features,” in CVPRWorkshop on Object Tracking and Classification Beyond the VisibleSpectrum, 2006.

[17] L. Gonzalez, C. Angulo, F. Velasco, and A. Catala, “Dual unificationof bi-class Support Vector Machine formulations,” Pattern Recognition,vol. 39, no. 7, pp. 1325–1332, 2006.

[18] C.-W. Hsu and C.-J. Lin, “A comparison of methods for multi-classsupport vector machines,” IEEE Trans. on Neural Networks, vol. 13,pp. 415–425, 2002.

[19] M. Hu, Y. Chen, and J. T.-Y. Kwok, “Building sparse multiple-kernelSVM classifiers,” IEEE Trans. on Neural Networks, vol. 20, no. 5, pp.827–839, 2009.

[20] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptivemixtures of local experts,” Neural Computation, vol. 3, pp. 79–87, 1991.

[21] T. Joachims, “Training linear SVMs in linear time,” in SIGKDD, 2006.[22] M. I. Jordan and R. A. Jacobs, “Hierarchical mixtures of experts and

the EM algorithm,” Neural Computation, vol. 6, pp. 181–214, 1994.[23] S. Kruger, M. Schaffoner, M. Katz, E. Andelic, and A. Wendemuth,

“Mixture of support vector machines for hmm based speech recogni-tion,” in Intl. Conf. on Pattern Recognition, 2006.

[24] J. T. Kwok, “Support vector mixture for classification and regressionproblems,” in Intl. Conf. on Pattern Recognition, 1998.

[25] G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan,“Learning the kernel matrix with semidefinite programming,” Journalof Machine Learning Research, vol. 5, pp. 27–72, 2004.

12

Fig. 5. Example skin classification results. Top row: pseudo-color images for sample human subjects; Second, third and fourth rows: classification resultsdelivered by our algorithm, the non-linear SVM and linear SVM.

[26] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learningapplied to document recognition,” in IEEE, vol. 86, no. 11, 1998, pp.2278–2324.

[27] D. Liu and J. Nocedal, “On the limited memory method for large scaleoptimization,” Mathematical Programming B, vol. 45, no. 3, pp. 503–528, 1989.

[28] W. H. Press, S. A. Teukolsky, W. T. Vetterlin, and B. P. Flannery,Numerical Recipes in C. Cambridge University Press, 1992.

[29] A. Rahimi and B. Recht, “Random features for large-scale kernelmachines,” in Advances in Neural Information Processing Systems,2007.

[30] V. Ramamurti and J. Ghosh, “Structural adaptation in mixtures ofexperts,” in Intl. Conf. on Pattern Recognition, 1996.

[31] J. Rissanen, Stochastic Complexity in Statistical Inquiry. WorldScientific, 1989.

[32] K. Saito and R. Nakano, “A constructive learning algorithm for an hme,”in Intl. Conf. on Neural Networks, 1996.

[33] B. Scholkopf and A. J. Smola, Learning with Kernels: Support VectorMachines, Regularization, Optimization, and Beyond. MIT Press, 2001.

[34] A. Smola and B. Scholkopf, “Sparse greedy matrix approximation

for machine learning,” in Advances in Neural Information ProcessingSystems, 2000.

[35] S. R. Waterhouse and A. J. Robinson, “Constructive algorithms forhierarchical mixtures of experts,” in Advances in Neural InformationProcessing Systems, 1996.

[36] C. Williams and M. Seeger, “Using the nystrom method to speedup kernel machines,” in Advances in Neural Information ProcessingSystems, 2001.

[37] L. Xu, M. I. Jordan, and G. E. Hinton, “An alternative model for mixturesof experts,” in Advances in Neural Information Processing Systems,1994.