Proceedings of International Joint Conference on Neural Networks, Montreal, Canada, July 31 - August 4, 2005

A Study of AdaBoost with SVM Based WeakLearners

Xuchun Li, Lei Wang, Eric SungSchool of Electrical and Electronic Engineering

Nanyang Technological University, Singapore, 639798E-mail: xuchunli@pmail.ntu.edu.sg, elwang@ ntu.edu.sg, eericsung@ntu.edu.sg

Abstract- In this article, we focus on designing an algorithm,named AdaBoostSVM, using SVM as weak learners for AdaBoost.To obtain a set of effective SVM weak learners, this algorithmadaptively adjusts the kernel parameter in SVM instead of usinga fixed one. Compared with the existing AdaBoost methods,the AdaBoostSVM has advantages of easier model selection andbetter generalization performance. It also provides a possible wayto handle the over-fitting problem in AdaBoost. An improvedversion called Diverse AdaBoostSVM is further developed to dealwith the accuracy/diveristy dilemma in Boosting methods. Byimplementing some parameter adjusting strategies, the distribu-tions of accuracy and diversity over these SVM weak learners aretuned to achieve a good balance. To the best of our knowledge,such a mechanism that can conveniently and explicitly balancesthis dilemma has not been seen in the literature. Experimentalresults demonstrated that both proposed algorithms achievebetter generalization performance than AdaBoost using otherkinds of weak learners. Benefiting from the balance betweenaccuracy and diversity, the Diverse AdaBoostSVM achieves thebest performance. In addition, the experiments on unbalanceddata sets showed that the AdaBoostSVM performed much betterthan SVM.

I. INTRODUCTION

One of the major developments in machine learning in thepast decade is the ensemble method, which finds a highlyaccurate classifier by combining many moderately accuratecomponent classifiers. Two of the commonly used techniquesfor constructing ensemble classifiers are Boosting [1] andBagging [2]. Comparing with Bagging, Boosting performsbetter in most situations [3]. As the most popular Boostingmethod, AdaBoost [4] creates a collection of weak learnersby maintaining a set of weights over training samples andadjusting these weights after each weak learning cycle adap-tively: the weights of the samples which are misclassified bycurrent weak learner will be increased while the weights ofthe samples which are correctly classified will be decreased.The success of AdaBoost can be explained as enlarging the

margin [5], which could enhance AdaBoost's generalizationcapability. Many studies that use decision trees [6] or neuralnetworks [7] [8] as weak learners for AdaBoost have beenreported. These studies showed the good generalization per-formance of AdaBoost. However, when decision trees are usedas weak learners, what is the suitable tree size? When RBFneural networks are used as weak learners, how to control theircomplexity to avoid overfitting? How to decide the numberof centers and how to set the width values of the RBFs?

All of these have to be carefully tuned in practical use ofAdaBoost. Furthermore, diversity is known as an importantfactor which affects the generalization accuracy of ensembleclassifiers [9][10]. It is also known that AdaBoost exists anaccuracy/diversity dilemma [6], which means that the moreaccurate two weak learners become, the less they can disagreewith each other. However, the existing AdaBoost algorithmshave not yet explicitly taken sufficient measurement to dealwith this dilemma. Finally, It is reported that AdaBoost mayoverfit the training samples [ I1] and result in poor generaliza-tion performance. Therefore, it is necessary to stop AdaBoost'slearning cycles at a suitable moment. But how to truncate theAdaBoost learning process to avoid overfitting is still an openproblem [ I1].

Support Vector Machine [ 12] was developed from the theoryof Structural Risk Minimization. By using a kernel trick tomap the training samples from an input space to a highdimensional feature space, SVM finds an optimal separatinghyperplane in the feature space and uses a regularizationparameter, C, to balance its model complexity and trainingerror. One of the popular kernels used by SVM is the RBFkernel, which has a parameter known as Gaussian width,a. Comparing with RBF networks, SVM with RBF kernel(RBFSVM) can automatically calculate the number and loca-tion of the centers and the weights [ 13]. Also, it can effectivelyavoid overfitting by selecting proper parameters C and a. Fromthe performance analysis of RBFSVM [ 14], we know that af isa more important parameter: although RBFSVM cannot learnwell when a very low value of C is used, its performancelargely depends on the af value if a roughly suitable C isgiven. This means that, for a given roughly suitable C, theperformance of RBFSVM can be conveniently changed bysimply adjusting the value of a.

Therefore, in this paper, we try to answer the followingquestions: Can SVM be used as a weak learner of AdaBoost?If yes, how about the generalization performance of thisAdaBoost? Does this AdaBoost have some advantages over theexisting ones, especially about the aforementioned problems?Also, compared with using a single SVM, what is the benefitof using this AdaBoost which is a combination of multipleSVMs? In this work, the RBFSVM is adopted as a weaklearner for AdaBoost. As mentioned above, the RBFSVMhas a parameter of a which has to be set beforehand. Anintuitive way is to simply apply a single a to all SVM weak

D-7803-9048-2/05/$20.00 ©2005 IEEE 196

learners. However, we observed that this way cannot lead tosuccessful AdaBoost due to over-weak or over-strong SVMweak learners. Although there may exist a single best a,searching for it will largely increase the computational load ofBoosting process and therefore should be avoided if possible.The following fact opens a door for us to avoid searching

for the single best (X. It is known that the classificationperformance of RBFSVM can be conveniently changed byadjusting the value of o. Based on this, the algorithm, Ad-aBoostSVM, is developed, where a set of moderately accurateRBFSVM is trained for AdaBoost by adaptively adjusting theof values instead of using a fixed one. This gives rise to asuccessful SVM based AdaBoost. Compared with the existingAdaBoost methods, proposed AdaBoostSVM has the followingadvantages: It needs not perform accurate parameter tuning.Instead, giving a rough range of o is often enough. By settingthe lower end of this range, proposed algorithm also providesa possible way to truncate the Boosting process to handlethe over-fitting problem when other kinds of weak learnersare used. Besides these, proposed algorithm achieves betterclassification performance than AdaBoost with other weaklearners such as neural networks.

Furthermore, since proposed AdaBoostSVM invents a con-venient way to control the classification accuracy of each weaklearner, it also provides an opportunity to deal with the well-known accuracy/diveristy dilemma in Boosting methods. Thisis a happy accident from the investigation of AdaBoost basedon SVM weak learners. Through some parameter adjustingstrategies, we can tune the distributions of accuracy anddiversity over these weak learners to achieve a good balance.To the best of our knowledge, there is no such a mechanismwhich can conveniently and explicitly balance this dilemmain the literature. The improved version of AdaBoostSVM iscalled Diverse AdaBoostSVM in this work. It is observed that,benefiting from the balance between accuracy and diversity, itgives better performance than AdaBoostSVM.

Finally, compared with a single SVM, proposed algorithms,which are the combination of multiple SVMs, can achievemuch better performance on unbalanced data sets. This alsojustifies, from another perspective, that AdaBoost with SVMweak learners is worth of investigation.

II. BACKGROUNDA. AdaBoost

Given a set of training samples, AdaBoost [15] maintainsa probability distribution, W, over these samples. This distri-bution is initially uniform. Then, AdaBoost algorithm callsWeakLearn algorithm repeatedly in a series of cycles. Atcycle t, AdaBoost provides training samples with a distributionWt to the WeakLearn algorithm. In response, the WeakLearntrains a classifier ht. The distribution Wt is updated aftereach cycle according to the prediction results on the trainingsamples. "Easy" samples that are correctly classified by theweak learner, ht, get low weights, and "hard" samples thatare misclassified get higher weights. Thus, AdaBoost focuseson the samples with more weights, which seem to be harder

for WeakLearn. This process continues for T cycles, and atlast, AdaBoost uses weighted vote to combine all the obtainedweak learners into a single final hypothesis f. Greater weightsare given to weak learners with lower errors. The importanttheoretical property of AdaBoost is that if the weak learnersconsistently have accuracy only slightly better than halt; thenthe error of the final hypothesis drops to zero exponentiallyfast. This means that the weak learners need be only slightlybetter than random.

TABLE I

Algorithm: AdaBoost [15]1. Input: a set of training samples with labels {(x1,yI), .(xN,yN)},WeakLearn algorithm, the number of cycles T.2. Initialize: the weight of samples: w1 = 1/N, for all i =1, ...,N.3. Do for t =1. T

(1 )Use WeakLearn algorithm to train the weak learner ht on theweighted training sample set.

(2)Calculate the training error of ht : Et= EN1 w /iYi $ ht(x1)(3)Set weight of weak learner ht at =I In

(4)Update training samples' weights: w - exp{J-rtyht(X )}where Ct is a normalization constant, and N w+ = 1.4. Output: f(x) = signl([T1 at ht(x)).

B. Support Vector MachineSupport Vector Machine (SVM) [12] was developed from

the theory of Structural Risk Minimization. In a binary clas-sification problem, SVM's decision function is

f(x) = (w, q(x)) + b (1)where 0(x) is a mapping of sample x from the input space toa high-dimensional feature space. (-, ) denotes the dot productin the feature space. The optimal values of w and b can beobtained by solving the following optimization problem,

INminin)inike: g(w,6,)= 2 |W||2+CEXi1 i=l

sutbjec t to: yi ((w, O(x2 )) + b) > 1- ,.(> 0

(2)

(3)

Here, (i is the i-th slack variable and C is the regularizationparameter. According to the Wolfe dual form, the aboveminimization problem can be written as

N 1 N N

mininize: W(a) = - a1i + 2E E yiyjaiajk(xi, xj)i= 1 i= '=I

N

slubject to: , yiai = 0, Vi : 0 < ai < Ci=l

(4)

(5)

where ai is a Lagrange multiplier which corresponds to thesample xi, k(-,-) is a kernel function that implicitly maps theinput vectors into a suitable feature space

k(xi, xj) = (q$(xi), 0(xj)) (6)

197

Compared with the RBF networks [131, SVM algorithm auto-matically computes the number and location of the centers, theweights, and the threshold in the following way: by the use of asuitable kernel function(in this paper, the RBF kernel is used),the samples are mapped nonlinearly into a high dimensionalfeature space. In this space, an optimal separating hyperplaneis constructed by the support vectors which are closest to thedecision boundary. Support vectors correspond to the centersof RBF kernels in the input space.

III. PROPOSED ALGORITHM: ADABOOSTSVMThis work uses the RBFSVM as weak learner for AdaBoost.

But how to set the a value for these weak learners'? Problemsare encountered when applying a single a to all weak learners.In detail, an over-large a often results in too weak RBFSVM.Its classification accuracy is often less than 50% and cannotmeet the requirement on a weak learner given in AdaBoost. Onthe other hand, a smaller a often makes the RBFSVM strongerand boosting them may become inefficient because the errorsof these weak learners are highly correlated. Furthermore, toosmall a can even make RBFSVM overfit the training samplesand they also cannot be used as weak learners. Hence, findinga suitable a for AdaBoost with SVM weak learners becomes aproblem. By using model selection techniques such as cross-validation or leave-one-out, a single best a may be found,with which the AdaBoost may achieve good classificationperformance. However, the process of model selection is time-consuming and should be avoided if possible.The classification performance of SVM is affected by its

parameters. For RBFSVM, they are the Gaussian width, a,and the regularization parameter, C. The variation of eitherof them leads to the change of classification performance.However, as reported in [141, although RBFSVM cannot learnwell when a very low value of C is used, its performancelargely depends on the a value if a roughly suitable C is given.This means that, for a given C, the performance of RBFSVMcan be changed by simply adjusting the value of a. Increasingthis value often reduces the learning model complexity andthen lowers down the classification performance, whereasdecreasing it can lead to more complex learning model andhigher performance. Therefore, this gives a chance to getaround the problem resulting from using a fixed a for allRBFSVM weak learners. In the following proposed algorithm,we generate a set of moderately accurate RBFSVM classifiersfor AdaBoost by adaptively adjusting the a values.When applying Boosting method to strong component

classifiers, these classifiers must be appropriately weakenedin order to benefit from Boosting [6]. Hence, if RBFSVMis used as weak learner for AdaBoost, a relatively largea value, which corresponds to a RBFSVM with relativelyweak learning ability, is preferred. Both re-sampling and re-weighting can be used to train AdaBoost. In the proposedalgorithm, without loss of generality, re-weighting techniqueis used. We proposed an AdaBoostSVM algorithm as follows(Table. II): Firstly, a large value is set to a, which correspondsto a RBFSVM classifier with very weak learning ability.

Then, RBFSVM with this a is used in as many cycles aspossible as long as more than half accuracy on weightedtraining samples can be obtained. Otherwise, this a value isdecreased slightly to make the RBFSVM with the new a valueobtain more than half accuracy. This process continues untilthe a is decreased to the given minimal a value. By doingso, the proposed AdaBoostSVM algorithm can often generatea set of moderately accurate SVM classifiers with possiblyuncorrelated errors.

TABLE 11

Algorithm: AdaBoostSVM1. Input: a set of training samples with labels {(XI,YI),..). (XN,YN)};the initial a, oini; the minimal a, amin; the step of a, 0'tep.2. Initialize: the weight of samples: wl = 1/N, for all i =1,., N.3. Do While (a > Cmzn)

(I)Use RBFSVM algorithm to train the weak leamer ht on theweighted training sample set.

(2)Calculate training error of ht = EN1 S. Yi $ ht(xi).(3)lf et > 0.5, decrease af value by oastep and goto (1).(4)Set weight of weak learner ht: at = 2 In( ).(5)Update training samples' weights: w. - wexp{(atiTt(x,)}

where Ct is a normalization constant, and EN=1 w = 14. Output: f(x) = sign([1: atht(x)).

IV. IMPROVEMENT: DIVERSE ADABOOSTSVMA. Accuracy/Diversity Dilemma of AdaBoost

Diversitv is known to be an important factor affectingthe generalization performance of ensemble methods [9][10],which means that the errors made by different componentclassifiers are uncorrelated. If each component classifier ismoderately accurate and these component classifiers disagreewith each other, the uncorrelated errors of these componentclassifiers will be removed by the voting process so as toachieve good ensemble results [16]. This also applies to Ad-aBoost. For AdaBoost, it is known that there exists a dilemmabetween weak learner's accuracy and diversity [61, whichmeans that the more accurate two weak learners become, theless they can disagree with each other. Hence, how to selectSVM weak learners for AdaBoost? Select accurate but notdiverse weak learners? Or select diverse but not too accurateones? In the proposed AdaBoostSVM, the obtained SVM weaklearners are mostly moderately accurate, which give chances toobtain more un-correlated weak learners. As aforementioned,through adjusting the af value, a set of SVM weak learnerswith different learning abilities is obtained. This provides anopportunity of selecting more diverse weak learners from thisset to deal with the accuracy/diversity dilemma. Hence, weproposed a Diverse AdaBoostSVM algorithm (Table. III), andit is hoped to achieve higher generalization performance thanAdaBoostSVM.

B. Diverse AdaBoostSVMAlthough how to measure and use diversity for ensemble

methods is still an open problem [10], recently some promising

198

results have been reported. By focusing on increasing diversity,these methods [9][17] achieved higher generalization accuracy.In the proposed Diverse AdaBoostSVM algorithm, we willuse the definition of diversity in [9], which measures thedisagreement between one weak learner and all the existingweak learners. In the Diverse AdaBoostSVM, the diversityis calculated as follows: If ht(xi) is the prediction label oft-th weak learner on sample xi, and f(xi) is the combinedprediction label of all the existing weak learners, the diversityof the t-th weak learner on sample xi is defined as:

dt(xi) =f0: if ht(xi) = f(xi) 71: if ht (xi) 0 f(xi)

and the diversity of AdaBoostSVM with T weak learners onN samples is defined as:

1T ND = TN xv a dt (Xi) (8)

t=1 Zi=1

In each cycle of Diverse AdaBoostSVM, this diversity valueis calculated first. If this value is larger than the predefinedthreshold, DIV, this new RBFSVM weak learner will beselected. Otherwise, this weak learner will be thrown away.Through this mechanism, a set of moderately accurate and yetdiverse SVM weak learners can be generated. This is differentfrom the AdaBoostSVM which simply takes all the availableSVM weak learners. As seen from the following experimentalresults, the Diverse AdaBoostSVM algorithm gives the bestperformance. We think that the improvement is due to itsexplicitly dealing with the accuracy/diversity dilemma.

TABLE III

Algorithm: Diverse AdaBoostSVM1. Input: a set of training samples with labels {(xI,y), (xN,yN)};the initial ar, aiui; the minimal or. amin; the step of a, o8,tep; thethreshold on diversity DIV.2. Initialize: the weight of samples: wl = 1/N, for all i = 1..t N.3. Do While (a > amin)

(1)Use RBFSVM algorithm to train the weak leaner ht on theweighted training sample set.

(2)Calculate training error of ht : et= Z=1 wi, yYi : ht(xi).(3)Calculate diversity of ht: Dt = Z$N1 dt(xi).(4)if et > 0.5 or Dt < DIV, decrease af by Ostep and goto (1).(5)Set weight of weak learner ht at = 1 ln().

2 Et(6)Update training samples' weights: t+1 = W expw{-atyihttxj}

where Ct is a normalization constant, and EN 1 w =1.4. Output: f (x) = sign(E[1 atht(x)).

V. EXPERIMENTAL RESULT

Since AdaBoost with neural networks weak learners per-forms better than those with decision trees weak learners [7],we compare proposed AdaBoostSVM and Diverse AdaBoost-SVM algorithms with AdaBoost with neural networks weaklearners. A large scale of experiments are generated on 13benchmark data sets and an unbalanced data sets.

A. Benchmark Data Sets

1) Data set information and parameter setting: The 13benchmark data sets come from UCI Repository, DELVE, andSTATLOG. The dimensions of these data sets range from 2 to60, the numbers of training samples range from 140 to 1300,the numbers of test samples range from 75 to 7000. Detailedinformation of these data sets can be found at [18] and all ofthese data sets can be downloaded there. For each data set,100 partitions are generated into training and test sets. Oneach partition, for each algorithm, a classifier is trained andthen test error is calculated. The final performance of eachalgorithm on a data set is its average performance over these100 partitions of this data set.

It is known that a mainly affects the performance ofRBFSVM classifier compared with C. Hence, C is set as avalue within 10 to 100 in proposed algorithms. The 0min isset as the average minimal distance between any two trainingsamples and the oi,i is set as about 10 to 15 times of theamin value or the scatter radius of the training samples in theinput space. Although the value of astep can affect the numberof AdaBoostSVM's learning cycles, it has less effect on thefinal generalization performance, as shown later. Therefore,Ustep is set as a value within I to 3. The 'DIV' value in theDiverse AdaBoostSVM algorithm is set as 0.7 to I times ofthe maximal diversity value obtained in previous cycles. "0.7"is used as the tolerance of the possible variation of diversity.

2) General performance: In Table IV, the average gener-alization performance (with standard deviation) for the fouralgorithms is shown. The test errors of AdaBoost with NeuralNetworks weak learners (AdaBoostNN) and SVM are directlyobtained from [8]. From these results, it can be found thatproposed AdaBoostSVM algorithm (AdaBoostsvAlz) performsbetter than AdaBoostNN algorithm, and is comparable toSVM algorithm. Proposed Diverse AdaBoostSVM (DiverseAdaBoostNN) performs a little better than SVM algorithm.

TABLE IV

Data set ABNN ABSVAM Diverse ABSVAIV SVMBanana 12.3+0.7 12.1±1.7 11.3±1.4 11.5+0.7

B. Cancer 30.4±4.7 25.5±5.0 24.8±4.4 26.0±4.7Diabetes 26.5±2.3 24.8±2.3 24.3±2.1 23.5+1.7German 27.5+2.5 23.4±2.1 22.3±2.1 23.6±2.1Heart 20.3±3.4 15.5±3.4 14.9±3.0 16.0±3.3Image 2.7±0.7 2.7±0.7 2.4±0.5 3.0±0.6

Ringnorm 1.9±0.3 2.1± 1.1 2.0±0.7 1.7±0.1F. Solar 35.7±1.8 33.8±1.5 33.7±1.4 32A.1.8Splice 10.1±0.5 11.1±1.2 11.0±1.0 10.9±0.7Thyroid 4.4±2.2 4.4±2.1 3.7±2.1 4.8±2.2Titanic 22.6±1.2 22.1±1.9 21.8±1.5 22.4±1.0Twonorm 3.0±0.3 2.6±0.6 2.5±0.5 3.0±0.2Waveform 10.8±0.6 10.3±1.7 10.2±1.2 9.9±0.4Average 16.0±1.6 14.6±1.9 14.2±1.7 14.5±i.5

Comparison among four algorithms: AdaBoost with neural networks weaklearners (ABNN), proposed AdaBoostSVM (ABSVAI), proposed DiverseAdaBoostSVM (Diverse ABSV AI) and SVM

3) Influence of C and ajinj In order to investigate theinfluence of parameter C on the proposed algorithms, we varythe value of C from 1 to 100, and perform experiments on

199

100 partitions of the "Titanic" data set of UCI benchmarkto obtain their average performances. Figure. 1 shows thecomparison result. It can be found that in a considerable scale(in this case, it is from 1 to 100), the variation of C has littleeffect (less than 1%) on the final generalization performanceof proposed algorithm. This is also consistent with the analysisof RBFSVM that C value has less effect on the performanceof RBFSVM. Note that the a value decreases from u1ini toUmin as the number of SVM weak learners increases (Seethe label of horizontal axis). The small platform at the lefttop corner of this figure means that the test error does notdecrease until the a decreases to a certain value. Then, thetest error decreases quickly to the lowest value and keepsthis value with a little variation. This shows that the cinivalue does not have much impact on the final performanceof AdaBoostSVM. Furthermore, the learning cycle will betruncated when a reaches the given Umin value. This makes itpossible to truncate the learning cycle to handle the overfittingproblem by setting a suitable amin. This will be investigated inour future work. This property is more valuable because whento stop AdaBoost's learning cycle is still an open problem forAdaBoost.

32%,1 ~~~~~~ F ~~~c=1

31% -i-C=50C=100

30%

29%

2 28%

" 27%-

i-26%-

25%-

24%-

23%-

20 40 60 80 100 120 140number of SVM weak learners (a.i. ->a.n)

Fig. 1. Compare the performance of AdaBoostSVM with different C values

4) Influence of uJstep: In order to see the influence of Ustepon the proposed AdaBoostSVM algorithm, we did a set ofexperiments with different aJstep values for AdaBoostSVMon the "Titanic" data set of UCI benchmark. Figure. 2 givesthe result. From this figure, we can find that although thenumber of learning cycles in AdaBoost changes with the valueof Ustep, the final generalization accuracy is relatively stable.Similar conclusions can also be drawn from other benchmarkdata sets that the value of Ustep has less effect on the finalgeneralization performance of AdaBoostSVM.From the above discussion of parameters C, a7ini, Ustep, it

can be concluded that, compared with AdaBoost with neuralnetworks weak learners, proposed AdaBoostSVM algorithm iseasier to do parameter tuning.

20 4u 1bro%VM 80 100 12020 Nuanber oSMweak learner

Fig. 2. Compare the performance of AdaBoostSVM with different (Jstep

5) Accuracy-Diversity diagram: In order to see the effectof Diverse AdaBoostSVM, we used an Accuracy-Diversitydiagram to visualize the distribution of accuracy and thediversity over the SVM weak learners. The Accuracy-Diversitydiagram is a scatterplot where each point corresponds to aweak learner. In this diagram, a point's x coordinate valueis the diversity value after the corresponding weak learner istrained while y coordinate value of this point is the accuracyrate of the corresponding weak learner. Similar diagram wasalso used in [6]. Due to the lack of space, we report only theAccuracy-Diversity diagrams of Diverse AdaBoostSVM andAdaBoost with neural networks weak learner algorithms onthe 'Splice' data set of the UCI benchmark. From Figure. 3, wecan observe that proposed Diverse AdaBoostSVM algorithmcan obtain more high-diversity and moderately accurate weaklearners, which may produce the uncorrelated error amongdifferent weak learners more efficiently.

100%

900/%

9'80%a;"70%

60%

Diverse AdaBoostSVM

f0 *

**, * #,*+"', 5 0.55 0.6 p.65 0.7 0.75

Diversity

(a)Diverse AdaBoostSVM

AdaBoost with Neural Network weak learner100%

90%

080% **W

070%/

60%

500/T6.3 0.4 0.5 0.6 0.7Diversitv

(b) Neural Networks weak learner

Fig. 3. Compare the accuracy and the diversity's distributions betweenDiverse AdaBoostSVM and AdaBoost with Neural Networks weak learners

B. Unbalanced Data SetsWe also find that the proposed AdaBoostSVM algorithm

performs much better than a single SVM when facing unbal-anced classification problems. In the case of binary classifica-tion, unbalanced problem means that the number of positive

200

samples is much larger than that of negative samples, or viceversa. It is known that standard SVM algorithm cannot handlesuch kind of problems well. Proposed AdaBoostSVM algo-rithm includes many moderately accurate SVM weak learners,some of which can focus on the misclassified samples of thenon-dominant class. By combining these moderately accurateSVM classifiers, AdaBoostSVM performs better than standardSVM on unbalanced problems. In the following experiments,we compare the performance of proposed AdaBoostSVM andstandard SVM on unbalanced data sets. We used the 'Splice'data set of the UCI benchmark. Splice data set has 483 positivetraining samples, 517 negative training samples and 2175 testsamples. In the following experiments, the number of negativesamples is fixed as 500 and the number of positive samples isreduced from 150 to 30. From Figure. 4, it can be found thatalong with the decreasing ratio of the number positive samplesto that of negtive samples, the improvement of AdaBoostSVMover SVM becomes more and more. When the ratio reaches30:500, SVM almost cannot work and performs like randomguess, while proposed AdaBoostSVM can still work witha relatively good performance. The good performance ofensemble methods to handle unbalanced problems was alsoobserved in [19] [20].

90%

85/& S AdaBoostSVMa m

80%-X \

"75% , ,,

670% \-+

EqZ65%

60% \

55%-\

:28500 100:500 80:5X0 60 500 30:500Number of postive samples: Number of negtive samples

Fig. 4. Compare SVM and AdaBoostSVM on unbalanced problem

VI. CONCLUSION AND DISCUSSION

AdaBoost with SVM weak learners is proposed in thispaper, which is achieved by adaptively adjusting the kernelparameter to get a set of effective weak learners. Experi-ments on benchmark data sets demonstrated that proposedAdaBoostSVM performs better than AdaBoost with neuralnetworks weak learners. In addition, it needs not accuratemodel selection and thus saves computational cost. Basedon AdaBoostSVM, an improved version is further developedto deal with the accuracy/diversity dilemma, and promisingresult is obtained. Besides these, it is found that proposedAdaBoostSVM algorithm has much better performance thanSVM on unbalanced problems. The reported in this paper is

a general framework of AdaBoost with SVM weak learners,which can be tailored for different purposes through changingthe parameter scales or selection criterion. In the future work,more theoretical analysis and comparison with AdaBoost withall kinds of weak learners will be conducted.

REFERENCES

[1] Robert E. schapire, "The boosting approach to machine learning: Anoverview," In MSRI Workshop on Nonlinear Estimation and Classifica-tion, 2002.

[2] Leo Breiman, "Bagging predictors," Machine Learning, vol. 24, pp.123-140, 1996.

[3] Eric Bauer and Ron Kohavi. "An empirical comparison of votingclassification algorithms: Bagging, boosting, and variants:' MachineLearning, vol. 36, no. 1, pp. 105-139, Jul 1999.

[4] Yoav Freund and Robert E. Schapire, "A decision-theoretic general-ization of on-line learning and an application to boosting," Jolrnal ofCfonputer and System Sciences, vol. 55(1), pp. 119-139, August 1997.

[5] R. E. Schapire, Y. Singer, P. Bartlett, and W. Lee, "Boosting the margin:A new explanation for the effectiveness of voting methods," Thte Annalsof Statistics, vol. 26, no. 5, pp. 1651-1686, 1998.

[6] Thomas G. Dietterich, "An experimental comparison of three methodsfor constructing ensembles of decision trees: Bagging, boosting, andrandomization," Machine Learning, vol. 40, no. 2, pp. 139-157, Aug2000.

[7] Holger Schwenk and Yoshua Bengio, "Boosting neural networks,"Nueral Cfonputtation, vol. 12, pp. 1869-1887, 2000.

[8] Gunnar Ratsch, "Soft margins for adaboost," Machine Learning, vol.42, no. 3, pp. 287-320, Mar 2001.

[9] Prem Melville and Raymond J. Mooney, "Creating diversity in ensem-bles using artificial data:' Injormnation Futsion, vol. 6, no. 1, pp. 99-111,Mar 2005.

[10] Ludmila 1. Kuncheva and Christopher J. Whitaker, "Measures ofdiversity in classifier ensembles and their relationship with the ensembleaccuracy:' Machine Learning, vol. 51, no. 2, pp. 181-207, May 2003.

[11] Wenxin Jiang, "Process consistency for adaboost:' Annals oJ Statistics,vol. 32, no. 1, pp. 13-29, 2004.

[12] Vladimir Vapnik, Statistical Learniing Theory, John Wiley and SonsInc., New York, 1998.

[13] Bernhard Scholkopf, Kah-Kay Sung, Chris Burges, Federico Girosi,Partha Niyogi, Tomaso Poggio, and Vladimir Vapnik, "Comparingsupport vector machines with gaussian kernels to radial basis functionclassifiers:' IEEE Transactions on Signal Processing, vol. 45, no. 11,pp. 2758-2765, 1997.

[14] Giorgio Valentini and Thomas G. Dietterich, "Bias-variance analysis ofsupport vector machines for the development of svm-based ensemblemethods:' Journal of Machtine Learning Research, vol. 5, pp. 725-775,Jul 2004.

[15] Robert E. Schapire and Yoram Singer, "Improved boosting algorithmsusing confidence-rated predictions," Machine Learning, vol. 37, no. 3,pp. 297-336, Dec 1999.

[16] H.W.Shin and S.Y.Sohn, "Selected tree classifier combination based onboth accuracy and error diversity:' Pattern Recognitiont, vol. 38, pp.191-197, 2005.

[17] Sanjoy Dasgupta and Philip M. Long, "Boosting with diverse baseclassifiers:' in Proceeding oJ the 16th Annual ConJerence on LearningTheory, Aug 2003, pp. 273-287.

[18] http://mlg.anu.edu.au/-raetsch/data.[19] Rong Yan, Yan Liu, Rong Jin, and Alex Hauptmann, "On predicting

rare class with svm ensemble in scene classification," in Proceedingof the IEEE International C'onjrence on Acoustics, Speech. and Signal2003, Apr 2003, pp. III - 21-4.

[20] Hyun-Chul Kim, Shaoning Pang, Hong-Mo Je, Daijin Kim, andSung Yang Bang, "Constructing support vector machine ensemble,"Pattern Recognition, vol. 36, no. 12, pp. 2757-2767, Dec 2003.

201