AutomaticAircraftRecognition usingDSmTandHMMXin-deLiJin-dong PanJean DezertAbstractIn this paper we propose a new method for solvingthe Automatic Aircraft Recognition (AAR) problemfromasequence of images of an unknown observed aircraft. Our methodexploits the knowledge extracted from a training image data set(a set of binary images of different aircrafts observed under threedifferent poses) with the fusion of information of multiple featuresdrawn fromthe image sequence using Dezert-SmarandacheTheory(DSmT)coupledwithHiddenMarkovModels(HMM).The rst stepof the methodconsists for eachimage of theobserved aircraft to compute both Hus moment invariants (therst features vector) and the partial singular values of the outlineoftheaircraft(thesecondfeaturesvector). Inthesecondstep,we use a probabilistic neural network (PNN) based on thetrainingimagedatasettoconstructtheconditional basicbeliefassignments (BBAs) of the unknown aircraft type within the setofapredenedpossibletargettypesgiventhefeaturesvectorsand pose condition. The BBAs are then combined altogether bythe Proportional Conict Redistribution rule #5 (PCR5) of DSmTtogetaglobal BBAaboutthetargettypeunderagivenposehypothesis. These sequential BBAs give initial recognition resultsthat feed a HMM-based classier for automatically recognizingtheaircraft inamultipleposes context. Thelast part of thispapershowstheeffectivenessof thisnewSequential Multiple-Features Automatic Target Recognition (SMF-ATR) method withrealistic simulation results. This method is compliant with real-time processing requirement for advanced AAR systems.Keywords: Information fusion; DSmT; ATR; HMM.I. INTRODUCTIONATR(AutomaticTarget Recognition) systemsplayama-jor roleinmodernbattleeldfor automaticmonitoringanddetection, identicationandfor precisionguidedweaponaswell. The Automatic Aircraft Recognition (AAR) problem isasubclass of theATRproblem. Manyscholars havemadeextensiveexplorationsforsolvingATRandAARproblems.The ATR method is usually based on target recognition usingtemplatematching[1],[2]andsinglefeature(SF)extraction[3][7] algorithms. Unfortunately, erroneous recognition oftenoccurswhenutilizingtargetrecognitionalgorithmsbasedonsingle feature only, specially if there exist important changes inpose and appearance of aircrafts during ight path in the imagesequence. In such condition, the informational content drawnfromsinglefeaturemeasurescannot helpenoughtomakeareliable classication. To overcome this serious drawback, newATRalgorithmsbasedonmultiplefeatures(MF)andfusiontechniqueshavebeenproposed[8][12]. AninterestingMF-ATRalgorithmbasedonBack-PropagationNeural Network(BP-NN), and Dempster-Shafer Theory (DST) of evidence[23] has been proposed by Yang et al. in [11] which has beenpartly the source of inspiration to develop our new improvedsequential MF-ATRmethodpresentedhere andintroducedbrieyin[12](inchinese). Inthispaperwewill explainindetails how our new SMF-ATR method works and we evaluateits performances on a typical real image sequence.Although MF-ATR approach reduces the deciency of SF-ATRapproachingeneral, therecognitionresultscansome-timesstillbeindeterminateformasingleimageexploitationbecausetheposeandappearanceof different kinds of air-craftscanbeverysimilarforsomeinstantaneousposesandappearances. To eliminate (or reduce) uncertainty and improvethe classication, it is necessary to exploit a sequence ofimages of theobservedaircraft duringits ight anddevel-opefcient techniques of sequential informationfusionforadvanced(sequential)MF-ATRsystems. Twopioneerworkson sequential ATRalgorithms using belief functions (BF)havebeenproposedinlast years. In2006, Huanget al. in[13] have developeda sequential ATRbasedonBF, Husmoment invariants(forimagefeaturesvector), aBP-NNforpatternclassication, andamodiedDempster-Shafer (DS)fusionrule1. ASF-ATRapproachusingBF, Hus momentinvariants, BP-NNandDSmTrulehas alsobeenproposedin [14] the same year. In these papers, the authors did clearlyshowthebenetoftheintegrationoftemporalSFmeasuresfor the target recognition, but the performances obtained werestill limitedbecauseof largepossiblechangesinposesandappearances of observed aircrafts (specially in high maneuvermodes as far as militaryaircrafts are under concern). Thepurpose of this paper is to develop a new (sequential) MF-ATRmethodabletoprovideahighrecognitionratewithagoodrobustness when face to large changes of poses and ppearancesof observed aircraft during its ight.The general principle of our SMF-ATR method is shown onFig.1. The upper part of Fig. 1 consists in Steps 1 & 2, whereasthelowerpartofFig.1consistsinSteps3&4respectivelydescribed as follows: Step1(Featuresextraction) : Weconsider andextractonlytwofeaturesvectorsinthiswork2(Husmoment1called the abortion method by the authors.2The introduction of extra features is possible and under investigations.Originally published as Li X.-D., Pan J.-D., Dezert J., Automatic Aircraft Recognition using DSmT and HMM, in Proc. of Fusion 2014 Int Conf onInformation Fusion, Salamanca, Spain, July 7-10, 2014, and reprinted with permission.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4441Probabilistic Neural Network no 1Probabilistic Neural Network no 2BBA fusionand decisionDecisionFig. 1: General principle of our sequential MF-ATR approach.invariants vector, and Singular Values Decomposition(SVD) features vector) from the binary images3 Step 2 (BBAs construction4) : For every image in the se-quence and from their two features vectors, two BayesianBBAs on possible (target type,target pose) are computedfromthe results of two PNNs trained on the imagedataset. The methodof BBAconstructionis differentfrom the one proposed in [12]. Step3(BBAscombination):Foreveryimage, saythek-thimage, inthesequence, thetwoBBAsof Step2arecombinedwiththePCR5fusionrule, fromwhichadecisionOkonthemost likelytarget typeandposeisdrawn. Step 4 (HMM-based classier) : Fromthe sequenceOK= {O1, . . . , Ok . . . , OK} ofKlocal decisions com-puted at Step 3, we feed several HMM-based classiersinparallel (eachHMMcharacterizes eachtarget type)and we nd nally the most likely target observed in theimage sequence which gives the output of our SMF-ATRapproach.Thenext sectionpresents eachstepof this newSMF-ATRapproach. Section3evaluatestheperformancesof thisnewmethod on real image datasets. Conclusions and perspectivesof this work are given in Section 4.II. THE SEQUENTIAL MF-ATR APPROACHIn this section we present the aforementioned steps neces-sary for the implementation of our new SMF-ATR method.3In this work, we use only with binary images because our image trainingdataset containsonlybinaryimageswithcleanbackgrounds, andworkingwithbinaryimagesiseasier todoandrequireslesscomputational burdenthan working with grey-level or color images. Hence it helps to satisfy real-time processing. The binarization of the images of the sequence under analysisis done with the the Flood Fill Method explained in details in [22] using thepoint of the background as a seed for the method.4The mathematical denition of a BBA is given in Section II-C.A. Step 1: Features extraction from binary imageBecause Aircraft poses in a ight can vary greatly, we needimagefeaturesthat arestableandremainunchangedundertranslation, rotation and scaling. In terms of aircraft features,twocategoriesarewidelyused: 1) moment featuresand2)contour features. Image moments have been widely used sincea long time specially for pattern-recognition applications [16].Moment features which are the descriptions of image regionalcharacteristics are mainly obtained from the intensity of eachpixel of target image. Contour features are extracted primarilyby discretizing the outline contour and they describe thecharacteristic of the outline of the object in the image. In termsof moment features, Hus moment invariants [6] are used here.As contour features, we use the SVD [15] of outlines extractedfrom the binary images. Hus momentsTwo-dimensional (p+q)-th order moments for p, q =0, 1, 2, ... of an image of sizeM Nare dened as follows:mpq M

m=1N

n=1mpnqf(m, n) (1)wheref(m, n) is the value of the pixel (m, n) of the binaryimage. Note thatmpqmay not be invariant whenf(m, n) bytranslation, rotatingorscaling. Theinvariantfeaturescanbeobtainedusingthe(p + q)-thordercentralmomentspqforp, q= 0, 1, 2, ... dened bypq M

m=1N

n=1(m x)p(n y)qf(m, n) (2)where x, and yare the barycentric coordinates of image (i.e.the centroidof the image). These values are computedby x=m10m00=1C

Mm=1

Nn=1 m f(m, n)and y=m01m00=1C

Mm=1

Nn=1 n f(m, n), where Cis a normalizationconstant givenbyC=m00=

Mm=1

Nn=1 f(m, n). Thecentroid moments pq is equivalent to the mpq moment whoseAdvances and Applications of DSmT for Information Fusion. Collected Works. Volume 4442center has been shifted to the centroid of the image. Therefore,pq are invariant to image translations. Scale invariance is ob-tained by normalization [6]. The normalized central momentspqare dened forp + q=2, 3, . . . bypq pq/00, with= (p+q+2)/2. Based on these normalized central momentsHu in [16] derived seven moment invariants that are unchangedunder image scaling, translation and rotation as follows120 + 022(20 02)2+ 42113(30 312)2+ (321 03)24(30 + 12)2+ (21 + 03)25(30 312)(30 + 12)[(30 + 12)23(21 + 03)2]+ (321 03)(21 + 03)[3(30 + 12)2(21 + 03)2]6(20 02)[(30 + 12)2(21 + 03)2]+ 411(30 + 12)(21 + 03)7(321 03)(30 + 12)[(30 + 12)23(21 + 03)2](30 312)(21 + 03)[3(30 + 12)2(03 + 21)2]In this work, we use only the four simplest Hus moments tocompute, thatis=[1234], tofeedtherstPNNof our sequential MF-ATR method5. SVD features of the target outlineTheSVDis widelyappliedsignal andimageprocessingbecauseit is anefcient tool tosolveproblems withleastsquares method [21]. The SVD theorem states that ifAmnwithm>n(representinginourcontexttheoriginalbinarydata) is a real matrix6, then it can be written using a so-calledsingular value decomposition of the formAmn= UmmSmnVTnnwhere Ummand Vnnare orthogonal7matrices. Thecolumns ofU are the left singular vectors.VThas rows thatare the right singular vectors. The real matrix S has the samedimensions asA and has the form8Smn=

Srr0r(nr)0r(mr)0(mr)(nr)

whereSrr=Diag{1, 2, . . . , r}with1 2, . . . r> 0 and1 r min(m, n).Calculating the SVD consists of nding the eigenvalues andeigenvectorsof AATandATA. Theeigenvectorsof ATAmakeupthecolumnsofV, theeigenvectorsofAATmakeup the columns ofU. The singular values1,. . . , rare thediagonal entriesof Srrarrangedindescendingorder, andthey are square roots of eigenvalues fromAATorATA.A method to calculate the set of discrete points{a1, a2, . . . , an} of a target outline froma binary imageis proposedin[17]. The SVDfeatures are thencomputed5It istheoreticallypossibletoworkwithall sevenHusmomentsinourMF-ATR method, but we did not test this yet in our simulations.6For a complex matrix A, the singular value decomposition is A=USVH, whereVHis the conjugate transpose ofV.7They verifyUTmmUmm= ImandVTnnVnn= Inn, whereImm and Inn are respectively the identity matrices of dimensions mmandn n.80pqis ap qmatrix whose all its elements are zero.fromtheeigenvalues of thecirculant matrixbuilt fromthediscretizedshapeof theoutlinecharacterizedbythevectord=[d1, d2, . . . , dn]wherediisthedistanceofthecentroidof the outline to the discrete pointsai, i = 1, 2, . . . , n of theoutline.In our analysis, it has been veried fromour imagedataset that only the rst components of SVD features vector=[1, 2, . . . , r] takeimportant values withrespect tothe other ones. The other components of tend quicklytowards zero. Therefore only few rst components ofplayanimportant roletocharacterizethemainfeaturesoftargetoutline. However, ifoneconsidersonlythesefewmainrstcomponentsof , onefailstocharacterizeefcientlysomespecicfeatures(details) of thetarget prole. Bydoingso,onewouldlimit theperformancesofATR. That iswhywepropose to use the partial SVDs of outline as explained in thenext paragraph.TocapturemoredetailsofaircraftoutlinewithSVD, onehas to taken into account also additional small singular valuesof SVD. This is done with the following procedure issued fromthe face recognition research community [24]. The normalizeddistance vectord = [d1,d2, . . . ,dn] is built fromdbytakingd=[1, d2/d1, . . . , dn/d1], where d1isthedistancebetweenthecentroidof outlineandtherst chosenpointsof thecontour of theoutlineobtainedbyaclassical9edgedetector algorithm. To capture the details of target outline andtoreducethecomputational burden, oneworkswithpartialSVDs of theoriginal outlinebyconsideringonlyl slidingsub-vectorsdwofd,wherewisthenumberofcomponentsofdw. For exampleif onetakes w=3pointsonlyinthesub-vectors andifd=[d1,d2, . . . ,d9], thenonewill takethe sub-vectorsd1w=[d1,d2,d3],d2w=[d4,d5,d6] andd3w=[d7,d8,d9] if wedont useoverlappingcomponentsbetween sub-vectors. Fromthe sub-vectors, one constructstheircorrespondingcirculant matrixandapplytheirSVDtoget partial SVD features vectors l=1w, l=2w, etc. The numberlof partial SVD of the original outline of the target is givenbyl =(n w)/(w m) + 1, wheremisthenumber ofcomponents overlapped by each two adjacent sub-vectors, andn is the total number of discrete contour points of the outlinegiven by the edge detector.B. Step 2: BBAs construction with PNNsIn order to exploit efciently fusion rules dealing withconictinginformationmodeledbybeliefmassassignments(BBAs) [18], [23], we need to build BBAs from all featurescomputedfromimagesofthesequenceunderanalysis. Theconstruction of the BBAs needs expert knowledge or knowl-edgedrawnfromtrainingusingimagedataset.Inthispaper,we propose to utilize probabilistic neural networks (PNN)initially developed in nineties by Specht [19] to construct theBBAsbecauseit isacommontechniqueusedinthetargetrecognition and pattern classication community that is able to9Inthiswork, weusethecvcontourfunctionofopencvsoftware[22]toextract the target outline from a binary image.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4443achieve with large training dataset performances close to thoseobtained by a human expert in the eld. The details of PNNssettingsfor BBAsconstructionaregivenin[12]. However,because the neural network after training to some extent has agood discriminant ability (close to an expert in the eld), theBBAisconstructedbytheneuralnetworkdirectlybasedonthe PNNs output, which is different from the construction ofthe BBA based on the confusion matrix described in [12].HerewepresenthowthetwoPNNs(showninFigure1)work. Inourapplication, wehaveNc=7typesofaircraftsinour trainingimagedataset. For eachtype, theaircraft isobserved withNp= 3 poses. Therefore we haveNcp= Nc Np= 21 types of distinct cases in our dataset. For each case,one hasNi= 30 images available for the training. Thereforethewholetrainingdataset containsNcpi=NcNpNi=7 3 30 = 630 binary images. For the rst PNN (fed by Husfeatures vector), the number of input layer neurons is 4 becausewe use only = [1, 2, 3, 4] Hus moment invariants inthis work. For the second PNN (fed by partial SVD featuresvector), thenumber of input layer neurons is constant andequal tol wbecausewetake l windowswiththewidthw(soonehas wsingular values of partial SVDfor everywindow). The number of hidden layer neurons of each PNN isthe number of the training samples,Ncpi= 630. The numberof output layer neurons is equal toNcp= 21 (the number ofdifferent possible cases).Our PNNsfedbyfeaturesinput vectors(Husmomentsand SVD outline) do not provide a hard decision on the typeand pose of the observed target under analysis because in ourbelief-based approach we need to build BBAs. Therefore thecompetitionfunctionoftheoutputlayerfordecision-makingimplementedclassicallyinthePNNschemeis not usedintheexploitation10phaseof our approach. Instead, thePNNcomputes the Ncp Ni(Euclidean) distances betweenthefeaturesvectorsoftheimageundertestandtheNcpi=630features vectors of the training dataset. AGaussian radialbasis function(G-RBF) is usedinthehiddenlayer of thePNNs [19] to transform its input (Euclidean) distance vectorofsize1 Ncpiintoanother 1 Ncpidistance(similarity)that feeds the output layer through a weighting matrix of sizeNcpiNcp= 63021 estimated from the training samples. Asa nal output of each PNN, we get an unnormalized similarityvector mof size (1Ncpi)(NcpiNcp) = 1Ncp= 121which is then normalized to get a Bayesian BBA on the frameof discernment ={(targeti, posej), i =1, . . . , c, j =1, . . . , p}. Because we use only two11PNNs in this approach,weareabletobuildtwoBayesianBBAs m1(.)andm2(.)dened on the same frame for every image of the sequenceto analyze.C. Step 3: Fusion of BBAs and local decisionA basic belief assignment (BBA), also called a (belief) massfunction, is a mappingm(.) : 2 [0; 1] such thatm() = 010when analyzing a new sequence of an unknown observed aircraft.11A rst PPN fed by Hus features, and a second PNN fed by SVD outlinefeatures see Fig. 1.and X2 m(X)=1, whereistheso-calledframeofdiscernmentoftheproblemunderconcernwhichconsistsofanitediscretesetofexhaustiveandexclusivehypotheses12i, i = 1, . . . , n, and where 2is the power-set of (the set ofall subsets of). This denition of BBA has been introducedinDempster-Shafer Theory(DST) [23]. Thefocal elementsof aBBAareall elements Xof 2suchthat m(X) >0.Bayesian BBAs are special BBAs having only singletons (i.e.the elements of) as focal elements.InDST, thecombinationofBBAsisdonebyDempstersrule of combination [23] which corresponds to the normalizedconjunctiveconsensus operator. Becausethis fusionruleisknowntobenotsoefcient(bothinhighlyandalsoinlowconicting) in some practical situations [25], many alternativerules have been proposed during last decades [18], Vol. 2.Toovercome the practical limitations of Shafers modelandinorder todeal withfuzzyhypotheses of the frame,Dezert and Smarandache have proposed the possibility toworkwithBBAs denedonDedekinds lattice13D[18](Vol.1) so that intersections (conjunctions) of elements of theframecanbeallowedinthefusionprocess, witheventuallysome given restrictions (integrity constraints). Dezert andSmarandache have also proposed several rules of combinationbased on different Proportional Conict Redistribution (PCR)principles. Among these new rules, the PCR5 and PCR6 rulesplay a major role because they do not degrade the specicity ofthe fusion result (contrariwise to most other alternative rule),and they preserve the neutrality of the vacuous BBA14. PCR5and PCR6 provide same combined BBAwhen combiningonly two BBAs m1(.) and m2(.), but they differ whencombining three (or more) BBAs altogether. It has beenrecently proved in [26] that PCR6 is consistent with empirical(frequentist) estimationof probabilitymeasure, unlikeotherfusionrules15.Thesetwomajor differenceswithDST, makethe basis of Dezert-Smarandache Theory (DSmT) [18].Inthecontext of thiswork, weproposetousePCR5tocombine the two (Bayesian) BBAs m1(.) and m2(.) built fromthetwoPNNsfedbyHusfeaturesvectorandSVDoutlinefeatures vector. Because for each image of the observed targetin the sequence, one has only two BBAs to combine, the PCR5fusionresult issameasthePCR6fusionresult. Of course,ifonewantstoincludeotherkindsoffeaturesvectorswithadditional PNNs, the PCR6 fusion rule is recommended. ThePCR principle consists in redistributing the partial conictingmasses16onlytothesetsinvolvedintheconictandpropor-tionallytotheirmass. ThePCR5(orPCR6)combinationof12This is what is called Shafers model of the frame in the literature.13Dedekinds lattice is the set of all composite subsets built from elementsof with and operators.14A vacuous BBA is the BBA such thatm() = 1.15except the averaging rule.16For two BBAs, a partial conicting mass is a product m1(X)m2(Y ) >0 of the elementX Ywhich is conicting, that is such thatX Y= .Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4444two BBAs is done according to the following formula17[18]mPCR5/6(X) =

X1,X22X1X2=Xm1(X1)m2(X2)+

Y 2\{X}XY =[m1(X)2m2(Y )m1(X) + m2(Y )+m2(X)2m1(Y )m2(X) + m1(Y )] (3)whereall denominators in(3) aredifferent fromzero, andmPCR5/6() =0. If a denominator is zero, that fractionis discarded. All propositions/sets areinacanonical form.Because we work here only with Bayesian BBAs, the previousfusionformulaisinfact rathereasytoimplement, see[18](Vol. 2, Chap. 4).In summary, the target features extraction in a sequence ofKimages allows us to generate, after Step 3, a set of BBAs{mImagek(.), k =1, 2, . . . , K}. EveryBBAmImagek(.) isobtainedbythe PCR5/6fusionof BBAs mImagek1(.) andmImagek2(.)built fromtheoutputsoftwoPNNs. FromthiscombinedBBA, alocal18decisionOkcanbedrawnaboutthe target type and target pose inImagekby taking the focalelement ofmImagek(.) having the maximum mass of belief.D. Step 4: Hidden Markov Model (HMM) for recognitionUsually(andspeciallyinmilitarycontext), thepostureofan aircraft can continuously change a lot during its ightpathmakingtarget recognitionbasedonlyonsingleobservation(image) verydifcult, becausesomeambiguities canoccurbetweenextractedfeatureswiththosestoredinthetrainingimage data set. To improve the target recognition performanceandrobustness, oneproposestousethesequenceof targetrecognition decision Ok drawn from BBAs {mImagek(.), k =1, 2, . . . , K} to feed HMM classiers in parallel. We suggestthis approach because the use of HMMhas already beenproved to be very efcient in speech recognition, naturallanguage and face recognition. We briey present HMM, andthen we will explain howHMMs are used for automaticaircraft recognition.Let us consider a dynamical system with a nite set of pos-sible statesS= {s1, s2, . . . , sN}. The state transitions of thesystem is modeled by a rst order Markov chain governed bythetransitionprobabilitiesgivenbyP(s(tk)=sj|s(tk1)=si, s(tk2)=sk, . . .)=P(s(tk)=sj|s(tk1)=si)=aij,wheres(tk)istherandomstateofthesystemattimetk. AHMM is a doubly stochastic processes including an underlyingstochastic process (i.e. a Markov chain for modeling the statetransitions of thesystem), andasecondstochasticprocessfor modeling the observation of the system(which is afunction of the random states of the system). A HMM, denoted = (A, B, ), is fully characterized by the knowledge of thefollowing parameters17Here we assume that Shafers model holds. The notation mPCR5/6means PCR5 and PCR6 are equivalent when combining two BBAs.18because it is based only on a single image of the unknown observed targetin the sequence under analysis.1) The numberNof possible statesS= {s1, s2, . . . , sN}of the Markov chain.2) Thestatetransitionprobabilitymatrix19A=[aij] ofsizeN N, whereaijP(s(tk) = si|s(tk1) = sj).3) Thepriormassfunction(pmf)oftheinitialstateofthe chain, that is = {1, . . . , N} with Ni=1 i= 1,wherei= P(s(t1) = si).4) Thenumber MofpossiblevaluesV = {v1, . . . , vM}taken by the observation of the system.5) The conditional pmfs of observed values given the statesof the system characterized by the matrixB = [bmi] ofsize M N, withbmiP(Ok=vm|s(tk) =si),where Ok is the observation of the system (i.e. the localdecision on target type with its pose) at timetk.Inthisworkweconsider aset of NcHMMsinparallel,whereeachHMMisassociatedwithagiventypeof targetto recognize. We consider the following state and observationmodels in our HMMs:- Statemodel: For agiventypeof aircraft, weconsider anite set of distinct aircraft postures available in our trainingimage dataset. In our application, we consider only three statescorrespondingtos1=top view, s2=side viewands3=front view as shown (for a particular aircraft) in Figure 2.Fig. 2: Example of HMM states.-Observationmodel:InourHMMs, weassumethat eachstate (posture) of aircraft is observable. Since we haveonly Np= 3 states S ={s1, s2, s3} for each aircraft,and we have Nc= 7 types of aircrafts in the trainingdataset, wehavetodeal withNcp=3 7=21possible20observations(local decisions)at eachtimetk. Asexplainedpreviously, at the endof Step3we have a set of BBAs{mImagek(.), k = 1, 2, ..., K} that helps to draw the sequenceof local decisions OK {O1, . . . , Ok, . . . , OK}. Thissequenceof decisions(calledalsorecognitionobservations)isusedtoevaluatethelikelihoodP(OK|i)ofthedifferentHMMs described by the parameter i= (Ai, Bi, i),i =1, 2, . . . , Nc. Thecomputationof theselikelihoodswillbe detailed at the end of this section. The nal decisionfor ATRconsists to infer the true target type based onthemaximumlikelihoodcriterion. Moreprecisely, onewilldecide that the target type isi

ifi

= arg maxi P(OK|i). Estimation of HMM parametersTo make recognition with HMMs, we need at rst to denea HMMfor each type of target one wants to recognize.More precisely, we needtoestimate the parameters i=19We assume that the transition matrix is known and time-invariant, i.e. allelementsaijdo not depend ontk1andtk.20We assume that the unknown observed target type belongs to the set oftypes of the dataset, as well as its pose.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4445(Ai, Bi, i), wherei =1, . . . , Ncisthetarget typeinthetraining dataset. The estimationof HMM parameters is donefromobservationsequencesdrawnfromthetrainingdatasetwith Baum-Welch algorithm [20] that must be initialized witha chosen value 0i= (A0i, B0i, 0i). This initial value is chosenas follows:1) State prior probabilities 0ifor a target of type i: For eachHMM, we consider only three distinct postures (states)s1,s2ands3for theaircraft. Weuseauniformprior probabilitymassdistributionforalltypesoftargets. Therefore, wetake0i=[1/3, 1/3, 1/3] for anytarget type i =1, . . . , Nctorecognize.2) StatetransitionmatrixA0iof atarget of type i: Thecomponents apq of the state transition matrix A0iare estimatedfrom the analysis of many sequences21of targeti as followsapq=

K1k=1(s(tk), sp) (s(tk+1), sq)

K1k=1(s(tk), sp)(4)where Npis the number of states of the Markov chain,(x, y) is the Kronecker delta function dened by (x, y) = 1if y = x, and (x, y) = 0 otherwise, and where Kisthe number of images in the sequence of target i avail-able in the training phase. For example, if in the train-ing phase and for a target of type i = 1, we have thefollowing sequence of (target type, pose) cases given by[(1, 1), (1, 1), (1, 2), (1, 1), (1, 3), (1, 1), (1, 1)],thenfromEq.(4) withK= 7, we get22A0i=1=2/4 1/4 1/41 0 01 0 03) Observation matrix B0ifor a target of type i: Theinitial observation matrix B0iis given by the confusion matrixlearnt from all images of the training dataset. More precisely,fromeveryimage of the trainingdataset, we extract Husfeaturesandpartial SVDoutlinefeaturesandwefeedeachPNNtoget twoBBAs accordingtoSteps 1-3. Fromthecombined BBA, we make the local decision (targeti, posej) ifm((targeti, posej)) is bigger than all other masses of beliefof theBBA. Thisprocedureisappliedtoall imagesinthetrainingdataset. Bydoingso, we canestimate empiricallythe probabilities todecide (targeti, posej) whenreal case(targeti , posej ) occurs. So we have an estimation of all com-ponents of the global confusion matrixB0= [P(decision =(targeti, posej) | reality =(targeti , posej ))]. FromB0we extract the c sub-matrices (conditional confusion matrices)B0i, i = 1, . . . , Nc by taking all the rows of B0correspondingtothetarget of typei. Inourapplication, onehas Nc=7types and Np= 3 postures (states) for each target type, henceone hasNcp=7 3=21 possibles observations. Thereforethe global confusion matrixB0has size21 21 is the stackof Nc=7sub-matrices B0i, i =1, ..., Nc, eachof sizeNp Ncp= 3 21.21The video streamof different (known) aircraft ights generate thesequences of images to estimate approximatelyapq22One veries that the probabilities of each raw of this matrix sum to 1. Exploitation of HMM for ATRGiven a sequence OKof Klocal decisions drawn from thesequence of K images, and given Nc HMMs characterized bytheir parameter i (i = 1, . . . , Nc), one has to compute all thelikelihoods P(OK|i), and then infer from them the true targettype based on the maximum likelihood criterion which is doneby deciding the target type i

if i

= arg maxi P(OK|i). Thecomputation ofP(OK|i) is done as follows [20]: generation of all possible state sequences of lengthK, SKl=[sl(t1)sl(t2) . . . sl(tK)], where sl(tk) S(k=1,. . . , K) andl = 1, 2, . . . , |S|K computationof P(OK|i)byapplyingthetotal proba-bility theorem as follows23P(SKl|i) = sl(t1)asl(t1)sl(t2). . .asl(tK1)sl(tK)(5)P(OK|i, SKl) = bsl(t1)O1 bsl(t2)O2 . . . bsl(tK)OK(6)P(OK|i) =|S|K

l=1P(OK|i, SKl)P(SKl|i) (7)III. SIMULATIONS RESULTSFor thesimulations of SMF-ATRmethod, wehaveusedNc= 7 types of aircrafts in the training image dataset. Eachimage of the sequence has1200 702 pixels. The sequencesof aircraft observations in the training dataset take 150 frames.The Np=3 poses of every aircraft is shown in Fig. 3.For evaluatingour approach, wehaveusedsequences (testsamples) of images of 7 different aircraft, more preciselythe Lockheed-F22, Junkers-G.38ce, Tupolev ANT 20 MaximeGorky, CaspianSeaMonster(KaspianMonster), Mirage-F1,Piaggio P180, and Lockheed-Vega, ying under conditions thatgeneratealot ofstate(posture)changesintheimages. Thenumber of theimagesineachsequencetotest variesfrom400to500. TheshapingparameteroftheG-RBFofPNNshas been set to 0.1. The simulation is done in two phases: 1)the training phase (for training PNNs and estimating HMMsparameters), and 2) the exploitation phase for testing the realperformances of the SMF-ATR with test sequences.A - Performances evaluationIn our simulations, we have tested SMF-ATRwith twodifferent fusionrules: 1) thePCR5rule(seeSectionII-C),and 2) Dempster-Shafer (DS) rule24[23]. The percentages ofsuccessful recognition(i.e. therecognitionrateRi)obtainedwiththesetwoSMF-ATRmethodsareshowninTableIforeachtypei =1, 2, . . . , Ncofaircraft. TheperformancesoftheseSMF-ATRversions aregloballyverygoodsinceoneis abletorecognizewithaminimumof 85.2%of successthetypesof aircraft includedintheimagesequencesundertest when using DS-based SMF-ATR, and with a minimum of23The indexi of components of Aiand Bimatrices has been omitted fornotation convenience in the last two formulas.24Because Dempsters rule is one of the basis of Dempster-Shafer Theory,wecall prefer tocall it Dempster-Shafer rule, or just DSrule. This rulecoincides here with Bayesian fusion rule because we combine two BayesianBBAs and we dont use informative priors.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4446Fig. 3: Poses of different types of aircrafts.93.5% of success with the PCR5-based SMF-ATR. In term ofcomputational time, it takes between 5ms and 6ms to processeachimageinthesequencewithnoparticular optimizationof our simulationcode, whichindicatesthat thisSMF-ATRapproach is close to meet the requirement for real-time aircraftrecognition. It canbeobservedthat PCR5-basedSMF-ATRoutperformsDS-basedSMF-ATRfor3typesofaircraft andgives similar recognition rate as with DS-based SMF-ATR forother types. So PCR5-based SMF-ATR is globally better thanDS-based SMF-ATR for our application.Target type 1 2 3 4 5 6 7Ri(PCR5 rule) 95.7 93.5 96.3 98.2 96.3 98.5 97.3Ri(DS rule) 95.7 93.5 85.2 97.8 96.3 98.5 97.2TABLE I: Aircraft recognition ratesRi(in %).B - Robustness of SMF-ATR to image scalingTo evaluate the robustness of (PCR5-based) SMF-ATR ap-proach to image scaling effects, we did apply scaling changes(zoomout)ofZO=1/2, ZO=1/4andZO=1/8intheimages of the sequences under test. The performances of theSMF-ATR are shown in Table II. One sees that the degradationof recognition performance of SMF-ATR due to scaling effectsis very limited since even with a 1/8 zoom out one gets 90%of successful target recognition. The performance will declinesharply if the targets zoom out goes beyond 1/16.C - Robustness to compound typeTable III gives the performances of SMF-ATR on sequenceswithtwotypesoftargets(475imageswithtype1, and382images with type 2).Thetwoleft columnsofTableIIIshowtheperformancesTarget type 1 2 3 4 5 6 7Ri(no ZO) 95.7 93.5 96.3 98.2 96.3 98.5 97.3Ri(ZO=1/2) 95.0 92.0 95.2 94.7 96.1 96.6 95.4Ri(ZO=1/4) 95.0 92.0 94.7 91.7 93.6 91.6 95.7Ri(ZO=1/8) 95.0 92.2 93.1 89.3 93.6 94.5 90.7TABLEII:AircraftrecognitionratesRi(in%)of(PCR5/6-based) SMF-ATR with different zoom out values.Aircraft Single Single CompoundType 1 Type 2 TypeRi(SMF-ATR) 96.3 % 98.5% 97.3%TABLE III: Robustness to target compound.obtained when recognizing each type separately in each sub-sequence. The last column shows the performance whenrecognizingthecompoundtypeType 1 Type 2. Oneseesthat the performance obtained with compound type (97.3%) isclose to the weighted average2597.5% recognition rate. Thisindicates that no wide range of recognition errors occurs whenthe targets type change during the recognition process, makingSMF-ATR robust to target type switch.D - Performances with and without HMMsWe have alsocomparedthe performances of SMF-ATR,with two methods using more features but which do not exploitsequences of images with HMM. More precisely, the recogni-tion is done locally from the combined BBA for every imagewithouttemporalintegrationprocessingbasedonHMM.Wecall thesetwoMultipleFeaturesFusionmethodsMFF1andMFF2 respectively. In MMF1, one uses Hus moments, NMI(Normalized Moment of Inertia), afne invariant moments, andSVD of outline, PNN and PCR5 fusion, whereas MMF2 usessamefeatures as MMF1but withBPnetworkas classierand DS rule of combination. The recognition performances areshown in Table IV. One sees clearly the advantage to use theimage sequence processing with HMMs because of signicantimprovement ofATRperformances. TherecognitionrateofMFF2declinesseriouslybecausetheconvergenceoftheBPnetwork is not good enough.Target type 1 2 3 4 5 6 7Ri(SMF-ATR) 95.7 93.5 96.3 98.2 96.3 98.5 97.3Ri(MFF1) 89.2 92.0 91.2 86.9 92.2 93.5 95.0Ri(MFF2) 64.9 51.6 82.8 82.2 70.8 48.3 58.9TABLE IV: Performances (in %) with and without HMMs.E - SMF-ATR versus SSF-ATRWe have also compared in Table Vthe performancesSMF-ATRwith those of two simple SSF-ATR26methods,calledSSF1-ATRandSSF2-ATR. TheSSF1-ATRusesonlyHus moments features whereas SSF2-ATRuses onlySVDof outline as features. SSF1-ATRexploits image sequenceinformationusingBPnetworksasclassierandDSruleforcombination, while SSF2-ATR uses PNN and PCR5/6 rule.25According to the proportion of the two types in the whole sequence.26SSF-ATR stands for Single-feature Sequence Automatic Target Recogni-tion.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4447Target type 1 2 3 4 5 6 7Ri(SMF-ATR) 95.7 93.5 96.3 98.2 96.3 98.5 97.3Ri(SFF1-ATR) 39.3 42.3 74.3 56.7 60.1 33.9 44.3Ri(SFF2-ATR) 88.8 66.4 86.7 66.9 73.6 52.9 63.8TABLE V: Performances (in %) of SMF-ATR and SFF-ATR.OneclearlyseestheseriousadvantageofSMF-ATRwithrespect toSFF-ATRduetothecombinationof informationdrawn from both kinds of features (Hus and SVD of outline)extracted from the images.IV. CONCLUSIONS AND PERSPECTIVESA new SMF-ATR approach based on features extraction hasbeen proposed. The extracted features from binary images feedPNNs for building basic belief assignments that are combinedwithDSmTPCRruletomakealocal (basedononeimageonly)decisionontarget type. Theset oflocal decisionsac-quired over time for the image sequence feeds HMMs to makethe nal recognition of the target. The evaluation of this newSMF-ATRapproachhasbeendonewithrealisticsequencesofaircraft observations. SMF-ATRisabletoachievehigherrecognition rates than classical approaches that do not exploitHMMs, or SSF-ATR. Another complementary analysis of therobustness of SMF-ATR to target occultation is currently underprogressandwill bepublishedinaforthcomingpaper. Ourverypreliminaryresultsbasedonlyonfewsequencesindi-catethat SMF-ATRseemsveryrobust totarget occultationsoccurring randomly in single (non consecutive) images, but aneranalysisbasedonMonte-Carlosimulationwillbedoneto evaluate quantitatively its robustness in different conditions(number of consecutive occultations in the sequences, the levelof occultation, etc). Asinterestingperspectives, wewant toextend SMF-ATR approach for detecting new target types thatarenot includedinimagedataset. Also, wewouldwant todeal with the recognition of multiple crossing targets observedin a same image sequence.ACKNOWLEDGMENTThis work was supported by NNSF of China (No.60804063, 61175091), and Excellent Young Teacher Founda-tion sponsored by Qing Lan Project.REFERENCES[1] G. Marsiglia, L. Fortunato, A. Ondini, G. Balzarotti, Template matchingtechniques for automaticIRtarget recognitioninreal andsimulatedscenarios: testsandevaluations, InProc. ofSPIE: Target RecognitionXIII, Vol. 5094, pp. 159169, 2003.[2] O.K. Kwon, D.G. Sim, R.H. Park, Robust Hausdorff distance matchingalgorithmsusingpyramidalstructures,PatternReco.,Vol.34(10),pp.20052013, 2001.[3] S. Marouani, A. Huertas, G. Medioni, Model-based aircraft recognitionin perspective aerial imagery, In Proc. of Int. Symp. on Computer Vision,pp. 371376, Coral Gables, FL, USA, 1995.[4] S. Das, B. Bhanu, Asystemfor model-basedobject recognitioninperspective aerial images, Pattern Reco., Vol. 31 (4), pp. 465491, 1998.[5] J.F. Gilmore, Knowledge-based target recognition systemevolution,Optical Eng., Vol. 30 (5), pp. 557570, 1991.[6] S.A. Dudani, K.J. Breeding, R.B. McGhee, Aircraft identicationbymoment invariants, IEEE Trans. on Comp., Vol. 100, no. 1, 1977.[7] Y.-J. Du, H.-C. Zhang, Q. Pan, Three-dimensionalaircraftrecognitionusing moments, Journal of Data Acquisition and Processing, Vol. 15 (3),pp. 390394, 2000.[8] L. Gu, Z.-Q. Zhuang, G.-Y. Zheng, Z.-J. Wang, Algorithmfor handshapematchingbasedonfeaturefusion, ComputerApplications, Vol.25 (10), pp. 22862288, 2005.[9] C.-Q. Deng, G. Feng, Content-based image re-trieval using combinationfeatures, Computer Applications, Vol. 23 (7), pp. 100102, 2003.[10] L. Chen, J. Chen, Multi-feature fusion method based on support vectormachine and K-nearest neighbor classier, Computer Applications, Vol.29 (3), pp. 833835, 2009.[11] F.-P. Yang, Z.-X. Bai, TargetRecognitionMethodbasedonCombina-tionOf BPNeural Networkwith-EvidenceTheory, FireControl andCommand Control, Vol. 31 (10), pp. 8890, 2006.[12] X.-D. Li, W.-D.Yang, J. Dezert, AnAirplane Image Targets Multi-featureFusionRecognitionMethod, ActaAutomaticaSinica, Vol. 38(8), pp. 12981307, 2012.[13] J. Huang, Y. Liang, Y.-M. Cheng, Q. Pan, J.-W.Hu, Automatictargetrecognition method based on sequential images, J. of Computer Appli-cations, Vol. 27 (1), pp. 8793, 2006.[14] J. Hou, Z. Miao, Q. Pan, Intelligent target recognition method ofsequential imagesbasedonDSmT, J. ofComputerApplications, Vol.26 (1), pp. 120122, 2006.[15] S. Chen, J. Feng, Visual patternrecognitionbymoment invariants, inProc. of 2010 IEEE Int. Conf. on Inform. and Autom., Harbin, China,pp. 857860, 2010.[16] M.K. Hu, Visual pattern recognition by moment invariants, IRE Trans.on Information Theory, Vol. IT-8 (2), pp. 179187, 1962.[17] J.-D. Pan, X.-D. Li, Anairplanetarget recognitionmethodbasedonsingular value and PNN, Aero Weaponry, Vol. 1, pp. 4550, 2013.[18] F. Smarandache, J. Dezert, Advances andapplications of DSmTforinformationfusion(Collectedworks), AmericanResearchPress, USAVol.1, 2 & 3, 20042009. http://www.onera.fr/staff/jean-dezert?page=2[19] D.F. Specht, Probabilistic neural networks, Neural networks, Vol. 3, No.1, pp. 109118, 1990.[20] L.R. Rabiner, Atutorial on Hidden Markov Models and selectedapplicationsinspeechrecognition, Proc. oftheIEEE, Vol. 77, No. 2,pp. 257286, 1989. http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/[21] J.C. Nash, The Singular-Value DecompositionandIts Use toSolveLeast-Squares Problems, Ch. 3 in Compact Numerical Methods forComputers: Linear Algebra and Function Minimisation, 2nd ed. Bristol,England: Adam Hilger, pp. 3048, 1990.[22] G. Bradski, A. Kaehler, LearningOpenCV: ComputervisionwiththeOpenCV library, OReilly, pp. 124129, 2008.[23] G. Shafer, AMathematical theoryof evidence, PrincetonUniversityPress, Princeton, NJ, U.S.A., 1976.[24] L. Cao Lin, D.-F. Wang, X.-J. Liu, M.-Y. Zou, Face recognition based ontwo-dimensional Gabor wavelets, Journal of Electronics & InformationTechnology, Vol. 28, No. 3, pp. 490494, 2006.[25] J. Dezert, A. Tchamova, On the validity of Dempsters fusion rule anditsinterpretationasageneralizationofBayesianfusionrule, Int. J. ofIntel. Syst., Vol. 29, No. 3, Dec. , 2013.[26] F. Smarandache, J. Dezert, On the consistency of PCR6 with theaveraging rule and its application to probability estimation, in Proc. ofFusion 2013, Istanbul, Turkey, July 9-12, 2013.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4448