linear discriminant analysis : a detailed...
TRANSCRIPT
Line a r disc rimin a n t a n alysis : a d e t ailed t u to ri al
Ga b er, T, Tha r w a t , A, Ib r a hi m, A a n d H a s s a nie n, AE
h t t p://dx.doi.o r g/10.3 2 3 3/AIC-1 7 0 7 2 9
Tit l e Line a r di sc ri min a n t a n alysis : a d e t ailed t u to ri al
Aut h or s Gab er, T, Tha r w a t , A, Ib r a hi m, A a n d H a s s a nie n, AE
Typ e Article
U RL This ve r sion is available a t : h t t p://usir.s alfor d. ac.uk/id/e p rin t/52 0 7 4/
P u bl i s h e d D a t e 2 0 1 7
U SIR is a digi t al collec tion of t h e r e s e a r c h ou t p u t of t h e U nive r si ty of S alford. Whe r e copyrigh t p e r mi t s, full t ex t m a t e ri al h eld in t h e r e posi to ry is m a d e fre ely availabl e online a n d c a n b e r e a d , dow nloa d e d a n d copied for no n-co m m e rcial p riva t e s t u dy o r r e s e a r c h p u r pos e s . Ple a s e c h e ck t h e m a n u sc rip t for a ny fu r t h e r copyrig h t r e s t ric tions.
For m o r e info r m a tion, including ou r policy a n d s u b mission p roc e d u r e , ple a s econ t ac t t h e Re posi to ry Tea m a t : u si r@s alford. ac.uk .
1
Linear Discriminant Analysis: A DetailedTutorial
Alaa Tharwat ∗
Department of Computer Science and Engineering,Frankfurt University of Applied Sciences, Frankfurtam Main, GermanyFaculty of Engineering, Suez Canal University, EgyptE-mail: [email protected]
Tarek Gaber ∗
Faculty of Computers and Informatics, Suez CanalUniversity, EgyptE-mail: [email protected]
Abdelhameed Ibrahim ∗
Faculty of Engineering, Mansoura University, EgyptEmail: [email protected]
Aboul Ella Hassanien∗
Faculty of Computers and Information, CairoUniversity, EgyptE-mail: [email protected]∗Scientific Research Group in Egypt, (SRGE),http://www.egyptscience.net
Linear Discriminant Analysis (LDA) is a very commontechnique for dimensionality reduction problems as a pre-processing step for machine learning and pattern classifica-tion applications. At the same time, it is usually used as ablack box, but (sometimes) not well understood. The aim ofthis paper is to build a solid intuition for what is LDA, andhow LDA works, thus enabling readers of all levels be ableto get a better understanding of the LDA and to know how toapply this technique in different applications. The paper firstgave the basic definitions and steps of how LDA techniqueworks supported with visual explanations of these steps.Moreover, the two methods of computing the LDA space, i.e.class-dependent and class-independent methods, were ex-plained in details. Then, in a step-by-step approach, two nu-merical examples are demonstrated to show how the LDAspace can be calculated in case of the class-dependent andclass-independent methods. Furthermore, two of the mostcommon LDA problems (i.e. Small Sample Size (SSS) andnon-linearity problems) were highlighted and illustrated, andstate-of-the-art solutions to these problems were investigated
and explained. Finally, a number of experiments was con-ducted with different datasets to (1) investigate the effect ofthe eigenvectors that used in the LDA space on the robust-ness of the extracted feature for the classification accuracy,and (2) to show when the SSS problem occurs and how it canbe addressed.Keywords: Dimensionality reduction, PCA, LDA, KernelFunctions, Class-Dependent LDA, Class-Independent LDA,SSS (Small Sample Size) problem,, eigenvectors artificial in-telligence
1. Introduction
Dimensionality reduction techniques are importantin many applications related to machine learning [15],data mining [6,33], Bioinformatics [47], biometric [61]and information retrieval [73]. The main goal of the di-mensionality reduction techniques is to reduce the di-mensions by removing the redundant and dependentfeatures by transforming the features from a higher di-mensional space that may lead to a curse of dimension-ality problem, to a space with lower dimensions. Thereare two major approaches of the dimensionality reduc-tion techniques, namely, unsupervised and supervisedapproaches. In the unsupervised approach, there is noneed for labeling classes of the data. While in the su-pervised approach, the dimensionality reduction tech-niques take the class labels into consideration [32,15].
There are many unsupervised dimensionality reduc-tion techniques such as Independent Component Anal-ysis (ICA) [31,28] and Non-negative Matrix Factor-ization (NMF) [14], but the most famous technique ofthe unsupervised approach is the Principal ComponentAnalysis (PCA) [71,4,67,62]. This type of data reduc-tion is suitable for many applications such as visualiza-tion [40,2], and noise removal [70]. On the other hand,the supervised approach has many techniques such asMixture Discriminant Analysis (MDA) [25] and Neu-ral Networks (NN) [27], but the most famous techniqueof this approach is the Linear Discriminant Analysis(LDA) [50]. This category of dimensionality reduction
AI CommunicationsISSN 0921-7126, IOS Press. All rights reserved
2
techniques are used in biometrics [36,12], Bioinfor-matics [77], and chemistry [11].
The LDA technique is developed to transform thefeatures into a lower dimensional space, which max-imizes the ratio of the between-class variance to thewithin-class variance, thereby guaranteeing maximumclass separability [76,43]. There are two types of LDAtechnique to deal with classes: class-dependent andclass-independent. In the class-dependent LDA, oneseparate lower dimensional space is calculated for eachclass to project its data on it whereas, in the class-independent LDA, each class will be considered as aseparate class against the other classes [1,74]. In thistype, there is just one lower dimensional space for allclasses to project their data on it.
Although the LDA technique is considered the mostwell-used data reduction techniques, it suffers from anumber of problems. In the first problem, LDA fails tofind the lower dimensional space if the dimensions aremuch higher than the number of samples in the datamatrix. Thus, the within-class matrix becomes singu-lar, which is known as the small sample problem (SSS).There are different approaches that proposed to solvethis problem. The first approach is to remove the nullspace of within-class matrix as reported in [79,56]. Thesecond approach used an intermediate subspace (e.g.PCA) to convert a within-class matrix to a full-rankmatrix; thus, it can be inverted [35,4]. The third ap-proach, a well-known solution, is to use the regulariza-tion method to solve a singular linear systems [38,57].In the second problem, the linearity problem, if differ-ent classes are non-linearly separable, the LDA can-not discriminate between these classes. One solution tothis problem is to use the kernel functions as reportedin [50].
The brief tutorials on the two LDA types are re-ported in [1]. However, the authors did not show theLDA algorithm in details using numerical tutorials, vi-sualized examples, nor giving insight investigation ofexperimental results. Moreover, in [57], an overviewof the SSS for the LDA technique was presented in-cluding the theoretical background of the SSS prob-lem. Moreover, different variants of LDA techniquewere used to solve the SSS problem such as Di-rect LDA (DLDA) [22,83], regularized LDA (RLDA)[18,37,38,80], PCA+LDA [42], Null LDA (NLDA)[10,82], and kernel DLDA (KDLDA) [36]. In addition,the authors presented different applications that usedthe LDA-SSS techniques such as face recognition andcancer classification. Furthermore, they conducted dif-ferent experiments using three well-known face recog-
nition datasets to compare between different variantsof the LDA technique. Nonetheless, in [57], there isno detailed explanation of how (with numerical exam-ples) to calculate the within and between class vari-ances to construct the LDA space. In addition, the stepsof constructing the LDA space are not supported withwell-explained graphs helping for well understandingof the LDA underlying mechanism. In addition, thenon-linearity problem was not highlighted.
This paper gives a detailed tutorial about the LDAtechnique, and it is divided into five sections. Section 2gives an overview about the definition of the main ideaof the LDA and its background. This section begins byexplaining how to calculate, with visual explanations,the between-class variance, within-class variance, andhow to construct the LDA space. The algorithms of cal-culating the LDA space and projecting the data ontothis space to reduce its dimension are then introduced.Section 3 illustrates numerical examples to show howto calculate the LDA space and how to select the mostrobust eigenvectors to build the LDA space. While Sec-tion 4 explains the most two common problems of theLDA technique and a number of state-of-the-art meth-ods to solve (or approximately solve) these problems.Different applications that used LDA technique are in-troduced in Section 5. In Section 6, different packagesfor the LDA and its variants were presented. In Section7, two experiments are conducted to show (1) the influ-ence of the number of the selected eigenvectors on therobustness and dimension of the LDA space, (2) howthe SSS problem occurs and highlights the well-knownmethods to solve this problem. Finally, concluding re-marks will be given in Section 8.
2. LDA Technique
2.1. Definition of LDA
The goal of the LDA technique is to project the orig-inal data matrix onto a lower dimensional space. Toachieve this goal, three steps needed to be performed.The first step is to calculate the separability betweendifferent classes (i.e. the distance between the meansof different classes), which is called the between-classvariance or between-class matrix. The second step is tocalculate the distance between the mean and the sam-ples of each class, which is called the within-class vari-ance or within-class matrix. The third step is to con-struct the lower dimensional space which maximizesthe between-class variance and minimizes the within-
3
class variance. This section will explain these threesteps in detail, and then the full description of the LDAalgorithm will be given. Figures (1 and 2) are used tovisualize the steps of the LDA technique.
2.2. Calculating the Between-Class Variance (SB)
The between-class variance of the ith class (SBi) rep-resents the distance between the mean of the ith class(µi) and the total mean (µ). LDA technique searchesfor a lower-dimensional space, which is used to max-imize the between-class variance, or simply maxi-mize the separation distance between classes. To ex-plain how the between-class variance or the between-class matrix (SB) can be calculated, the following as-sumptions are made. Given the original data matrixX = x1,x2, . . . ,xN, where xi represents the ith sam-ple, pattern, or observation and N is the total num-ber of samples. Each sample is represented by M fea-tures (xi ∈ R M). In other words, each sample is repre-sented as a point in M-dimensional space. Assume thedata matrix is partitioned into c = 3 classes as follows,X = [ω1,ω2,ω3] as shown in Fig. (1, step (A)). Eachclass has five samples (i.e. n1 = n2 = n3 = 5), where nirepresents the number of samples of the ith class. Thetotal number of samples (N) is calculated as follows,N = ∑
3i=1 ni.
To calculate the between-class variance (SB), theseparation distance between different classes which isdenoted by (mi−m) will be calculated as follows:
(mi−m)2 = (W T µi−W T µ)2 =W T (µi−µ)(µi−µ)TW(1)
where mi represents the projection of the mean of theith class and it is calculated as follows, mi = W T µi,where m is the projection of the total mean of allclasses and it is calculated as follows, m = W T µ, Wrepresents the transformation matrix of LDA1, µi(1×M) represents the mean of the ith class and it is com-puted as in Equation (2), and µ(1×M) is the total meanof all classes and it can be computed as in Equation (3)[83,36]. Figure (1) shows the mean of each class andthe total mean in step (B and C), respectively.
µ j =1n j
∑xi∈ω j
xi (2)
1The transformation matrix (W ) will be explained in Sect. 2.4
µ =1N
N
∑i=1
xi =c
∑i=1
ni
Nµi (3)
where c represents the total number of classes (in ourexample c = 3).
The term (µi − µ)(µi − µ)T in Equation (1) repre-sents the separation distance between the mean of theith class (µi) and the total mean (µ), or simply it repre-sents the between-class variance of the ith class (SBi ).Substitute SBi into Equation (1) as follows:
(mi−m)2 =W T SBi W (4)
The total between-class variance is calculated as fol-lows, (SB = ∑
ci=1 niSBi ). Figure (1, step (D)) shows first
how the between-class matrix of the first class (SB1 ) iscalculated and then how the total between-class matrix(SB) is then calculated by adding all the between-classmatrices of all classes.
2.3. Calculating the Within-Class Variance (SW )
The within-class variance of the ith class (SWi ) rep-resents the difference between the mean and the sam-ples of that class. LDA technique searches for a lower-dimensional space, which is used to minimize the dif-ference between the projected mean (mi) and the pro-jected samples of each class (W T xi), or simply min-imizes the within-class variance [83,36]. The within-class variance of each class (SW j ) is calculated as inEquation (5).
∑xi∈ω j , j=1,...,c
(W T xi−m j)2
= ∑xi∈ω j , j=1,...,c
(W T xi j−W T µ j)2
= ∑xi∈ω j , j=1,...,c
W T (xi j−µ j)2W
= ∑xi∈ω j , j=1,...,c
W T (xi j−µ j)(xi j−µ j)TW
= ∑xi∈ω j , j=1,...,c
W T SW j W
(5)
From Equation (5), the within-class variance foreach class can be calculated as follows, SW j = dT
j ∗d j =
∑n ji=1(xi j − µ j)(xi j − µ j)
T , where xi j represents the ith
sample in the jth class as shown in Fig. (1, step (E,
4
WTSBWff
WTSwWargfmax=
μ1 μ
μ1
μ
dMx1E
d1xME
SB1=
Between-ClassfMatrixfdSBE
dn1xME
Sw1= -
μ1
ω1
-
ω1 μ1Within-ClassfMatrixfdSWE
D=
dNxME
ω1-μ1
ω2-μ2
ω3-μ3
x1-μ1x2-μ1
xn1-μ1xdn1L1E-μ2
xdn1Ln2E-μ2
xN-μ3
d1xME
μ1
μ2
μ3
MeanfoffeachfClassfdμiE
-
μ
d1xME
TotalfMeanfdμE
Vk∈RMxK
dMxn1E
X =
SB1
SB1 SB2 SB3
SB=
SB
L L =
X
-
- -
=
SW1
SW1 SW2 SW3
SW=
SW
L L =
v1 v2 vk vM
λ1λ2 λM
SortedfEigenvalues
λk
LargestfkfEigenvalues
kfSelectedEigenvectors
LowerfDimensionalf
SpacedVkE
dMxME dMxME
B
C
D F
G
E
ConstructfthefLowerfDimensionalfSpacefdVkE
ω1
ω2
ω3
X=
x1x2
xn1
dNxME
xn1L1xn1L2
xn1Ln2
xN
DatafMatrixfdXE
A
Mean-CenteringfDatafdDi=ωi-μiE
W
Fig. 1. Visualized steps to calculate a lower dimensional subspace of the LDA technique.
F)), and d j is the centering data of the jth class, i.e.d j = ω j−µ j = xi
n ji=1−µ j. Moreover, step (F) in the
figure illustrates how the within-class variance of thefirst class (SW1 ) in our example is calculated. The totalwithin-class variance represents the sum of all within-class matrices of all classes (see Fig. (1, step (F))), andit can be calculated as in Equation (6).
5
Table 1Notation.
Notation Description Notation DescriptionX Data matrix xi ith sample
N Total number of samples in X MDimension of X or
the number of features of X
W Transformation matrix Vk The lower dimensional space
ni Number of samples in ωi c Total number of classes
µiThe mean ofthe ith class
miThe mean of the ith class
after projection
µTotal or global mean
of all samplesm
The total mean of allclasses after projection
SWiWithin-class variance or scatter
matrix of the ith class (ωi)SW Within-class variance
SBiBetween-class variance
of the ith class (ωi)SB Between-class variance
V Eigenvectors of W λ Eigenvalue matrix
Vi ith eigenvector λi ith eigenvalue
xi j The ith sample in the jth class Y Projection of the original data
kThe dimension of the lower
dimensional space (Vk)ωi ith Class
SW =3
∑i=1
SWi
= ∑xi∈ω1
(xi−µ1)(xi−µ1)T
+ ∑xi∈ω2
(xi−µ2)(xi−µ2)T
+ ∑xi∈ω3
(xi−µ3)(xi−µ3)T
(6)
2.4. Constructing the Lower Dimensional Space
After calculating the between-class variance (SB)and within-class variance (SW ), the transformation ma-trix (W ) of the LDA technique can be calculated as inEquation (7), which is called Fisher’s criterion. Thisformula can be reformulated as in Equation (8).
arg maxW
W T SB WW T SW W
(7)
SW W = λSB W (8)
where λ represents the eigenvalues of the transfor-mation matrix (W ). The solution of this problemcan be obtained by calculating the eigenvalues (λ =
λ1,λ2, . . . ,λM) and eigenvectors (V = v1,v2, . . . ,vM)of W = S−1
W SB, if SW is non-singular [83,36,81].The eigenvalues are scalar values, while the eigen-
vectors are non-zero vectors, which satisfies the Equa-tion (8) and provides us with the information about theLDA space. The eigenvectors represent the directionsof the new space, and the corresponding eigenvaluesrepresent the scaling factor, length, or the magnitude ofthe eigenvectors [59,34]. Thus, each eigenvector rep-resents one axis of the LDA space, and the associatedeigenvalue represents the robustness of this eigenvec-tor. The robustness of the eigenvector reflects its abilityto discriminate between different classes, i.e. increasethe between-class variance, and decreases the within-class variance of each class; hence meets the LDAgoal. Thus, the eigenvectors with the k highest eigen-values are used to construct a lower dimensional space(Vk), while the other eigenvectors (vk+1,vk+2,vM)are neglected as shown in Fig. (1, step (G)).
6
Projection
Y∈RNxk
Data1After1Projectionk
Vk∈RMxK
Lower1Dimensional1
Space(Vk)
ω1
ω2
ω3
X=
x1x2
xn1
(NxM)
xn1+1xn1+2
xn1+n2
xN
Data1Matrix1(X)
ω1
ω2
ω3
y1y2
yn1yn1+1yn1+2
yn1+n2
yN
Y=XVk
Fig. 2. Projection of the original samples (i.e. data matrix) on the lower dimensional space of LDA (Vk).
Figure (2) shows the lower dimensional space ofthe LDA technique, which is calculated as in Fig. (1,step (G)). As shown, the dimension of the original datamatrix (X ∈ R N×M) is reduced by projecting it ontothe lower dimensional space of LDA (Vk ∈ R M×k) asdenoted in Equation (9) [81]. The dimension of thedata after projection is k; hence, M− k features are ig-nored or deleted from each sample. Thus, each sample(xi) which was represented as a point a M-dimensionalspace will be represented in a k-dimensional space byprojecting it onto the lower dimensional space (Vk) asfollows, yi = xiVk.
Y = XVk (9)
Figure (3) shows a comparison between two lower-dimensional sub-spaces. In this figure, the originaldata which consists of three classes as in our exampleare plotted. Each class has five samples, and all sam-ples are represented by two features only (xi ∈ R 2)to be visualized. Thus, each sample is representedas a point in two-dimensional space. The transforma-tion matrix (W (2× 2)) is calculated using the stepsin Sect. 2.2, 2.3, and 2.4. The eigenvalues (λ1 and λ2)and eigenvectors (i.e. sub-spaces) (V = v1,v2) of Ware then calculated. Thus, there are two eigenvectorsor sub-spaces. A comparison between the two lower-dimensional sub-spaces shows the following notices:
– First, the separation distance between differentclasses when the data are projected on the firsteigenvector (v1) is much greater than when thedata are projected on the second eigenvector (v2).As shown in the figure, the three classes are effi-ciently discriminated when the data are projectedon v1. Moreover, the distance between the means
μ1
μ2
μ3 μS W
2
SW
1
SW
3
SB3
SB2
SB
1
x1
x2
m2m1
m3
The distance
between classes
The
vara
ince
with
in e
ach
clas
s First
Eigenvector (v1)
m1
m2
m3
m1-m2
m 1-m
2
SW1
S W1
Secon
d
Eigen
vect
or (v
2)
Fig. 3. A visualized comparison between the two lower-dimensionalsub-spaces which are calculated using three different classes.
of the first and second classes (m1−m2) when theoriginal data are projected on v1 is much greaterthan when the data are projected on v2, which re-flects that the first eigenvector discriminates thethree classes better than the second one.
– Second, the within-class variance when the dataare projected on v1 is much smaller than when itprojected on v2. For example, SW1 when the dataare projected on v1 is much smaller than when thedata are projected on v2. Thus, projecting the dataon v1 minimizes the within-class variance muchbetter than v2.
7
From these two notes, we conclude that the first eigen-vector meets the goal of the lower-dimensional spaceof the LDA technique than the second eigenvector;hence, it is selected to construct a lower-dimensionalspace.
2.5. Class-Dependent vs. Class-Independent Methods
The aim of the two methods of the LDA is to cal-culate the LDA space. In the class-dependent LDA,one separate lower dimensional space is calculated foreach class as follows, Wi = S−1
WiSB, where Wi represents
the transformation matrix for the ith class. Thus, eigen-values and eigenvectors are calculated for each trans-formation matrix separately. Hence, the samples ofeach class are projected on their corresponding eigen-vectors. On the other hand, in the class-independentmethod, one lower dimensional space is calculated forall classes. Thus, the transformation matrix is calcu-lated for all classes, and the samples of all classes areprojected on the selected eigenvectors [1].
2.6. LDA Algorithm
In this section the detailed steps of the algorithmsof the two LDA methods are presented. As shown inAlgorithms (1 and 2), the first four steps in both al-gorithms are the same. Table (1) shows the notationswhich are used in the two algorithms.
2.7. Computational Complexity of LDA
In this section, the computational complexity forLDA is analyzed. The computational complexity forthe first four steps, common in both class-dependentand class-independent methods, are computed as fol-lows. As illustrated in Algorithm (1), in step (2), tocalculate the mean of the ith class, there are niM ad-ditions and M divisions, i.e., in total, there are (NM +cM) operations. In step (3), there are NM additionsand M divisions, i.e., there are (NM +M) operations.The computational complexity of the fourth step isc(M +M2 +M2), where M is for µi− µ, M2 for (µi−µ)(µi− µ)T , and the last M2 is for the multiplicationbetween ni and the matrix (µi−µ)(µi−µ)T . In the fifthstep, there are N(M +M2) operations, where M is for(xi j − µ j) and M2 is for (xi j − µ j)(xi j − µ j)
T . In thesixth step, there are M3 operations to calculate S−1
W , M3
is for the multiplication between S−1W and SB, and M3
to calculate the eigenvalues and eigenvectors. Thus,in class-independent method, the computational com-
Algorithm 1. : Linear Discriminant Analysis (LDA): Class-Independent
1: Given a set of N samples [xi]Ni=1, each of which
is represented as a row of length M as in Fig. (1,step(A)), and X(N×M) is given by,
X =
x(1,1) x(1,2) . . . x(1,M)
x(2,1) x(2,2) . . . x(2,M)...
......
...x(N,1) x(N,2) . . . x(N,M)
(10)
2: Compute the mean of each class µi(1×M) as inEquation (2).
3: Compute the total mean of all data µ(1×M) as inEquation (3).
4: Calculate between-class matrix SB(M×M) as fol-lows:
SB =c
∑i=1
ni(µi−µ)(µi−µ)T (11)
5: Compute within-class matrix SW (M×M), as fol-lows:
SW =c
∑j=1
n j
∑i=1
(xi j−µ j)(xi j−µ j)T (12)
where xi j represents the ith sample in the jth class.6: From Equation (11 and 12), the matrix W that
maximizing Fisher’s formula which is defined inEquation (7) is calculated as follows, W = S−1
W SB.The eigenvalues (λ) and eigenvectors (V ) of W arethen calculated.
7: Sorting eigenvectors in descending order accord-ing to their corresponding eigenvalues. The first keigenvectors are then used as a lower dimensionalspace (Vk).
8: Project all original samples (X) onto the lower di-mensional space of LDA as in Equation (9).
plexity is O(NM2) if N >M; otherwise, the complexityis O(M3).
In Algorithm (2), the number of operations to cal-culate the within-class variance for each class SW j inthe sixth step is n j(M + M2), and to calculate SW ,N(M + M2) operations are needed. Hence, calculat-ing the within-class variance for both LDA methodsare the same. In the seventh step and eighth, thereare M3 operations for the inverse, M3 for the multi-plication of S−1
WiSB, and M3 for calculating eigenval-
ues and eigenvectors. These two steps are repeated for
8
Algorithm 2. : Linear Discriminant Analysis (LDA): Class-Dependent
1: Given a set of N samples [xi]Ni=1, each of which
is represented as a row of length M as in Fig. (1,step(A)), and X(N×M) is given by,
X =
x(1,1) x(1,2) . . . x(1,M)
x(2,1) x(2,2) . . . x(2,M)...
......
...x(N,1) x(N,2) . . . x(N,M)
(13)
2: Compute the mean of each class µi(1×M) as inEquation (2).
3: Compute the total mean of all data µ(1×M) as inEquation (3).
4: Calculate between-class matrix SB(M×M) as inEquation (11)
5: for all Class i, i = 1,2, . . . ,c do6: Compute within-class matrix of each class
SWi(M×M), as follows:
SW j = ∑xi∈ω j
(xi−µ j)(xi−µ j)T (14)
7: Construct a transformation matrix for each class(Wi) as follows:
Wi = S−1Wi
SB (15)
8: The eigenvalues (λi) and eigenvectors (V i) ofeach transformation matrix (Wi) are then calcu-lated, where λi and V i represent the calculatedeigenvalues and eigenvectors of the ith class, re-spectively.
9: Sorting the eigenvectors in descending order ac-cording to their corresponding eigenvalues. Thefirst k eigenvectors are then used to construct alower dimensional space for each class V i
k .10: Project the samples of each class (ωi) onto their
lower dimensional space (V ik ), as follows:
Ω j = xiVj
k , xi ∈ ω j (16)
where Ω j represents the projected samples ofthe class ω j.
11: end for
each class which increases the complexity of the class-dependent algorithm. Totally, the computational com-
plexity of the class-dependent algorithm is O(NM2) ifN > M; otherwise, the complexity is O(cM3). Hence,the class-dependent method needs computations morethan class-independent method.
In our case, we assumed that there are 40 classesand each class has ten samples. Each sample is rep-resented by 4096 features (M > N). Thus, the com-putational complexity of the class-independent methodis O(M3) = 40963, while the class-dependent methodneeds O(cM3) = 40×40963.
3. Numerical Examples
In this section, two numerical examples will be pre-sented. The two numerical examples explain the stepsto calculate the LDA space and how the LDA tech-nique is used to discriminate between only two differ-ent classes. In the first example, the lower-dimensionalspace is calculated using the class-independent method,while in the second example, the class-dependentmethod is used. Moreover, a comparison between thelower dimensional spaces of each method is presented.In all numerical examples, the numbers are rounded upto the nearest hundredths (i.e. only two digits after thedecimal point are displayed).
The first four steps of both class-independent andclass-dependent methods are common as illustrated inAlgorithms (1 and 2). Thus, in this section, we showhow these steps are calculated.
Given two different classes, ω1(5×2) and ω2(6×2)have (n1 = 5) and (n2 = 6) samples, respectively. Eachsample in both classes is represented by two features(i.e. M = 2) as follows:
ω1 =
1.00 2.002.00 3.003.00 3.004.00 5.005.00 5.00
and ω2 =
4.00 2.005.00 0.005.00 2.003.00 2.005.00 3.006.00 3.00
(17)
To calculate the lower dimensional space usingLDA, first the mean of each class µ j is calculated. Thetotal mean µ(1×2) is then calculated, which representsthe mean of all means of all classes. The values of themean of each class and the total mean are shown below,
µ1 =[3.00 3.60
], µ2 =
[4.67 2.00
], and
µ =[ 5
11 µ1611 µ2
]=[3.91 2.727
] (18)
9
The between-class variance of each class (SBi(2×2)) and the total between-class variance (SB(2×2) arecalculated. The values of the between-class variance ofthe first class (SB1 ) is equal to,
SB1 = n1(µ1−µ)T (µ1−µ) = 5[−0.91 0.87]T [−0.91 0.87]
=
[4.13 −3.97−3.97 3.81
](19)
Similarly, SB2 is calculated as follows:
SB2 =
[3.44 −3.31−3.31 3.17
](20)
The total between-class variance is calculated a fol-lows:
SB = SB1 +SB2 =
[4.13 −3.97−3.97 3.81
]+
[3.44 −3.31−3.31 3.17
]=
[7.58 −7.27−7.27 6.98
](21)
To calculate the within-class matrix, first subtractthe mean of each class from each sample in that classand this step is called mean-centering data and it is cal-culated as follows, di = ωi − µi, where di representscentering data of the class ωi. The values of d1 and d2are as follows:
d1 =
−2.00 −1.60−1.00 −0.600.00 −0.601.00 1.402.00 1.40
and d2 =
−0.67 0.000.33 −2.000.33 0.00−1.67 0.000.33 1.001.33 1.00
(22)
In the next two subsections, two different methodsare used to calculate the LDA space.
3.1. Class-Independent Method
In this section, the LDA space is calculated using theclass-independent method. This method represents thestandard method of LDA as in Algorithm (1).
After centring the data, the within-class variancefor each class (SWi(2× 2)) is calculated as follows,
SW j = dTj ∗ d j = ∑
n ji=1(xi j − µ j)
T (xi j − µ j), where xi j
represents the ith sample in the jth class. The totalwithin-class matrix (SW (2× 2)) is then calculated asfollows, SW = ∑
ci=1 SWi . The values of the within-class
matrix for each class and the total within-class matrixare as follows:
SW1 =
[10.00 8.008.00 7.20
], SW2 =
[5.33 1.001.00 6.00
],
SW =
[15.33 9.009.00 13.20
] (23)
The transformation matrix (W ) in the class-independentmethod can be obtained as follows, W = S−1
W SB, andthe values of (S−1
W ) and (W ) are as follows:
S−1W =
[0.11 −0.07−0.07 0.13
]and W =
[1.36 −1.31−1.48 1.42
](24)
The eigenvalues (λ(2×2)) and eigenvectors (V (2×2)) of W are then calculated as follows:
λ =
[0.00 0.000.00 2.78
]and V =
[−0.69 0.68−0.72 −0.74
](25)
From the above results it can be noticed that, thesecond eigenvector (V2) has corresponding eigenvaluemore than the first one (V1), which reflects that, thesecond eigenvector is more robust than the first one;hence, it is selected to construct the lower dimen-sional space. The original data is projected on thelower dimensional space, as follows, yi = ωi V2, whereyi(ni×1) represents the data after projection of the ith
class, and its values will be as follows:
y1 = ω1V2 =
1.00 2.002.00 3.003.00 3.004.00 5.005.00 5.00
[
0.68−0.74
]=
−0.79−0.85−0.18−0.97−0.29
(26)
Similarly, y2 is as follows:
y2 = ω2V2 =
1.243.391.920.561.181.86
(27)
10
−9 −8 −7 −6 −5 −4 −3 −2 −1 00
0.2
0.4
0.6
0.8
1
1.2
1.4
Projected Data
P(P
roje
cte
d D
ata
)
Class 1
Class 2
SB
SW2SW1
(a)
−1 0 1 2 3 4 50
0.2
0.4
0.6
0.8
1
1.2
1.4
Projected Data
P(P
roje
cte
d D
ata
)
Class 1
Class 2SB
SW2SW1
(b)
Fig. 4. Probability density function of the projected data of the firstexample, (a) the projected data on V1, (b) the projected data on V2.
Figure (4) illustrates a probability density function(pdf) graph of the projected data (yi) on the two eigen-vectors (V1 and V2). A comparison of the two eigenvec-tors reveals the following :
– The data of each class is completely discrimi-nated when it is projected on the second eigen-vector (see Fig. (4)(b)) than the first one (seeFig. (4a)). In other words, the second eigenvectormaximizes the between-class variance more thanthe first one.
– The within-class variance (i.e. the variance be-tween the same class samples) of the two classesare minimized when the data are projected on thesecond eigenvector. As shown in Fig. ((4)(b)), thewithin-class variance of the first class is smallcompared with Fig. ((4)(a)).
3.2. Class-Dependent Method
In this section, the LDA space is calculated using theclass-dependent method. As mentioned in Sect. 2.5,the class-dependent method aims to calculate a sepa-rate transformation matrix (Wi) for each class.
The within-class variance for each class (SWi(2×2)) is calculated as in class-independent method. Thetransformation matrix (Wi) for each class is then cal-culated as follows, Wi = S−1
WiSB. The values of the two
transformation matrices (W1 and W2) will be as fol-lows:
W1 = S−1W1
SB =
[10.00 8.008.00 7.20
]−1 [ 7.58 −7.27−7.27 6.98
]=
[0.90 −1.00−1.00 1.25
][7.58 −7.27−7.27 6.98
]=
[14.09 −13.53−16.67 16.00
](28)
Similarly, W2 is calculated as follows:
W2 =
[1.70 −1.63−1.50 1.44
](29)
The eigenvalues (λi) and eigenvectors (Vi) for eachtransformation matrix (Wi) are calculated, and the val-ues of the eigenvalues and eigenvectors are shown be-low.
λω1 =
[0.00 0.000.00 30.01
]and Vω1 =
[−0.69 0.65−0.72 −0.76
](30)
λω2 =
[3.14 0.000.00 0.00
]and Vω2 =
[0.75 0.69−0.66 0.72
](31)
where λωi and Vωi represent the eigenvalues and eigen-vectors of the ith class, respectively.
From the results shown (above) it can be seen that,the second eigenvector of the first class (V 2ω1 ) has cor-responding eigenvalue more than the first one; thus,the second eigenvector is used as a lower dimensionalspace for the first class as follows, y1 = ω1 ∗V 2ω1 ,where y1 represents the projection of the samples ofthe first class. While, the first eigenvector in the secondclass (V 1ω2 ) has corresponding eigenvalue more thanthe second one. Thus, V 1ω2 is used to project the dataof the second class as follows, y2 =ω2 ∗V
1ω2 , where y2
represents the projection of the samples of the secondclass. The values of y1 and y2 will be as follows:
y1 =
−0.88−1.00−0.35−1.24−0.59
and y2 =
1.683.762.430.931.772.53
(32)
11
Figure (5) shows a pdf graph of the projected data(i.e. y1 and y2) on the two eigenvectors (V 2ω1 and V 1ω2 )and a number of findings are revealed the following:
– First, the projection data of the two classes areefficiently discriminated.
– Second, the within-class variance of the projectedsamples is lower than the within-class variance ofthe original samples.
−1 0 1 2 3 4 50
0.2
0.4
0.6
0.8
1
1.2
1.4
Projected Data
P(P
roje
cted
Dat
a)
Class 1Class 2
Fig. 5. Probability density function (pdf) of the projected data usingclass-dependent method, the first class is projected on V 2ω1 , while
the second class is projected on V 1ω2 .
3.3. Discussion
In these two numerical examples, the LDA space iscalculated using class-dependent and class-independentmethods.
Figure (6) shows a further explanation of the twomethods as following:
– Class-Independent: As shown from the figure,there are two eigenvectors, V1 (dotted black line)and V2 (solid black line). The differences betweenthe two eigenvectors are as follows:
∗ The projected data on the second eigenvec-tor (V2) which has the highest correspondingeigenvalue will discriminate the data of thetwo classes better than the first eigenvector. Asshown in the figure, the distance between theprojected means m1−m2 which represents SB,increased when the data are projected on V2than V1.
∗ The second eigenvector decreases the within-class variance much better than the first eigen-vector. Figure (6) illustrates that the within-class variance of the first class (SW1 ) was muchsmaller when it was projected on V2 than V1.
∗ As a result of the above two findings, V2 is usedto construct the LDA space.
– Class-Dependent: As shown from the figure, thereare two eigenvectors, V 2ω1 (red line) and V 1ω2(blue line), which represent the first and secondclasses, respectively. The differences between thetwo eigenvectors are as following:
∗ Projecting the original data on the two eigen-vectors discriminates between the two classes.As shown in the figure, the distance betweenthe projected means m1−m2 is larger than thedistance between the original means µ1−µ2.
∗ The within-class variance of each class is de-creased. For example, the within-class varianceof the first class (SW1 ) is decreased when it isprojected on its corresponding eigenvector.
∗ As a result of the above two findings, V 2ω1 andV 1ω2 are used to construct the LDA space.
– Class-Dependent vs. Class-Independent: The twoLDA methods are used to calculate the LDAspace, but a class-dependent method calculatesseparate lower dimensional spaces for each classwhich has two main limitations: (1) it needsmore CPU time and calculations more than class-independent method; (2) it may lead to SSS prob-lem because the number of samples in each classaffects the singularity of SWi
2.
These findings reveal that the standard LDA techniqueused the class-independent method rather than usingthe class-dependent method.
4. Main Problems of LDA
Although LDA is one of the most common data re-duction techniques, it suffers from two main problems:the Small Sample Size (SSS) and linearity problems.In the next two subsections, these two problems willbe explained, and some of the state-of-the-art solutionsare highlighted.
4.1. Linearity problem
LDA technique is used to find a linear transforma-tion that discriminates between different classes. How-ever, if the classes are non-linearly separable, LDA cannot find a lower dimensional space. In other words,LDA fails to find the LDA space when the discrimina-tory information are not in the means of classes. Fig-
2SSS problem will be explained in Sect. 4.2
12
1
2
3
4
5
6
1 2 3 4 5 6
x2
µ1
Vω21
Vω12
SamplesPofPClassP1Class-DependentP
LDAPsubspaces
Projec
tionP
onPV 1
µ2
µ
SamplesPofPClassP2
7
7
x1
Cla
ss-In
depe
nden
tP
LDAP
subs
pace
s
S W1
SW
1
SW
1
S W1
m1 -m
2
m1
m2
m 1-m
2
ProjectionP
onPV2
m1 -m
2µ1 -µ
2
µ1 -µ
2
MeanPofPClassP1
MeanPofPClassP2
ProjectedPMeanPofPClassP1
ProjectedPMeanPofPClassP2
TotalPMean
Fig. 6. Illustration of the example of the two different methods of LDA methods. The blue and red lines represent the first and second eigenvectorsof the class-dependent approach, respectively, while the solid and dotted black lines represent the second and first eigenvectors of class-indepen-dent approach, respectively.
ure (7) shows how the discriminatory information doesnot exist in the mean, but in the variance of the data.This is because the means of the two classes are equal.The mathematical interpretation for this problem is asfollows: if the means of the classes are approximatelyequal, so the SB and W will be zero. Hence, the LDAspace cannot be calculated.
One of the solutions of this problem is based on thetransformation concept, which is known as a kernelmethods or functions [50,3]. Figure (7) illustrates howthe transformation is used to map the original data intoa higher dimensional space; hence, the data will be lin-early separable, and the LDA technique can find thelower dimensional space in the new space. Figure (8)graphically and mathematically shows how two non-
separable classes in one-dimensional space are trans-formed into a two-dimensional space (i.e. higher di-mensional space); thus, allowing linear separation.
The kernel idea is applied in Support Vector Ma-chine (SVM) [49,66,69] Support Vector Regression(SVR)[58], PCA [51], and LDA [50]. Let φ representsa nonlinear mapping to the new feature space Z. Thetransformation matrix (W ) in the new feature space (Z)is calculated as in Equation (33).
F(W ) = max
∣∣∣∣∣W T Sφ
BW
W T Sφ
WW
∣∣∣∣∣ (33)
where W is a transformation matrix and Z is the newfeature space. The between-class matrix (Sφ
B) and thewithin-class matrix (Sφ
W ) are defined as follows:
13
µ µ2µ1= =x x
ω1
Mapping or Transformation
Non-Linearly Seprable Classes
Linearly Seprable Classes
ω1
ω1
ω1
ω1 ω1 ω2
ω2
ω2
ω2
ω2
ω2
Fig. 7. Two examples of two non-linearly separable classes, toppanel shows how the two classes are non-separable, while the bot-tom shows how the transformation solves this problem and the twoclasses are linearly separable.
Sφ
B =c
∑i=1
ni(µφ
i −µφ)(µφ
i −µφ)T (34)
Sφ
W =c
∑j=1
n j
∑i=1
(φxi j−µφ
j )(φxi j−µφ
j )T (35)
where µφ
i = 1ni
∑nii=1 φxi and µφ = 1
N ∑Ni=1 φxi =
∑ci=1
niN µφ
iThus, in kernel LDA, all samples are transformed
non-linearly into a new space Z using the functionφ. In other words, the φ function is used to map theoriginal features into Z space by creating a nonlin-ear combination of the original samples using a dot-products of it [3]. There are many types of kernelfunctions to achieve this aim. Examples of these func-tion include Gaussian or Radial Basis Function (RBF),K(xi,x j) = exp(−||xi− x j||2/2σ2), where σ is a posi-tive parameter, and the polynomial kernel of degree d,K(xi,x j) = (
⟨xi,x j
⟩+ c)d , [3,51,72,29].
1 2
Mapping
0
1
2
3
4
-1-2
1 20-1-2
X
Φ(x)=(x,x2)=(z1,z2)=Z
z2
z1
(1)→ (1,1)
(-1)→(-1,1)
(2)→ (2,4)
(-2)→(-2,4)
Samples are Non-linearly Seprable
Samples are linearly Seprable
Class 1
Class 2
Fig. 8. Example of kernel functions, the samples lie on the top panel(X) which are represented by a line (i.e. one-dimensional space) arenon-linearly separable, where the samples lie on the bottom panel(Z) which are generated from mapping the samples of the top spaceare linearly separable.
4.2. Small Sample Size Problem
4.2.1. Problem DefinitionSingularity, Small Sample Size (SSS), or under-
sampled problem is one of the big problems of LDAtechnique. This problem results from high-dimensionalpattern classification tasks or a low number of train-ing samples available for each class compared with thedimensionality of the sample space [30,38,85,82].
The SSS problem occurs when the SW is singular3.The upper bound of the rank4 of SW is N − c, whilethe dimension of SW is M×M [38,17]. Thus, in mostcases M >> N − c which leads to SSS problem. Forexample, in face recognition applications, the size ofthe face image my reach to 100×100 = 10000 pixels,which represent high-dimensional features and it leadsto a singularity problem.
3A matrix is singular if it is square, does not have a matrix in-verse, the determinant is zeros; hence, not all columns and rows areindependent
4The rank of the matrix represents the number of linearly inde-pendent rows or columns
14
4.2.2. Common Solutions to SSS Problem:There are many studies that proposed many solu-
tions for this problem; each has its advantages anddrawbacks.
– Regularization (RLDA): In regularization method,the identity matrix is scaled by multiplying it bya regularization parameter (η > 0) and adding itto the within-class matrix to make it non-singular[18,38,82,45]. Thus, the diagonal components ofthe within-class matrix are biased as follows,SW = SW + ηI. However, choosing the value ofthe regularization parameter requires more tuningand a poor choice for this parameter can degradethe performance of the method [38,45]. Anotherproblem of this method is that the parameter η isjust added to perform the inverse of SW and hasno clear mathematical interpretation [38,57].
– Sub-space: In this method, a non-singular inter-mediate space is obtained to reduce the dimen-sion of the original data to be equal to the rankof SW ; hence, SW becomes full-rank5, and thenSW can be inverted. For example, Belhumeur etal. [4] used PCA, to reduce the dimensions of theoriginal space to be equal to N− c (i.e. the upperbound of the rank of SW ). However, as reportedin [22], losing some discriminant information is acommon drawback associated with the use of thismethod.
– Null Space: There are many studies proposed toremove the null space of SW to make SW full-rank;hence, invertible. The drawback of this method isthat more discriminant information is lost whenthe null space of SW is removed, which has a neg-ative impact on how the lower dimensional spacesatisfies the LDA goal [83].
Four different variants of the LDA technique that areused to solve the SSS problem are introduced as fol-lows:
PCA + LDA technique: In this technique, the orig-inal d-dimensional features are first reduced to h-dimensional feature space using PCA, and then theLDA is used to further reduce the features to k-dimensions. The PCA is used in this technique to re-duce the dimensions to make the rank of SW is N−c asreported in [4]; hence, the SSS problem is addressed.However, the PCA neglects some discriminant infor-mation, which may reduce the classification perfor-mance [57,60].
5A is a full-rank matrix if all columns and rows of the matrix areindependent, (i.e. rank(A)= # rows= #cols) [23]
Direct LDA technique: Direct LDA (DLDA) is one ofthe well-known techniques that are used to solve theSSS problem. This technique has two main steps [83].In the first step, the transformation matrix, W, is com-puted to transform the training data to the range spaceof SB. In the second step, the dimensionality of thetransformed data is further transformed using someregulating matrices as in Algorithm 4. The benefit ofthe DLDA is that there is no discriminative features areneglected as in PCA+LDA technique [83].
Regularized LDA technique: In the Regularized LDA(RLDA), a small perturbation is add to the SW matrixto make it non-singular as mentioned in [18]. This reg-ularization can be applied as follows:
(SW +ηI)−1SBwi = λiwi (36)
where η represents a regularization parameter. The di-agonal components of the SW are biased by adding thissmall perturbation [18,13]. However, the regularizationparameter need to be tuned and poor choice of it candegrade the generalization performance [57].
Null LDA technique: The aim of the NLDA techniqueis to find the orientation matrix W, and this can beachieved using two steps. In the first step, the rangespace of the SW is neglected, and the data are projectedonly on the null space of SW as follows, SWW = 0. Inthe second step, the aim is to search for W that satis-fies SBW = 0 and maximizes |W T SBW |. The higher di-mensionality of the feature space may lead to compu-tational problems. This problem can be solved by (1)using the PCA technique as a pre-processing step, i.e.before applying the NLDA technique, to reduce the di-mension of feature space to be N−1; by removing thenull space of ST = SB + SW [57], (2) using the PCAtechnique before the second step of the NLDA tech-nique [54]. Mathematically, in the Null LDA (NLDA)technique, the h column vectors of the transformationmatrix W = [w1,w2, . . . ,wh] are taken to be the nullspace of the SW as follows, wT
i SW wi = 0, ∀i = 1 . . .h,where wT
i SBwi 6= 0. Hence, M− (N− c) linearly inde-pendent vectors are used to form a new orientation ma-trix, which is used to maximize |W T SBW | subject to theconstraint |W T SWW |= 0 as in Equation (37).
W = arg max|W T SW W |=0
|W T SBW | (37)
15
5. Applications of the LDA technique
In many applications, due to the high number of fea-tures or dimensionality, the LDA technique have beenused. Some of the applications of the LDA techniqueand its variants are described as follows:
5.1. Biometrics Applications
Biometrics systems have two main steps, namely,feature extraction (including pre-processing steps) andrecognition. In the first step, the features are extractedfrom the collected data, e.g. face images, and in thesecond step, the unknown samples, e.g. unknown faceimage, is identified/verified. The LDA technique andits variants have been applied in this application. Forexample, in [83,10,41,75,20,68], the LDA techniquehave been applied on face recognition. Moreover, theLDA technique was used in Ear [84], fingerprint [44],gait [5], and speech [24] applications. In addition, theLDA technique was used with animal biometrics as in[65,19].
5.2. Agriculture Applications
In agriculture applications, an unknown sample canbe classified into a pre-defined species using computa-tional models [64]. In this application, different vari-ants of the LDA technique was used to reduce the di-mension of the collected features as in [9,26,46,21,64,63].
5.3. Medical Applications
In medical applications, the data such as the DNAmicroarray data consists of a large number of featuresor dimensions. Due to this high dimensionality, thecomputational models need more time to train theirmodels, which may be infeasible and expensive. More-over, this high dimensionality reduces the classifica-tion performance of the computational model and in-creases its complexity. This problem can be solved us-ing LDA technique to construct a new set of featuresfrom a large number of original features. There aremany papers have been used LDA in medical applica-tions [54,52,53,16,39,55,8].
6. Packages
In this section, some of the available packages thatare used to compute the space of LDA variants. Forexample, WEKA6 is a well-known Java-based datamining tool with open source machine learning soft-ware such as classification, association rules, regres-sion, pre-processing, clustering, and visualization. InWEKA, the machine learning algorithms can be ap-plied directly on the dataset or called from person’sJava code. XLSTAT7 is another data analysis and sta-tistical package for Microsoft Excel that has a widevariety of dimensionality reduction algorithms includ-ing LDA. dChip8 package is also used for visualiza-tion of gene expression and SNP microarray includingsome data analysis algorithms such as LDA, cluster-ing, and PCA. LDA-SSS9 is a Matlab package, and itcontains several algorithms related to the LDA tech-niques and its variants such as DLDA, PCA+LDA, andNLDA. MASS10 package is based on R, and it hasfunctions that are used to perform linear and quadraticdiscriminant function analysis. Dimensionality reduc-tion11 package is mainly written in Matlab, and it hasa number of dimensionality reduction techniques suchas ULDA, QLDA, and KDA. DTREG12 is a softwarepackage that is used for medical data and modelingbusiness, and it has several predictive modeling meth-ods such as LDA, PCA, linear regression, and decisiontrees.
7. Experimental Results and Discussion
In this section, two experiments were conducted toillustrate: (1) how the LDA is used for different appli-cations, (2) what is the relation between its parame-ter (Eigenvectors) and the accuracy of a classificationproblem, (3) when the SSS problem could appear anda method for solving it.
6http:/www.cs.waikato.ac.nz/ml/weka/7http://www.xlstat.com/en/8https://sites.google.com/site/dchipsoft/home9http://www.staff.usp.ac.fj/sharma al/index.htm
10http://www.statmethods.net/advstats/discriminant.html11http://www.public.asu.edu/*jye02/Software/index.html12http://www.dtreg.com/index.htm
16
7.1. Experimental Setup
This section gives an overview of the databases,the platform, and the machine specification used toconduct our experiments. Different biometric datasetswere used in the experiments to show how the LDAusing its parameter behaves with different data. Thesedatasets are described as follows:
– ORL dataset13 face images dataset (Olivetti Re-search Laboratory, Cambridge) [48], which con-sists of 40 distinct individuals, was used. In thisdataset, each individual has ten images taken atdifferent times and varying light conditions. Thesize of each image is 92×112.
– Yale dataset14 is another face images datasetwhich contains 165 grey scale images in GIF for-mat of 15 individuals [78]. Each individual has11 images in different expressions and configura-tion: center-light, happy, left-light, with glasses,normal, right-light, sad, sleepy, surprised, and awink.
– 2D ear dataset (Carreira-Perpinan,1995)15 imagesdataset [7] was used. The ear data set consists of17 distinct individuals. Six views of the left pro-file from each subject were taken under a uniform,diffuse lighting.
In all experiments, k-fold cross-validation tests haveused. In k-fold cross-validation, the original samples ofthe dataset were randomly partitioned into k subsets of(approximately) equal size and the experiment is run ktimes. For each time, one subset was used as the testingset and the other k− 1 subsets were used as the train-ing set. The average of the k results from the folds canthen be calculated to produce a single estimation. Inthis study, the value of k was set to 10.
The images in all datasets resized to be 64×64 and32× 32 as shown in Table (2). Figure (9) shows sam-ples of the used datasets and Table (2) shows a descrip-tion of the datasets used in our experiments.
In all experiments, to show the effect of the LDAwith its eigenvector parameter and its SSS problem onthe classification accuracy, the Nearest Neighbour clas-sifier was used. This classifier aims to classify the test-ing image by comparing its position in the LDA spacewith the positions of training images. Furthermore,class-independent LDA was used in all experiments.
13http://www.cam-orl.co.uk14http://vision.ucsd.edu/content/yale-face-database15http://faculty.ucmerced.edu/mcarreira-perpinan/software.html
Fig. 9. Samples of the first individual in: ORL face dataset (top row);Ear dataset (middle row), and Yale face dataset (bottom row).
Table 2Dataset description.
Dataset Dimension (M)No. of
Samples (N)No. of
classes (c)ORL64×64 4096
400 40ORl32×32 1024
Ear64×64 4096102 17
Ear32×32 1024
Yale64×64 4096165 15
Yale32×32 1024
Moreover, Matlab Platform (R2013b) and using a PCwith the following specifications: Intel(R) Core(TM)i5-2400 CPU @ 3.10 GHz and 4.00 GB RAM, underWindows 32-bit operating system were used in our ex-periments.
7.2. Experiment on LDA Parameter (Eigenvectors)
The aim of this experiment is to investigate the re-lation between the number of eigenvectors used in theLDA space, and the classification accuracy based onthese eigenvectors and the required CPU time for thisclassification.
As explained earlier that the LDA space consistsof k eigenvectors, which are sorted according to theirrobustness (i.e. their eigenvalues). The robustness ofeach eigenvector reflects its ability to discriminate be-tween different classes. Thus, in this experiment, itwill be checked whether increasing the number ofeigenvectors would increase the total robustness ofthe constructed LDA space; hence, different classescould be well discriminated. Also, it will be testedwhether increasing the number of eigenvectors wouldincrease the dimension of the LDA space and the pro-jected data; hence, CPU time increases. To investi-
17
10 20 30 40 50 60 70 80 90 10030
40
50
60
70
80
90
100
Percentage of EIgenvectors (%)
Acc
urac
y (%
)
ORL DatasetEar DatasetYale Dataset
(a)
10 20 30 40 50 60 70 80 90 1000.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Percentage of EIgenvectors (%)
CP
U ti
me
(sec
s)
ORL DatasetEar DatasetYale Dataset
(b)
Fig. 10. Accuracy and CPU time of the LDA techniques using dif-ferent percentages of eigenvectors, (a) Accuracy (b) CPU time.
gate these issue, three datasets listed in Table (2) (i.e.ORL32×32, Ear32×32, Yale32×32), were used. Moreover,seven, four, and eight images from each subject in theORL, ear, Yale datasets, respectively, are used in thisexperiment. The results of this experiment are pre-sented in Fig. (10).
From Fig. (10) it can be noticed that the accu-racy and CPU time are proportional with the numberof eigenvectors which are used to construct the LDAspace. Thus, the choice of using LDA in a specific ap-plication should consider a trade-off between these fac-tors. Moreover, from Fig. (10a), it can be remarked thatwhen the number of eigenvectors used in computingthe LDA space was increased, the classification accu-racy was also increased to a specific extent after whichthe accuracy remains constant. As seen in Fig. (10a),this extent differs from application to another. For ex-ample, the accuracy of the ear dataset remains con-
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
The Index of the Eignvector
Wei
ght o
f the
Eig
enve
ctor
ORL DatasetEar DatasetYale Dataset
Fig. 11. The robustness of the first 40 eigenvectors of the LDA tech-nique using ORL32×32, Ear32×32, and Yale32×32 datasets.
stant when the percentage of the used eigenvectors ismore than 10%. This was expected as the eigenvec-tors of the LDA space are sorted according to their ro-bustness (see Sect. 2.6). Similarly, in ORL and Yaledatasets the accuracy became approximately constantwhen the percentage of the used eigenvectors is morethan 40%. In terms of CPU time, Fig. (10b) shows theCPU time using different percentages of the eigenvec-tors. As shown, the CPU time increased dramaticallywhen the number of eigenvectors increases.
Fig. (11) shows the weights16 of the first 40 eigen-vectors which confirms our findings. From these re-sults, we can conclude that the high order eigenvectorsof the data of each application (the first 10 % of eardatabase and the first 40 % of ORL and Yale datasets)are robust enough to extract and save the most discrim-inative features which are used to achieve a good accu-racy.
These experiments confirmed that increasing thenumber of eigenvectors will increase the dimension ofthe LDA space; hence, CPU time increases. Conse-quently, the amount of discriminative information andthe accuracy increases.
7.3. Experiments on the Small Sample Size problem
The aim of this experiment is to show when the LDAis subject to the SSS problem and what are the methodsthat could be used to solve this problem. In this experi-ment, PCA-LDA [4] and Direct-LDA [83,22] methods
16The weight of the eigenvector represents the ratio of its cor-responding eigenvalue (λi) to the total of all eigenvalues (λi, i =1,2, . . . ,k) as follows, λi
∑kj=1 λ j
18
are used to address the SSS problem. A set of exper-iments was conducted to show how the two methodsinteract with the SSS problem, including the number ofsamples in each class, the total number of classes used,and the dimension of each sample.
As explained in Sect. 4, using LDA directly maylead to the SSS problem when the dimension of thesamples are much higher than the total number of sam-ples. As shown in Table (2), the size of each image orsample of the ORL dataset is 64×64 is 4096 and the to-tal number of samples is 400. The mathematical inter-pretation of this point shows that the dimension of SWis M×M, while the upper bound of the rank of SW isN−c [79,82]. Thus, all the datasets which are reportedin Table (2) lead to singular SW .
To address this problem, PCA-LDA and direct-LDAmethods were implemented. In the PCA-LDA method,PCA is first used to reduce the dimension of the origi-nal data to make SW full-rank, and then standard LDAcan be performed safely in the subspace of SW as in Al-gorithm (3). For more details of the PCA-LDA methodare reported in [4]. In the direct-LDA method, the null-space of SW matrix is removed to make SW full-rank,then standard LDA space can be calculated as in Al-gorithm (4). More details of direct-LDA methods arefound in [83].
Table (2) illustrates various scenarios designed totest the effect of different dimensions on the rank of SWand the accuracy. The results of these scenarios usingboth PCA-LDA and direct-LDA methods are summa-rized in Table (3).
As summarized in Table (3), the rank of SW is verysmall compared to the whole dimension of SW ; hence,the SSS problem occurs in all cases. As shown inTable (3), using the PCA-LDA and the Direct-LDA,the SSS problem can be solved and the Direct-LDAmethod achieved results better than PCA-LDA becauseas reported in [83,22], Direct-LDA method saves ap-proximately all important information for classifica-tion while the PCA in PCA-LDA method saves the in-formation with high variance.
8. Conclusions
In this paper, the definition, mathematics, and im-plementation of LDA were presented and explained.The paper aimed to give low-level details on how theLDA technique can address the reduction problem byextracting discriminative features based on maximiz-ing the ratio between the between-class variance, SB,
Algorithm 3. : PCA-LDA.1: Read the training images (X = x1,x2, . . . ,xN),
where xi(ro×co) represents the ith training image,ro and co represent the rows (height) and columns(width) of xi, respectively, N represents the totalnumber of training images.
2: Convert all images in vector representation Z =z1,z2, . . . ,zN, where the dimension of Z is M×1,M = ro× co.
3: calculate the mean of each class µi, total mean ofall data µ, between-class matrix SB(M×M), andwithin-class matrix SW (M×M) as in Algorithm(1, Step(2-5)).
4: Use the PCA technique to reduce the dimension ofX to be equal to or lower than r, where r representsthe rank of SW , as follows:
XPCA =UT X (38)
where, U ∈ R M×r is the lower dimensional spaceof the PCA and XPCA represents the projected dataon the PCA space.
5: Calculate the mean of each class µi, total mean ofall data µ, between-class matrix SB, and within-class matrix SW of XPCA as in Algorithm (1,Step(2-5)).
6: Calculate W as in Equation (7) and then calculatethe eigenvalues (λ) and eigenvectors (V ) of the Wmatrix.
7: Sorting the eigenvectors in descending order ac-cording to their corresponding eigenvalues. Thefirst k eigenvectors are then used as a lower dimen-sional space (Vk).
8: The original samples (X) are first projected on thePCA space as in Equation (38). The projection onthe LDA space is then calculated as follows:
XLDA =V Tk XPCA =V T
k UT X (39)
where XLDA represents the final projected on theLDA space.
and within-class variance, SW , thus discriminating be-tween different classes. To achieve this aim, the paperfollowed the approach of not only explaining the stepsof calculating the SB, and SW (i.e. the LDA space) butalso visualizing these steps with figures and diagramsto make it easy to understand. Moreover, two LDAmethods, i.e. class-dependent and class-independent,are explained and two numerical examples were given
19
Table 3Accuracy of the PCA-LDA and direct-LDA methods using thedatasets listed in Table (2).
Dataset Dim (SW )# Training
images# Testingimages
Rank (SW )Accuracy (%)
PCA-LDA Direct-LDA
ORL32×32 1024×10245 5 160 75.5 88.57 3 240 75.5 97.59 1 340 80.39 97.5
Ear32×32 1024×10243 3 34 80.39 96.084 2 51 88.24 94.125 1 68 100 100
Yale32×32 1024×10246 5 75 78.67 90.678 3 105 84.44 97.7910 1 135 100 100
ORL64×64 4096×40965 5 160 72 87.57 3 240 81.67 96.679 1 340 82.5 97.5
Ear64×64 4096×40963 3 34 74.5 96.084 2 51 91.18 96.085 1 68 100 100
Yale64×64 4096×40966 5 75 74.67 928 3 105 95.56 97.7810 1 135 93.33 100
The bold values indicate that the corresponding methods obtain best performances
and graphically illustrated to explain how the LDAspace using the two methods can be constructed. Inall examples, the mathematical interpretation of therobustness and the selection of the eigenvectors aswell the data projection were detailed and discussed.Also, LDA common problems (e.g. the SSS and lin-earity) were mathematically explained using graphi-cal examples, then their state-of-the-art solutions arehighlighted. Moreover, a detailed implementation ofLDA applications was presented. Using three standarddatasets, a number of experiments were conducted to(1) investigate and explain the relation between thenumber of eigenvectors and the robustness of the LDAspace, (2) to practically show when the SSS problemoccurs and how it can be addressed.
References
[1] S. Balakrishnama and A. Ganapathiraju. Linear discriminantanalysis-a brief tutorial. Institute for Signal and informationProcessing, 1998.
[2] E. Barshan, A. Ghodsi, Z. Azimifar, and M. Z. Jahromi. Su-pervised principal component analysis: Visualization, classifi-cation and regression on subspaces and submanifolds. PatternRecognition, 44(7):1357–1371, 2011.
[3] G. Baudat and F. Anouar. Generalized discriminant analysisusing a kernel approach. Neural computation, 12(10):2385–2404, 2000.
[4] P. N. Belhumeur, J. P. Hespanha, and D. Kriegman. Eigenfacesvs. fisherfaces: Recognition using class specific linear projec-tion. IEEE Transactions on Pattern Analysis and Machine In-telligence, 19(7):711–720, 1997.
[5] N. V. Boulgouris and Z. X. Chi. Gait recognition using radontransform and linear discriminant analysis. IEEE transactionson image processing, 16(3):731–740, 2007.
[6] M. Bramer. Principles of data mining. Springer, Second Edi-tion, 2013.
[7] M. Carreira-Perpinan. Compression neural networks for fea-ture extraction: Application to human recognition from ear im-ages. MS thesis, Faculty of Informatics, Technical Universityof Madrid, Spain, 1995.
[8] H.-P. Chan, D. Wei, M. A. Helvie, B. Sahiner, D. D. Adler,M. M. Goodsitt, and N. Petrick. Computer-aided classificationof mammographic masses and normal tissue: linear discrimi-nant analysis in texture feature space. Physics in medicine andbiology, 40(5):857, 1995.
[9] L. Chen, N. C. Carpita, W.-D. Reiter, R. H. Wilson, C. Jeffries,and M. C. McCann. A rapid method to screen for cell-wall mu-tants using discriminant analysis of fourier transform infraredspectra. The Plant Journal, 16(3):385–392, 1998.
[10] L.-F. Chen, H.-Y. M. Liao, M.-T. Ko, J.-C. Lin, and G.-J. Yu.A new lda-based face recognition system which can solve the
20
Algorithm 4. : Direct-LDA.1: Read the training images (X = x1,x2, . . . ,xN),
where xi(ro×co) represents the ith training image,ro and co represent the rows (height) and columns(width) of xi, respectively, N represents the totalnumber of training images.
2: Convert all images in vector representation Z =z1,z2, . . . ,zN, where the dimension of Z is M×1,M = ro× co.
3: Calculate the mean of each class µi, total mean ofall data µ, between-class matrix SB(M×M), andwithin-class matrix SW (M×M) of X as in Algo-rithm (1, Step(2-5)).
4: Find the k eigenvectors of SB with non-zeros eigenvalues, and denote them as U =[u1,u2, . . . ,uk] (i.e. UT SbU > 0).
5: Calculate the eigenvalues and eigenvectors ofUT SWU , and then sort the eigenvalues and discardthe eigenvectors which have high values. The se-lected eigenvectors are denoted by V ; thus, V rep-resents the null space of SW .
6: The final LDA matrix (Ψ) consists of the range 17
SB and the null space of SW as follows, Ψ =UV .7: The original data are projected on the LDA space
as follows, Y = XΨ = XUV .
small sample size problem. Pattern recognition, 33(10):1713–1726, 2000.
[11] D. Coomans, D. Massart, and L. Kaufman. Optimization bystatistical linear discriminant analysis in analytical chemistry.Analytica Chimica Acta, 112(2):97–122, 1979.
[12] J. Cui and Y. Xu. Three dimensional palmprint recognitionusing linear discriminant analysis method. In Proceedings ofthe second International Conference on Innovations in Bio-inspired Computing and Applications (IBICA), 2011, pages107–111. IEEE, 2011.
[13] D.-Q. Dai and P. C. Yuen. Face recognition by regularized dis-criminant analysis. IEEE Transactions on Systems, Man, andCybernetics, Part B (Cybernetics), 37(4):1080–1085, 2007.
[14] D. Donoho and V. Stodden. When does non-negative ma-trix factorization give a correct decomposition into parts? InAdvances in neural information processing systems 16, pages1141–1148. MIT Press, 2004.
[15] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classification.John Wiley & Sons, Second Edition, 2012.
[16] S. Dudoit, J. Fridlyand, and T. P. Speed. Comparison of dis-crimination methods for the classification of tumors using geneexpression data. Journal of the American statistical associa-tion, 97(457):77–87, 2002.
[17] T.-t. Feng and G. Wu. A theoretical contribution to the fastimplementation of null linear discriminant analysis method us-ing random matrix multiplication with scatter matrices. arXivpreprint arXiv:1409.2579, 2014.
[18] J. H. Friedman. Regularized discriminant analysis. Journal ofthe American Statistical Association, 84(405):165–175, 1989.
[19] T. Gaber, A. Tharwat, A. E. Hassanien, and V. Snasel. Bio-metric cattle identification approach based on webers local de-scriptor and adaboost classifier. Computers and Electronics inAgriculture, 122:55–66, 2016.
[20] T. Gaber, A. Tharwat, A. Ibrahim, V. Snael, and A. E. Has-sanien. Human thermal face recognition based on random lin-ear oracle (rlo) ensembles. In International Conference on In-telligent Networking and Collaborative Systems, pages 91–98.IEEE, 2015.
[21] T. Gaber, A. Tharwat, V. Snasel, and A. E. Hassanien. Plantidentification: Two dimensional-based vs. one dimensional-based feature extraction methods. In 10th international confer-ence on soft computing models in industrial and environmentalapplications, pages 375–385. Springer, 2015.
[22] H. Gao and J. W. Davis. Why direct lda is not equivalent to lda.Pattern Recognition, 39(5):1002–1006, 2006.
[23] G. H. Golub and C. F. Van Loan. Matrix computations, vol-ume 3. Johns Hopkins University Press, Fourth Edition, 2012.
[24] R. Haeb-Umbach and H. Ney. Linear discriminant analysisfor improved large vocabulary continuous speech recognition.In IEEE International Conference on Acoustics, Speech, andSignal Processing (1992), volume 1, pages 13–16. IEEE, 1992.
[25] T. Hastie and R. Tibshirani. Discriminant analysis by gaus-sian mixtures. Journal of the Royal Statistical Society. Series B(Methodological), pages 155–176, 1996.
[26] K. Heberger, E. Csomos, and L. Simon-Sarkadi. Principal com-ponent and linear discriminant analyses of free amino acids andbiogenic amines in hungarian wines. Journal of agriculturaland food chemistry, 51(27):8055–8060, 2003.
[27] G. E. Hinton and R. R. Salakhutdinov. Reducing thedimensionality of data with neural networks. Science,313(5786):504–507, 2006.
[28] K. Honda and H. Ichihashi. Fuzzy local independent compo-nent analysis with external criteria and its application to knowl-edge discovery in databases. International journal of approxi-mate reasoning, 42(3):159–173, 2006.
[29] Q. Hu, L. Zhang, D. Chen, W. Pedrycz, and D. Yu. Gaus-sian kernel based fuzzy rough sets: Model, uncertainty mea-sures and applications. International Journal of ApproximateReasoning, 51(4):453–471, 2010.
[30] R. Huang, Q. Liu, H. Lu, and S. Ma. Solving the small samplesize problem of lda. In Proceedings of 16th International Con-ference on Pattern Recognition, 2002., volume 3, pages 29–32.IEEE, 2002.
[31] A. Hyvarinen, J. Karhunen, and E. Oja. Independent compo-nent analysis, volume 46. John Wiley & Sons, 2004.
[32] M. Kirby. Geometric data analysis: an empirical approach todimensionality reduction and the study of patterns. John Wiley& Sons, 2000.
[33] D. T. Larose. Discovering knowledge in data: an introductionto data mining. John Wiley & Sons, First Edition, 2014.
[34] T. Li, S. Zhu, and M. Ogihara. Using discriminant analy-sis for multi-class classification: an experimental investigation.Knowledge and information systems, 10(4):453–472, 2006.
21
[35] K. Liu, Y.-Q. Cheng, J.-Y. Yang, and X. Liu. An efficient algo-rithm for foley–sammon optimal set of discriminant vectors byalgebraic method. International Journal of Pattern Recognitionand Artificial Intelligence, 6(05):817–829, 1992.
[36] J. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos. Facerecognition using lda-based algorithms. IEEE Transactions onNeural Networks, 14(1):195–200, 2003.
[37] J. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos. Regular-ized discriminant analysis for the small sample size problemin face recognition. Pattern Recognition Letters, 24(16):3079–3087, 2003.
[38] J. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos. Regu-larization studies of linear discriminant analysis in small sam-ple size scenarios with application to face recognition. PatternRecognition Letters, 26(2):181–191, 2005.
[39] B. Moghaddam, Y. Weiss, and S. Avidan. Generalized spectralbounds for sparse lda. In Proceedings of the 23rd internationalconference on Machine learning, pages 641–648. ACM, 2006.
[40] W. Muller, T. Nocke, and H. Schumann. Enhancing the visu-alization process with principal component analysis to supportthe exploration of trends. In Proceedings of the 2006 Asia-Pacific Symposium on Information Visualisation, volume 60,pages 121–130. Australian Computer Society, Inc., 2006.
[41] S. Noushath, G. H. Kumar, and P. Shivakumara. (2d) 2 lda:An efficient approach for face recognition. Pattern recognition,39(7):1396–1400, 2006.
[42] K. K. Paliwal and A. Sharma. Improved pseudoinverse lineardiscriminant analysis method for dimensionality reduction. In-ternational Journal of Pattern Recognition and Artificial Intel-ligence, 26(01):1250002, 2012.
[43] F. Pan, G. Song, X. Gan, and Q. Gu. Consistent feature selec-tion and its application to face recognition. Journal of Intelli-gent Information Systems, 43(2):307–321, 2014.
[44] C. H. Park and H. Park. Fingerprint classification using fastfourier transform and nonlinear discriminant analysis. PatternRecognition, 38(4):495–503, 2005.
[45] C. H. Park and H. Park. A comparison of generalized lin-ear discriminant analysis algorithms. Pattern Recognition,41(3):1083–1097, 2008.
[46] S. Rezzi, D. E. Axelson, K. Heberger, F. Reniero, C. Mariani,and C. Guillou. Classification of olive oils using high through-put flow 1 h nmr fingerprinting with principal component anal-ysis, linear discriminant analysis and probabilistic neural net-works. Analytica Chimica Acta, 552(1):13–24, 2005.
[47] Y. Saeys, I. Inza, and P. Larranaga. A review of fea-ture selection techniques in bioinformatics. Bioinformatics,23(19):2507–2517, 2007.
[48] F. S. Samaria and A. C. Harter. Parameterisation of a stochas-tic model for human face identification. In Proceedings of theSecond IEEE Workshop on Applications of Computer Vision,1994., pages 138–142. IEEE, 1994.
[49] B. Scholkopf, C. J. Burges, and A. J. Smola. Advances in kernelmethods: support vector learning. MIT press, 1999.
[50] B. Scholkopf and K.-R. Mullert. Fisher discriminant analysiswith kernels. In Proceedings of the 1999 IEEE Signal Process-ing Society Workshop Neural Networks for Signal ProcessingIX, Madison, WI, USA, pages 41–48, 1999.
[51] B. Scholkopf, A. Smola, and K.-R. Muller. Nonlinear compo-nent analysis as a kernel eigenvalue problem. Neural computa-tion, 10(5):1299–1319, 1998.
[52] A. Sharma, S. Imoto, and S. Miyano. A between-classoverlapping filter-based method for transcriptome data anal-ysis. Journal of bioinformatics and computational biology,10(05):1250010, 2012.
[53] A. Sharma, S. Imoto, and S. Miyano. A filter based featureselection algorithm using null space of covariance matrix fordna microarray gene expression data. Current Bioinformatics,7(3):289–294, 2012.
[54] A. Sharma, S. Imoto, S. Miyano, and V. Sharma. Null spacebased feature selection method for gene expression data. In-ternational Journal of Machine Learning and Cybernetics,3(4):269–276, 2012.
[55] A. Sharma and K. K. Paliwal. Cancer classification by gradientlda technique using microarray gene expression data. Data &Knowledge Engineering, 66(2):338–347, 2008.
[56] A. Sharma and K. K. Paliwal. A new perspective to null lineardiscriminant analysis method and its fast implementation usingrandom matrix multiplication with scatter matrices. PatternRecognition, 45(6):2205–2213, 2012.
[57] A. Sharma and K. K. Paliwal. Linear discriminant analysisfor the small sample size problem: an overview. InternationalJournal of Machine Learning and Cybernetics, pages 1–12,2014.
[58] A. J. Smola and B. Scholkopf. A tutorial on support vectorregression. Statistics and computing, 14(3):199–222, 2004.
[59] G. Strang. Introduction to linear algebra. Wellesley-CambridgePress, Massachusetts, Fourth Edition, 2003.
[60] D. L. Swets and J. J. Weng. Using discriminant eigenfeaturesfor image retrieval. IEEE Transactions on pattern analysis andmachine intelligence, 18(8):831–836, 1996.
[61] M. M. Tantawi, K. Revett, A. Salem, and M. F. Tolba. Fiducialfeature reduction analysis for electrocardiogram (ecg) basedbiometric recognition. Journal of Intelligent Information Sys-tems, 40(1):17–39, 2013.
[62] A. Tharwat. Principal component analysis-a tutorial. Interna-tional Journal of Applied Pattern Recognition, 3(3):197–240,2016.
[63] A. Tharwat, T. Gaber, Y. M. Awad, N. Dey, and A. E. Has-sanien. Plants identification using feature fusion technique andbagging classifier. In The 1st International Conference on Ad-vanced Intelligent System and Informatics (AISI2015), Novem-ber 28-30, 2015, Beni Suef, Egypt, pages 461–471. Springer,2016.
[64] A. Tharwat, T. Gaber, and A. E. Hassanien. One-dimensionalvs. two-dimensional based features: Plant identification ap-proach. Journal of Applied Logic, 2016.
[65] A. Tharwat, T. Gaber, A. E. Hassanien, H. A. Hassanien, andM. F. Tolba. Cattle identification using muzzle print imagesbased on texture features approach. In Proceedings of the FifthInternational Conference on Innovations in Bio-Inspired Com-puting and Applications IBICA 2014, pages 217–227. Springer,2014.
[66] A. Tharwat, A. E. Hassanien, and B. E. Elnaghi. A ba-based al-gorithm for parameter optimization of support vector machine.Pattern Recognition Letters, 2016.
22
[67] A. Tharwat, A. Ibrahim, A. E. Hassanien, and G. Schaefer.Ear recognition using block-based principal component analy-sis and decision fusion. In International Conference on Pat-tern Recognition and Machine Intelligence, pages 246–254.Springer, 2015.
[68] A. Tharwat, H. Mahdi, A. El Hennawy, and A. E. Hassanien.Face sketch synthesis and recognition based on linear regres-sion transformation and multi-classifier technique. In The1st International Conference on Advanced Intelligent Systemand Informatics (AISI2015), November 28-30, 2015, Beni Suef,Egypt, pages 183–193. Springer, 2016.
[69] A. Tharwat, Y. S. Moemen, and A. E. Hassanien. Classificationof toxicity effects of biotransformed hepatic drugs using whaleoptimized support vector machines. Journal of Biomedical In-formatics, 2017.
[70] C. G. Thomas, R. A. Harshman, and R. S. Menon. Noise reduc-tion in bold-based fmri using component analysis. Neuroim-age, 17(3):1521–1537, 2002.
[71] M. Turk and A. Pentland. Eigenfaces for recognition. Journalof cognitive neuroscience, 3(1):71–86, 1991.
[72] V. Vapnik. The nature of statistical learning theory. SpringerScience & Business Media, Second Edition, 2013.
[73] J. Venna, J. Peltonen, K. Nybo, H. Aidos, and S. Kaski. Infor-mation retrieval perspective to nonlinear dimensionality reduc-tion for data visualization. The Journal of Machine LearningResearch, 11:451–490, 2010.
[74] P. Viszlay, M. Lojka, and J. Juhar. Class-dependent two-dimensional linear discriminant analysis using two-pass recog-nition strategy. In Proceedings of the 22nd European SignalProcessing Conference (EUSIPCO), pages 1796–1800. IEEE,2014.
[75] X. Wang and X. Tang. Random sampling lda for face recogni-tion. In Proceedings of the 2004 IEEE Computer Society Con-ference on Computer Vision and Pattern Recognition (CVPR),volume 2, pages II–II. IEEE, 2004.
[76] M. Welling. Fisher linear discriminant analysis. Department ofComputer Science, University of Toronto, 3, 2005.
[77] M. C. Wu, L. Zhang, Z. Wang, D. C. Christiani, and X. Lin.Sparse linear discriminant analysis for simultaneous testingfor the significance of a gene set/pathway and gene selection.Bioinformatics, 25(9):1145–1151, 2009.
[78] J. Yang, D. Zhang, A. F. Frangi, and J.-y. Yang. Two-dimensional pca: a new approach to appearance-based face rep-resentation and recognition. IEEE Transactions on PatternAnalysis and Machine Intelligence, 26(1):131–137, 2004.
[79] L. Yang, W. Gong, X. Gu, W. Li, and Y. Liang. Null space dis-criminant locality preserving projections for face recognition.Neurocomputing, 71(16):3644–3649, 2008.
[80] W. Yang and H. Wu. Regularized complete linear discriminantanalysis. Neurocomputing, 137:185–191, 2014.
[81] J. Ye, R. Janardan, and Q. Li. Two-dimensional linear discrim-inant analysis. In Proceedings of 17th Advances in Neural In-formation Processing Systems (NIPS), pages 1569–1576, 2004.
[82] J. Ye and T. Xiong. Computational and theoretical analysisof null space and orthogonal linear discriminant analysis. TheJournal of Machine Learning Research, 7:1183–1204, 2006.
[83] H. Yu and J. Yang. A direct lda algorithm for high-dimensionaldata with application to face recognition. Pattern recognition,34(10):2067–2070, 2001.
[84] L. Yuan and Z.-c. Mu. Ear recognition based on 2d images. InProceedings of the first IEEE International Conference on Bio-metrics: Theory, Applications, and Systems, 2007. BTAS 2007.,pages 1–5. IEEE, 2007.
[85] X.-S. Zhuang and D.-Q. Dai. Inverse fisher discriminate cri-teria for small sample size problem and its application to facerecognition. Pattern Recognition, 38(11):2192–2194, 2005.