general and interval type-2 fuzzy face-space approach to emotion recognition

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS: SYSTEMS, VOL. 43, NO. 3, MAY 2013 587

General and Interval Type-2 Fuzzy Face-SpaceApproach to Emotion Recognition

Anisha Halder, Amit Konar, Rajshree Mandal, Aruna Chakraborty,Pavel Bhowmik, Nikhil R. Pal, Fellow, IEEE, and Atulya K. Nagar

Abstract—Facial expressions of a person representing similaremotion are not always unique. Naturally, the facial features ofa subject taken from different instances of the same emotion havewide variations. In the presence of two or more facial features, thevariation of the attributes together makes the emotion recognitionproblem more complicated. This variation is the main source ofuncertainty in the emotion recognition problem, which has beenaddressed here in two steps using type-2 fuzzy sets. First a type-2fuzzy face space is constructed with the background knowledge offacial features of different subjects for different emotions. Second,the emotion of an unknown facial expression is determined basedon the consensus of the measured facial features with the fuzzy facespace. Both interval and general type-2 fuzzy sets (GT2FS) havebeen used separately to model the fuzzy face space. The intervaltype-2 fuzzy set (IT2FS) involves primary membership functionsfor m facial features obtained from n-subjects, each having l-in-stances of facial expressions for a given emotion. The GT2FS in ad-dition to employing the primary membership functions mentionedabove also involves the secondary memberships for individualprimary membership curve, which has been obtained here byformulating and solving an optimization problem. The optimiza-tion problem here attempts to minimize the difference betweentwo decoded signals: the first one being the type-1 defuzzificationof the average primary membership functions obtained from then-subjects, while the second one refers to the type-2 defuzzifiedsignal for a given primary membership function with secondarymemberships as unknown. The uncertainty management policyadopted using GT2FS has resulted in a classification accuracy of98.333% in comparison to 91.667% obtained by its interval type-2counterpart. A small improvement (approximately 2.5%) in clas-sification accuracy by IT2FS has been attained by pre-processingmeasurements using the well-known interval approach.

Index Terms—Emotion recognition, facial feature extraction,fuzzy face space, interval and general type-2 fuzzy sets, intervalapproach (IA).

Manuscript received June 25, 2011; revised November 26, 2011 andMarch 31, 2012; accepted May 24, 2012. Date of publication March 7, 2013;date of current version April 12, 2013. This paper was recommended byAssociate Editor S.-M. Chen.

A. Halder, A. Konar, and R. Mandal are with the Department of Electronicsand Tele-Communication Engineering, Jadavpur University, Calcutta 700 032,India (e-mail: [email protected]; [email protected];[email protected]).

A. Chakraborty is with the St. Thomas’ College of Engineering andTechnology, Khiddirpore, Calcutta 700023, India (e-mail: [email protected]).

P. Bhowmik is with the Department of Electrical Engineering, University ofFlorida, Gainesville, FL 32611 USA (e-mail: [email protected]).

N. R. Pal is with the ECSU, Indian Statistical Institute, Barrackpur, Calcutta700 035, India (e-mail: [email protected]).

A. K. Nagar is with the Department of Math and Computer Science, Liver-pool Hope University, L16 9JD Liverpool, U.K. (e-mail: [email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TSMCA.2012.2207107

I. INTRODUCTION

EMOTION recognition is currently gaining importance forits increasing scope of applications in human–computer

interactive systems. Several modalities of emotion recogni-tion, including facial expression, voice, gesture, and posturehave been studied in the literature. However, irrespective ofthe modality, emotion recognition comprises two fundamentalsteps involving feature extraction and classification [36]. Fea-ture extraction refers to determining a set of features/attributes,preferably independent, which together represents a given emo-tional expression. Classification aims at mapping emotionalfeatures into one of several emotion classes.

Performance of an emotion recognition system greatly de-pends on feature selection and classifier design. A good clas-sification algorithm sometimes cannot yield high classificationaccuracy for poorly selected features. On the other hand, evenusing a large set of features, describing an emotion, we oc-casionally fail to recognize the emotion correctly because ofa poor classifier. Most commonly used techniques for featureselection in the emotion recognition problem include principalcomponent analysis (PCA) [59], independent component anal-ysis [60], rough sets [42], [61], Gabor filter [62], and Fourierdescriptors [25]. Among the popularly used techniques foremotion classification, neural net-based mapping [3], [4], [18],fuzzy relational approach [14], linear discriminate analysis[60], support vector machine (SVM) [8], and hidden Markovmodel [59], [62] need special mention. A brief overview of theexisting research on emotion recognition is given next.

Ekman and Friesen took an early attempt to recognizefacial expression from the movements of cheek, chin, andwrinkles [24]. Their experiments confirmed the existence of agood correlation between basic movements of the facial actionunits [13], [19] and facial expressions [1], [2], [5], [7], [10],[19]–[22]. Kobayashi and Hara [15]–[17] designed a schemefor the recognition of human facial expressions using thewell-known back-propagation neural networks [38], [43]. Theirscheme is capable of recognizing six common facial expres-sions depicting happiness, sadness, fear, anger, surprise, anddisgust. Yamada proposed an alternative method of emotionrecognition through classification of visual information [49].

Fernandez-Dols et al. proposed a scheme for decoding emo-tions from facial expressions and content [50]. Kawakami et al.[43] designed a method for the construction of emotion spaceusing neural networks. Busso and Narayanan [51] analyzed thescope of facial expressions, speech, and multi-modal informa-tion in emotion recognition. Metallinou et al. [71] employed

2168-2216/$31.00 © 2013 IEEE

588 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS: SYSTEMS, VOL. 43, NO. 3, MAY 2013

content-sensitive learning for audio-visual emotion recognition.In [73], Metallinou et al. proposed a novel approach to visualemotion recognition using a compact representation of faceand viseme information. In [74], Metallinou et al. presentedan approach to decision level fusion for handling multi-modalinformation in emotion recognition. Lee et al. [75] employed ahierarchical binary tree for emotion recognition. Mower et al.designed an interesting scheme about human perception ofaudio-visual synthetic emotion character in the presence ofconflicting information [76]. Cohen et al. [52] developed ascheme for emotion recognition from the temporal variationsin facial expressions obtained from the live video sequence ofthe subjects. They used hidden Markov model to automaticallysegment and recognize facial expression. Gao et al. presenteda scheme for facial expression recognition from a single facialimage using line based caricatures [53]. Among other signifi-cant contributions in emotion recognition, the works presentedin [6], [8], [9], [11], [12], [15]–[17], [23]–[28], [30]–[32], [35],[40], [46], [56], [57], [60], [70], [72], [77]–[80] need specialmention. For a more complete literature survey, which cannotbe given here for space restriction, readers may refer to twooutstanding papers by Pantic et al. [57], [67].

Emotional features greatly depend on the psychologicalstates of the subjects. For example, facial expressions of asubject, while experiencing the same emotion, have widervariations, resulting in significant changes in individual feature.Further, different subjects experiencing the same emotion havedifferences in their facial features. Repeated experiments witha large number of subjects, each having multiple instances ofsimilar emotional experience, reveal that apparently there existsa small but random variation of facial features around specificfixed points [65]. The variation between different instances offacial expression for similar emotive experience of an individ-ual can be regarded as an intra-personal level uncertainty [41].On the other hand, the variation in facial expression of individ-uals for similar emotional experience can be treated as inter-personal level uncertainty [41].

The variations in features can be modeled with fuzzy sets.Classical (type-1 (T1)) fuzzy sets, pioneered by Zadeh [66],have widely been used over the last five decades for modelinguncertainty of ill-defined systems. T1 fuzzy sets employ a sin-gle membership function to represent the degree of uncertaintyin measurements of a given feature. Hence, it can capturethe variation in measurements of a given feature for differentinstances of a specific emotion experienced by a subject. In[14], the authors have considered a fixed membership functionto model the uncertainty involved in a feature for a given emo-tion, disregarding the possibility of variation in the membershipcurves for different subjects.

This paper, however, models the above form of inter-personallevel uncertainty by interval type-2 (T2) fuzzy sets (IT2FS).IT2FS employs an upper and a lower membership function(UMF and LMF) to capture the uncertainty involved in agiven measurement of a feature within the bounds of its twomembership curves at the point of the measurement. However,the degree of correct assignment of membership for eachmembership curve embedded between the UMF and LMF inIT2FS is treated as unity, which is not always appropriate.

General T2 fuzzy set (GT2FS) can overcome the above problemby considering a secondary membership grade that representsthe correctness in (primary) membership assignment at eachmeasurement points. Naturally, GT2FS is expected to give usbetter results in emotion classification for its representationaladvantage over IT2FS.

One fundamental problem in GT2FS that limits its appli-cation in classification problems, perhaps, is due to users’ in-ability to correctly specify the secondary memberships. In thispaper, we determine the secondary memberships by extractingcertain knowledge from the individual primary assignments foreach feature of a given emotion for a subject. The knowledgeextracted is encoded as an optimization problem with secondarymemberships as unknown. The solution to the optimizationproblem carried out offline provides the secondary grades.The secondary grades are later aggregated with the primarymemberships of individual feature for all subjects at the givenmeasurement point to obtain modified primary memberships.

The paper provides two alternative approaches to emotionrecognition from an unknown facial expression, when the emo-tion class of individual facial expression of a large number ofexperimental subjects is available. The first approach deals withIT2FS to construct a fuzzy face space based on the measure-ments of a set of features from a given set of facial expressionscarrying different emotions. An unknown facial expression isclassified into one of several emotion classes by determiningthe maximum support of individual emotion classes to a givenset of measurements of a facial expression. The class having themaximum support is declared as the emotion of the unknownfacial expression. In spirit, this is similar to how a fuzzy rule-based system for classification works.

The second approach employs GT2FS to construct a fuzzyface space, comprising both primary and secondary member-ship functions, obtained from known facial expressions of sev-eral subjects containing multiple instances of the same emotionfor each subject. The emotion class of an unknown facial ex-pression is determined by computing the support of each classto the given facial expression. The class with the maximumsupport is the winner. The maximum support evaluation hereemploys both primary and secondary memberships, and thus isslightly different than the IT2FS-based classification.

Experiments reveal that the classification accuracy of emo-tion of an unknown person by the GT2FS-based scheme isas high as 98%. When secondary memberships are ignored,and classification is performed with IT2FS, the classificationaccuracy falls by a margin of 7%. The additional 7% classi-fication accuracy obtained by GT2FS, however, has to pay aprice for additional complexity of (m× n× k) multiplications,where m, n, and k denote the number of features, numberof subjects, and number of emotion classes, respectively. A2.5% improvement in classification accuracy by IT2FS hasbeen attained by pre-processing measurements and selectingmembership functions using the well-known interval approach(IA) [68].

The paper is divided into eight sections. Section II providesfundamental definitions associated with T2 fuzzy sets, whichwill be required in the rest of the paper. In Section III, wepropose the principle of uncertainty management in fuzzy face

HALDER et al.: GENERAL AND INTERVAL TYPE-2 FUZZY FACE-SPACE APPROACH TO EMOTION RECOGNITION 589

space for emotion recognition. Section IV deals with secondarymembership evaluation procedure for a given T2 primarymembership function. A scheme for selection of membershipfunction and data filtering to eliminate poor measurements toimprove the performance of IT2FS-based recognition is givenin Section V. Experimental details are given in Section VI,and two methods of performance analysis are undertaken inSection VII. Conclusions are listed in Section VIII.

II. PRELIMINARIES ON T2 FUZZY SETS

In this section, we define some terminologies related to T1and T2 fuzzy sets. These definitions will be used throughoutthe paper.

Definition 1: Given a universe of discourse X , a conven-tional T1 fuzzy set A defined on X , is given by a 2-D mem-bership function, also called T1 membership function. The(primary) membership function, denoted by μA(x), is a crispnumber in [0, 1] for a generic element x ∈ X . Usually, thefuzzy set A is expressed as a two tuple [36], given by

A = {(x, μA(x)) |∀x ∈ X} . (1)

An alternative representation of the fuzzy set A is also foundin the literature as given in (2).

A =

∫x∈X

μA(x)|x (2)

where∫

denotes union of all admissible x.Definition 2: A T2 fuzzy set A is characterized by a 3-D

membership function, also called T2 membership function,which itself is fuzzy. The T2 membership function is usuallydenoted by μA(x, u), where x ∈ X , and u ∈ Jx ⊆ [0, 1] [39].Usually, the fuzzy set A is expressed as a two tuple:

A = {((x, u), μA(x, u)) |x ∈ X,u ∈ Jx ⊆ [0, 1]} (3)

where μA(x, u) ∈ [0, 1]. An alternative form of representationof the T2 fuzzy set is given in (4)

A =

∫x∈X

∫u∈Jx

μA(x, u)|(x, u), Jx ⊆ [0, 1] (4)

=

∫x∈X

⎡⎢⎣

∫u∈Jx

fx(u)

u

⎤⎥⎦ /x, Jx ⊆ [0, 1] (5)

where fx(u) = μA(x, u) ∈ [0, 1]. The∫ ∫

denotes union overall admissible x and u [39].

Definition 3: At each point of x, say x = x/, the 2-D planecontaining axes u and μ(x/, u) is called the vertical slice ofμA(x, u). A secondary membership function is a vertical sliceof μA(x, u). Symbolically, it is given by μA(x, u) at x = x/ forx/ ∈ X and ∀u ∈ Jx/ ⊆ [0, 1]

μA(x = x′, u) =

∫u∈J

x/

fx/(u)|u, Jx/ ⊆ [0, 1] (6)

where 0 ≤ fx/(u) ≤ 1. The amplitude of a secondary member-ship function is called secondary grade (of membership). In (6)Jx/ is the primary membership of x/.

Definition 4: Uncertainty in the primary membership of a T2fuzzy set A is represented by a bounded region, called footprintof uncertainty (FOU) [39], which is the defined as the union ofall primary memberships, i.e.,

FOU(A) = ∪x∈U

Jx. (7)

If all the secondary grades of a T2 fuzzy set A are equal to 1,i.e.,

μA(x, u) = 1∀x ∈ X,∀u ∈ Jx ⊆ [0, 1] (8)

then A is called IT2FS. The FOU is bounded by two curves,called the Lower and the Upper Membership functions, denotedby μ

A(x) and μA(x), respectively, where μ

A(x) and μA(x) at

all x, respectively, take up the minimum and the maximum ofthe membership functions of the embedded T1 fuzzy sets [38]in the FOU.

III. UNCERTAINTY MANAGEMENT IN FUZZY

FACE SPACE FOR EMOTION RECOGNITION

This section provides a general overview of the proposedscheme for emotion recognition using T2 fuzzy sets. Here,the emotion recognition problem is considered as uncertaintymanagement in fuzzy space after encoding the measured facialattributes by T2 fuzzy sets.

Let F = {f1, f2, . . . , fm} be the set of m facial features. LetμA(fi) be the primary membership in [0,1] of the feature fito be a member of set A, and μ(fi, μA(fi)) be the secondarymembership of the measured variable fi in [0,1]. A primaryand secondary membership function corresponds to a particularemotion class c, are denoted by μAc(fi) and μ(fi, μAc(fi)),respectively. If the measurement of a facial feature, fi, isperformed p times on the same subject experiencing the sameemotion, and the measurements are quantized into q intervalsof equal size, we can evaluate the frequency of occurrence ofthe measured variable fi in q quantized intervals. The intervalcontaining the highest frequency of occurrence then can beidentified, and its center, mi, approximately represents themode of the measurement variable fi. The second moment,σi, around mi is determined and a bell-shaped (Gaussian)membership function centered at mi and with a spread σi

is used to represent the membership function of the randomvariable fi. This function represents the membership of fi tobe CLOSE-TO the central value, mi. It may be noted that abell-shaped (Gaussian-like) membership curve would have apeak at the center with a membership value one, indicating thatmembership at this point is the largest for an obvious reason ofhaving the highest frequency of fi at the center.

On repetition of the above experiment for variable fi on nsubjects, each experiencing the same emotion, we obtain n suchmembership functions, each one for one individual subject.Naturally, the measurement variable fi now has both intra-and inter-personal level uncertainty. The intra-level uncertainty


Fig. 1 Experimental FOU for feature fi = Mouth-Opening.

occurs due to the pre-assumption of a specific (Gaussian)primary membership function, and the inter-level uncertaintyoccurs due to multiplicity of the membership functions forn subjects. Thus, a new measurement for an unknown facialexpression can be encoded using all the n-membership curves,giving n possible membership values, thereby giving rise touncertainty in the fuzzy space.

The uncertainty involved in the present problem has beenaddressed here by three distinctive approaches: 1) IT2FS,2) IA-IT2FS, and 3) GT2FS. The first approach is simple,but more error prone as it ignores the intra-level uncertainty.The second and the third approaches are robust as they arecapable to take care of both the uncertainties. However, themodality of uncertainty management by the second and thethird approaches is significantly different. The second approachmodels each subject’s interval using a uniform probabilitydistribution, and thus the mean and variance of each intervalare mapped into an embedded T1 fuzzy set. The third approachhandles intra- and inter-personal level uncertainty compositelyby fusing the primary and the secondary membership functionsinto an embedded interval T2 membership function. All threeapproaches have many common steps. Hence, we first presentthe steps involved in IT2FS and then explain the two techniqueswithout repeating the common steps further.

A. Principles Used in the IT2FS Approach

The primary membership functions for a given featurevalue fi corresponding to a particular emotion c taken fromn-subjects together forms a IT2FS Ac, whose FOU is boundedby a lower and an upper membership curves μ

Ac(fi) and

μAc(fi), respectively, where

μAc

(fi) =Min{μ1Ac

(fi), μ2Ac

(fi), . . . , μnAc

(fi)}, (9)

μAc(fi) =Max{μ1Ac

(fi), μ2Ac

(fi), . . . , μnAc

(fi)}

(10)

are evaluated for all fi, and μj

Ac(fi), 1 ≤ j ≤ n denotes the

primary membership function of feature fi for subject j inIT2FS Ac.

Fig. 1 provides the FOU for a given feature fi.Now, for a given measurement f

/i , we obtain an interval

[μAc

(f/i ), μAc(f

/i )], representing the entire span of uncertainty

of the measurement variable f/i in the fuzzy space, induced by

n primary membership distributions: μj

Ac(fi), 1 ≤ j ≤ n. The

interval [μAc

(f/i ), μAc(f

/i )] is evaluated by replacing fi by f

/i

in (9) and (10), respectively.If there exist m different facial features, then for each feature,

we would have such an interval, and consequently we obtain msuch intervals given by[μAc

(f/1

), μAc

(f/1

)],[μAc

(f/2

), μAc

(f/2

)], . . . . . . . . . ,[

μAc

(f/m

), μAc

(f/m

)].

The proposed IT2FS reasoning system employs a particularformat of rules, commonly used in fuzzy classification prob-lems [47]. Consider for instance a fuzzy rule, given by Rc:if f1 is A1 AND f2 is A2 . . .. AND fm is Am then emotionclass is c.

Here, fi for i = 1 to m are m-measurements (feature values)and A1, A2, . . . , Am are IT2FS on the respective domains

Ai =[μAc

(fi), μAc(fi)], ∀i. (11)

Since an emotion is characterized by all of these m features,to find the overall support of the m features (m measurementsmade for the unknown subject) to the emotion class c repre-sented by the n primary memberships, we use the fuzzy meetoperation

Sminc =Min

{μAc

(f/1

), μ

Ac

(f/2

). . . , μ

Ac

(f/m

)}(12)

Smaxc =Min

{μAc

(f/1

), μAc

(f/2

), . . . , μAc

(f/m

)}. (13)

Thus, we can say that the unknown subject is experiencingthe emotion class c at least to the extent smin

c , and at most tothe extent smax

c .To reduce the nonspecificity associated with the interval

sc−i = [sminc , smax

c ], different approaches can be taken. Forexample, the most conservative approach would be to use lowerbound, while the most liberal view would be to use the upperbound of the interval as the support for the class c. In theabsence of any additional information, a balanced approachwould be to use center of the interval as the support for the classc by the n primary memberships to the unknown subject. Thisidea is supported by Mendel [42] and Lee [48]. We compute thecenter Sc of the interval Sc−i

Sc =

(sminc + smax

c

)2

. (14)

Thus, Sc is the degree of support that the unknown facialexpression is in emotion class c. Now, to predict the emotion ofa person from his facial expression, we determine Sc for eachemotion class. Presuming that there exist k emotion classes, letus denote the degree by which the emotion classes 1, 2, . . . , ksupport the unknown facial expression be S1, S2, . . . ., Sk, re-spectively. Since a given facial expression may convey differentemotions with different degrees, we resolve the conflict by


ranking the Si for i = 1 to k, and thus determine the emotionclass r, for which Sr >= Si for all i.

The principle of selection of the emotion class r from a setof competitive emotions, satisfying the above inequality holds,since the joint occurrence of the fuzzy memberships, inducedby (12)–(14), for all the features of the given facial expressionfor emotion r is the greatest among the same values for all otheremotions.

B. Principles Used in the GT2FS Approach

The previous approach employs a reasoning mechanism tocompute the degree of support of k emotion classes inducedby m features for each class to an unknown facial expressionusing a set of k ×m IT2FS. The GT2FS-based reasoningrealized with measurements taken from n-subjects, however,requires k ×m× n GT2FSs to determine the emotion class ofan unknown facial expression. The current approach tunes theprimary membership values for the given measurements usingthe secondary memberships of the same measurement, and thusreduces the degree of intra-level uncertainty of the primarydistributions. The reduction in the degree of uncertainty helpsin improving the classification accuracy of emotion at the costof additional complexity required to evaluate T2 secondarydistributions and also to reason with k ×m× n fuzzy sets.

Let fi be the measurement of the ith feature for a subject withan unknown emotion class. Now, by consulting the n primarymembership functions that were generated from n-subjects inthe training data for a given emotion class, c, we obtain n pri-mary membership values corresponding to fi for emotion classc as given by μ1

Ac(fi), μ

2Ac

(fi), . . . , μnAc

(fi). Let the secondarymembership values for each primary membership value, respec-tively, be μ(fi, μ

1Ac

(fi)), μ(fi, μ2Ac

(fi)), . . . , μ(fi, μnAc

(fi)).Note that, these secondary membership values correspond toemotion class c. Unless clarity demands, we have avoided (hereand elsewhere) use of a subscript to represent the emotionclass. We now fuse (aggregate) the evidences provided bythe primary and secondary membership values to obtain themodified primary membership supports. A plausible way offusing would be to use a T-norm. Here, we use the product. Theproduct always lies within the FOU and thus satisfies Mendel-John Representation Theorem [39]. Further higher is the sec-ondary membership, higher is the product representing newembedded fuzzy membership. Since the secondary membershiprepresents the degree of correctness in primary membership,the product helps in reduction of intra-level uncertainty.Thus, for subject j of the training data representing emotionclass c, we obtain

modμj

Ac(fi)=μj

Ac(fi)×μ

(fi, μ

j

Ac(fi)

)∀j=1, . . . , n (15)

where modμj

Ac(fi) denotes the modified primary membership

value for jth training subject for cth emotion class. The sec-ondary membership values used in the above product functionare evaluated using their primary memberships obtained by aprocedure discussed in Section IV.

The next step is to determine the range of modμj

A(f

/i ) for

j = 1 to n, comprising the minimum and the maximum givenby [modμ

A(f

/i ),

mod μA(f/i )], where

modμA

(f/i

)=Min

{modμ1

A

(f/i

),

modμ2A

(f/i

), . . . ,modμn

A

(f/i

)}(16)

modμA

(f/i

)=Max

{modμ1

A

(f/i

),

modμ2A

(f/i

), . . . ,modμn

A

(f/i

)}. (17)

Now, for m features, the rule-based T2 classification isperformed in a similar manner as in the previous section withthe replacement of μ

A(f

/i ) and μA(f

/i ) by modμ

A(f

/i ) and

modμA(f/i ), respectively.

C. Methodology

We briefly discuss the main steps involved in fuzzy face-space construction based on the measurements of m facial fea-tures for n-subjects, each having l instances of facial expressionfor a particular emotion. We need to classify a facial expressionof an unknown person into one of k emotion classes.

IT2FS-Based Emotion Recognition:

1) We extract m facial features for n subjects, each havingl (l could be different for different emotion classes)instances of facial expression for a particular emotion.The above features are extracted for k-emotion classes.

2) We construct a fuzzy face space for each emotion classseparately. The fuzzy face space for an emotion classcomprises a set of n primary membership functions foreach feature. Thus, we have m groups (denoted by m rowsof blocks in Fig. 2) of n-primary membership functions(containing n blocks under each row of Fig. 2). Eachprimary membership curve is constructed from l-facialinstances of a subject attempted to exhibit a particularemotion in her facial expression by acting.

3) For a given set of features f/1 , f

/2 , . . . , f

/m obtained from

an unknown facial expression, we determine the range ofmembership for feature f

/i , given by [μ

A(f

/i ), μA(f

/i )],

where A is an IT2FS with a primary membership functiondefined as CLOSE-TO-center-value-m of the respectivemembership function.

4) Now, for an emotion class j, we take fuzzy meet operationover the ranges for each feature to evaluate the range ofuncertainty for individual emotion class. The meet opera-tion here is computed by taking cumulative t-norm (herewe use min) of μ

A(f

/i ) and μA(f

/i ) separately for i = 1

to m, and thus obtaining Sminj and Smax

j , respectively(see top of Fig. 2).

5) The support of the j-th emotion class to the measure-ments is evaluated by computing the average Sj of Smin

j

and Smaxj .


Fig. 2. The IT2FSS-based emotion recognition.

6) Now, we determine the maximum support offered by allthe k emotion classes, and declare the unknown facialexpression to have emotion r, if Sr ≥ Si for all emotionclass i = 1 to k. The suffix j in [μmin

A (f/i ), μ

maxA (f

/i )]j

refers to the range in that interval for emotion j.

GT2FS-Based Emotion Recognition:

1) This step is same as the step 1 of IT2FS-based emotionrecognition.

2) The construction of the primary membership functionshere follows the same procedure as given in step 2 ofIT2FS-based recognition scheme. In addition, we need toconstruct secondary membership functions for individualprimary membership curves. The procedure for construc-tion of secondary membership functions will be discussedin Section IV. The complete scheme of construction ofT2FFS, considering all k emotion classes, is given inFig. 3.

3) For a given feature f/i , we consult each primary and

secondary membership curve under a given emotion

class, and take the product of primary and secondarymembership at fi = f

/i . The resulting membership value

obtained for the membership curves for the subject w inthe training data is given by

modμwA

(f/i

)= μw

A

(f/i

)× μ

(f/i , μ

wA

(f/i

))(18)

where the notations have their usual meaning. Now, forw = 1 to n, we evaluate modμw

A(f

/i ), and thus obtain the

minimum and the maximum values of modμwA(f

/i ), to

obtain a range of uncertainty [modμA(f

/i ),

mod μA(f/i )].

This is repeated for all features under each emotion class.In Fig. 4 we, unlike conventional approaches, presentsecondary membership functions against feature f

/i , for

i = 1 to m. Such representation is required to demon-strate the computation of modμ

A(f

/i ).

4) Step 4 is the same as that in IT2FS-based recognitionscheme with the replacement of μ

A(f

/i ) and μA(f

/i ),


Fig. 3. General type-2 fuzzy face-space construction for m features, k emo-tion classes, and n subjects.

respectively, by modμA(f

/i ) and modμA(f

/i ). Steps 5 and

6 are exactly similar to those in IT2FS-based recognitionscheme. A complete scheme for GT2FS-based emotionrecognition, considering support of k-emotion classes isgiven in Fig. 5.

IV. FUZZY T2 MEMBERSHIP EVALUATION

In this, we discuss T2 membership evaluation [37]–[39].Although theoretically very sound, T2 fuzzy set has limitedlybeen used over the last two decades because of the users’inadequate knowledge to correctly assign the secondary mem-berships. This paper, however, overcomes this problem byextracting T2 membership function from its T1 counterpart byan evolutionary algorithm. A brief outline to the construction ofsecondary membership function is given in this section.

Intuitively, when an expert assigns a grade of membership,she is relatively more certain to determine the location of thepeaks and the minima of the function, but may not have enoughbackground knowledge to correctly assign the membership val-ues at other points. Presuming that the (secondary) membershipvalues at the peak and the minima are close to 1, we attempt tocompute secondary memberships at the remaining part of thesecondary membership function. The following assumptionsare used to construct an objective function, which is minimizedto obtain the solution of the problem.

1) Let x = xp and x = xq be two successive optima(peak/minimum) on the primary membership functionμA(x). Then, at any point x lying between xp and xq,the secondary membership μ(x, μA(x)) will be smallerthan both μ(xp, μA(xp)) and μ(xq, μA(xq)).

2) The fall-off in secondary membership at a point x awayfrom its value at a peak/minimum μ(xp, μA(xp)) is expo-nential, given by

μ (x, μA(x)) = μ (xp, μA(xp)) . exp (−|x− xp|) . (19)

3) The secondary membership at any point x between twoconsecutive optima at x = xp and x = xq in the primarymembership is selected from the range [α, β], where

α = μ (xp, μA(xp)) . exp (−|x− xp|)β = μ (xq, μA(xq)) . exp (−|x− xq|)

}. (20)

T1 defuzzification over the average of n primary member-ship functions should return the same value as obtainedby T2 defuzzification for a given primary membershipfunction for any given source. This assumption holdsbecause the two modalities of defuzzification, represent-ing the same real-world parameter, should return closevalues, ignoring the average inter-personal level of uncer-tainty while taking the average of n-primary membershipfunctions.

4) The unknown secondary membership at two values ofx separated by a small positive δ should have a smalldifference. This is required to avoid sharp changes in thesecondary grade.

Let the primary membership functions for feature fi = xfrom n sources be μ1

A(x), μ2

A(x), . . . ., μn

A(x). Then, the aver-

age membership function which represents a special form offuzzy aggregation is given by

μA(x) =

n∑i=1

μiA(x)

n, ∀x (21)

i.e., at each position of x = xj , the above membership aggre-gation is employed to evaluate a new composite membershipprofile μA(x). The defuzzified signal obtained by the centroidmethod [36] from the averaged primary membership functionis given by

c =

∑∀x

x.μA(x)∑∀x

μA(x). (22)


Fig. 4. Computing support of the general type-2 fuzzy FS for emotion class i.


Fig. 5. GT2FFS-based emotion classification.

Further, the T2 centroidal defuzzified signal obtained fromthe ith primary and secondary membership functions here isdefined as

ci =

∑∀x

x.μiA(x).μ

(x, μi

A(x)

)∑∀x

μiA(x).μ

(x, μi

A(x)

) . (23)

The products of primary and secondary memberships areused in (23) to refine the primary memberships by the degreeof certainty of the corresponding secondary values.

Using assumptions 3 and 4, we construct a performanceindex Ji to compute secondary membership for the ith subjectfor a given emotion

Ji=(ci−c)2+

xR−1∑x=x1

{μ((x+δ), μi

A(x+δ)

)−μ

(x, μi

A(x)

)}2.

(24)

The second term in (24) acts as a regularizing term to preventabrupt changes in the membership function. In (24),x1 andxR are the smallest and the largest values of a given featureconsidered over R sampled points of μi

A(x). In (24), δ = (xR −

x1)/(R− 1) and xk = x1 + (k − 1). δ for k = 1, . . . ., R. Thesecondary membership evaluation problem now transforms tominimization of Ji by selecting μ(x, μi

A(x)) from a given

range [α, β], where α and β are the secondary membershipsat the two optima in secondary membership around the pointx. Expressions (20) are used to compute α and β for eachx separately. Note that, for each subject carrying individualemotion, we have to define (23) and (24) and find the optimalsecondary membership functions.

Any derivative-free optimization algorithm can be used tominimize Ji with respect to secondary memberships, and obtainμ(x, μi

A(x)) at each x except the optima on the secondary

membership. Differential evolution (DE) [34] is one suchderivative-free optimization algorithm, which has fewer con-trol parameters, and has outperformed the well-known binarycoded genetic algorithm [54] and particle swarm optimization

algorithms [55] with respect to standard benchmark functions[45]. Further, DE is simple and involves only a few lines code,which motivated us to employ it to solve the above optimizationproblem.

An outline to basic DE [34] is given in the Appendix. Analgorithm to compute the secondary membership function of aT2 fuzzy set from its primary counterpart using DE is givenbelow.

1) Obtain the averaged primary membership function μA(x)from the primary membership functions μi

A(x) obtained

from n sources, i.e., i = 1, . . . ., n. Evaluate c, and alsoci for a selected primary membership distribution μi

A(x)

using (22) and (23), respectively.2) Find the optima on μj

A(x) for a given j. Let the set of

x corresponding to the optima be S. Set the secondarymembership μ(x, μj

A(x)) to 0.99 (close to one) for all

x ∈ S.3) For each x ∈ X , where x /∈ S, identify the optima closest

around x from S. Let they be located at x = xp and x =xq , where xp < x < xq . Determine α and β for each x,given by (20).

4) For each x, where μ(x, μj

A(x)) lies in [α, β], minimize Jj

by DE.5) Obtain μ(x, μj

A(x)) for all x after the DE converges.

6) Repeat step 2 onwards for all j.For a Gaussian primary membership function, the minimum

occurs at infinity, but the minimum value is practically zerowhen x is m± 4σ, where m and σ are mean and standarddeviation of x. In Step 2, the minimum is taken as m± 4σ,and we obtain x by dividing the range [m− 4σ,m+ 4σ] intoequal intervals of same length (here 20 intervals).

An illustrative plot of secondary membership function for agiven primary is given in Fig. 6.

V. FILTERING UNWANTED DATA POINTS IN FEATURE

SPACE USING INTERVAL APPROACH

The IT2FS-based scheme for emotion recognition given inSection III is computationally efficient with good classificationaccuracy. However, its performance depends greatly on themeasurements obtained from facial expressions of the experi-mental subjects. In order to reduce the effect of outliers, we herepresent a scheme of data pre-processing/filtering and selectionof membership functions following the well-known IA [68].

The important steps of IA used in the present context arere-structured for the present application as outlined below. Let[a(i), b(i)] be the end-point interval of measurements of a givenfacial feature for the ith subject obtained from l instances of herfacial expressions for a specific emotion.

Step 1) (Outlier processing): This step divides the two setsof lower and upper data end-points: a(i) and b(i),respectively, for i = 1 to n subjects in quartiles,and tests the acceptability of each data end-point bysatisfying the following criteria:

a(i) ∈ [Qa(0.25)−1.5IQRa, Qa(0.75)+1.5IQRa]b(i) ∈ [Qb(0.25)−1.5IQRb, Qb(0.75)+1.5IQRb]L(i) ∈ [QL(0.25)−1.5IQRL, QL(0.75)+1.5IQRL]

⎫⎬⎭ (25)


Fig. 6. (a) The primary membership function for a given feature and (b) itscorresponding secondary membership function obtained by minimizing Ji.

where Qj(x) denotes the quartile ranges containingthe first x% of the data points in the i-th data set.Here, j ∈ {a, b, L} and a, b denote lower, upper endpoints of intervals, and L is the length of an interval.IQR denotes intra-quartile range and is defined byQ(0.75) minus Q(0.25). The suffixes a, b and Lin IQR denote the IQR for left, right end pointsand interval length, respectively. L(i) is defined asthe length of data interval = b(i) − a(i), for i = 1 ton. The reduced set of data end-points after outlierprocessing is n/.

Step 2) (Tolerance limit processing): This step deals withtolerance limit processing by presuming the datadistributions to be Gaussian, and testing whetherlower/upper data end-points: a(i), b(i) and intervallength L(i) lie within mean plus/minus k(= 2.752)times the standard deviation of the data points. Thenumber 2.752 appears in the scenario for statisticalvalidation with 20 data end-point intervals for 20subjects [68].

If a data interval [a(i), b(i)] and its length L(i)

satisfy (26), the interval is accepted, otherwiserejected:

a(i) ∈ [ml − ksl,ml + ksl]b(i) ∈ [mr − ksr,mr + ksr]L(i) ∈ [mL − ksL,mL + ksL]

⎫⎬⎭ (26)

where, mj and sj denotes sample mean and stan-dard deviation for j ∈ {l, r, L}, for the n/ set ofdata points/intervals. After tolerance processing, thereduced set of data end-points is n//.

Step 3) (Reasonable-interval test): This step checks whetherintervals are reasonable, i.e., they are over-lapped. This has been performed by computingξ∗, given in (27) and then by testing whetherlower bounds of each interval a(i) < ξ∗ and upperbound b(i) > ξ∗, where ξ∗is one of the possiblevalues of

ξ∗

=

(mrσ

2l −mlσ

2r

)±σlσr

[(ml−mr)

2+2(σ2l −σ2

r

)ln(

σl

σr

)] 12

σ2l−σ2

r

(27)

where ml and σl are sample mean and variance ofthe n// left endpoints and mr and σr are samplemean and variance of the n// right endpoints. Ifml <= ξ∗ <= mr is satisfied, then the data inter-vals are retained and dropped otherwise. The re-maining number of data points after the drop ofsome intervals is called n///.

Step 4) (FOU selection): This step is used for the selectionof the right FOU among triangle, left shoulder, andright shoulder. For each FOU, the criteria can befound in [68]. We here reproduce the results fortriangular FOU only, as our results to be given inSection VI yields triangular FOU. For triangularFOU, the conditions are

mr ≤ 5.831ml − 1.328 sc√n///

mr ≤ 0.171ml + 8.29− 1.328 sd√n///

mr ≥ ml

⎫⎬⎭ (28)

where sc = standard deviation of [b(i) − 5.831a(i)]for i = 1 to n///, sd = standard deviation of [b(i) −0.17a(i) − 8.29] for i = 1 to n///.

Step 5) (FOU parameter evaluation): This step deals withparameter evaluation of the triangular membershipfunctions for the existing data intervals [a(i), b(i)].For each interval [a(i), b(i)], we obtain the param-eters a

(i)MF and b

(i)MF representing the end-points

of the x-coordinates of the base for a symmet-ric triangular membership function as reproducedbelow [68]:

a(i)MF = 1

2

[(a(i) + b(i)

)−√2(b(i) − a(i)

)]b(i)MF = 1

2

[(a(i) + b(i)

)+√2(b(i) − a(i)

)]}. (29)

We use these membership functions in place ofGaussian membership functions in our IT2FS ap-proach and call this approach as IT-IT2FS.


Fig. 7. Facial features.

Fig. 8. (a) Localized eye search region, and (b) detection of eye features.

VI. EXPERIMENTS DETAILS

In this section, we present the experimental detailsof emotion recognition using the principles introduced inSections III–V. Here, we consider the following k(= 5) emo-tion classes: anger, fear, disgust, happiness, and relaxation. Theexperiment is conducted with two sets of subjects: 1) the firstset of n(= 20) subjects is considered for designing the fuzzyface space and 2) the other set of 40 facial expressions takenfrom six unknown subjects is considered to validate the result ofthe proposed emotion classification scheme. Five facial features(i.e., m = 5) have been used here to design the T2 fuzzy facespace.

We now briefly overview the main steps of feature extrac-tion followed by fuzzy face-space construction and emotionrecognition of an unknown subject using the pre-constructedface space.

A. Feature Extraction

Feature extraction is a fundamental step in emotion recog-nition. This paper considers extraction of features from emo-tionally rich facial expressions synthesized by the subjects byacting. Existing research results [14], [28] reveal that the mostimportant facial regions responsible for the manifestation ofemotion are the eyes and the lips. This motivated us to select thefollowing features: Left eye opening (EOL), right eye opening(EOR), Distance between the Lower Eyelid to the Eyebrowfor the Left Eye (LEEL), distance between the lower eyelid toeyebrow for the right eye (LEER), and the maximum mouthopening (MO) including the lower and the upper lips. Fig. 7explains the above facial features on a selected facial image.

For extraction of any of the features mentioned above, thefirst step that needs to be carried out is to separate out the skinand the non-skin regions of the image.

Estimation of Eye Features (EOL, LEEL, EOR, and LEER):To compute the eye features, we first localize the eye region asshown in Fig. 8(a). The image in Fig. 8(a) is now transformedto gray scale, and average intensity over each row of pixels isevaluated. Now, we identify the row with the maximum dip in

Fig. 9. (a) Mouth search area, (b) lip cluster, and (c) graph of average intensityover each row against the row position.

average intensity, while scanning the image from top. This rowindicates the first dark region from top, i.e., the eyebrow region(Fig. 8(b)). Similarly, we detect the lower eyelid by identifyingthe row with the maximum dip in intensity in the gray scaleimage, while scanning the face up from the bottommost row.The location of the top eyelid region is identified by scanningthe face up from the marked lower eyelid until the maximumdip occurs in the gray scale image.

Estimation of MO: In order to estimate the MO, we firstlocalize the mouth region as shown in Fig. 9(a). Then, aconversion from R-G-B to perceptually uniform L∗ − a∗ − b∗

color space is undertaken in order to represent the perceptualdifference in color by Euclidean distance [69]. The k-meansclustering algorithm is applied next on this image to segmentthe image into three clusters, namely skin, lip, and teeth regions.Each cluster is now transformed to gray scale, and the onewith the highest average gradient of the boundary points (inintensity) is declared as the lip region. Now, to obtain the MO,we plot the average intensity over each row of Fig. 9(b) againstthe row number. The width of the zero-crossing zone in the plot(Fig. 9(c)) provides a measure of MO.

Experiments are undertaken both on colored image databasesuch as the Indian Women (Jadavpur University) database, andgray scale images including Japanese Female Facial Expression(JAFFE) and Cohn-Kanade databases. The principles of featureextraction introduced above are equally applicable in bothcolor and gray scale images. However, for color images, weneed a conversion to gray scale to determine the features ofeye and mouth of the subject. In addition, for the gray scalefacial images, segmentation of lip-, skin-, and teeth-regions isperformed with intensity data only, unlike the case in colorimages, where we use the 3-D data points (L∗, a∗, b∗) as theinput to the k-means algorithm for segmentation.

Selective images from three facial expression databases aregiven in Fig. 10. Training and test image data partition for threeexperimental databases is given in Table I. The training data inTable I include l instances for n subjects for k distinct emotions.

The following explanation in this section is given with re-spect to Indian Woman Database (Jadavpur University).

B. Creating the T2 Fuzzy Face Space

The interval T2 fuzzy face space contains only the primarymembership distributions for each facial feature. Since wehave five facial features, and the experiment includes fivedistinct emotions of 20 subjects, we obtain 20× 5× 5 = 500primary membership curves. To compute primary member-ships, ten instances of a given emotion are used. These 500membership curves are grouped into 25 heads, each containing


Fig. 10. Experiment done on different databases: (a) JAFFE, (b) Indianwomen database (Jadavpur University), (c) Cohn-Kanade database.

TABLE ITRAINING AND TEST DATA FOR THREE DATABASES

20 membership curves of 20 subjects for a specific feature for agiven emotion. Fig. 11 gives an illustration of one such group of20 membership functions for the feature EOL for the emotion:Anger.

For each primary membership function, we have a corre-sponding secondary membership function. Thus, we obtain500 secondary membership functions. Two illustrative T2 sec-ondary memberships for subjects 1 and 2 for the feature EOL

for the emotion anger are given in Fig. 12. The axes in the figurerepresent feature (EOL), primary and secondary membershipvalues as indicated.

Fig. 11. Membership distributions for emotion anger and feature EOL.

Fig. 12. (a) Secondary membership curve of subject 1. (b) Secondary mem-bership curve of subject 2 for emotion anger.

C. Emotion Recognition of an Unknown Facial Expression

The emotion recognition problem addressed here attemptsto determine the emotion of an unknown person from herfacial expression. To keep the measurements in an emotionalexpression normalized and free from distance variation fromthe camera focal plane, we construct a bounding box, coveringonly the face region, and the reciprocal of the diagonal of thebounding box is used as a scale factor for normalization of themeasurements. The normalized features obtained from Fig. 13are listed in Table II. We now briefly explain the experimentalresults obtained by two alternative reasoning methodologiesincorporating IT2FS and GT2FS.


Fig. 13. Facial image of an unknown subject.

TABLE IIEXTRACTED FEATURES OF FIG. 13

TABLE IIICALCULATED TYPE-2 PRIMARY MEMBERSHIP VALUES

FOR THE FEATURE: EOL UNDER EMOTION: DISGUST

TABLE IVCALCULATED RANGES OF PRIMARY MEMBERSHIPS

AND CENTRE VALUE FOR EACH EMOTION

IT2FS-Based Recognition: The IT2FS-based recognitionscheme considers a fuzzy face space of five sets of 20 primarymembership functions as in Fig. 11, where each set refers to oneparticular feature obtained from 20 sources for an individualemotion. Consequently, for five distinct emotions, we have 25such sets of primary membership functions. Table III providesthe evaluation of T2 primary membership values for the feature,EOL, consulting 20 primary functions obtained from 20 sub-jects, representing the facial expression for disgust. The rangeof these memberships is given in the last row of Table III.For each feature, we obtain five tables like Table III, each onefor a given emotion. Thus, for five features, we would havealtogether 25 such tables.

Table IV provides the results of individual range in primarymembership for each feature experimented under differentemotional conditions. For example, the entry (0–0.18) corre-sponding to the row anger and column EOL gives an idea aboutthe extent of the EOL for the unknown subject matches withknown subjects from the emotion class anger. The results ofcomputing fuzzy meet operation over the range of individual

TABLE VRESULTS OF EXECUTION OF IA ON FEATURE

EOL DATA SET FOR EMOTION: ANGER

Fig. 14. Graphical selection of FOU by testing that the point (ml,mr) =(0.0755, 0.1257) plotted in the figure lies under the triangular zone obtainedby satisfying inequalities in (28).

features taken from facial expressions of the subjects under thesame emotional condition are given in Table IV. The average ofthe ranges along with its center value is also given in Table IV.It is observed that the center has the largest value (= 0.3435)for the emotion: happiness.

IT2FS-Based Recognition With Pre-Processing of FeaturesUsing the Interval Approach (Hereafter IA-IT2FS): The IA in-troduced in Section V has two fundamental merits. It eliminatesnoisy data points obtained from facial data of the subjects.It also helps in identifying the primary membership functionsfor each feature of a facial expression representing a specificemotion by a statistically meaningful approach. The results ofexecution of adapted IA algorithm given in the last section forthe feature EOL for the emotion anger are given in Table V forconvenience. After similar tables for all features of all possibleemotions are determined, we use the statistically significantFOU for each feature of each emotion. In Fig. 14, we provide anillustrative experimental FOU for the feature EOL for emotion


Fig. 15. Constructed symmetric triangular membership functions using (29).

TABLE VICALCULATED TYPE-2 MEMBERSHIP VALUES FOR

THE FEATURE: EOL UNDER EMOTION: DISGUST

anger by performing step 4 of Section V. The parameters of theFOU, here triangles, are evaluated by step 5 of Section V. Now,for an unknown facial expression, we follow the steps of IT2FS-based approach to recognize the emotion exhibited in the facialexpression. Our experiments reveal that the pre-processingsteps by IA help in improving the recognition accuracy of theIT2FS scheme by 2.5% (Fig. 15).

GT2FS-Based Recognition: We now briefly illustrate theGT2FS-based reasoning for emotion classification. Here, thesecondary membership function corresponding to the individ-ual primary membership function of five features obtainedfrom facial expressions carrying five distinct emotions for 20different subjects are determined using membership functionslike Fig. 12.

Table VI provides the summary of the primary and secondarymemberships obtained for EOL for the emotion: disgust. Therange computation for the feature EOL is also shown in thelast column of Table VI. The same computations are repeatedfor all emotions, and the range evaluated in the last column ofTable VII indicates that the center of this range here too has thelargest value (= 0.301) for the emotion: happiness.

TABLE VIICALCULATED RANGES OF PRIMARY MEMBERSHIP

CENTRE VALUE FOR EACH EMOTION

TABLE VIIIPERCENTAGE ACCURACY OF OUR PROPOSED

METHODS OVER THREE DATABASES

VII. PERFORMANCE ANALYSIS

Performance analysis for emotion recognition itself is anopen-ended research problem, as there is a dearth of literatureon this topic. This paper, compares the relative performanceof the proposed GT2FS algorithms with five traditional emo-tion recognition algorithms/techniques and the IA-IT2FS andIT2FS-based schemes introduced here, considering a commonframework in terms of their features and databases. The al-gorithms used for comparison include linear SVM classifier[28], (T1) fuzzy relational approach [14], PCA [33], multi-layer perceptron (MLP) [1], [29], radial basis function network(RBFN) [1], [29], IT2FS, and IA-IT2FS [68].

Table VIII shows the classification accuracy of our pro-posed three algorithms using three facial image databases, i.e.,JAFFE, Indian Women Face Database (Jadavpur University),and Cohn-Kanade database. Experimental classification accu-racy obtained for different other algorithms mentioned aboveusing the three databases is given in Table X.

Two statistical tests called McNemar’s test [58] andFriedman test [59], and one new test, called root mean squareerror test are undertaken to analyze the relative performance ofthe proposed algorithms over existing ones.

A. McNemar’s Test

Let fA and fB be two classifiers obtained by algorithms Aand B, when both the algorithms have a common training set R.

Let n01 be the number of examples misclassified by fA butnot by fB , and n10 be the number of examples misclassifiedby fB but not by fA. Then, under the null hypothesis that


TABLE IXSTATISTICAL COMPARISON OF PERFORMANCE USING

MC NEMAR’S TEST WITH THREE DATABASES

both algorithms have the same error rate, the statistic Z in (30)follows a χ2 with degree of freedom equals to 1 [59]:

Z =(|n01 − n10| − 1)2

n01 + n10. (30)

Let A be the proposed GT2FS algorithm and B is one of theother seven algorithms. We thus evaluate Z = Z1 through Z7,where Zj denotes the comparator statistic of misclassificationbetween the GT2FS (Algorithm: A) and the jth of the sevenalgorithms (Algorithm: B), where the suffix j refers to thealgorithm in row number j of Table IX.

Table IX is evaluated to obtain Z1 through Z7 and thehypothesis has been rejected, if Zj > χ2

1, 0.95 = 3.84, whereχ21, 0.95 = 3.84 is the value of the chi square distribution for 1

degree of freedom at probability of 0.05 [81].The last inequality indicates that if the null hypothesis is true,

then the probability of χ2 to be more than 3.84 is less than 0.05.If the hypothesis is not rejected, we consider its acceptance.The decision about acceptance or rejection is also included inTable IX.

It is evident from Table IX that McNemar’s test cannot dis-tinguish the performance of the five classification algorithms:IT2FS, IA-IT2FS, SVM, fuzzy relational approach, and PCAthat support the hypothesis. Hence, next we use the Friedmantest for ranking the algorithms.

B. Friedman Test

The Friedman test [58] ranks the algorithms for each datasets separately. The best performing algorithm gets rank 1. Incase of ties, average ranks are used.

TABLE XAVERAGE RANKING OF CLASSIFICATION ALGORITHMS BY FRIEDMAN

TEST, WHERE, CA = Classifier Algorithm, A = GT2FS, B1 = SVM,B2 = IT2FS, B3 = IA-IT2FS, B4 = Fuzzy Relational Approach,

B5 = PCA, B6 = MLP, B7 = RBFN

Let rji be the rank of jth algorithm on the ith data set. Theaverage rank of algorithm j then is evaluated by

Rj =1

N

∑∀i

rji . (31)

The null hypothesis here states that all the algorithms areequivalent, so their individual ranks Rj should be equal. Underthe null hypothesis, for large enough N and k, the Friedmanstatistic χ2

F in (32) is distributed as a χ2 with k-1 degreesof freedom. Here, k = 8 and N = 3. A larger N of courseis desirable; however, emotion databases being fewer, findinglarge N is not feasible. Here, we consider percentage accu-racy of classification as the basis of rank. Table X providesthe percentage accuracy of classification with respect to threedatabases, JAFFE, Indian Woman (Jadavpur University), andCohn-Kanade and the respective ranks of the algorithm

χ2F =

12N

k(k + 1)

⎡⎣∑

j

R2j −

k(k + 1)2

4

⎤⎦ . (32)

Now, using N = 3, k = 8, and the ranks in Table X, weobtain χ2

F = 17.67 > χ27,0.95 = 14.067 [81], where χ2

7, 0.95 =14.067 is the value of the chi square distribution for 7◦ offreedom at probability of 0.05 [81]

χ2F =

12N

k(k + 1)

⎡⎣∑

j

R2j −

k(k + 1)2

4

⎤⎦

=17.67 > χ27,0.95(14.067).

Thus, the hypothesis that the algorithms are equivalent isrejected. Therefore, the performances of the algorithms aredetermined by their ranks only. The order of ranking of thealgorithm is apparent from their average ranks. The smallerthe average rank, the better is the algorithm. Let “>” be acomparator of relative ranks where x > y means the algorithmx is better in rank than algorithm y. Table X indicates that the


relative order of ranking of the algorithm by Friedman test as,GT2FS> IA−IT2FS > SVM > IT2FS > PCA >Fuzzy Rela-tional Approach > MLP > RBFN. It is clear from Table Xthat the average rank of GT2FS is 1 and average rank of IT2FSand IA-IT2FS are 4 and 2, respectively, claiming GT2FSoutperforms all the algorithms by Friedman test.

VIII. CONCLUSION

The paper presents three automatic emotion recognitionsystems based on IT2FS, IA-IT2FS, and GT2FS. In order toclassify an unknown facial expression, these systems make useof the background knowledge about a large face database withknown emotion classes. The GT2FS-based recognition schemerequires T2 secondary membership functions, which are ob-tained using an innovative evolutionary approach that is alsoproposed in this paper. All the schemes first construct a fuzzyface space, and then infer the emotion class of the unknownfacial expression by determining the maximum support of theindividual emotion classes using the pre-constructed fuzzy facespace. The class with the highest support is assigned as theemotion of the unknown facial expression.

The IT2FS-based recognition scheme takes care of the inter-subject level uncertainty in computing the maximum supportof individual emotion class. The GT2FS-based recognitionscheme, however, takes care of both the inter- and intra-subjectlevel uncertainty, and thus offers higher classification accuracyfor the same set of features. Using three data sets, the classifi-cation accuracy obtained by employing GT2FS is 98.333%, byIT2FS is 91.667%, and by IA-IT2FS is 94.167%.

The more the number of subjects used for constructing thefuzzy face space, the better would be the fuzzy face space,and thus better would be the classification accuracy. Since thefuzzy face space is created offline, the online computationalload to recognize emotion is insignificantly small in IT2FS.The computational load in GT2FS, however, is large as itincludes an optimization procedure to determine the secondarymembership for each emotion and for each subject. However,this additional complexity in GT2FS, offers approximately 7%improvement in classification accuracy in comparison to thatby IT2FS. The IA-IT2FS has around 2.5% gain in classificationaccuracy with no more additional computational complexitythan IT2FS. It may be noted that the necessary computations inIA-IT2FS for data filtering and membership function selectionis performed offline. The statistical tests employed clearlyindicate that GT2FS outperforms the seven selected algorithms.

The problems that may be taken up as future research arebriefly outlined below. First, new alternative strategies are to bedesigned to determine secondary memberships without usingoptimization techniques. Second, a more formal and systematicapproach to fuse secondary and primary memberships to reduceuncertainty in the fuzzy face space is to be developed. Last,we would try to explore the power of fuzzy logic to determineemotion classes in absence of sufficient (or even no) mea-surements. Facial features, for example MO, may be directlyencoded into fuzzy features with fuzzy sets, such as “a little,”“more,” and “not so large,” and then an IT2FS-based modelmay be adopted to recognize emotion of unknown subjects.

Classification accuracy under this circumstance could be poor,but a more human-like interpretation of emotion can be givenin the absence of precise measurements.

APPENDIX

THE CLASSICAL DIFFERENTIAL

EVOLUTION ALGORITHM [34]

An iteration of the classical DE algorithm consists of the fourbasic steps—initialization of a population of vectors, mutation,crossover or recombination, and finally selection. The mainsteps of classical DE are given below:

I. Set the generation number t = 0 and randomly initializea population of NP individuals �Pt = { �X1(t), �X2(t), . . . ,�XNP (t)} with �X1(t) = {xi,1(t), xi,2(t), . . . ., xi,D(t)}and each individual uniformly distributed in the range[ �Xmin, �Xmax], where Xmin = {xmin,1, xmin,2, . . . ,

xmin,D} and �Xmax = {xmax,1, xmax,2, . . . ., xmax,D}with i = [1, 2, . . . ., NP ].

II. while stopping criterion is not reached, dofor i = 1 to NP

a. Mutation:Generate a donor vector �V (t) =

{vi,1(t), vi,2(t), . . . . . . , vi,D(t)} corresponding to the ithtarget vector �X1(t) by the following scheme �V1(t) =�Xr1(t) + F ∗ ( �Xr2(t)− �Xr3(t)) where r1, r2 and r3 aredistinct random integers in the range [1, NP]

b. Crossover:Generate trial vector �Ui(t) =

{ui,1(t), ui,2(t), . . . ., ui,D(t)} for the ith target vector�X1(t) by binomial crossover as

�ui,j(t) =�vi,j(t) if rand (0, 1) < Cr

= �xi,j(t) otherwise.

c. Selection:Evaluate the trial vector �Ui(t)

if f(�Ui(t)) ≤ f( �Xi(t)),then vecXi(t+ 1) = vecUi(t)

f( �Xi(t+ 1)) =f(vecUi(t))

end ifend ford. Increase the counter value t = t + 1.

end while

The parameters used in the algorithm namely scaling factor“F ” and crossover rate “Cr” should be initialized before callingthe “while” loop. The terminate condition can be defined inmany ways, a few of which include: 1) fixing the number ofiterations N , 2) when best fitness of population does not changeappreciably over successive iterations, and 3) either of 1) and2), whichever occurs earlier.


ACKNOWLEDGMENT

The authors gratefully acknowledge the funding they ob-tained from UGC UPE-II program in Cognitive science,Perception Engineering Project funded by DIT at Jadavpur Uni-versity, and the Grant received from Liverpool Hope University,Liverpool, U.K. for publication of the paper.

REFERENCES

[1] D.-J. Kim, S.-W. Lee, and Z. Bien, “Facial emotional expression recogni-tion with soft computing techniques,” in Proc. 14th IEEE Int. Conf. FuzzySyst., 2005, pp. 737–742.

[2] R. Cowie, E. Douglas-Cowie, J. G. Taylor, S. Ioannou, M. Wallace, andS. Kollias, “An intelligent system for facial emotion recognition,” in Proc.IEEE ICME, Amsterdam, The Netherlands, Jul. 2005, pp. 904–907.

[3] A. Bhavsar and H. M. Patel, “Facial expression recognition using neuralclassifier and fuzzy mapping,” in Proc. IEEE INDICON, Chennai, India,2005, pp. 383–387.

[4] Y. Guo and H. Gao, “Emotion recognition system in images basedon fuzzy neural network and HMM,” in Proc. 5th IEEE ICCI, 2006,pp. 73–78.

[5] M. H. Bindu, P. Gupta, and U. S. Tiwary, “Cognitive model based emotionrecognition from facial expressions for live human computer interaction,”in Proc. IEEE Symp.CIISP, 2007, pp. 351–356.

[6] M. Rizon, M. Karthigayan, S. Yaacob, and R. Nagarajan, “Japaneseface emotions classification using lip features,” in Proc. GMAI, 2007,pp. 140–144.

[7] N. Esau, E. Wetzel, L. Kleinjohann, and B. Kleinjohann, “Real-time facialexpression recognition using a fuzzy emotion model,” in Proc. IEEE Int.Fuzzy Syst. Conf., 2007, pp. 1–6.

[8] H. Zhao, Z. Wang, and J. Men, “Facial complex expression recognitionbased on fuzzy kernel clustering and support vector machines,” in Proc.3rd ICNC, 2007, pp. 562–566.

[9] Y. Lee, C. W. Han, and J. Shim, “Fuzzy neural networks and fuzzyintegral approach to curvature-based component range facial recognition,”in Proc. Int. Conf. Converg. Inf. Technol., 2007, pp. 1334–1339.

[10] A. Khanam, M. Z. Shafiq, and M. U. Akram, “Fuzzy based facialexpression recognition,” in Proc. Congr. Image Signal Process., 2008,pp. 598–602.

[11] G. U. Kharat and S. V. Dudul, “Neural network classifier for human emo-tion recognition from facial expressions using discrete cosine transform,”in Proc. 1st Int. Conf. Emerging Trends Eng. Technol., 2008, pp. 653–658.

[12] J. M. Sun, X. S. Pei, and S. S. Zhou, “Facial emotion recognition inmodern distant system using SVM,” in Proc. 7th Int. Conf. Mach. Learn.Cybern., Kunming, China, Jul. 2008, pp. 3545–3548.

[13] H. Zhao and Z. Wang, “Facial action units recognition based on fuzzykernel clustering,” in Proc. 5th Int. Conf. Fuzzy Syst. Knowl. Discov.,2008, pp. 168–172.

[14] A. Chakraborty, A. Konar, U. K. Chakraborty, and A. Chatterjee, “Emo-tion recognition from facial expressions and its control using fuzzy logic,”IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 39, no. 4, pp. 726–743, Jul. 2009.

[15] H. Kobayashi and F. Hara, “Measurement of the strength of six basic facialexpressions by neural network,” Trans. Jpn. Soc. Mech. Eng. (C), vol. 59,no. 567, pp. 177–183, 1993.

[16] H. Kobayashi and F. Hara, “Recognition of mixed facial expressionsby neural network,” Trans. Jpn. Soc. Mech. Eng. (C), vol. 59, no. 567,pp. 184–189, 1993.

[17] H. Kobayashi and F. Hara, “The recognition of basic facial expressions byneural network,” Trans. Soc. Instrum. Contr. Eng, vol. 29, no. 1, pp. 112–118, 1993.

[18] Y. Sun, Z. Li, and C. Tang, “An evolving neural network for authenticemotion classification,” in Proc. 5th Int. Conf. Natural Comput., 2009,pp. 109–113.

[19] C. Chang, Y. Huang, and C. Yang, “Personalized facial expression recog-nition in color image,” in Proc. 4th Int. Conf. Innov. Comput., Inf. Contr.,2009, pp. 1164–1167.

[20] M. Fu, Y. Kuo, and K. Lee, “Fusing remote control usages and facial ex-pression for emotion recognition,” in Proc. 4th Int. Conf. Innov. Comput.,Inf. Contr., 2009, pp. 132–135.

[21] C. Huang and Y. Huang, “Facial expression recognition using model-based feature extraction and action parameters classification,” J. Vis.Commun. Image Represent., vol. 8, no. 3, pp. 278–290, Sep. 1997.

[22] K. Huang, S. Huang, and Y. Kuo, “Emotion recognition based on anovel triangular facial feature extraction method,” in Proc. IJCNN, 2010,pp. 1–6.

[23] H. Tsai, Y. Lai, and Y. Zhang, “Using SVM to design facial expressionrecognition for shape and texture features,” in Proc. 9th Int. Conf. Mach.Learn. Cybern., Qingdao, China, Jul. 11–14, 2010, pp. 2697–2704.

[24] P. Ekman and W. V. Friesen, Unmasking the Face: A Guide to Recogniz-ing Emotions From Facial Clues. Englewood Cliffs, NJ: Prentice-Hall,1975.

[25] O. A. Uwechue and S. A. Pandya, Human Face Recognition Using Third-Order Synthetic Neural Networks. Boston, MA: Kluwer, 1997.

[26] B. Biswas, A. K. Mukherjee, and A. Konar, “Matching of digital imagesusing fuzzy logic,” AMSE Publ., vol. 35, no. 2, pp. 7–11, 1995.

[27] M. Paleari, R. Chellali, and B. Huet, “Features for multimodal emotionrecognition: an extensive study,” in Proc. CIS, 2010, pp. 90–95.

[28] S. Das, A. Halder, P. Bhowmik, A. Chakraborty, A. Konar, andA. K. Nagar, “Voice and facial expression based classification of emotionusing linear support vector,” in Proc. 2nd Int. Conf. Develop. Syst. Eng.,2009, pp. 377–384.

[29] M. Gargesha and P. Kuchi, “Facial expression recognition using a neuralnetwork,” Artif. Neural Comput. Syst., pp. 1–6, 2002.

[30] J. M. Mendel, “Fuzzy sets for words: A new beginning,” in Proc. IEEEInt. Conf. Fuzzy Syst., 2003, pp. 37–42.

[31] Y. Zhang, Q. J. I., “Active and dynamic information fusion for facialexpression understanding from image sequences,” IEEE Trans. PatternAnal. Mach. Intell., vol. 27, no. 5, pp. 699–714, May 2005.

[32] C. Liu and H. Wechsler, “Probabilistic reasoning models for face recogni-tion,” in Proc. Comput. Vis. Pattern Recog. Conf., 1998, pp. 827–832.

[33] G. U. Kharat and S. V. Dudul, “Emotion recognition from facial ex-pression using neural networks,” in Proc. Human Comput. Sys. Interact.,2009, pp. 207–219.

[34] K. V. Price, R. M. Storn, and J. A. Lampinen, Eds., Differential Evolution:A Practical Approach. New York: Springer-Verlag, 2005.

[35] A. Konar and A. Chakraborty, Emotional Intelligence: A CyberneticApproach. Heidelberg, Germany: Springer-Verlag, 2009.

[36] H. J. Zimmermann, Fuzzy Set Theory and its Applications. Berlin,Germany: Springer-Verlag, 2001.

[37] J. M. Mendel and H. Wu, “Type-2 Fuzzistics for symmetric interval type-2 fuzzy sets: Part 1, forward problems,” IEEE Trans. Fuzzy Syst., vol. 14,no. 6, pp. 718–792, Dec. 2006.

[38] J. M. Mendel and D. Wu, Perceptual Computing. Piscataway, NJ: IEEEPress, 2010.

[39] J. M. Mendel and R. I. John, “Type-2 fuzzy sets made simple,” IEEETrans. Fuzzy Syst., vol. 10, no. 2, pp. 117–127, Apr. 2002.

[40] A. Konar and A. Chakraborty, Eds., Advances in Emotion Recognition.New York: Wiley, 2011.

[41] J. M. Mendel, R. I. John, and F. Liu, “Interval type-2 fuzzy logic systemsmade simple,” IEEE Trans. Fuzzy Syst., vol. 14, no. 6, pp. 808–821,Dec. 2006.

[42] J. M. Mendel, “On the importance of interval sets in type-2 fuzzy logicsystems,” in Proc. Joint 9th IFSA World Congr. 20th NAFIPS Int. Conf.,Vancouver, BC, Canada, Jul. 25–28, 2001, pp. 1647–1652.

[43] F. Kawakami, S. Morishima, H. Yamada, and H. Harashima, “Construc-tion of 3-D emotion space using neural network,” in Proc. 3rd Int. Conf.Fuzzy Logic, Neural Nets Soft Comput., Iizuka, Japan, 1994, pp. 309–310.

[44] H. Wu, Y. Wu, and J. Luo, “An interval type-2 fuzzy rough set model forattribute reduction,” IEEE Trans. Fuzzy Syst., vol. 17, no. 2, pp. 301–315,Apr. 2009.

[45] P. N. Suganthan, N. Hansen, J. J. Liang, K. Deb, Y.-P. Chen, A. Auger,and S. Tiwari, “Problem definitions and evaluation criteria for the CEC2005 special session on real-parameter optimization,” Nanyang Technol.Univ., Singapore, 2005, Tech. Rep..

[46] J. Romero Agüero and A. Vargas, “Calculating functions of intervaltype-2 fuzzy numbers for fault current analysis,” IEEE Trans. Fuzzy Syst.,vol. 15, no. 1, pp. 31–40, Feb. 2007.

[47] O. Cordón, M. J. del Jesus, and F. Herrera, “A proposal on reasoning meth-ods in fuzzy rule-based classification systems,” Int. J. Approx. Reasoning,vol. 20, no. 1, pp. 21–45, Jan. 1999.

[48] C.-H. Lee, J.-L. Hong, Y.-C. Lin, and W.-Y. Lai, “Type-2 fuzzy neuralnetwork systems and learning,” Int. J. Comput. Cogn., vol. 1, no. 4,pp. 79–90, Dec. 2003.

[49] H. Yamada, “Visual information for categorizing facial expression ofemotion,” Appl. Cogn. Psychol., vol. 7, no. 3, pp. 257–270, Jun. 1993.

[50] J. M. Fernandez-Dols, H. Wallbotl, and F. Sanchez, “Emotion cate-gory accessibility and the decoding of emotion from facial expres-sion and context,” J. Nonverbal Behav., vol. 15, no. 2, pp. 107–123,Jun. 1991.


[51] C. Busso and S. Narayanan, “Interaction between speech and facialgestures in emotional utterances: A single subject study,” IEEE Trans.Audio, Speech Language Process., vol. 15, no. 8, pp. 2331–2347,Nov. 2007.

[52] I. Cohen, N. Sebe, A. Garg, L. S. Chen, and T. S. Huang, “Facial ex-pression recognition from video sequences: Temporal and static mod-eling,” Comput. Vis. Image Understand., vol. 91, no. 1/2, pp. 160–187,Jul. 2003.

[53] Y. Gao, M. K. H. Leung, S. C. Hui, and M. W. Tananda, “Facial expressionrecognition from line-based caricatures,” IEEE Trans. Syst., Man, Cybern.A, Syst., Humans, vol. 33, no. 3, pp. 407–412, May 2003.

[54] Z. Michalewicz, Genetic Algorithms + Data Structure = EvolutionPrograms. Berlin, Germany: Springer-Verlag, 1992.

[55] P. Engelbrecht, Fundamentals of Computational Swarm Intelligence.New York: Wiley, 2005.

[56] R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional in-telligence: Analysis of affective physiological state,” IEEE Trans. PatternAnal. Mach. Intell., vol. 23, no. 10, pp. 1175–1191, Oct. 2001.

[57] M. Pantic and I. Patras, “Dynamics of facial expression: Recognitionof facial actions and their temporal segments from face profile imagesequences,” IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 2,pp. 433–449, Apr. 2006.

[58] J. Demsar, “Statistical comparisons of classifiers over multiple data sets,”J. Mach. Learn. Res., vol. 7, pp. 1–30, Dec. 2006.

[59] T. G. Dietterich, “Approximate statistical tests for comparing super-vised classification learning algorithms,” Neural Comput., vol. 10, no. 7,pp. 1895–1923, Oct. 1998.

[60] A. Halder, P. Rakshit, S. Chakraborty, A. Konar, A. Chakraborty, E. Kim,and A. K. Nagar, “Reducing uncertainty in interval type-2 fuzzy sets forqualitative improvement in emotion recognition from facial expressions,”in Proc. IEEE FUZZ, Brisbane, Australia, 2012, to be published.

[61] D. Datcu and L. J. M. Rothkrantz, “Emotion recognition using bimodaldata fusion,” in Proc. 12th Int. Conf. Comput. Syst. Technol., 2011,pp. 122–128.

[62] M. Zia Uddin, J. J. Lee, and T.-S. Kim, “Independent shape component-based human activity recognition via hidden Markov model,” Appl. Intell.,vol. 33, no. 2, pp. 193–206, Oct. 2010.

[63] Y. Yang, Z. Chen, Z. Liang, and G. Wang, “Attribute reduction for massivedata based on rough set theory and mapReduce,” in Proc. RSKT , 2010,pp. 672–678.

[64] C.-Y. Chang, S.-C. Li, P.-C. Chung, J.-Y. Kuo, and Y.-C. Tu, “Automaticfacial skin defect detection system,” in Proc. BWCCA, 2010, pp. 527–532.

[65] H. Zhang, D. Liu, and Z. Wang, Controlling Chaos: Suppression,Synchronization and Chaotification. New York: Springer-Verlag, 2009.

[66] L. A. Zadeh, “Fuzzy sets,” Inf. Contr., vol. 8, no. 3, pp. 338–353,Jun. 1965.

[67] M. Pantic and L. J. M. Rothkrantz, “Automatic analysis of facial ex-pressions: The state of the art,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 22, no. 12, pp. 1424–1445, Dec. 2000.

[68] F. Liu and J. M. Mendel, “Encoding words into interval type-2 fuzzysets using an interval approach,” IEEE Trans. Fuzzy Syst., vol. 16, no. 6,pp. 1503–1521, Dec. 2008.

[69] L. Guan, S.-Y. Kung, and J. Larsen, Multimedia Image and Video Pro-cessing. Boca Raton, FL: CRC Press, 2000.

[70] A. Metallinou, A. Katsamanis, and S. Narayanan, “A hierarchical frame-work for modeling multimodality and emotional evolution in affectivedialogs,” in Proc. ICASSP, Kyoto, Japan, Mar. 2012, pp. 2401–2404.

[71] A. Metallinou, M. Wollmer, A. Katsamanis, F. Eyben, B. Schuller, andS. Narayanan, “Context-sensitive learning for enhanced audiovisual emo-tion classification,” IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 184–198, 2012.

[72] A. Metallinou, A. Katsamanis, Y. Wang, and S. Narayanan, “Trackingchanges in continuous emotion states using body language and prosodiccues,” in Proc. ICASSP, 2011, pp. 2288–2291.

[73] A. Metallinou, C. Busso, S. Lee, and S. S. Narayanan, “Visual emotionrecognition using compact facial representations and viseme informa-tion,” in Proc. ICASSP, 2010, pp. 2474–2477.

[74] A. Metallinou, S. Lee, and S. S. Narayanan, “Decision level combinationof multiple modalities for recognition and analysis of emotional expres-sion,” in Proc. ICASSP, 2010, pp. 2462–2465.

[75] C.-C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan, “Emotionrecognition using a hierarchical binary decision tree approach,” SpeechCommun., vol. 53, no. 9/10, pp. 1162–1171, Nov. 2011.

[76] E. Mower, M. Mataric, and S. Narayanan, “Human perception of audio-visual synthetic character emotion expression in the presence of ambigu-ous and conflicting information,” IEEE Trans. Multimedia, vol. 11, no. 5,pp. 843–855, Aug. 2009.

[77] M. Grimm, E. Mower, K. Kroschel, and S. Narayanan, “Primitives basedestimation and evaluation of emotions in speech,” Speech Commun.,vol. 49, no. 10/11, pp. 787–800, Oct. 2007.

[78] C. M. Lee and S. Narayanan, “Emotion recognition using a data-drivenfuzzy inference system,” in Proc. EUROSPEECH, Geneva, Switzerland,2003, pp. 157–160.

[79] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim,J. Chang, S. Lee, and S. Narayanan, “Interactive emotional dyadic motioncapture database,” J. Language Resourc. Eval., vol. 42, no. 4, pp. 335–359, Dec. 2008.

[80] A. Metallinou, C.-C. Lee, C. Busso, S. Carnicke, and S. Narayanan, “TheUSC Creative IT database: A multimodal database of theatrical improvi-sation,” in Proc. MMC, Valletta, Malta, May 2010.

[81] J. N. Kapur and H. C. Saxena, Mathematical Statistics. New Delhi,India: S. Chand & Company Ltd., 1980.

Anisha Halder received the B.Tech. degree in elec-tronics and communication engineering from HaldiaInstitute of Technology, Midnapore, India, and theM.E. degree in control engineering from Electronicsand Telecommunication Engineering Department,Jadavpur University, Kolkata, India, in 2007 and2009, respectively. Currently, she is working towardthe Ph.D. degree in facial expression analysis foremotion recognition jointly under the guidance ofProf. Amit Konar and Dr. Aruna Chakraborty atJadavpur University, Kolkata.

Her principal research interests include artificial intelligence, pattern recog-nition, cognitive science, and human–computer interaction. She is an authorof over 25 papers published in top international journals and conferenceproceedings.

Amit Konar received the B.E. degree from BengalEngineering and Science University (B.E. College),Howrah, India, in 1983, and the M.E. Tel E, M.Phil.,and Ph.D. (Engineering) degrees from Jadavpur Uni-versity, Calcutta, India, in 1985, 1988, and 1994,respectively.

In 2006, he was a Visiting Professor with theUniversity of Missouri, St. Louis. Currently, he isa Professor with the Department of Electronics andTelecommunication Engineering, Jadavpur Univer-sity, where he is the Founding Coordinator of the

M.Tech. program on intelligent automation and robotics. He has supervised15 Ph.D. theses. He has over 200 publications in international journal andconference proceedings. He is the author of eight books, including two populartexts Artificial Intelligence and Soft Computing (CRC Press, 2000) and Compu-tational Intelligence: Principles, Techniques, and Applications (Springer, 2005).He serves as the Associate Editor of IEEE TRANSACTIONS ON SYSTEMS,MAN, AND CYBERNETICS, PART-A, and IEEE TRANSACTIONS ON FUZZY

SYSTEMS. His research areas include the study of computational intelligencealgorithms and their applications to the various domains of electrical en-gineering and computer science. Specifically, he worked on fuzzy sets andlogic, neurocomputing, evolutionary algorithms, Dempster–Shafer theory, andKalman filtering, and applied the principles of computational intelligence inimage understanding, very large scale integration design, mobile robotics,pattern recognition, brain-computer interfacing, and computational biology. Hewas the Recipient of All India Council for Technical Education-accredited1997–2000 Career Award for Young Teachers for his significant contributionin teaching and research.


Rajshree Mandal received the B.E. degree in BirlaInstitute of Technology, Mesra, Ranchi, in 2004. Hereceived the M.E. degree in communication engi-neering at Jadavpur University, Calcutta, India, dur-ing 2009–2011.

She served as a member of the Digital SignalProcessing group in Tata Elxsi, Bangalore during2004–2006. She also worked in Interra Systems inCalcutta during 2006–2009. She has published overten papers in international journals and conferences.Her current research interest includes image com-

pression, digital image processing, and multi-modal emotion recognition.

Aruna Chakraborty received the M.A. degree incognitive science and the Ph.D. degree on emotionalintelligence and human–computer interactions fromJadavpur University, Calcutta, India, in 2000 and2005, respectively.

Currently, she is an Associate Professor in theDepartment of Computer Science and Engineering,St. Thomas’ College of Engineering and Technol-ogy, Calcutta. She is also a Visiting Faculty ofJadavpur University, Calcutta, India, where she of-fers graduate-level courses on intelligent automation

and robotics, and cognitive science. She, with her teacher Prof. A. Konar, haswritten a book on Emotional Intelligence: A Cybernetic Approach, which hasappeared from Springer, Heidelberg, in 2009. She is also co-editing a bookwith Prof. A. Konar on Advances in Emotion Recognition, which is shortly toappear from Willey Interscience. She serves as an Editor to the InternationalJournal of Artificial Intelligence and Soft Computing, Inderscience, U.K. Hercurrent research interest includes artificial intelligence, emotion modeling, andtheir applications in next-generation human–machine interactive systems. Sheis a nature lover and loves music and painting.

Pavel Bhowmik received the B.E. degree fromJadavpur University, Kolkata, India, in 2010. Cur-rently, he is working toward the M.S. degreein electrical engineering at University of Florida,Gainesville.

He has published more than 15 research articlesin peer-reviewed journals and international confer-ences. He served as a Reviewer in IEEE CEC 2011,ICCSII 2012, ISIEA 2012, PECON 2012. His re-search interests include various aspects of artificialintelligence (AI) like machine learning, fuzzy logic,

evolutionary algorithm, and emotion recognition. He is also interested incontrol and communication issues of smart grid and the application of variousAI techniques in this aspect.

Nikhil R. Pal (M’91–SM’00–F’05) is a Professor inthe Electronics and Communication Sciences Unit ofthe Indian Statistical Institute. His current researchinterest includes bioinformatics, brain science, fuzzylogic, image and pattern analysis, neural networks,and evolutionary computation.

He was the Editor-in-Chief of the IEEE TRANS-ACTIONS ON FUZZY SYSTEMS for the periodJanuary 2005–December 2010. He has served/beenserving on the editorial/advisory board/steering com-mittee of several journals including the International

Journal of Approximate Reasoning, Applied Soft Computing, Neural Informa-tion Processing—Letters and Reviews, International Journal of Knowledge-Based Intelligent Engineering Systems, International Journal of Neural Sys-tems, Fuzzy Sets and Systems, International Journal of Intelligent Computing inMedical Sciences and Image Processing, Fuzzy Information and Engineering:An International Journal, IEEE TRANSACTIONS ON FUZZY SYSTEMS, andthe IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS—B.

He has given many plenary/keynote speeches in different premier interna-tional conferences in the area of computational intelligence. He has served asthe General Chair, Program Chair, and Co-Program chair of several confer-ences. He is a Distinguished Lecturer of the IEEE Computational IntelligenceSociety (CIS) and a member of the Administrative Committee of the IEEECIS. He is the General Chair of 2013 IEEE International Conference on FuzzySystems.

He is a Fellow of the National Academy of Sciences, India, a Fellow of theIndian National Academy of Engineering, a Fellow of the Indian National Sci-ence Academy, and a Fellow of the International Fuzzy Systems Association.

Atulya K. Nagar holds the Foundation Chair, asa Professor of Computer and Mathematical Sci-ences, at Liverpool Hope University, and is the Headof the Department of Mathematics and ComputerScience. A mathematician by training, ProfessorNagar possesses multi-disciplinary expertise inNatural Computing, Bioinformatics, Operations Re-search, and Systems Engineering. He has an ex-tensive background and experience of working inUniversities in the U.K. and India.

He has been an expert Reviewer for the Biotech-nology and Biological Sciences Research Council Grants peer-review commit-tee for Bioinformatics Panel and serves on Peer-Review College of the Arts andHumanities Research Council as a scientific expert member.

He has coedited volumes on Intelligent Systems and Applied Mathematics;he is the Editor-in-Chief of the International Journal of Artificial Intelligenceand Soft Computing and serves on editorial boards for a number of prestigiousjournals as well as on International Program Committee for several interna-tional conferences. He received a prestigious Commonwealth Fellowship forworking toward his Doctorate in applied non-linear mathematics, which heearned from the University of York in 1996. He received the B.Sc. (Hons.),M.Sc., and MPhil (with Distinction) degrees from the MDS University ofAjmer, India. Prior to joining Liverpool Hope, Prof. Nagar was with theDepartment of Mathematical Sciences, and later at the Department of SystemsEngineering, at Brunel University, London.

general and interval type-2 fuzzy face-space approach to emotion recognition

Documents