Computing with Words Model for Emotion Recognition by Facial Expression Analysis Using

Interval Type-2 Fuzzy Sets

Anisha Halder ETCE Department, Jadavpur

University, Kolkata-32. halder.anisha@gmail.com

Aruna Chakraborty

Department of CSE, St. Thomas’ College of

Engineering and Technology Kolkata

aruna_stcet@rediffmail.com

Amit Konar

ETCE Department, Jadavpur University, Kolkata-32. konaramit@yahoo.co.in

Atulya K. Nagar Department of Math and

Computer Science, Liverpool Hope University,

Liverpool, UK. nagara@hope.ac.uk

Abstract— The paper provides a novel approach to emotion recognition of subjects from the user-specified word description of their facial features. The problem is solved in two phases. In the first phase, an interval type-2 fuzzy membership space for each facial feature in different linguistic grades for different emotions is created. In the second phase, a set of fuzzy emotion-classifier rules is instantiated with fuzzy word description about facial features to infer the winning emotion class. The most attractive part of this research is to autonomously transform user specified word descriptions into membership functions and construction of footprint of uncertainty for each facial feature in different linguistic grades. The proposed technique for emotion classification is very robust as it is sensitive to changes in word description only rather than the absolute measurement of features. Besides it offers a good classification accuracy of 87.8% and thus comparable with existing techniques.

Keywords- Computing with Words, Interval Type-2 Fuzzy Reasoning, IT2FS FOU, UMF, LMF, Affective Computing, Emotion Recognition.

I. INTRODUCTION Emotion recognition is currently gaining popularity for its increasing applications in human-computer interfaces (HCI). Several modalities of emotion recognition have been reported in the literature. Of these, the modality of facial expression analysis is given priority for its simplicity in realization in current generation HCI-system. This paper attempts to provide a novel solution to the well-known emotion recognition problem by facial expression analysis. Existing research on emotion recognition by facial expression analysis employ several models of machine learning and pattern recognition, including Bayesian classifier [1], Back-propagation neural classifier [2], [3], Support Vector Machine (SVM) [4], [5] and Hidden Markov models (HMM) [6]. Recently researchers are taking keen interest to employ the logic of fuzzy sets in emotion recognition by facial expression analysis. In [7], Chakraborty et al. employed classical fuzzy sets to recognize emotion by facial expression analysis.

In a more recent work [8], researchers considered type-2 fuzzy sets as a basic model for facial expression representation of subjects (carrying same/similar emotions), and later employed type-2 fuzzy reasoning to classify unknown facial expression into one of five known emotion classes. The latter method requires measurements of facial features to determine the emotion class of the subject by type-2 fuzzy analysis. Unfortunately, the precise measurements of the facial features requires segmentation, localization and feature extraction on the unknown facial image of the subject, and thus adds overhead to the computational complexity of the classifier algorithm. The present paper, however, overcomes the above problem by labeling the features of a facial image into fuzzy word descriptions, which are directly submitted to a fuzzy classifier for emotion recognition. Fuzzy quantifiers like LARGE, SMALL and MODERATE, and fuzzy linguistic hedges like VERY, NOT SO etc. are used to describe the qualitative variation in the fuzzy quantification. A user on observing an unknown facial expression describes the facial features using the fuzzy linguistic hedges and quantifiers. A computer receives the linguistic descriptions about the face and classifies the emotion of the subject to one of five distinct emotion classes through a process of Interval Type-2 Fuzzy Reasoning (IT2FR). The contribution of the present paper is briefly outlined below. First, the paper proposes a novel approach to translate user-defined word descriptions about facial features into emotion using IT2FR. Automatic transformation of word description of features into emotions being humanlike enhances the scope of interaction between humans and machines in the next generation human-computer interface (HCI). Second, construction of the space of footprint of uncertainty (FOU) for each facial feature (such as mouth-opening) in different linguistic grades (such as LARGE) for different emotions from the available word description obtained from several assessors carries novelty in the literature of Affective Computing. Third, the proposed scheme does not require absolute measurement of facial features as input to the emotion

classifier system. Rather, it requires only word description of facial features of a subject by a user. Naturally, the proposed method saves significant computational overhead required due to pre-processing, segmentation and localization of the facial features. Lastly, the lack of precision in word description of features in comparison to absolute measurement of features does not significantly influence the classification accuracy, indicating the robustness of the word description models. It is apparent that humans can autonomously determine emotion from facial expression of the subjects, but the machinistic classification of emotion from facial attributes of subjects is useful in certain application, where human intervention is not desirable. For example, recognition of emotion of psychological patients throughout the day carries more information than those obtained by nurses at specific time intervals. Further, the effect of human bias positive/negative arousal of emotion cannot be ignored particularly for psycho-patients. Recognition of human emotion by machines have significance in HCI, specifically to monitor the mental state of psychological patients.

The paper is divided into seven sections. In section II, the preliminaries of Type 2 fuzzy sets are given. Section III is concerned with the principles and methodology. Section IV provides the details of the fuzzy classifier. Experimental details are given in section V. Conclusions are listed in section VI.

II. PRELIMINARIES OF TYPE-2 FUZZY SETS We now define a few terminologies related to Type-1 (T1) and Type-2 (T2) fuzzy sets to develop the rest of the paper.

Definition 1: A conventional type-1 fuzzy set A defined on a universe of discourse X, is described by a two-dimensional membership function, hereafter called type-1 membership function. For any generic element ,Xx∈ the membership function (MF) )(xAμ is a crisp number in [0, 1]. Generally, a fuzzy set A is two tuple [10], [28] given by,

}.|))(,{( XxxxA A ∈∀= μ (1) Alternatively, a fuzzy set A is defined as given in (2) [11].

∫

∈

=Xx

A xxA |)(μ (2)

where ∫ denotes union of all admissible x.

Definition 2: A type-2 fuzzy set A~ is described by a three dimensional membership function, hereafter called type-2 membership function, which also is fuzzy. The type-2 membership function is usually expressed as ),,(~ uxAμ where

Xx∈ , and ].1,0[⊆∈ xJu Generally, the fuzzy set A~ is described as a two tuple:

]}1,0[,|)),(),,{((~~ ⊆∈∈= xA JuXxuxuxA μ (3)

where ].1,0[),(~ ∈uxAμ An alternative representation of type-2 fuzzy set is presented in (4).

]1,0[),,(|),(~

~ ⊆= ∫ ∫∈ ∈

xXx Ju

A JuxuxAx

μ (4)

]1,0[,/])([ ⊆= ∫ ∫∈ ∈

xXx Ju

x Jxuufx

(5)

where ]1,0[),()( ~ ∈= uxuf Ax μ . The notation ∫∫ represents a

union over all admissible values of x and u.

Definition 3: The two-dimensional plane containing axes u and ),( / uxμ at each point of x=x/ is called the vertical slice of

),(~ uxAμ [12]. A secondary membership function thus is a vertical slice of ).,(~ uxAμ Symbolically, the secondary

membership function ),(~ uxAμ at x = x/ for Xx ∈/ and

]1,0[/ ⊆∈∀ xJu is defined by

]1,0[,|)(),(/

/~ ⊆= ∫∈

/

x

xJu

xA J uufuxμ (6)

where 1)(0 / ≤≤ uf x. The magnitude of a secondary

membership function is called secondary grade [12]. /x

J in (6) is referred to as the primary membership of x/.

Definition 4: Uncertainty in the primary membership of a type-2 fuzzy set A~ is described by a bounded region, called footprint of uncertainty (FOU). The FOU is the defined as the union of all primary memberships [24], i.e.,

xUxJAFOU

∈∪=)~( (7)

In [12] , )~(AFOU has alternatively been defined as AD~ , where

XxJxAD x ∈∀=)(~ . (8) Thus (4) reduces to

∫ ∫ ∈=

ADux A uxuxA ~),(~ ),(),(~ μ (9)

In case all the secondary grades of a type-2 fuzzy set A~ are equal to 1, i.e.,

]1,0[1),(~ ⊆∈∀∈∀= xA JuX,x uxμ (10)

we call A~ an interval type-2 fuzzy set (IT2FS). The FOU in a IT2FS is bounded by two curves: the Lower and the Upper Membership Functions )(~ x

Aμ and )(~ xAμ respectively, where

)(~ xA

μ and )(~ xAμ at all x is computed by evaluating the

minimum and the maximum of the membership functions of the embedded type-1 fuzzy sets [13] inside the FOU.

III. PRINCIPLES AND METHODOLOGY Suppose that there are 10 assessors, who have definite opinions about the linguistic gradations of the facial features, such as VERY LARGE (VL), LARGE (L), MODERATE (M), SMALL (S) and VERY SMALL (VS) based on the range of specific facial attribute, say eye-opening. Thus for five linguistic grades and 10 assessors we have a matrix A (Table- I) of (10×5), where the row index i indicates assessor number i and the column index j indicates the j-th linguistic grade, and the element aij = [kfmin , kfmax] denotes the range of a facial attribute allocated to j-th linguistic grade by assessor i for the k-th feature.

Now, presume that we have a face database containing 50 facial images of 10 subjects for five distinct emotions. We now consider each group of 10 images carrying the same emotion of different subjects. These facial images are assessed by 10 assessors to attach one fuzzy linguistic grade to individual facial images based on the manual examination of the facial features by the assessors and their opinion about ranges for each linguistic grade. The linguistic grades of a facial feature for 10 facial images carrying the same emotion assessed by 10 assessors is now recorded in matrix B (Table-II), the i-th row index of which denotes assessor i, the j-th column index of which denotes image number j, and the element bij denotes the linguistic grades assigned to image j by assessor i. Thus for 7 features, we have 7 such B-like matrices.

TABLE I MATRIX A: RANGE OF 5 LINGUISTIC GRADATIONS OF A PARTICULAR

FACIAL FEATURE ACCORDING TO 10 ASSESSORS

TABLE II MATRIX B/: OPINION OF DIFFERENT ASSESSORS ABOUT A PARTICULAR

FEATURE OF 10 FACIAL IMAGES AND CORRESPONDING MOST LIKELY LINGUISTIC GRADE FOR A SPECIFIC EMOTION

Now, we identify the linguistic grade having highest frequency of occurrence in each row of B matrix, and save it in the last column of the same row in the augmented B matrix, hereafter called B/. The last column information in i-th row of B/ represents the assessment by assessor i about the most likely linguistic grade of the images carrying same emotion. Now, for each most likely membership grade, such as LARGE, VERY LARGE etc. obtained from the last column of matrix B/, we construct a IT2FS describing the concept for a

linguistic phrase like mouth-opening is LARGE. Construction of FOUs requires two steps. First, for each assessor, consulting the range of linguistic variable for each feature as given in Table-1, we construct a type-1 Membership Function (MF). Now, for each linguistic grade, say LARGE, obtained in the last column of Table-2, we consider the union of MFs representing mouth-opening is LARGE. The union of the type-1 MFs for the fuzzy proposition: mouth-opening is LARGE represents the FOU of the IT2FS describing the same concept. Thus for every linguistic grade found in the last column of Table-II we would have an FOU describing the linguistic grade. So, for 5 emotions and 6 features and 5 linguistic grades for each feature, we could have a maximum number of 5 × 6 × 5= 150 FOUs. However, as seen in Table-II the last column contains only 2 out of 5 grades for mouth-opening in emotion happiness. So, in practice the number of FOUs is much less than 150. The FOUs together is referred to as fuzzy face space. Once the fuzzy face space is created, we use it during the recognition phase. In the recognition phase, given an unknown facial image, a user manually measures the linguistic grades of each feature. Presume that the user in general is not an assessor. Since the parametric ranges of features say mo, for the linguistic grades like VERY HIGH is not known for a user, who is not an assessor, we use the average of the centre of the ranges of feature assigned by 10 assessors to the linguistic grade the user refers to describe a facial feature. Thus the average of the centre values of the j-th column of Matrix A is the measure of the gradation of the unknown feature. Thus for 6 features of the unknown facial expression, we have 6 such derived measurements. After we obtain the derived measurements about the 6 facial features, we instantiate the respective FOUs with the derived measurements following the rule. Let the derived measurement of the 6 features: 621 ,....,, fff be 621

ˆ,....,ˆ,ˆ fff respectively. Now, for the FOUs of a given emotion i, we evaluate the UMF and LMF for each feature 6 1 ,ˆ toiff ii =∀= . Now we evaluate

,))ˆ((

,))ˆ((

6

1

6

1

iii

iii

LMFfLMF

UMFfUMF

T

T

=

=

=

= (11)

and define score 2/)( iii LMFUMFsc += . We repeat the

above computation of ,, isci ∀ and regard emotion j to be the emotion of the unknown subject, if iscsc ij ∀≥ , . If a single rule only is instantiated by the word descriptions of the user, the rule will be fired and the inferred score sci of the fired rule-i represents the degree of strength of the recognized emotion as stated in the consequence of the fired classifier rule.

G1

(VS)

Gj

(M)

G5

(VL)

A1

A2

Ai

A10

[kfmin , kfmax]

I1 I2 I10

Most likely Linguistic Grade

A1

A2

A10

L M M L M M

M M M L L M

M

L

Grade

Assessor

Images (Ij)

Assessor (Ai)

IV. THE PROPOSED FUZZY CLASSIFIER The traditional fuzzy models employed in emotion recognition [7], [8], [9] requires measurements of the facial features to instantiate membership functions in the antecedent part of fuzzy rules designed for emotion classification. This calls for image segmentation, localization and measurements of the facial features in the localized images, adding computational overhead in the emotion recognition scheme. The proposed method, however, does not require (exact) measurements of facial features. A user provides her opinion about facial measurements in fuzzy linguistic grades, such as mouth-opening is VERY- SMALL, and the fuzzy word engine can determine the emotion of the subject from the fuzzy quantification about facial features. The proposed fuzzy classifier includes a set of fuzzy rules and a IT2FS inferential procedure to recognize the emotion class from the word descriptions of facial features. The other important issue to be addressed in this section is to calibrate the MFs based on individual users’ opinion about the ranges of facial features to describe linguistic grades.

A. Membership Function Selection MFs about facial features of a subject based on the fuzzy (linguistic) assessment of different assessors are determined from the assessors’ specification about the ranges of attributes to qualify a specific linguistic grade. For example, consider the facial feature: mouth-opening (mo), which is categorized into five linguistic grades by an assessor according to his own choice of the ranges for the attribute. An illustrative grading for the linguistic variable mouth-opening by an assessor is given below.

VERY SMALL(VS): when 4≤mo≤6, SMALL (S): when 6≤mo≤10 MODERATE (M): when 11≤mo≤14 LARGE (L): when 15≤mo≤18 VERY LARGE (VL): when 19≤mo

In our experiments, we consider n (=10) assessors to provide ranges for five linguistic grades stated above. To determine the MFs for fuzzy linguistic grade, we consider two issues: 1) restricting the base of individual MF based on the available range of the feature representing the linguistic grade, and 2) select a mathematical function to represent the MF, whose parameters are to be tuned intuitively as illustrated below.

TABLE III MODELS AND MODEL PARAMETERS FOR REPRESENTING TYPE-I

MEMBERSHIP FUNCTIONS

The list of intuitively selected membership function for five linguistic grades: VS, S, M, L and VL along with the set of unknown parameters of the MFs are given in Table-III. The parameters used in the MFs are determined with an aim to satisfy its characteristics in the range of intervals provided by the assessor. Examples 1 and 2 demonstrate the selection of parameters for five distinct grades of MFs. Example 1: This example illustrates the selection of k in the membership function: mokmo

L e .)( 1 −−=μ for the fuzzy concept mouth-opening is LARGE. Suppose, we want k to attain 70% of the fuzzy membership 1 at mo= 14 pixels. Then 7.01 )14.( =− −ke which returns 56599.0)3.0ln()14/1( =−=k Example 2: This example illustrates the selection of the parameters of MODERATE MFs for mouth-opening. Since for the given assessor, mo for MODERATE grade lies in 11 to 14 pixels, we set mo as the average of 11 and 14 pixels, and thus 5.12=mo pixels. The standard deviation sigma here is evaluated by considering a Gaussian MF for MODERATE grade, and so σ3+mo approximately equal to 14 pixels, yielding σ = 0.5. For the grade VS and S we select k1 =100 and k2=10 intuitively. For the grade VL, we intuitively set k4= k3

2.

B. The Classifier Rules The IT2FS reasoning system introduced here employs a particular format of rules, commonly used in fuzzy classification problems [14]. The general format of a fuzzy rule is given by

Rc: if f1 is 1~A AND f2 is 2

~A …. AND fm is mA~ then emotion class is c.

Here, fi for i=1 to m are m-measurements (feature values) in the interval type-2 fuzzy sets 1

~A , 2~A , …, mA~ respectively.

C. FOUs Principles of FOU construction have been outlined in section III. It is apparent from section III that we construct FOU for all possible linguistic grades appearing in the last column of Table-II for all the features appearing in the facial expression representative of a specific emotion. These FOUs are used in subsequent reasoning phase to classify emotion from word description about facial features, such as LARGE mouth-opening, SMALL eye-opening etc. Details of reasoning using classifier rules, FOUs and word description about facial features are given below.

D. Reasoning Technique for Emotion Classification Given a set of word descriptions about facial features, we need to determine the emotion of the subject. The following steps are undertaken to handle the problem. First, for each word description about a facial feature, such as mouth-opening is LARGE, we identify the range of the feature (here, mouth-

Gradation Selected MF Parameter of the MF VS )exp( .2

1 mok− k1

S moke .− k2

M 22 /)( σmomoe −− 2,σmo

L moke .31 −− k3

VL moke .41 −− k4

opening) for each assessor by consulting Table-I, and determine the average of the centre of the intervals describing the word description. This numerical value of the feature is used to determine the UMF and LMF of the already constructed IT2FS at the pre-determined value of the feature. Now, based on the word descriptions of m features, we obtain m feature values by the above procedure. Naturally, a classifier rule Rc introduced above is instantiated with the measurement of features, undertaken above, and the FOU for each IT2FS representing the given word description is instantiated with the derived feature values to obtain a set of m UMFs and m LMFs. We now take the minimum of the UMFs and LMFs separately and take the average of the resulting minima. The result thus obtained denotes the degree of membership of the facial expression to lie in the emotion class c following Rule Rc. If more than one rules fire with the given word descriptions about facial features, then the rule yielding largest membership of the emotion class is declared as the winner, and the corresponding emotion class c is the emotion class of the unknown subject.

V. EXPERIMENTS In this section, we present the experimental details of emotion recognition using the principles and algorithms introduced in section III and IV. We here consider 6 facial features, (i.e., m=6) and 5 emotion classes, (i.e., k=5) including anger, fear, disgust, happiness and relaxation.

A. Facial Feature Selection Existing research reports [5], [7], [8] indicate that the eyes and the lips are the most important facial regions responsible for the manifestation of emotion. This inspired us to select the following features: Left Eye Opening (eoL), Right Eye Opening (eoR), Distance between the Lower Eyelid to the Eyebrow for the Left Eye (leeL), Distance between the Lower Eyelid to Eyebrow for the Right Eye (leeR), Maximum Mouth opening (mo) including the lower and the upper lips and Eye-brow Constriction (ebc). These features are obtained from emotionally rich facial expressions synthesized by the subjects by acting. Fig. 1 explains the above facial features on a selected facial image.

B. Creating the Interval Type-2 Fuzzy Membership Space

There are 10 assessors, who have definite opinions about the linguistic gradations of the facial features, such as VERY LARGE, LARGE, SMALL etc. based on the range of specific

facial attribute. The feature: Mouth-Opening according to assessor 1 is Very Small (VS) when mo≤7, Small (S) when 8≤mo≤10 and so on as indicated in row 1 of Table-IV. Rest of the rows in Table- IV indicates gradation of mouth-opening by other assessors. So, for 6 features there are 6 such tables like Table-IV.

TABLE IV OPINION OF DIFFERENT ASSESSORS ABOUT THE GRADATIONS OF

FEATURE: MO

TABLE V OPINION OF DIFFERENT ASSESSOR ABOUT THE FEATURE: MO OF 10

INSTANCES IN FIG. 2 AND CORRESPONDING MOST LIKELY MF

Thus when an assessor examines 10 facial expressions of different subjects carrying a specific emotion (Fig. 2), he assigns one of five possible grades (VS, S, M, L, and VL) to the features of the facial expressions. Thus for 10 assessors,

Assessors Ranges of each Linguistic Grade

VS S M L VL

A1 mo≤7 8≤mo≤10 11≤mo≤13 14≤mo≤18 mo≥19

A2 mo≤5 6≤mo≤9 10≤mo≤13 14≤mo≤19 mo≥20

A3 mo≤9 10≤mo≤14 15≤mo≤18 19≤mo≤23 mo≥24

A4 mo≤6 7≤mo≤8 8≤mo≤14 15≤mo≤18 mo≥19

A5 mo≤7 8≤mo≤9 9≤mo≤12 13≤mo≤20 mo≥21

A6 mo≤8 9≤mo≤10 10≤mo≤15 16≤mo≤21 mo≥22

A7 mo≤4 5≤mo≤7 6≤mo≤9 10≤mo≤16 mo≥17

A8 mo≤3 4≤mo≤7 5≤mo≤10 11≤mo≤16 mo≥17

A9 mo≤7 8≤mo≤10 9≤mo≤14 15≤mo≤21 mo≥22

A10 mo≤6 7≤mo≤8 8≤mo≤13 14≤mo≤18 mo≥19

Assessors

Facial Images for a given emotion Most likely MF I1 I2 I3 I4 I5 I6 I7 I8 I9 I10

A1 L L VL L L M L VL VL L L

A2 L VL VL VL VL L L VL VL VL VL

A3 L L VL L L L L VL L L L

A4 M VL VL M L L M VL VL VL VL

A5 L L VL L L L L VL L L L

A6 L L VL L L M L VL VL VL L

A7 M VL VL L VL L M VL VL L VL

A8 L L VL L L L L VL L L L

A9 L L VL L L M M VL L M L

A10 L L VL L L L L VL VL L LFig. 1 Facial Features

EOL

LEEL

MO

LEER EOR

Fig. 2. Different Facial Expressions from Subjects for emotion: Happy

we obtain Table V, where all columns except the last column indicates the assigned grade of a specific feature in 10 facial expressions (carrying a specific emotion) by 10 assessors. The last column in each row indicates the most likely grade (i.e., grade with highest frequency count) assigned in that row. So, for the 6 features of each set of facial expressions like Fig. 2 there are 6 such tables like Table-V and for 5 emotional sets of facial images there are 5×6= 30 such tables. We now create an IT2FS by constructing type-1 membership function (MF) using the most likely grade of the assignment by the individual assigner. In other words, for the last entries in Table-V we construct MF in definite ranges of the facial parameter as defined in Table-IV. For each linguistic grade, here, LARGE and VERY LARGE, obtained in the last column of Table-V, we consider the union of (embedded) type-1 MFs representing mouth-opening is LARGE and the union of (embedded) type-1 MFs representing mouth-opening is VERY-LARGE to obtain the FOUs. Thus we have 2 FOUs describing the linguistic grade for emotion Happiness and feature mo. Fig. 3 gives an illustration of two such FOUs for the feature mo, emotion: Happiness, grade: LARGE and VERY-LARGE. Thus for 5 emotions, 6 features and 5 possible linguistic grades, a maximum number of 5×6×5= 150 FOUs can be constructed. But practically, the number of FOUs is much less than 150 as there are fewer valid linguistic grades, like LARGE and VERY LARGE for the feature mo in happy expressions. The FOUs together is referred to as fuzzy membership space. Once the fuzzy membership space is created, we use it during the recognition phase.

0 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

0 10 20 30 40 50 600

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

very large

C. Emotion recognition of an unknown subject Suppose we want to recognize the correct emotion class of

the facial image in Fig. 4.

TABLE VI

OPINION OF A USER ABOUT THE FEATURES OF FIG. 4

For recognizing the correct emotion of an unknown face,

first the 6 facial features i.e., eoL, eoR, mo, leeL, leeR and ebc are assessed by a user. For the facial expression shown in Fig. 4, the opinion of a user is listed in the 2nd row of Table VI.

Now, we consider the average of the centre values of each column of Table IV as the measure of the gradation of the unknown feature. The grade of mo from the 2nd row of Table VI is found to be L. So, the value of the feature mo to be M is the average of the centre value of the 4th column of Table IV, i.e., 16.55. The calculated feature values for each assigned grade for each feature are listed in the last row of Table VI. After we obtain the derived measurements about the 6 facial features, we instantiate the respective FOUs with the derived measurements following the rule. The numerical value of the feature is used to determine the UMF and LMF of the already constructed IT2FS at the pre-determined value of the feature. We now take the minimum of the UMFs and LMFs separately and take the average of the resulting minima to obtain the degree of membership of the facial expression to lie in the emotion class c following Rule Rc. For the unknown facial expression shown in Fig. 4, suppose the word descriptions are: eo is LARGE; mo is LARGE; lee is LARGE and ebc is LARGE. Here 4 features are considered among the 6 because the values of the features eoL and eoR, and leeL and leeR are almost remains same always. Suppose, the above word descriptions instantiate two rules of competitive emotion class. The first rule infers anger, while the second infers fear. Rules:

1. If eo is LARGE/VERY LARGE; mo is LARGE; lee is LARGE and ebc is LARGE- emotion is ANGER.

2. If eo is LARGE; mo is MODERATE/LARGE; lee is LARGE and ebc is LARGE- emotion is FEAR.

As the above two rules are concurrently fired with the given instantiation of word descriptions, the rule yielding largest membership of the emotion class is declared as the winner. To evaluate the degree of strength of individual emotions, we now consider four FOUs for the given word descriptions for two emotion classes: Fear and Anger (Fig. 5).

Feature eoL eoR mo leeL leeR ebc Grade of the feature L L L L L L

Numerical value of the feature

12.4 12.4 16.55 20.5 20.5 15

(a)

Fig. 4 Facial Image of an unknown subject

Fig. 3. (a) Type-1 MFs for mouth-opening is LARGE for Emotion:Happiness based on the opinion of 10 assessors and its correspondingFOU (b) Type-1 MFs for mouth-opening is VERY-LARGE and itscorresponding FOU

(b)

2 4 6 8 10 12 14 16 18 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

2 4 6 8 10 12 14 16 18 200

0. 1

0. 2

0. 3

0. 4

0. 5

0. 6

0. 7

0. 8

0. 9

1

large

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

large

The value of LMF and UMF (membership ranges) for each We now instantiate the FOUs in Fig. 5 with derived feature values listed in row 2 of Table VI, take the minimum of the UMFs and LMFs separately at the feature values and finally take the average of the two minima. Table VII provides the results of evaluation of UMF and LMF for each feature, the minimum of UMFs, the minimum of LMFs and the average of the two minima. It is observed that the average has the largest value (=0.335) for the emotion: fear. So, we can conclude that the subject in Fig. 4 carries the emotion: fear.

TABLE VII UMFS, LMFS AND AVERAGE OF THE MMINIMA OF UMFS AND LMFS

OBTAINED FROM FIG. 5

The proposed technique returns a classification accuracy of 87.8%, which is comparable with the performance of traditional emotion classifiers. But the main advantage of the proposed technique lies in determining emotion from facial word description, where we save some time by avoiding image processing for feature extraction.

VI. CONCLUSION The paper introduced a word-description model to recognize emotion of subjects from word description about facial features using IT2FS. The merit of the work lies in automatic membership function construction from word descriptions and construction of FOUs for each facial feature in different grades for different emotional faces. The proposed approach is insensitive to small changes in facial features as it deals with relatively coarse information about facial features and thus is very robust. Reasoning with word description about facial expression undertaken here is

unique and novel and is expected to have wider applications in the next generation human-computer interfaces.

REFERENCES [1] N. Sebe, M.S. Lew, I. Cohen, A. Garg, T.S. Huang, “Emotion

recognition using a Cauchy Naive Bayes classifier,” in 16th International Conference on Pattern Recognition, vol.1, pp. 17-20, 2002.

[2] Hiroshi Kobayashi and Fumio Hara, “Recognition of Six Basic Facial Expressions and Their Strength by Neural Network,” in IEEE International Workshop on Robot and Human Communication, pp. 381-386 , 1992.

[3] J. Zhao and G. Kearney, “Classifying Facial Emotions by Backpropagation Neural Networks with Fuzzy Inputs,” Proc. Conf. Neural Information Processing, vol. 1, pp. 454-457, 1996.

[4] Nello Cristianini, John Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-based Learning Methods, Cambridge University Press, 2000.

[5] Sauvik Das, Anisha Halder, Pavel Bhowmik, Aruna Chakraborty, Amit Konar , A.K. Nagar, “Voice and Facial Expression Based Classification of Emotion Using Linear Support Vector Machine”, DESE Conf., Abu Dhabi, UAE, 2009.

[6] Jen-Chun Lin, Chung-Hsien Wu, and Wen-Li Wei, “ErrorWeighted Semi-Coupled Hidden Markov Model for Audio-Visual Emotion Recognition,” in IEEE Trans. on Multimedia, vol. 14, no. 1, Feb. 2012.

[7] Chakraborty, A., Konar, A., Chakraborty, U, and Chatterjee, A., “Emotion recognition and control using fuzzy logic,” IEEE Trans. on Systems, Man and Cybernetics Part-A, vol. 39, no. 4, pp. 726-743, July 2009.

[8] Rajshree Mandal, Anisha Halder, Pavel Bhowmik, Aruna Chakraborty and Amit Konar, “Uncertainty Management in Type-2 Fuzzy Face-Space for Emotion Recognition,” Proc. of FUZZ-IEEE, Taiwan, Chaina, 2011.

[9] Anisha Halder, Pratyusha Rakshit, Sumantra Chakraborty, Amit Konar, Eunjin Kim, and Atulya K. Nagar, “Reducing Uncertainty in Interval Type-2 Fuzzy Sets for Qualitative Improvement in Emotion Recognition from Facial Expressions,” in FUZZ-IEEE, 2012, Brisbane, Australia.

[10] H.J Zimmermann, Fuzzy Set Theory and its Applications, Springer, Netherlands, 2001.

[11] Julio Romero Agüero, and Alberto Vargas “Calculating Functions of Interval Type-2 Fuzzy Numbers for Fault Current Analysis,” IEEE Trans. on Fuzzy Systems, Vol. 15, No. 1, Feb 2007.

[12] H. Wu, Y. Wu, and J. Luo, “An Interval Type-2 Fuzzy Rough Set Model for Attribute Reduction,” in IEEE Trans. on Fuzzy System, vol. 17, no. 2, April 2009.

[13] J.M Mendel and D. Wu, Perceptual Computing, IEEE Press, Wiley Publications, 2010.

[14] O. Cordon, M.J del Jesus, F. Herrera, “A proposal on reasoning methods in fuzzy rule-based classification systems,” Int. J. of Approximate Reasoning, Vol. 20, pp. 21-45, 1999.

[15] G.U. Kharat and S.V. Dudul , “Emotion Recognition from Facial Expression Using Neural Networks,” Human-Computer Sys. Intera., AISC 60, pp. 207–219 Springer-Verlag Berlin Heidelberg 2009

[16] Dae-Jin Kim, Sang-Wan Lee and Zeungnam Bien, “Facial Emotional Expression Recognition with Soft Computing Techniques,” 2005 IEEE International Conference on Fuzzy Systems

[17] M.Gargesha and P.Kuchi, “Facial expression recognition using a neural network,” Artificail Neural Computation Systems, 2002.

[18] Thomas G. Dietterich, “ Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms,” Neural Computation 10(7): 1895-1923 (1998)

[19] J. M. Mendel and Hongwei Wu, “Type-2 Fuzzistics for Nonsymmetric Interval Type-2 Fuzzy Sets: Forward Problem,” IEEE Trans. Fuzzy Systems, vol. 15, no. 5, pp. 916-930, Oct. 2007.

[20] J. M. Mendel and Hongwei Wu, “Type-2 fuzzistics for symmetric interval type-2 fuzzy sets: Part 1, forward problems,” IEEE Trans. Fuzzy Systems, vol. 14, no. 6, December 2006.

[21] H. Wu and J. M. Mendel, “Uncertainty bounds and their use in the design of interval type-2 fuzzy logic systems,” IEEE Trans. on Fuzzy Systems, vol. 10, pp. 622-639, Oct. 2002.

Emotions UMF and LMF of each Facial Feature

Minimum of UMF and LMF

Average Value eo mo lee ebc

ANGER 0.21,0.79

0.18, 0.61

0.13,0.7

0.11, 0.8

0.11, 0.61 0.25

FEAR 0.14,0.89

0.18, 0.81

0.15,0.9

0.18, 0.87

0.14, 0.81 0.335

Fig. 5 FOUs at the numerical values of TABLE VI for (a) eo is L, mo is L, lee is L and ebc is L for emotion ANGER, and (b) eo is L, mo is L, lee is L

and ebc is L for emotion FEAR

(a)

(b)

[22] Q. Liang and J. M. Mendel, “Designing Interval Type-2 Fuzzy Logic Systems Using An SVD-QR Method: Rule Reduction,” IEEE Trans. on Fuzzy Syst., vol. 15, pp. 939-957, 2000.

[23] J. M. Mendel, “Computing Derivatives in Interval Type-2 Fuzzy Logic Systems,” IEEE Trans. on Fuzzy Syst., vol. 12, no. 1, pp. 84-98, Feb., 2004.

[24] J.M. Mendel and R.I. John, “Type-2 fuzzy sets made simple”, IEEE Trans. on Fuzzy Systems, vol. 10, pp. 117-127, 2002.

[25] L. A. Zadeh, “Fuzzy Sets”, Information and Control, vol. 8, no. 3, pp. 338-353, 1965.

[26] J.M. Mendel. Type-2 fuzzy sets and systems: An overview. IEEE Computational Intelligence, pp. 20–29, February 2007.

[27] N. N. Karnik and J. M. Mendel, “Centroid of type-2 fuzzy set,” Information Sciences, vol. 132, pp. 195-220, 2001.

[28] L. A. Zadeh, The concept of a linguistic variable and its application to approximate reasoning I, Information Sciences, 8, 199-249, 1975.