eyedetectioninafaceimageusinglinearandnonlinear...

�Current address: General Electric Co./Imaging Architecture,Integrated Imaging Solutions, 800 Business Center Drive, Mt.Prospect, IL 60056, USA.

*Corresponding author. Tel.: #1-301-405-4526; fax: #1-301-314-9115.

E-mail address: [email protected] (A. Rosenfeld).

Pattern Recognition 34 (2001) 1367}1391

Eye detection in a face image using linear and nonlinear "lters

Saad A. Sirohey�, Azriel Rosenfeld*

Computer Vision Laboratory, Center for Automation Research, University of Maryland, College Park, MD 20742-3275, USA

Received 25 August 1999; received in revised form 12 May 2000; accepted 30 May 2000

Abstract

This paper describes two methods of eye detection in a face image. The face is "rst detected as a large #esh-coloredregion, and anthropometric data are then used to estimate the size and separation of the eyes. When a linear "lteringmethod, using "lters based on Gabor wavelets, was then applied to detect the eyes in the gray-level image of the face, thedetection rate was good (80% on one dataset, 95% on another), but there were many false alarms. A nonlinear "lteringmethod was therefore developed to detect the corners of the eyes in the color image of the face. This method gave a 90%detection rate with no false alarms. � 2001 Pattern Recognition Society. Published by Elsevier Science Ltd. All rightsreserved.

Keywords: Face detection; Eye detection; Linear "lters; Nonlinear "lters; Corner detection

1. Introduction

Detection of faces and facial features using machinevision techniques has many useful applications. Theseapplications range from feature analysis, i.e., eye, nose,and mouth detection, speech reading, facial expressionanalysis, etc., to holistic analysis, i.e., face recognition,gender classi"cation, etc. Though human beings accom-plish these tasks countless times a day, they are verychallenging for machine vision. For a comprehensivesurvey see Ref. [1]. This paper deals with the problem ofdetecting faces in color images, with the speci"c goal oflocating the eyes. A review of the relevant literature isgiven in Section 2.In a frontal digital image of a face it is not easy to see

any 3-D structure. However, the eyes form prominent2-D features under uniform or smoothly varyingillumination. In almost all cases the eyes come in

pairs and each eye, if it is open, can be described ashaving a white sclera and a dark iris. These facts providea basis for detecting the eyes, once the face has beendetected.In this paper we perform face region detection using#esh-tone signatures in color images, as described inSection 3. We estimate the approximate scale of the facefrom the `face bloba thus obtained. Using this scaleestimate we de"ne "ltering methods for detecting theeyes, as described in Section 4. Successful eye detectionalso serves to verify the correctness of the face regiondetection.

2. Literature review

This section reviews pertinent literature on both facedetection (Section 2.1) and facial feature detection, withemphasis on the eyes (Section 2.2). Literature dealingwith the holistic analysis or recognition of faces is gener-ally not covered here; for a review the reader is referred toRef. [1].

2.1. Face detection

We "rst review face detection in gray-level images; facedetection in color images will then be reviewed.

0031-3203/01/$20.00 � 2001 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.PII: S 0 0 3 1 - 3 2 0 3 ( 0 0 ) 0 0 0 8 2 - 0

One of the earliest papers that reported the presence orabsence of a face in an image was Ref. [2]. An edge mapextracted from the image is matched to an oval templatewhose position and size can vary. At positions wherepotential matches are reported, the face hypothesis iscon"rmed by checking for edges at the expected positionsof the eyes, mouth, etc. The performance of this techniquewas sensitive to the illumination direction.Kelly [3] introduced a top}down image analysis ap-

proach which he called `planninga for automaticallyextracting the head and body outlines from an image andsubsequently locating the eyes, nose, and mouth. Hishead extraction algorithm worked as follows: Reduced-size versions of the original image, obtained by localaveraging and sampling, are "rst searched for edges thatmay form the outline of a head. The extracted edgelocations are then projected back to the original image,and a "ne search is performed locally for edges thatbelong to the head outline. Heuristics are used to connectthese edges. Once a connected head outline is obtained,the expected locations of the eyes, nose and mouth areexamined to locate these features; heuristics are againemployed in this search process.Govindaraju et al. [4] de"ne a computational model

for locating the face in a cluttered image. Their techniqueutilizes a deformable template which is slightly di!erentfrom that of Yuille et al. [5] (see Section 2.2). The tem-plate is composed of three segments that are de"ned bythe major curvature discontinuities of the head outline.These three segments correspond to the right side, the leftside and the hairline. Each of these segments is assigneda four-tuple consisting of the arc length of the curve, thechord in vector form, the area enclosed between the curveand the chord, and the centroid of this area. If all three ofthe segments are present in particular orientations, a faceis detected. The center of the three segments gives thelocation of the center of the face. The templates areallowed to translate, scale and rotate according tospring-basedmodels, and a cost function is used to evalu-ate hypothesized face candidates. Their method wastested on about 10 images; they claim to have nevermissed a face, but they also got false alarms.Craw et al. [6] describe a method for extracting the

head area from an image, subject to constraints on thelocation of the head in the image. Resolutions of 8�8,16�16, 32�32, 64�64 and 128�128 (full scale) are usedin a multiresolution scheme. At the lowest resolutiona template of the head outline is constructed. Edge mag-nitudes and directions are calculated from the gray-levelimage using a Sobel mask, and a line follower is used toconnect the outline of the head. After the head outlinehas been located a search for lower-level featuressuch as eyes, eyebrows, and lips is conducted, guidedby the location of the head outline, using a similarline following method; but this search was not alwayssuccessful.

Another method of "nding the face in an image isdescribed by Burt [7]. It utilizes a coarse-to-"ne templatematching approach to locate the head. To illustrate theusefulness of the method, it was applied in a `smarttransmissiona system which could locate and track thehead in a video sequence and use the head locationinformation in an object-based compression algorithm.Craw et al. [8] describe a system for detecting and

measuring facial features. Their work was motivated inpart by automated indexing of police mug-shots. Theyendeavored to locate 40 feature points in a gray-scaleimage; these feature points were chosen according to thesystem de"ned by Shepherd [9], which was also used asa criterion of judgment. Their system too used a hier-archical coarse-to-"ne search, and a template de"nedusing the principle of polygonal random transformationof Grenander et al. [10]. The approximate location, scaleand orientation of the head are obtained by iterativedeformation of the template by random scaling, transla-tion and rotation. Constraints are imposed to ensure thatthese transformations do not lead to results that haveno resemblance to the human head. Optimization isachieved by simulated annealing [11]. After a rough ideaof the location of the head is obtained, re"nement is doneby transforming individual sides of the polygon. Theauthors claimed successful segmentation of the head inall 50 images that were tested. In 43 of these imagesa complete outline of the head was obtained; in theremaining seven they failed to "nd the chin. The detailedtemplate of the face included eyes, nose, mouth, etc.; inall, 1462 possible feature points were searched for. Theauthors claimed to have identi"ed 1292 of these featurepoints. They attribute the 6% incorrect identi"cationrate to be due to the presence of beards and mustaches,which caused mistakes in locating the chin and themouth. Because of its use of optimization and randomtransformation, their system was computationally inten-sive.In earlier work by the author [12] the face is seg-

mented from a moderately cluttered background, usingan approach that makes use of both the intensity imageof the face and the edge image found using the Cannyedge "nder [13]. Pre-processing tasks include locatingintersection points of edges (occlusions of objects), as-signing labels to contiguous edge segments, and linkingmost likely similar edge segments at intersection points.The face is approximated by an ellipse. An accuracy ofabove 80% was obtained when the process was appliedto a set of 48 cluttered images.Yow and Cipolla [14] describe a feature-based face

detection system. They model the face using horizontallines indicating the mouth, end of nose, eyes, and eye-brows. To compensate for occlusion or partial responses,the face model is subdivided into four components eachcomprising a di!erent subset of the original model.This subset is further divided into "ve groups that are

1368 S.A. Sirohey, A. Rosenfeld / Pattern Recognition 34 (2001) 1367}1391

perceptually paired into horizontal pairs and verticalpairs. Features are detected using an edge detector anda line linker. The results are grouped according to rulesthat assign a probability of each sublevel linking to thenext higher level. A region in the image is selected asa face if this probability is greater than a threshold.Experiments were conducted on a set of 110 images offaces at di!erent scales, orientations and viewpoints. An85% success rate was achieved. They reported that al-though their algorithm was insensitive to scale to someextent, it was not completely scale-invariant.An information-based discrimination method for face

detection is used in Ref. [15]. The Kullback relativeinformation was used to discriminate between the faceand non-face sets. The training set consisted of 11�11-pixel face images taken from the FERET database. Thepre-processing phase involved only histogram equaliza-tion. Non-face examples were constructed by using gen-eral scene images. The system was tested on the samedatabase as that of Ref. [16]. The authors reported noimprovement in the detection and false alarm rate butmention that the algorithm is an order of magnitudefaster than those in Refs. [16,17].In Ref. [18] a support vector machine (SVM) is trained

for face detection. SVM training is computationally quiteexpensive, but it results in "nding the optimum boundarybetween di!erent classes; for a detailed account of theSVM the reader is referred to Ref. [19]. Pre-processingwas identical to that in Ref. [17]. Detection was carriedout using an exhaustive hierarchical multi-scale "ltering ofthe input image with the SVM classi"er. Experiments wereconducted on two sets of images and a comparison ofSVM with the system of Sung and Poggio [17] wasperformed. Results showed an improvement over Ref. [17]for the "rst set, but no improvement for the second set.Sung and Poggio [17] used an example-based learning

approach. The training set consisted of a 19�19-pixelface image. The pre-processing was identical to that inRef. [16]. The face set is modeled by dividing it into sixGaussian clusters, each with a mean and a covariancematrix, by elliptical k-means clustering. For the non-faceclass they determined a negative region in face space by`bootstrappinga; the misclassi"ed faces are also modeledby a set of six clusters. The normalized Mahalanobisdistance is used as the metric on a subset of 75 eigenvec-tors. Two distance measures are de"ned for each of thesix face and six non-face clusters; one is the projectivedistance from the cluster mean, while the other is theorthogonal distance. For each test image, 12 pairs ofdistance measures are determined and these 12 pairs arefed to a set of three classi"ers. Experiments were per-formed on two datasets, with results of 96.7 and 84.6%success reported using a single-layer perceptron as theclassi"er.In Ref. [16] a neural network based upright face detec-

tion system is outlined. A 20�20-pixel face was used to

train the network. Pre-processing steps included using anoval mask to eliminate the corners, a brightness plane "tto remove brightness variations, and "nally histogramequalization. Training is done using multiple face imagesobtained by minor variations in scale and rotation of anoriginal and mirrored set of face images. For the non-faceset, general scenes are initially used. During the trainingphase a bootstrap technique is used to augment thenon-face region with misclassi"ed faces. In the testingphase a brute force hierarchical pyramid scheme is usedto check for faces at every location and every scale. Afterinitial "ltering with the face "lter, the results are post-processed using another neural network. Experimentswere conducted on two di!erent datasets. Test Set 1 (con-sisting of 130 images with 507 frontal faces) yielded a per-formance of 92.5%. Test Set 2 was the FERET database,which yielded a 99.5% detection rate.Ref. [20] is an extension to Ref. [16]. By augmenting

the algorithm with a router network, which determinesthe angle of rotation, the authors were successful indetecting frontal face images in various rotated orienta-tion. The new system was able to detect 76% of the facesin two test sets, with very few false alarms.A common thread in most of the face detection work

described up till now has been the fact that scale informa-tion cannot be obtained from a gray-scale image; there-fore, a brute force search has to be performed at everylocation at all possible scales to detect faces. Color im-ages provide the needed ingredient to make initialsegmentation and scale selection for faces possible.The remaining part of this section reviews work doneusing color images for segmentation and detection offaces.In Ref. [21] a face detection and tracking method is

outlined using normalized color information. The #eshregion of the face is extracted and its color distribution isdetermined. Normalized red and green values are used tocompensate for brightness. Along with this color in-formation, motion is used to determine an area in theimage that is #esh-colored and is moving. Combiningthese two information sources a head tracking systemwas developed using a neural network. Testing was doneon two sequences of 1860 and 460 frames, yielding 96 and100% accuracy, respectively.Ref. [22] describes a lip and face real-time tracker

using the same color distribution technique as in Ref.[21]. For tracking, Kalman "ltering was used. A hiddenMarkov model was used to classify head motion as wellas facial expression. A near-perfect detection rate wasreported.In Ref. [23] a color facial region template-based ap-

proach is used to detect faces in color images. The faceis divided into six rectangular regions; the forehead, twoeye regions, two cheek regions, and the mouth region.A priori constraints on chromatics and smoothnessassociated with each of these regions are used for their

S.A. Sirohey, A. Rosenfeld / Pattern Recognition 34 (2001) 1367}1391 1369

selection by assigning to each region an invariantmeasure. A multiresolution search is performed on theimage using a "xed-size template. Although the templateand its regions do not give accurate feature detections,the face region is claimed to be robustly detected, evenunder varying pose.

2.2. Facial feature detection

Nixon [24] used the Hough transform for eye detec-tion. The eye was modeled by a circle for the iris anda `tailoreda ellipse for the sclera boundary. The tailoringwas necessary because of the pointed nature of the cor-ners of the eyes. The Sobel gradient operator was appliedto the initial image. Thresholding the resulting gradientimage produced an edge image. Further constraints wereplaced on the positions of the eyes in the image (thesearch was performed on the upper half of the image).Testing was done on a set of six images taken undersimilar illumination conditions. Results of the testingindicated a small di!erence between the experimentalposition and the measured position of the iris center.Yuille et al. [5] extracted facial features using deform-

able templates. These templates are allowed to translate,rotate and deform to "t the best representation of theirshape present in the image. Preprocessing is done todetect the peaks and valleys in the initial intensity image.The template for the eye has 11 parameters consisting ofthe upper and lower arcs of the eye; the circle of the iris;the center points; and the angle of inclination of the eye.This template is "t to the image in an energy minimiz-ation sense. Energy functions of valley potential, edgepotential, image potential, peak potential and internalpotential are determined. Coe$cients are selected foreach potential and an update rule is employed to deter-mine the best parameter set. In the experiments it wasfound that the starting location of the template is criticalfor determining the exact location of the eye. When thetemplate was started above the eyebrow, the algorithmfailed to distinguish between the eye and the eyebrow.Another drawback to this approach is its computationalcomplexity. Generally speaking, template-based ap-proaches to feature extraction are more logical. Theproblem lies in the description of the templates. When-ever analytical approximations are made to the image,the system has to be tolerant to certain discrepanciesbetween the template and the actual image. This toler-ance tends to average out the di!erences that makeindividual faces unique.A statistically motivated approach to detecting and

recognizing the human eye in an intensity image with theconstraint that the face is in a frontal pose is described byHallinan [25]. A template-based approach to detectingthe eye in an image is presented. The template is depictedas having two regions of uniform intensity. The "rst is theiris region and the other is the white region of the eye.

The approach constructs an àrchetypala eye and modelsvarious distributions as variations of it. For the ìdealaeye a uniform intensity is chosen for both the iris andwhite. In an actual eye certain discrepancies from theideal are found which hamper the uniform intensitychoice. These discrepancies can be modeled as `noiseacomponents added to the ideal image. For instance, thewhite region might have speckle (spot) points dependingon scale, lighting direction, etc. Likewise, the iris can have`whitea spots within it. The author uses an �-trimmeddistribution for both the iris and the white. A `blobadetection system is developed to locate the intensityvalley caused by the iris enclosed by the white. Using�-trimmed means and variances and a parameter set forthe template of the blob, a cost function is determined forvalley detection. A deformable human eye template isconstructed around the valley detection scheme. Thesearch for candidates uses a coarse-to-"ne approach.Minimization is achieved using the steepest descentmethod. After locating the candidate a goodness-of-"tcriterion is used for veri"cation purposes. The inputsused in the experiments were frontal face intensity im-ages. In all, three sets of data were used. One consisted of25 images used as a testing set, another had 107 positiveeyes, and the third consisted of images with most prob-ably erroneous locations which could be chosen as candi-date templates. For locating the valleys the author re-ported as many as 60 false alarms for the "rst data set, 30for the second and 110 for the third. An increase in hitrate was reported when using the �-trimmed distribution.The overall best hit rate reported was 80%.Reisfeld and Yeshurun [26] use a generalized sym-

metry operator for "nding the eyes and mouth in a face.Their motivation stems from the almost symmetric na-ture of the face about a vertical line through the nose.Subsequent symmetries lie within features such as theeyes, nose and mouth. The symmetry operator locatespoints in the image corresponding to high values ofa symmetry measure discussed in detail in Ref. [26]. Theyindicate their procedure's superiority over other correla-tion-based schemes like that of Baron [27] in the sensethat their scheme is independent of scale or orientation.However, since no a priori knowledge of face location isused, the search for symmetry points is computationallyintensive. The authors mention a success rate of 95% ontheir face image database, with the constraint that theface occupies between 15 and 60% of the image.In an extension to their &&eigenface'' method, Ref. [28]

uses a modular èigenspacea for detecting features in theface. They construct principal component projectivespaces for the eyes, nose and mouth and call themèigeneyesa, eigennosea and èigenmoutha, respectively.Detection of these features involves a hierarchical searchin scale and space for a region in the image whoseprojection matches the signature of the feature in ques-tion. Detection rates of 94, 80, and 56%were reported for


Fig. 1. Average scatter plot of #esh tone rescaled to 100�100.

�http://pics.psych.stir.ac.uk/ (click on image database).

the eyes, nose and mouth, respectively, on a large datasetof 7562 images.Gabor [29] "lters are used to locate features of the

eyes. These features include the corners of the eyes andthe iris. In a model-based approach, steerable Gabor"lters are used to locate and track the corners of the eyesas well as "nding the iris. A sequential search is employedby "rst "nding a particular edge, e.g., the left corner ofthe iris, then using steerable "lters to track the edge of theiris. A similar method is used to "nd the corners of theeyes. Experiments were performed on 100 high(512�512) resolution face images with di!use illumina-tion. The authors reported an overall iris detection errorrate of 2.35%. On the remaining images the outer eyecorners were found successfully while a 91.6% successrate was reported for the inner eye corners.Most of the literature reviewed here involved brute-

force, hierarchical search for the detection of faces ingray-scale images. Color images, along with #esh-toneinformation, provide a means of estimating the size of theface. We will augment the face detection method de-scribed in Ref. [21] and use a "lter-based approach tolocate the eyes for face detection veri"cation as well asfacial feature detection. More speci"cally, we will de-scribe two "lter-based approaches, one using Gaborwavelet-based linear "lters for eye detection, and theother using non-linear "lters for eye corner detection. Aswe will see, the linear method has good detection perfor-mance, but gives many false alarms; this led us to developthe non-linear method.Our methods will be tested on the Aberdeen, UK face

image database,� as well as on video frames collected inour laboratory. The methods described in the literaturehave been tested on di!erent datasets; this makes itdi$cult to compare the performance of di!erentmethods. At the algorithm level, our approach has signif-icantly less complexity and should perform better thanthose in the literature.

3. Face detection

In this section we describe in detail the method wehave used to detect and delineate the face region from thebackground. Our method is closely related to that de-scribed in Ref. [21].

3.1. Color segmentation

Let I represent a color image with RGB values. Thisimage is normalized to r"R/(R#G#B) andg"G/(R#G#B) which we callI

�[21]. We determine

the #esh tone color space by manually selecting #eshregions in color images and histogramming their r andg values (both rescaled to 100), giving H

�(r, g), where

subscript i is the image number. The average histogram isH"1/n��

��H

�(r, g); it is shown as a 2D scatter plot in

Fig. 1. In Ref. [21] motion information was used todecide if a region of interest was a face or not. We do nothave motion information, as our database consists ofsingle images. We assume that the face is the most prom-inent #esh-tone region in the image, and we verify ourassumption with the aid of detecting internal featuressuch as eyes.Many methods of analyzing histograms can be found

in the literature [30]. We can model the scatter plot byeither a single or a mixture of Gaussian distributions. Byexamining the scatter plot of the average histogram wefound that a single 2D Gaussian was an adequate model.The "tted Gaussian had the following parameters:

�"�42.91

32.28�,

�"�19.28 !5.45

!5.45 8.55�. (1)

Let < be the two-dimensional vector of r and g, with��and �

�the respective means and � representing the

covariance. The transformation

<K "A<#E[<],

where

A"B�� (2)


Fig. 2. Images 1}36 of the Aberdeen database.

(with B and � being the eigenvectors and eigenvalues of�, respectively) will give uncorrelated r and g values. Thisprocedure transforms the r and g values to the axes of theGaussian cluster. We found that the resulting rotationwas negligible and a simple rectangular bounding boxaround the cluster was su$cient for our experiments.This analysis gives us D

��, which is the #esh tone

color space. D��

is used to segment the image intoregions of interest and background. This binary imageF

��"I

�:I

�(r, g)3D

��is further processed by con-

nected component labeling, resulting in the formation ofblobs of #esh tone.Seventy-two color images were taken from the Aber-

deen, UK database. These images were collected underslightly varying illumination and scale. All of these im-ages have a frontal view of the face. The original imagescan be seen in Figs. 2 and 3 and the results of #esh tone"ltering are shown in Figs. 4 and 5. A bounding rectangle

is used to enclose the face blob. As can be seen from theresults of the #esh tone "ltering, illumination changes,especially highlights, tend to a!ect the color of the #eshregion. This is apparent from the large number of `holesain the face blob. Although there exist features (regions) inthe face blob which should deviate from the #esh tone, forinstance, the eyes, nostrils and in some cases the lips, wecannot rely on this being the case in general.It should be noted that the normalized color space for#esh tone is dependent on the color of the illuminationsource. This creates a need to initialize the #esh tone fora particular acquisition scenario. Once initialization hasbeen done, the #esh tone is relatively immune to indi-vidual variations in #esh tone, even across di!erent races[21].As a secondary database we used a sequence of 21

(nonconsecutive) video frames acquired in our laborat-ory. This sequence was collected in a semi-controlled


Fig. 3. Images 37}72 of the Aberdeen database.

environment. The background is more cluttered than theone for the Aberdeen images. Histogram analysis gave usthe following parameter values:

�"�52.66

29.99�,

�"�68.94 !30.89

!30.89 18.11�. (3)

As in the case of the Aberdeen images we used a simplerectangular bounding box to specify the #esh tone region.It should be noted that the parameter values for thisdatabase are di!erent from those for the Aberdeendatabase.A sample image from the frame sequence and the result

of #esh tone "ltering are shown in Figs. 6(a) and(b), respectively. The region of interest in Fig. 6

and its connected components of #esh tone are shownin Fig. 7.

3.2. Scale selection

The literature reviewed in Section 2.1 indicates thedi$culty of determining the scale of the face in anintensity image without brute-force search. This task issomewhat easier when color images are used and pre-screening is done using #esh-tone color information, buteven in a color segmentation the exact scale (or size) ofthe face is not directly evident. This is clear from the colorsegmentation results in Figs. 4 and 5. We have useda rectangular bounding box to determine the face area.This area may contain part of the neck as well as somehair (if the hair color matches the #esh color).Our goal is to determine the size of the eyes (orbits). If

the mandible size (Fig. 8) is known then it is very easy to


Fig. 4. Delineated #esh regions for the images of Fig. 2.

determine the orbit size using the reference values in Ref.[31]. The face blob has the face region as a subset, sincethe neck and sometimes the hair (or lack thereof) alsoshow up as #esh tone. There is no easy way to determinethe exact size of the mandible. However, we can approx-imate the mandible size by the square root of the area ofthe face blob. Using reference values from Ref. [31] wedetermine the mean and variance of the ratio betweenthe blob size and the orbit size. This ratio can be used toestimate the size of the eye. We have experimentallyfound this estimate to correctly determine the size of theeye in the majority of our images. Minor exceptionsoccur when long hair matching the #esh tone is present.Examples of this can be seen in Figs. 4 and 5.In summary, after the #esh-tone colored blobs are

segmented, an estimate of the orbit size is computedusing anthropometric ratios. We will now describe twomethods of detecting the eyes using this information.

4. Eye detection

In this section we describe two methods of detectingthe eyes in the face image. The "rst method, which isdescribed in greater detail in Ref. [32], uses linear "ltersto detect the eyes in the gray level image of the face, asdescribed in Section 4.1. This method gave good detec-tion results, but it also gave many false alarms whenapplied to our face databases. We therefore developeda second method in which nonlinear "lters were used todetect the corners of the eyes, as described in Section 4.2.This method also gave good detection results, with nofalse alarms.

4.1. Eye detection by linear xltering

Our linear eye detection "lter is constructed out oforiented Gabor wavelets which form an approximation


Fig. 5. Delineated #esh regions for the images of Fig. 3.

Fig. 6. (a) Sample image from our sequence. (b) Result of #esh-tone "ltering.

Fig. 7. Region of interest in Fig. 6 and its connected compo-nents.

to the eye. Gabor wavelets have been used in a variety offeature extraction and face recognition algorithms; seeRefs. [29,33,34]. One of the most attractive reasons forusing Gabor wavelets is their energy localization prop-erty in both the space and frequency domains. In Ref.[29] steerable Gabor wavelets were used to track the leftand right corners of the eyes.


Fig. 8. Scale determination of orbits.

Fig. 9. Sketch of an eye.

Fig. 10. (a) Real part of Gabor wavelet. (b) Imaginary part of Gabor wavelet.

Fig. 11. Schematic plan of the placement of the Gabor wavelets.

The general shape of the eye is elongated with the longaxis horizontal. Fig. 9 is a sketch of an eye as a 2D object.The size of the eye is not constant even if the distance ofthe face from the camera is "xed. An exact matched"ltering technique will run into problems due to thesevariations. Fig. 9 suggests "nding the eyes by detectingthe right and left sclera regions as well as the iris region.There is a disparity in intensity values between the#esh region of the face and the sclera. We use Gaborwavelets to detect the edges of the eye away from thecorners of the eye. At an appropriate scale and orienta-tion, "ltering with a single Gabor wavelet gives a promin-ent response to the corresponding pattern of intensity

variations in the image. If we look at the real and imagi-nary parts of the wavelet (Figs. 10(a) and (b)) we noticethat the real part is akin to a spot detector and theimaginary part resembles a step edge detector. Our ap-proach utilizes the ability of Gabor wavelets to "ndoriented edges and applies it to detect the edges of theeye's sclera.Each eye is modeled by four Gabor wavelets in a pre-

scribed formation (Fig. 11). Although most of the workdone on Gabor wavelet "ltering for detection of featuresinvolves the real part [33], we have found that the imagi-nary part is ideally suited for detecting the boundarybetween the #esh region of the face and the sclera. Theimaginary part of the Gabor wavelet is asymmetrical inthe direction of the sinusoid propagation. To capture thesclera, we orient the positive parts of the wavelets towardthe inside of the eye. This yields a high response if all fourwavelet "lters "nd the eye edges.Fig. 11 shows the orientations of the Gabor wavelets

that maximize the detection of all four intensitytransition regions of the eye. In addition to these four"lters we also use a Gaussian "lter to detect the darkcircle of the iris. Fig. 12 is a graphical rendition of thecombined wavelet and Gaussian "lter.


Fig. 12. Graphical view of the eye "lter.

Fig. 13. (a) Left opposing wavelets, (b) right opposing wavelets and (c) central Gaussian for iris detection.

A tradeo! is needed to optimize the selectivity of the"lter and its immunity to illumination variations. Thisled us to decompose the "lter into three componentswhich we call the left opposing wavelets (Fig. 13(a)), theright opposing wavelets (Fig. 13(b)), and the centralGaussian for iris detection (Fig. 13(c)).The image is convolved with these three "lters separ-

ately. To select a feature point, threshold values areselected at the mean plus 1.5 times the standard deviationfor each of the three convolutions. A point in the image isclassi"ed as a feature if all three convolutions at thatpoint exceed their thresholds.The scale parameter is set to be 55% of the size of the

orbit that is determined by the scale selection algorithm(Section 3.2). The horizontal and vertical displacementsare kept at a ratio of 3 : 1 in favor of the horizontaldirection. The angular parameters are selected as $303for the left and right opposing wavelets, respectively.

To eliminate false alarms and validate correct detec-tions, we employ two stages of post-processing. The "rststage acts as a pre-screener using one-dimensional inten-sity variations to verify the eye hypothesis, and the sec-ond stage uses scale information to pair these screenedresponses. We "rst describe the veri"cation stage in somedetail, because it is also used in our nonlinear "lteringmethod of eye detection; we then brie#y describe thepairing stage.The rationale behind the veri"cation stage is to obtain

independent con"rmation for the detection of the eye.Instead of performing two-dimensional spatial patternmatching we project a portion of the image onto onedimension and analyze the projection to determine if theimage has `eyenessa properties. `Eyenessa can be de-scribed by the light to dark to light intensity variationfrom the left part of the sclera to the iris to the right partof the sclera.A compact light source tends to produce di!ering

intensity values across the image. This was encounteredin the #esh tone classi"cation method used for face detec-tion as well, where it produced holes in the face blob, asseen previously. A compact light source creates a high-light on the iris. The position of this highlight is depen-dent on (a) the direction of the light source and (b) thegaze direction of the eye. The eye is a spherical object andwill re#ect the light source at the point where the normalto the eye bisects the angle between the light source andthe camera. This highlight manifests itself as a high inten-sity value on the iris. A simple light to dark to light


Fig. 14. (a) t��, (b) t

��, (c) t

��, and (d) t

�� .

approach to describing èyenessa will fail due to this falsebright intensity value at a point where the dark irisshould be present. This highlight could be modeled asspot or specular noise, and the "rst thought might be touse a smoothing "lter to reduce its e!ect. However, sincethe spatial resolution of the eye is very small, anysmoothing operation will result in corrupting the naturalvariations of the sclera and iris that we are trying todetect.At every point where an eye has been hypothesized we

use the size of the estimated orbit to de"ne a region in theimage centered at that point. Let this region be e and itssize be m�n (in rows and columns). e is projected ontofour vectors of n elements by taking the minimum, max-imum, average and median values of each column (seebelow for the rationale for these four projected spaces).The vectors are thus given by

t��( j)"min(e (i, j), i"1,2,m),

t��( j)"max(e (i, j), i"1,2,m),

t��( j)"mean(e (i, j), i"1,2,m),

t��

( j)"median(e (i, j), i"1,2,m). (4)

Examples of these projections are shown in Fig. 14.The original èyenessa criterion would have given rise

to a Ùa shaped pattern, with the left sclera forming onepeak, the right sclera forming the other peak, and the irisforming a trough between these two peaks. When a high-

light is present in the iris, the Ùa turns into a `Wa (seeFig. 15).We found that the highlight on the iris sometimes

creates a problem for t��. Slightly o!set detection of the

eye sometimes causes the dark area of the eyelashes to beincluded in e, causing t

��to give erroneous responses.

Using t��

sometimes fails when the eye is rolled to theside and t

�� sometimes gives erroneous values when

the eye detection is not on the central horizontal axis.To compensate for these individual shortcomings oft��, t

��, t

��, and, t

�� , we used all of the projections

and constructed a consensus-based decision criterion.Each of the projected vectors t

�: i3�min,max, avg,

and median� is divided into two equal halves t��and

t��representing the left and right halves of the eye. We

locate the maximum point in each of the halves; letx��

"max(t��) be the maximum for the left half and

x��

"max(t��) the maximum for the right half. For the

iris we determine the minimum point between x��

andx��

and call it x��. Consensus is obtained by using

a voting strategy. We allow a tolerance of one positionand the four projected vectors are equally weighted.Voting is done by the left half and right half maxima ofevery projected vector. A decision is reached if any pointon the vector has more than two candidates voting for itand the ratio between the mean value of these two pointsin the maximum projection and the minimum point inthe minimum projection is above a set threshold.


Fig. 15. Average value projection (b) shows `Wa caused by the highlight on the iris (a).

Table 1Intermediate and "nal results of linear "ltering for images 1}36

Image no. Initialcenters

Veri"edcenters

Pairedcenters

im1 14 2 0im2 30 7 3im3 28 7 2im4 40 11 6im5 24 4 1im6 55 8 3im7 52 10 4im8 53 6 2im9 44 8 2im10 70 8 0im11 54 8 2im12 61 10 7im13 57 9 4im14 50 9 5im15 70 13 6im16 61 11 7im17 80 12 10im18 58 10 2im19 48 6 1im20 32 7 8im21 37 7 11im22 41 8 0im23 148 16 1im24 158 13 2im25 48 5 4im26 67 6 2im27 70 6 2im28 75 6 0im29 79 14 2im30 95 16 5im31 64 11 2im32 51 13 6im33 43 12 8im34 55 16 8im35 106 14 3im36 117 17 4

Finally, we brie#y describe the pairing stage. The eyesin a frontal image of the face almost always come in pairs.Since we have the scale information for the face, we canimpose spatial distance and orientation conditions on theresponses that have been passed by the veri"cation stage.Pairing is achieved by looking for responses that arewithin a distance tolerance of d

��and have an angular

deviation of at most $� degrees from the horizontal;� was set at 7%.Tables 1 and 2 show the results of applying the linear"lter, followed by the veri"cation and pairing post-pro-cessing stages, to the Aberdeen database. These tablesshow the image number in the "rst column, the numberof responses after applying the linear "lter in the secondcolumn, the number of points which satisfy the `eyenessacriterion in the third column, and the "nal number ofpairs that are selected as eye candidates in the fourthcolumn. We see that the linear "ltering approach pro-vides adequate eye detection, but its false alarm rate israther high.An examination of the results leads to the conclusion

that this approach is best suited for uniform illuminationsituations where the eyes are wide open, closely resem-bling the "gure of the eye as seen in Fig. 9. A reason formissing the detection of an eye is that, due to varyingillumination across the face, thresholding (based on thestatistics of the "lter responses) may produce good detec-tions on the side of the face closest to the illuminationsource and poor detections on the side away from thesource. In such a situation only one eye will be detected,and since we are enforcing a pair of eyes condition for the"nal selection, this results in a missed detection. Lower-ing the threshold decreases the missed detection rate butincreases the number of false alarms. Another factorresponsible for false alarms is detections that are close toeach other.The linear "ltering method was also applied to our

frame sequence. The o!set m was again kept at 1.5 withthe scale parameter s set at 65% of the orbit size deter-mined by the scale selection procedure. d

��is again set


Table 2Intermediate and "nal results of linear "ltering for images 37}72


Veri"edcenters

Pairedcenters

im37 118 15 2im38 102 20 3im39 23 6 0im40 12 6 0im41 11 5 0im42 58 12 7im43 58 11 6im44 46 11 2im45 21 1 0im46 57 10 4im47 42 10 2im48 57 17 14im49 38 6 4im50 51 9 6im51 50 7 3im52 33 2 1im53 32 4 2im54 21 6 3im55 126 14 0im56 72 25 9im57 65 24 5im58 66 31 4im59 41 10 3im60 40 6 1im61 56 7 0im62 55 11 1im63 41 4 0im64 28 5 6im65 65 5 2im66 66 4 2im67 63 12 6im68 68 8 6im69 60 5 5im70 27 6 2im71 24 3 0im72 24 4 1

Table 3Intermediate and "nal results of linear "ltering for frames 1}21


Veri"edcenters

Pairedcenters

im1 31 16 3im2 38 17 4im3 33 14 3im4 37 13 3im5 32 12 3im6 30 13 3im7 41 11 2im8 39 15 2im9 41 17 5im10 36 13 3im11 35 13 4im12 36 12 2im13 38 9 2im14 43 17 1im15 46 13 2im16 43 16 3im17 48 20 3im18 37 16 3im19 36 14 3im20 42 18 5im21 37 15 4

by the scale selection and the rotation tolerance �"73.Table 3 shows the results of applying the various stagesto the 21 frames of the sequence. The eyes in these imagesclosely resemble the ideal eye depicted in Fig. 9. Thisleads to better detection rates, as seen from the results.False alarms, however, are still present.Tables 1}3 show that our linear "ltering method, fol-

lowed by postprocessing, had an 80% detection rate onthe Aberdeen database and a 95% detection rate on theframe sequence, but is su!ered from many false alarms.The high false alarm rate of the linear "ltering method ledus to develop a non-linear approach based on eye cornerdetection, as described in Section 4.2.

4.2. Eye detection by nonlinear xltering

The linear "ltering-based method described in Section4.1 was applied to gray-level images; color was used onlyto detect the face and determine the scale for the "lter.Color images provide additional information which wecan also use in detecting the eyes. This information ispresent in the form of semi-distinct normalized color dis-tributions of the sclera and the #esh. The semi-distinctnesscan be seen in Fig. 16 and its reason is explained below.The eyes can be thought of as islands in a sea of #esh

tone. They have a distinct spatial pattern at each corner.This pattern resembles a wedge of sclera color sur-rounded by #esh tone. It should be noted here that we areignoring the iris color, because it has a large degree ofvariability across individuals.In this section we de"ne non-linear "lters to detect the

left and right corners of the eyes in a color image. These"lters are constructed to take into account the variabilityof squint in the eye, a factor we were unable to accountfor in our linear "ltering method. The design and imple-mentation of the "lters is described in Section 4.2.1.Since our "lters are designed to detect left and right

corners separately, their responses provide rich informa-tion for detecting the eyes. We will describe the use of thisinformation in Section 4.2.2. Experimental results of test-ing the non-linear "ltering-based method on the sameimages that were used in Section 4.1 are given in Section4.2.3.


Fig. 16. Normalized r and g color distributions for (a) #esh tone and (b) sclera, both scaled to 100.

Fig. 17. Wedge "lters.

4.2.1. Filter design and implementationOur "lters are applied to color images and make use of

information about the normalized color distribution ofthe #esh tone in the face [21]. In the same way that theface region's color distribution is determined, we alsodetermine the color distribution of the sclera region. Letthe normalized color distribution for the face beD

��(Fig. 16(a)) and that for the sclera beD

��(Fig. 16(b))

(these normalized distributions are empirically deter-mined from r and g as in Section 3.1). The sclera regioncan contain some #esh-tone-like characteristics due tothe presence of blood vessels in it. In addition to this,highlights on the face tend to push the #esh-tone towardthe eye-tone. These factors cause the color distribution ofthe sclera region to overlap that of the #esh-tone region,as can be seen in Fig. 16. Discrimination between the#esh region and the sclera region has to take this overlapinto account. This is accomplished by experimentallydetermining a boundary in the normalized (r, g) space.Given the #esh-tone and eye-tone distributions in the

normalized color space, we can construct non-linear "l-ters to detect the left and right corners of the eye. Weknow that the eye corners are surrounded by #esh re-

gions, and in turn surround the sclera of the eye. Our"lters are tuned to look for positions in the image whichhave this characteristic.At every location in the face region we apply left and

right eye corner "lters. These `wedge "ltersa are made upof three wedge-shaped parts as shown in Fig. 17. Thelargest wedge looks for #esh tone and the smallest wedgelooks for sclera tone. There is a `don't carea regionbetween the sclera wedge and the #esh wedge; this allowsfor possible squinting of the eye. A corner, either right orleft, is detected if the average value of the pixels in the#esh wedge satis"es D

��and the average value of the

pixels in the sclera wedge satis"es D��. (We also tested

approaches not based on the average, e.g., determiningthe numbers of #esh-colored and sclera-colored pixels inthe wedge. We found that this approach was much lesstolerant to noise than the averaging approach.)A lot of work has been done on corner detection in

gray-scale images [35}38]. Some methods make use ofvariable wedge-shaped neighborhoods [39,40]. Ourmethod di!ers because we use color images and we knowthe approximate orientation and scale of the corners thatwe are interested in. We also use `don't carea regions,


which are a standard idea in edge detection (for examplethe zeros in the Sobel operator). Non-linear feature de-tectors, especially for lines, are discussed extensively inRef. [41]. The discussion in Ref. [41] provides a frame-work for de"ning such detectors for other types of fea-tures, such as corners. We have also made use of thedistinction in Ref. [41] between non-linear and semi-linear "lters.Our corner "lters, like their linear counterparts in

Section 4.1, are scale-dependent. The "lters shown in Fig.17 have a radius of 15 pixels. The inner wedge has anangular size of 203, and the outer wedge has an angularsize of 703. The don't care region thus has two parts of253 each. In rescaling to take into account the size of theface, the only change needed is in the radius of the "lter.The extracted face subimage (Section 3.1) is treated as

the input to the "ltering process. Scale determination(Section 3.2) allows us to select the radius of the "lters.Although scale has some e!ect on the performance of the"lters, it is not as critical as in the case of the linear "lter.For this reason we determined the average scale of ourdata set and used a single scale in the "ltering process.Let the "lter radius be n pixels. LetWL

�andWL

�be

the outer and inner left corner wedges, and letWR�and

WR�be the corresponding right corner wedges. It should

be noted that these are binary "lters. The "lter responseis an `ANDINGa of an image region corresponding tothe size of the wedge with the wedge, resulting in selectionof all the image values at locations where the wedge hasvalue 1. This process is done for all four wedges at everypossible image location.LetFL

��and FL

��be the normalized r responses of

"ltering with the outer and inner left corner wedges, andlet FR

��and FR

��be the normalized r responses using

the outer and inner right corner wedges. Then

FL��(i, j)"mean(I

�(i!n : i#n, j!n : j#n)�WL

�),


�(i!n : i#n, j!n : j#n)�WL

�),

FR��(i, j)"mean(I

�(i!n : i#n, j!n : j#n)�WR

�),


�(i!n : i#n, j!n : j#n)�WR

�),

(5)

whereI�is the extracted image of normalized r values as

determined in Section 3.1. Similarly, the responses for thenormalized g image are


�(i!n : i#n, j!n : j#n)�WL

�),


�(i!n : i#n, j!n : j#n)�WL

�),


�(i!n : i#n, j!n : j#n)�WR

�),


�(i!n : i#n, j!n : j#n)�WR

�).

(6)

Eqs. (5) and (6) determine four pairs of values of everypossible pixel location in the image. To determine if animage location quali"es as either a left corner or a rightcorner we need to check its normalized color valuesagainst the #esh tone and sclera tone values that we havedetermined above. Let ¸ indicate a left corner detectionimage andR indicate a right corner detection image; then

¸(i, j)"(FL��(i, j),FL

��(i, j))3D

��

�(FL��(i, j),FL

��(i, j))3D

��,

R(i, j)"(FR��(i, j),FR

��(i, j))3D

��

�(FR��(i, j),FR

��(i, j))3D

��. (7)

Using mean values in the "lter application tends toincrease the number of responses. These responses tendto form clusters. We use pruning operations to reduce thee!ect of this clustering. The simplest operation would beto perform connected component labeling and use thecenter of mass of each cluster as the pruned response.However, we are using wedge-shaped "lters for both leftand right corner detection, and we know the approxim-ate orientation of the eyes. This justi"es the followingpruning heuristic: For the left corner clustered responseswe use the mean value in the vertical direction but theminimum value in the horizontal direction. Likewise, forthe right corner clustered responses we use the meanvalue in the vertical direction but the maximum value inthe horizontal direction.Figs. 18(a) and (b) show the clustering of the left and

right corner responses, respectively. Figs. 18(c) and (d) arethe responses after the pruning operations describedabove. We can see the e!ect of using these pruningoperations by the results shown in Fig. 18. The left corneroperation looks for the leftmost midpoint of the cluster,thus e!ectively picking up the farthest left response. Sim-ilarly, the right corner operation looks for the rightmostmidpoint of the cluster. Averaging the masked valueswould introduce responses that are displaced away fromthe actual corners, e.g., a left corner response would bedisplaced to the right, into the sclera region. Our pruningoperations e!ectively negate this artifact.Figs. 18(c) and (d) still show a lot of false responses.

This demonstrates the need to do further post-process-ing. It should be noted that the responses are identi"ed asright or left responses, and we also have scale informa-tion from the face detection and scale selection operation.Combining this information gives us more #exibility thanwe had in the linear "ltering approach. In the nextsection we will describe the post-processing that we per-form after non-linear "ltering, using the corner detectionresults as a starting point.

4.2.2. Post-processingThe corner detection algorithm gives responses for left

corners and right corners. We also have scale information


Fig. 18. Non-linear "lter responses for (a) left corner and (b)right corner detection. (c) and (d) show the responses afterperforming the pruning operations.

Fig. 19. Block diagram of post-processing steps.

about the face, speci"cally, an estimate of the orbit size.Using reference values we can easily "nd the inter-oculardistance from the estimates of the orbit size [31]. In thissection we describe post-processing methods that makeuse of this geometric information.Fig. 19 presents a block diagram of our post-process-

ing procedure. We pair the left and right responses inthree di!erent ways. The "rst one (indicated by the LRchannel in Fig. 19) pairs a left response with the corre-sponding right response using the orbit size as the pairingcriterion. This pairing is termed LR pairing. We havefound that this simple pairing alone does not e!ectivelyeliminate the false alarms. The second and third pairingsare shown in the LL RR channel of Fig. 19. In this casewe use the interocular distance as the pairing criterion.We pair left responses that satisfy this criterion and termthe result LL, and we similarly pair right responses andterm the result RR. We then use the `eyenessa veri"ca-tion procedure described in Section 4.1 to reduce thenumber of pairings. Finally, we combine the survivingLR, LL and RR pairs into "nal decisions. We will de-scribe the all of these steps in detail in the subsectionsthat follow.

4.2.2.1. Pairing responses. The "rst pairing that we dois for left and right corner responses that satisfy the orbitsize criterion. We are looking for the left and right cor-ners of an eye. Let the left corner responses bel(i), i"1,2, k where l(i) is the coordinate pair for the ithresponse and k is the total number of left responses.Similarly, r(i), i"1,2,m represents the ith right cornerresponse and m is the total number of right corner re-sponses. Let v

��be the k�m vector joining each left

corner response to every right corner response. Let d��be

the k�m distance matrix representing the ¸�distance

between the left and right corner responses. Also, let��be the k�m slope matrix representing the slope of the

line connecting each left corner response with each rightcorner response. Then

d��(i, j)"��v

��(i, j)��, ∀(i, j)3N��,

��(i, j)"L(v

��(i, j)), ∀(i, j)3N��. (8)

Let the estimated orbit size range be o( b��

to o( b��.

Since we are dealing with frontal upright faces, we do notanticipate signi"cant rotation. For this reason we havea tight bound on the allowable slope value of the lineconnecting the two corners of the eye. Let this bound be��. A pair of left and right corner responses is said to

form an LR pair if the following conditions hold:

o( b��

)d��(i, j))o( b

��,

��(i, j)�)

��. (9)

Left corner response l(i) then forms an LR pair with rightcorner response r( j). Fig. 20 shows the resulting LR pairsformed from the left and right responses shown in Figs.18(c) and (d).The second type of pairing joins left (or right) eye

corners that could be the left (or right) corners of twoeyes. We denote a left/right pairing by LL anda right/right pairing by RR. Using the same notation forleft and right responses we denote by v

��and v

��the k�k

and m�m vectors joining each left corner with everyother left corner and each right corner with every other


Fig. 20. Pairing left/right responses.

Fig. 21. Left/left pairing and (b) right/right pairing.

right corner, respectively. d��is the k�k distance matrix

for LL pairings with the corresponding slope matrix ��.

Similarly, d��is them�m distance matrix for RR pairings

with the corresponding slope matrix ��. These matrices

are determined by

d��(i, j)"��v

��(i, j)��, ∀iOj3N��,

��(i, j)"L(v

��(i, j)), ∀iOj3N��,

d��(i, j)"��v

��(i, j)��, ∀iOj3N��,

��(i, j)"L(v

��(i, j)), ∀iOj3N��. (10)

Let the inter-ocular distance range between o( c��

ando( c

��and the let the slope tolerance for LL and RR

pairings be ��. An LL pairing is formed between l(i) and

l( j) if

o( c��

)d��(i, j))o( c

��,

��(i, j)�)

��(11)

are satis"ed. Similarly, an RR pairing is formed betweenr(i) and r( j) if

o( c��

)d��(i, j))o( c

��,

��(i, j)�)

��(12)

are satis"ed. Fig. 21(a) shows all the LL pairs in Fig. 20while Fig. 21(b) shows the RR pairs.

4.2.2.2. Verifying Èyenessa. Once we have the LR, LLand RR pairings, we proceed to the next phase wherescreening for èyenessa is done. Èyenessa refers to thecharacteristic `Wa -shaped patterns that are expected tobe present in the eyes. The èyenessa veri"cation processis the same as the one described in Section 4.1, except thatthe extracted regions are di!erent; in fact, these regionsdi!er even among the three classes themselves.The pairing procedure results in three sets of pairs, one

each for the LR, LL and RR cases. Each set of pairscorresponds to two points on the image. We will analyze

the èyenessa property for each of these sets individually.Since the main di!erence among the three sets is theextracted region (or regions) that is to be analyzed, we"rst describe these regions.

LR: Let lx(i), ly(i) denote the left and rx(i), ry(i) the rightcoordinates for the ith LR pair. Then the extracted regione��is the region with left and right boundaries at lx(i) and

rx(i), respectively, and upper and lower boundaries deter-mined by the vertical scale of the eye (determined by theaspect ratio of the eye) centered on the horizontal linejoining the mean values of ly(i) and ry(i). The aspect ratiois the mean ratio between the horizontal and verticalextents of the eye. We use the ratio of 3 : 1 in favor of thehorizontal axis.

LL: Let llx(i), lly(i) be the left and lrx(i), lry(i) the rightcoordinates for the ith LL pair. In this case we expect aneye to be to the right of each of these points, and we usethe estimated orbit size o( b to determine the size of theextracted region. Let e

��(l) and e

��(r) be the extracted

regions for the left and right eyes, respectively. The leftboundary for e

��is llx(i) while the right boundary is

determined by using o( b. For the upper and lower bound-aries, we center the eye on the horizontal line, take themean of lly(i) and lry(i), and then use the vertical extent ofthe eye (determined by o( b and the aspect ratio of the eye)to determine these boundaries. Similarly, for e

��(r) we use

lrx(i) as the left boundary and follow the proceduredescribed for e

��(l).

RR: Let rlx(i), rly(i) be the left and rrx(i), rry(i) the rightcoordinates for the ith RR pair. Here we expect the eyesto be to the left of each of these points. We select tworegions e

��(l) and e

��(l) in a symmetrical manner to the

regions extracted for the LL pairs.Once we have obtained the extracted regions, we per-

form the èyenessa test as described in Section 4.1. Forthe case of LR pairs this becomes a test of the singleregion e

��. After performing the test on all LR pairs we

keep only those pairs that pass the test. Fig. 22 showsonly those LR pairs that passed the èyenessa test.


Fig. 22. Veri"ed LR pairs of responses.

Fig. 23. Veri"ed LL and RR pairs of responses.

Similarly, we test the pairs of extracted regions for LLand RR pairs. Fig. 23 shows only those pairs of LL andRR responses that passed this test.We see from Figs. 22 and 23 that some false alarms still

remain. The next section will discuss a natural method ofcombining the paired LR, LL and RR responses.

4.2.2.3. Combining verixed pairs of responses. In theexample that we have been treating so far we have a leftand right corner detection for each eye, but we also havesome false alarms that need to be suppressed. Thisexample illustrates one reason for using corner detection"lters. In many cases we can use partial corner detectionresponses to make useful decisions. For instance, supposewe have both corner responses for the left eye but haveonly the left corner response for the right eye. After thepairing and veri"cation phase we get an LR pair and anLL pair. Individually these pairings do not provide verystrong evidence for detecting a pair of eyes. However, ifwe combine the two responses, we arrive at strongersupport for the hypothesis of a pair of eyes.Using this rationale we have developed a weighted

decision procedure that depends on the number of pairsthat contribute to the eye hypothesis. The procedure isbased on grouping pairs of responses; it results in a non-linear decision space. This space is divided into sevenregions which we number 0, 1, 1.5, 2, 3, 3.5 and 4, and

denote by d�, d

�, d

��, d

�, d

�, d

��and d

. These regions

are de"ned as follows:

d�

No LR, LL or RR pairs are present in the image.d�

One LR pair is present.d��

Either one LL pair or one RR pair is present.d�

Two LR pairs or an LL pair and an RR pair arepresent, conforming to the criteria for a pair of eyes.

d�

One LR pair, coinciding with either an LL pairor an RR pair, is present.

d��

Two LR pairs are supported by either one LL pairor one RR pair, or an LL pair and an RR pair aresupported by one LR pair.

d

Two LR pairs are supported by one LL pair andone RR pair.

It should be noted that the subscripts for the decisionregions are numerically in the same orders as the degreesof con"dence we would assign to the decisions.The decision is reached in two stages. The "rst stage

pairs up LR pairs and LL and RR pairs. For the case ofLR pairing, let k and m represent the indices of two LRpairs. k and m will form a (k,m) pair of pairs if

1. lx(k)+lx(m) and rx(k)+rx(m),2. ly(k)(ly(m) and ry(k)(ry(m), and ry(k)(ly(m).

The "rst condition ensures that the two pairs are in anapproximate straight horizontal line. The second condi-tion enforces an ordering requirement on the corners ofthe two eyes. This condition ensures that the left cornerand right corner of the left eye and the left corner andright corner of the right eye occur in that order, from leftto right. When both of these conditions are satis"ed wesay that (k,m) forms a pair of LR pairs.For the case of the pth LL pair and the nth RR pair we

look for the following conditions:

1. llx(p)+rlx(n) and lrx(p)+rrx(n),2. lly(p)(rly(n) and lry(p)(rry(n), and rly(n)(lry(p).

If these conditions are satis"ed then we say that the pthLL pair and the nth RR pair form a (p, n) pair of pairs.The second stage looks for those pairs of pairs which

are supported by other pairs of pairs. This means that ifwe have a (k,m) pair of LR pairs and a (p, n) pair of LLand RR pairs in which the corners are the same, i.e.,(llx(p), lly(p))"(lx(k), ly(k)) and (lrx(p), lry(p))"(lx(m),ly(m)) and (rlx(n), rly(n))"(rx(k), ry(k)) and (rrx(n),rry(n))"(rx(m), ry(m), then we have full support for thepair of pairs, with the result falling in the decision regiond. An example of d

is shown in Fig. 24(d).

The decision region d��

is obtained when we havea (k,m) pair of LR pairs which is supported by one pair ofeither LL or RR, or we have a (p,m) pair of LL and RRpairs which is supported by one LR pair. Fig. 24(c) is anexample of this decision.


Fig. 24. Detection of eyes according to con"dence measure.

d�is the decision region where we do not have a pair of

pairs of either kind. Here we combine LR pairs witheither LL pairs or RR pairs. This decision is reachedwhen we encounter only three corners for the two eyes.For instance, if we have the left and right corners of theleft eye and an LL pair, then we are in decision region d

�.

For this decision region we can de"ne a binary 4-tuplewhich indicates the missing corner; in the example justgiven, the binary string 1110 indicates the absence of theright corner of the right eye. Fig. 24(b) shows an instanceof this decision with the binary string 1011.The decision region d

�indicates that a pair of pairs has

been found but there is no cross-support. This indicatesthat either we have a (k,m) pair of LR pairs or we havea (p, n) pair of LL and RR pairs.

d��

is the presence of one LL or one RR pair, indicat-ing a possible match for two eyes. d

�is the presence of

only one LR pair, indicating one eye's presence, as shownin Fig. 24(a). d

�is the null decision region in which no

eyes have been found.

4.2.3. Experimental resultsOur eye corner detection algorithm was tested on the

same two sets of images as in Section 4.1. In the Aberdeendatabase we set the radius of the "lter at 17 pixelsand kept it constant, as described in Section 4.2.1.The normalized and rescaled #esh-tone region D

��and

eye-tone region D��

were as follows:

D��

"(r, g)339)r)48, 30)g)35, (13)

D��

"(r, g)332)r)38, 30)g)36. (14)

For pairing responses, d��, d

��and d

��were determined by

the scale selection method, while the rotation toleranceswere set at

��"103 and

��"

��"73.

Tables 4 and 5 shown the results of all the steps inapplying non-linear "ltering and post-processing to theAberdeen database. The "rst column gives the imagenumber; the second and third columns give the numbersof right corner and left corner responses, respectively; thefourth, "fth and sixth columns give the numbers of LR,LL and RR pairings, respectively. The `Veri"ed LR andLL/RRa columns give the numbers of these pairs thatpass the èyenessa criterion. The `Finala column indi-cates the "nal number of points obtained by combiningthe veri"ed pairs of responses, and the `da column givesthe decision region that the image lies in; this indicatesthe level of con"dence.In the introduction to this section we alluded to the

fact that detecting left and right corners of the eye pro-vides rich information for the subsequent post-process-ing. As can be seen in Tables 4 and 5, we can make usefuldecisions even when all four corners of the two eyes arenot detected. An example is provided by the decisionregion d

�obtained for image ìm14a by combining one

LL pair with one LR pair. The binary string associatedwith this decision was 1011. The non-linear "ltering pro-cess failed to "nd the right corner of the left eye; this isseen from the binary string associated with the decision.This additional information is lacking in the linear "lter-ing method and is a contributing factor in the betterperformance of the non-linear "ltering method.When we applied the non-linear "ltering procedure to

the frame sequence the radius for the "lter was set at 11.The parameter values for D

��and D

��are also di!er-

ent; they are

D��

"(r, g)346)r)55, 29)g)34, (15)

D��

"(r, g)335)r)44, 30)g)36. (16)

For pairing responses, d��, d

��and d

��were determined by

the scale-selection procedure and the values of ��,

��and

��were the same as the ones used in the "rst dataset.Table 6 gives the results of all the intermediate steps in

applying non-linear "ltering and post-processing to theframe sequence, and Fig. 25 shows the 21 frames withthe "nal responses (having con"dence levels 1.5 orgreater) marked by dots. In this sequence the subject ismoving her head as well as her eyes. Analyzing the resultsshows an evolving change in the number of detectedcorners as the iris is rolled from one side to the other.From the detections observed in the latter part of thesequence (corresponding to images ìm17a}ìm21a in


Table 4Intermediate and "nal results of non-linear "ltering for images 1}36

Image no. Right Left LR LL RR Pass LR Pass LL/RR Final d

im1 10 7 3 1 1 2 0 4 2.0im2 13 12 2 2 1 2 2 4 4.0im3 15 14 3 2 2 2 3 4 4.0im4 9 10 5 5 2 4 3 4 4.0im5 12 12 3 1 4 2 3 4 4.0im6 23 16 4 2 8 2 2 4 4.0im7 24 19 5 4 10 2 2 4 4.0im8 22 18 5 4 3 3 2 4 4.0im9 19 17 5 2 3 2 2 4 4.0im10 8 8 4 2 2 2 2 4 4.0im11 9 10 4 1 1 2 2 4 4.0im12 9 9 4 2 1 2 2 4 4.0im13 3 4 2 1 1 2 2 4 4.0im14 3 4 1 1 1 1 1 3 3.0im15 6 13 2 3 1 2 2 4 4.0im16 6 13 2 1 1 2 2 4 4.0im17 8 17 3 5 3 2 2 4 4.0im18 12 20 8 4 1 2 2 4 4.0im19 10 15 3 1 1 2 2 4 4.0im20 11 7 2 1 1 2 2 4 4.0im21 6 11 2 1 1 2 2 4 4.0im22 7 10 4 1 1 2 2 4 4.0im23 6 9 1 0 1 1 0 2 1.0im24 9 10 2 0 3 2 1 4 3.5im25 4 6 2 1 1 2 2 4 4.0im26 9 4 2 1 1 2 2 4 4.0im27 9 5 2 1 1 2 2 4 4.0im28 18 11 4 1 4 1 0 2 1.0im29 29 44 12 15 4 4 2 4 4.0im30 24 16 7 2 2 3 3 4 4.0im31 25 22 9 1 7 2 2 4 4.0im32 5 6 2 1 1 2 2 4 4.0im33 7 8 2 1 1 2 2 4 4.0im34 10 13 3 1 1 2 2 4 4.0im35 5 7 3 0 0 0 0 0 0.0im36 8 6 2 1 1 2 2 4 4.0

Table 6), it is evident that as the irises move towards theleft corners of their respective eyes we are able to detectthe opposing (right) corners. If we consider this as a singleimage the con"dence measure would be very low, 1.5;however, considering each image as part of the sequence,we are able to reach valuable conclusions about themovement of the iris. Such qualitative analyses are ofgreat importance when tracking of eye gaze direction isrequired.

5. Conclusions

Our method of eye corner detection involves nonlinear"ltering using color-based wedge-shaped "lters. This ap-

proach proved to be superior to the one using Gaborwavelet-based linear "ltering due to the fact that it wasnot sensitive to squint, and it also provided rich informa-tion, in the form of right and left corner responses, forpost-processing. For the Aberdeen database a detectionrate of 90% was achieved with no false alarms, whensetting the con"dence threshold at 2. We found that mostof the failures were due to incorrect estimation of theorbit size from the face blob. For the frame sequence weset the con"dence threshold at 1.5 and achieved a detec-tion rate of 90% with no false alarms. We used a lowercon"dence threshold in this sequence to allow for the eyegaze variations. As we see from Fig. 25, when the iris isshifted to either the right or left corner of the eye, only theopposite corner is detected.


Table 5Intermediate and "nal results of non-linear "ltering for images 37}72


im37 6 4 0 1 1 0 1 2 1.5im38 4 9 2 2 1 2 1 4 3.5im39 6 16 3 2 1 1 2 4 3.5im40 6 6 2 1 1 2 1 4 3.5im41 7 10 2 2 1 1 1 3 3.0im42 7 11 2 1 1 2 1 4 3.5im43 4 12 2 1 1 2 2 4 4.0im44 14 18 5 3 5 2 1 4 3.5im45 12 6 2 1 1 2 2 4 4.0im46 5 11 3 2 1 2 2 4 4.0im47 10 13 4 2 3 2 1 4 3.5im48 23 13 7 2 12 2 2 4 4.0im49 9 10 4 1 2 2 2 4 4.0im50 10 9 2 1 1 2 2 4 4.0im51 10 14 2 2 1 2 2 4 4.0im52 7 5 3 1 1 2 2 4 4.0im53 4 4 2 1 1 2 2 4 4.0im54 13 9 4 2 3 2 2 4 4.0im55 27 28 10 7 9 2 0 0 0.0im56 24 23 9 6 6 3 3 4 4.0im57 23 24 9 6 8 3 3 4 4.0im58 13 21 4 1 4 1 0 2 1.0im59 11 9 5 2 3 1 2 4 3.5im60 17 11 4 4 4 2 2 4 4.0im61 7 12 3 3 1 1 1 3 3.0im62 5 11 3 1 0 0 0 0 0.0im63 13 9 2 1 1 2 1 4 3.5im64 8 7 2 1 1 2 2 4 4.0im65 25 22 7 5 6 2 2 4 4.0im66 23 19 7 4 4 2 2 4 4.0im67 9 11 4 3 2 2 2 4 4.0im68 17 13 2 1 2 2 2 4 4.0im69 18 13 6 1 2 3 2 4 4.0im70 10 10 4 2 1 2 2 4 4.0im71 9 11 4 3 1 2 2 4 4.0im72 7 17 2 2 1 2 2 4 4.0

Table 6Intermediate and "nal results of non-linear "ltering for frames 1}21


im1 4 4 0 1 0 2 2 4 4.0im2 7 5 1 3 2 2 2 4 4.0im3 4 5 1 0 1 2 2 4 4.0im4 8 6 2 1 3 2 2 4 4.0im5 4 6 0 0 0 2 2 4 4.0im6 6 9 0 0 0 2 1 4 3.5im7 7 11 0 0 0 2 2 4 3.5im8 7 7 0 0 0 3 1 0 0.0im9 9 10 3 0 0 2 2 4 4.0im10 8 6 2 0 1 3 3 4 4.0im11 9 9 1 1 3 3 3 4 4.0im12 9 9 2 1 0 1 1 3 3.0im13 8 10 0 0 0 1 1 3 3.0im14 7 10 1 0 0 0 1 2 1.5im15 7 8 1 1 1 1 1 3 3.0im16 9 7 2 1 5 1 2 0 0.0im17 10 4 1 1 6 0 1 2 1.5im18 11 7 3 0 6 1 1 3 3.0im19 5 4 0 1 2 0 1 2 1.5im20 6 2 0 0 2 0 1 2 1.5im21 4 2 0 0 0 0 1 2 1.5


Fig. 25. Final results of the non-linear "ltering method on theframe sequence.

In summary, the non-linear "ltering method had ahigher level of performance in detecting the eyes thantraditional linear "ltering approaches. As a by-product,detection of the corners of the eyes provides landmarkpoints that can be used in eye gaze tracking.

Acknowledgements

The support of the O$ce of Naval Research underGrant N00014-95-1-0521 is greatfully acknowledged.The authors also thank Prof. Rama Chellappa for hisguidance in the development of the linear "lteringmethod.

References

[1] R. Chellappa, C.L. Wilson, S. Sirohey, Human and ma-chine recognition of faces: a survey, Proc. IEEE 83 (1995)705}740.

[2] T. Sakai, M. Nagao, S. Fujibayashi, Line extraction andpattern recognition in a photograph, Pattern Recognition1 (1969) 233}248.

[3] M.D. Kelly, Visual identi"cation of people by computer,Technical Report AI-130, Stanford University, Stanford,CA, 1970.

[4] V. Govindaraju, S.N. Srihari, D.B. Sher, A computationalmodel for face location, Proceedings, Third InternationalConference on Computer Vision, 1990, pp. 718}721.

[5] A. Yuille, D. Cohen, P. Hallinan, Feature extraction fromfaces using deformable templates, Proceedings, IEEEComputer Society Conference on Computer Vision andPattern Recognition, 1989, pp. 104}109.

[6] I. Craw, H. Ellis, J. Lishman, Automatic extraction of facefeatures, Pattern Recognition Lett. 5 (1987) 183}187.

[7] P.J. Burt, Multiresolution techniques for image repres-entation, analysis, and &smart' transmission, SPIE Pro-ceedings, Vol. 1199: Visual Communications and ImageProcessing IV, 1989, pp. 2}15.

[8] I. Craw, D. Tock, A. Bennett, Finding face features,Proceedings, Second European Conference on ComputerVision, 1992, pp. 92}96.

[9] J.W. Shepherd, An interactive computer system for retriev-ing faces, in: H.D. Ellis, M.A. Jeeves, F. Newcombe, A.Young (Eds.), Aspects of Face Processing, Nijho!, Dor-drecht, 1986, pp. 398}409.

[10] U. Grenander, Y. Chow, D. Keenan, Hands: A PatternTheoretic Study of Biological Shapes, Springer, NewYork,1991.

[11] S. Geman, D. Geman, Stochastic relaxation, Gibbs distribu-tion and Bayesian restoration of images, IEEE Trans. Pat-tern Anal. Mach. Intell. 6 (1984) 721}741.

[12] S.A. Sirohey, Human face segmentation and identi"cation,Technical Report CAR-TR-695, Center for AutomationResearch, University ofMaryland, College Park, MD, 1993.

[13] J. Canny, A computational approach to edge detection,IEEE Trans. Pattern Anal. Mach. Intell. 8 (1986) 679}689.

[14] K.C. Yow, R. Cipolla, Feature-based human face detection,Technical Report CUED/F-INFENG/TR 249,Department of Engineering, University of Cambridge,Cambridge, England, 1996.

[15] A.J. Colmenarez, T.S. Huang, Face detection with informa-tion-based maximum discrimination, Proceedings, IEEEComputer Society Conference on Computer Vision andPattern Recognition, 1997, pp. 782}787.

[16] H.A. Rowley, S. Baluja, T. Kanade, Neural network-basedface detection, IEEE Trans. Pattern Anal. Mach. Intell. 20(1998) 23}38.

[17] K.K. Sung, T. Poggio, Example-based learning for view-based human face detection, IEEE Trans. Pattern Anal.Mach. Intell. 20 (1998) 39}51.

[18] E. Osuna, R. Freund, F. Girosi, Training support vectormachines: An application to face detection, Proceedings,IEEE Computer Society Conference on Computer Visionand Pattern Recognition, 1997, pp. 130}136.

[19] V. Vapnik, The Nature of Statistical Learning Theory,Springer, New York, 1995.

[20] H.A. Rowley, S. Baluja, T. Kanade, Rotation invariantneural network-based face detection, Proceedings, IEEEComputer Society Conference on Computer Vision andPattern Recognition, 1998, pp. 38}44.


[21] H. Hunke, Locating and tracking human faces with neuralnetworks, Technical Report CMU-CS-94-155, Carnegie-Mellon University, Pittsburgh, PA, 1994.

[22] N. Oliver, A.P. Pentland, F. Bernard, Lafter: lips and facereal time tracker, Proceedings, IEEE Computer SocietyConference on Computer Vision and Pattern Recognition,1997, pp. 123}129.

[23] X. Zhang, C. Podilchuk, Face location with template-basedinvariants, Proceedings, IEEE Workshop on Image andMultidimensional Signal Processing, 1996, pp. 16}17.

[24] M. Nixon, Eye spacing measurement for facial recognition,SPIE Proceedings, Vol. 575: Applications of Digital ImageProcessing VIII, 1985, pp. 279}285.

[25] P.W.Hallinan, Recognizing human eyes, SPIE Proceedings,Vol. 1570: Geometric Methods in Computer Vision, 1991,pp. 214}226.

[26] D. Reisfeld, Y. Yeshurun, Robust detection of facial featuresby generalized symmetry, Proceedings, International Con-ference on Pattern Recognition, 1992, pp. 117}120.

[27] R. Baron, Mechanisms of human facial recognition, Int. J.Man-Mach. Stud. 15 (1981) 137}178.

[28] A. Pentland, B. Moghaddam, T. Starner, M. Turk, View-based and modular eigenspaces for face recognition, Pro-ceedings, IEEEComputer Society Conference on ComputerVision and Pattern Recognition, 1994, pp. 84}91.

[29] R. Herpers, M. Michaelis, K.H. Lichtenauer, G. Sommer,Edge and keypoint detection in facial regions, Proceedings,Second International Conference on Automatic Face andGesture Recognition, 1996, pp. 212}217.

[30] R. Duda, P. Hart, Pattern Classi"cation and Scene Analysis,Wiley, New York, 1973.

[31] L.G. Farkas, T.A. Hreczko, M.J. Katic, Craniofacial normsin North American caucasians from birth (one year) to

young adulthood, in: L.G. Farkas (Ed.), Anthropometry ofthe Head and Face, 2nd Edition, Raven Press, New York,1994, pp. 241}336.

[32] S. Sirohey, A. Rosenfeld, Eye detection, Technical ReportCAR-TR-896, Center for Automation Research, Universityof Maryland, College Park, MD, 1998.

[33] M. Lades, J.C. Vorbruggen, J. Buhmann, J. Lange, C.v.d.Malsburg, R.P. Wurtz, Distortion invariant object recogni-tion in the dynamic link architecture, IEEE Trans. Comput.42 (1993) 300}311.

[34] B.S. Manjunath, R. Chellappa, V. Shekhar, C.v.d. Malsburg,A robust method for detecting image features with applica-tion to face recognition and motion correspondence, Pro-ceedings, International Conference on Pattern Recognition,1992, pp. 208}212.

[35] M. Michaelis, G. Sommer, Junction classi"cation by mul-tiple orientation detection, Proceedings, European Confer-ence on Computer Vision, Vol. I, 1994, pp. 101}108.

[36] J.A. Noble, Finding corners, Image Vision Comput. 6 (1988)121}128.

[37] K. Rohr, Recognizing corners by "tting parametric models,Int. J. Comput. Vision 9 (1992) 213}230.

[38] K. Rohr, On the precision in estimating the location ofedges and corners, J. Math. Imaging Vision 7 (1997) 7}22.

[39] W. Yu, K. Daniilidis, G. Sommer, Low-cost junction charac-terization using polar averaging "lters, Proceedings, Interna-tional Conference on Image Processing, 1998, pp. 41}44.

[40] W. Yu, K. Daniilidis, G. Sommer, Rotated wedge aver-aging method for junction characterization, Proceedings,IEEE Computer Society Conference on Computer Visionand Pattern Recognition, 1998, pp. 390}395.

[41] A. Rosenfeld, A.C. Kak, Digital Picture Processing, Vol. 2,Academic Press, New York, 1982.

About the Author*SAAD A. SIROHEY received his B.Sc. with highest honors in electrical engineering from King Fahd University ofPetroleum andMinerals, Dhahran, Saudi Arabia in 1990, and hisMS and Ph.D. degrees from the University ofMaryland, College Park,also in electrical engineering. He is currently working at General Electric Medical Systems in medical imaging. His research interests aresignal/image processing and image understanding. He has done research in automated face detection/recognition and tracking withemphasis on eye gaze tracking. He is currently working on bio-medical imaging.

About the Author*AZRIEL ROSENFELD is a tenured Research Professor, a Distinguished University Professor, and Director of theCenter for Automation Research at the University of Maryland in College Park. He also holds a$liate professorships in theDepartments of Computer Science, Electrical Engineering, and Psychology. He holds a Ph.D. in mathematics from ColumbiaUniversity (1957), rabbinic ordination (1952) and a Doctor of Hebrew Literature degree (1955) from Yeshiva University, and honoraryDoctor of Technology degrees from LinkoK ping University, Sweden (1980) and Oulu University, Finland (1994).Dr. Rosenfeld is widely regarded as the leading researcher in the world in the "eld of computer image analysis. Over a period of 35

years he has made many fundamental and pioneering contributions to nearly every area of that "eld. He wrote the "rst textbook in the"eld (1969); was founding editor of its "rst journal (1972); and was co-chairman of its "rst international conference (1987). He haspublished over 30 books and over 600 book chapters and journal articles, and has directed over 50 Ph.D. dissertations. In 1985 he servedas chairman of a panel appointed by the National Research Council to brief the President's Science Advisor on the subject of computervision; he has also served (1985}1988) as a member of the Vision Committee of the National Research Council. In honor of his 65thbirthday, a book entitled `Advances in Image Understanding * A Festschrift for Azriel Rosenfelda, edited by Kevin Bowyer andNarendra Ahuja, was published by IEEE Computer Society Press in 1996.He is a Fellow of the Institute of Electrical and Electronics Engineers (1971), and won its Emanuel Piore Award in 1985; he is

a founding Fellow of the American Association for Arti"cial Intelligence (1990) and of the Association for ComputingMachinery (1993);he is a Fellow of the Washington Academy of Sciences (1988), and won its Mathematics and Computer Science Award in 1988; he wasa founding Director of the Machine Vision Association of the Society of Manufacturing Engineers (1985}1988), won its President'sAward in 1987 and is a certi"ed Manufacturing Engineer (1988); he was a founding member of the IEEE Computer Society's TechnicalCommittee on Pattern Analysis and Machine Intelligence (1965), served as its Chairman (1985}1987), and received the Society'sMeritorious Service Award in 1986, its Harry Goode Memorial Award in 1995, and became a Golden Core member of the Society in


1996; he received the IEEE Systems, Man, and Cybernetics Society's Norbert Wiener Award in 1995; he received an IEEE StandardsMedallion in 1990, and the Electronic Imaging International Imager of the Year Award in 1991; he was a founding member of theGoverning Board of the International Association for Pattern Recognition (1978}1985), served as its President (1980}1982), won its "rstK.S. Fu Award in 1988, and became one of its founding Fellows in 1994; he received the Information Science Award from theAssociation for IntelligentMachinery in 1998; he was a ForeignMember of the Academy of Science of the GermanDemocratic Republic(1988}1992), and is a Corresponding Member of the National Academy of Engineering of Mexico (1982).


eyedetectioninafaceimageusinglinearandnonlinear...

Documents