0018-9162/00/$10.00 © 2000 IEEE50 Computer

Face Recognitionfor SmartEnvironments

Smart environments, wearable computers, andubiquitous computing in general are the com-ing “fourth generation” of computing andinformation technology.1-3 These devices willbe everywhere—clothes, home, car, and

office—and their economic impact and cultural sig-nificance will dwarf those of the first three generations.At the very least, they represent some of the most excit-ing and economically important research areas ininformation technology and computer science.

But before this new generation of computers can bewidely deployed, those working on interfaces mustinvent new methods of interaction without a keyboardor mouse. To win wide consumer acceptance, these inter-actions must be friendly and personalized, which impliesthat next-generation interfaces will be aware of the peo-ple in their immediate environment and, at a minimum,know who they are.

MEANS OF IDENTIFICATIONGiven the requirement for determining people’s iden-

tities, the obvious question is, what technology is best?A wide variety of identification technologies are avail-able, and many have been in widespread commercialuse for years. The most common personal verificationand identification methods today are password/PIN(personal identification number) systems and token sys-tems (using tokens such as your driver’s license).Because such systems are vulnerable to forgery, theft,and lapses in users’ memories, biometric identificationsystems, which use pattern recognition techniques toidentify people by their physiological characteristics,are attracting considerable interest. Fingerprints are aclassic example of a biometric; newer technologiesinclude retina and iris recognition.

While they are appropriate for bank transactionsand entry into secure areas, such biometric technolo-

gies have the disadvantage of being intrusive, bothphysically and socially. They require users to positiontheir bodies relative to the sensor and then pause fora second to “declare” themselves. This pause-and-declare interaction is unlikely to change because of thefine-grained spatial sensing required. There is an “ora-cle-like” aspect to the interaction as well. Since peopledo not recognize each other by such things as retinascans, these types of identification feel intrusive.

The pause-and-declare interaction is useful in high-security applications (the interruption makes peoplesecurity conscious), but it is exactly the opposite of whatis required for a store that recognizes its best customers,an information kiosk that remembers you, or a housethat knows the people who live there. Face and voicerecognition have a natural place in these next-genera-tion smart environments. They are unobtrusive (theyrecognize at a distance without a pause-and-declareinteraction), they are usually passive (needing no specialelectromagnetic illumination), they do not restrict usermovement, and they are now both low power and inex-pensive. Perhaps most important, though, is the factthat humans identify other people by their faces andvoices and are likely to be comfortable with systemsthat use similar means of recognition.

ACHIEVING FACE RECOGNITIONTwenty years ago, the problem of face recognition

was considered among the hardest in artificial intelli-gence and computer vision. Surprisingly, however, overthe past decade, a series of successes have made gen-eral personal identification appear not only technicallyfeasible but also economically practical.

The apparent tractability of the face recognitionproblem, combined with the dream of smart environ-ments, has produced a huge surge of interest fromfunding agencies and from researchers themselves. It

Computers of the future will interact with us more like humans. A keyelement of that interaction will be their ability to recognize our faces andeven understand our expressions.

Alex (Sandy)PentlandTanzeemChoudhuryMIT MediaLaboratory

C O V E R F E A T U R E

has also spawned several thriving commercial enter-prises. There are now several companies that sell com-mercial face recognition software that is capable ofhigh-accuracy recognition with databases of morethan 1,000 people.

These early successes came from the combinationof well-established pattern recognition techniqueswith a fairly sophisticated understanding of the imagegeneration process. In addition, researchers realizedthat they could capitalize on regularities that are pecu-liar to people. For instance, human skin colors lie ona one-dimensional manifold in color space, with colorvariation primarily due to melanin concentration.Human facial geometry is limited and essentially two-dimensional when people are looking toward the cam-era. Today, researchers are working on relaxing someconstraints of existing face recognition algorithms toachieve robustness under changes due to lighting,aging, rotation in depth, and expression. They are alsostudying how to deal with variations in appearancedue to such things as facial hair, glasses, andmakeup—problems that already have partial solu-tions.

Typical representational frameworkThe dominant representational approach that has

evolved is descriptive rather than generative. Thisapproach uses training images to characterize the rangeof two-dimensional appearances of objects the systemmust recognize. Although researchers initially used verysimple modeling methods, they now principally char-acterize appearance by estimating the probability den-sity function (PDF) of the image data for the target class.

For instance, given several examples of a target classΩ—for example, faces—in a low-dimensional repre-sentation of the image data, it is straightforward tomodel the PDF P(x|Ω) of its image-level features x asa simple parametric function—a mixture of Gaussiandistribution functions—thus obtaining a low-dimen-sional, computationally efficient appearance modelfor the target class. In other words, we can use exam-ple face images to obtain a simple mathematical modelof facial appearance in image data.

Once we have learned the PDF of the target class, wecan use Bayes’ rule to perform maximum a posteriori(MAP) detection and recognition. The result is typicallya very simple, neural-net-like representation of the tar-get class’s appearance, which a system can use to detectoccurrences of the class, to compactly describe itsappearance, and to efficiently compare different exam-ples from the same class. Indeed, this representationalframework is so efficient that some current face recog-nition methods can process video data at 30 frames persecond. Several systems can compare an incoming faceto a database of thousands of people in under one sec-ond—and all on a standard PC!

Dealing with dimensionalityTo obtain an appearance-based representa-

tion, the image must first be transformed into alow-dimensional coordinate system that pre-serves the general perceptual quality of the tar-get object’s image. This transformation isnecessary to address the problem of dimen-sionality: The raw image data has so manydegrees of freedom that it would require mil-lions of examples to directly learn the range of appear-ances. Typical methods for reducing dimensionalityinclude

• Karhunen-Loève transform (also called principalcomponents analysis),

• Ritz approximation (also called example-basedrepresentation),

• Sparse-filter representations (for example, Gaborjets and wavelet transforms),

• Feature histograms, and• Independent-component analysis.

These methods all allow efficient characterization ofa low-dimensional subspace within the large space ofraw image measurements. Once you obtain a low-dimensional representation of the target class—face,eye, or hand—you can use standard statistical para-meter estimation methods to learn the range ofappearances that the target exhibits in the new, low-dimensional coordinate system. Because of the lowerdimensionality, obtaining a useful estimate of eitherthe PDF or the interclass discriminant functionrequires relatively few examples.

An important variation on this methodology is dis-criminative models, which attempt to model the differ-ences between classes rather than the classes themselves.Such models can often be learned more efficiently andaccurately than by directly modeling the PDF. A simplelinear example of such a difference feature is the Fisherdiscriminant. Systems can also employ discriminantclassifiers, such as support vector machines, whichattempt to maximize the margin between classes.

FACE RECOGNITION EFFORTSThe subject of face recognition is as old as computer

vision because of the topic’s practical importance andtheoretical interest from cognitive scientists. Despitethe fact that other identification methods (such as fin-gerprints or iris scans) can be more accurate, facerecognition has always been a major research focusbecause it is noninvasive and seems natural and intu-itive to users.

Perhaps the most famous early example of a facerecognition system is that of Teuvo Kohonen of theHelsinki University of Technology,4 who demon-strated that a simple neural net could perform face

February 2000 51

Face recognitioncapitalizes on

regularities that are peculiar to humans.

52 Computer

recognition for aligned and normalized images offaces. The network he employed computed a facedescription by approximating the eigenvectors of theimage’s autocorrelation matrix. These eigenvectorsare now known as eigenfaces.

Kohonen’s system was not a practical success, how-ever, because it relied on precise alignment and nor-malization. In the following years, many researcherstried face recognition schemes based on edges, inter-feature distances, and other neural-net approaches.While several were successful with small databases ofaligned images, none successfully addressed the morerealistic problem of large databases where the face’slocation and scale are unknown.

Michael Kirby and Lawrence Sirovich of BrownUniversity5 later introduced an algebraic manipula-tion that made it easy to directly calculate the eigen-faces. They also showed that it required fewer than100 eigenfaces to accurately code carefully alignedand normalized face images. Matthew Turk and AlexPentland of MIT6 then demonstrated that the resid-ual error when coding with the eigenfaces could beused to detect faces in cluttered natural imagery andto determine the precise location and scale of faces inan image. They then demonstrated that coupling thismethod for detecting and localizing faces with theeigenface recognition method could achieve reliable,real-time recognition of faces in a minimally con-strained environment. This demonstration that a com-bination of simple, real-time pattern recognitiontechniques could create a useful system sparked anexplosion of interest in face recognition.

Current workBy 1993, researchers claimed that several algorithms

provided accurate performance in minimally con-strained environments. To better understand the poten-tial of these algorithms, the US Defense Ad-vanced

Research Projects Agency and the US Army ResearchLaboratory established the Feret (face recognition tech-nology) program with the goals of evaluating their per-formance and encouraging advances in the technology.7

Feret identified three algorithms that demonstratedthe highest level of recognition accuracy with largedatabases (1,196 people or more) under double-blindtesting conditions: those of the University of SouthernCalifornia (USC), illustrated in Figure 1;8 the Universityof Maryland (UMD);9 and the Massachusetts Instituteof Technology (MIT) Media Laboratory, illustrated inFigure 2.10 Only two algorithms, those from USC andMIT, are capable of both minimally constrained detec-tion and recognition; the UMD system requiresapproximate eye locations to operate. RockefellerUniversity developed a fourth algorithm, illustrated inFigure 3, that was an early contender, but it was with-drawn from testing to form a commercial enterprise.11

The MIT and USC algorithms have also become thebasis for commercial systems.

The MIT, Rockefeller, and UMD algorithms all useversions of the eigenface transform followed by dis-criminative modeling. The UMD algorithm uses a lin-ear discriminant, while the MIT system employs aquadratic discriminant. The Rockefeller system uses asparse version of the eigenface transform followed bya discriminative neural network. The USC system, incontrast, takes a very different approach. It begins bycomputing Gabor jets from the image and then doesa flexible template comparison of image descriptionsusing a graph-matching algorithm.

The Feret database testing employs faces with vari-able positions, scales, and lighting in a manner consis-tent with mug shot or driver’s license photography.With databases of fewer than 200 people and imagestaken under similar conditions, all four algorithms per-form nearly perfectly. Interestingly, even simple corre-lation matching can sometimes achieve similar accuracyfor databases of only 200 people.7 This is strong evi-dence that any new algorithm should be tested withdatabases of at least 200 individuals and should achieveperformance over 95 percent on mug-shot-like imagesto be considered potentially competitive.

In the larger Feret testing (with 1,196 or moreimages), the performance of the four algorithms aresimilar enough that it is difficult or impossible to makemeaningful distinctions among them (especially ifadjustments for date of testing are made). With frontalimages taken on the same day, the typical first-choicerecognition performance is 95 percent accuracy. Forimages taken with different cameras and lighting, typ-ical performance drops to 80 percent accuracy. Forimages taken one year later, the typical accuracy isapproximately 50 percent. Still, it should be noted thateven 50 percent accuracy is hundreds of times chanceperformance.

Figure 1. USC’s system uses elastic graph matching for face recognition. It creates aface bunch graph from 70 face models to obtain a general representation called (a) anobject-adapted grid. The system then (b) matches a given image to the face bunchgraph to find the fiducial points. It creates an image graph using elastic graph matchingand then compares that image to a database of faces for recognition.

(a) (b)

February 2000 53

−k

Ω

|| ∆ ||

–

gp

S(p,g)

| ∆)P(ΩIP(∆ | )

IΩ

S(p,g)

ΩE

ΩI

g

–

p

U

Figure 2. MIT’s system of using eigenfaces for face recognition relies on (a) appearance and (b) discriminative models.

Figure 3. Rockefeller University’s system of using local feature analysis for face recognition. The parts marked on the image to the left correspond toreceptive fields for the (a) mouth, (b) nose, (c) eyebrow, (d) jawline, and (e) cheekbone. (Reprinted with permission of NYT Pictures)

1. The system collects a database of faceimages.

2. It generates a set of eigenfaces by per-forming principal component analysis(PCA) on the face images. Approxi-mately 100 eigenvectors are enough tocode a large database of faces.

3. The system then represents each faceimage as a linear combination of theeigenfaces.

4. Given a test image, the system approx-imates it as a combination of eigen-faces. A distance measure indicates thesimilarity between two images.

1. The system obtains data sets ΩI and ΩE bycomputing intrapersonal differences(matching two views of each individual inthe data set) and by computing extraper-sonal differences (matching different indi-viduals in the data set).

2. It generates two sets of eigenfaces by per-forming PCA on each class.

3. The system derives a similarity scorebetween two images by calculating S =P(ΩI|∆), where ∆ is the difference betweena pair of images. If S is less than 0.5, thesystem considers the two images to be ofthe same individual.

(a) (b)

(a) (b) (c) (d) (e)

54 Computer

Commercial systems and applicationsSeveral face recognition products are com-

mercially available. Algorithms developed by thetop contenders in the Feret competition are thebases for some available systems; others weredeveloped outside of the Feret testing frame-work. While it is extremely difficult to judge,three systems—from Visionics, Viisage, andMiros—seem to be the current market leaders.

• Visionics’ FaceIt software is based on thelocal feature analysis algorithm developed at

Rockefeller University. A commercial companyin the UK is incorporating FaceIt into a closed-circuit television anticrime system calledMandrake. This system searches for known crim-inals in video data acquired from 144 closed-cir-cuit camera locations. When a match occurs, thesystem notifies a security officer in the controlroom.

• Viisage, another leading face recognition com-pany, uses the eigenface-based recognition algo-rithm developed at the MIT Media Laboratory.Companies and government agencies in many USstates and several developing nations use Viisage’ssystem in conjunction with identification cards—for example, driver’s licenses and similar gov-ernment ID cards.

• Miros uses neural network technology for itsTrueFace face recognition software. TrueFace isused by the Mr. Payroll Corp. system in its checkcashing system and has been deployed at casinosand similar sites in many US states.

NOVEL APPLICATIONSFace recognition systems are no longer limited to

identity verification and surveillance tasks. Growingnumbers of applications are using face recognition asthe initial step toward interpreting human actions,intentions, and behavior as a central part of next-gen-eration smart environments. Many actions and behav-iors humans display can only be interpreted if you alsoknow the identities of the individuals and the peoplearound them. Examples are a valued repeat customerentering a store, behavior monitoring in an elder careor child care facility, and command-and-control inter-faces in a military or an industrial setting. In each ofthese applications, identity information is crucial toproviding machines with the background knowledgeneeded to interpret measurements and observationsof human actions.

Face recognition for smart environmentsResearchers today are actively building smart envi-

ronments—visual, audio, and haptic interfaces withenvironments such as rooms, cars, and office desks.1,2

In these applications, a key goal is to give machinesperceptual abilities that allow them to function natu-rally with people—to recognize people and remembertheir preferences and peculiarities, to know what theyare looking at, and to interpret their words, gestures,and unconscious cues, such as vocal prosody and bodylanguage. Researchers are exploring applications forthese perceptually aware devices in health care, enter-tainment, and collaborative work.

Facial-expression recognition interacts with othersmart-environment capabilities. For example, a smartsystem should know whether the user looks impa-tient—because information is being presented tooslowly—or confused because it is coming too fast.Facial expressions provide cues for identifying anddistinguishing between these states. Recently, mucheffort has gone into creating a person-independentexpression recognition capability. While there are sim-ilarities in expressions across cultures and betweenpeople, for anything but the most gross facial expres-sions, the analysis must account for the person’s nor-mal resting facial state—something that definitely isn’tthe same between people. Consequently, facial-expres-sion research has so far been limited to recognition ofa few discrete expressions rather than addressing theentire spectrum of expressions along with their subtlevariations. Before a system can achieve a really usefulexpression analysis capability, it must first be able torecognize and tune its parameters to a specific person.

Wearable recognition systemsWhen we build computers, cameras, microphones,

and other sensors into a person’s clothes, the com-puter’s view moves from a passive third-person to anactive first-person vantage point.3 These wearabledevices can adapt to a specific user and be more inti-mately and actively involved in the user’s activities. Thewearable-computing field is rapidly expanding and justrecently became a full-fledged technical committeewithin the IEEE Computer Society. Consequently, wecan expect to see rapidly growing interest in the largelyunexplored area of first-person image interpretation.

Face recognition is an integral part of wearable sys-tems like memory aids and context-aware systems.Thus, developers will integrate many future recogni-tion systems with clothing and accessories. Forinstance, if you build a camera into your eyeglasses,then face recognition software can help you rememberthe name of the person you are looking at by whis-pering it in your ear. The US Army has started testingsuch devices for use by border guards in Bosnia, andresearchers at the University of Rochester’s Center for Future Health are looking at them for patients with Alzheimer’s disease (see http://wearables.www.media.mit.edu/projects/wearables and http://www.futurehealth.rochester.edu).

Growing numbers of applications are using face

recognition as theinitial step towardinterpreting human

actions.

FUTURE WORKToday’s face recognition systems work very well

under constrained conditions, such as frontal mugshot images and consistent lighting. All current facerecognition algorithms fail under the vastly varyingconditions in which humans can and must identifyother people. Next-generation recognition systemswill need to recognize people in real time and in muchless constrained situations.

We believe that identification systems that arerobust in natural environments—in the presence ofnoise and illumination changes—cannot rely on a sin-gle modality; thus, fusion with other modalities isessential. Technology used in smart environments hasto be unobtrusive and allow users to act freely.Wearable systems in particular require the sensingtechnology to be small, low power, and easily inte-grable with clothing. Considering all the requirements,systems that use face and voice identification seem tohave the most potential for widespread application.

Cameras and microphones today are very small andlightweight, and have been successfully integrated withwearable systems. Audio- and video-based recognitionsystems have the critical advantage of using the modal-ities humans use for recognition. Finally, researchersare beginning to demonstrate that unobtrusive audio-and video-based personal identification systems canachieve high recognition rates without requiring theuser to be in a highly controlled environment.12

The goal of smart environments is to create aspace where computers and machines are morelike helpful assistants, rather than inanimate

objects. Face recognition technology could play amajor part in achieving this goal, and it has come along way in the past 20 years. But to achieve the goalof widespread application in smart environments,next-generation face recognition systems will have tofit naturally within the pattern of normal humaninteractions and conform to human intuitions aboutwhen recognition is likely. This implies that futuresmart environments should use the same modalitiesas humans and have approximately the same limita-tions. Although substantial research remains to bedone, these goals now appear to be within reach.

References1. M. Weiser, “The Computer for the 21st Century,”

Scientific American, Mar. 1991, pp. 66–76.2. A. Pentland, “Smart Rooms, Smart Clothes,” Scientific

American, Apr. 1996, pp. 68–76.3. A. Pentland, “Wearable Intelligence,” Scientific Ameri-

can Presents, Apr. 1998, pp. 90–95.4. T. Kohonen, Self-Organization and Associative Mem-

ory, Springer-Verlag, Berlin, 1989.

5. M. Kirby and L. Sirovich, “Application of the Karhunen-Loeve Procedure for the Characterization of HumanFaces,” Trans. IEEE Pattern Analysis and MachineIntelligence, Jan. 1990, pp. 103–108.

6. M. Turk and A. Pentland, “Eigenfaces for Recognition,”J. Cog. Neuroscience, Jan. 1991, pp. 71–86.

7. P. Phillips et al., “The Feret Database and EvaluationProcedure for Face Recognition Algorithms,” Image andVision Computing, May 1998, pp. 295–306.

8. L. Wiskott et al., “Face Recognition by Elastic BunchGraph Matching,” Trans. IEEE Pattern Analysis andMachine Intelligence, July 1997, pp. 775–779.

9. K. Etemad and R. Chellappa, “Discriminant Analysisfor Recognition of Human Face Images,” J. Optical Soc.of America, pp. 1724–1733.

10. B. Moghaddam and A. Pentland, “Probabilistic VisualRecognition for Object Recognition,” Trans. IEEE Pat-tern Analysis and Machine Intelligence, July 1997, pp.696–710.

11. P. Penev and J. Atick, “Local Feature Analysis: A Gen-eral Statistical Theory for Object Representation,” Net-work: Computation in Neural Systems, Mar. 1996, pp.477–500.

12. T. Choudhury et al., “Multimodal Person RecognitionUsing Unconstrained Audio and Video,” Proc. 2nd Conf.Audio- and Video-Based Biometric Person Authen-tication, Univ. of Maryland, College Park, Md., 1999,pp. 176–181.

Alex (Sandy) Pentland is academic head of the MITMedia Laboratory, Toshiba professor, and codirectorof the Center for Future Health. He is interested inthe development of new technologies that empowerpeople, including those from developing nations. Pent-land has a PhD from MIT. He is a cofounder of theIEEE Face and Gesture Recognition Conference andthe IEEE Computer Society Technical Committee onWearable Information Devices. Contact him atsandy@media.mit.edu.

Tanzeem Choudhury is a graduate student at the MITMedia Laboratory. Her research interests include facerecognition, real-time multimodal person identifica-tion, and facial expression analysis. Choudhury hasan SM from MIT and a BS in electrical engineeringfrom the University of Rochester. Contact her attanzeem@media.mit.edu.

February 2000 55