Skin Detail Analysis for Face Recognition

Jean-Sebastien Pierrard, Thomas VetterUniversity of Basel, Department of Computer Science

Bernoullistrasse 16, CH - 4056 Basel, Switzerland{Jean.Pierrard, Thomas.Vetter}@unibas.ch

AbstractThis paper presents a novel framework to localize in a

photograph prominent irregularities in facial skin, in par-ticular nevi (moles, birthmarks). Their characteristic con-figuration over a face is used to encode the person’s identityindependent of pose and illumination. This approach ex-tends conventional recognition methods, which usually dis-regard such small scale variations and thereby miss poten-tially highly discriminative features. Our system detects po-tential nevi with a very sensitive multi scale template match-ing procedure. The candidate points are filtered accordingto their discriminative potential, using two complementarymethods. One is a novel skin segmentation scheme basedon gray scale texture analysis that we developed to performoutlier detection in the face. Unlike most other skin detec-tion/segmentation methods it does not require color input.The second is a local saliency measure to express a point’suniqueness and confidence taking the neighborhood’s tex-ture characteristics into account. We experimentally evalu-ate the suitability of the detected features for identificationunder different poses and illumination on a subset of theFERET face database.

1. IntroductionFacial skin exhibits various small scale structures in the

surface (wrinkles, scars) and the texture (nevi – a generalterm for pigment lesions like birthmarks and moles) thatstand out from normal skin appearance and representpotentially valuable references for individual distinction.Among such skin irregularities moles are especially suitedfor identification. Their predictable appearance, alsounder changing illumination, facilitates detection. Andtheir numerous appearance in conjunction with uniquedistribution patterns scales well with extensive galleries.Furthermore moles require no abstract encoding, in contrastto most other facial features. This fact could be exploited toquery a database without having to provide a sample face,e.g. “search all faces with a birthmark near the upper rightlip”.

The goal of this paper is to present techniques for de-tection and validation of moles that are prominent enoughto be used for identification. Relying on such small scalevariations is an unusual approach in face recognition. Con-ventional recognition algorithms are designed to work onlow resolution images. For example the well known Eigen-faces approach [12], representative for linear appearancebased subspace methods, performs dimensionality reduc-tion using PCA on the raw image data and thereby implic-itly treats local variations as noise. Also model based al-gorithms like the Active Appearance Model in 2D [4] orthe Morphable Model in 3D [1] use PCA to model intraclass variations. These methods cannot capture small unex-pected details in their reconstruction without severe overfit-ting, which would render the whole method useless. Thereexist many techniques based on local descriptors using e.g.textons, DCT coefficients or Gabor wavelet features. How-ever, none of these methods involve an explicit represen-tation of one of the aforementioned skin features. To ourknowledge the only other attempt to exploit mole-like fea-tures for identification was presented by Lin and Tang [8].Their work comprises a multilayer representation of a facein global appearance, facial features (organs), skin textureand irregularities, which all contribute to the identity. TheSIFT framework is used for detection and description of ir-regular skin details which are then combined in an elasticgraph for recognition. Their approach also tackles stabil-ity and distinctiveness issues by validating interest regionsusing multiple gallery samples per person and by ensuringdissimilarity to normal skin regions. Unfortunately the au-thors do not provide methods how to compute the partition-ing and correspondence for the local regions (organs andskin).

We introduce a novel framework capable of detectingprominent moles. We attach importance to the fact that,in contrast to [8], our methods are not bound to a certainrecognition scenario. That means in particular that we workindependent of pose and illumination and that we are ableto deduce personally identifying features from one gallery

1-4244-1180-7/07/$25.00 ©2007 IEEE

Figure 1. Overview of processing steps and dependencies in ourmole detection framework. The left lane shows the main pro-cessing steps to obtain locations and saliency measures for moles.Starting from a Morphable Model reconstruction, the right lanesillustrate how the prior knowledge of the 3D face model is incor-porated into the system.

image per individual. Our system is divided into three mainsteps corresponding to the three properties that characterizea local region as birthmark, see also Figure 1.Appearance: From distance a mole appears simply assmall dark region of circular shape surrounded by a brighterarea, i.e. a so called ’blob’. This description also holdsunder varying viewing conditions (pose/illumination). Weemploy a very sensitive multi scale detection scheme, seesection 3, to identify even the most subtle mole candidates.Location: Due to its sensitivity, the detector also respondsto typical facial features such as nostrils, corners of eyes,eyebrows and mouth as well as to unexpected deviationslike hair strands. These points are not discriminative acrossindividuals and it is crucial for our scheme that they are re-jected. With prior knowledge derived from the MorphableModel we compute in section 4 a binary segmentationof the face in order to rule out points in those non-skinareas. In contrast to most other skin detection/segmentationschemes [13] our approach is texture based and thereforrequires no color input or training.Context: Finally the notion of saliency is introduced insection 5 which allows us to assess the importance ofeach birthmark candidate for recognition. This proceduretakes the relationship between a point’s size and contrastand the texture of it’s neighborhood into account. Inessence it represents a combined measure of uniquenessand confidence. Points below a certain saliency thresholdare immediately discarded.

Figure 2. Illustration of the feature/region mapping techniquebased on dense correspondence. In this example a binary mask,marking the cheeks, was selected in the reference frame. Via afitting of the Morphable Model this mask can be mapped to thephotograph and vice versa detected skin features can be mappedto the model.

2. Face representation

Our framework incorporates face-class specific knowl-edge from the 3D Morphable Model developed by Blanzand Vetter [1]. The model is derived from 3D laser scans,registered to a common reference in which shape and tex-ture are parameterized. A statistical analysis of this data isperformed by Principal Component Analysis. The result-ing sets of orthogonal eigenvectors represent facial proto-types which can be linearly combined to form new faces.For a set of model parameters (linear coefficients) and addi-tional parameters for pose and illumination, a realistic faceimage can be synthesized. Given a photograph of a face,an analysis-by-synthesis approach [2] is used to fit theseparameters such that the rendered face model matches theinput image. The result is a complete 3D reconstruction ofthe face where shape and texture in occluded regions (due topose) have been automatically replenished from the model.

While the model itself delivers features that can be usedfor recognition [2], we utilize it primarily as a preprocessingto establish a dense correspondence between the pixels of aface photograph and the fixed reference coordinate systemin which the model’s vertices are parameterized. Therebywe benefit from the model’s capabilities in two ways:

• Vertices marking certain features or regions in a facecan be consistently selected in the reference frame.Given an image and it’s Morphable Model reconstruc-tion, these vertices can then be projected into the imagedomain, either as point sets or as triangular mesh. Thistechnique, shown in Figure 2, allows us to roughly lo-calize the facial organs as well as to determine regionswhich are likely to contain only skin. The latter willserve our skin segmentation algorithm as initialization.

• With the reference frame acting as intermediary, loca-tions of feature points in different images can be en-coded and compared in a pose-independent manner.That means, once we localized interesting moles, theirpositions within the face are mapped to the referenceframe. In these “universal” coordinates we then matchthe individual mole configurations for recognition.

The optimization algorithm which fits the model to animage [2] requires manually defined feature points like thetip of the nose or corners of the eyes for initialization. Oncea 3D reconstruction is computed the corresponding pointsfrom the 3D model can be projected back into the image.We compute the average distance between manually de-fined and reconstructed coordinates to get an indicator forthe quality of the fit in terms of correspondence. In the fol-lowing we shall refer to this measure as alignment error.If not stated otherwise we perform all computations onlywithin the support region of the model. This is sufficientsince the detected moles must be compared in a commoncoordinate system which is only defined within the spatiallimits of the reconstruction.

3. Mole candidate detectionWe detect moles by means of normalized cross correla-

tion (NCC) matching. A Laplacian-of-Gaussian filter maskserves as template, because of its very close resemblance tothe blob-like appearance of moles. NCC is not scale invari-ant and the object size is not known a priori. This requiresus to compute the matching for several resolutions, usingtemplates of varying scale. With a growing number of res-olutions a straight forward implementation becomes veryinefficient. Therefor, inspired by Mikolajczyk et al. [9], wecarry the matching out in separated steps for candidate pointlocalization in space and scale respectively.

First NCC is computed for a small subset of scales, dis-tributed across the desired search range. In the output imageof each scale sk we then find all local maxima (xi, yi; sk)to pinpoint candidate positions in 2D. Only these points arefurther considered. In the second step we compute corre-lation coefficients for the remaining points, using templatesthat correspond to mole sizes in the range [.5sk, 2sk]. Ifthe maximum response across these scales is below a fixedthreshold the point is discarded. Otherwise the templatewith maximal correlation defines the points scale for sub-sequent processing.

Handling scale and space independently has the draw-back of causing duplicate point detections, meaning candi-dates located at different scales and/or coordinates but actu-ally responding to the same feature in the image. We iden-tify such cases and remove all duplicates except for the onewith largest scale.

Another problem arises in areas of changing brightnessas cause of shading (changing shape or illumination). Theintensity gradients surrounding a mole conflict with the uni-form area assumption coded in the mole templates. An ex-ample for which the described method fails, can be seen inFigure 3. The two obvious solutions to handle such situa-tions are not applicable for us. 1) Lowering the correlationthreshold would produce too many false positives in lessproblematic facial regions. 2) Matching against additional

Figure 3. Example of two prominent moles where detection inoriginal image (a) fails. The magnified section (b) shows multi-ple gradients in vicinity of both moles. After applying our illumi-nation compensation to this region (c,d), the detector succeeds onboth moles. Intensities in (b,d) are normalized.

templates on multiple scales that also incorporate skin shad-ing would dramatically increase the computational effort.Instead, we compensate for the shading in the input image.In section 4.2, in the context of skin segmentation, we in-troduce an image transformation that attempts to removegradients from shading. The mole detector is then simplyapplied to the output of this procedure, with better results.

Before conducting the experiments, we hand-labeled ina few gallery images the locations of moles that we deemedsalient. The number of scales (range & sample steps) andthe NCC threshold were then chosen such that all markedpoints could be located. Template detection typically re-duced the number of candidates for further processing to1-2h of the pixels representing a face.

4. Facial skin segmentation

The template detector does not incorporate any specificknowledge as to where moles can appear. As consequenceit may nominate any facial feature with similar appear-ance, e.g. pupils, nostrils or corners of the mouth. More-over we must expect sporadic hits in areas with hair (beard,hairstyle). Since none of these findings are characteristicfor a person, they have a negative impact on the recognitionperformance and must therefore be eliminated. We tacklethis problem by computing a binary segmentation of theface into skin and “non-skin” regions. Mole candidates ly-ing outside the skin segment can then be rejected. Note thatit is not possible to reverse the execution order of detectionand then rejection. This becomes clear when we look at thesegmentation results at the end of section 4.1.

The non-skin region is composed of two parts. One partis derived directly from the 3D reconstruction with the Mor-phable Model. In the reference frame of the model we havedefined the subset of vertices which belong to the eyes, nos-trils and the lips and then projected this selection to the im-age (see section 2). Due to imperfect reconstructions the re-sulting mask may not be very precise. We take this into ac-count by dilating the mask according to the measured align-ment error.

Figure 4. Display of reconstructed locations for eyebrows, eyes,cheeks and lips on two faces with problematic model fitting. Im-portant features are globally misaligned, due to outliers/hairstyle(left) and expression (right). These shortcomings necessitate analternative method to mask such regions.

The second non-skin component marks outliers. By thisterm we refer to all kind of unexpected objects in the sensethat they do not appear in every face. Unfortunately theMorphable Model offers no clear strategy how to deal withoutliers. On one hand eyebrows, beard and the hair line canbe reproduced in the texture. On the other hand hairstyleor open mouth (any expressions) are not represented. Evenworse, if larger areas of the face contain outliers this canseriously perturb the model parameters and corrupt a recon-struction in several ways: 1) Due to the holistic representa-tion, adapting the modeled texture to outliers comes at thecost of higher reconstruction errors in other regions. Asresult differences between the real and the rendered image“even out”. 2) The estimated illumination parameters arediverted and can introduce cast shadows into the synthe-sized image. 3) The reconstructed shape deteriorates andleads to bad correspondence and thereby misaligned fea-tures. Two such problematic examples can be seen in Fig-ure 4. Our conclusion from these shortcomings is that wecannot trust the model’s reconstruction error to reveal alloutliers. Instead, we utilize only the most reliable contri-butions of the model, i.e. pose and parts of the shape, toinitialize a general purpose segmentation algorithm.

In gray scale skin and non-skin are only distinguish-able by their luminosity and/or texture. It is often thecase, that their respective intensity distributions show sig-nificant overlap, which makes per pixel decisions inappro-priate. The techniques presented below operate on localneighborhoods. In the next section we propose a simpleprocedure to find skin regions by example. Then we intro-duce a preprocessing step to render the results more robustagainst varying lighting conditions, see section 4.2. For bet-ter understanding we initially motivate these methods usingonly thresholding as binary segmentation. Later, in section4.3 we compare our results with a state-of-the-art segmen-tation algorithm.

Figure 5. Texture similarity procedure applied to multi-texturedimage (left), with Itgt = Isrc and Iseed shown in color. The cor-responding matching error Ek

ts (right) clearly marks regions con-taining the sample texture.

4.1. Segmentation based on texture similarity

In the work of Efros and Leung [5] textures are synthe-sized by repeatedly matching the neighborhoods around un-processed pixels in the synthesis image against all possiblesource patches extracted from a sample texture. The centerpixels of the minimum error patches then build up the syn-thesized texture. With modifications this idea can be usedas analysis tool to compute a measure of texture similarityfor one image (target) with respect to a given sample of thetexture (source).

Let Itgt be the target image for which the similarityshould be computed. Further we denote with Isrc a sourceimage and with Iseed an associated binary mask, both defin-ing a texture sample region. The similarity is then computedfor each pixel p ∈ Itgt independently by taking its localneighborhood N tgt

p and searching within the seed region ofIsrc for the best matching patch N src

q . Unlike [5] we do notimpose different weights on the neighborhood’s pixels. Thetexture similarity error per target pixel p is:

Ets(p) := minq|N src

q ⊂(Isrc∩Iseed )

∥∥N tgtp −N src

q

∥∥SSD

(1)

This measurement does not yet take the statistics of the sam-ple texture into account. In order to determine how likelya target pixel may originate from this texture we actuallycompute the k-nearest-neighbors to N tgt

p . The error Ekts is

then defined, analogous to equation (1), as the average ofthe corresponding closest-patch distances. Figure 5 demon-strates that ideally this procedure will indeed distinguish be-tween textures.

The method is easily adopted to our segmentation prob-lem. We have chosen the cheeks as facial area which is un-likely to contain outliers. As illustrated in Figure 2, we areable to determine the corresponding region in a novel facefrom its Morphable Model reconstruction, which providesthe skin seed mask Iseed . Recalling Figure 4, we also seethat this particular mapping is very robust in case of prob-lematic fittings. The algorithm is first applied to the originalface image with Itgt = Isrc . Under the assumption that theselected seed contains only skin, the output Ek

ts inside thisarea defines the range of matching errors one can expect forsimilarly textured regions. The maximum of this range is

Input + seed Ekts (inverted) Segmentation

Figure 6. Segmentation by thresholding on the output of texturesimilarity. In first and second row texture similarity was computedon original image with a very good result in the top row. The badresult in the second row shows how this method can be negativelyaffected by shading. Performing the same task on an illuminationcompensated version of the image solved this problem (3rd row).

used as threshold to the entire Ekts and we obtain a segmen-

tation:

Iskin(p) :=

1 if Ekts(p) ≤ max

q∈IseedEk

ts(q)

0 otherwise(2)

Notice that, without averaging over the k-NN, all errors in-side Iseed would be zero and this idea could not work. Re-sults for two faces are displayed in Figure 6. We see thatshading may provoke large gaps in the segments in lighterareas of the skin. This problem is dealt with in the nextsection. Another point to notice is that this segmentationmethod also treats larger moles as outliers. We employ asimple heuristic to prevent such areas from being excludedfrom further processing: if a mole candidate is located in-side a hole of the skin segment, we still accept it if the gap’ssize is less than two times larger than the candidate’s scale.Obviously this rule forces us to perform mole detection first.

4.2. Illumination compensation

In 3 and the previous section we pointed out that sig-nificant changes in the skin’s luminosity can have a neg-ative impact on the performance of the mole detector andthe texture similarity algorithm. Therefor we introduce amethod to compensate for this effect by performing illumi-nation compensation, based on a variant of homomorphicfiltering [6]. The underlying simplified reflectance modelassumes that for each pixel location (x, y) the image canbe described by the product of reflectance and illumination:I(x, y) = R(x, y) · L(x, y). Thus, to recover R one wouldsimply need to divide the image by the illumination. Un-fortunately L is unknown. However, the model further sug-gests that lighting changes slowly and smoothly across animage while reflectance manifests itself in high frequencycomponents. The idea is now to approximate L by a low-pass filtered version of the image, here denoted by Flp(I).Since the frequencies of function products are not directlyseparable (see [6]) this is done in the log-domain. The re-flectance becomes:

log (R(x, y)) = log (I(x, y))− log (L(x, y))≈ log (I(x, y))− [Flp (log(I))] (x, y)

(3)

The exact type and application (spectral or spacial domain)of filter vary among different homomorphic filtering ap-proaches. In our case an approximation to the illuminationterm is computed by locally fitting smooth functions to thelogarithm of image brightness surface.

Given an image I the fitting procedure works as follows.For each pixel p we interpret pixels in it’s neighborhoodNp as points on a 3D surface. Np is translated into lo-cal coordinates (xi, yi, Ii) such that the center pixel p be-comes (0, 0, 0). Then we compute a least-squares fit ofthe quadratic function z = f(x, y) = a

2x2 + bxy + c2y2

to these points. Let zp(q) denote the LSQ solution forpatch Np evaluated at pixel q ∈ Np. The approxima-tion induces an error on each pixel of the fitted patch (ex-cept on the center pixel). As this procedure is repeated forthe whole image, every pixel p ∈ I receives errors fromseveral patches, namely those neighborhoods which some-where overlap with p. We accumulate these error contribu-tions separated into positive and negative components:

E−ic(p) :=

1|Np|

∑{q|p∈Nq}

min(0, I(p)− zq(p)

)(4)

The definition of E+ic is analogous, but rather computes

max(0, ·). If this procedure is applied to log(I), the errorscan be interpreted as the right side in equation (3), where thelow-pass filter has been implemented as average of smoothfunction approximations of the neighborhood. Taking theexponential, brings us back to the image domain and re-sults in two reflectance images. The advantage of separat-ing positive and negative errors in equation (4) is that we

Figure 7. Comparison of skin segments obtained from GrabCut(bottom) and thresholding (top). GrabCut has the advantage ofproducing fewer small and isolated segments with the downsidethat it is too conservative in shaded regions.

can isolate different reflectance contributions. For exam-ple R− = exp(E−

ic) represents the details with darker ap-pearance like creases, moles or pupils, whereas R+ capturesbrightness peaks like sharp specular highlights. For the fi-nal results in our paper we use only the R− image as inputfor the mole detector as well as for the texture similarityalgorithm.

4.3. Thresholding vs. “GrabCut”

In order to verify, if we could further improve our skinsegmentation, this section compares the results computedby simple thresholding of the texture similarity output Ek

ts

(obtained on the R− input) with a modern algorithm basedon graph cuts. In graph cut segmentation techniques the pix-els of an image are represented as nodes in a graph with theedges reflecting their spatial relationship. The edge weightsare defined as some measure of similarity on the connectedpixels. Via a cost function, that takes the edges betweensegments and regional properties into account, the segmen-tation problem is reformulated into one of partitioning thegraph, such that the cost function is minimized.

For the case of two labels (object or background) Boykovand Jolly [3] presented a graph cut formulation that canbe efficiently computed with a combinatorial optimizationtechnique and leads to a globally optimal binary labeling.Their cost function is influenced by two models, one foreach label, of the pixel values in the respective segment.The models are learned from seed regions in the image, forwhich the labeling is known. However, with no prior knowl-

Figure 8. Illustration of density estimation for saliency. A pixelgroup ~q (red) and all other similarly shaped constellations (green)within it’s neighborhood populate a multidimensional featurespace. The density is estimated by the number the samples lyingwithin a spherical Parzen window around the feature point ~q.

edge on the location of object and background, this place-ment of seeds demands for user interaction. Rother et al.[11] extended this approach. Their method, called GrabCutembeds the “one-shot“ algorithm of [3] into an EM scheme,where model parameters and labeling are iteratively refined.One major benefit of this procedure is, that it suffices toprovide seeds for only one label. Thanks to this propertyGrabCut is directly applicable to our problem setting.

Figure 7 compares results obtained by our own imple-mentation of this algorithm with those from thresholding.In both cases the inputs are the texture similarity error Ek

ts

computed on the illumination compensated R− image andthe cheek region seed. It is important to note that compu-tations are limited to the support of the Morphable Modelreconstruction. This is not relevant for thresholding but forGrabCut, because it prevents pixels from the background(clothes, etc.) to “pollute“ the statistics associated with thetwo labels. We observe that GrabCut (with manually tunedbut constant parameters) tends to generate less scatteredsegments with smoother segment boundaries. However, itcuts off too many pixels in highly shaded regions (especiallyaround the nose) and thus produces larger gaps in the skinsegment. This is unacceptable for our current application,since we may lose important moles (also due to the heuristicwe implement on segment holes, see section 4.1).

5. Local saliencySaliency is commonly used as synonym for discrimina-

tive power. The more salient a feature is, the better it shouldbe distinguishable from others. The exact definition, how-ever, depends on the actual application. In [14] Walker etal. formulate this notion over the probability density in fea-ture space and reason that salient features should lie in lowdensity areas. Hence, intuitively saliency corresponds torarity. In their paper the pdf is approximated with mixturesof Gaussian kernels. Hall et al. [7] take on the same defini-tion but use a more accurate Parzen windows technique fordensity estimation, which we also adopt here.

Having constrained the detected mole candidates to skinregions, our goal is now to define a measure that allows usto differentiate between prominent and more or less coinci-dental hits. The latter may occur in “noisy” regions, e.g. in

Figure 9. Filtering of mole candidates according to saliency. Cir-cles mark points with saliency salε ≥ 1 which are later used toidentify this face. Zoomed neighborhoods of four candidates withcorresponding patches from exp(E−ic) show that our saliency mea-sure indeed relates a point’s size and contrast to the surroundingnoise and delivers an intuitive measure of importance.

the presence of freckles or stubble, where a single dark spothas no significance. A point’s scale and correlation coeffi-cient (from detection) contribute to this assessment but arenot sufficient. We therefore combine two more properties,the contrast and the uniqueness of a point wrt. it’s neighbor-hood, into a saliency value.

Consider a mole candidate composed of a group of d pix-els stored as vector ~q. Further assume a square neighbor-hood N~q centered around the same location. In our case thewidth of the neighborhood is fixed and corresponds roughlyto the pixel distance between both nostrils in a frontal view.We extract from N~q all possible translated and mirrored re-gions ~ri that have a shape similar to ~q but do not share anypixel with ~q. Let’s assume there are M such regions, whichpopulate a d-dimensional feature space. We then considera hypersphere with radius ε and volume Vε around ~q anddetermine the number k of feature points lying within thesphere, as shown in Figure 8. The ratio k/M

Vεis then an

estimate for the probability density at ~q. Based on the mea-surement of k we define our saliency as:

salε(~q) :=

min~ri⊂N~q,~ri∩~q=∅

‖~q−~ri‖ε for k = 1

M−kM for k > 1

(5)

The radius is chosen as ε = d · σ2N~q

, with σN~qdenoting the

standard deviation of all pixels in N~q but not in ~q. Let ustake a closer look at the two cases in equation (5):• k > 1: As more points fall within the ε-sphere, the

estimated density around ~q increases by the ratio kM .

The saliency simply decreases by the same rate, takingvalues in the range [0, 1).

• k = 1: No other feature is closer than ε to ~q. Wecompute the distance to it’s nearest neighbor inmultiples of the sphere radius. Since ε is related tothe sample variance in N~q the saliency becomes anormalized measure of how much the pixels in ~q standout from the noise in it’s neighborhood, ranging from[1,∞).

We apply the described procedure at every mole can-didate location in the illumination compensated imageexp(E−

ic), constrained to skin segments. An example ofevaluated points is shown in Figure 9. In the left image allpoints delivered by the mole detection process have beenhighlighted. The red squares mark candidates which lie ei-ther in non-skin regions or which have a computed saliencysalε < 1. The remaining points are deemed salient andwill be used for identification. Of course not all acceptedpoints are equally “interesting”. Figure 9 also depicts theprocessed patches (right column) of the three most salientmoles and one of the rejected points. The non-skin partsare masked out and the remaining pixels are normalized.Clearly the saliency correlates with mole size and is higherfor points with less variation (noise) in the surrounding.

6. ExperimentsWe perform identification purely based on the previ-

ously detected moles on a subset (reported in [2]) of theFERET [10] face database, for which we obtained the Mor-phable Model reconstructions. This subset consists of graylevel images with resolutions in the range of 50-80 pix-els eye distance. It contains images of 194 individuals in11 poses from which we chose the sets ba (frontal view)for the gallery, bc-bh (head rotated by ±40◦,±25◦,±15◦)and bk (frontal view with different illumination) as probefaces. The recognition experiments are limited to personsfor which the respective gallery image contains at least onemole with a saliency greater than some threshold. Our sim-ilarity measure is based on the mole locations in the Mor-phable Model reference coordinates and their associatedsaliency values. We compare two given faces F and G asfollows:• The saliency values of all moles of a face are trans-

formed to relative weights wi = sali/∑n

j=1 salj .• A proximity threshold σthr is defined as the average of

the alignment errors of both faces.• For each mole location i in F we find the closest point

j from G. If their distance is smaller than 3σthr thepoint i is considered matched and we define a match-ing value vi = min(wF

i , wGj )/ max(wF

i , wGj ). In this

case the point j from G is removed so that it cannotmatch any other locations in F . Otherwise (distancethan proximity threshold) i remains unmatched, we setvi = 0 and proceed to the next item.

Saliency threshold (Gallery subset size)5 (156) 10 (107) 15 (83)

Probe Fail Perf. Fail Perf. Fail Perf.bc 69 55.77 39 63.55 26 68.67bd 34 78.20 13 87.85 8 90.36be 17 89.10 7 93.45 4 95.18bf 20 87.18 5 95.32 5 93.97bg 47 69.87 24 77.57 17 79.51bh 68 56.41 30 71.96 21 74.70bk 42 73.07 22 79.44 13 84.33

Table 1. Performance of identification purely based on detectedmoles. The gallery (frontal views, ba) and probe are limited tofaces, which contain at least one mole in the gallery with a saliencygreater than the denoted threshold. Performance is listed as num-ber of unidentified faces from the gallery subset (Fail) and in per-cent (Perf ).

Gallery /Probe

Saliencythreshold

Gallerysubset size Fail Perf.

ba / be 1 194 42 78.35ba / bf 46 76.28bb / bc 5 180 17 90.55bi / bh 5 184 21 88.58

Table 2. Identification from moles on full gallery with ±15◦ ro-tated probes and on two non-frontal galleries.

• After processing all point from F the matching scoreis computed as

∑nF

i=1 vi/ max(nF , nG).

From the identification results displayed in Table 1 and 2we notice that: 1) Performance drops with increasing rota-tion angle independent of the gallery pose. This is obvious,since the overlapping area in which moles from both facescan be matched shrinks. Recognition under the different il-lumination (bk set) suffers from the lower contrast betweenmoles and skin which results in lower saliency values andthus more rejections. The total number of detected moles is1600, whereas in all other sets we can account for more than2200 moles. 2) At least 80% of the faces have some promi-nent moles (saliency ≥ 5) for which we obtain recognitionperformances above 87%. This is quite remarkable, consid-ering that in average about 5-10 locations, representing lessthan 0.3% of the pixels in a face, determine it’s identity. En-forcing more prominent moles leads to better performancebut greatly reduces the number of usable faces. Somewherebetween saliency thresholds of 10-15 is the limit beyondwhich the number of misclassified faces decreases less thanthe number of available faces.

7. ConclusionThis paper presented a novel approach to exploit local

skin irregularities as features for face identification. We fo-cused on the methodology of detection and evaluation of

such regions and showed that it is possible to determinea persons identity based on only a few well-chosen pix-els. Future work comprises refinements in the comparisonof local skin features (e.g. valuing the absence of salientmoles as exclusion criterion) as well as fusion with otherface recognition methods to support cases where no molesare present.

References[1] V. Blanz and T. Vetter. A morphable model for the synthesis

of 3D faces. In Siggraph 1999, Computer Graphics Proceed-ings, pages 187–194, Los Angeles, 1999. Addison WesleyLongman.

[2] V. Blanz and T. Vetter. Face recognition based on fitting a 3dmorphable model. IEEE Transactions on Pattern Analysisand Machine Intelligence, 25(9):1063–1074, 2003.

[3] Y. Boykov and M.-P. Jolly. Interactive graph cuts for op-timal boundary and region segmentation of objects in n-dimages. In International Conference on Computer Vision,(ICCV), pages 105–112, 2001.

[4] T. F. Cootes, G. J. Edwards, and C. J. Taylor. Active appear-ance models. IEEE Transactions on Pattern Analysis andMachine Intelligence, 23:681–685, Jan. 2001.

[5] A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In IEEE International Conference onComputer Vision, pages 1033–1038, Corfu, Greece, Septem-ber 1999.

[6] R. C. Gonzalez and R. E. Woods. Digital Image Process-ing. Addison-Wesley Longman Publishing Co., Inc., Boston,MA, USA, 2001.

[7] D. Hall, B. Leibe, and B. Schiele. Saliency of interest pointsunder scale changes. In British Machine Vision Conference(BMVC’02), Cardiff, UK, September 2002.

[8] D. Lin and X. Tang. Recognize high resolution faces: Frommacrocosm to microcosm. In CVPR ’06: Proceedings of the2006 Conference on Computer Vision and Pattern Recogni-tion, pages 1355–1362, 2006.

[9] K. Mikolajczyk and C. Schmid. Indexing based on scaleinvariant interest points. In Proceedings of the 8th Interna-tional Conference on Computer Vision, Vancouver, Canada,pages 525–531, 2001.

[10] P. J. Phillips, H. Wechsler, J. S. Huang, and P. J. Rauss.The FERET database and evaluation procedure for face-recognition algorithms. Image and Vision Computing,16(5):295–306, 1998.

[11] C. Rother, V. Kolmogorov, and A. Blake. “GrabCut”: inter-active foreground extraction using iterated graph cuts. ACMTransactions on Graphics, 23(3):309–314, Aug. 2004.

[12] M. Turk and A. Pentland. Eigenfaces for recognition. Jour-nal of Cognitive Neuroscience, 3(1):71–86, 1991.

[13] V. Vezhnevets, V. Sazonov, and A. Andreeva. A survey onpixel-based skin color detection techniques. In Proc. Graph-icon, 2003.

[14] K. Walker, T. Cootes, and C. Taylor. Locating salient ob-ject features. In British Machine Vision Conference (BMVC),volume 2, pages 557–566. BMVA Press, 1998.