Heat Front Propagation Contours for 3D Face Recognition

Mostafa Abdelrahman1, Aly A. Farag1, and Moumen El-Melegy1,2

1Computer Vision and Image Processing LaboratoryUniversity of Louisville, Louisville, KY, 40292, USA

2Electrical Engineering Department, Assiut University, Assiut 71516, Egypthttp://www.cvip.uofl.edu

Abstract

Face recognition is a key biometric method aim-ing at identifying individuals by the features of face.Due to the challenges facing face recognition from2D images, researchers have resorted to 3D facerecognition. Our work in this paper is motivatedby the recent and remarkable success of heat-basedfeatures for 3D object classification and retrieval.We propose an approach for 3D face recognitionbased on the front contours of heat propagationover the face surface. The front contours are ex-tracted automatically as heat is propagating start-ing from a detected set of landmarks. The propaga-tion contours are used to successfully discriminatethe various faces. The proposed approach is eval-uated on the largest publicly available database of3D facial images and successfully compared to thestate-of-the-art approaches in the literature.

1. Introduction

Face recognition is one of the biometric meth-ods identifying individuals by the features of face.Automatic face recognition has evolved from smallscale research systems to a wide range of commer-cial products. Therefore, computer vision method-ology for automatic face recognition has becomean attractive research area in the past three decades(for more details see [23, 1]).

Face recognition from 3D has some advantages

Figure 1. The basic idea of the proposed approach: Theheat propagation front contours for a virtual heat sourceat some facial landmarks (nose tip here) are used as dis-criminating features between different faces. First andsecond faces are for the same subject thus having simi-lar contours, while the third face has different contoursbecause it is for another subject.

over 2D facial images. Their pose can be easilycorrected by rigid rotations in 3D space. They alsoprovide structural information about the face (e.g.,surface curvature and geodesic distances), whichcannot be obtained from a single 2D image. Lastly,3D face recognition algorithms have been shownto be robust to variations in illumination conditionsduring image acquisition.

3D face recognition approaches can be dividedto three main categories. The first category ofLocal features approaches utilizes local features,such as SIFT for meshes and face symmetry [18].In the second category, deformable template-basedapproaches have been proposed. As an exam-ple, Kakadiaris et al. [13] utilize an annotated facemodel to study geometrical variability across faces.The annotated face model is deformed elastically

to fit each face, thus matching different anatom-ical areas such as the nose, eyes and mouth. Inthe third category of surface-distance based ap-proaches, distances between feature points on theface surface are employed. Gupta et al. [12] use Eu-clidean/geodesic distances between fiducial points,in conjunction with linear classifiers. As stated ear-lier, the problem of automated detection of fidu-cial points is non-trivial and hinders automation ofthese methods. Also Bronstein et al. [5] provide alimited experimental illustration of this invarianceby comparing changes in surface distances with theEuclidean distances between corresponding pointson a canonical face surface.

Our work in this paper is motivated by the re-cent and remarkable success of heat-based featuresfor 3D object classification and retrieval [4, 2]. Wepropose an approach for 3D face recognition basedon the front contours of heat propagation over theface surface, see Figure 1. The front contours areextracted automatically as heat is propagating start-ing from a detected set of landmarks. The ex-traction of those fiducial landmarks is fully auto-mated. Our approach encodes the local face fea-tures as well as the diffusion distance over the sur-face around these landmarks. The propagation con-tours are used to successfully discriminate the var-ious faces. The proposed approach is evaluated onthe Texas 3D Face Recognition Database [12] asit is the largest publicly available database of 3Dfacial images acquired using a stereo imaging sys-tem. It is also compared to the state-of-the-art ap-proaches in the 3D face recognition literature.

The paper is organized as follows. Section 2 de-scribes the proposed approach starting from the fa-cial feature detection then heat equation solution,the describe form of the Laplace-Beltrami, and howto construct the heat kernel. Section 3 presents ourexperimental results. Conclusions and future workare given in Section 4.

2. Proposed ApproachWe present an approach for 3D face recogni-

tion based on heat kernel (HK). We develop a newapproach to extract the geometric points that havethe same diffusion distance from a source landmark

point. These points lie on a contour around a sourcelandmark point. These contours are unique for eachface, and using it as a face descriptor is able todiscriminate between human faces. In this sectionwe will describe how the landmarks points are de-tected. Then we will give the details about the so-lution of heat equation over a manifold, and the re-construction of the heat kernel based on the eigen-functions and eigenvalues of the Laplace-Beltramioperator. This will allow us to explain how the con-tours are extracted around the facial landmark.

2.1. Facial Landmark Detection

The relationship among facial feature positionsis commonly modeled as a single Gaussian dis-tribution function [7], which is the model usedby the Active Appearance Model (AAM) and Ac-tive Shape Model (ASM) algorithms. Everinghamet.al [9] modelled the probability distribution overthe joint position of the features using a mixture ofGaussian trees. The appearance of each facial fea-ture is assumed independent of the other featuresand is modelled using a variation of the AdaBoostalgorithm [19].

In this paper we use the method proposed in [9]to detect a small set of facial landmarks (exactlynine points). The landmarks are extracted fromthe 2D image and then mapped to the 3D surface.These points are used to initialized the ExtendedActive Shape Model (STASM) [15] for mesh fit-ting, where the STASM base mesh is warped tothese nine points. The output of this step is 68facial landmarks. Figure 2 illustrates an examplefor fitting STASM-based meshes on sample facesof Texas 3D dataset. Later we will show that wedo not need all these 68 points, and we will use asubset of 12 points.

2.2. Heat Front Propagation Contours

We provide a solution of the heat equation basedon a linear Finite Element Method (FEM) approx-imation to derive a discrete heat kernel. We solvefor the heat kernels at each detected landmark pointfor different time scales. Then a set of 3D contourson the face surface are extracted based on the heatkernel. These contours can be used to discriminate

Figure 2. Facial landmarks detected on sample faces fromTexas 3D faces dataset.

between the different faces.

2.2.1 The Heat Equation and Heat Kernel

Heat is energy transferred from one system to an-other by thermal interaction, it is always accompa-nied by a transfer of entropy. The heat equationis an important partial differential equation whichdescribes the distribution of heat (or variation intemperature) in a given region over time. One wayto solve this equation is to use eigenfunctions andeigenvalues of the Laplace-Beltrami operator. Inthis section we will describe the Laplace-Beltramioperator, its discretization, and the solution of heatequation on a manifold based on finite element ap-proximation.

Modeling the flow of heat at time t on a man-ifold M, the heat equation is a second orderparabolic partial differential equation, and usuallywritten as

4Mu(x, t) = − ∂

∂tu(x, t), (1)

where 4M denotes the positive semi-definiteLaplace- Beltrami operator of M , which is Rie-mannian equivalent of the Laplacian. The solu-tion u(x, t) of the heat equation with initial condi-tion u(x, 0) = u0(x) describes the amount of heaton the surface at point x in time t. u(x, t) is re-quired to satisfy the Dirichlet boundary conditionu(x, t) = 0 for all x ∈ ∂M and all t.

Figure 3. Discretization of the Laplace-Betrami operatorusing cotangent weights. Left:A vertex xi and its adja-cent faces Middle:the definition of the angles αij and βijRight: the definition of the area Ai.

2.2.2 Laplace-Beltrami Framework

Mathematically, the Laplace-Beltrami operator isa generalization of the Laplace operator to real-valued (twice differentiable) functions f ∈ C2 onany Riemannian manifold M. Let X be our shape,equipped with the Riemannian metric g, and f :X→ R be a real-valued function, with f ∈ C2, de-fined on a Riemannian manifold M. The Laplace-Beltrami operator can be defined as [17]:

∆Xf = −div(∇f) (2)

In order to process shapes represented on amesh, one needs to discretize the Laplace-Beltramioperator. To construct a discrete version of theLaplace-Beltrami operator 4Mu, we assume thatthe shape M is sampled at N points x1, . . . ,xN

and represented as a triangular mesh TX. Assumealso that a function u on the shape is discretizedand given as a vector with elements ui = u(xi) fori = 1, . . . , N.

Here we adopt the discretization method pro-posed by Desbrun et al. [8], which is straightfor-ward to implement and as accurate as the more re-cent methods [22, 21]. According to Desbrun etal. [8], the value of4Mu at a vertex xi is approxi-mated as

(4Mu)i = − 1

Ai

∑j∈Nei(i)

(cotαij+cotβij)(ui−uj),

(3)where (4Mu)i for a mesh function u denotesits discrete Laplacian evaluated at vertex i (fori = 1; 2; ....;N , N number of vertices); Ai is theVoronoi area at ith mesh vertex [10], as shown in

Figure 4. HK induced from three different landmarkspoint detected on the face shape at seven different timesamples. Colors represent the values of the heat inducedfrom the source point to each vertex of the shape. Forsmall time, the HK is more local. As time progresses,the HK becomes more global.

Figure 3, and αij and βij are the two angles sup-porting the edge connecting vertices i and j. Thisdiscretization preserves many important propertiesof the continuous Laplace-Beltrami operator [20],such as positive semi-definiteness, symmetry, andlocality, and it is numerically consistent.

In a matrix form we can write

(4Mu)i = A−1Lu, (4)

where A = diag(Ai), L = diag(∑

l 6=i wil)− wij ,and wij = (cotαij + cotβij). The first k small-est eigenvalues and eigenfunctions of the Laplace-Beltrami operator discretized according to (4) arecomputed by solving the generalized eigen decom-position problem Wφ = λAφ, where L = φΛφT ,Λ is a diagonal matrix of eigenvalues,and φ is aN × (k + 1) matrix whose columns correspond tothe right eigenvectors of L.

The first n = 100 eigenfunctions and eigenval-ues of the Laplace-Beltrami operator will be usedto construct the heat kernels in the following sub-section.

2.2.3 Heat Kernel

The heat kernel quantitatively encodes the heat flowacross a manifold M and is uniquely defined for

any two vertices i and j on the manifold. The heatdiffusion propagation over M is governed by theheat equation (1).

Given an initial heat distribution f : M ⊆Rd → R, as a scalar function defined on a compactmanifoldM , let u(x, t) denote the heat distributionat time t. The heat operator u(x, t) satisfies the heatequation for all t, and limt→0 u(x, t) = f .

Patane and Falcidieno [16] use a linear FEM ap-proximation to derive a discrete heat kernel that islinear, stable to sampling density of the input sur-face, and scale covariant. According to [16] thediscrete solution of the heat equation K(·, t) canbe written as

K(·, t) = ϕD(t)ϕTBf (5)

where ϕ = [ϕ1, ϕ2, . . . , ϕn], and D(t) =diag(exp(− 1

2λ1t), exp(− 12λ2t), . . . , exp(− 1

2λnt)),and λi and ϕi are the eigenvalues and eigenfunc-tions of the Laplace-Beltrami operator and n istheir number. And B is defined as:

B(i, j) :=

{|tr|+|ts|

12 if j ∈ Ne(i)∑k∈Ne(i)|tk|

6 if i = j(6)

where Ne(i) is the 1-star neighbors of vertex i,| tk | is the area of triangle, tk, tr and ts are the tri-angles that shares the edge (i, j) as defined in [16].

Then the heat kernel H(·, t) is given by:

H(·, t) := ϕD(t)ϕT B. (7)

Thus H(·, t) is N ×N matrix for N vertex shape,but we consider calculatingH at only small numberof critical points detected on the 3D object surfaceas described in next section.

Figure 4 shows the HKs induced from three dif-ferent critical point for one sample face at differenttime samples. For small time, the signature cap-tures local shape information. As time elapses, thesignature tends to capture more global shape de-tails.

2.3. Contours Matching

After calculating the heat kernels at each point,the 3D contours are extracted at the 3D point on the

Figure 5. Upper row: The selected 12 landmarks, Lowerrow: The reconstructed one contour around each point.These 12 contours are what we use as our face descriptor.

face surface that have equal heat values. We usea pre-determined number of the contours aroundeach point. Then each contour is sampled with afixed number of points. This representation givesa finite and ordered set of 3D points per face. Tomatch two faces, we simply use the Iterative Clos-est Point (ICP) algorithm to estimate rigid trans-formation parameters between the correspondingpoint sets for the found contours on the two faces.The L2norm distance between the contour pointsof the probe face and gallery faces after registra-tion is used as the distance measure, and the galleryfaces are ranked based on this distance measure.

3. Experimental ResultsTo assess the performance of the proposed ap-

proach we test it on the Texas 3D Face Recogni-tion Database [12, 11]. Currently, it is the largestpublicly available database of 3D facial images ac-quired using a stereo imaging system. The databasecontains 1149 3D models of 118 adult human sub-jects. The number of images of each subject variesfrom 1 per subject to 89 per subject. The 3D meshare created by triangulating the depth image. In ourexperiment we used a gallery set of 105 subjects(only one 3D model for each subject is used). Theprobe set is selected from 480 sessions (3D faces).

We did a series of initial experiments to test dif-ferent number of contours around each facial land-mark. It was found that using as few as one con-tour can provide good recognition rate. Using morecontours could improve the results further but at thecost of increased computational time. In the results

Figure 6. The semi-log CMC curves, for the eigensur-faces, fishersurfaces, ICP algorithms, anthroface 3D al-gorithm based on 25 manually located points, the an-throface 3D algorithm that employed 10 automaticallylocated points [12], and our proposed approach HFPC

reported below, one contour per each landmark isemployed. Also only 12 landmark points are usedin this experiment, Figure 5 shows sample of 3Dfaces with the used feature points and the extractedcontours.

For the sake of comparison, we show the resultsof five methods: the eigensurfaces of Chang et al.[6], fishersurfaces of BenAbdelkader and Griffin[3], and ICP algorithms Lu et al. [14], the anthro-face 3D algorithm [12], based on 25 manually lo-cated points, and the anthroface 3D algorithm thatemployed 10 automatically located points [12]. us-ing the same probe and gallery sets.

Figure 6 shows the cumulative rank curves(CMC) for the five algorithms together with ourproposed approach results. The proposed approachhas achieved a 100% recognition rate at rank 4. Theresults indeed confirm the superior performance ofthe proposed approach over the state-of-the-art ap-proaches for 3D face recognition.

4. Conclusion

This paper has addressed the problem of 3D facerecognition. We have presented a new approachbased on the front contours of heat propagationover the face surface. This was motivated by the re-cent and remarkable success of heat-based featuresfor 3D object classification and retrieval [4, 2]. Thefront contours are extracted automatically as heat

is propagating starting from a detected set of land-marks. The extraction of those fiducial landmarksis fully automated. Our approach encodes the localface features as well as the diffusion distance overthe surface around these landmarks.

The proposed approach has been evaluated onthe Texas 3D Face Recognition Database [12] asit is the largest publicly available database of 3Dfacial images acquired using a stereo imaging sys-tem. Our results have demonstrated the superiorperformance of the proposed approach over severalstate-of-the-art approaches in the 3D face recogni-tion literature. Future work may include testing theproposed approach on different databases.

References[1] A. F. Abate, M. Nappi, D. Riccio, and G. Sabatino.

2d and 3d face recognition: A survey. PatternRecogn. Lett., 28:1885–1906, 2007.

[2] M. Abdelrahman, M. T. El-Melegy, and A. A.Farag. 3d object classification using scale invari-ant heat kernels with collaborative classification. In(NORDIA) ECCV Workshops, pages 22–31, 2012.

[3] C. BenAbdelkader and P. A. Griffin. Compar-ing and combining depth and texture cues for facerecognition. Image Vision Comput., 23(3):339–352, 2005.

[4] A. M. Bronstein, M. M. Bronstein, L. J. Guibas,and M. Ovsjanikov. Shape google: Geometricwords and expressions for invariant shape retrieval.ACM Trans. Graph., 30(1):1, 2011.

[5] A. M. Bronstein, M. M. Bronstein, and R. Kimmel.Three-dimensional face recognition. InternationalJournal of Computer Vision, 64(1):5–30, 2005.

[6] K. I. Chang, K. W. Bowyer, and P. J. Flynn.An evaluation of multimodal 2d+3d face biomet-rics. IEEE Trans. Pattern Anal. Mach. Intell.,27(4):619–624, 2005.

[7] D. Cristinacce and T. F. Cootes. Facial feature de-tection using adaboost with shape constraints. InBMVC, pages 1–10, 2003.

[8] M. Desbrun, M. Meyer, P. Schroder, and A. H. Barr.Implicit fairing of irregular meshes using diffusionand curvature flow. In SIGGRAPH, pages 317–324,1999.

[9] M. Everingham, J. Sivic, and A. Zisserman. Hello!my name is... buffy” – automatic naming of charac-ters in tv video. In BMVC, pages 899–908, 2006.

[10] E. Grinspun, Y. Gingold, J. Reisman, and D. Zorin.Computing discrete shape operators on gen-eral meshes [Eurographics 2006 Best Paper, 3rdPlace]. Eurographics (Computer Graphics Forum),25(3):547–556, 2006.

[11] S. Gupta, K. R. Castleman, M. K. Markey, andA. C. Bovik. Texas 3d face recognition database.http://live.ece.utexas.edu/research/texas3dfr/index.htm.

[12] S. Gupta, M. K. Markey, and A. C. Bovik. Anthro-pometric 3d face recognition. International Journalof Computer Vision, 90(3):331–349, 2010.

[13] I. A. Kakadiaris, G. Passalis, G. Toderici, M. N.Murtuza, Y. Lu, N. Karampatziakis, and T. Theo-haris. Three-dimensional face recognition in thepresence of facial expressions: An annotated de-formable model approach. IEEE Trans. PatternAnal. Mach. Intell., 29(4):640–649, 2007.

[14] X. Lu, A. K. Jain, and D. Colbry. Matching 2.5dface scans to 3d models. IEEE Trans. Pattern Anal.Mach. Intell., 28(1):31–43, 2006.

[15] S. Milborrow and F. Nicolls. Locating facial fea-tures with an extended active shape model. ECCV,2008.

[16] G. Patane and B. Falcidieno. Multi-scale featurespaces for shape processing and analysis. In ShapeModeling International, pages 113–123, 2010.

[17] P. H. R.O. Duda and D. Stork. Eigenvalues inRiemannian geometry. London: Academic Press,,1984.

[18] D. Smeets, J. Keustermans, J. Hermans, P. Claes,D. Vandermeulen, and P. Suetens. Symmetricsurface-feature based 3d face recognition for par-tial data. In IJCB, pages 1–6, 2011.

[19] P. A. Viola and M. J. Jones. Rapid object detec-tion using a boosted cascade of simple features. InCVPR (1), pages 511–518, 2001.

[20] M. Wardetzky, S. Mathur, F. Kalberer, and E. Grin-spun. Discrete laplace operators: no free lunch. InSymposium on Geometry Processing, pages 33–37,2007.

[21] G. Xu. Convergent discrete laplace-beltrami oper-ators over triangular surfaces. In GMP, pages 195–204, 2004.

[22] G. Xu. Discrete laplace-beltrami operators andtheir convergence. Computer Aided Geometric De-sign, 21:767–784, 2004.

[23] W. Zhao, R. Chellappa, and A. Rosenfeld. Facerecognition: a literature survey. ACM ComputingSurveys, 35:399–458, 2003.