recognition of human faces: from biological to artifical vision massimo tistarelli computer vision...

Post on 24-Jan-2016

220 Views

Category:

Documents

2 Downloads

Preview:

Click to see full reader

TRANSCRIPT

Recognition of Human Recognition of Human Faces: from Biological Faces: from Biological

to Artifical Visionto Artifical Vision

Massimo TistarelliMassimo TistarelliComputer Vision Laboratory

University of Sassari – Italytista@uniss.it

October 30 2007 - IstanbulMassimo Tistarelli 2

The Computer Vision Lab

Started in 1990 at University of Genova

Moved to Alghero, Sardinia in 2002

October 30 2007 - IstanbulMassimo Tistarelli 3

The Computer Vision Lab

October 30 2007 - IstanbulMassimo Tistarelli 4

The Computer Vision Lab

Second European Conference on Computer Vision – May 1992 – ECCV 1992 – S. Margherita Ligure, Italy

Int.l Workshop on Facial Image Analyisis and Recognition Technology – May 1998 – Freiburg, Germany

Int.l Workshop on Biometric Authentication – June 2002 Copenhagen, Denmark

Int.l Summer School for Advanced Studies on Biometrics:– 1st Authentication and Recognition – June 2-6 2003– 2nd Multimodality and System Integration – June 6-10 2005– 3rd New Sensors, Databases and Evaluation – June 12-16 2006– 4th New Technologies and Embedded Systems – June 11-15 2007– 5th New Technologies for Security and Privacy – June 9-13 2008

Fifth IEEE AutoId Workshop – June 7-8 2007, Alghero, Italy

Third IAPR/IEEE Int.l Conference on Biometrics – June 2-5 2009, Alghero, Italy

October 30 2007 - IstanbulMassimo Tistarelli 5

Credits

The laboratory staff:

– Manuele Bicego

– Gavin Brelstaff

– Linda Brodo

– Enrico Grosso

– Dakshina Kisku (on leave from IITK)

– Andrea Lagorio

– Ajita Rattani (joint Phd with UNICA)

– Elif Surer

October 30 2007 - IstanbulMassimo Tistarelli 6

Concise outline

Recognition of human faces and Biometrics

Issues from the Human Visual System

Applied technologies for face recognition

Exploiting knowledge from biological systems

New “trends and hints” ?

October 30 2007 - IstanbulMassimo Tistarelli 7

Biometrics

•Biometrics (as in statistics): “Statistical study

of biological observations and phenomena”

(Webster)

•Biometrics (as in security industry): “A

measurable, physical characteristic or personal

behavioral trait used to recognize the identity, or

verify the claimed identity, of an enrollee”

•Biometric recognition: Personal recognition

based on “who you are or what you do” as

opposed to “what you know” (password) or

“what you have” (ID card)

October 30 2007 - IstanbulMassimo Tistarelli 8

Most natural for humans

Highly acceptable and non-intrusive

Highly applicable:

– Static identity verification

– Uncontrolled face detection and identification from video

Medium to Low performances

Not unique (twins)

Aging and time effects

Why face recognition?

October 30 2007 - IstanbulMassimo Tistarelli 9

Are both from the same Are both from the same subject?subject?

How do we recognize faces?

October 30 2007 - IstanbulMassimo Tistarelli 10

Are both from the same Are both from the same subject?subject?

How do we recognize faces?

October 30 2007 - IstanbulMassimo Tistarelli 11

Score

Automated face recognition

Face Detection and Selection

Facial featuresextraction

Registrationand

Representation

Matching(comparison)

Input (grabbed) sequence

Individual face model

October 30 2007 - IstanbulMassimo Tistarelli 12

Automatic classification of faces

A class separation problem:

Fal

se

Raj

ecti

on

False Acceptance

?

October 30 2007 - IstanbulMassimo Tistarelli 13

Fk

Fh

L1 L2

S1

S3

S2

= (Fh,Fk)

Identification from human faces

A class separation problem:

• Choice of optimal representation • Inter-class similarity vs Intra-class variability

October 30 2007 - IstanbulMassimo Tistarelli 14

• Two different people may have very similar appearance

Twins Father and son

www.marykateandashley.com news.bbc.co.uk/hi/english/in_depth/americas/2000/us_elections

Inter-class SIMILARITY

October 30 2007 - IstanbulMassimo Tistarelli 15

Intra-class VARIABILITY

October 30 2007 - IstanbulMassimo Tistarelli 16

Recognition/Identification and Authentication/Verification

Many facts come into play:

– The dimensionality of the data and feature space

– The noise in the data

– The data alignment/localizazion

– The acquisition device

– The algorithmic classification

– The time variability of the data

– MORE….

October 30 2007 - IstanbulMassimo Tistarelli 17

Data dimensionality

... Not many ... (20x14)

It’s more a question of spatial distribution and …

proper frequency tuning

How many pixels should be processed to reliably How many pixels should be processed to reliably recognize a face?recognize a face?

October 30 2007 - IstanbulMassimo Tistarelli 18

Local analysis (Receptive fields)

The density and size of the cones decreases linearly from the retina center (fovea) to the periphery

October 30 2007 - IstanbulMassimo Tistarelli 19

Contextual analysis (Visual attention)

Eye movements while watching a girl’s face (A.L. Yarbus, “Eye Movements and Vision”, Plenum Press, 1967)

October 30 2007 - IstanbulMassimo Tistarelli 20

Space-variant representations

J. Bigun et al. 1997

M. Tistarelli et al. 1996

October 30 2007 - IstanbulMassimo Tistarelli 21

Where is the “Mom’s face” neuron?

October 30 2007 - IstanbulMassimo Tistarelli 22

Brain anatomy

Coronal anatomical image of a brain:STS, Superior temporal sulcus

MTG, middle temporal gyrus

ITS, inferior temporal sulcus

ITG, inferior temporal gyrus

FG, fusiform gyrus

FFrom rom fMRIfMRI, f, face-selective neurons have been found in the ace-selective neurons have been found in the inferior inferior temporal areastemporal areas, the , the superior temporal sensory areasuperior temporal sensory area, the , the amygdalaamygdala, the , the ventral striatumventral striatum ( (receivingreceiving input from the amygdala) and the input from the amygdala) and the inferior inferior convexityconvexity..

Specific regions have been reported also in the fusiform gyrus responding Specific regions have been reported also in the fusiform gyrus responding significantly to viewing faces.significantly to viewing faces. A. R. Damasio, J. Damasio and G. W. Van Hoesen. “Prosopagnosia: “Prosopagnosia:

Anatomic basis and behavioral mechanisms”. Anatomic basis and behavioral mechanisms”. Neurology, vol. 32, 331–341, 1982.

October 30 2007 - IstanbulMassimo Tistarelli 23

Brain activation

• Parts of the Parts of the inferiorinferior and and medialmedial temporal cortextemporal cortex may work may work together to process faces,together to process faces, the the amygdalaamygdala being being responsible for responsible for assigning significance to faces, assigning significance to faces, and thus and thus affecting both affecting both attention and mnemonic attention and mnemonic aspects of face processingaspects of face processing..

• Autism and Asperger syndromAutism and Asperger syndrom

C. A. Nelson. “The development and neural bases of face recognition”. “The development and neural bases of face recognition”. Infant and Child Development, vol. 10, 3-18, 2001.J. P. Aggleton, M. J. Burton and R. E. Passingham. “Cortical and subcortical afferents to the amygdala of the rhesus monkey “Cortical and subcortical afferents to the amygdala of the rhesus monkey (Macaca mulatta)”. (Macaca mulatta)”. Brain Research, vol. 190, 347–368, 1980.

October 30 2007 - IstanbulMassimo Tistarelli 24

Brain activation – fMRI maps

C. L. Leveroni et al. “Neural Systems Underlying the Recognition of Familiar and Newly Learned Faces”, The Journal of Neuroscience, January 15, 2000, 20(2):878–886

Recognition of 50 Familiar Faces (FF) vs 50 Newly Learned Faces (NL) and compared to rejection of 50 Foil (FO -False Objective) faces.Encoding (EN) session for learning new faces.

October 30 2007 - IstanbulMassimo Tistarelli 25

Brain activation areas

C. L. Leveroni et al. “Neural Systems Underlying the Recognition of Familiar and Newly Learned Faces”, The Journal of Neuroscience, January 15, 2000, 20(2):878–886

October 30 2007 - IstanbulMassimo Tistarelli 26

Facial features and brain activation

Aylward et al. Brain Activation during Face Perception: Evidence of a Developmental Change. J. Cogn. Neurosci..2005; 17: 308-319.

October 30 2007 - IstanbulMassimo Tistarelli 27

Aylward et al. Brain Activation during Face Perception: Evidence of a Developmental Change. J. Cogn. Neurosci..2005; 17: 308-319.

Eyes vs radialAll stimuli

Facial features and brain activation

October 30 2007 - IstanbulMassimo Tistarelli 28

Vaina, L.M., Solomon, J., Chowdhury, S., Sinha, P., Belliveau, J.W., “Functional Neuroanatomy of Biological Motion Perception in Humans”. Proc. of the National Academy of Sciences of the United States of America, Vol. 98, No. 20 (Sep. 25, 2001) , pp. 11656-11661

Face and motion perception

Biological Motion

Non Rigid Motion

October 30 2007 - IstanbulMassimo Tistarelli 29

Neural architecture of face perception - 1

J.V. Haxby, E.A. Hoffman, and M.I. Gobbini. “The distributed human neural system for face perception”. Trends in Cognitive Sciences, vol. 4, No. 6, 223--233, June 2000

Motion

Static

October 30 2007 - IstanbulMassimo Tistarelli 30

A. O’Toole, D.A. Roark, and H. Abdi. “ Recognizing moving faces: A psychological and neural synthesis”.Trends in Cognitive Science, 6:261--266, 2002.

Neural architecture of face perception - 2

October 30 2007 - IstanbulMassimo Tistarelli 31

Cooperative/competitive visual tracking AND recognition

Evidence

Face trackingprocess

Feature extractionregistration process

Incrementalmatching process

Expectation

Biological face recognition

October 30 2007 - IstanbulMassimo Tistarelli 32

Face representation

It is not clear how faces are represented in the HVS, but the representation is formed by a dynamically updated collection of visual fixations.– Both foveal and peripheral vision are involved, the former

responsible for a more accurate representation.J.M. Henderson et al. “Eye movements are functional during face learning”, Memory & Cognition

2005, 33 (1), 98-106.D.R. Melmoth et al. “The Effect of Contrast and Size Scaling on Face Perception in Foveal and

Extrafoveal Vision”, Investigative Ophthalmology and Visual Science. 2000;41:2811-2819.

This representation includes both iconic data as well as information about the spatial relationship among face elements.

– What about a “face template”?

October 30 2007 - IstanbulMassimo Tistarelli 33

Pose-dependent

Algorithms

Pose-invariant

Pose-dependency

Matching features

Appearance-based (Holistic)PCA, LDA, ICA, LPPKernel . . .

Feature-based (Analytic)Gabor setsLBP . . .

HybridLFAEGBM/JETsAAM . . .

Viewer-centered Images

Object-centered Models

Face representation

A step back:Taxonomy of face recognition Algorithms

• Gordon et al., 1995 3D from 2D

• Lengagne et al., 1996 3D from stereo

• Atick et al., 1996 LFA

• Yan et al., 1996 Geometric modeling

• Zhao et al., 2000 Shape from Shading • Zhang et al., 2000 3D from video

October 30 2007 - IstanbulMassimo Tistarelli 34

Automatic face recognition

Face recognition involves several sequential processes:

Face detection

Facial featureslocalization

Registration/Representation

Classification

Normalization

October 30 2007 - IstanbulMassimo Tistarelli 35

Face detection

Sebastian Marcel - http://www.idiap.ch/~marcel/demos/comparison/opencv/alt/image_66-detect.jpg

October 30 2007 - IstanbulMassimo Tistarelli 36

The Viola-Jones face detector

• At present the most efficient face detector

implemented and largely adopted

• Publicly available within the Open-CV library

• Several improvements are being developed

P. Viola, M. J. Jones, "Robust Real-Time Face Detection", International Journal Computer Vision, 57(2), pp. 137-154, 2004.

October 30 2007 - IstanbulMassimo Tistarelli 37

Some improvements…

Improve localization accuracy by post-filtering the face region

October 30 2007 - IstanbulMassimo Tistarelli 38

Automatic face recognition

Face recognition involves several sequential processes:

Face detection

Facial featureslocalization

Registration/Representation

Classification

Normalization

October 30 2007 - IstanbulMassimo Tistarelli 39

THE Problem

Illumination compensation

Original ImageHistogram equalization

October 30 2007 - IstanbulMassimo Tistarelli 40

Image re-lighting

),(),(),( yxLyxRyxI ),(

),(),(

yxL

yxIyxR

dxdyLLdxdyyxIyxLyxLF yx )()),(),()(,()( 222

L(x,y)

(1)

?

Isotropic diffusion (Gaussian filtering)

Anisotropic diffusion (Lagrange solution of (1))

R. Gross and V. Brajovic, “An Image Preprocessing Algorithm for Illumination Invariant Face Recognition”, International Conference on Audio- and Video-Based Biometric Person Authentication, 2003.

D. Jobson, Z. Rahmann and G. Woodell, “A Multiscale Retinex for Bridging the Gap Between Color Images and the Human Observations of Scenes”, IEEE Transanctions on Image Processing, volume 6, Issue 7, 1997.

October 30 2007 - IstanbulMassimo Tistarelli 41

Automatic face recognition

Face recognition involves several sequential processes:

Face detection

Facial featureslocalization

Registration/Representation

Classification

Normalization

October 30 2007 - IstanbulMassimo Tistarelli 42

Understanding Facial features

Gray level oriented patterns/photometric properties

Physical Landmarks

October 30 2007 - IstanbulMassimo Tistarelli 43

Human face structure

Cranial circumferenceMax. cranial breadthMin. frontal breadthBigonial breadthUpper facial heightBasion-Prosthion lengthNasal breadth (max.)Lower nasal breadthOrbital breadthBiorbital breadthForamen magnum breadthCranial heightMax. cranial lengthBizygomatic breadthTotal facial heightBasion-Nasion lengthBasal heightUpper nasal breadthOrbital heightInterorbital breadthPalate-external breadth & lengthPalate-internal breadth & length…. (more)

Craniometric measurements (1)

October 30 2007 - IstanbulMassimo Tistarelli 44

Facial Region

L_eye R_eye Nose Mth L_ear R_ear Chn L_chk R_chk F_head

Me

an

Pe

rce

nt

En

tere

d

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Mean percentage of times (averaged across viewers) that each facial region was fixated at least once.

How to define facial features?

J.H. Henderson et al. “Gaze Control for Face Learning and Recognition by Humans and Machine”; in T. Shipley and P. Kellman (Eds.), From Fragments to Objects: Segmentation and Grouping in Vision

October 30 2007 - IstanbulMassimo Tistarelli 45

Facial features as 2D/3D landmarks

• 2D landmarks can be defined and tracked on face images

• Simple 2D vs complex 3D representations

October 30 2007 - IstanbulMassimo Tistarelli 46

• The LFA-based approach (Local Feature Analyisis) uses localized kernels, which are constructed from PCA-based eigenvectors, for extracting pre-defined topographic facial features (e.g., eyebrows, cheek, mouth, etc.)

Marked average face image

•Five topographic kernels are shown in the top row

•Five corresponding residual correlations (response) in the bottom row.

Facial features as 2D/3D landmarks

October 30 2007 - IstanbulMassimo Tistarelli 47

Facial features as 2D patterns

Gabor wavelets• Provide a description of the local

structure of the facial patterns

• Convolution with a bank of frequency-tuned filters

October 30 2007 - IstanbulMassimo Tistarelli 48

Local Binary Patterns (LBP)

• Pixels are labeled by thresholding the 3x3neighbourhood with the center value and considering the result as a binary number.

• The histogram of the labels is used as a texture descriptor.

Facial features as 2D patterns

October 30 2007 - IstanbulMassimo Tistarelli 49

Automatic face recognition

Face recognition involves several sequential processes:

Face detection

Facial featureslocalization

Registration/Representation

Classification

Normalization

October 30 2007 - IstanbulMassimo Tistarelli 50

Face displacement and perception

… … But the data is still thereBut the data is still there

October 30 2007 - IstanbulMassimo Tistarelli 51

In cognitive psychology it is called “perceptual organization”

The “registration” problem

October 30 2007 - IstanbulMassimo Tistarelli 52

The “registration” problem

S. Arca, P. Campadelli, and R. Lanzarotti. A face recognition system based on automatically determined facial fiducial points. Pattern Recognition, 39(3):432–443, 2006.

October 30 2007 - IstanbulMassimo Tistarelli 53

Automatic face recognition

Face recognition involves several sequential processes:

Face detection

Facial featureslocalization

Registration/Representation

Classification

Normalization

October 30 2007 - IstanbulMassimo Tistarelli 54

Linear Linear algorithmalgorithm

SVM, MPM, PCA, CCA, FDA…

DataData Embed dataEmbed data

x1

xn

if data described by numerical vectors: embedding ~ (non-linear) transformation

Kernel methods and learning

October 30 2007 - IstanbulMassimo Tistarelli 55

Support Vector Machines

Support Vector Machines are Support Vector Machines are binarybinary classifiers classifiers

Class 1Class 1

Class 2Class 2V. Vapnik, S.E. Golowich, A.J. Smola: Support Vector Method for Function Approximation, Regression Estimation and Signal Processing. Neural Information Processing Systems 1996: 281-287

October 30 2007 - IstanbulMassimo Tistarelli 56

The separating hyperplane can be very complex

Problems may occur with outliers

The shape of the hyperplane depends on the population of the classes

Need for an “impostor” class

Fk

Fh

ScSi

Support Vector Machines

October 30 2007 - IstanbulMassimo Tistarelli 57

One-Class Support Vector Machines are One-Class Support Vector Machines are unaryunary classifiers classifiers

One-Class Support Vector Machines

Class 1Class 1

ImpostorsImpostorsBen-Hur, A., Horn, D., Siegelmann, H., , Vapnik, V.: « Support vector clustering ». Journal of Machine Learning Research 2 (2001) 125–137

October 30 2007 - IstanbulMassimo Tistarelli 58

The separating surface is a hyperspehere

Selectivity can be adjusted by two parameters

No need for direct “impostor” training

Fk

Fh

ScSi

...122 iRaxi

One-Class Support Vector Machines

October 30 2007 - IstanbulMassimo Tistarelli 59

5 .5 0 %N O N EO n e - C la s s S V M

6 .1 9 %M i x e d d a ta s e t

6 .0 0 %Y a l e

5 .5 0 %S ti r li n g

6 .0 0 %B e rnB in a r y S V M

E q u a l E r r o r R a te

(E E R )I m p o s to r s e tM e t h o d

5 .5 0 %N O N EO n e - C la s s S V M

6 .1 9 %M i x e d d a ta s e t

6 .0 0 %Y a l e

5 .5 0 %S ti r li n g

6 .0 0 %B e rnB in a r y S V M

E q u a l E r r o r R a te

(E E R )I m p o s to r s e tM e t h o d

0 . 5 6 %O n e th r e s h o l d p e r

s u b j e c t

3 . 3 8 %

1 2 2 5 2 02 2 7 5

S in g l e th r e s h o l d

E q u a l E r r o r R a t e

( E E R )I m p o s t o r t e s t sC l ie n t t e s t sT h r e s h o ld

0 . 5 6 %O n e th r e s h o l d p e r

s u b j e c t

3 . 3 8 %

1 2 2 5 2 02 2 7 5

S in g l e th r e s h o l d

E q u a l E r r o r R a t e

( E E R )I m p o s t o r t e s t sC l ie n t t e s t sT h r e s h o ld

Poor Poor representationrepresentation

(10 frames/sbj)(10 frames/sbj)

Rich Rich representationrepresentation

(100 (100 frames/sbj)frames/sbj)

One-Class Support Vector Machines

October 30 2007 - IstanbulMassimo Tistarelli 60

Test Test ParameterFalse Reject

Rate False Accept

Rate

Fingerprint

FVC[2004]

Exaggerated distortion 2% 2%

FpVTE[2003]

US govt. operational data 0.1% 1%

Face

FRVT[2002]

Varied lighting, outdoor/indoor

10% 1%

FRGC[2006]

Time lapse, varied lighting/expression,

outdoor/indoor10% 0.1%

IrisITIRT

[2005] Indoor environment,

multiple visits0.99% 0.94%

VoiceNIST

[2004]Text independent, multi-

lingual5-10% 2-5%

Face recognition performances

October 30 2007 - IstanbulMassimo Tistarelli 61

Human vs Machine face recognition performances

A. O’Toole, J. Phillips, F. Jiang, Ayyad, Pénard & A. O’Toole, J. Phillips, F. Jiang, Ayyad, Pénard & AbdiAbdi, “Benchmarking Algorithms Against Humans” , “Benchmarking Algorithms Against Humans” IEEE:T-PAMI, 2007 IEEE:T-PAMI, 2007 (in press)(in press) Identity Matching for Easy Face Pairs

0

0.2

0.4

0.6

0.8

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False Accept Rate

Veri

ficati

on

Rate

CMU

NJIT

Algorithm B

Algorithm D

Viisage

Algorithm C

Algorithm A

Human Performance

Chance Performance

Identity Matching for Difficult Face Pairs

0

0.2

0.4

0.6

0.8

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False Accept Rate

Veri

ficati

on

Rate

NJIT

CMU

Viisage

Human Performance

Algorithm A

Algorithm B

Algorithm C

Algorithm D

Chance Performance

October 30 2007 - IstanbulMassimo Tistarelli 62

We are still HERE!

…Where is the “Mom’s face” neuron?

October 30 2007 - IstanbulMassimo Tistarelli 63

Automatic face recognition

WHERE DO WE GO NEXT?

Face detection

Facial featureslocalization

Registration/Representation

Classification

Normalization

October 30 2007 - IstanbulMassimo Tistarelli 64

Where do we go next?

ATTENTIVE AND DYNAMIC

PROCESSES

October 30 2007 - IstanbulMassimo Tistarelli 65

Are both from the same Are both from the same subject?subject?

What was missing?What was missing?

A simple recognition experiment…

October 30 2007 - IstanbulMassimo Tistarelli 66

Visual attention

Eye movements while watching a girl’s face (A.L. Yarbus, “Eye Movements and Vision”, Plenum Press, 1967)

October 30 2007 - IstanbulMassimo Tistarelli 67

The analysis of visual tracking patterns is sometimes misleading…

R.C. Miall, University Laboratory of Physiology, Oxford, UKJohn Tchalenko, Camberwell College of Arts, London, UK

Visual tracking

October 30 2007 - IstanbulMassimo Tistarelli 68

Subject-specific representation

For recognition we are not interested on what faces have in common but rather what differentiate one face from another.

For localization and tracking we are interested on what every face has in common (to tell a face from “non-faces”)

October 30 2007 - IstanbulMassimo Tistarelli 69

SVMfeature space

Distinctive patches

Face 1Face 2

Confusion: most similar patches

Distinctive patches

Fk

Fh

Bicego M., Brelstaff G., Brodo L., Grosso E., Lagorio A. and Tistarelli M. (2007) “Distinctiveness of faces: a computational approach”, ACM Transactions on Applied Perception, Vol. 5, n. 2, 2008.(in press)

Subject-specific representation

October 30 2007 - IstanbulMassimo Tistarelli 70

Three examples of differences extracted from pairs of images of the same person. The 100 most weighted patches are

shown

Subject-specific representation

October 30 2007 - IstanbulMassimo Tistarelli 71

A

A B

B

(A) perceptual and (B) computational test results of saliency

of local facial features. Red and green points in (A) represent the two positions selected by each of the 45 subjects (13 male, 32 female).

Subject-specific representation

October 30 2007 - IstanbulMassimo Tistarelli 72

(A) perceptual and (B) computational test results of saliency

of local facial features. Red and green points in (A) represent the two positions selected by each of the 45 subjects (13 male, 32 female).

A B

A B

Subject-specific representation

October 30 2007 - IstanbulMassimo Tistarelli 73

52 subjects from the BANCA database (26 M and 26 F).

Matched Controlled (MC) BANCA protocol.

The Weighted Error Rate (WER) is computed from the threshold from G1 on G2 (26 subjects each).

Image areas are selectively deleted from the face image.

Authentication test

October 30 2007 - IstanbulMassimo Tistarelli 74

Authentication test

Quantitative results by selectively masking image areas

October 30 2007 - IstanbulMassimo Tistarelli 75

Comparative tests

Lowe [2004] (SIFT),

Salah et al. [2002],

Walther [2006],

Ullman et al. [2002].

Authentication test

October 30 2007 - IstanbulMassimo Tistarelli 76

Hidden Markov Models

Statistical analysis of sequences of patterns

… …

October 30 2007 - IstanbulMassimo Tistarelli 77

Hidden Markov Models

Modelling (learning) sequences of symbols

– capture the underlying structure of a set of symbol strings

– stochastic generalisation of finite-state automata, where both transitions between states and generation of output symbols are governed by probabilistic distributions

Markovian Models, where states are not directly observable

– a probability function is associated to each state describing the probability that a symbol is emitted from this state.

October 30 2007 - IstanbulMassimo Tistarelli 78

Coding– the image is divided into a sequence of T sub-images

– for each sub-image the DCT coefficients are computed

– only the most important D coefficients are retained

– the final sequence is composed by DxT symbols

Learning : train one HMM for each subject:– the number of states is fixed a priori

– at the end we have one HMM for each subject

Recognition : – Bayesian scheme: assuming a priori equi probable classes, the object

is assigned to the highest likelihood class

Hidden Markov Models

October 30 2007 - IstanbulMassimo Tistarelli 79

Recognition and selective attention

Starting point: HMM based classification of faces

“Walking on the face” for obtaining HMM sequences

Standard raster scan-path Saliency-based scan-path

Attention drives face

scanning

October 30 2007 - IstanbulMassimo Tistarelli 80

Model of attention-based classification

Inhibition of Return

Face sampleSaliency Master Map Feature

MapsNormalization and Integration

Hidden Markov Model

Fovea

'What'Information

Feature Extraction

content

location 'Where'Information

Associative Level

Attentive Level

classification

October 30 2007 - IstanbulMassimo Tistarelli 81

Experiments on BANCA protocol MC

Gabor wavelets for saliency map construction

Employed features: gray levels, DCT coefficients, Haar

wavelets

A. A. Salah, M. Bicego, L. Akarun,  E. Grosso, M. Tistarelli: "Hidden Markov model-based face recognition using selective attention", Human Vision and Electronic Imaging XII, Proc. of SPIE, vol. 6492, (2007)

Authentication test

October 30 2007 - IstanbulMassimo Tistarelli 82

About 40% of the saccades are sufficient to obtain the same result of raster scan

In some cases further saccades may decrease the classification performances

The system can be improved by exploiting also the where information

Authentication test

October 30 2007 - IstanbulMassimo Tistarelli 83

Face recognition from video

Dynamics in a video stream conveys far more Dynamics in a video stream conveys far more information than a collection of single snapshotsinformation than a collection of single snapshots

October 30 2007 - IstanbulMassimo Tistarelli 84

Individual motion can be highly distinctiveIndividual motion can be highly distinctive

Face recognition from video

October 30 2007 - IstanbulMassimo Tistarelli 85

… … and even more distinctiveand even more distinctive

Face recognition from video

October 30 2007 - IstanbulMassimo Tistarelli 86

Face recognition from video

Advantages from video:

• More data available• Temporal integration• Behavioral cues• Spatial and temporal sampling

October 30 2007 - IstanbulMassimo Tistarelli 87

Effects of motion on representation

Facerotation

Facescaling

Facetranslation

Face sub-spaces manifold

October 30 2007 - IstanbulMassimo Tistarelli 88

The curse of dimensionality…

…the risk is to have too much data to be processed

How to exploit the added information in video?How to exploit the added information in video?

Face recognition from video

Standard VGA (640x480)

1 frame: 300 KByte30 frames: 1 MByte

Standard video: ~1 MByte/Sec

October 30 2007 - IstanbulMassimo Tistarelli 89

Not just more data to be processed:

Data selection (pose, expression, illumination, noise…)

Multi-data fusion (decision/score/feature level)

3D reconstruction/virtual views

Resolution enhancement

Expression and emotion analysis

Behavioral analysisBehavioral analysis

DynamicDynamic video templates video templates…?

Face recognition from video

October 30 2007 - IstanbulMassimo Tistarelli 90

A. O’Toole, D.A. Roark, and H. Abdi. “ Recognizing moving faces: A psychological and neural synthesis”.Trends in Cognitive Science, 6:261--266, 2002.

Neural architecture of face perception - 2

October 30 2007 - IstanbulMassimo Tistarelli 91

How to represent photometric and dynamic How to represent photometric and dynamic information at the same time?information at the same time?

Face recognition from video

October 30 2007 - IstanbulMassimo Tistarelli 92

Hidden Markov Models

Statistical analysis of sequences of patterns

… …

October 30 2007 - IstanbulMassimo Tistarelli 93

Standard HMMs can be extended to multi-dimensional patterns and sequences … in many ways

1. Each image is modeled as a single HMM and the sequence of images as a sequence of HMMs (A. Hadid and M. Pietikainen. “An experimental investigation about the integration of facial dynamics in video-based face recognition”. Electronic Letters on Computer Vision and Image Analysis, 5(1):1-13, 2005.)

2. The entire video is modeled as a single HMM

(X. Liu and T. Chen. “Video-based face recognition using adaptive hidden Markov models”. In Proc. Int. Conf.

on Computer Vision and Pattern Recognition, 2003.)

3. The images and the sequence itself are modeled as a complex, hierarchical HMM-based structure (M. Bicego, E.Grosso, M. Tistarelli. “Person authentication from video of faces: a behavioral and physiological approach using Pseudo Hierarchical Hidden Markov Models”, Int.l Conference on Biometric Authentication 2006, Hong Kong, China, January 2006. )

Dynamic Hidden Markov Models

October 30 2007 - IstanbulMassimo Tistarelli 94

Physiological and behavioral featuresPhysiological and behavioral features

HMMHMM

PH-HMMPH-HMM

Pseudo Hyerarchical HMM

October 30 2007 - IstanbulMassimo Tistarelli 95

Pseudo-Hierarchical Pseudo-Hierarchical Hidden Markov ModelHidden Markov Model

The image stream is represented as a collection of sub-streams, each corresponding to a different facial expression or motion.

The change in expression is capturede by the transition matrix of the PH-HMM

Pseudo Hyerarchical HMM

October 30 2007 - IstanbulMassimo Tistarelli 96

Recogniton from video: performances

Classification Results (recognition rate)Still images: one HMM for class 11/21 (52.38%)

Still images: one HMM for cluster of class 14/21 (66.67%)

Still images: one HMM for image 12/21 (57.14%)

Video: Pseudo Hierarchical HMM 21/21 (100%)_________________________________________

Verification Results (EER)Client tests: 20 - Impostor tests: 420

Still images: one HMM for class 20.24%

Still images: one HMM for cluster of class 10.60%

Still images: one HMM for image 13.81%

Video: Pseudo Hierarchical HMM 6.07 %

M. Bicego, E. Grosso, M. Tistarelli: "Person Authentication from video of faces: a behavioral and physiological approach using Pseudo Hierarchical Hidden Markov Models" . Int. Conf. On Biometrics (ICBA06) , LNCS 3832, D. Zhang, A.K. Jain Eds, pp 113-120, (2006)

October 30 2007 - IstanbulMassimo Tistarelli 97

Industrial FR systemsA4Vision, Inc. (3D scanner)

AcSys Biometrics Corp.

Animetrics Inc. (3D shape)

C-VIS Computer Vision und Automation GmbH (Neural Networks)

Cognitec Systems GmbH (LFA)

Cybula Ltd. (3D shape/2D texture)

DreamMirh Co., Ltd. (2D ??? )

Geometrix, Inc. (3D shape)

Iconquest (2D Fractal-based ??? )

Identix Inc. (LFA)

Imagis Technologies Inc. (VISIPHOR Advanced Face Recognition ???)

Neven Vision, Inc. (2D Gabor wavelets; feature-based )

Takumi Vision Technologies, Inc. (Hi-Tech algorithms ??? )

Viisage (Template matching ??? )

VisionSphere Technologies Inc. (2D features with Holistic Feature Code)

Source: www.face-rec.org

October 30 2007 - IstanbulMassimo Tistarelli 98

Near-infra red imaging

Authentimetric F1

October 30 2007 - IstanbulMassimo Tistarelli 99

Conclusion

• Many algorithmic and system solutions have been proposed.

• More investments needed.

• More basic research on open issues.

• Need for “universal” databases for evaluation and testing on key applications.

October 30 2007 - IstanbulMassimo Tistarelli 100

New technologies for Security and Privacy

Contact: tista@uniss.it

http://biometrics.uniss.it

5th Int.l Summer School for Advanced Studies on

Biometrics for secure authentication:

Alghero, Italy - June, 9 – 13 2008

October 30 2007 - IstanbulMassimo Tistarelli 101

Contact: tista@uniss.it

http://icb07.uniss.it

3rd IAPR/IEEE Int.l Conference onBiometrics

Alghero, Italy - June, 2 – 5 2009

top related