robust low-resolution face identification and verification
TRANSCRIPT
![Page 1: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/1.jpg)
Robust Low-Resolution Face Identification and Verification Using
High-Resolution Features
Dr. Pablo H. Hennings Yeomans1, Prof. B.V.K. Vijaya Kumar1, Dr. Simon Baker2
1. ECE Department, Carnegie Mellon University2. Microsoft Research, Microsoft Corporation
![Page 2: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/2.jpg)
2
![Page 3: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/3.jpg)
3
![Page 4: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/4.jpg)
Surveillance and Forensics
• In many surveillance scenarios people may be far from the camera and their faces may be smallsmall.
• Looking for suspects involves parsing through hundreds ofparsing through hundreds of hours of video.
• A terrorist crime was solved in Italy in 2002 by analysis of 52,000 hours of surveillance videos installed in rail stations.
6
![Page 5: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/5.jpg)
Problem
• As media usage moves from images to video, low-resolution content is now more common than before.
• The problem from the biometrics perspective is the mismatch between training and testing resolutions.
Training Images Probe Images
8
![Page 6: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/6.jpg)
Outline
• Conventional approaches• Proposed framework• Generalizations that improve recognition• Generalizations that improve recognition• Experiments and Results• Conclusions
10
![Page 7: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/7.jpg)
Face Recognition ofFace Recognition of Low-resolution Images
• Classification degrades considerably as the resolution of probe faces decreases to very low resolutions
LR
HigherError Rates
0
11
Higher resolutions0
LR
![Page 8: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/8.jpg)
Reconstruction of Low-resolution Images
• We could try to reconstruct the face image• Reconstruction algorithms find estimates of missing HR pixels in a LR
image by assuming a model
LR HRLR HR
12
![Page 9: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/9.jpg)
Super-Resolution with Face Priors
* From hallucinating faces software distribution
13S. Baker and T. Kanade, “Limits on Super-Resolution and How to Break Them”,IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 9, September, 2002, pp. 1167 - 1183.
![Page 10: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/10.jpg)
The Image Formation Process
• Inverting the image formation process
high-resolution image
low-resolution image warp matrix
detector noise
low resolution image warp matrixblur matrix
decimation matrix
• Super-resolution algorithms aim to invert this process either directly or indirectly.
S f l
14A. Zomet and S. Peleg, “Super-Resolution from Multiple Images having Arbitrary Mutual Motion”, in S. Chaudhuri (Editor), Super-Resolution Imaging, Kluwer Academic, Sept. 2001, pp. 195-209.
See, for example:
![Page 11: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/11.jpg)
Possible Solutions
1. We can apply a super-resolution algorithm and then classify the result
2. We can downsample the gallery image and match at the resolution of the proberesolution of the probe
16
![Page 12: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/12.jpg)
Proposed Solution: S2R2
1. We can apply a super-resolution algorithm and then classify the result
2. We can downsample the gallery image and match at the resolution of the proberesolution of the probe
• We propose an alternative approach which jointly uses pp j ysuper-resolution methods and includes face features for recognition (S2R2)
17
![Page 13: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/13.jpg)
S2R2 Block Diagram
18
![Page 14: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/14.jpg)
S2R2 Simultaneous fit
• Minimize the regularized functional given by Index of class to match
Low-res probeImage formation matrix
First-derivativesmatrix
Feature extractionmatrix
Features from galleryimage
Super-resolution models Feature extraction
• Regularization parameters are trained to produce distortions that are discriminatory
model
19
discriminatory
![Page 15: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/15.jpg)
20
![Page 16: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/16.jpg)
21
![Page 17: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/17.jpg)
S2R2 Classification Main Idea
• By computing measures of fit from these simultaneous-fit results we extract new featuresextract new features.
• A binary classifier is trained h th t th fsuch that the scores of
true-fits and false-fits using the new features are as separate as possible.p p
• Here, a linear discriminant is used.
22Scores
![Page 18: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/18.jpg)
S2R2 Classification
• Compute measures-of-fit norms and form a new feature vector rk
• Classify with rk using conventional classification methods
23
classification methods
![Page 19: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/19.jpg)
Verification with S2R2
• In verification a claimed class is input with the LR probe
24
![Page 20: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/20.jpg)
Scoring with S2R2
• In general, a classifier can output a score instead of a hard decision
25
![Page 21: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/21.jpg)
Identification with S2R2
• In identification the input probe is scored against K classes
26
![Page 22: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/22.jpg)
S2R2 Implementation
• The simultaneous fit of S2R2 is an LS problem:
• Typically solved using iterative methods
• But for every LR input image y, and for every claimed class k we have to solve a simultaneous-fit and extract measures-of-fit f tfeatures
We need to dothousands or millionsof comparisons
27
of comparisons.
![Page 23: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/23.jpg)
Simultaneous Fit Using SVD methods
• Let
• Then we can write the reconstructed image as
• This becomes
28Can be pre-computed
before testing this probe image against each classifier
![Page 24: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/24.jpg)
Finding Regularization ParametersFinding Regularization Parametersand Final Classifier
• In training, all the regularization parameters are found simultaneously with the final classifier w
⟨Final w
®
At every step compute error metric of classifier to rank each (⟨ ®)
30
At every step, compute error metric of classifier to rank each (⟨, ®)
![Page 25: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/25.jpg)
S2R2 is a General Framework
• It consists of a simultaneous fit formulation
• And measures-of-fit features and classifierVerification
Identification
32
![Page 26: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/26.jpg)
Using Features at Multiple ResolutionsUsing Features at Multiple Resolutions (MFS2R2)
• Since we know the image formation process
• Features defined asF FF FL
B
33
Gallery image form class k
![Page 27: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/27.jpg)
Simultaneous Fits Using Face Priors
• In training, we can compute super-resolution priors for every gallery image
FL FL
34
Gallery image form class k
![Page 28: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/28.jpg)
S2R2 Features Using RelativeS R Features Using Relative Residuals
• Using a simultaneous fit given by
• Normalization of features helps the final classification stage
• We refer to this version of the proposed framework as S2R2e
35
• We refer to this version of the proposed framework as S2R2e
![Page 29: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/29.jpg)
Database
• The MultiPIE database– Total of 337 subjects, compared to 68 of PIE– Subjects are captured in several recording sessions with different poses,
illuminations and expressions, as in PIE
• Data set for experiments– Using frontal view, neutral expressions, different flash illuminations– Sequestered 73 subjects as generic training setq j g g– Sequestered 40 subjects to learn regularization parameters– The rest 224 subjects are used as gallery and probes– Gallery images are not under flash illumination– We use only 1 image per gallery classWe use only 1 image per gallery class– There are a total of 2912 probe images
38
![Page 30: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/30.jpg)
Sample from Multi-PIE
Original
24x24
12x12
6x6
39
![Page 31: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/31.jpg)
Experimental Setup
• Proposed framework settings– Using a single input (still image) probe– The image formation process is assumed known– Using smoothness constraints as first derivatives– Face-features are 25 Fisherfaces– Final classifier is a Fisher discriminant
T i i l ti– Training resolution is 24x24 pixels
– S2R2 using face-features at training resolution
Identification Accuracy (%)
and probe resolution.
40pixels
![Page 32: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/32.jpg)
MFS2R2e Results Using Multi-PIE (I)
• Magnification factor of 2
44
![Page 33: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/33.jpg)
MFS2R2e Results Using Multi-PIE (II)
• Magnification factor of 4
45
![Page 34: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/34.jpg)
ROC Curves
M ifi ti f t f 4M ifi ti f t f 2 Magnification factor of 4Magnification factor of 2
45
![Page 35: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/35.jpg)
Conclusions• Before our work, there were two conventional approaches for recognition of low-pp g
resolution faces, both with clear disadvantages.
• Most of the effort in related works focuses on image reconstruction using super-resolution, with face recognition addressed only as an afterthought.
N ith th d S2R2 f k l ti th d b• Now, with the proposed S2R2 framework, super-resolution methods can be tailored to face recognition, rather than just reconstruction.
• The proposed framework can use existing super-resolution and feature extraction models. It extracts new features by finding a template that fits i l l i h il bl d l d f fsimultaneously into the available models and face-features.
• We have shown that with simple linear discriminants using these new (measures-of-fit) features we can produce better recognition performance than conventional approaches.
• Finally, the proposed framework is flexible. Our formulation can be easily expanded or generalized to use video, multiple cameras, and even other image representations (such as wavelets) and non-linear features.
54
![Page 36: Robust Low-Resolution Face Identification and Verification](https://reader030.vdocuments.net/reader030/viewer/2022032806/623ec629e4dff439531bd681/html5/thumbnails/36.jpg)
Thank youThank you