how to evaluate accuracy of biometric systems
TRANSCRIPT
Intro
● PeWe member○ 2006 - 2010
● Innovatrics○ fingerprint-based biometrics
■ SDKs■ large-scale fingerprint matching (AFIS)■ end-products
○ 300M+ people
Typical Biometric Modalities
● Face● Iris● Retina● Signature● …● Fingerprints
Examples of Biometric Systems
● National ID● Social Insurance● Health Insurance● Border Control● Driving Licenses● Voter’s Lists● Ghost Workers
Identity Management Systems
Accuracy
● Enrollment:○ FTE: Fail to Enroll
● Verification:○ FMR: False Match Rate○ FNMR: False Non-Match Rate
● Identification:○ FPIR: False Positive Identification Rate○ FNIR: False Negative Identification Rate
AFIS
● Automated Fingerprint Identification System○ CAFIS, MegaMatcher, ExpressID AFIS
● 100 000 000+ fingerprint comparisons / second / CPU
How to Compute Accuracy1. Enroll 1000 different records for the first timeEvery record must be unique, we label them A = {a1, a2, …, a1000}2. Enroll same people againWe label them B = {b1, b2, …, b1000}. We know that a and b with the same index are from the same person3. Perform verification of all records from A against all records from BIn total we will have 1 million matching results for every pair.4. Analyze 1000 scores having the same indexThis is called genuine distribution.5. Analyze 999 000 scores having different indexThis is called impostor distribution.6. Calculate FNR and FNMR for different scores to get ROC curve
How to Compute Accuracy
Should: Accept Should: Reject
Reality: Accepted TA FA
Reality: Rejected FR TR
How to Compute Accuracy
Should: Accept Should: Reject
Reality: Accepted TA (1000) FA (0)
Reality: Rejected FR (0) TR (999 000)
The false non-match rate is the expected probability that Ai will be falsely declared not to match to Bi.FNMR = FR / (FR + TA) = 0 : 1000
How to Compute Accuracy
Should: Accept Should: Reject
Reality: Accepted TA (999) FA (0)
Reality: Rejected FR (1) TR (999 000)
The false non-match rate is the expected probability that Ai will be falsely declared not to match to Bi.FNMR = FR / (FR + TA) = 1 : 1000
How to Compute Accuracy
Should: Accept Should: Reject
Reality: Accepted TA (1000) FA (0)
Reality: Rejected FR (0) TR (999 000)
The false match rate is the expected probability that a sample will be falsely declared to match a single randomly-selected “non-self”.FMR = FA / (FA + TR) = 0 : 999 000
FNMR FMR
Similarity score
ROC CurveFNMR
FMR
Examples of Real-life Accuracies
● iPhone 5S○ Verification, 1 finger, FMR 1:50 000
● Time Attendance System○ Verification, population 10-1000, 1 finger, FMR < 1:1000
● Population 4.5M, 6 fingers○ Identification, FPIR < 1:100 000, FNIR < 2% (1:50)
How to Influence Accuracy
● Threshold○ Security vs. comfort
● Fingerprints○ how many, which positions○ quality○ position anonymization
● Template extractor● Matching speed● Discriminative ability of bio. modality
○ Dataset size ~ FMR
Customer and Accuracy
● Not aware● Aware, but ignoring● Cooperating● Demanding
Datasets
● physical access● huge difference in accuracy due to quality of fingerprints● annotated datasets
Independent Accuracy Tests
● NIST PFT○ Proprietary Fingerprint Template Evaluation○ Verification
● NIST FpVTE○ Fingerprint Vendor Technology Evaluation○ Identification
● NIST MINEX○ Minutia Exchange
NIST PFT II
Resources
● INDIA UID● Introduction to Biometrics
○ Springer, 2011● Best Practices in Testing and Reporting Performance of Biometric Devices
○ http://ftp.sas.ewi.utwente.nl/open/courses/intro_biometrics/Mansfield02.pdf
Other Keywords
● Deterrence effect● Fingerprint quality (NFIQ)● Speed● Template extraction
○ basic pattern, minutiae points, pattern● Segmentation● ABIS● Positive/Negative identification● Criminal/Civil AFIS● India UID, Indonesia eKTP● iPhone● FAR, FRR