face recognition vendor test frvt 2018 performance of ... · pdf fileface recognition vendor...
TRANSCRIPT
FaceRecognitionVendorTestFRVT2018
PerformanceofAutomatedFaceIdentificationAlgorithmsConcept,EvaluationPlanandAPI
VERSION0.5
THISDOCUMENTISOPENFORCOMMENTS:[email protected]
PatrickGrotherMeiNgan
KayeeHanaokaInformationAccessDivision
InformationTechnologyLaboratory
October26,2017
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page2of18
TableofContents11. FRVT2018.........................................................................................................................................................................32
1.1. Scope......................................................................................................................................................................331.2. Audience.................................................................................................................................................................341.3. Schedule.................................................................................................................................................................351.4. Reporting................................................................................................................................................................361.5. VersionControl.......................................................................................................................................................371.6. Background.............................................................................................................................................................481.7. FRVT2018:Changesfrompriorevaluations...........................................................................................................491.8. Relationtothe1:1FRVTevaluation.......................................................................................................................4101.9. Coreaccuracymetrics.............................................................................................................................................4111.10. Applicationrelevance.............................................................................................................................................512
2. Rulesforparticipation......................................................................................................................................................5132.1. Participationagreement.........................................................................................................................................5142.2. Validation................................................................................................................................................................5152.3. Hardwarespecification...........................................................................................................................................5162.4. Operatingsystem,compilation,andlinkingenvironment......................................................................................6172.5. Softwareanddocumentation.................................................................................................................................6182.6. Runtimebehavior...................................................................................................................................................7192.7. Timelimits..............................................................................................................................................................8202.8. Templatesizelimits................................................................................................................................................821
3. DatastructuressupportingtheAPI..................................................................................................................................9223.1. Requirement...........................................................................................................................................................9233.2. Fileformatsanddatastructures.............................................................................................................................924
4. APIspecification.............................................................................................................................................................13254.1. Namespace...........................................................................................................................................................13264.2. Overview...............................................................................................................................................................13274.3. API.........................................................................................................................................................................1528
29ListofTables30
Table1–Scheduleandallowednumberofsubmissions.........................................................................................................331Table3–Processingtimelimitsinseconds,per640x480colorimage,onasingleCPU.......................................................832Table4–Structureforasingleimage......................................................................................................................................933Table5–LabelsdescribingcategoriesofImages.....................................................................................................................934Table6–Structureforasetofimagesfromasingleperson...................................................................................................935Table7–Structureforapairofeyecoordinates...................................................................................................................1036Table8–Labelsdescribingtemplaterole..............................................................................................................................1037Table9–Enrollmentdatasettemplatemanifest...................................................................................................................1138Table10–Labelsdescribinggallerycomposition..................................................................................................................1239Table11–Structureforacandidate......................................................................................................................................1240Table12–Enumerationofreturncodes................................................................................................................................1241Table13–ReturnStatusstructure.........................................................................................................................................1342Table16–Proceduraloverviewofthe1:Ntest.....................................................................................................................1343Table17–Templatecreationinitialization............................................................................................................................1544Table19–Enrollmentfinalization..........................................................................................................................................1645Table21–Identificationsearch.............................................................................................................................................1746Table23–Insertionoftemplateintoagallery......................................................................................................................1847Table24–Removaloftemplatefromagallery.....................................................................................................................184849
50
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page3of18
1. FRVT201851
1.1. Scope52Thisdocumentestablishesaconceptofoperationsandanapplicationprogramminginterface(API)forevaluationofone-53to-manyfacerecognitionalgorithmsappliedtofacesappearingin2Dstillphotographs.Theprimaryfocusofthetestis54cooperativeportraitimages,e.g.mugshots1.Thetestwillincludealsosearchofnon-cooperativeimages.TheAPI55communicatesthetypeofimagetothealgorithm.56
1.2. Audience57
Participationisopentoanyorganizationworldwide,primarilyresearchersanddevelopersofFRalgorithms.WhileNIST58intendstoevaluatestabletechnologiesthatcouldbereadilymadeoperational,thetestisalsoopentoexperimental,59prototypeandothertechnologies.AllalgorithmsmustbesubmittedasimplementationsoftheAPIdefinedinthis60document.Thereisnochargeforparticipation.61
1.3. Schedule62InconsultationwithUSGovernmentcollaborators,NISTwillexecutetheFRVT2018ontheschedulegiveninTable1.Note63thatNISTwillreportresultspubliclyattheendofPhases2and3.TheendofPhase2correspondscloselytotheendof64thefinancialyearandresultsfromtheFRVTarerequiredbythatdate.65
Developersmaysubmitthenumberofalgorithmsidentifiedintherightmostcolumn.66
Table1–Scheduleandallowednumberofsubmissions67Phase Date Milestone Maximumnumberof
implementationsAPIDevelopment
2017-10-26 Draftevaluationplanavailableforpubliccomments 2017-11-16 Finalevaluationplanpublished
Phase1
2018-01-22 Participationstarts:AlgorithmsmaybesenttoNIST 2018-02-16 LastdayforsubmissionofalgorithmstoPhase1 32018-05-23 Interimresultsreleased
Phase2
2018-06-21 LastdayforsubmissionofalgorithmstoPhase2 22018-09-20 Resultsreleased
Phase3
2018-10-19 LastdayforsubmissionofalgorithmstoPhase3 22018-12-19 Releaseoffinalpublicreport
1.4. Reporting68AttheconclusionofPhase1,NISTwillprovideresultsforallalgorithmstoalldevelopers,andtoUSGovernmentpartners.69
AttheconclusionofPhase2,NISTwillpublishresultsforallalgorithmsonitswebsite.70
AttheconclusionofPhase3,NISTwillpublishanInteragencyReportsummarizingtheentireFRVT1:N.71
Important:ThisisanopentestinwhichNISTwillidentifythealgorithmandthedevelopingorganization.Algorithm72resultswillbeattributedtothedeveloper.Resultswillbemachinegenerated(i.e.scripted)andwillincludetiming,73accuracyandotherperformanceresults.Thesewillbepostedalongsideresultsfromotherimplementations.74
NISTmayadditionallyreportresultsinworkshops,conferences,conferencepapersandpresentations,journalarticlesand75technicalreports.76
1.5. VersionControl77Developersmustsubmitaversion.txtfileinthedoc/folderthataccompaniestheiralgorithm–seeSection2.5.3.The78stringinthisfileshouldallowthedevelopertoassociateresultsthatappearinNISTreportswiththesubmittedalgorithm.791TheMEDSdatabaseincludessamplemugshots.
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page4of18
Thisisintendedtoallowend-userstoobtainproductizedversionsoftheprototypessubmittedtoNIST.NISTwillpublish80thecontentsofversion.txt.NISThaspreviouslypublishedMD5hashesofthecorelibrariesforthispurpose.81
1.6. Background82NISThasconductedevaluationsoffacerecognitionprototypessincethefirstFRVTin2000.Until2006,thosetrials83simulatedone-to-manysearchaccuracyusingsetsofone-to-onecomparisons.In2010,insupportoftheFBI’s84procurementofafacesearchcapability,NISTreported[NISTIR7709]accuracyandspeedofend-to-endone-to-many85searchimplementationswithenrolledpopulationsupto1.8million.Usingupdatedalgorithms,thattestwasrepeated86andextendedin2013[NIST8009].Thosetestsfocusedoncooperativeportraitimages.Inlate2017,NISTwillpublishthe87resultsfromtheFaceRecognitionPrizeChallengewhichassessedcapabilityofcontemporarysearchalgorithmsonless88constrainedimages[FRPC2017].89
1.7. FRVT2018:Changesfrompriorevaluations90Givenmassivechangesinfacerecognitionsincethelastone-to-manyevaluationin2013,NISTseekstoassessbenefits91thathaveaccruedtotheuseofthosetechnologieswithcooperativeimages.92
Thefollowingarenewaspects:93
― Effectofincreasedpopulationsize:Nisexpectedtoexceed107representinganorderofmagnitudeincreasinginthe94numberofuniquefaces.95
― Accuracywithencounter-basedgalleries:PriorNISTevaluationsconstructedgalleriesinwhichallknownimagesof96anenrolleewereprovidedtothealgorithmtogether,underasingleidentity.Thissupportedalgorithmswhichmight97fuseimagesortemplates.However,thissubject-basedenrollmentdiffersfromthat(minorityof)operational98deployments–encounter-basedapplications-inwhichmultipleimagesofapersonarepresentinadatabase99withoutanyspecifiedidentityorlink.Thistestwillcompareaccuracyforbothsubject-basedandencounter-based100galleries.Section3.2.6containsnarrativeandAPIsupporttoinformthealgorithmofthegallerytype.101
― Expenseofdeletionandinsertionfunctions:PriorNISTevaluationsconstructedagallerywhichwassearched102withoutanymodification.Thisfacilitatedmeasurementofaccuracyandspeedbutdidnotallowformeasurementof103computationalexpenseofaddingordeletingelementsfromthegallery.Thesefunctionsmaynotbetrivial,for104example,iftheunderlyingimplementationusesfast-searchdatastructures.105
1.8. Relationtothe1:1FRVTevaluation106SinceFebruary2017,NISThasbeenrunninganongoingevaluationofone-to-onefaceverificationalgorithms,FRVT1:1.107Thisallowsanydevelopertosubmitalgorithmsatanytime,onceeverythreecalendarmonths,therebybetteraligning108developmentandevaluationschedules.TheFRVT1:1includessixdifferentdatasets,oneofwhichismugshots,similarto109theprimarysetproposedforinclusionontheFRVT1:Ndefinedherein.Itmaybenefitdeveloperstosubmittheircore1:1110recognitionalgorithmstotheFRVT1:1processbytheFRVT1:Ndeadline.Ifsuchcpu-basedalgorithmsaresubmittedby111January22,wewillpublishresultsbeforethe1:Ndeadline.112
ParticipationintheFRVT1:1isnotrequiredforparticipationinFRVT1:N.113
1.9. Coreaccuracymetrics114Thistestwillexecuteopen-universesearches.Thatis,someproportionofsearcheswillnothaveanenrolledmate.From115thecandidatelistsreturnedbyalgorithms,NISTwillcomputeandreportaccuracymetrics,primarily:116
― Falsenegativeidentificationrate(FNIR)–theproportionofmatedsearcheswhichdonotyieldamatewithinthetop117Rranksandatorabovethreshold,T.118
― Falsepositiveidentificationrate(FPIR)–theproportionofnon-matedsearchesreturningany(1ormore)candidates119atoraboveathreshold,T.120
― Selectivity–thenumberofnon-matedcandidatesreturnedatoraboveathreshold,T.Thisquantityhasavalue121runningfrom0toL,thenumberofcandidatesrequested.Itmaybefractional,asitisestimatedasacountdividedby122thenumberofnon-matesearches.123
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page5of18
ThesequantitiesareestimatedfromcandidatelistsproducedbyrequestingthetopLmostsimilarcandidatestothe124search.Wedonotintendtoexecutesearchesrequestingonlythosecandidatesaboveaspecifiedinputthreshold.125
WewillreportFNIRandFPIRbysweepingthethresholdovertheinterval[0,infinity).Errortradeoffplots(FNIRvs.FPIR,126parametriconthreshold)willbetheprimaryreportingmechanism.127
WewillreportalsoFNIRbysweepingarankRovertheinterval[1,L]toproduce(thecomplementof)thecumulative128matchcharacteristic(CMC).129
1.10. Applicationrelevance130NISTanticipatesreportingFNIRintwoFPIRregimes:131
― Investigationmode:Givencandidatelistsandathresholdofzero,theCMCmetricisrelevanttoinvestigational132applicationswherehumanexaminerswilladjudicatecandidatesindecreasingorderofsimilarity.Thisiscommonin133lawenforcement“leadgeneration”.134
― Identificationmode:Wewillapply(high)thresholdstocandidatelistsandreportFNIRvaluesrelevantto135identificationapplicationswherehumanlaborismatchedtothetolerablenumberoffalsepositivesperunittime.136Thisisusedinduplicatedetectionsearchesforcredentialissuanceand,moreso,insurveillanceapplications.137
Giventhatmultiplealgorithmsmaybesubmitted,developersareencouragedtosubmitvariantstailoredtominimizeFNIR138inthetwoFPIRregimes,andtoexplorethespeed-accuracytradespace.139
2. Rulesforparticipation140
2.1. Participationagreement141Aparticipantmustproperlyfollow,complete,andsubmittheFRVTParticipationAgreement.Thismustbedoneonce,142eitherpriororinconjunctionwiththeveryfirstalgorithmsubmission.Itisnotnecessarytodothisforeachsubmitted143implementationthereafter.144
NOTE OrganizationsthathavealreadysubmittedaparticipationagreementforongoingFRVT1:1donotneedtosend145inanewparticipationagreement,astheirsubmissionofa1:Nalgorithmindicatestheyagreetothesametermsand146conditionsarticulatedinthatagreement.147
NOTE Ifanorganizationupdatestheircryptographicsigningkey,theymustsendanewcompletedparticipation148agreementsubmissionforthisevaluation,withthefingerprintoftheirpublickey.149
150
2.2. Validation151AllparticipantsmustruntheirsoftwarethroughtheprovidedFRVT1:Nvalidationpackagepriortosubmission.The152validationpackagewillbemadeavailableathttps://github.com/usnistgov/frvt.Thepurposeofvalidationistoensure153consistentalgorithmoutputbetweentheparticipant’sexecutionandNIST’sexecution.154
2.3. Hardwarespecification155NISTintendstosupporthighperformancebyspecifyingtheruntimehardwarebeforehand.Thereareseveraltypesof156computerbladesthatmaybeusedinthetesting.Eachmachinehasatleast192GBofmemory.Weanticipatethat16157processescanberunwithouttimeslicing,thoughNISTwillhandleallmultiprocessingworkviafork()2.Participant-158initiatedmultiprocessingisnotpermitted.159
Allimplementationsshalluse64bitaddressing.160
NISTintendstosupporthighlyoptimizedalgorithmsbyspecifyingtheruntimehardware.Thereareseveraltypesof161computersthatmaybeusedinthetesting.Thefollowinglistgivessomedetailsaboutpossiblecomputearchitectures:162
2http://man7.org/linux/man-pages/man2/fork.2.html
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page6of18
― DualIntel®Xeon®X56903.47GHzCPUs(6coreseach)3163
― DualIntel®Xeon®[email protected](10coreseach)4164
― DualIntel®Xeon®[email protected](14coreseach)4165
ThistestwillnotsupporttheuseofGraphicsProcessingUnits(GPUs).TheFRVT1:1activity,whichremainsopen,166documentsrelativeGPUvsCPUspeed.167
2.4. Operatingsystem,compilation,andlinkingenvironment168
Theoperatingsystemthatthesubmittedimplementationsshallrunonwillbereleasedasadownloadablefileaccessible169fromhttp://nigos.nist.gov:8080/evaluations/CentOS-7-x86_64-Everything-1511.iso,whichisthe64-bitversionofCentOS1707.2runningLinuxkernel3.10.0.171
Forthistest,MacOSandWindows-compiledlibrariesarenotpermitted.AllsoftwaremustrununderCentOS7.2.172
NISTwilllinktheprovidedlibraryfile(s)toourC++languagetestdrivers.Participantsarerequiredtoprovidetheirlibrary173inaformatthatisdynamically-linkableusingtheC++11compiler,g++version4.8.5.174
Atypicallinklinemightbe175g++ -std=c++11 -I. -Wall -m64 -o frvt1N frvt1N.cpp -L. –lfrvt1N_acme_0 176
TheStandardC++libraryshouldbeusedfordevelopment.Theprototypesfromthisdocumentwillbewrittentoafile177"frvt1N.h"whichwillbeincludedvia#include.178
Theheaderfileswillbemadeavailabletoimplementersathttps://github.com/usnistgov/frvt.Allalgorithmsubmissions179willbebuiltagainsttheofficiallypublishedheaderfiles–developersshouldnotaltertheheaderfileswhencompilingand180buildingtheirlibraries.181
Allcompilationandtestingwillbeperformedonx86_64platforms.Thus,participantsarestronglyadvisedtoverify182library-levelcompatibilitywithg++(onanequivalentplatform)priortosubmittingtheirsoftwaretoNISTtoavoidlinkage183problemslateron(e.g.symbolnameandcallingconventionmismatches,incorrectbinaryfileformats,etc.).184
2.5. Softwareanddocumentation185
2.5.1. Libraryandplatformrequirements186ParticipantsshallprovideNISTwithbinarycodeonly(i.e.nosourcecode).Theimplementationshouldbesubmittedin187theformofadynamically-linkedlibraryfile.188
ThecorelibraryshallbenamedaccordingtoTable2.Additionalsupplementallibrariesmaybesubmittedthatsupport189this“core”libraryfile(i.e.the“core”libraryfilemayhavedependenciesimplementedintheseotherlibraries).190Supplementallibrariesmayhaveanyname,butthe“core”librarymustbedependentonsupplementallibrariesinorder191tobelinkedcorrectly.TheonlylibrarythatwillbeexplicitlylinkedtotheFRVTtestdriveristhe“core”library.192
IntelIntegratedPerformancePrimitives(IPP)®librariesarepermittediftheyaredeliveredasapartofthedeveloper-193suppliedlibrarypackage.Itistheprovider’sresponsibilitytoestablishproperlicensingofalllibraries.TheuseofIPP194librariesshallnotpreventrunningonCPUsthatdonotsupportIPP.PleasetakenotethatsomeIPPfunctionsare195multithreadedandthreadedimplementationsareprohibited.196
NISTwillreportthesizeofthesuppliedlibraries.1973cat/proc/cpuinforeturnsfpuvmedepsetscmsrpaemcecx8apicsepmtrrpgemcacmovpatpse36clflushdtsacpimmxfxsrssesse2sshttmpbesyscallnxpdpe1gbrdtscplmconstant_tscarch_perfmonpebsbtsrep_goodnoplxtopologynonstop_tscaperfmperfpnipclmulqdqdtes64monitords_cplvmxsmxesttm2ssse3cx16xtprpdcmpciddcasse4_1sse4_2popcntaeslahf_lmidaaratdthermtpr_shadowvnmiflexpriorityeptvpid4cat/proc/cpuinforeturnsfpuvmedepsetscmsrpaemcecx8apicsepmtrrpgemcacmovpatpse36clflushdtsacpimmxfxsrssesse2sshttmpbesyscallnxpdpe1gbrdtscplmconstant_tscarch_perfmonpebsbtsrep_goodnoplxtopologynonstop_tscaperfmperfeagerfpupnipclmulqdqdtes64monitords_cplvmxsmxesttm2ssse3fmacx16xtprpdcmpciddcasse4_1sse4_2x2apicmovbepopcnttsc_deadline_timeraesxsaveavxf16crdrandlahf_lmabm3dnowprefetchidaaratepbplnptsdthermtpr_shadowvnmiflexpriorityeptvpidfsgsbasetsc_adjustbmi1hleavx2smepbmi2ermsinvpcidrtmcqmrdseedadxsmapxsaveoptcqm_llccqm_occup_llc
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page7of18
Table2–Implementationlibraryfilenameconvention198Form libfrvt1N_provider_sequence.endingUnderscoredelimitedpartsofthefilename
libfrvt1N provider sequence ending
Description Firstpartofthename,requiredtobethis.
Singleword,non-infringingnameofthemainproviderEXAMPLE:Acme
Aonedigitdecimalidentifiertostartat0andincrementedby1foreachdistinctalgorithmsenttoNIST.Donotincrementthisnumberwhensubmittingbug-fixedupdates.
.so
Example libfrvt1N_acme_0.so
2.5.2. Configurationanddeveloper-defineddata199Theimplementationundertestmaybesuppliedwithconfigurationfilesandsupportingdatafiles.Thesemightinclude,200forexample,model,calibrationorbackgroundfeaturedata.NISTwillreportthesizeofthesuppliedconfigurationfiles.201
2.5.3. Submissionfolderhierarchy202Participantsubmissionsshallcontainthefollowingfoldersatthetoplevel203
― lib/-containsallparticipant-suppliedsoftwarelibraries204
― config/-containsallconfigurationanddeveloper-defineddata205
― doc/-containsversion.txt,whichdocumentsversioninginformationforthesubmittedsoftwareandanyother206participant-provideddocumentationregardingthesubmission207
― validation/-containsvalidationoutput208
2.5.4. Installationandusage209Theimplementationshallbeinstallableusingsimplefilecopymethods.Itshallnotrequiretheuseofaseparate210installationprogramandshallbeexecutableonanynumberofmachineswithoutrequiringadditionalmachine-specific211licensecontrolproceduresoractivation.Theimplementationshallnotusenorenforceanyusagecontrolsorlimitsbased212onlicenses,numberofexecutions,presenceoftemporaryfiles,etc.Theimplementationshallremainoperableforat213leasttwelvemonthsfromthesubmissiondate.214
2.6. Runtimebehavior215
2.6.1. Modesofoperation216ImplementationsshallnotrequireNISTtoswitch“modes”ofoperationoralgorithmparameters.Forexample,theuseof217twodifferentfeatureextractorsmusteitheroperateautomaticallyorbesplitacrosstwoseparatelibrarysubmissions.218
2.6.2. Interactivebehavior,stdout,logging219Theimplementationwillbetestedinnon-interactive“batch”mode(i.e.withoutterminalsupport).Thus,thesubmitted220libraryshall:221
― Notuseanyinteractivefunctionssuchasgraphicaluserinterface(GUI)calls,oranyothercallswhichrequireterminal222interactione.g.readsfrom“standardinput”.223
― Runquietly,i.e.itshouldnotwritemessagesto"standarderror"andshallnotwriteto“standardoutput”.224
― OnlyifrequestedbyNISTfordebugging,includealoggingfacilityinwhichdebuggingmessagesarewrittentoalog225filewhosenameincludestheproviderandlibraryidentifiersandtheprocessPID.226
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page8of18
2.6.3. Exceptionhandling227Theapplicationshouldincludeerror/exceptionhandlingsothatinthecaseofafatalerror,thereturncodeisstill228providedtothecallingapplication.229
2.6.4. Externalcommunication230ProcessesrunningonNISThostsshallnotside-effecttheruntimeenvironmentinanymanner,exceptformemory231allocationandrelease.Implementationsshallnotwriteanydatatoexternalresource(e.g.server,file,connection,or232otherprocess),norreadfromsuch,norotherwisemanipulateit.Ifdetected,NISTwilltakeappropriatesteps,including233butnotlimitedto,cessationofevaluationofallimplementationsfromthesupplier,notificationtotheprovider,and234documentationoftheactivityinpublishedreports.235
2.6.5. Statelessbehavior236Allcomponentsinthistestshallbestateless,exceptasnoted.Thisappliestofacedetection,featureextractionand237matching.Thus,allfunctionsshouldgiveidenticaloutput,foragiveninput,independentoftheruntimehistory.NIST238willinstituteappropriateteststodetectstatefulbehavior.Ifdetected,NISTwilltakeappropriatesteps,includingbutnot239limitedto,cessationofevaluationofallimplementationsfromthesupplier,notificationtotheprovider,and240documentationoftheactivityinpublishedreports.241
2.6.6. Single-threadrequirementandparallelization242Implementationsmustruninsingle-threadedmode,becauseNISTwillparallelizethetestbydividingtheworkloadacross243manycoresandmanymachines.Implementationsmustensurethattherearenoissueswiththeirsoftwarebeing244parallelizedviathefork()function.245
2.7. Timelimits246TheelementalfunctionsoftheimplementationsshallexecuteunderthetimeconstraintsofTable3.Thesetimelimits247applytothefunctioncallinvocationsdefinedinsection4.Assumingthetimesarerandomvariables,NISTcannotregulate248themaximumvalue,sothetimelimitsare90-thpercentiles.Thismeansthat90%ofalloperationsshouldtakelessthan249theidentifiedduration.Timingwillbeestimatedfromatleast1000separateinvocationsofeachelementalfunction.250
Thetimelimitsapplyperimage.251
Table3–Processingtimelimitsinseconds,per640x480colorimage,onasingleCPU252
Function 1:NTemplateGeneration 11:Nfinalization(ongalleryof1millionenrolledtemplates) 400001:Nsearchfor:
- N=1millionenrolledtemplates- L=100returnedcandidates
25
2.8. Templatesizelimits253NISTanticipatesevaluatingperformancewithNwellinexcessof107.Forimplementationsthatrepresentagalleryin254memorywithalineardatastructure,thememoryofourmachinesimpliesalimitontemplatesizes.Thus,foratemplate255sizeB,thetotalmemoryrequirementwouldbeaboutNB.NISTanticipatesrunningthelargestNvaluesonmachines256equippedwith768GBormemory.WithN=25million,templatesshouldnotexceed32KB.257
TheAPI,however,supportsmulti-stagesearchesandreadaccessofthediskduringthe1:Nsearch.Diskaccesswould258likelybeveryslow.Inallcases,algorithmsshallconformtothesearchdurationlimitsgiveninTable3,withlinearscaling.259
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page9of18
3. DatastructuressupportingtheAPI260
3.1. Requirement261FRVT1:NparticipantsshallimplementtherelevantC++prototypedinterfacesofsection4.C++waschoseninorderto262makeuseofsomeobject-orientedfeatures.263
3.2. Fileformatsanddatastructures264
3.2.1. Overview265
Inthisfacerecognitiontest,anindividualisrepresentedbyK³1two-dimensionalfacialimages.Mostimageswillcontain266exactlyoneface.Inasmallfractionoftheimages,other,smaller,faceswillappearinthebackground.Algorithmsshould267detectoneforegroundface(thebiggestone)ineachimageandproduceonetemplate.268
Table4–Structureforasingleimage269C++codefragment Remarkstypedef struct Image { uint16_t width; Numberofpixelshorizontally uint16_t height; Numberofpixelsvertically uint16_t depth; Numberofbitsperpixel.Legalvaluesare8and24. std::shared_ptr<uint8_t> data; Managedpointertorasterscanneddata.EitherRGBcoloror
intensity.Ifimage_depth==24thispointsto3WHbytesRGBRGBRGB...Ifimage_depth==8thispointstoWHbytesIIIIIII
Label description; Singledescriptionoftheimage.TheallowedvaluesforthisfieldarespecifiedintheenumerationinTable5.
} Image;
AnImagewillbeaccompaniedbyoneofthelabelsgivenbelow.Facerecognitionimplementationsshouldtolerate270Imagesofanycategory.271
Table5–LabelsdescribingcategoriesofImages272LabelasC++enumeration Meaningenum class Label { UNKNOWN=0, Eitherthelabelisunknownorunassigned. ISO, Frontal,intendedtobeinconformitytoISO/IEC19794-5:2005. MUGSHOT, Fromlawenforcementbookingprocesses.Nominallyfrontal. PHOTOJOURNALISM, Theimagemightappearinanewssourceormagazine.Theimagesare
typicallytakenbyprofessionalphotographerandarewellexposedandfocusedbutexhibitposeandilluminationvariations.
EXPLOITATION, Theimageistakenfromachildexploitationdatabase.Thisimageryhashighlyunconstrainedposeandillumination,expressionandresolution.
WILD Unconstrainedimage,takenbyanamateurphotographer,exhibitingwidevariationsinpose,illumination,andresolution.
};
Table6–Structureforasetofimagesfromasingleperson273C++codefragment Remarksusing Multiface = std::vector<Image>; VectorofImageobjects
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page10of18
3.2.2. Datastructureforeyecoordinates274Implementationsshallreturneyecoordinatesofeachfacialimage.Thisfunction,whilenotnecessaryforarecognition275test,willassistNISTinassuringthecorrectnessofthetestdatabase.TheprimarymodeofusewillbeforNISTtoinspect276imagesforwhicheyecoordinatesarenotreturned,ordifferbetweenimplementations.277
TheeyecoordinatesshallfollowtheplacementsemanticsoftheISO/IEC19794-5:2005standard-thegeometric278midpointsoftheendocanthionandexocanthion(seeclause5.6.4oftheISOstandard).279
Sense:Thelabel"left"referstosubject'slefteye(andsimilarlyfortherighteye),suchthatxright<xleft.280
Table7–Structureforapairofeyecoordinates281C++codefragment Remarkstypedef struct EyePair { bool isLeftAssigned; Ifthesubject’slefteyecoordinateshavebeencomputedandassigned
successfully,thisvalueshouldbesettotrue,otherwisefalse. bool isRightAssigned; Ifthesubject’srighteyecoordinateshavebeencomputedandassigned
successfully,thisvalueshouldbesettotrue,otherwisefalse. uint16_t xleft; XandYcoordinateofthecenterofthesubject'slefteye.Iftheeye
coordinateisoutofrange(e.g.x<0orx>=width),isLeftAssignedshouldbesettofalse.
uint16_t yleft;
uint16_t xright; XandYcoordinateofthecenterofthesubject'srighteye.Iftheeyecoordinateisoutofrange(e.g.x<0orx>=width),isRightAssignedshouldbesettofalse.
uint16_t yright;
} EyePair;
3.2.3. Templaterole282Labelsdescribingthetype/roleofthetemplatetobegeneratedwillbeprovidedasinputtotemplategeneration.This283supportsasymmetricalgorithmswheretheenrollmentandrecognitiontemplatesmaydifferincontentandsize.284
Table8–Labelsdescribingtemplaterole285LabelasC++enumeration Meaningenum class TemplateRole { Enrollment_1N, Enrollmenttemplatefor1:Nidentification Search_1N Searchtemplatefor1:Nidentification};
3.2.4. Datatypeforsimilarityscores286Identificationandverificationfunctionsshallreturnameasureofthesimilaritybetweenthefacedatacontainedinthe287twotemplates.Thedatatypeshallbeaneight-bytedoubleprecisionreal.Thelegalrangeis[0,DBL_MAX],wherethe288DBL_MAXconstantislargerthanpracticallyneededanddefinedinthe<climits>includefile.Largervaluesindicatemore289likelihoodthatthetwosamplesarefromthesameperson.290
Providersarecautionedthatalgorithmsthatnativelyproducefewuniquevalues(e.g.integerson[0,127])willbe291disadvantagedbytheinabilitytosetathresholdprecisely,asmightberequiredtoattainafalsepositiveidentification292rateofexactly0.0001,forexample.293
3.2.5. Filestructureforenrolledtemplatecollection294Tosupportthis1:Ntest,NISTwillconcatenateenrollmenttemplatesintoasinglelargefile,theEDB(forenrollment295database).TheEDBisasimplebinaryconcatenationofproprietarytemplates.Thereisnoheader.Thereareno296delimiters.TheEDBmaybemanygigabytesinlength.297
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page11of18
Thisfilewillbeaccompaniedbyamanifest;thisisanASCIItextfiledocumentingthecontentsoftheEDB.Themanifest298hastheformatshownasanexampleinTable9.IftheEDBcontainsNtemplates,themanifestwillcontainNlines.The299fieldsarespace(ASCIIdecimal32)delimited.Therearethreefields.Strictlyspeaking,thethirdcolumnisredundant.300
Important:Ifacalltothetemplategenerationfunctionfails,ordoesnotreturnatemplate,NISTwillincludetheTemplate301IDinthemanifestwithsize0.Implementationsmusthandlethisappropriately.302
Table9–Enrollmentdatasettemplatemanifest303Fieldname TemplateID TemplateLength PositionoffirstbyteinEDBDatatyperequired std::string uint64_t uint64_tExamplelinesofamanifestfileappeartotheright.Lines1,2,3andNappear.
90201744 1024 0person01 1536 10247456433 512 2560... subject12 1024 307200000
304TheEDBschemeavoidsthefilesystemoverheadassociatedwithstoringmillionsofsmallindividualfiles.305
3.2.6. GalleryType306
307Figure1–Illustrationofconsolidatedversusunconsolidatedenrollmentdatabase5308
Figure1illustratestwotypesofgalleries:309
― Consolidated:Thedatabaseisformedbyenrollingallimagesofasubjectunderacommonidentitylabel.Theresult310isagallerywithNidentitiesandNtemplates.Thistypeofgallerypresentsuswiththecleanestexperimentaldesign,311“oneneedleinahaystack”scenario.Itallowsalgorithmstoperformimageandfeaturelevelfusion.Operationallyit312requireshighintegritybiographicalinformationtomaintain.313
― Unconsolidated:Thedatabaseisformedbyenrollingphotographswithoutregardtowhetherthesubjectalreadyhas314alreadybeenenrolledornot.Underthisscheme,differentimagesofthesamepersoncanexistinthegalleryunder315differentsubjectidentifiers,thatis,thereareNidentities,andM>Ndatabaseentries.316
Duringgalleryfinalization,algorithmswillbeprovidedwithanenumeratedlabelfromTable10whichspecifiesthetypeof317gallerybeingprocessed.318
5ThefaceimagescontainedinthisfigurearefromthepubliclyavailableSpecialDatabase32-MultipleEncounterDataset(MEDS).https://www.nist.gov/itl/iad/image-group/special-database-32-multiple-encounter-dataset-meds
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page12of18
Table10–Labelsdescribinggallerycomposition319LabelasC++enumeration Meaningenum class GalleryType { Consolidated, Consolidated,subject-basedenrollment Unconsolidated Unconsolidated,event-basedorphoto-basedenrollment};
3.2.7. Datastructureforresultofanidentificationsearch320AllidentificationsearchesshallreturnacandidatelistofaNIST-specifiedlength.Thelistshallbesortedwiththemost321similarmatchingentrieslistfirstwithlowestrank.ThedatastructureshallbethatofTable11.322
Table11–Structureforacandidate323 C++codefragment Remarks1. typedef struct Candidate 2. { 3. bool isAssigned; Ifthecandidatecomputationsucceeded,thisvalueissettotrue.Falseotherwise.
Ifvalueissettofalse,similarityScoreandtemplateIdwillbeignoredentirely.4. std::string templateId; TheTemplateIDfromtheenrollmentdatabasemanifestdefinedinclause3.2.5.5. double similarityScore; Measureofsimilaritybetweentheidentificationtemplateandtheenrolledcandidate.
Higherscoresmeanmorelikelihoodthatthesamplesareofthesameperson.
Analgorithmisfreetoassignanyvalue[0,DBL_MAX]toacandidate.Thedistributionofvalueswillhaveanimpactonthefalse-negativeandfalse-positiveidentificationrates.
6. } Candidate;
3.2.8. DatastructureforreturnvalueofAPIfunctioncalls324
Table12–Enumerationofreturncodes325ReturncodeasC++enumeration Meaningenum class ReturnCode { Success=0, Success ConfigError, Errorreadingconfigurationfiles RefuseInput, Electiverefusaltoprocesstheinput,e.g.becausecannothandlegreyscale ExtractError, Involuntaryfailuretoprocesstheimage,e.g.aftercatchingexception ParseError, Cannotparsetheinputdata TemplateCreationError, Electiverefusaltoproducea“non-blank”template(e.g.insufficientpixels
betweentheeyes) VerifTemplateError, Formatching,eitherorbothoftheinputtemplateswereresultoffailed
featureextraction FaceDetectionError, Unabletodetectafaceintheimage NumDataError, Theimplementationcannotsupportthenumberofimages TemplateFormatError, Templatefileisinanincorrectformatordefective EnrollDirError, Anoperationontheenrollmentdirectoryfailed(e.g.permission,space) InputLocationError, Cannotlocatetheinputdata–theinputfilesornamesseemincorrect MemoryError, Memoryallocationfailed(e.g.outofmemory) NotImplemented, Functionisnotimplemented VendorError, Vendor-definedfailure.Vendorerrorsshallreturnthiserrorcodeand
documentthespecificfailureintheReturnStatus.infostringfromTable13.};
326327328
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page13of18
Table13–ReturnStatusstructure329C++codefragment Meaningstruct ReturnStatus { ReturnCode code; ReturnCode std::string info; Optionalinformationstring // constructors };
330
4. APIspecification331
PleasenotethatincludedwiththeFRVT1:Nvalidationpackage(availableathttps://github.com/usnistgov/frvt)isa“null”332implementationofthisAPI.Thenullimplementationhasnorealfunctionalitybutdemonstratesmechanicallyhowone333couldgoaboutimplementingthisAPI.334
4.1. Namespace335AlldatastructuresandAPIinterfaces/functioncallswillbedeclaredintheFRVTnamespace.336
4.2. Overview337The1:Nidentificationapplicationproceedsinthreephases:enrollment,finalizationandidentification.Theidentification338phaseincludesseparateprobefeatureextractionandsearchstages.339
Thedesignreflectsthefollowingtestingobjectivesfor1:Nimplementations.340
- supportdistributedenrollmentonmultiplemachines,withmultipleprocessesrunninginparallel- allowrecoveryafterafatalexception,andmeasurethenumberofoccurrences- allowNISTtocopyenrollmentdataontomanymachinestosupportparalleltesting- respecttheblack-boxnatureofbiometrictemplates- extendcompletefreedomtotheprovidertousearbitraryalgorithms- supportmeasurementofdurationofcorefunctioncalls- supportmeasurementoftemplatesize- supportmeasurementoftemplateinsertionandremovaltimesintoanenrollmentdatabase
Table14–Proceduraloverviewofthe1:Ntest341
Phase # Name Description PerformanceMetricstobe
reportedbyNIST
Enrollm
ent
E1 Initialization initializeTemplateCreation(TemplateRole=Enrollment_1N)
Givetheimplementationthenameofadirectorywhereanyprovider-suppliedconfigurationdatawillhavebeenplacedbyNIST.Thislocationwillotherwisebeempty.
Theimplementationispermittedread-onlyaccesstotheconfigurationdirectory.
E2 ParallelEnrollment createTemplate(TemplateRole=Enrollment_1N)
ForeachofNindividuals,passK=1imageoftheindividualtotheimplementationforconversiontoatemplate.Theimplementationwillreturnatemplatetothecallingapplication.
NIST'scallingapplicationwillberesponsibleforstoringalltemplatesasbinaryfiles.Thesewillnotbeavailabletotheimplementationduringthisenrollmentphase.
Multipleinstancesofthecallingapplicationmayrunsimultaneouslyorsequentially.Thesemaybeexecutingondifferentcomputers.
Statisticsofthetimesneededtoenrollanindividual.
Statisticsofthesizesofcreatedtemplates.
Theincidenceoffailedtemplatecreations.
E3 Finalization finalizeEnrollment()
Permanentlyfinalizetheenrollmentdirectory.Thissupports,forexample,adaptation
Sizeoftheenrollmentdatabaseasafunctionof
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page14of18
oftheimage-processingfunctions,adaptationoftherepresentation,writingofamanifest,indexing,andcomputationofstatisticalinformationovertheenrollmentdataset.
Theimplementationispermittedread-write-deleteaccesstotheenrollmentdirectoryduringthisphase.
populationsizeN.
Durationofthisoperation.Thetimeneededtoexecutethisfunctionshallbereportedwiththeprecedingenrollmenttimes.
Prob
eTemplateCreatio
n
S1 Initialization initializeTemplateCreation(TemplateRole=Search_1N)
Givetheimplementationthenameofadirectorywhereanyprovider-suppliedconfigurationdatawillhavebeenplacedbyNIST.Thislocationwillotherwisebeempty.
Theimplementationispermittedread-onlyaccesstotheconfigurationdirectory.
Statisticsofthetimeneededforthisoperation.
S2 Templatepreparation
createTemplate(TemplateRole=Search_1N)
Foreachprobe,createatemplatefromK>=1image.
Theresultofthisstepisasearchtemplate.Multipleinstancesofthecallingapplicationmayrunsimultaneouslyorsequentially.Thesemaybeexecutingondifferentcomputers.
Statisticsofthetimeneededforthisoperation.
Statisticsofthesizeofthesearchtemplate.
Search
S3 Initialization initializeIdentification()
Telltheimplementationthelocationofanenrollmentdirectory.Theimplementationshouldreadallorsomeoftheenrolleddataintomainmemory,sothatsearchescancommence.
Theimplementationispermittedread-onlyaccesstotheenrollmentdirectoryduringthisphase.
Statisticsofthetimeneededforthisoperation.
S4 Search identifyTemplate()
Atemplateissearchedagainsttheenrollmentdatabase.
DevelopersshallnotattempttoimprovethedurationoftheidentifyTemplate()functionbyoffloadinganyofitsprocessingintothecreateTemplate()function.
Statisticsofthetimeneededforthisoperation.
Accuracymetrics-TypeI+IIerrorrates.
Failurerates.
GalleryIn
sertand
Rem
ove
G1 Initialization initializeIdentification()
Telltheimplementationthelocationofanenrollmentdirectory.Theimplementationshouldreadallorsomeoftheenrolleddataintomainmemory,sothatsearchescancommence.
Theimplementationispermittedread-onlyaccesstotheenrollmentdirectoryduringthisphase.
Statisticsofthedurationoftheoperation.
G2 Templateinsertion/removalinto/fromgallery
galleryInsertID()/galleryDeleteID()
galleryInsertID()isexecutedoneormoretimestoinsertatemplatecreatedwithcreateTemplate(TemplateRole=Enrollment_1N)intothegallery.
galleryDeleteID()isexecutedoneormoretimestoremoveatemplatefromthegallery.
Statisticsofthedurationoftheoperation.
G3 Search identifyTemplate()
Atemplateissearchedagainsttheenrollmentdatabase.
Statisticsofthedurationoftheoperation.
Accuracymetrics-TypeI+IIerrorrates.
342
343
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page15of18
4.3. API344
4.3.1. Interface345ThesoftwareundertestmustimplementtheinterfaceIdentInterfacebysubclassingthisclassandimplementing346eachmethodspecifiedtherein.347
C++codefragment Remarks1. Class IdentInterface 2. {
public:
3. static std::shared_ptr<IdentInterface> getImplementation(); FactorymethodtoreturnamanagedpointertotheIdentInterfaceobject.ThisfunctionisimplementedbythesubmittedlibraryandmustreturnamanagedpointertotheIdentInterfaceobject.
4. // Other functions to implement 5. };
Thereisoneclass(static)methoddeclaredinIdentInterface.getImplementation()whichmustalsobe348implemented.Thismethodreturnsasharedpointertotheobjectoftheinterfacetype,aninstantiationofthe349implementationclass.Atypicalimplementationofthismethodisalsoshownbelowasanexample.350
C++codefragment Remarks #include “frvt1N.h”
using namespace FRVT; NullImpl:: NullImpl () { } NullImpl::~ NullImpl () { } std::shared_ptr<IdentInterface> IdentInterface::getImplementation() { return std::make_shared<NullImpl>(); } // Other implemented functions
4.3.2. Initializationoftemplatecreation351Beforeanyfeatureextraction/templatecreationcallsaremade,theNISTtestharnesswillcalltheinitializationfunctionof352Table15.ThisfunctionwillbecalledBEFOREanycallstofork()aremade.353
Table15–Templatecreationinitialization354
Prototype ReturnStatusinitializeTemplateCreation( conststd::string&configDir, InputTemplateRolerole); Input
Description
Thisfunctioninitializestheimplementationundertestandsetsallneededparametersinpreparationfortemplatecreation.ThisfunctionwillbecalledN=1timesbytheNISTapplication,priortoparallelizingM>=1callstocreateTemplate()viafork(). Thisfunctionwillbecalledfromasingleprocess/thread.
InputParameters configDir Aread-onlydirectorycontaininganydeveloper-suppliedconfigurationparametersorrun-timedatafiles.
role AvaluefromtheTemplateRoleenumerationthatindicatestheintendedusageofthetemplatetobegenerated.Inthiscase,eitherEnrollment_1NorSearch_1N.
OutputParameters
None
ReturnValue SeeTable12forallvalidreturncodevalues.
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page16of18
4.3.3. TemplateCreation356AMultifaceisconvertedtoasingletemplateusingthefunctionofTable16.357
Table16–TemplateCreation/FeatureExtraction358
Prototypes ReturnStatuscreateTemplate( constMultiface&faces, InputTemplateRolerole, Inputstd::vector<uint8_t>&templ,std::vector<EyePair>&eyeCoordinates);
OutputOutput
Description TakesaMultifaceandoutputsaproprietarytemplateandassociatedeyecoordinates.Thevectortostorethetemplatewillbeinitiallyempty,anditisuptotheimplementationtopopulateitwiththeappropriatedata.
Note:Intheeventthatmorethanonefaceisdetectedinanimage,featuresshouldbeextractedfromtheforegroundface,thatis,thelargestfaceintheimage.
Forenrollmenttemplates(TemplateRole=Enrollment_1N):Ifthefunctionexecutescorrectly(i.e.returnsasuccessfulreturncode),thetemplatewillbeenrolledintoagallery.TheNISTcallingapplicationmaystoretheresultingtemplate,concatenatemanytemplates,andpasstheresulttotheenrollmentfinalizationfunction(seesection4.3.4).Theresultingtemplatemayalsobeinsertedimmediatelyintopreviouslyfinalizedgallery.Whentheimplementationfailstoproduceatemplate(i.e.returnsanon-successfulreturncode),itshallstillreturnablanktemplate(whichcanbezerobytesinlength).Thetemplatewillbeincludedintheenrollmentdatabase/manifestlikeallotherenrollmenttemplates,butisnotexpectedtocontainanyfeatureinformation.
IMPORTANT:NIST'sapplicationwritesthetemplatetodisk.Anydataneededduringsubsequentsearchesshouldbeincludedinthetemplate,orcreatedfromthetemplatesduringtheenrollmentfinalizationfunctionofsection4.3.4.
Foridentification/probetemplates(TemplateRole=Search_1N):TheNISTcallingapplicationmaycommitthetemplatetopermanentstorage,ormaykeepitonlyinmemory(thedeveloperimplementationdoesnotneedtoknow).Ifthefunctionreturnsanon-successfulreturnstatus,theoutputtemplatewillnotbeusedinsubsequentsearchoperations.
InputParameters
face InputMultifacerole Labeldescribingthetype/roleofthetemplatetobegenerated.Inthiscase,itwilleitherbe
Enrollment_1NorSearch_1N.OutputParameters
templ Theoutputtemplate.Theformatisentirelyunregulated.Thiswillbeanemptyvectorwhenpassedintothefunction,andtheimplementationcanresizeandpopulateitwiththeappropriatedata.
eyeCoordinates Thefunctionshallreturntheestimatedeyecentersfortheinputfaceimage.ReturnValue SeeTable12forallvalidreturncodevalues.
4.3.4. Finalization359Afteralltemplateshavebeencreated,thefunctionofTable17willbecalled.Thisfreezestheenrollmentdata.Afterthis360calltheenrollmentdatasetwillbeforeverread-only.361
Thefunctionallowstheimplementationtoconduct,forexample,statisticalprocessingofthefeaturedata,indexingand362datare-organization.Thefunctionmayalterthefilestructure.Itmayincreaseordecreasethesizeofthestoreddata.363Nooutputisexpectedfromthisfunction,exceptareturncode.364
Implementationsshallnotmovetheinputdata.Implementationsshallnotpointtotheinputdata.Implementations365shouldnotassumetheinputdatawillbereadableafterthecall.Implementationsmust,ataminimum,copytheinput366dataorotherwiseextractwhatisneededforsearch.367
Table17–Enrollmentfinalization368
Prototypes ReturnStatusfinalizeEnrollment( conststd::string&enrollmentDir, Inputconststd::string&edbName, Inputconststd::string&edbManifestName, InputGalleryTypegalleryType); Input
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page17of18
Description Thisfunctiontakesthenameofthetop-leveldirectorywheretheenrollmentdatabase(EDB)anditsmanifesthavebeenstored.Thesearedescribedinsection3.2.5.Theenrollmentdirectorypermissionswillberead+write.
Thefunctionsupportspost-enrollment,developer-optional,book-keepingoperations,statisticalprocessinganddatare-orderingforfastin-memorysearching.Thefunctionwillgenerallybecalledinaseparateprocessafteralltheenrollmentprocessesarecomplete.
Thisfunctionshouldbetolerantofbeingcalledtwoormoretimes.Secondandthirdinvocationsshouldprobablydonothing.
Thisfunctionwillbecalledfromasingleprocess/thread.
InputParameters
enrollmentDir Thetop-leveldirectoryinwhichenrollmentdatawasplaced.Thisvariableallowsanimplementationtolocateanyprivateinitializationdataitelectedtoplaceinthedirectory.
edbName Thenameofasinglefilecontainingconcatenatedtemplates,i.e.theEDBofsection3.2.5.Whilethefilewillhaveread-write-deletepermission,theimplementationshouldonlyalterthefileifitpreservesthenecessarycontent,inotherfilesforexample.Thefilemaybeopeneddirectly.Itisnotnecessarytoprependadirectoryname.ThisisaNIST-providedinput–implementersshallnotinternallyhard-codeorassumeanyvalues.
edbManifestName ThenameofasinglefilecontainingtheEDBmanifestofsection3.2.5.Thefilemaybeopeneddirectly.Itisnotnecessarytoprependadirectoryname.ThisisaNIST-providedinput–implementersshallnotinternallyhard-codeorassumeanyvalues.
galleryType AlabelfromTable10specifyingthecompositionofthegallery.
OutputParameters
None
ReturnValue SeeTable12forallvalidreturncodevalues.
369
4.3.5. SearchInitialization370ThefunctionofTable18willbecalledoncepriortooneormorecallsofthesearchingfunctionofTable19.Thefunction371mightsetstaticinternalvariablessothattheenrollmentdatabaseisavailabletothesubsequentidentificationsearches.372ThisfunctionwillbecalledBEFOREanycallstofork()aremade.373
Table18–Identificationinitialization374
Prototype ReturnStatusinitializeIdentification( conststring&configDir, Inputconststring&enrollmentDir); Input
Description ThisfunctionreadswhatevercontentispresentintheenrollmentDir,forexampleamanifestplacedtherebythefinalizeEnrollment()function.Thisfunctionwillbecalledfromasingleprocess/thread.
InputParameters configDir Aread-onlydirectorycontaininganydeveloper-suppliedconfigurationparametersorrun-timedatafiles.
enrollmentDir Theread-onlytop-leveldirectoryinwhichenrollmentdatawasplaced.ReturnValue SeeTable12forallvalidreturncodevalues.
4.3.6. Search375ThefunctionofTable19comparesaproprietaryidentificationtemplateagainsttheenrollmentdataandreturnsa376candidatelist.377
Table19–Identificationsearch378
Prototype ReturnStatusidentifyTemplate( conststd::vector<uint8_t>&idTemplate, Inputconstuint32_tcandidateListLength, Inputstd::vector<Candidate>&candidateList, Outputbool&decision); Output
FaceRecognitionVendorTest1:N
NIST Concept,EvaluationPlanandAPI Page18of18
Description
Thisfunctionsearchesatemplateagainsttheenrollmentset,andoutputsalistofcandidates.ThecandidateListvectorwillinitiallybeempty,andtheimplementationshallpopulatethevectorwithcandidateListLengthentries.
InputParameters idTemplate AtemplatefromcreateTemplate(TemplateRole=Search_1N)-Ifthevaluereturnedbythatfunctionwasnon-zerothecontentsofidTemplatewillnotbeusedandthisfunction(i.e.identifyTemplate)willnotbecalled.
candidateListLength ThenumberofcandidatesthesearchshouldreturnOutputParameters
candidateList Avectorcontaining"candidateListLength"objectsofcandidates.Thedatatypeisdefinedinsection3.2.6.Eachcandidateshallbepopulatedbytheimplementation.Thecandidatesshallappearindescendingorderofsimilarityscore-i.e.mostsimilarentriesappearfirst.
decision Abestguessatwhetherthereisamatewithintheenrollmentdatabase.Iftherewasamatefound,thisvalueshouldbesettotrue,Otherwise,false.ManysuchdecisionsallowasinglepointtobeplottedalongsideaDET.
ReturnValue SeeTable12forallvalidreturncodevalues.379
NOTE: Ordinarilythecallingapplicationwillsettheinputcandidatelistlengthtooperationallytypicalvalues,say0£L£380200,andL<<N.WewillmeasurethedependenceofsearchdurationonL.381
4.3.7. Galleryinsertionandremovaloftemplates382Thefunctionsofthissectioninsertanewtemplateinto,andremovesanexistingtemplatefrom,afinalizedgallery,383respectively.384
Table20–Insertionoftemplateintoagallery385
Prototype ReturnStatusgalleryInsertID( conststd::vector<uint8_t>&templ, Inputconststd::string&id); Input
Description
Thisfunctioninsertsatemplatewithanassociatedidintoanexistingfinalizedgallery.InvocationofthisfunctionwillalwaysbeprecededbyacalltoinitializeIdentification(),whichwillprovidethelocationofthefinalizedgallerytobeloadedintomemory.OneormorecallstoidentifyTemplate()maybemadeaftercallingthisfunction.
ThetemplateIDwillnotexistinthedatabasealready,soa1:Nduplicatesearchisnotnecessary.
Thisfunctionwillbecalledfromasingleprocess/thread.
InputParameters templ AtemplatecreatedviacreateTemplate(TemplateRole=Enrollment_1N)id Anidentifierassociatedwiththeenrollmenttemplate
ReturnValue SeeTable12forallvalidreturncodevalues.
Table21–Removaloftemplatefromagallery386
Prototype ReturnStatusgalleryDeleteID( conststd::string&id); Input
Description
Thisfunctiondeletesanexistingtemplatefromafinalizedgallery.InvocationofthisfunctionwillalwaysbeprecededbyacalltoinitializeIdentification(),whichwillprovidethelocationofthefinalizedgallery.OneormorecallstoidentifyTemplatemaybemadebeforeandaftercallingthisfunction.
ThetemplateIDwillexistinthedatabase.
Thisfunctionwillbecalledfromasingleprocess/thread.
InputParameters id AnidentifierassociatedwiththeenrollmenttemplateReturnValue SeeTable12forallvalidreturncodevalues.387