face recognition vendor test frvt 2018 performance of ... · pdf fileface recognition vendor...

Click here to load reader

Post on 15-Mar-2018

221 views

Category:

Documents

2 download

Embed Size (px)

TRANSCRIPT

  • FaceRecognitionVendorTestFRVT2018

    PerformanceofAutomatedFaceIdentificationAlgorithmsConcept,EvaluationPlanandAPI

    VERSION0.5

    THISDOCUMENTISOPENFORCOMMENTS:[email protected]

    PatrickGrotherMeiNgan

    KayeeHanaokaInformationAccessDivision

    InformationTechnologyLaboratory

    October26,2017

  • FaceRecognitionVendorTest1:N

    NIST Concept,EvaluationPlanandAPI Page2of18

    TableofContents11. FRVT2018.........................................................................................................................................................................32

    1.1. Scope......................................................................................................................................................................331.2. Audience.................................................................................................................................................................341.3. Schedule.................................................................................................................................................................351.4. Reporting................................................................................................................................................................361.5. VersionControl.......................................................................................................................................................371.6. Background.............................................................................................................................................................481.7. FRVT2018:Changesfrompriorevaluations...........................................................................................................491.8. Relationtothe1:1FRVTevaluation.......................................................................................................................4101.9. Coreaccuracymetrics.............................................................................................................................................4111.10. Applicationrelevance.............................................................................................................................................512

    2. Rulesforparticipation......................................................................................................................................................5132.1. Participationagreement.........................................................................................................................................5142.2. Validation................................................................................................................................................................5152.3. Hardwarespecification...........................................................................................................................................5162.4. Operatingsystem,compilation,andlinkingenvironment......................................................................................6172.5. Softwareanddocumentation.................................................................................................................................6182.6. Runtimebehavior...................................................................................................................................................7192.7. Timelimits..............................................................................................................................................................8202.8. Templatesizelimits................................................................................................................................................821

    3. DatastructuressupportingtheAPI..................................................................................................................................9223.1. Requirement...........................................................................................................................................................9233.2. Fileformatsanddatastructures.............................................................................................................................924

    4. APIspecification.............................................................................................................................................................13254.1. Namespace...........................................................................................................................................................13264.2. Overview...............................................................................................................................................................13274.3. API.........................................................................................................................................................................1528

    29ListofTables30Table1Scheduleandallowednumberofsubmissions.........................................................................................................331Table3Processingtimelimitsinseconds,per640x480colorimage,onasingleCPU.......................................................832Table4Structureforasingleimage......................................................................................................................................933Table5LabelsdescribingcategoriesofImages.....................................................................................................................934Table6Structureforasetofimagesfromasingleperson...................................................................................................935Table7Structureforapairofeyecoordinates...................................................................................................................1036Table8Labelsdescribingtemplaterole..............................................................................................................................1037Table9Enrollmentdatasettemplatemanifest...................................................................................................................1138Table10Labelsdescribinggallerycomposition..................................................................................................................1239Table11Structureforacandidate......................................................................................................................................1240Table12Enumerationofreturncodes................................................................................................................................1241Table13ReturnStatusstructure.........................................................................................................................................1342Table16Proceduraloverviewofthe1:Ntest.....................................................................................................................1343Table17Templatecreationinitialization............................................................................................................................1544Table19Enrollmentfinalization..........................................................................................................................................1645Table21Identificationsearch.............................................................................................................................................1746Table23Insertionoftemplateintoagallery......................................................................................................................1847Table24Removaloftemplatefromagallery.....................................................................................................................184849

    50

  • FaceRecognitionVendorTest1:N

    NIST Concept,EvaluationPlanandAPI Page3of18

    1. FRVT201851

    1.1. Scope52Thisdocumentestablishesaconceptofoperationsandanapplicationprogramminginterface(API)forevaluationofone-53to-manyfacerecognitionalgorithmsappliedtofacesappearingin2Dstillphotographs.Theprimaryfocusofthetestis54cooperativeportraitimages,e.g.mugshots1.Thetestwillincludealsosearchofnon-cooperativeimages.TheAPI55communicatesthetypeofimagetothealgorithm.56

    1.2. Audience57Participationisopentoanyorganizationworldwide,primarilyresearchersanddevelopersofFRalgorithms.WhileNIST58intendstoevaluatestabletechnologiesthatcouldbereadilymadeoperational,thetestisalsoopentoexperimental,59prototypeandothertechnologies.AllalgorithmsmustbesubmittedasimplementationsoftheAPIdefinedinthis60document.Thereisnochargeforparticipation.61

    1.3. Schedule62InconsultationwithUSGovernmentcollaborators,NISTwillexecutetheFRVT2018ontheschedulegiveninTable1.Note63thatNISTwillreportresultspubliclyattheendofPhases2and3.TheendofPhase2correspondscloselytotheendof64thefinancialyearandresultsfromtheFRVTarerequiredbythatdate.65

    Developersmaysubmitthenumberofalgorithmsidentifiedintherightmostcolumn.66

    Table1Scheduleandallowednumberofsubmissions67Phase Date Milestone Maximumnumberof

    implementationsAPIDevelopment

    2017-10-26 Draftevaluationplanavailableforpubliccomments 2017-11-16 Finalevaluationplanpublished

    Phase1

    2018-01-22 Participationstarts:AlgorithmsmaybesenttoNIST 2018-02-16 LastdayforsubmissionofalgorithmstoPhase1 32018-05-23 Interimresultsreleased

    Phase2

    2018-06-21 LastdayforsubmissionofalgorithmstoPhase2 22018-09-20 Resultsreleased

    Phase3

    2018-10-19 LastdayforsubmissionofalgorithmstoPhase3 22018-12-19 Releaseoffinalpublicreport

    1.4. Reporting68AttheconclusionofPhase1,NISTwillprovideresultsforallalgorithmstoalldevelopers,andtoUSGovernmentpartners.69

    AttheconclusionofPhase2,NISTwillpublishresultsfora