ongoing face recognition vendor test (frvt) part 3: … · 2019. 12. 19. · .ir.8280 2019/12/19...

This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0

NISTIR 8280

Ongoing Face RecognitionVendor Test (FRVT)

Part 3: Demographic Effects

Annex 16 : Identification error characteristics by race andsex

This document is an annex of NIST Interagency Report 8280:https://doi.org/10.6028/NIST.IR.8280

2019/12/19

https://pages.nist.gov/frvt/reports/demographics/annexes/annex_16.pdf

This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0

2019/12/19 08:17:56 FRVT - FACE RECOGNITION VENDOR TEST - DEMOGRAPHICS 1

1 Overview

This annex includes figures that shows rank-based and threshold-based error characteristics for mugshot im-ages from three identification experiments. Each experiment was designed to isolate just one demographicfactor from all demographic factors related to race, sex, and age. The experiments also remove any effects fromimbalance, specially by composing galleries from two demographic groups in equal proportion. The experi-ments were designed to be readily interpretable. They were not designed to mimic any specific operationalscenario.

The three experiments are:

. Black-White Males The left panels in each figure summarize one-to-many search results for galleries inwhich all subjects are males in the age range 20 to 40, and the only known covariate is race, here blackvs. white. The metadata labels are described in Annex Annex 1 .

. Black Male-Female The center panels in each figure summarize one-to-many search results for galleriesin which all subjects are black in the age range 20 to 40, and the only known covariate is sex.

. White Male-Female As previously, with white instead of black subjects.

In the above we refer the “known” covariates. The word here is used to o express that a large number of (weassume) less influential factors could be at play: e.g. the height of the subject1. In virtually all modeling effors,e.g. building regression models, there are unknown indepedent variables. Full knowledge of all covariateswould mean the model would have the complexity of the system itself.

2 Data

The mugshot dataset and its metadata is described in Annex Annex 1 .

The figures in this annex were generated from the three experiments detailed in Table 1.

3 Fixed Threshold

A false positive is declared if a non-mated search yields any candidates (one or more) with similarity at orabove, a threshold. The error tradeoff characteristics sweep the threshold over at least 1280 representativethreshold values, to effect FPIR statements over four decades from one in ten thousand, up to one. Doing thisfor the various demographic curves individually yields the colored traces in the second row of panels.

The error tradeoff characteristics also include three grey lines. These connect points of fixed threshold, andthereby expose difference in FNIR and FPIR between two demographic groups.

The threshold value could be any value germane to that comparison algroithm. Here we set it to achieve oneof three FPIR values on, arbitrarily, white males. The three FPIR values are 0.003, 0.03, 0.3.

1This may matter because pitch angle relative to the camera is known to affect most face recognition algorithms, and many cameras wouldnot be adjusted for height for each subject. The mugshot photographic configuration includes a larger stand-off distance, around 2 meters,so pitch angle variation is expected to be small. In addition, subject are usually directed to look at the camera.

Links: EXEC. SUMMARYTECH. SUMMARY

False positive: Incorrect association of two subjects 1:1 FMR 1:N FPIRFalse negative: Failed association of one subject 1:1 FNMR 1:N FNIR

T� 0 → FMR, FPIR→ 0→ FNMR, FNIR→ 1



This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


ENROLLMENT SEARCH

TYPE SEE POPULATION MATE NON-MATE

SECTION ?? FILTER N-SUBJECTS

N-IMAGES N-SUBJECTS

N-IMAGES N-SUBJECTS

N-IMAGES

Demographics1 RECENT MALE,

AGE21-40, ∆T≤ 5 YR, BLACKAND WHITEBALANCED

800 000 B +800 000 W

800 000 B +800 000 W

100 000 B +100 000 W

100 000 B +100 000 W

100 000 B +100 000 W

100 000 B +100 000 W

2 RECENT WHITE,AGE21-40, ∆T≤ 5 YR, MALEAND FEMALEBALANCED

800 000 F +800 000 M

800 000 F +800 000 M

100 000 F +100 000 M

100 000 F +100 000 M

100 000 F +100 000 M

100 000 F +100 000 M

3 RECENT BLACK,AGE21-40, ∆T≤ 5 YR, MALEAND FEMALEBALANCED

500 000 F +500 000 M

500 000 F +500 000 M

97 000 F +97 000 M

97 000 F +97 000 M

100 000 F +100 000 M

100 000 F +100 000 M

Table 1: Enrollment and search sets. Each row summarizes one identification trial. Unless stated otherwise, all entriesrefer to mugshot images. The term “natural” means that subjects were selected without heed to demographics, i.e. in thedistribution native to this dataset. The probe images were collected in a different calendar year to the enrollment image.Missing values in rows 2-12 are the same as in row 1.

4 Plot

Each page contains two figures corresponding to two algorithm. Each figure contains two rows. The firstshows FNIR as a function of rank, a metric appropriate to investigate “lead generation” applications. Thesecond rows shows FNIR as a function of FPIR, plotted parametrically on threshold, T. Higher thresholds andlower FPIR are at the left of each plot.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/3divi_0.pdf

BW_male_20_40_balanced MF_black_20_40_balanced MF_white_20_40_balanced

1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.01

0.02

0.03

0.05

0.07

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.01

0.02

0.03

0.05

0.07

0.10

0.20

0.30

0.50

0.70

False positive identification rate, FPIR(T)

Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite

Thresholds3.3583.2383.1322.987set for whitemales for FPIR0.00030.00300.03000.3000

FemaleMale

Figure 1: FNIR by sex and race for mugshost, 3divi-0. The three columns of panels represent three one-to-many search trialswith, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/3divi_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.02

0.03

0.05

0.07

0.10

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.02

0.03

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) Thresholds3.5713.4923.4063.275set for whitemales for FPIR0.00030.00300.03000.3000

FemaleMale

race

BlackWhite

Figure 2: FNIR by sex and race for mugshost, 3divi-3. The three columns of panels represent three one-to-many search trialswith, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/alchera_0.pdf

BW_male_20_40_balanced MF_white_20_40_balanced

1 3 10 30 50 1 3 10 30 50

0.01

0.02

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset: MugshotsN=1600000

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

)

race

BlackWhite

Thresholds61.62202356.68058049.466983set for whitemales for FPIR0.00300.03000.3000

FemaleMale

Figure 3: FNIR by sex and race for mugshost, alchera-0. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/anke_002.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T


FemaleMale

race

BlackWhite

Figure 4: FNIR by sex and race for mugshost, anke-002. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/aware_0.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.03

0.05

0.07

0.10

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 5: FNIR by sex and race for mugshost, aware-0. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/aware_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.01

0.02

0.03

0.05

0.07

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.02

0.03

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T


FemaleMale

race

BlackWhite

Figure 6: FNIR by sex and race for mugshost, aware-3. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/ayonix_0.pdf


1 3 10 30 50 1 3 10 30 50

0.2

0.3

0.5

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset: MugshotsN=1600000

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

0.2

0.3

0.5

0.7


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

)

race

BlackWhite

Thresholds0.9862490.9774850.953753set for whitemales for FPIR0.00300.03000.3000

FemaleMale

Figure 7: FNIR by sex and race for mugshost, ayonix-0. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/camvi_1.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.07

0.10

0.20

0.30

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 8: FNIR by sex and race for mugshost, camvi-1. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0




1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.05

0.07

0.10

0.20

0.30

0.50

0.70

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale





This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0




1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.03

0.05

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.03

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale





This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/cogent_0.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 11: FNIR by sex and race for mugshost, cogent-0. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/cogent_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 12: FNIR by sex and race for mugshost, cogent-3. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/cognitec_2.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

0.020

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 13: FNIR by sex and race for mugshost, cognitec-2. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/dahua_1.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 14: FNIR by sex and race for mugshost, dahua-1. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/deepsea_001.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.003

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 15: FNIR by sex and race for mugshost, deepsea-001. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/dermalog_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.02

0.03

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.01

0.02

0.03

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 16: FNIR by sex and race for mugshost, dermalog-5. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/dermalog_6.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 17: FNIR by sex and race for mugshost, dermalog-6. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/everai_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

0.007

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T


FemaleMale

race

BlackWhite

Figure 18: FNIR by sex and race for mugshost, everai-3. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/everai_paravision_004.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.001

0.002

0.003

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.001

0.0020.003

0.0050.0070.010

0.0200.030

0.0500.0700.100

0.2000.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 19: FNIR by sex and race for mugshost, everai-paravision-004. The three columns of panels represent three one-to-many search trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/eyedea_0.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.07

0.10

0.20

0.30

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 20: FNIR by sex and race for mugshost, eyedea-0. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/eyedea_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.02

0.03

0.05

0.07

0.10

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.02

0.03

0.05

0.07

0.10

0.20

0.30

0.50

0.70


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 21: FNIR by sex and race for mugshost, eyedea-3. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/hik_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 22: FNIR by sex and race for mugshost, hik-5. The three columns of panels represent three one-to-many search trialswith, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/idemia_4.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

0.020

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 23: FNIR by sex and race for mugshost, idemia-4. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/idemia_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

0.020

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T


FemaleMale

race

BlackWhite

Figure 24: FNIR by sex and race for mugshost, idemia-5. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/imperial_000.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.001

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 25: FNIR by sex and race for mugshost, imperial-000. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/incode_004.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 500.002

0.003

0.005

0.007

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 26: FNIR by sex and race for mugshost, incode-004. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/isystems_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 27: FNIR by sex and race for mugshost, isystems-3. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/kedacom_001.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 28: FNIR by sex and race for mugshost, kedacom-001. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/megvii_1.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.007

0.010

0.020

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 29: FNIR by sex and race for mugshost, megvii-1. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/microsoft_0.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 30: FNIR by sex and race for mugshost, microsoft-0. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/microsoft_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.001

0.002

0.003

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.001

0.0020.003

0.0050.0070.010

0.0200.030

0.0500.0700.100

0.2000.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 31: FNIR by sex and race for mugshost, microsoft-5. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/nec_2.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.001

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 32: FNIR by sex and race for mugshost, nec-2. The three columns of panels represent three one-to-many search trialswith, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/nec_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 33: FNIR by sex and race for mugshost, nec-3. The three columns of panels represent three one-to-many search trialswith, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/neurotechnology_007.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.003

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 34: FNIR by sex and race for mugshost, neurotechnology-007. The three columns of panels represent three one-to-many search trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/neurotechnology_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 35: FNIR by sex and race for mugshost, neurotechnology-5. The three columns of panels represent three one-to-many search trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/ntechlab_007.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T


FemaleMale

race

BlackWhite

Figure 36: FNIR by sex and race for mugshost, ntechlab-007. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/ntechlab_6.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

0.007

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 37: FNIR by sex and race for mugshost, ntechlab-6. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/rankone_006.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 38: FNIR by sex and race for mugshost, rankone-006. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/rankone_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

0.020

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T


FemaleMale

race

BlackWhite

Figure 39: FNIR by sex and race for mugshost, rankone-5. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/realnetworks_003.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 40: FNIR by sex and race for mugshost, realnetworks-003. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/realnetworks_2.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 41: FNIR by sex and race for mugshost, realnetworks-2. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/sensetime_0.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.003

0.005

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 42: FNIR by sex and race for mugshost, sensetime-0. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0




1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.01

0.02

0.03

0.05

0.07

0.10

0.20

0.30

0.50


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale





This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0




1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.003

0.005

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale





This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/tech5_001.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.002

0.003

0.005

0.007

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+000.001

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 45: FNIR by sex and race for mugshost, tech5-001. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/tevian_4.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

0.020

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 46: FNIR by sex and race for mugshost, tevian-4. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/toshiba_1.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 47: FNIR by sex and race for mugshost, toshiba-1. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/visionlabs_008.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.001

0.002

0.003

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.001

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 48: FNIR by sex and race for mugshost, visionlabs-008. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/visionlabs_7.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.001

0.002

0.003

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.001

0.002

0.003

0.0050.0070.010

0.020

0.030

0.0500.0700.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 49: FNIR by sex and race for mugshost, visionlabs-7. The three columns of panels represent three one-to-manysearch trials with, respectively, balanced galleries of black and white males, black males and females, and white malesand females. The upper row shows false negative identification rate vs. rank for investigative search applications withzero threshold. The lower row shows false negative vs. false positive identification rates for higher threshold applicationswhere only strong hits should be returned, either because that causes an acceptable decision or because labor is availableto review the candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to giveFPIR of 0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/vocord_3.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.005

0.007

0.010

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.003

0.005

0.007

0.010

0.020

0.030

0.050

0.070

0.100

0.200

0.300

0.500

0.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 50: FNIR by sex and race for mugshost, vocord-3. The three columns of panels represent three one-to-many searchtrials with, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




This

publ

icat

ion

isav

aila

ble

free

ofch

arge

from

:htt

ps:/

/doi

.org

/10.

6028

/NIS

T.IR

.828

0


reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/yitu_5.pdf


1 3 10 30 50 1 3 10 30 50 1 3 10 30 50

0.003

0.005

Rank

Fals

e ne

gativ

e id

entif

icat

ion

rate

(F

NIR

)

race

BlackWhite

Dataset:MUGSHOTS

FemaleMale


1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+001e−04

3e−041e−03

3e−031e−02

3e−021e−01

3e−011e+00

1e−043e−04

1e−033e−03

1e−023e−02

1e−013e−01

1e+00

0.002

0.003

0.0050.007

0.010

0.020

0.030

0.0500.070

0.100

0.200

0.300

0.5000.700


Fals

e ne

gativ

e id

entif

icat

ion

rate

, FN

IR(T

) race

BlackWhite


FemaleMale

Figure 51: FNIR by sex and race for mugshost, yitu-5. The three columns of panels represent three one-to-many search trialswith, respectively, balanced galleries of black and white males, black males and females, and white males and females.The upper row shows false negative identification rate vs. rank for investigative search applications with zero threshold.The lower row shows false negative vs. false positive identification rates for higher threshold applications where onlystrong hits should be returned, either because that causes an acceptable decision or because labor is available to reviewthe candidates returned The grey lines join points of equal threshold. The three thresholds are chosen to give FPIR of0.0003, 0.003, 0.03, 0.3 respectively one baseline demographic, here white males.




ongoing face recognition vendor test (frvt) part 3: … · 2019. 12. 19. · .ir.8280 2019/12/19...

Documents