evaluation methodology for analyzing usability factors in biometrics

Evaluation Methodology for Analyzing

Usability Factors in Biometrics

Belen Fernandez-Saavedra, Raul Alonso-Moreno, Jaime Uriarte-Antonio & Raul Sanchez-Reillo University Carlos III of Madrid - Dpt. Electronics Technology

ABSTRACT

In recent years, biometrics is being used more and more in security applications. This fact has led suppliers and researchers to analyze biometric algorithms powers and vulnerabilities, as to improve the feasibility of this technology. Nevertheless, as many authors claim, biometric performance also depends on other factors such as usability and/or user acceptance, which can influence significantly their performance. Only a few of these factors have already been studied, using specific approaches and only for certain biometric modalities, such us fingerprint and face. However, there Is no general and independent methodology implemented to assess how these factors affect biometric system performance and to produce intercom parable results.

Based on previous works and following procedures and requirements addressed in the International Standard ISOIIEC 19795-2 for scenario evaluations (1), these authors have developed a general methodology to analyze end-to-end system performance when some usability factors are modified. Such factors cover different ways of presenting biometric characteristics to the sensor and also tbe biometric characteristic variability caused by illness or climatic changes. A generic and controlled scenario has been modelled to carry on all sets of trials. Then, the methodology has been particularized defining specific protocols, methods, and considerations for each parameter to assess. Furthermore, details for analysing these parameters through different modalities have been defined. In addition, this methodology has been checked for one modality considering different usability aspects to obtain the feedback necessary to test its validity and viability and to detect points of interest for improvement. Results, main conclusions, and suggestions for test operators will be presented.

Author's Current Address: B. Fernandez-Saavedra. R. Alonso-Moreno, J. Uriarte-Antonio and R. Sancbcz·Reillo University Carlos III of Madrid - Opt. Eleetronics Technology, University Group for

'

Identification Technologies (GUTI), Avda. del Mar Mediterraneo, 22, E-28914 - Leganes (Madrid), Spain.

Based ?D a presentation at ICCST 2009; reviewed & recommended by the Program ComlDlttee.

0885/89851101 $26.00 @ 2010 IEEE

20

INTRODUCTION

Nowadays biometrics is already a technology very widespread in the market. This fact has caused suppliers to not only worry about the implementation of powerful algorithms and secwity systems, but also about other aspects such as usability or user acceptance for improving their products. However, an these aspects are interdependent. It is well-known that usability factors affect biometric system performance. Depending on the ease of presenting the biometric characteristics to the sensor and the usage of the biometric system, biometric performance can change. At the same time, both usability and biometric performance could influence user acceptance because if one system has a high performance rate or its usage is difficult, users will not want

to use it. In addition, user acceptance could also influence biometric performance since if a user does not trust a biometric system, that user will not use the system properly. Consequently, all these dependencies are very complex and it is difficult to analyze them. Furthermore, some factors related to user acceptance are very subjective.

One approach to quantify these influences and dependencies is to study such aspects separately. On the one hand,

. how usability influences biometric performance may be

exammed and, on the other hand, how user acceptance is affected by usability and biometric performance. Both analyses are depicted in Tables 1 and 2. These tables are composed of factors to consider during the evaluation and indicate what parameters should be varied for each factor.

The usability factors of Table 1 should be quantified by means of performance rates or error rates, throughput rates or sample quality, meanwhile factors of Table 2 concerning user acceptance should be analyzed using questionnaires and surveys where people can express their opinions after they have used the biometric system. These questionnaires or surveys should be developed and studied using a psychological perspective.

Some of these usability factors have already been analyzed using proper methods, but only for one biometric product or modality [2, 3]. There is no general methodology for doing it and it is essential to obtain repeatable and intercomparable results. General testing procedures are addressed in ISOIIEC 19795-2 [1], the International Standard for technology and scenario

IEEE A&E SYSTEMS MAGAZINE, AUGUST 201 0

Factors

Ease of presentation of biometric cbaracteristics

Easiness of user-system interaction

Learning process

Table 1. Usability Factors Tbat Affect Biometric Performance

Parameters to Consider

Sensor Different Positions

Modified

Anatomic Cbanges Climatic Cbanges

Conditions to present biometric

cbaracteristic

Age Knowledge

Guidance

Feedback

Age Knowledge

Temporal Illness

Moved

Moved

Modified

Otbers

Habituated user Unbabituated user

Witbout guides Non-attended guide

Attended guide

With feedback Witbout feedback

Habituated user Unbabituated user

Heigbts Inclinations

Damage or dirty

Temperature, Humidity, Illumination, Pressure, Noise

Cuts, scratcb, bites, burns, bruises, colds, allergies, inflammations,

conjunctivitis, etc.

Translation directions: up, down, left, right Rotations: roll, yaw

Spots of oils, paints, ink, creams, makeup, wet, dry, etc.

Accessories: gloves, scarves, bats, caps, glasses, contact lens, bair, rings, piercings,

etc.

Bebaviour: cbewing gum, expressions, emotions, different languages, etc.

Witb Tecbnical Knowledge Witbout Tecbnical Knowledge

Video Poster or pictograms

Audio

DUring tbe process At tbe end of the process

Witb Tecbnical Knowledge Witbout Tecbnical Knowledge

evaluation, where scenario evaluation includes the assessment of end-to-end biometric system perfonnance.

However, it does not specify particular approaches for testing the usability or user acceptance factors.


Table 2. Factors That Influence User Acceptance

Factors

User Perception

Parameters To. Consider

Different usability grades

Trusted Privacy

Satisfaction

Necessity

User cooperation

Healthy Security Safety

Assets

In order to solve this lack of test procedures, these authors have developed an evaluation methodology for analyzing the usability factors in biometric performance, but only for the most important usability parameters: biometric sensor and biometric characteristic variations (the first three parameters of Table 1). Trying to cover all of them is a very complex and hard task. These parameters are not only significant as usability parameters, but also as zero effort impostor attempts. This methodology is based on ISOIIEC 19795-2 [I] and is generic for all biometric modalities. It consists of describing a general scenario evaluation following ISOIIEC 1 9795-2 directives and then specifying particular requirements to assess how biometric performance is influenced by such usability parameters. These specifications include the essential changes needed regarding general scenarios previously defined.

The next section describes the common scenario for testing and all general considerations. It also explains the procedures for performing trials, data to collect, and results to obtain. In the following section specific scenarios are defined. It comprises the explanation of requirements and methods for analyzing each particular usability parameter and the main changes to perform in a common scenario. Then, in the Evaluation Results section, results of applying this methodology to a fingerprint biometric system considering several usability aspects are shown. Finally, conclusions and future work are presented.

COMMON EVALUATION SCENARIO

This is a common scenario for performing trials during the evaluation. To produce repeatable results, these authors specified the main characteristics the scenario must possess.

It is important to emphasize that the enrollment process must be executed in this scfDarlo, while the verification process shall be performed first in this scenario and then in

22

Table 3, Typical Indoor Conditions

Temperature

Humidity

Illumination

Noise

Pressure

22 °C * 4°C (standard conditions)

40 % to60% (standard conditions)

500 lux to 25 00 lux (Recommendations in [5 ) for indoor work places )

30 dB to 40 dB

(I'tj computer sound level)

1 atm (standard conditions)

similar scenarios with corresponding modifications according to the usability parameters under test.

General The first step of a scenario evaluation is to describe the

system under test and the objectives of the evaluation. This scenario has been designed to evaluate an end-ta-end system performance of biometric verification systems. For identification systems, the protocols for the verification process and some test procedures have to be adapted. Also different error rates and throughput rates should be obtained. Nevertheless, this scenario and most of these generic requirements could be applied for database collections in technology evaluations or similar performance tests.

Environment Environment is a factor that should be specified in two

senses: ambient conditions that surround biometric system and where and how biometric system is placed.

1) Ambient conditions: Because biometric system performance is sensitive to this aspect and is not the target of evaluation to measure this effect, ambient conditions are set to typical indoor conditions. Such conditions are shown in Table 3. These conditions shall be controlled depending on the biometric modality to assess because not all of them influence all modalities. Table 4 shows these dependencies.

This table is based on ISOIIEC TR 19795-3 [4], that addresses modality specifications.

IEEE A&E SYSTEMS MAGAZINE, AUGUST 2010

Table 4. Ambient Factors That Affect Each Biometric Modality

Ambient Factor Affected Biometric Modalities

Temperature Face, Fingerprint, Vascular, Voice and Hand Geometry

Humidity Face, Fingerprint, Vascular, Voice and Hand Geometry

Illumination Face, Vascular, Iris and Fingerprint (optical sensors)

Noise Voice and audio guides for all modalities

Pressure Signature

2) Biometric system placement: Sensor must be located as suppliers recommend. If there is no guidance, the sensor shall be placed straight, in front of test subjects, without inclinations. The height depends on the type of sensor:

• Desk device: between 70 to 75cm (standard height table).

• Wall device: sensor has to be placed according the average human height and the biometric characteristic to evaluate.

Test Crew Scenario evaluation shall be executed using a

representative test crew for a real-word application. In order to assess usability aspects for different biometric systems, this test crew must be general. The features of the appropriate population shall be as follows.

1) Demographics: For some biometric modalities, biologic differences between men and women or adults and children could influence biometric performance. In a generic scenario such factors have to be balanced to minimize their effects. These authors propose to consider the same characteristics defined at ISO/IEC 19795-5 [6] for access control scenario evaluations.

IEEE A&E SYSTEMS MAGAZINE AUGUST 2010

• Age: people 1 8 to 70 years old. The number of test subjects shall be equally distributed for each range of age: 18 to 30, 31 to 50, and 5 1 to 70. It shall be from 25 to 40% of the test crew.

• Gender: 40% to 60% of men and 60010 to 40% of women.

2) Size: is a difficult parameter to define. ISOIIEC 19795-1 [7] suggests applying the rule of three or the rule of thirty. Both rules consist of choosing the number of test subjects depending on error probability to obtain and the number of errors that take place. Nevertheless, it is necessary to know the expected error rate and have independent samples and is not always possible. For an error rate of 0.1 % (FNMR or FRR) without observed errors and considering the rule of three for a confidence level of 90%, 2,000 genuine comparisons should be executed. However, this quantity involves enrolling 2,000 test subjects (for independent samples). This is expensive and entails much effort. A reasonable test size is nearly 200 people. Such people should go twice and provide at least 5 genuine transactions per visit. This allows us to achieve the desired number of genuine comparisons, but it is necessary to know that samples are not totally independent. In addition, test subjects have to carry out 5 impostor transactions per visit to obtain FMR and FAR rates. We recommend an increase in the number of test subjects because some subjects could leave the evaluation process before its conclusion. These authors suggest adding another 10%.

3) Selection: Test subjects shall be randomly selected or volunteer, but they must meet age and gender requirements mentioned above. They do not have to be involved during the biometric system implementation or tune.

4) Training and Guidance o/ Test Subjects: It is another parameter that could modifY biometric system performance. It influences test subject behaviour, and therefore, some requirements have to be specified.

• Test information: must include general information about the evaluation: objectives of evaluation; number of visits; and legal issues related to the participation of test SUbjects. These authors recommend not providing information about the real purpose of the evaluation so test subjects do not change their behaviour.

23

• Test instructions: A generic explanation about the biometric system under test and the particular modality shall be provided to test subjects. Then, phases of evaluation (enrollment and verification) and tasks to execute during each phase have to be described.

• Training: Previous evaluation, two or three practical trials shall be performed by test subjects in the presence of test operators; they shaJI correct test subjects mistakes. It is very useful if test operators also show test subjects some examples about correct vs. incorrect ways to interact with the system.

These authors suggest having more than one sensor. One sensor might be installed for training in a previous step to perform enrollment or verification transactions with another sensor installed in the actual biometric system.

• Guidance: Test subjects shall be guided during training and enrollment process . During verification process, one test operator must be checking that the test subject presents the right biometric characteristic to the system.

• Feedback: All available information related to the capture of biometric samples (quality) and the comparison (the outcome of enrollment or verification attempts) will be captured, and shall be shown to test subjects during their interactions and at the end, respectively.

5) Visits: The number of visits should be at least two. More than one visit improves the crew training. At the first visit test, subjects shaJI perform two or three trials for training, and then complete the enrollment process and the set of verification transactions. At subsequent visits, test subjects will have to carry out only the set of verification transactions. The second and subsequent visits should be planned for at least one week later. During different visits, and in all processes, transactions shall be done with disengagement from the device.

Levels of Effort and Deeision Policies If the biometric system does not have a fixed restriction

for the maximum number of transactions and maximum time per transaction, it shall be:

24

• Two transactions for enrollment. If the first transaction fails, a second transaction must be executed.

• Three attempts per each verification transaction.

• The maximum time for enrollment and verification attempts depends on how long these processes take in the case of the system under test. We suggest 2.5 times the average time for carrying out this process. If this time is unknown, the test operator should execute trials for measuring this time before the evaluation.

Error Protocols Some protocols must be specified for typical errors that

could happen during the evaluation. These are as follows :

1) General errors: If the sensor or other parts of the biometric system fail, the test operator has to stop the evaluation and solve the problem. Once repaired, the test operator shall check that everything works correctly and shall continue with the evaluation. He shall report the error and the approach to solving it.

2) Enrollment and verification errors: If the test operator introduces a wrong identifier, he must cancel the process and execute a new transaction. If the test subject presents a wrong biometric characteristic, the test operator must cancel the process, and inform the test subject about the mistake. A new transaction shall be carried out.

Data to Record and Test Results Biometric performance is obtained by means of

calculating metrics and statistics using data collected during the evaluation. Such results, as well as all information about test planning and test execution, must be reported at the end of the tests.

Relevant information to be saved and rates to be generated are as follows:

1) Data to record

• Total number of enrolled test subjects: men and women.

• Number of failures at enrollment process.

• Enrollment time for each attempt.

• Number of failures to acquire.

• Total number of attempts : number of genuine and impostor attempts.

• Number of fail attempts at verification process: number of genuine and impostor attempts.

• Verification time for each attempt.

• Total number of transactions at verification process : number of genuine and impostor transactions.

IEEE ME SYSTEMS MAGAZINE, AUGUST 201 0

• Number offailed transactions at verification process: number of genuine and impostor transactions.

• Similarity scores.

• Hardware and software information: computer settings and operating system.

2} Test results

• Error rates and graphics

FTE, FTA, FNMR, FMR, FAR, FRR.

DET, ROC and distribution curves.

Uncertainties of estimates.

• Throughput rates: These rates are composed of average, median, maximum, minimum and standard deviation of the following times:

• Enrollment time: This time begins when the test subject interacts with the biometric application to save his template, and ends when the biometric system shows the outcome of the operation.

• Verification time: This time begins when the test subject interacts with the biometric application to verify his identity, and ends when the biometric system shows a match decision.

Tests Proeedures and Execution Sequence The course of the evaluation shall follows a specific order

so we can be assured that all evaluation tasks are executed. These authors suggest working according to the following:

I} Pre-test activities

• Analyze the biometric system under test.

• Implement the essential application for performing the evaluation. This application has to conform with the levels of effort and decision policies and has to collect all relevant information.

• Plan the evaluation scenarios including materials, tools, and test instruments to control scenario settings.

• Plan the evaluation schedule and the recruitment of test subjects.


• Develop information, legal and data forms and guides for test subjects.

• Instruct test operators about system operation, error protocols and other evaluation features.

• Calibrate test instruments.

2} Test activities

• Procedures before the first visit

Recruit test subjects.

Establish operational environment for enrollment (common scenario settings).

Install the biometric system in the operational environment.

Conduct a system operability analysis for enro]Jment and verification processes.

Prepare resources for evaluation scenarios.

• First visit

Explain test instructions to test subjects and how to fill in user forms (data and legal forms). At this time, test subjects become acclimatized.

Carry out user training for enrollment.

Enrollment: Test subjects shall perform the first transaction and if it fails, they must perform a second transaction.

• Verification in scenarios.

Train test subject for the verification process.

Peiform the first session of genuine and impostor transactions in the common scenario.

Modify the scenario. When modifications entail operational environment changes, the biometric system shall be installed in this scenario again and its operability shall be checked. Test subjects shall wait some time for acclimatation; otherwise, when these modifications mean changing the biometric characteristic or user behaviour, the test operators must instruct the test subjects.

Peiform the first session of genuine and impostor transactions for such scenarios.

25

Table S. Scenarios For Assessing Sensor Position

Height

Standard Height Variation

Inclination

Scenario

Very low height

Low height

High height

Very high height

Little slope

Slope

Value

-30cm

-lScm

+15 em

+30cm

The last two steps have to be done for the evaluation of all the specific scenarios.

• Subsequent Visits

Establish a common scenario and install in it the biometric system; then, check that it works.

Remind test subjects about the test instructions for the verification process.

Perform a session of genuine and impostor transactions in such common scenarios.

Execute the specific scenario modifications and verification transactions as indicated at the first visit for usability scenarios.

3) Post-test activities

Analyze all information collected during the evaluation.

Calculate performance rates, graphics, and metrics as mentioned above.

Generate reports including test planning, test procedures, test crew features, results, and all evaluation details for each specific scenario.

USABILITY FACTORS AND ITS EVALUATION

Once the common evaluation scenario has been specified, all variable parameters to modify each usability factor will be described.

26

Table 6. Modifications Depending Of The Type Of Sensor

Type Of Sensor Modifications

Sensors with camera Dust

Sensors without camera (swipe, capacitive, etc.)

Grated surface or lens

Pen tablets or Dirt (drops, grease, fluff, etc.) digitizing tablets

Microphone Dust

Sensor

Dirt (drops, grease, fluty, etc.)

One of the most important parts to analyze in usability evaluation is the biometric sensor. Its localization and condition could val)' depending on the application and over time. Both parameters are considered in the following scenarios.

1) Different positions: To assess this parameter, test operators only have to modify the placement of the sensor. The other factors are similar to those of the generic scenario. Specific values for sensor location are shown in Table S.

2) Modified sensor: This parameter involves damaging or dirtying the biometric sensor. Test operators must have more than one sensor to evaluate these aspects during the same evaluation of others. In Table 6, the major modifications that could suffer from damager or dirty sensors are detaiJed.

Anatomic Changes Another important part to analyze in usability tests is the

biometric characteristic. It could be changed by means of environmental influence or due to diseases. Next, the specific evaluation scenarios for such variations will be specified.

1) Environmental anatomic changes: To analyze these changes, the operational environment shall be adjusted to produce specific variations. Scenarios to assess are summarized in Table 7. The biometric characteristic and biometric system must


Table 7 . Scenarios For Testing Environmental Anatomic Changes

Temperature ( 0C)

Humidity RH (%)

Illumination (lux) Infrared

Outdoor

Darkness

Noise (dB)

Pressure (atm)

be exposed to environmental factors during a period of 10 minutes, at least, before starting verifications. Also, environments must be controlled and scenario conditions kept during test subject interactions.

As mentioned above, not all ambient factors affect all biometric characteristics equally. Depending on the biometric modality of the system under test, some scenarios have to

be taken into account. These authors suggest basing selections according to Table 4.

2) Changes produced by illness: There are several illnesses that could affect the biometric characteristic or biometric behaviour. Usability evaluation only


Seenario

Cold

Veryeold

Very high

Very low humidity

Very high humidity

Visible

NIR

Visible

NIR

Visible

NIR

High noise

Very high noise

Low pressure

Very low pressure

Value

-SOC-ooc

-lO°C--soC

400C-SOoC

0 -/0- 1 0 0/.

90%- 100%

1 ,0 00 to 2,500 lux

° to 3 ,000 lux

1 ,0 00 to 3 S,000 lux

° to 1 0 ,000 lux

Oto 1 00 lux

Oto 1 00 lux

60 dB

7 SdB

0.79 atm ( 1'::1 1 500 m)

0.66 atm (1'::1 3000 m)

covers mild illnesses that users could suffer from but not prevent them using biometric sensors. Maybe for usability analysis, chronic illnesses should also be studied, but they are not covered herein. To test this usability factor, only features concerning the test crew should be modified. People must be enrolled with their healthy biometric characteristic and after that, they must be verified when they are ill. The problem is how to get enough people to obtain significant results. In Table 8, a set of the most common illnesses and the affected biometric modality are listed.

Conditions to Present Biometric Characteristic Finally, the last usability parameter to determine is a

combination of the biometric characteristic and sensor. It is

27

Table 8. Mild Illness

Illness

Cuts. scratcbs, bites burns, bruises

Conjunctivitis

Cold

Allergies

Inflammations

Affected Biometric Modalities

Face, Fingerprint, Vascular, Hand Geometry,

Iris, and Signature

Face and Iris

Face, Iris, and Voice

Face, Fingerprint, Vascular, Hand Geometry,

Iris, Voice, and Signature

Face, Fingerprint, Vascular, Hand Geometry, Iris, Voice, and Signature

the way to present the biometric characteristic to the sensor. There are two aspects to assess. On the one hand, if the biometric characteristic is moved with respect to the common position, and on the other hand, if the biometric characteristic has suffered modification due to cosmetics, chemical products, accessories, user behaviour, etc. Scenarios to evaluate these parameters are described below.

1) Moved: This test analyzes performance when the sample is not centred. To check this factor, the way the users present their biometric characteristic to the sensor has to be modified, thus, training and user guides shall be adapted for these scenarios.

Test scenarios must cover the fact that the test subjects have to present their biometric characteristic including four translational positions (up, down, right, left) and two rotations (roll and yaw). Figure 1 shows the area and the centre for each translational position and the potential turns for each rotation.

All of these positions are possible for sensors such as cameras or devices with a useful square surface. For tablets used for signing, roll rotation is not possible. In the case of the voice modality, any of these movements are possible but

28

roll

Fig. 1 . Translations and Rotations

Table 9. Modifications of Biometrics Cbaracteristics

Modifications

Cream and spots of oil, ink, etc.

Wet or dry

Gloves

Scarves

Hats, caps, and glasses

Contact lens

Make up, bair and/or fringe

Moustacbe, beard

Rings

Biometric Modalities

Face, Fingerprint, and Vascular

Face, Fingerprint, Iris, and Vascular

Fingerprint, Vascular, Signature, and

Hand Geometry

Face and Voice

Face and Iris

Iris

Face and Iris

Face

Vascular, Hand Geometry, and Signature

Piercings, ortbodontics, Voice and Face cbewing gum

Expressions

Emotions

Different languages

Face and Iris

Voice, Face, and Iris

Voice and Signature

they have to be quantified taking into account the distance to the microphone.

No specific values are set. Depending on the biometric sensor, test operators must select the type of scenario and quantify the movement.


2) Biometric characteristic modifications: For testing biometric performance when the biometric characteristic is modified in daily life. Usability scenarios for several modifications and the biometric characteristic that is affected are shown in Table 9, which has been generated according to ISOIIEC TR 19795-3 [4].

EVALUATION RESULTS

Once the evaluation methodology and usability scenarios have been defined, a series of tests have been carried out for a fmgerprint verification system to validate it The overall evaluation and results obtained will be explained in the following paragraphs.

The biometric system under test consisted of an optical fmgerprint sensor connected to a common PC. Using its SDK, a verification system was implemented. It conformed with the level of effort and decision policies specified.

• Seenario 1: Common Scenario. It is mandatory for the evaluation for testing the system when it is used normally.

• Seenario 2: Sensor Position Scenario. The biometric system was used when its sensor was placed with a slope of 400. It is an example of a case in which this sensor (desk device) is placed in an A TM where the surface is usually sloping.

• Seenarios 3 and 4: two diJlerent scenarios related to conditions to present the biometric characteristic and moved presentations.

In Seenario 3 , test subjects were instructed to present their biometric characteristic one centimetre up from the centre of the usable fingerprint area; and

in Scenario 4, one centimetre down. The supplier does not recommend using the system in that way but authors chose these scenarios to check the methodology for factors that are known to be influent

• Scenario 5 : one more scenario related to conditions to present the biometric characteristic, but taking into account finger modifications. For this scenario fingers were modified using hand cream (e.g., a system close to the beach).

Evaluation settings conformed with the common scenario. It was done in typical indoor conditions with a temperature


of25.4°C and a relative humidity of 47%. Illumination was fluorescent, in the visible range and its intensity varied between 1,500 and 2,200 lux depending on the specific wavelength. Other environmental factors were not considered for this sensor. Regarding the system placement, as the sensor was a desk device, it was located in a standard table, straight for all scenarios except for Scenario 2 that was tilted at an angle of 40°.

The platform of the biometric system was a Core 2 duo U7600 laptop with 1.2 GHz, 2GB of RAM, and using WindowsXP.

Test crew size was selected taking into account the supplier claim, the rule of three with a 95% of confidence level, and the time and effort enabled us to find test SUbjects.

According to the supplier, the sensor has an FRR = 0.34% (for zero FAR); hence, applying the rule of three, means that at least 883 genuine comparisons have to be executed. These authors decided to perform the same number of impostor comparisons. Due to the difficulty of finding this number of persons and considering that this evaluation aim was only the assessment of the proposed methodology, this quantity of comparisons was achieved but using samples of the same person (not totally independent).

Ten individuals were recruited and each person provided his middle and index finger of both hands. These authors selected this amount of individuals because they would provide enough samples, and always taking into account that someone could leave the evaluation. The biometric characteristics were chosen because these are the fingers recommended by the supplier. Each person had to perform five genuine and five impostor comparisons, at least, per visit and the number of visits was two. This number of comparisons is similar to the one suggested in the common scenario. It was chosen like that to check how much effort it entailed for test subjects.

Test subjects were instructed by the test operator during the first visit for enrollment and verification process. To explain how to use the device, user guides provided by suppliers were used and the test operator showed correct vs. incorrect usage with examples. Also, test subjects took part in practical trials before carrying out real transactions.

For Scenarios 3 and 4, specific instructions were given about how to present the biometric characteristic to the sensor with the corresponding movement, up or down.

Visual feedback was shown to test subjects during their interactions with the device and at the end of each attempt. This way, they knew image quality when they presented their biometric characteristic and also the match decision.

In addiction� test operators guided each test subject only at the enrollment process and they applied policies and error protocols addressed in the common scenario.

The evaluation was performed following the test sequence and all test procedures that were described in the common scenario. Once it was finished, outcomes were analyzed and the most important metrics and graphics were obtained.

29

Table 10. Results of Enrollment

Enrollment Results Common Scenario

Number of enrolled users 10 (6 women, 4 men)

Number of templates 40 (right and left index and middle finger

per person)

FTE 5% first enrollment 0% second enrollment

Time to enroll

Average

Minimum

Maximum

Standard Deviation

16.282 s

8. 15 s

82.86 s

1 1 .728 s

These are depicted in Table 1 0 that contains the enrollment results and also in Table I I and Figure 2 showing verification results.

Enrollment was carried out at the common scenario. The number of enrolled users and templates corresponded to the test crew characteristics mentioned above. FTE error rate indicates that the decision policy of performing a second enrollment is a good decision. Some test subjects made mistakes during the first enrollment even though they received training before. Executing a second enrollment after the correction of the errors provides successful templates.

This process did not take too much time because it included capturing three images to create a template and also guiding test subjects when necessary. This is why maximum time is higher, because during enrollment, the test operator has to reinstruct the test SUbjects.

Verifications were performed in all scenarios described previously. Table 1 1 summarizes the results obtained during the evaluation. At the end, the number of comparisons exceeds the number calculated for obtaining significant results, in spite of the fact that some users did not perform the second session as was expected. The differences in this amount between scenarios are due to some test subjects not attending the second visit and others performed more than 5 comparisons.

After the evaluation, these authors suggest implementing some methods to help the test operators during the count of the number of comparisons if the evaluation is to be strict in this aspect.

30

FT A error rates have already disclosed the differences of usability scenarios. This rate is higher for Scenarios 3 and 4, in which the way to present the biometric characteristic was modified. Time statistics show that usability parameters do not affect throughput rates; these are similar for all scenarios.

Finally, DET curves for scenarios are illustrated in Figure 2. It reveals the influence of usability parameters in FNMR and FMR error rates and it is possible to measure and compare. For the biometric system under test, performance is modified by all tested usability parameters. For Scenarios 3 and 4, influence is considerable; but for Scenarios 2 and 5 it is not too much. Depending on the specific application and the grade of security level to achieve, this system will be appropriate or not, but now suppliers and customers may quantify it using this methodology.

DET 1 0

2��r-nn

H H ... - - -I - � -I - 1 -1 ... H I- - - I- - I- ... .... ... H H - - - 1 - - 1 - t- I- H + 1 " r l T - - , - T -. -" T " r - - r - r- i , . T n n - - - , - - 1 - r r 'i T I '"'1 rl t - - ...., - "'t ':"1 - 1 '"1 T 1"1 r- - - r- - r- ..,. ..,. T H H - - - t- - 1 - r r '""1 T I , _, 1.' 1. _ _ J _ .1 _, _oj 1. t±L _ --1...._ '- .1 .1 .1 U IJ _ _ _ , _ _ , _ L L ,-, .1 '

I I I I I I I I I I I I I I I -,.-......... w I I I I I I I I I I I 1 I I I ' I I I 1 1 1 1'·11 _ ..... ' I I I I I I I

1 0 I =' � I � = = � = � =t =, � _ - t: = t: l. :t r tl c = = = �;:I = t: � ::1 :I I

:� �: � � � � � ��} r ,:1 � � � _ � � � � � � � � � � :� �:s. �:3 �: I ...J L I l. _ _ .J _ .l _I _I .J l. l.J _ .i l. U U _ _ _ I _ _ I _ L. �1...J 1.1 I...J Ll l. _ _ .J _ .1 -1 _I .J L I.J L _ _ L _ U _ _ _ 1 _ _ 1 _ L L ��' 1 1 1 I I I 1 1 1 1 1 1 1 1 I 1 1 1 1 1 1 I I I I I I I I I n r - , - T -I -" T n r - - r - r i T T n - - 1 - - 1 - r r II I 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

I...J L I _ _ .... .1 _I _I J .L IJ L _ _ L _ L J. .1 .L U U _ _ . _1 _ L L I...J I I� � I ! :: = � �'! =' :t � i;3�··�.§ E � § � � � g � � � � , _ = = = � :j � • Scenario 1 , T I' I - """� - r ,. ,. r n n - - - , - - 1 - rt

'I T 11 1- � - I ·""'�I I T '-I II - - - , - - , - , 1-" • Scenario 2 -, r ,-, ,- - - r - r , -p T

. - - - , - -, - r

Scenario 3

1 0" Scenario 4 I I I ' I I I I I I I I I I " I I I I I

� Scenario S H :� � � � � � � � H � � = � �:,!�, = � � :� �, - - - - - - - - - - - - - - - - - - - - - - - -

FMR (%)

Fig. 2. DET Curves for Different Scenarios

CONCLUSIONS AND FUTURE WORKS

Herein, a general and independent methodology has been implemented to assess the influence of usability factors in biometric perfonnance. First, a common and controlled scenario was specified based on ISOIIEe 1 9795-2 and then, some modifications were defined (taking into account usability parameters to evaluate). After that, the methodology was validated with a fingerprint biometric system analyzing different usability aspects. Results have disclosed that this methodology is feasible for carrying out independent and intercomparable tests and for quantifying usability effects.

Nevertheless, this methodology cannot be checked with other modalities and there are usability parameters that could not be considered herein. Both issues are relevant and it is important to continue research in this area.


Table 1 1. Results of Verifications

Verification Results 1

No. of samples 2,080

No. genuine compo 1,056

No. impostor compo 1,024

FTA (%) 0.913

FTA gen. (%) 0.189

FTA imp. (%) 1.660

Time to capture (s) Av. 2.02

Min. 1 .14

Max. 8.48

St. Des. 0.517

ACKNOWLEDGMENTS

We thank all of our test subjects that collaborated during the evaluation process. Thanks very much for your time and your patience. This work was developed within the F PU fellowship program, funded by the Spanish Ministry of Science and Education.

REFERENCES

[IJ ISO/IEC 1 9795-2:2007, Information Technology - Biometric Performance Testing and Reporting - Part 2: Testing methodologies for technology and scenario evaluation.

[2] M. Theofanos, S. Orandi, R. Michaels, B. Stanton and N. Zhang, Effects of Scanner Height on Fingerprint Capture,

National Institute of Standards and Technology, NISTIR 7382, December 14, 2006.

IEEE A&:E SYSTEMS MAGAZINE, AUGUST 2010

2 Scenarios

3 4 5

2,046 1 ,923 1,944 1,920

1 ,026 978 982 960

1,020 945 962 960

0.684

0.097

1.275

2.15

1.19

7.65

0.556

8.736 18.46 0.365

10.94 19.24 0.208

6.455 17.67 0.521

1 .92 1 .839 1 .786

0.68 0.93 0.91

9.41 10.43 4.53

0.548 0.573 0.391

[3] M. Tbeofanos, B. Stanton, C. Sheppard, R. Michaels, N. Zhan, J. Wydler, L. Nadel and W. Rubin,

Usability Testing of Height and Angles of Ten-Print Fingerprint Capture,

National Institute of Standards and Technology, NISTIR 7504, June 2008.

[4J ISOIIEC TR 1 9795-3, Information Technology - Biometric Performance Testing and Reporting - Part 3 : Modality specific testing.

[5] CEN EN 1 2464-\ Light and lighting - Lighting of work places Part 1 : Indoor work places.

[6] ISO/IEC CD 1 9795-5, Information Technology - Biometric Performance Testing and Reporting - Part 5: Grading Scheme for Access Control Scenario Evaluation.

[7] ISOIIEC 1 9795-1 :2006, Information Technology - Biometric Performance Testing and Reporting - Part I : Principles and

framework. 4

3 1

evaluation methodology for analyzing usability factors in biometrics

Documents