hiring the most qualified firefighter while avoiding lawsuits

Fire & Police Selection, Fire & Police Selection, Inc.Inc.

Hiring the Most Qualified Hiring the Most Qualified Firefighters….Firefighters….

……While Avoiding While Avoiding

(and Defending) Lawsuits(and Defending) Lawsuits

Copyright Copyright © 2011 © 2011 Fire & Police Selection, Inc.Fire & Police Selection, Inc.

Contact InformationContact Information

Dan A. Biddle, Ph.D.Dan A. Biddle, Ph.D.Stacy L. Bell, M.S.Stacy L. Bell, M.S.CEO, Biddle Consulting Group, Inc./Fire & Police CEO, Biddle Consulting Group, Inc./Fire & Police

Selection, Inc.Selection, Inc.193 Blue Ravine, Ste. 270193 Blue Ravine, Ste. 270Folsom, CA 95630Folsom, CA 956301-888-990-34731-888-990-3473www.biddle.comwww.biddle.com | www.FPSI.com | www.FPSI.com

Email: Email: [email protected] | [email protected] | [email protected]@fpsi.com

http://www.biddle.com/

mailto:[email protected]


Biddle Consulting Group, Biddle Consulting Group, Inc.Inc.

• Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense)

• EEO Litigation Support / OFCCP (federal contracting) Audit Support• Compensation Analyses / Test Development and Validation• Published: Adverse Impact and Test Validation, 2nd Ed., as a

practical guide for HR professionals• Editor & Publisher: EEO Insight an industry e-Journal • Creator and publisher of a variety of productivity Software/Web Tools:

– OPAC® (Administrative Skills Testing)– CritiCall® (9-1-1 Dispatcher Testing)– AutoAAP™ (Affirmative Action Software and Services)– C4™ (Contact Center Employee Testing)– Encounter™ (Video Situational Judgment Test)– Adverse Impact Toolkit™ (free online at

www.disparateimpact.com)– AutoGOJA® (Automated Guidelines Oriented Job Analysis®)

IndustryLeader


Fire & Police Selection, Inc.Fire & Police Selection, Inc.

• Over 80 Physical Ability Test installations at fire departments Over 80 Physical Ability Test installations at fire departments across the U.S.across the U.S.

• Over 800 personnel and fire and police clients across the U.S. Over 800 personnel and fire and police clients across the U.S. and Canadaand Canada

• Entry-level written, physical, interview assessments, Entry-level written, physical, interview assessments, manipulative skills tests, cognitive ability tests, and personality manipulative skills tests, cognitive ability tests, and personality tests available for selection of public safety personneltests available for selection of public safety personnel

• Post-Academy certification testsPost-Academy certification tests• Fire Promotional tests and assessment exercisesFire Promotional tests and assessment exercises• Work Styles Inventory for Work Styles Inventory for predicting firefighter and police predicting firefighter and police

officer job performanceofficer job performance• Comprehensive Examination Battery for Entry-Level FirefightersComprehensive Examination Battery for Entry-Level Firefighters• Situational Judgment/Writing Ability Test for Entry-Level Police Situational Judgment/Writing Ability Test for Entry-Level Police

OfficersOfficers• NewNew National Fire Select Test (NFST National Fire Select Test (NFST™) for Entry-Level ™) for Entry-Level

FirefightersFirefighters


Fire & Police Selection, Inc.Fire & Police Selection, Inc.

• Overview of Today’s Presentation:Overview of Today’s Presentation:• The modern litigation landscapeThe modern litigation landscape• Litigation triggersLitigation triggers• Validation 101: Building and using testsValidation 101: Building and using tests• A quick tour through an “ideally A quick tour through an “ideally

weighted” test for firefightersweighted” test for firefighters


Approach to Test ValidationApproach to Test Validation

The Quickly Changing Climate of The Quickly Changing Climate of Personnel Testing…Personnel Testing…

•EEOC•DOJ•DOL (OFCCP—Office of Federal Contract Compliance Programs)•State EEO Offices of Fair Employment•Private Plaintiff Groups


The Quickly Changing Climate The Quickly Changing Climate of Personnel Testing…of Personnel Testing…

Title VII Cases Filed with EEOC

10,000

14,000

18,000

22,000

26,000

30,000

34,000

FY1997

FY1998

FY1999

FY2000

FY2001

FY2002

FY2003

FY2004

FY2005

FY2006

FY2007

FY2008

FY2009

FY2010

Year

# C

ases


Adverse Impact & Test ValidationAdverse Impact & Test Validation-Uniform Guidelines & Civil Rights Act of -Uniform Guidelines & Civil Rights Act of 19911991

Amends Section 703 of the 1964 Civil Rights Act (Title VII)Amends Section 703 of the 1964 Civil Rights Act (Title VII)(k)(1)(A). An unlawful employment practice based on(k)(1)(A). An unlawful employment practice based ondisparate impact is established under this title only if:disparate impact is established under this title only if:

A(i) a complaining party demonstrates that a respondent uses a A(i) a complaining party demonstrates that a respondent uses a

particular employment practice that causes a disparate impact particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent practice is job-related for the position in question and consistent with business necessity; with business necessity; OROR,,

A(ii) the complaining party makes the demonstration described A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an in subparagraph (C) with respect to an alternate employment alternate employment practicepractice, and the respondent refuses to adopt such alternative , and the respondent refuses to adopt such alternative employment practice.employment practice.


How Can Testing Practices be How Can Testing Practices be Challenged? Title VII Disparate Impact Challenged? Title VII Disparate Impact Discrimination FlowchartDiscrimination Flowchart

Practice,Practice,Procedure,Procedure,

or Test (PPT)or Test (PPT)

AdverseAdverseImpact?Impact?

YESYES NONO

Is the PPTIs the PPTValid?Valid?

YESYES NONO

Alternative Alternative EmploymentEmployment

Practice?Practice?

NONODefendant PrevailsDefendant Prevails

YESYESPlaintiff Plaintiff PrevailsPrevails

ENDEND

PlaintifPlaintifff PrevailsPrevails

““OROR””


Validation StandardsValidation Standards

Court PrecedenceCourt Precedence Title VII: Uniform Guidelines on Employee Selection Title VII: Uniform Guidelines on Employee Selection

Procedures (1978): EEOC, DOJ, CSB, DOLProcedures (1978): EEOC, DOJ, CSB, DOL Principles for the Validation and Use of Personnel Principles for the Validation and Use of Personnel

Selection Procedures (2003)Selection Procedures (2003)– Division 14 of the American Psychological Association (the Division 14 of the American Psychological Association (the

Society for Industrial and Organizational Psychology, or Society for Industrial and Organizational Psychology, or SIOP) SIOP)

Standards for Educational and Psychological Testing, Standards for Educational and Psychological Testing, published by:published by:– American Educational Research AssociationAmerican Educational Research Association– American Psychological Association, and American Psychological Association, and – National Council on Measurement in EducationNational Council on Measurement in Education


What is Validation?What is Validation?

Traditionally, it is making sure a Traditionally, it is making sure a selection procedure (i.e., test) selection procedure (i.e., test) measures what it is designed to measures what it is designed to measuremeasure

In a legal realm, a selection procedure In a legal realm, a selection procedure is valid if it can be proven by an is valid if it can be proven by an employer that it is “…job related employer that it is “…job related for for the position in question the position in question and consistent and consistent with business necessity.”with business necessity.”


Uniform Guidelines Style Uniform Guidelines Style Validation…Validation…

Content ValidityContent Validity – Section14CSection14C

Criterion ValidityCriterion Validity– Section 14BSection 14B

Construct ValidityConstruct Validity– Section 14DSection 14D


What Types of Validation are What Types of Validation are REALLY USED in Practice?REALLY USED in Practice?

ContentContent (80%) (80%)CriterionCriterion (15%) (15%)ConstructConstruct (5%) (5%)

Why?Why?– SimplicitySimplicity– DefensibilityDefensibility


CannotCannot alone support inferences alone support inferences about mental processesabout mental processes– Cannot measure traits (such as Cannot measure traits (such as

intelligence, aptitude, personality, intelligence, aptitude, personality, commonsense, judgment, leadership, commonsense, judgment, leadership, spatial ability)spatial ability)

NotNot appropriate for those KSAs that appropriate for those KSAs that will be expected to learn on the jobwill be expected to learn on the job

Content ValidityContent Validity


Content Validity Essential #1: Content Validity Essential #1: Job Analysis ResearchJob Analysis Research

Level 1: Job Description/Job Analysis Level 1: Job Description/Job Analysis ReviewReview

Level 2: Test Customization Survey Level 2: Test Customization Survey (TCS)(TCS)

Level 3: Full Job Analysis (GOJA)Level 3: Full Job Analysis (GOJA)


What Ratings are Typically Collected What Ratings are Typically Collected During a Content Validity Job Analysis?During a Content Validity Job Analysis?(Level 2-3)(Level 2-3)

For For Job DutiesJob Duties ratings are typically collected for: ratings are typically collected for:– Frequency Frequency – ImportanceImportance* *

Guidelines , 14.C[1,2,4]Guidelines , 14.C[1,2,4]– Best WorkerBest Worker* *

Guidelines , 14.C[9]Guidelines , 14.C[9]– FundamentalFundamental

Americans with Disabilities ActAmericans with Disabilities Act– AssignableAssignable

Americans with Disabilities ActAmericans with Disabilities Act

* = required under federal Guidelines for Content Validity


For For KSAKSAs ratings are typically collected for:s ratings are typically collected for:– FrequencyFrequency– ImportanceImportance* *

Guidelines, 14.C[1,2,4]Guidelines, 14.C[1,2,4]– Best WorkerBest Worker

Guidelines, 14.C[9]Guidelines, 14.C[9]– Minimum v. Desirable QualificationsMinimum v. Desirable Qualifications

Guidelines, 14.C[4]Guidelines, 14.C[4]– Level needed upon entryLevel needed upon entry

Guidelines, 5.F, 14.C[1]Guidelines, 5.F, 14.C[1]– Level needed for successLevel needed for success

knowledge onlyknowledge only

* = required under federal Guidelines for Content Validity

What Ratings are Typically Collected During a What Ratings are Typically Collected During a Content Validity Job Analysis? (Level 2-3)Content Validity Job Analysis? (Level 2-3)


Content Validity Essential #2: Targeting Content Validity Essential #2: Targeting Key KSAs for Test BuildingKey KSAs for Test Building

Only certain KSAs can be Only certain KSAs can be tested…tested…– Only “concrete” KSAs can be tested Only “concrete” KSAs can be tested

under content validity (cannot test under content validity (cannot test abstract KSAs)abstract KSAs)

– KSAs need to be “operationally KSAs need to be “operationally defined”defined”

– Important/Critical for the jobImportant/Critical for the job– Needed “Day One” on the jobNeeded “Day One” on the job– Duty/KSA LinkagesDuty/KSA Linkages


Firefighter Job Analysis Firefighter Job Analysis DutiesDuties SampleSample

Firefighter/EMT-I, Firefighter-PM and Fire Engineer Duties

Supervisor Ratings

Domain A Station Duties

Frequency

Importance

BestWorker

Fundamental

Assignable

A1

Arrives on time, receives and follows orders, instructions, regarding the work to be done.

8.75 4.88 5 yes no

A2Recognizes the need and makes recommendations for entries in daily log.

8.25 3.13 4 yes no

A3

Takes inventory and records and reports equipment and apparatus missing or needing repair or replacement.

8.25 4.00 4 yes yes


Firefighter Job Analysis Firefighter Job Analysis KSAPCsKSAPCs SampleSample

Firefighter/EMT-I, Firefighter-PM, and Fire Engineer KSAPCs

Supervisor Ratings

Domain O Knowledges, Skills, Abilities, and Personal Characteristics (KSAPCs): The following knowledge skills, abilities, and personal characteristics affect work behavior and are necessary or important for successful job performance.

This KSAPC is linked to

the following duties.

Frequency

Importance

MQ vs. HQ

Best Worke

r

Level Needed

Upon Entry

O53

Willing to be resourceful, goal-oriented, and proactive in bringing tasks to completion.

A1, A2, A24, C5

7.25 4.13 MQ 1 4

O54

Willing to remain calm under pressure or heavy workloads, not easily given to hostility, anxiousness, or vulnerability. Wiling to make sensible decisions under pressure.

A5, A20, E1, F8, G2

7.63 4.63 MQ 1 4

O55Ability to maintain respect for other’s emotional needs following a crisis.

A5, A20, C6, F11

6.38 4.13 HQ 1 3


Selection PlanSelection Plan

Identify appropriate weights for the testing components.Identify appropriate weights for the testing components.

– Completing a selection plan is the step in the validation process Completing a selection plan is the step in the validation process where the key, measurable KSAPCs and Physical Requirements where the key, measurable KSAPCs and Physical Requirements are laid out as targets to be assessed by the selection process.are laid out as targets to be assessed by the selection process.

– A selection plan distills the complete list of KSAPCs and Physical A selection plan distills the complete list of KSAPCs and Physical Requirements into only those that can and should be tested by Requirements into only those that can and should be tested by one or more selection procedures in the overall selection process.one or more selection procedures in the overall selection process.

– The result is a detailed plan that identifies the critical KSAPCs to The result is a detailed plan that identifies the critical KSAPCs to measure, five necessary ratings, and how the device should be measure, five necessary ratings, and how the device should be used (e.g., pass/fail, banding, etc.)used (e.g., pass/fail, banding, etc.)

– The final step is to identify the weights for the various The final step is to identify the weights for the various components.components.


Selection Plan—Selection Plan—Sample Sample

KSAPC/KSAPC/

Phys Req.Phys Req.KSAPC/Physical Requirement KSAPC/Physical Requirement

RatingRatingSelection Procedure and Selection Procedure and Use (Use (PPass/Fail, ass/Fail, RRank, or ank, or

CCombine*)ombine*)

Knowledge, Skills, Knowledge, Skills, Ability, Personal Ability, Personal Characteristic, or Characteristic, or Physical Physical RequirementRequirement

Best W

ork

er

Best W

ork

er

(1-5

)(1

-5)

Level N

eed

ed

Up

on

En

tryLevel N

eed

ed

Up

on

En

try (1

-4)

(1-4

)

Level N

eed

ed

(Kn

ow

led

ges O

nly

) (1-

Level N

eed

ed

(Kn

ow

led

ges O

nly

) (1-

4)

4)

Imp

orta

nce

Imp

orta

nce

(1-5

)(1

-5)

Min

imu

m/H

elp

ful Q

ualifi

catio

n

Min

imu

m/H

elp

ful Q

ualifi

catio

n

(MQ

/HQ

)(M

Q/H

Q)

Ap

plic

atio

n F

orm

Ap

plic

atio

n F

orm

Writte

n T

est

Writte

n T

est

Stru

ctu

red

S

tructu

red

In

terv

iew

Inte

rvie

w

Backg

rou

nd

Backg

rou

nd

Ch

eck

Ch

eck

Interpersonal and Interpersonal and team work skillsteam work skills

4.24.2 3.23.2 N/AN/A 44 MQMQ RR

Verbal Verbal communication communication skillsskills

3.93.9 3.53.5 N/AN/A 3.73.7 MQMQ RR

Knowledge of Fire Knowledge of Fire CodesCodes

2.62.6 3.53.5 3.43.4 3.93.9 MQMQ P/FP/F

Upper body Upper body strengthstrength

1.91.9 3.93.9 N/AN/A 3.43.4 MQMQ

Basic math skillsBasic math skills 1.41.4 3.53.5 N/AN/A 3.13.1 MQMQ P/FP/F

*The score from this selection procedure will be combined with scores from other selection procedures in the *The score from this selection procedure will be combined with scores from other selection procedures in the selection process.selection process.


Job Duties

Operationally defined KSAOs

Other KSAOs

Selection Devices

(e.g., applicationform, tests,interviews)

Content Validity Essential #3: Content Validity Essential #3: Connecting the Test to the JobConnecting the Test to the Job

Content Valid!


Content Validity Essential #4: Content Validity Essential #4: (Test Usage)(Test Usage)

Pass/Fail Cutoffs:Pass/Fail Cutoffs:– ““Normal Expectations of Acceptable Proficiency in the Normal Expectations of Acceptable Proficiency in the

Workplace” (Guidelines, 5H)Workplace” (Guidelines, 5H)– Modified Angoff (U.S. v. South Carolina, USSC)Modified Angoff (U.S. v. South Carolina, USSC)

Banding:Banding:– Substantially Equally Qualified ApplicantsSubstantially Equally Qualified Applicants– Statistically Driven (use Std. Error of Difference)Statistically Driven (use Std. Error of Difference)

Ranking:Ranking:– Is there adequate score dispersion?Is there adequate score dispersion?– Does the test have high reliability? (e.g., >.85)Does the test have high reliability? (e.g., >.85)– Is the KSA performance differentiating?Is the KSA performance differentiating?

Weighted/combined with other testsWeighted/combined with other tests– How are the weights related to the jobHow are the weights related to the job– Do they come from the job analysis or SME ratings?Do they come from the job analysis or SME ratings?

Using Test Scores• Ranking assumes one applicant is

reliably more qualified than the other

• Banding considers the unreliability of the test battery and “ties” applicants

• Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified”

Applicant Score

Tom 100Stacy 100Bob 100

Frank 100Julie 99

Rozanne 99Mark 98Luke 98Henry 97Paul 97Peter 96

Rebecca 96Alyssa 95

Matthew 94John 93

Annette 93Ray 92

Thomas 91Julissa 90


Pass/Fail CutoffsPass/Fail Cutoffs

Some major considerations:Some major considerations:– NOT ADVISED: NOT ADVISED:

Using an arbitrary cutoff (e.g., 70%)Using an arbitrary cutoff (e.g., 70%) Using applicant scores to benchmark (e.g., setting cutoff Using applicant scores to benchmark (e.g., setting cutoff

scores at mean-SD of applicant scores)scores at mean-SD of applicant scores)– ADVISED: ADVISED:

Consider “Normal expectations of acceptable proficiency in Consider “Normal expectations of acceptable proficiency in the workplace” (Guidelines, 5H)the workplace” (Guidelines, 5H)

– Usually requires SME-level data Usually requires SME-level data oror ratings ratings # Applicants that can be feasibly processed# Applicants that can be feasibly processed

– CONSIDER:CONSIDER: Is the test supported by content validity information or Is the test supported by content validity information or

criterion-related information? criterion-related information? How critical are the KSAs measured?How critical are the KSAs measured? Does the test measure “baseline” or “differentiating” KSAs?Does the test measure “baseline” or “differentiating” KSAs? How would current incumbents perform on this test?How would current incumbents perform on this test?


Using Test ScoresUsing Test Scores

Tests can be used on a pass/fail basis to ensure that Tests can be used on a pass/fail basis to ensure that

candidates possess the candidates possess the minimum competencyminimum competency levels levels needed for the job.needed for the job.

Tests can be used in a ranked fashion whereby criterion-Tests can be used in a ranked fashion whereby criterion-related validity evidence supports that higher test related validity evidence supports that higher test performance statistically predicts better job performance performance statistically predicts better job performance (or, content validity can also be used per UGESP 14C9).(or, content validity can also be used per UGESP 14C9).

Tests can be used by weighting/combining multiple test Tests can be used by weighting/combining multiple test scores whereby a “compensatory” or combined score is scores whereby a “compensatory” or combined score is used. used.

Banding is a recommended “balanced” solution because it Banding is a recommended “balanced” solution because it uses the test results in ways that are supportable by the uses the test results in ways that are supportable by the tests themselves.tests themselves.


Comparison Between “Test Use” Comparison Between “Test Use” OptionsOptions

FactorFactor RankingRanking BandingBanding Pass/Fail Pass/Fail CutoffsCutoffs

Validation Validation RequirementsRequirements

HighHigh ModerateModerate LowLow

Adverse ImpactAdverse Impact HighHigh ModerateModerate LowLow

DefensibilityDefensibility Low Low HighHigh HighHigh

Litigation "Red Flag"Litigation "Red Flag" HighHigh ModerateModerate LowLow

UtilityUtility HighHigh ModerateModerate LowLow

CostCost Low Low ModerateModerate HighHigh

Applicant FlowApplicant Flow Restrictive/Restrictive/ControllableControllable

Moderate/Moderate/ControllableControllable

HighHigh

Development TimeDevelopment Time Low Low ModerateModerate HighHigh

Reliability Reliability RequirementsRequirements

HighHigh ModerateModerate LowLow

# Item Requirements# Item Requirements HighHigh ModerateModerate LowLow


Job PerformanceJob Performance

Cognitive Cognitive AbilityAbility

Weighting/Combining Weighting/Combining ScoresScores

Experience/Experience/BackgroundBackground

InterpersonalInterpersonal

PersonalityPersonalityFactorsFactors


Sample Data—Cognitive and Sample Data—Cognitive and Personality Test Scores Personality Test Scores Total Total ScoresScores

GenderGender EthnicityEthnicityCognitive Cognitive

ScoreScorePersonalitPersonalit

y Scorey Score

MM WW 7272 175175

MM BB 7171 158158

FF WW 7070 175175

MM BB 7070 161161

MM WW 6969 156156

MM BB 6868 225225

MM WW 6363 156156


Sample Data—Cognitive and Sample Data—Cognitive and Personality Test Scores Personality Test Scores Combined Combined Test ScoresTest Scores

GenderGender EthnicityEthnicityCognitive Cognitive % Score% Score

PersonalitPersonality Scorey Score

Combined Combined % Score% Score

MM WW 7272 175175 83%83%

MM BB 6868 225225 83%83%

FF WW 7070 175175 81%81%

MM BB 7171 158158 81%81%

MM BB 7070 161161 81%81%

MM WW 6969 156156 79%79%

MM WW 6363 156156 73%73%


Steps for Weighting Tests Steps for Weighting Tests Using Content Validity Using Content Validity MethodsMethods

1.1. Select a panel of 4-12 Subject-Matter Experts (SMEs) who are truly Select a panel of 4-12 Subject-Matter Experts (SMEs) who are truly experts in the content and “functional areas” of the target position. experts in the content and “functional areas” of the target position. Ensure diversity on the panel.Ensure diversity on the panel.

2.2. Provide a copy of the Job Analysis for each SME. Be sure that the Job Provide a copy of the Job Analysis for each SME. Be sure that the Job Analysis itemizes the various job duties and KSAPCs that are important Analysis itemizes the various job duties and KSAPCs that are important or critical to the job.or critical to the job.

3.3. Provide each SME with a copy of each test (or a highly detailed Provide each SME with a copy of each test (or a highly detailed description of the content of the test if confidentiality issues prohibit description of the content of the test if confidentiality issues prohibit SMEs from viewing the actual test).SMEs from viewing the actual test).

4.4. Explain the confidential nature of the workshop, the overall goals and Explain the confidential nature of the workshop, the overall goals and outcomes, and ask the SMEs to sign confidentiality agreements.outcomes, and ask the SMEs to sign confidentiality agreements.

5.5. Discuss and review with SMEs the content of each test and the KSAPCs Discuss and review with SMEs the content of each test and the KSAPCs measured by each. Also discuss the extent to which certain tests may measured by each. Also discuss the extent to which certain tests may be better measures of certain KSAPCs than others. Factors such as the be better measures of certain KSAPCs than others. Factors such as the vulnerability of certain tests to fraud, reliability issues, and others vulnerability of certain tests to fraud, reliability issues, and others should be discussed.should be discussed.

6.6. Provide a survey to SMEs that asks them to distribute 100 points among Provide a survey to SMEs that asks them to distribute 100 points among the tests that will be combined. Be sure that they consider the the tests that will be combined. Be sure that they consider the importance levels of the KSAPCs measured by the tests, and the job importance levels of the KSAPCs measured by the tests, and the job duties to which they are linked, when completing this step.duties to which they are linked, when completing this step.

7.7. Detect and remove outlier SMEs from the data set.Detect and remove outlier SMEs from the data set.8.8. Calculate the average weight for each test. These averages are the Calculate the average weight for each test. These averages are the

weights to use when combining the test into a composite score.weights to use when combining the test into a composite score.


Standardizing ScoresStandardizing Scores

Before individual test scores can be weighted Before individual test scores can be weighted and combined, they should be and combined, they should be standard standard scoredscored. This process normalizes the scores and . This process normalizes the scores and is a necessary step to place different tests on a is a necessary step to place different tests on a level playing field.level playing field.

Standardizing test scores is simple:Standardizing test scores is simple:– Convert raw scores into Convert raw scores into Z scoresZ scores by subtracting by subtracting

each applicant’s score from the average (mean) each applicant’s score from the average (mean) score of all applicants and dividing this value by the score of all applicants and dividing this value by the standard deviation (of all applicant total scores).standard deviation (of all applicant total scores).

– Multiply the Z scores by the respective weights of Multiply the Z scores by the respective weights of the test components and a final score for each the test components and a final score for each applicant can then be calculated.applicant can then be calculated.


Building a Balanced Hiring Program by Building a Balanced Hiring Program by Sorting the Key Firefighter Competencies Sorting the Key Firefighter Competencies for Successfor Success

In 2008, FPSI obtained input from over 140 chief-level In 2008, FPSI obtained input from over 140 chief-level officers who were asked to weigh the importance of officers who were asked to weigh the importance of key critical skills and abilities required for successful key critical skills and abilities required for successful entry-level firefighter job performance entry-level firefighter job performance

Developed cutoff score by working with a panel of 12 Developed cutoff score by working with a panel of 12 ChiefsChiefs

ResultsResults The process identified the skills and abilities that are most The process identified the skills and abilities that are most

critical for entry-level firefighters. With input from over 140 critical for entry-level firefighters. With input from over 140 fire departments, these ratings are being used to build a fire departments, these ratings are being used to build a written test that properly weighs the most important written test that properly weighs the most important qualifications for the entry-level firefighter job. With this type qualifications for the entry-level firefighter job. With this type of input from fire executives across the country, the of input from fire executives across the country, the National Fire Select Test™ National Fire Select Test™ is the first test of its type!is the first test of its type!


U.S. Fire Department Input U.S. Fire Department Input & Opinions& Opinions

PersonalCharacteristic

s

55%Cognitive/Academic

Skills & Abilities

45%

National Fire Select Test™

Cognitive/Academic Component

StrategicDecision-Making

14%

ProblemSolving

14%

Map Reading

10%

Writing Ability15%

ReadingAbility18%

MathematicalReasoning

15%

MechanicalAbility14%

Personal Characteristics Component

Thoroughness/AttentionTo Detail

9%

PublicRelations

9%

Sensitivity8%

EmotionalStability

10% Proactive/Goal-Oriented

8%

Truthfulness/Integrity

13%

Allegiance/Loyalty

10%

WorkingUnder Stress

10%

Teamwork12%

Following Orders11%

Proposed Solutions for Testing Cognitive/Academic Competencies

for Entry-Level FirefightersCognitive/Academic

(45% OverallImportance)

Weight

Proposed Testing SolutionTypical

Validation

Method

Math 10% Use written or “work sample” format; measure using a limited number ofmultiple-choice items. Balance various types of math skills (add/subtract/multiple, etc.).

CV

Reading 14% Measure using either (1) “Test Preparation Manual” approach (where applicants are given amanual and asked to study it for a few weeks prior to taking the test based on the Manual); or (2) a short reading passage containing material at a similar difficulty/context to the jobthat applicants are allowed to study during the testing session and answer related test items.

CV

Verbal Communication 15% While a Structured Interview is the best tool for measuring this skill (because the skillincludes verbal and non-verbal aspects), some level of this skill can be measured usingword recognition lists or sentence clarity items.

CV

Writing 12% Measure using writing passages or word recognition lists, sentence clarity, and/orgrammar evaluation items.

CV

Map Reading 8% Measure using maps and related questions asking applicants how they would maneuverto certain locations. Include directional awareness.

CV

Problem-Solving 15% Measure using word problems measuring reasoning skills in job-rich contexts. CRV

Strategic Decision-Making

13% While a Structured Interview is the best tool for measuring this skill (because the applicantcan be asked to apply this skill in firefighter-specific scenarios), some level of this skill can bemeasured using word problems or other contexts supplied in written format where applicantscan consider cause/effect of certain actions.

CV

Mechanical Ability 12% Using CV, measure mechanical comprehension skills such as leverage, force, and mechanical/physics contexts regarding weights, shapes, and distances. Also can measure spatialreasoning (when using a CRV validation strategy).

CV/CRV

Notes: CV: Content Validity; CRV: Criterion-related validity.

Proposed Solutions for Personal Characteristics for Entry-Level Firefighters

Personal Characteristics(55% Overall)

Weight Proposed Testing Solution Typical Validation

Method

Teamwork 12% Under a CV strategy, a Situational Judgment Test (SJT)can be used for measuring these skills. Alternatively, acustom personality test can be developed using CRV.while these types of assessments can measure if anapplicant knows the most appropriate response (using anSJT) or the best attitude or disposition (personality test),they are limited in that they cannot measure whether anapplicant would actually respond in such a way. For thesereasons, measuring the underlying traits that tend togenerate these positive behaviors is typically the mosteffective strategy. Structured Interviews can also provideuseful insight into these types of competencies, as wellas background and reference evaluations. However, thesetools are time consuming and expensive, so measuringthese areas in the testing stage is an effective strategy.

CV/CRV

Working Under Stress 10%

Allegiance/Loyalty 10%

Truthfulness/Integrity 13%

Public Relations 9%

Emotional Stability 10%

Sensitivity 8%

Proactive/Goal-Oriented 8% These competencies can be effectively measured usingeither an SJT (using a CV or CRV strategy) or a Conscientiousness (CS) scale (using a CRV strategy). A CStest can be developed using just 20-30 items (using likert-type responses). Such tests are typically successful inpredicting job performance in fire settings.

CV/CRVThoroughness/Attention to Detail 9%

Following Orders 11%

Notes: CV: Content Validity; CRV: Criterion-related validity.


Selecting the Ideal Selecting the Ideal FirefighterFirefighter

Critical Skills Critical Skills &&

AbilitiesAbilitiesTest “A” Test “B”Reading Ability

Mathematical Ability

Spatial Abilities

Mechanical Aptitudes

Writing AbilityInterpersonal Skills

Decision MakingTest “C”

Map Reading

NFST™NFST™


Challenges to Building a Balanced Testing Challenges to Building a Balanced Testing Program and Recommendations for Program and Recommendations for SuccessSuccess

The most significant challenge to building an effective testing The most significant challenge to building an effective testing program for entry-level firefighters lies with testing the “soft program for entry-level firefighters lies with testing the “soft skills” (personal characteristics). This is because these skills—skills” (personal characteristics). This is because these skills—such as teamwork and interpersonal skills—are crucial such as teamwork and interpersonal skills—are crucial ingredients for success but they are the most difficult to ingredients for success but they are the most difficult to measure in a typical testing format. measure in a typical testing format.

For example, developing a math test is easy; developing a For example, developing a math test is easy; developing a test for measuring teamwork skills is not—but the latter was test for measuring teamwork skills is not—but the latter was rated as more important for overall job success!rated as more important for overall job success!

The reason for this is that many skills and abilities are The reason for this is that many skills and abilities are “concrete” (opposed to theoretical and abstract). An “concrete” (opposed to theoretical and abstract). An applicant’s math skills can readily be tapped using questions applicant’s math skills can readily be tapped using questions that measure numerical skills at the same level these skills that measure numerical skills at the same level these skills are required on the job. Abstract or soft skills like teamwork are required on the job. Abstract or soft skills like teamwork are more difficult to measure during a two-hour testing are more difficult to measure during a two-hour testing session. session.


Failure to Use a “Balanced Failure to Use a “Balanced Test”Test”

May have limited validity.May have limited validity.

Less minority representation in the final eligibility list.Less minority representation in the final eligibility list.

A testing process that leaves out (or underestimates) A testing process that leaves out (or underestimates) important cognitive/academic skills will likely result in a important cognitive/academic skills will likely result in a significant number of cadets washing out of the academy, and significant number of cadets washing out of the academy, and can also lead to poor performance on the job.can also lead to poor performance on the job.

A testing process that over-measures cognitive/academic skills A testing process that over-measures cognitive/academic skills and under-measures personal characteristics could lead to a and under-measures personal characteristics could lead to a group of book-smart firefighters who lack the skills necessary group of book-smart firefighters who lack the skills necessary to work cooperatively in the close living conditions required by to work cooperatively in the close living conditions required by firefighters.firefighters.

Greater potential for lawsuits which may result in lowering Greater potential for lawsuits which may result in lowering standards.standards.


Fire & Police Selection, Inc. Fire & Police Selection, Inc. Contact us at:Contact us at:

Mailing AddressMailing Address 193 Blue Ravine Rd. Suite 270193 Blue Ravine Rd. Suite 270

Folsom, CA. 95630Folsom, CA. 95630

WebsiteWebsite www.FPSI.comwww.FPSI.com www.NationalFireSelect.comwww.NationalFireSelect.com

TelephoneTelephone 916.294.4242 x. 245916.294.4242 x. 245

E-MailE-Mail [email protected]@fpsi.com

hiring the most qualified firefighter while avoiding lawsuits

Business