steps for developing a validated work sample physical ... · 5 listening to today ... using content...
TRANSCRIPT
11
Steps for Developing a Validated
Work Sample Physical Ability Test
(PAT) Using Content Validity
Copyright © 2013 Biddle Consulting Group, Inc.
2
Visit BCGi Online
www.BCGinstitute.org
� If you enjoy this webinar,
� Don’t forget to check out our other training opportunities through the
BCGi website.
� BCGi Standard Membership (free)
� Online community
� Monthly webinars on EEO compliance topics
� EEO Insight Journal (e-copy)
� BCGi Platinum (paid) Membership
� Fully interactive online community
� Includes validation/compensation analysis books
� EEO Tools including those needed to conduct AI analyses
� EEO Insight Journal (e-copy and hardcopy)
� Members only webinars and training and much more…
3
� BCGi is a Human Resource Certification Institute
(HRCI) Preferred Provider
� Continuing Education (CE) Credits are available
for attending this webinar for no charge
� Only those who remain with us for at least 80%
of the webinar will be eligible to receive the
HRCI training completion form for CE
submission
HRCI Credits Available for
Today’s Presentation
4
About Our Sponsor: Biddle Consulting
Group, Inc. (BCG)
Affirmative Action Plan
(AAP) Consulting and
Fulfillment
� Thousands of AAPs developed each year
� Audit and compliance assistance
� AutoAAP™ Enterprise software
HR Assessments
� AutoGOJA™ online job analysis system
� TVAP™ test validation & analysis program
� CritiCall™ pre-employment testing for 911 operators
� OPAC™ pre-employment testing for admin
professionals
� Video Situational Assessments (General and Nursing)
EEO Litigation Consulting
/Expert Witness Services
� Over 200+ cases in EEO/AA (both plaintiff and defense)
� Focus on disparate impact/validation cases
Compensation Analysis� Proactive and litigation/enforcement pay equity studies
� COMPare™ compensation analysis software
Publications/Books� EEO Insight™: Leading EEO Compliance Journal
� Adverse Impact (3rd ed.) / Compensation (1st ed.)
BCG Institute for
Workforce Development
� 4,000+ members
� Free webinars, EEO resources/tools
Nation-Wide Speaking
and Training� Regular speakers on the national speaking circuit
5
Listening to Today’s Presentation
� To eliminate potential long-distance telephone charges,
you can hear the audio to this presentation through your
computer starting at 10 a.m. Pacific Time. We
recommend using a headset.
� Make certain the volume on your computer is not muted and
turned up sufficiently to hear the presentation through your
computer
� If that does not work, you can call into the long-distance
number provided in your webinar invitation to hear the
sound portion of the presentation. Once connected via
telephone, you must enter the Telephone Access Code
listed in your invitation or Outlook schedule.
66
Today’s presentation by
Jim Kuthy, Ph.D.Principal Consultant
Heather Patchell, M.A. Consultant
Steps for Developing a Validated
Work Sample Physical Ability Test
(PAT) Using Content Validity
Copyright © 2013 Biddle Consulting Group, Inc.
7
A Little About Presenters
� Jim Kuthy holds Masters and Doctorate Degrees
in Industrial & Organizational Psychology
� Heather Patchell has a Masters Degree in
Psychology
� More than twenty-five combined years of
experience in the employment selection field
� They have designed selection and promotion
devices for dozens of employers, including
physical ability tests
� Validated many selection devices, including
conducting validation studies that have been
successfully defended in court or passed review
by federal agencies
The “Fine Print” about
Today’s Presentation
� The information provided in the presentation is
not intended as legal advice
� We note there may be differences in Equal
Employment Opportunity law in the
jurisdiction where your organization is located.
� Please seek additional information regarding
questions you may have regarding specific
types of testing for your organization
� Information contained in this presentation is, to
the best of our knowledge, current as of
November 2013
8
Developing/Validating Work Sample
Physical Ability Tests (PAT)
Using Content Validity
� Physical Ability Tests (PATs) are tests that measure an
applicant’s ability to perform the physical parts of the job
� They can be administered in pre- or post-offer settings as
long they are “non medical”
� Medically-related tests can only be administered post offer
� PATs are highly useful and necessary tools in a wide variety
of employment situations
� Some typs of PATs can result in adverse impact against
women, and must therefore be validated
� Using “Work Sample” style PATs to mimic the actual job are
generally preferable to more abstract types of PATs
� Using a content validation strategy can be preferable in
most situations where PATs will be used
Why is this Topic Important to
HR/EEO Professionals?� Why do I need to know about Physical Ability Testing?
� PATs are typically the #1 cause of adverse impact against women in hiring settings
� PATs that have not been validated, but exhibit adverse impact, will quickly lose if challenged
� Physically demanding jobs should use a PAT, or “false positives” will enter the workplace
� What are the key essentials I need to know about this topic?
� Using a content validation strategy is fairly straight-forward for demonstrating the job-relatedness of PATs
� The Uniform Guidelines on Employee Selection Procedures and the Americans with Disabilities Act provide helpful guidance for setting up defensible PATs
� What are the consequences surrounding these issues?
� Qualified workers will be unfairly screened out unless a PAT is valid; unqualified workers will be hired if no testing is used
� The “start up” cost to fight test validation challenges in court is typically between $30k and $80k
Presentation Overview
� Steps for Developing and Validating a Work-
Sample Physical Ability Test Using Content –
Related Validity
� Administering the Test
� Scoring Physical Ability Tests
� The Americans with Disabilities Act and
Physical Ability Testing
� Sample Test Event Description: Ladder
Removal/Carry
� Suggested “Best Practices”
� Summary
12
What is Validity as
Specified in the Courts?
� The process of gathering and evaluating necessary
evidence to support test score interpretations.
� A selection procedure is said to be valid if it can be
proven by an employer that it is “…job related and
consistent with business necessity”
12
13
Why do we validate?
� It makes good business sense
� Validation often results in selection devices that are
better at helping to choose the best people for the
job, thereby potentially improving productivity
� Can help to reduce the amount of time spent
training and/or supervising newly-hired employees
� Often helps to reduce turnover
� Sends the message that you care about doing the
“right thing”
� Helps to minimize potentially-negative legal
liability
13
14
Why do we validate?� It is required under some circumstances
� If there is adverse impact against a protected group of test takers (federal Uniform Guidelines on Employee Selection Procedures, 1978)
� See www.uniformguidelines.com
� Go to Section 14C and 15C for Content-Related Validity specifications
� For federal contractors: If selection testing is identified as a problem during an audit (Executive Order 11246, As Amended)
� Check with your AAP consultant to determine if validation is required if an audit should occur
A Brief Overview of Three
Types of Validity (in plain English!)
� Content Validity: A non-statistical connection between the important parts of the job and the test
� Typically based on a Job Analysis and on ratings from Job Experts (Subject Matter Experts - SMEs)
� Criterion Validity: A mathematical study that shows the test predicts or is related to job performance:
� A statistical study with results that are “statistically significant” ( less than 5% by chance)
� Construct Validity: A connection between a test, a trait, and job performance:
� “Empirical evidence” connecting the test to the trait and the trait to the job (requires both Content and Criterion Validity, plus more)
Steps for Developing a Physical
Ability Test using Content Validity
� Step 1. Conduct a solid job analysis that focuses
on physical aspect of the job
� Job analysis identifies the important and critical work
behaviors (duties), as well as the abilities required to
perform those duties
� Abilities should be operationally defined in terms of
observable aspects of work behavior of the job
� Uniform Guidelines Section 14C(4): “The closer the
content and the context of the selection procedure are
to work samples or work behaviors, the stronger is the
basis for showing content validity.”
16
Steps for Developing a Physical
Ability Test using Content Validity
� Step 2a. Identify parts of the job that are
typically require continuous physical exertion
for over 10 minutes
� This may include repeatedly performed tasks
� This may consist of unrelated tasks where a rapid
pace is required
� Step 2b. Identify other physically strenuous
parts of the job
� Identify the most strenuous in terms of weights,
heights, distances, etc.
17
� Step 3 – Option A. Work with supervisors,
trainers, and/or employees to assemble a
continuously-timed job simulation PAT
� The included events must be those where continuous
pace is required for physically-demanding job duties
� Step 3 – Option B. Work to assemble discrete
test events for measuring physically-
demanding parts of the job that are not
typically performed in rapid succession or
continuously, if appropriate
18
Steps for Developing a Physical
Ability Test using Content Validity
Continuous vs. Discrete Events
Continuous Events
� For tasks that are
performed
continuously and/or
sequentially
� Combine events into a
continuous test event
with a single, job-
related time limit, only
if supported by the job
analysis
Discrete Events
� Sort of like “snapshots”
of strenuous tasks
� Allow time between
events in a job-related
fashion
� Time limit on each
event must be job
related
� Focus on the most
physically demanding
aspects of the job
19
�Step 4. Run a representative sample of
Job Experts (20-30, if possible; 7-10
minimally acceptable)
�Attempt to oversample
underrepresented groups to help insure
fairness
20
Steps for Developing a Physical
Ability Test using Content Validity
Steps for Developing a Physical
Ability Test using Content
Validity
� Step 4. (continued)
� Use a validation survey to collect the following data:
� Record actual time it took to complete the PAT event
� Total time if using a continuously timed test; each event’s
time if using discrete test events
� Ask Job Experts for their opinion…
� “… given your time to complete the physical ability test,
your current fitness level, and your level of job experience,
what time should a minimally-qualified applicant use when
taking this test?”
� You only need to collect this data once per Job Expert
21
Step 4. Validation Survey (Yes/No Questions
asked of Job Experts)
�Does the PAT measure skills/abilities that are
important/critical (essential for the performance of the
job)?
�Does the PAT measure skills/abilities that are
necessary on the first day of the job (i.e., before
training)?
�Does the PAT replicate/simulate actual work behaviors
in a manner, setting, and level of complexity similar to
the job?
�Do the events in the PAT need to be completed on the
job in a rapid and safe manner (i.e., is speed
important)?22
Steps for Developing a Physical
Ability Test using Content Validity
Step 4. Validation Survey ( More Yes/No Questions
asked of Job Experts)
�Are the weights and distances involved in the PAT
representative of the job?
�Is the duration that the objects/equipment are carried or
handled in the PAT similar to what is required of a single
person on the job?
�Is the PAT free from any “special techniques” that
employees learn on the job that would allow current job
incumbents to perform the event more easily than
others?
�Does the PAT require the same or less exertion of the
applicant than is required on the job?
23
Steps for Developing a Physical
Ability Test using Content Validity
� Step 5. Analyze the survey data
�At least 70% of experts should
respond “Yes” to each and every
Yes/No survey question
� If less than 70%, go back and redesign
the test as necessary
� Then, re-survey until at least 70%
endorse all of the survey items
24
Steps for Developing a Physical
Ability Test using Content Validity
� Step 5. Analyze the survey data
�Compute the average of the Job
Experts’ time-limit suggestions
gathered in Step 4B
�Use the Standard Error of Measurement
(determined in Step 6) to adjust
average times suggested by the Job
Experts to create final time limit
� Add 1 or 2 SEMs to obtain final time limit(s)
for testing to give “benefit of a doubt” to the
test taker25
Steps for Developing a Physical
Ability Test using Content Validity
� 5. Analyze the survey data (continued)
� Compute the average of the Job Expert’s actual
time
� Add 1.645 Standard Errors of Difference, which
provides 90% confidence
� or 1.96 Standard Deviations, which provides 97.5%
confidence
� This level constitutes the lowest score boundary of
the “normal expectations of acceptable
proficiency in the workplace” that is required
when setting cutoff scores (Section 15H of the
Uniform Guidelines)
� Ideally, any required time limits should be greater
than the adjusted actual time26
Steps for Developing a Physical
Ability Test using Content Validity
� Step 6. Conduct a test-retest study of 60+
applicants or incumbents to determine the
Standard Error of Measurement of the PAT
�σxis the standard deviation of test scores
�rxxis the test-retest reliability
27
)1( rSEMx
−=σ
� 6. (continued). Use the SEM to determine the
Standard Error of Difference (which will be used
to adjust the time limit)
�SED = Standard Error of Difference
�SEM = Standard Error of Measurement
28
2*SEMSED =
Warning� Any pace, weights, or other limitations used during
testing must be job related (i.e., similar to actual pace,
distances, weights, or other limitations that are required
on the job)
� Do not become overly creative when developing substitute events
� EEOC v. Dial Corp. – a unanimous decision by the
U.S. Court of Appeals for the Eighth Circuit in Case
Nos. 05-4183 and 05-4311
� Test was more difficult than the job because the pace did
not include “natural” breaks between tasks that occurred in
the workplace
� The decision affirmed an award of approximately $3.3
million to 52 rejected female job applicants 29
� Train test administrators/raters
� Run practice sessions; determine proficiency at both
administration and scoring
� Prior to testing, inform candidates of the contents
of the event(s)
� Give them sufficient opportunity to practice and
prepare in advance of the test
� Standardize instructions (i.e., read and/or show
video)
� If appropriate, demonstrate proper and/or safe
technique
� Make certain candidates understand the
instructions30
Administering the Test
� If candidate appears confused or frustrated, ask,
“Do you need me to repeat the instructions?”
� Determine in advance about what to do if the ability to
understand and follow the instruction is part of the test
(as identified during the job analysis)
� Alternatively, you can consider the need for a re-
reading of the instructions when scoring, if job related
� There should be a penalty during testing only if there
would be some penalty for that type of behavior on the job
� Allow multiple attempts to complete the test
event(s) (if appropriate)
31
Administering the Test
�Your paramount concern should be safety
� Candidates should be provided safety rules and
description of safe work/testing practices in advance
of the testing process
� Read those to the test takers again prior to testing
� Immediately stop testing if candidate fails to follow
a safety rule or safe work practice
� Explain to the candidate what they did incorrectly, and if
necessary, demonstrate proper behavior
� Allow candidate to continue testing unless it is obvious injury
or harm to the candidate, others, and/or company property
will occur
� Document this!
32
Administering the Test
� If a candidate violates a safety rule or performs in a
way the test administrator believes demonstrates
that candidate does not possess the level of safety-
related knowledge that a minimally-qualified
applicant should possess, a complete and accurate
description of the violation or unsafe practice
should be recorded
� Have the information about the violation or unsafe
practice reviewed by a panel of Job Experts
� Candidates who the experts believe do not possess the level of
safety-related knowledge or ability that a minimally-qualified
candidate should possess prior to training on the job should be
disqualified
33
Administering
the Test
� If candidate appears to be injured during testing,
ask, “Are you injured?”
� If answer is “yes,” or if it is obvious the candidate
has suffered an injury, stop the event and obtain
assistance immediately
� Develop a plan in advance on how to handle
injuries
� Clearly mark the boundaries and start/end points
of the test (perhaps mark the ground)
� Base the limits on solid, job-related factors
34
Administering the Test
Diagram
(and mark)
the Test
Event in
advance to
Minimize
Potential
Confusion(See book for narrative)
35
Scoring PATs� Use only objective criteria for passing
� Did the test taker successfully complete the task
asked of them?
� Do not score on subjective factors, such as
“appeared to be struggling,” “was breathing hard,”
or even “had to stand on toes to reach”
� While these are observable, only use a candidate’s ability
to successfully complete the task within the pre-set time
limit as the scoring criteria
� Carefully records times and distances
� It can be helpful to have multiple judges and/or
timing devices
� Include timer in video, if appropriate
� Record/document information carefully36
Americans with
Disabilities Act
� Limit testing to abilities related to essential job functions that
employees are required to perform on the job with or
without reasonable accommodation
� 1. Function is the reason the positions exists
� 2. There are a limited number of employees available to perform that
job function
� 3. The function is highly specialized so the person is hired for their
expertise or ability to perform that particular function
� Medically-based tests can only be given after a bona fide
offer of employment has been given
� Employers must provide reasonable accommodation
during testing even if the employer knows they will not be
able to provide that person with a reasonable
accommodation on the job
37
ADA: Pre-Employment
Medical Testing� Medically-based physical employment testing is allowed under
the Americans with Disabilities Act with limitations
� Must be given post-job offer
� Must be given to all employees who enter that job category
� These can include measures of heart rate and VO2 max
� These types of criteria should be criterion validated and/or
medically defensible; check for alternative measures
� If an employee is screened out because of a disability, the
employer must be able to demonstrate that the rejection is
“job-related and consistent with business necessity” (i.e.,
valid)
� If person is screened out for safety reasons, the employer must
demonstrate that the individual poses a “direct threat” (i.e.,
significant risk of substantial harm to themselves or others that
cannot be reduced through reasonable accommodation)
� Make certain record-keeping requirements are addressed38
What about Fitness Testing?
� Fitness tests (fitness evaluations or fitness
assessments) includes a series of measurements
that help determine the health status and physical
fitness of an individual
� “Job candidates must run one-and-a-half miles in 15
minutes”
� “The candidate will complete a 3 minute step-test with a
12 inch step. They will step to the beat of a metronome
set to 96 beats-per-minute (24 steps per minute). Heart
rate will start to be measured within 5 seconds of the end
of the test, and measured for a total of 1 minute”
� “The candidate will do 15 pushups and 10 pull-ups”
39
What about Fitness Testing?
� Fitness testing (e.g., hear rate, measures of VO2 max)
requires a statistical linkage between test performance
and job performance (criterion-related validity) unless
the test task mimics work behaviors
� Fitness tests generally cannot be validated using the
content-related approach described in the current
presentation
� In other words, the employer must be able to demonstrate a statistically
significant relationship between test scores and job performance to use
fitness testing
� Tests that are designed to measure “strength in the
abstract” (being strong for the sake of being
strong without obvious connection to job activities)
must be validated using a criterion-related study
40
Proceed Cautiously
� Caution – Tests that use arbitrary cutoffs, such as “must
perform at the 40th percentile for the general population,”
are generally not defensible
� Caution – Tests that measure physiological characteristics,
such as heart rate, are considered medical tests under the
Americans with Disabilities Act, and can only be given post
offer
� Our experience has been that Step-Test scores frequently do not
correlate with injury rates and/or accident occurrences
� This may be because employees who are more fit may also be more
likely to take physical risks and not abide by company safety
protocols
� Caution – Some fitness testing requires age and gender to
be considered. The issue of age and/or gender-norming
fitness tests has not been definitively settled in the courts
and can result in liability41
Fancy Machines Do Not
Automatically Increase Validity
� Highly technical and/or complex testing devices do not
automatically result in more valid testing
� Using highly technical machines can be less valid as defined in
the Uniform Guidelines than more straight-forward testing
� Also, determining defensible cutoff scores can be problematic
� Machines that “mimic” work tasks can be content validated
� We suggest you have any claims of validity reviewed by an
expert
� According to the EEOC: “Many industrial and personnel psychologists
validate selection procedures, review published evidence of validity and
make recommendations with respect to the use of selection procedures.
Many of these individuals are members or fellows of Division 14
(Industrial and Organizational Psychology) or Division 5 (Evaluation and
Measurement) of the American Psychological Association.”
� BCG has several Ph.D.-level Division 14 members on staff to assist you
42
“Best” Practices
� Consider video-taping the testing process
� This can minimize potential liability by demonstrating
how the candidate actually performed during testing
� Have both administrator and test taker sign the scoring
sheet, if possible
� Limit verbal feedback to candidates and to onlookers
during testing
� If feedback is given, it should be similar across all
candidates
� We recommend that spectators not be allowed
� Do not become lazy or sloppy over time
� We typically recommend rotating administrators and/or
raters to prevent “testing fatigue or complacency”
43
“Best” Practices
� Candidates can be asked to sign a waiver of
liability (and indicate they are ready for testing)
� Liability waiver should be signed before testing begins
� Liability release forms should include a description of
the test, so candidates can make an intelligent waiver
� If the employer so decides, candidates may be
required to obtain a medical release prior to
physical-ability testing
� If you require a medical release of one candidate, you
must require the same release from all candidates
� Do not use different rules for different candidates
� Have test taker remain for a little while after testing to insure
he/she has sufficiently recovered from the testing process
44
45
� “The closer the content and the context of the
selection procedure are to work samples or work
behaviors, the stronger is the basis for showing
content validity”
� Uniform Guidelines Section 14C(4)
� Standardize administration and scoring of testing
� Document, document, document
� Abide by Americans with Disabilities Act
� Have fun! (But be professional)
Final Thoughts about Content-
Related Physical Ability Testing