overcoming the fear of employment testing: what exactly · overcoming the fear of employment...
TRANSCRIPT
Overcoming the Fear of Employment Testing: What Exactly is “Testing”?
Presented by: Chris Atkinson, M. S. and Jim
Kuthy Ph.D.
Visit BCGi Online
• If you enjoy this webinar,
– Don’t forget to check out our other training opportunities through the BCGi website.
• BCGi Standard Membership (free)
– Online community
– Monthly webinars on EEO compliance topics
– EEO Insight Journal (e-copy)
• BCGi Platinum (paid) Membership ($299/year)
– Fully interactive online community
– Includes validation/compensation analysis books
– EEO Tools including those needed to conduct AI analyses
– EEO Insight Journal (e-copy and hardcopy)
– Members only webinars and training and much more…
www.BCGinstitute.org
HRCI Credit
• BCGi is an HRCI Preferred Provider
• CE Credits are available for attending this
webinar
• Only those who remain with us for at least 80%
of the webinar will be eligible to receive the
HRCI training completion form for CE
submission
44
About Our Sponsor: BCG
• Assisted hundreds of clients with cases involving Equal Employment
Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense)
• Compensation Analyses / Test Development and Validation
• Published: Adverse Impact and Test Validation, 3rd Ed., as a practical
guide for HR professionals
• Editor & Publisher: EEO Insight an industry e-Journal
• Creator and publisher of a variety of productivity
Software/Web Tools:
– OPAC®
(Administrative Skills Testing)
– CritiCall®
(9-1-1 Dispatcher Testing)
– AutoAAP™ (Affirmative Action Software and Services)
– C4™ (Contact Center Employee Testing)
– Encounter™ (Video Situational Judgment Test)
– AutoGOJA®
(Automated Guidelines Oriented Job Analysis®
)
– COMPare: Compensation Analysis in Excel
Contact Information
Jim Kuthy, Ph.D.Principal Consultant
(800) 999-0438 x 239
Chris Atkinson, M.S.Consultant
(800) 999-0438 x 120
The Presenters…
• Jim holds Masters and Doctorate Degrees in Industrial/ Organizational Psychology
• Jim has taught Psychology and Business-related courses at the University of Akron and California State University, Sacramento
• Chris holds a Masters Degrees in Industrial/Organizational Psychology
• More than twenty five combined years of experience in the employment selection field
• They have designed and validated a variety of employment tests for many employers
Outline
What is testing?
A Brief Overview of Adverse Impact
A Brief Overview of Test Validation
Commonly Used Employment Tests
Using a “Whole Person” Approach to Testing
Common Testing Myths
What is Employment Testing?
• Assessment Standpoint – A test is a
standardized series of problems or questions
that assess a persons knowledge, skills, abilities,
or other characteristics
• Legal Standpoint - Uniform Guidelines on
Employee Selection Procedures defines a test as any
method used to make an employment decision
U.S Employment Test Usage
• 41% of employers test job applicants in basic literacy and/or math skills.
• 34% of job applicants tested in 2000 lacked sufficient skills for the positions
they sought.
• 68% of employers engage in various forms of job skill testing.
• 29% of employers use one or more forms of psychological measurement or
assessment.
• 20% of employers use cognitive ability tests
• 8% of employers use interest inventories.
• 14% of employers use some form of managerial assessments
• 13% of employers use personality tests 10% of employers use physical
simulations of job tasks
***This information is based on survey data collected in 2001 from 1,627 HR managers who
are members of the American Management Association. The survey accurately reflects the
AMA membership base, and is therefore more reflective of larger organizations than all
employers in the U.S. economy.
Reasons for Testing
1. Testing leads to savings in the decision-making
process
2. The costs of making a wrong decision are high
3. The job requires attributes that are hard to
develop or change
4. Hard-to-get information can be obtained more
easily and efficiently
5. Individuals are treated consistently
6. There are a lot of applicants
A Brief Overview of Adverse Impact
Men
Pass (50)
Men
Fail (50)
Men Passing Rate
(50%)
Women
Pass (35)
Women
Fail (65)
Women Passing
Rate (35%)
Adverse Impact
• 80% Test
• Statistical Significance
A substantially different rate of selection in hiring, promotion, or other
employment decision which works to the disadvantage of members of
a race, sex, or ethnic group
Adverse Impact (i.e., significant difference in rates)
Sufficient Evidence of
Job Relatedness
/Validity
Disparate Impact
Discrimination
According to the 1964/1991 Civil
Rights Act(s)
A Brief Overview of Test Validation
• Traditionally, validation is making sure a selection
practice, procedure, or test (PPT) appropriately measures
what it is designed to measure
• In a legal realm, a selection procedure is valid if it can be
proven by an employer that it is “…job related and
consistent with business necessity”
• It is the inferences regarding the specific uses of a test or
other measurement procedure that are validated, not the
test itself
What is Validity?
What Needs to be Validated?
• Any practice, procedure, or test (PPT) exhibiting
adverse impact
– Written tests
– Interviews
– Physical ability tests
– Resume screens
– Virtually any process that is used for making
employment-related decisions
Validity Overview
• Content: A connection between the important parts of the
job and the test
– Requires demonstrating a connection between the job (using a job
analysis) and the content of the test
• Criterion: A mathematical study that proves the test
predicts job performance:
– It ALWAYS requires: A statistical study and results that are
“statistically significant” (<5% chance)
• Construct: A connection between a test, a trait, and job
performance:
– It ALWAYS requires “empirical evidence” connecting the test to
the trait and the trait to the job (all 3)
Job Duties Operationally
defined KSAs
Other
KSAs
Selection Devices
(e.g., applicationform, tests,
Interviews, BQs)
Content Validation Process
Job Duties
Other
KSAs
Selection Devices
(e.g., applicationform, tests,
Interviews, BQs)
Content Valid!
Operationally defined KSAs
Content Validation Process
20
Criterion-Related Study
0
10
20
30
40
50
60
70
0 20 40 60 80 100
Test Score
Pe
rfo
rma
nc
e M
ea
su
re
Score on some “Criteria” (e.g.,
job performance, days missed
work, etc.)
Score on a
“Test”
Criterion-related Validity
Commonly Used Employment Tests
Commonly Used Employment Tests
Assessment Centers
Personality Testing
Biodata
Cognitive Ability Tests
Job Knowledge Tests
Situational Judgement
Testing
Employment Interviews
Training Education and
Experience (TEE)
Work Sample Tests
Cognitive Ability Tests
• Cognitive ability tests assess abilities involved in
thinking (e.g., reasoning, perception, memory,
verbal and mathematical ability, and problem
solving)
• Examples: “General mental ability”, math,
verbal analogies, reading comprehension, spatial
reasoning
Cognitive Ability Tests
• Validity: High validity for a variety of jobs. The more
complex the job, the higher the validity
• Applicant Reactions: Tests developed for specific jobs
have higher face validity than off-the-shelf tests
• Adverse Impact: High against minorities
• Development Costs: Purchasing off-the shelf tests is
typically less expensive than developing custom tests
• Administration: Easy to administer to many applicants
• Use: Best used for jobs requiring specific cognitive
skills and more complex jobs
Job Knowledge Tests
• Assesses technical or professional expertise in
specific knowledge areas.
• Not appropriate when applicants will be trained
after selection in the critical knowledge areas
needed for the job.
• Examples: Basic accounting principles,
computer programming, and financial
management
Job Knowledge Tests
• Validity: High - Customized job knowledge tests have
been shown to have slightly higher validity than off-the-
shelf tests
• Applicant Reactions: Tend to be favorable
• Adverse Impact: High against minorities
• Development Costs: Frequent updates to the test
content and validation may be needed to keep up with
changes in the job
• Use: Best used for jobs requiring specific job
knowledge on the first day of the job
Situational Judgement Tests
• Situational judgment tests (SJTs) present a description
of a work problem or critical situation related to the job
and asks applicants to identify how they would
respond
• SJTs measure effectiveness in social functioning
dimensions such as conflict management, interpersonal
skills, or problem solving,
• SJT questions and alternatives are typically based on
critical incidents generated by job experts. Scores are
based on subject matter experts' judgments of the best
and worst alternatives.
Encounter Situational Judgment Testing
Situational Judgement Tests
• Validity: High content validity, moderately high
criterion-related validity
• Applicant Reactions: Very positive
• Adverse Impact: Moderate against minorities
• Use: Typically used for managerial positions or
other jobs requiring interpersonal interactions
Employment Interviews
• Interviews are subject to the requirements of the
federal Uniform Guidelines on Employee Selection
Procedures (1978), similar to any other type of test
or selection procedure
• Unstructured Interviews
– No constraints on types of questions asked
– Global evaluation of applicant responses
• Structured Interviews
– Asking exact same questions to all applicants
– Applicant responses evaluated against rating scales
Interview Validity
Least Validity Highest Validity
Unstructured
Single Rater
Generic “one size fits all”
Open Scoring/No Scoring
Structured
Multiple Raters
Job Specific
BARS
Structured Interviews
• Applicant Reactions – Unstructured interviews
are more favorable than structured interviews
• Adverse Impact – Low
• Typically used late in the assessment process as a
final screen
Training, Education & Experience
(TEE) Requirements
• A systematic method used to assess previous
experience, education, and training information
provided by job applicants
• Often collected as a supplement to the application
• This information is evaluated against education and
experience benchmarks to generate scores for
selection purposes blank
Training, Education & Experience (TEE)
Requirements
• Training requirements posted by employers as hiring or
promotion criteria.
– Certifications, credentials, training institution completions,
licensing requirements
• Educational requirements, including:
– Degrees, specific course requirements, etc.
• Experience requirements used as hiring or promotion
criteria, including:
– Closed systems (e.g., time-in-grade requirements)
– Open systems (inside/outside experience counts)
Training, Education & Experience
(TEE) Requirements
• Validity: Highly representative of the job, but
do not relate well to performance on the job.
• Applicant Reactions: Typically more positive
reactions from more experienced applicants
• Adverse Impact: Little to none against women
or minorities
• Uses: Commonly used as a screening device
prior to another selection tool (e.g., structured
interview).
Work Sample Tests
• Work sample testing refers to “hands-on”
performance tests in which a job candidate is
required to perform (or describe) activities/tasks
that are performed on the job
Callinan, M., & Robertson, I. T. (2000). Work sample testing. International Journal of Selection and Assessment, 8(4), 248-260.
Work Sample Test Examples
• Mechanic– Repair a problem on a car
– Use a repair manual
• Clerical– Typing Test
– Proofreading test
• Police Officer– Writing test
– Climbing, running
• Computer Programmer– Create a simple program
– Debug a current program
Adapted from “Selection Assessment Methods,” Pulakos 2005
Work Sample Tests
• Validity: One of the most valid forms of testing
• Adverse Impact: Typically low – depends on the
competencies being assessed
• Applicant Reactions: Favorable – Provides a realistic job
preview
• Development and Administration Costs – Can be high
• Best Used When:
– Measured competencies are critical for successful performance
– Limited number of applicants
– Small number of applicants being tested
Assessment Centers
• Employs multiple assessment methods and
exercises to evaluate a wide range of
competencies
• Applicant performance is usually observed and
evaluated by multiple assessors (i.e., raters)
• Examples: In-basket exercises, leaderless group
discussion
Assessment Centers
• Validity: Moderate to high
• Applicant Reactions: Favorable - Many exercises
serve as a realistic job preview
• Adverse Impact: Low to moderate
• Development Costs: High in terms of time and
money
• Administration Costs: High - Requires many raters
• Use: Particularly effective for assessing higher-level
managerial and leadership competencies as well as sales
and customer service
Personality Testing
• Personality tests elicit information about a person’s motivations,
preferences, interests, emotional make-up and style of interacting with
people and situations
• Self-report inventories typically ask applicants to rate their agreement
level on a various items designed to assess various traits
• Examples: 16PF, OPQ, Hogan Personality Inventory, Big 5
• Specific traits are more predictive for particular jobs
• Personality tests designed to assess psychiatric conditions are
considered medical examinations
Personality Testing
• Validity: Moderate levels of validity for a wide range of
outcome measures in a variety of settings
• Applicant Reactions: Not very favorable, may appear
as invasive
• Adverse Impact: Few, if any differences between men
and women, or ethnicities
• Administration: Easy to administer to large groups as
a screen
• Use: Often used when job performance requires a great
deal of interpersonal interaction or work in team
settings
Biodata
• “Biodata measures (also known as biographical
data, autobiographical information, and life
histories) are instruments assessing an
individual's previous life experiences, usually
with a focus on targeting those previous
experiences that predict future behaviors of
interest.”
Cooper, A. L., & Stokes, S. G. (2003). Biodata. In Hersen, M., Thomas, C. J (Eds.), Comprehensive Handbook of Psychological Assessment, Industrial and Organizational Assessment (Volume 4). Hoboken, NJ: Wiley
The Many Faces of Biodata
• Biodata can tap into many different dimensions
– Cognitive Ability: “What was your academic rank in your
high school graduating class?”
– Personality: “How often did you attend social events during
high school”?
– Vocational Interests: “How often do you read science
related publications”?
– Values: “During high school, how important was it to you to
be regarded as successful?
Cooper, A. L., & Stokes, S. G. (2003). Biodata. In Hersen, M., Thomas, C. J (Eds.), Comprehensive Handbook of Psychological Assessment, Industrial and Organizational Assessment (Volume 4). Hoboken, NJ: Wiley
Biodata
• Validity: Moderate – predictive of other criteria
such as turnover attendance, and safety behavior
• Applicant Reactions: Typically not as
favorable as other assessment methods
• Administration: Can be administered to many
applicants at once
• Adverse Impact: Low against minority groups
Using a “Whole Person” Approach to Testing
As you add more
devices or tests that
measure different
aspects of a person,
you will learn more
about that person
The U.S. Department
of Labor refers to this
as the
“Whole Person”
approach to selection
Observable
Behavior
Performed
During Work
Sample Test
Test of
Attitude,
Personality,
Biodata, Values,
etc.
Written
Knowledge
Test
Interview
Imagine you had the time and resources to conduct
multiple types of selection testing
Test of
Attitude,
Personality,
Biodata Values,
etc.
Written
Knowledge
TestInterview
Research shows work
sample tests are
typically among the
most directly related
to observable job
performance
Observable
Behavior
Performed
During Work
Sample Test
These generally include samples of work performance (e.g., data
entry, video-based tests)
Written
Knowledge
Test
Interview
Adding a test of
attitude,
personality,
values, biodata,
etc. can provide
additional helpful
information
about the job
applicant, but is
typically
somewhat less
directly related to
observable job
performance
Test of
Attitude,
Personality,
Biodata, Values,
etc.
Observable
Behavior
Performed
During Work
Sample Test
Test of
Attitude,
Personality,
Values, Biodata,
etc.
Written
Knowledge
Test
InterviewObservable
Behavior
Performed
During Work
Sample Test
Multiple
Selection
Devices can
measure the
same factors
about an
applicant in
the same way,
the same
factors in
different ways,
and/or
different
factors in the
same way
It is typically
best to try to
measure as
many different
factors about a
person as
possible
during testing
while
balancing
available
resources
Interview Interview
Work
Sample Test
Personality
Test
Knowledge,
Skill, Ability
Test
Personality
Test
Multiple-HurdleCombined Score without Hurdles
Knowledge,
Skill, Ability
Test
Work
Sample Test
Use Test Scores Appropriately
• Combining personality test scores with scores from other
selection devices may result in lower adverse impact
• You can combine scores from various types of tests, but you
should
– Only use scores from those who scored at or above some minimum level (i.e.,
they must “pass” that section of the selection process)
– Weight scores from each selection device appropriately
– Statistically standardize the scores from different tests into a single metric, such as
T-scores or Z-scores, before combining them together
• We suggest that you consider using personality test scores from
attitude/personality tests as a “pass or fail” hurdle
– Giving a personality test earlier in the process can frequently reduce potential
adverse impact during a testing process
• Ranking assumes one applicant is reliably more qualified than the other
• Banding considers the unreliability of the test battery and “ties” applicants
• Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified”
• Weighting/combining test scores can be done using “compensatory” or using cutoff on each test then weighting results
Applicant Score
Tom 100
Stacy 100
Bob 100
Frank 100
Julie 99
Rozanne 99
Mark 98
Luke 98
Henry 97
Paul 97
Peter 96
Rebecca 96
Alyssa 95
Matthew 94
John 93
Annette 93
Ray 92
Thomas 91
Julissa 90
Test Scores Use
Comparison Between “Test Use” Options
Factor Ranking Banding Pass/Fail
CutoffsValidation Requirements High Moderate Low
Adverse Impact High Moderate Low
Defensibility Low High High
Litigation "Red Flag" High Moderate Low
Utility High Moderate Low
Cost Low Moderate High
Applicant Flow Restrictive/
Controllable
Moderate/
Controllable
High
Development Time Low Moderate High
Reliability Requirements High Moderate Low
# Item Requirements High Moderate Low
Common Employment Testing Myths
57
A good test or interview will guarantee the people
you hire will be successful
• Selection devices help you to improve the likelihoodyou are hiring the best person for the job
– They cannot guarantee success (no matter what the sales person says)
• This is why we should try to capture as much as possible about the job candidate using multiple devices, attempting to measure the “whole person”
Myth #1
Myth #2
The federal Uniform Guidelines on Employee Selection Procedures(Section 7A ) say that it is your responsibility, NOT the developers of the selection devices you use, to ensure that they are fair (www.uniformguidelines.com)
– Be proactive…
o Make sure the selection device you use is valid -- Obtain copies of the validity report
o Have the validity report reviewed by professionals– Many Universities have Industrial & Organizational Psychology programs
that may be able to help you
– Ask the right people for assistance
o Stay current in your research– What was good yesterday, may not be good today
The test publisher is responsible for making
sure the test is fair and valid
59
It is better not to test at all because then
you cannot be sued
1. Your chances of obtaining qualified employees without testing is just that… a chance (and an unlikely one at that!)
2. Well constructed and validated tests are used successfully by the best organizations with very few or no problems
3. You are actually more likely to have problems if you do not use appropriately constructed & validated selection devices because
– That allows for discrimination to sneak in
– You have no evidence to back your employment decisionso Without evidence the courts will infer discrimination
Myth #3
60
You should only use tests that have absolutely no
adverse impact
• It is virtually impossible to eliminate adverse impact when attempting to measure some types of skills and abilities
• That being said, minimizing adverse impact is one factor to consider
• However, there are other factors to consider, such as making certain the job candidate can perform at least at a minimally qualified level
• Using only tests with no adverse impact means that you will not be measuring many of the important knowledge, skill, or abilities that are necessary to perform the job
Myth #4
61
It’s Too Expensive to Test• Testing can be expensive, but hiring the wrong person can
be even more expensive– Compute the costs of a “bad” hire
• You can actually reduce employee costs by choosing the most qualified person for the job– Less supervision
– Less training
– Reduced turnover
• You can sometimes reduce testing costs through better planning– Set and enforce valid basic qualifications to minimize the number of applicants to
be tested
– Use Realistic Job Previews to have those who are not truly interested “select” themselves out of the testing process
Myth #5
62
Testing is Too Time Consuming
• How much time are you willing to spend to
find the best people for the job?
• Balance the amount of time it takes to hire
the best person for the job against how long
you would like that person to STAY on the
job
Myth #6
Conclusion
• Employment testing can be a valuable method
to use in your hiring process
• Test validation is required if there is adverse
impact with the use of a test
– Test validation makes good business sense regardless
• Consider many factors in test selection
• Use a “Whole Person” approach to testing
Questions?
800-999-0438