testing the test: validation, litigation & risk management kathleen k. lundquist, ph.d. john c....
TRANSCRIPT
Testing the Test:Validation, Litigation & Risk
ManagementKathleen K. Lundquist, Ph.D.
John C. Scott, Ph.D.
2 2
About APTMetrics
Global Talent Management Solutions Provider Comprised of:
• Ph.D. industrial/organizational psychologists• Human resource consultants • Information technology specialists
What Sets APTMetrics Apart:• Professional integrity• Evidence-based approach• Technical expertise• Customer service
Diversity Supplier • Certified as a women-owned business
by WBENC• Certified as a women-owned small
business by the US SBA
Global Strategies for Talent Management.
3 3
Our Areas of Expertise
• Leader Assessment• Employee Selection• Litigation Support• Diversity Strategy &
Measurement• Job Analysis• Competency Modeling• Performance Management• Staffing for Mergers &
Acquisitions• Organizational Surveys
Global Strategies for Talent Management.
4 4
Our Web-Based Solutions Platform
APTMetrics®
SelectionMetrics®
Employee Selection System
LeadINsm
Leadership Assessment Suite
JobMetrics®
Job Analysis System
360Metrics®
360-Degree Feedback System
SurveyMetrics®
Organizational Survey System
5 5
APTMetrics’ U.S. Offices
6 6
• Expert Witness Testimony• Court appointed expert in settlement
• Ford • Abercrombie & Fitch• Morgan Stanley• Sodexo • The Coca-Cola Company
• Invited testimony on testing before EEOC• Negotiations with OFCCP• Proactive HR Process Audits• Development and validation of new selection
systems
Our Background with Testing and Litigation
7 7
• Are you concerned about a legal challenge to your tests or interviews?• Yes• No• I don’t know
Polling Question #1
8 8
The Federal Uniform Guidelines on Employee Selection Procedures (1978) define a test as:
Definition
“Any measure…used as a basis for any employment decision.”
9 9
What is a Test?
Work Simulation Exercises
Cognitive Ability Tests
Job Knowledge Tests
Personality and Interest Inventories
Honesty/Integrity Tests
Assessment Centers
Physical Abilities Tests
Application Form Data
Psychological Assessment
Education/Experience Requirements Reference Checks
Performance Evaluations
Interviews
10 10
• Is your company currently using written tests?• No, not at all• No, but considering using in the future• Yes, for a small number of jobs• Yes, for a wide range of jobs• I have no idea!
Polling Question #2
11 11
Validity…and the Law
12 12
Why are Tests Challenged?
• Adverse impact
• Validity• None
• Inadequate
• Dated
• Reliance on Studies Outside the Company
• Less Adverse Alternatives
• Inconsistent Administration
13 13
• Disproportionately fewer protected group applicants pass the test than majority group applicants
• Usually determined by:• 4/5ths or 80% rule• Standard deviation test
• The presence of adverse impact may be a given for many types of tests
Adverse Impact Definition
HOWEVER… tests with adverse impact can besuccessfully defended
14 14
4/5ths Rule Illustration
• 3 out of 5 African-Americans pass the test
• 60% pass rate
• 4 out of 5 Whites pass the test
• 80% pass rate
60% / 80% = 75%
Adverse Impact
15 15
Trends in Testing Case Law
• More specific evidence of adverse impact required• Beyond 80% rule• Practical and Statistical
Significance• Sample size issues
• Disparate treatment testing cases• Inconsistent
administration• Selective use of test• Jury trial and punitive
damages
• Greater emphasis on “validity”
of passing scores• Rationale• Set at acceptable
performance, not applicant flow
• Plaintiffs’ burden to identify less adverse alternatives• Must be knowable • Must demonstrate would be
less adverse• Must demonstrate
substantially the same validity
16 16
• Job analyses are conducted
• Procedures have face validity
• They document a search for less adverse alternatives
Employers Tend to be Successful When…
17 17
• Valid procedures are administered inconsistently
• Cut scores are set too high
Plaintiffs Tend to be Successful When…
18 18
249 Industrial and Organizational Psychologists answered this and related questions in a recent APT survey (July 2008)• Given a competently conducted criterion-related or content
validation study, how old would the validation study need to be to necessitate a new validation study?
• Average = 5 years old (Blue Collar, Supervisory, Managerial, Executive Tests)
• Technical Tests judged to have shorter shelf life = 3 to 3.5 years old
• Administrative/Customer Service/Sales = 4 to 4.5 years old
What is the “Shelf Life” of a Validation Study?
19 19
• Assuming a job’s tasks or work behaviors do not change, or change very little, what is the shelf life of a job analysis? That is, after how much time would one need to update the job analysis, even when there are little or no changes to the major tasks or work behaviors of the job?• Average = 5 to 6 years old
• What is the shelf life of the cut score? That is, how long would you recommend using a cut score before conducting additional research to determine if the cut score needs to change? • Average = 3 to 3.5 years old
Other Survey Results
20 20
• Conditions that shorten the shelf life of a validation study:
• Change in the nature of the job duties and or KSAs
• Conditions that result in the emergence of adverse impact or legal challenges
• Changes in applicant population• Changes in administration mode• Organizational changes (e.g., merger,
downsizing)
Other Survey Results
21 21
• These are professionally derived “rulesof thumb”
• Use these findings and your professional judgment
• Seek additional professional judgment• Seek legal input
And the Survey Says…
22 22
What must you do to validate
a test?
23 23
Test Validity Looks Like…
People who score high on the test are also high performers on the job
People who score low on the test are also low performers on the job
X
Y
Jo
b P
erf
orm
an
ce
Test Scores
24 24
• Selection procedures provide samples of behavior which allow us to make inferences about:• What basic abilities a person possesses• What the person knows• What the person can do• What a person is willing to do• How a person will behave in the future
The Validity of Selection Procedure Inferences
25 25
• Validity refers to the degree to which test scores are job related
• The process of validation involves accumulating evidence to provide a sound scientific basis for the proposed use of the test
Validation Defined
26 26
• Evidence Based Upon Test Content (Content Validity)• Demonstration that the content of a test is representative of
important aspects of performance on the job
• Evidence Based on Relations to Other Variables (Criterion Validity)• Statistical demonstration of a relationship between scores on a
test and job performance of a sample of employees
• Evidence Based on Internal Structure (Construct Validity)• Demonstration that test measures a construct (something
believed to be an underlying human trait or characteristic, such as conscientiousness) and the construct is important for successful job performance
Sources of Validity Evidence
27 27
Content Validity Study
1. Job Analysis
2. Test Development
3. SME Validation
4. Set Passing Scores
Profile job by identifying essential functions (WABs), knowledge, skills, abilities, and performance standards
Develop representative samples of performance domains - job sample tests, job skill tests, or job knowledge tests
Use subject matter expert judgment to document relationships between test content and performance domains
Use incumbents and subject matter experts to establish passing scores
28 28
• A comprehensive job analysis
• Competence in test construction
• Test content related to job’s content
• Test content representative of job’s content
• Examination of less adverse alternatives
• A passing score that selects those who can
better perform the job
Key Issues in Content Validity
29 29
Criterion-Related Validity Study
Establish Administrative Use
Job Analysis
Develop or Acquire Tests
Try Out/Pilot
Collect Test Data (Applicants or Employees)
Relate Test Scores & Performance Measures
Collect Performance Data
Try Out/Pilot
Develop Performance Measures
30 30
• Adequacy of job performance criteria• The psychometric quality of test and
criterion measure• Degree of correlation necessary to
establish validity• Examination of less adverse alternatives• Appropriateness of the passing score
Key Issues in Criterion-Related Validity
31 31
• No adverse impact• Transporting validity from another job
or location• Generalizing validity from other
studies of similar jobs
When Can You NOT Validate?
32 32
• Validation is the joint responsibility of test developer and test user
• When the use of a test differs from that supported by the test developer, the test user bears special responsibility for validation
Responsibility for Validation
33 33
• Job Analysis will identify what to assess
• It is not necessary to measure every important KSAO!
• It is necessary that every KSAO measured be important!
What Can You Test For?
34 34
Job Analysis is the Foundation
Job
Job
Job
Job
Analysis
Work Activities Performed
Scope and Effect of Work
Technical Skills Required
Competencies Required
Education Requirements
Experience Needed
Minimum & Preferred Qualifications
Test Specifications
Uniform Standardsfor Promotions
Structured Interviews
Performance Standards
35 35
• Does your company use formal job analyses as the basis for selection procedures?• No• Yes
Polling Question #3
36 36
Example of Test Specification Matrix
37 37
Types of Tests to Consider
ENTRY LEVEL
• Cognitive Ability Testing• Non-cognitive Measures• Interview
PROMOTIONAL
• Knowledge Testing• Performance Assessment• Interview
38 38
FaceValidity
39 39
• “High Fidelity” measurement tools (work samples, video, assessment centers) are more acceptable to candidates
• Once test specifications have been developed, decide:
• Custom design or identify commercially available test?
• What is appropriate testing medium?• Higher Fidelity = More Costly
• Whether custom or commercially available, must validate for your jobs
Selecting a Test
40 40
Combine Assessments
Cognitive
Job Performanc
e
Non-Cognitive
Structured Interview
41 41
• Cut-off scores should:• Be consistent with normal expectations of
proficiency within the workforce• Permit the selection of qualified applicants• Allow an organization to meet affirmative
action goals• Have a documented rationale
Criteria For Establishing Appropriate Cut-off Scores
42 42
What Makes Tests Fair and Defensible?
Validity
Based on job analysis
Standardized
Consistent
Validated
Implementation
Training
Ongoing monitoring
Appeals process
Communication
43 43
Ongoing Monitoring• Adverse Impact
• Test Content
• Administration Issues
44 44
1. “Aren’t tests harder to defend than other selection procedures?”
2. “The vendor says: ‘The test has no adverse impact.’ I have no problems, right?”
3. “The vendor says: ‘The test is validated – Trust us!’ I have no problems, right?”
4. “Candidates can take this new test at home over the Internet…so we can reduce overhead. I have no problems, right?”
5. “Once the test is validated, we’re done, right?”
Top Five Questions About Testing
45 45
For a copy of the results of APT’s recent Test Validity Shelf Life Survey email your request to:
Test Validity Survey
46 46
APTMetrics, Inc.One Thorndal CircleSecond FloorDarien, CT 06820
203.655.7779 [email protected]
Contact Information
Global Strategies for Talent Management.