minimizing adverse impact: learning from some possible serious
TRANSCRIPT
![Page 1: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/1.jpg)
Minimizing Adverse Impact: Learning From Some Possible Serious Flaws in the Wonderlic
Personnel Test
Joel P. Wiesen, Ph.D, Director Applied Personnel [email protected]
29th Annual IPMAAC ConferenceOrlando, Florida; June, 2005
![Page 2: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/2.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Why Discuss the Wonderlic
• Perhaps the most widely used commercial test in our field
• Enjoying success in court
![Page 3: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/3.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Features
• Short, timed test: 12 minutes
• Aims to test general cognitive ability
• Yields one score
• Many validity studies
![Page 4: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/4.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
One Major Problem• Adverse Impact (AI)
• B-W mean score difference almost 1 s.d.
• Not unique to the Wonderlic
![Page 5: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/5.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Possible Causes of AI
• Many possible causes of AI
• Dozens of possible reasons for AI
• Nevertheless, let’s not simply assume the test is unbiased.
• We will look closely at the Wonderlic
![Page 6: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/6.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Indications of Possible Test Bias
• Copyright before AI was discussed: 1942
• Cursory review of questions raises red flag
![Page 7: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/7.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Review of Test Materials
• Test Items
• Test Manual
![Page 8: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/8.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Instructions
First Sample Question
REAP the opposite of 1 obtain, 2 cheer, 3 continue, 4 exist, 5 sow
![Page 9: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/9.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Instructions
Third Sample Question
MINER MINOR -- Do these words1 have similar meanings,2 have contradictory meanings,3 mean neither the same or opposite?
![Page 10: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/10.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Personnel Test
Question omitted from this file for the sake of security.
Confidential. For Conference Only. Do Not Reproduce or Distribute.
![Page 11: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/11.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Personnel Test
Question omitted from this file for the sake of security.
Confidential. For Conference Only. Do Not Reproduce or Distribute.
![Page 12: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/12.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Personnel Test
Question omitted from this file for the sake of security.
Confidential. For Conference Only. Do Not Reproduce or Distribute.
![Page 13: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/13.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Personnel Test
Question omitted from this file for the sake of security.
Confidential. For Conference Only. Do Not Reproduce or Distribute.
![Page 14: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/14.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Reasoning or Vocabulary
If you do not know the words used, an item measures vocabulary rather than reasoning.
![Page 15: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/15.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Personnel Test
Question omitted from this file for the sake of security.
Confidential. For Conference Only. Do Not Reproduce or Distribute.
![Page 16: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/16.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Personnel Test
Question omitted from this file for the sake of security.
![Page 17: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/17.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Possibly Biased Questions
• About 9 such potentially biased questions
• B-W mean score difference is about 7
![Page 18: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/18.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
New Types of Test Items
• “The field of testing suffers from ideational stagnation”
Robert Sternberg1998 American Psychologist
![Page 19: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/19.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
New Types of Test Items
• Examples of new or improved items
• Examples of new item types
![Page 20: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/20.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
New or Improved Items
• Verbal Reasoning• Math• Clerical Speed• Face Recognition• Short-Term
Memory
• Oral Comprehension
• Mechanical Ability• Intelligence• Spatial Ability• Physical Ability
![Page 21: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/21.jpg)
New Math ItemA store is having a 20% off sale, and you
see a jacket selling for $100. How much did the jacket cost before the sale?
Old (Wonderlic) Math ItemQuestion omitted from this file for the
sake of security.
![Page 22: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/22.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
New Clerical Speed Item
• Names not all Anglo-Saxon
• Different fonts (more common today)
![Page 23: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/23.jpg)
New Clerical Speed ItemHow many of the six pairs of names listedbelow are exactly the same?
Shiva, Vandanna Shiva, VandanaRivierra, Luis Riviera, LuisFajardo, Makeda Fajardo, MakiedaRasheed, Fouad Rashed, FouadAbdul, Hakeem Abdul, HakeimHernandez, Luis Hernandes, Luis
![Page 24: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/24.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Problem Solving or Concept Formation
• Seldom used item type, but similar to items in the Wonderlic
• Simple instruction:- Select the one that does not belong.
![Page 25: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/25.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Problem Solving Item Type
• Select the one that does not belong.truckcarmotorcyclebicycle
![Page 26: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/26.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Problem Solving Item Type
• Select the one that does not belong.wallfloordoorchair
![Page 27: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/27.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Hard to Identify Bias by Content
• Content reviews are generally ineffective in detecting items with large B-W differences
• Many suggest it is futile to try
• The issue of test bias is scientifically dead (Hunter & Schmidt, 2000)
![Page 28: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/28.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
SAT Has Ethnic Bias
• Most B-W DIF with analogies and antonyms (which provide least context)
• Least B-W DIF with reading comprehension items (with maximal verbal context) (Freedle, 2003)
![Page 29: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/29.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Success in Reducing Adverse Impact
• The WTMA, my test of mechanical aptitude, reduces adverse impact on women as compared with the Bennett
• WTMA uses every day objects
• The test is available from PAR and Ramsay Corporation
![Page 30: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/30.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Second Major Problem
• Face validity appears low
• Face validity promotes user acceptance
![Page 31: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/31.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Conclusion to Item Review
• Be creative in writing items
• Consider new item types
• Pursue face validity to maximize user acceptance
![Page 32: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/32.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Wonderlic Test Manual
• Not clear what the test was designed to measure
• No test outline
• No mention of adverse impact
![Page 33: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/33.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Why Does the Wonderlic Work?
• Proxy for schooling
• Proxy for knowledge of certain American subculture
![Page 34: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/34.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Not Just the Wonderlic
• Other published tests have similar content
• Perhaps because published tests tend to be quite old.
![Page 35: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/35.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Some Closing Comments
• Don’t be content with status quo• Be vigilant in cultural bias review• Conduct research on fairness in job
performance evaluations. • Consider developing or using tests of non-
traditional abilities
![Page 36: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/36.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
Call for Research Partners
• A new General Employment Test is being developed
• Contact me to discuss being a beta or validation site
![Page 37: Minimizing Adverse Impact: Learning From Some Possible Serious](https://reader030.vdocuments.net/reader030/viewer/2022021508/586f9abd1a28ab66058c0ff2/html5/thumbnails/37.jpg)
By Dr. Joel P. Wiesen; Applied Personnel Research; (617) 244-
8859; IPMAAC Conference June
References• Freedle, R.O. (2003) Correcting the SAT’s ethnic and social-class
bias: A method for reestimating SAT scores. Harvard Educational Review, 73, 1-43.
• Hunter, J.E. & Schmidt, F.L. (2000) Racial and Gender Bias in Ability and Achievement Tests. Psychology, Public Policy, and Law, 6, 151-158.
• Sternberg, R.L. (1998) American Psychologist, 53, 576-77.
• Wiesen, J.P. (2005) Possible Reasons for the Black-White Mean Score Differences Seen With Many Cognitive Ability Tests http://personnelselection.com/adverse.impact.htm