using tests to predict job performance calswec - 2007
DESCRIPTION
Using tests to predict job performance CALSWEC - 2007. Cynthia Parry, PhD C.F. Parry Associates Michelle Graef, PhD Center on Children, Families & the Law – University of Nebraska, Lincoln Todd Franke, PhD Department of Social Work – University of California at Los Angeles - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/1.jpg)
Using tests to predict job performanceCALSWEC - 2007
Cynthia Parry, PhDC.F. Parry Associates
Michelle Graef, PhDCenter on Children, Families & the Law –
University of Nebraska, Lincoln
Todd Franke, PhDDepartment of Social Work –
University of California at Los Angeles
Henry Ilian, PhDNew York City Administration for Children’s Services
James Satterwhite Academy for Child Welfare Training
![Page 2: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/2.jpg)
Using tests to predict job performanceCALSWEC - 2007
• Goals:– Define high-stakes testing– Review technical complexities and professional
standards– Discuss the risks, rewards & consequences of
high-stakes testing
• Anticipated Outcome– A phone call to one or more of the presenters prior
any venture in high-stakes testing development
![Page 3: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/3.jpg)
Using tests to predict job performanceCALSWEC - 2007
Presentation in 4 parts:1. Overview of high stakes testing
2. Validity considerations
3. Validity issues in high stakes testing and training of child welfare workers
4. The ethics of using high-stakes testing and the ethics of not using high stakes testing
![Page 4: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/4.jpg)
Part 1 of 4:Overview of High Stakes Testing
Cindy Parry, PhD
![Page 5: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/5.jpg)
What do we mean by high stakes?
• Any use of a test that might affect– Your ability to get or keep a job– Your chances for promotion or raises– Your supervisor’s evaluation of your work– Testing out of training requirements or needing further
coaching or other interventions• Such as
– Licensing exams– Civil service hiring tests– Certification/credentialing tests– Any testing where individual results are reported to a
supervisor/manager or used to predict future competent performance
![Page 6: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/6.jpg)
What is the Law and Policy Context?
• 14th Amendment Requirements– Equal protection– Due process
• Civil Rights Acts– Disparate treatment of protected group– Disparate impact on a protected group– Business necessity
• ADA– Accommodations for test takers
Taken from Mehrens, W. Legal and Professional Bases for Licensure Testing. In Impara, James (Ed.). (1995) Licensure Testing: Purposes, Procedures and Practices. Buros Institute of Mental Measurements: University of Nebraska, Lincoln.
![Page 7: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/7.jpg)
Considerations in Developing a High Stakes Test
• Use professional standards to guide test development – The Standards for Educational and Psychological
Testing (AERA, APA, NCME) – Uniform Guidelines on Employee Selection
Procedures (EEOC)– Principles for the Validation and Use of Personnel
Selection Procedures (SIOP)
![Page 8: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/8.jpg)
Key Considerations in Validation
• Validity is the most fundamental consideration in developing any test
• We validate the use of a test score for a specific purpose — not a test or an item
• Content validity is the most common type of evidence cited in child welfare applications– Requires full specification of the domain or construct it is
intended to measure (1.6)– Standards require complete description of procedures used
to develop test content (1.6)• Use of subscales requires validation of the inferences
made from subscale scores as well as overall score (1.12)
Standards cited from: American Educational Research Association, American Psychological Association and National Council on Measurement in Education. (2002). Standards for Educational and Psychological Testing. American Educational Research Association: Washington, DC.
![Page 9: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/9.jpg)
Other Considerations for High Stakes Testing
• Adequate reliability for overall scores and any sub-scores that will be used in decision making
• Investigations of differential item functioning and test fairness
• Comparability of alternate forms or tests given over time
• Documentation of the method of setting the cut score and the qualifications of judges
• Notice of testing and due process procedures• Policy and procedures for accommodating persons
with disabilities • Policy and procedures for test security and
identification of test takers
![Page 10: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/10.jpg)
Part 2 of 4:Validity Considerations
Michelle Graef, PhD
![Page 11: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/11.jpg)
Can we use our training evaluation knowledge test to make decisions about candidates’ ability to
perform the job?Three points to consider:• Used alone, a written knowledge test is unlikely to be
adequate to make inferences about the full range of KSAOs needed to perform child welfare work.
• The target of your inferences indicates the appropriate content domain to be sampled. The typical training evaluation knowledge test (“post-test”) items have been sampled from the curriculum domain, which may not accurately represent the job domain.
• Even if you have strong evidence of content validity, evidence derived from other validation strategies (criterion-related validity) is recommended to support an employment decision. The implied inference is prediction of future job performance.
![Page 12: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/12.jpg)
Job Domain (KSAOs): Determined through a Job Analysis
Knowledge
Skills
Other Characteristics (values, beliefs, attitudes, personality)
Abilities
![Page 13: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/13.jpg)
Training Curriculum Content [K,S,O]
Test Item Content [K]
Sampling for development of a training evaluation knowledge test designed to assess trainees’ “mastery of training content”
Test Item Content [K]
![Page 14: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/14.jpg)
Training Curriculum Content [K,S,O]
Test Item Content [K]
Job Domain [K,S,A,O]
Example of a training evaluation knowledge test that is inappropriate for assessing individuals’ “ability to perform the job”
![Page 15: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/15.jpg)
Training Curriculum Content [K,S,O]
Test Item Content [K,S,O]
Job Content [K,S,A,O]
IDEAL: Appropriate sampling for training evaluation test designed to assess individuals’ “ability to perform the job”
![Page 16: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/16.jpg)
Predictor Measure (e.g., test)
Criterion Measure (e.g., supervisor rating, case review scores)
Criterion-related validity
Predictor Construct Domain (e.g., knowledge of child maltreatment, critical thinking skill, conscientiousness)
Criterion Construct Domain (e.g., job performance)
Content validityConstruct validity
Adapted from: Standards for Educational & Psychological Testing (1999)
Conceptual basis for test validation strategies
![Page 17: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/17.jpg)
Criterion-related validation study
• Demonstrate relationship between predictor and criterion using statistical significance testing
• Feasibility depends upon:– Availability of appropriate criterion measures– Representativeness of research sample– Adequacy of statistical power
• Variety of designs:– Predictive– Concurrent– Use of incumbents or job applicants
• Development of predictor and criterion measures that are relevant, uncontaminated, not deficient, free from bias, and demonstrate reliability
• Refer to Principles for the Validation and Use of Personnel Selection Procedures (2003)
![Page 18: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/18.jpg)
Recommendations
Strategies:
1) retrofit: map your existing knowledge test to the job domain and supplement it with other assessments (e.g., skills) to more adequately represent the job domain
2) start over: develop a test specifically for this purpose that directly samples the job domain, such as a work sample test, simulation, or assessment center
3) gather evidence of criterion-related validity in support of the use of your existing training evaluation knowledge test for employment decisions
![Page 19: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/19.jpg)
Part 3 of 4:Validity issues in high stakes testing and training of child
welfare workers
Todd Franke, PhD
![Page 20: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/20.jpg)
Validity issues in high stakes testing and training of child welfare workers
Are we measuring what is important?
![Page 21: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/21.jpg)
Los Angeles Context
Evidence based practice
Evidence based training
High stakes in Los Angeles
Minimally competent CPS workersWhat would one look like?
![Page 22: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/22.jpg)
Translation validity -- Focuses on whether the operationalization (i.e., measure) is a good translation of the construct
Face Validity -- On its face, does the operationalization look like a good translation of the construct?
Content Validity -- Operationalization is checked against the relevant content domain for the construct
Criterion-related Validity
Predictive Validity -- Operationalization’s ability to predict something it should theoretically be able to predict (L3)
Concurrent Validity
Convergent Validity
Discriminant Validity
Construct Validity
![Page 23: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/23.jpg)
So what constitutes the domains relevant in training child welfare workers?What should be trained/assessed?
It depends who you ask!
Knowledge of
Policies ( over 384 policies in LA-DCFS)
Procedures
submitting mileage
cellular phone reimbursement
use of logs (court calendars, visitation)
new case to-do’s
cubicle organization and systems
![Page 24: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/24.jpg)
Examples of current topics
Appreciating multiculturalism
Family preservation/Alternative response
Adoption assessments
Home assessments
Team Decision Making
Going to court/testifying
Court report writing
SDM tools
Basic interviewing
Adult substance abuse
Child neglect
Concurrent planning
Legal foundations
Risk assessment
Sexual abuse
Legal foundations
Legal permanence
Social work values
Worker safety
Physical abuse
![Page 25: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/25.jpg)
What else might be important?Suggestions from supervisors/trainees
Readiness to learn
Ability to multi-task
Willingness to accept supervision
Ability to make hard decisions
Ability to transfer knowledge to field
Social support
Problem solving
Are these training issues or hiring criteria?
Writing
Self-reliance
How to deal with emotional aspect of job
Critical thinking
Issues related to burn-out
Anything that links directly to case work practice
![Page 26: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/26.jpg)
What’s next?
How should an assessment be used?
Is knowledge only one of the domains that define the construct?
Does it have any predictive validity?Is it a valid measure of future job performance?
![Page 27: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/27.jpg)
Part 4 of 4:The Ethics of Using High-Stakes
Testing and the Ethics of Not Using High Stakes Testing
Henry Ilian, PhD
![Page 28: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/28.jpg)
High-stakes Tests Accomplish Three Things
• Screen out trainees who cannot demonstrate a specified level of mastery
• Compel Studying
• Enforce Fidelity to the Curriculum
![Page 29: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/29.jpg)
Compel Studying N Items Pretest Posttest
CW Common Core 74 80 39.70% 86.10%
(High Stakes)
Core Essentials for 64 32 44.30% 63.40%
Experienced CWs (Not High Stakes)
Enforce Fidelity to the CurriculumCore Essentials for Experienced CWs
Posttest 1 64 15 72.80%
Posttest 2 64 17 55.10%
A Comparison from NYC
![Page 30: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/30.jpg)
Using and Not Using High-Stakes Tests Each Has Consequences
• Using High-Stakes Tests– The requirements for a professionally developed
testing program require significant agency resources
– Some people with the potential to be good child protective workers may not pass
• Not Using High-Stakes Tests– Risk to families, children, self and colleagues– Difficulty maintaining employee standards and
agency morale
![Page 31: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/31.jpg)
The Major Ethical Issue is the Potential for Harm
• There is a need to balance potential harm – to caseworker trainees – to children and families served by CPS
agencies – to co-workers and the agencies
themselves
![Page 32: Using tests to predict job performance CALSWEC - 2007](https://reader036.vdocuments.net/reader036/viewer/2022062518/568148c2550346895db5dff5/html5/thumbnails/32.jpg)
Concerns for the Evaluator
• The organization may be reluctant/unable to make available the time and resources to develop and validate instruments
• Testing decisions are often made in a political or administrative context
• Administratively imposed testing may not meet professional standards