online standard setting workshop€¦ · process and validation for setting standards (reckase,...
TRANSCRIPT
![Page 1: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/1.jpg)
Online Standard Setting Workshop
Charalambos Kollias
![Page 2: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/2.jpg)
Brief introduction to Standard Setting
![Page 3: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/3.jpg)
What is standard setting? “In the field of achievement testing, standard setting is regarded as a process of decision making concerning the classification of the test/exam results in several successive, but limited number of levels of achievement.”
![Page 4: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/4.jpg)
(Kaftandjieva, 2010 p. 12) A task • to determine cutscore on test corresponding to description of performance High level of importance • falling above or below cut score on test has importance consequences on test taker (Reckase, 2009)
![Page 5: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/5.jpg)
Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard
![Page 6: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/6.jpg)
Elaborated Description Test Design Translation of Policy and Elaborated Content to Value on Test Score Scale
![Page 7: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/7.jpg)
Agency that Calls for the Standard & Policy Definition of Standard (Reckase, 2009) Standards • established/ set up Agencies call for standards
![Page 8: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/8.jpg)
• Standards – established/ set up Policy definitions of standards – whether standard is difficult to reach (academic recognition) or whether standard is for the minimum requirements of all individuals – first setting of a standard
![Page 9: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/9.jpg)
Elaborated Description Test Design (Reckase, 2009) Elaboration of policy definitions • Description of what someone exceeding standard can do Standard operationalized as score on a test.
![Page 10: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/10.jpg)
Test Design and Content • Test design to be consistent with policy definition – Test to represent content specified in elaborated description.
![Page 11: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/11.jpg)
Translation of Policy and Elaborated Content to Value on Test Score Scale (Reckase, 2009) Standard Setting • Policy descriptions & elaborated descriptions
![Page 12: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/12.jpg)
translated into numerical language of test score Translation requirements: • people who are ... – fluent in language of policy definitions – elaborated descriptions – language of test score
![Page 13: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/13.jpg)
Translation of Policy and Elaborated Content to Value on Test Score Scale (Reckase, 2009) Problem: • Difficult to find people with all capabilities
![Page 14: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/14.jpg)
Solution: • Processes to help with translation process
– standard setting methods Standard setting methods • methods for facilitating translation from verbal descriptions to numerical descriptions Participants • Judges/panelists = translators
![Page 15: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/15.jpg)
Standard setting methods Test centered • Subject matter experts (SMEs) review & discuss characteristics of students within or at “border” of achievement level, then carefully consider knowledge and skill demands of each item on test. Then make judgment regarding
![Page 16: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/16.jpg)
probability of examinee at borderline of achievement level getting the answer correct. Examineecentered • Testing groups of examinees already classified in achievement levels and using their performance to establish cut scores.
![Page 17: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/17.jpg)
Facilitating the derivation of valid standards 1. Selecting qualified and sufficient number of panelists 2. Providing adequate training of panelists 3. Facilitating panelists’ discussion of ratings 4. Providing qualitative feedback on provisional judgments 5. Facilitating discussions
![Page 18: Online Standard Setting Workshop€¦ · Process and Validation for Setting Standards (Reckase, 2009) Agency that Calls for the Standard Policy Definition of Standard](https://reader034.vdocuments.net/reader034/viewer/2022051813/60345d0a0d56667eee08dfa9/html5/thumbnails/18.jpg)
on final cut score for consensus 6. Conducting a comprehensive survey of panelists’ impressions 7. Deriving cutscore intervals (Sireci, Randall, & Zenisky, 2012)