![Page 1: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/1.jpg)
Developing Valid Rubrics for Assessing Global Awareness
and Global Perspective
Dr. Stephanie DoscherFlorida International University
AAC&U General Education and Assessment ConferenceBoston, Massachusetts, March 1st, 2013
![Page 2: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/2.jpg)
Session Outline Global Learning @ FIU
Rubric anatomy
Developing the rubrics
Scorer training
“Rubric for rubrics”
Gathering evidence of validity and reliability
![Page 3: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/3.jpg)
Text
![Page 4: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/4.jpg)
FIU’s Global Learning SLOs
Global Awareness: knowledge of the interrelatedness of local, global, international and intercultural issues, trends, and systems
Global Perspective: ability to develop a multi-perspective analysis of local, global, international, and intercultural problems
Global Engagement: willingness to engage in local, global, international, and intercultural problem solving
![Page 5: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/5.jpg)
Active Learning Strategies/ Performance Assessments
![Page 6: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/6.jpg)
![Page 7: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/7.jpg)
![Page 8: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/8.jpg)
Developing the Rubrics• Faculty learning community
• Pilot case studies, questions, scoring criteria
• Revisions based on response trends
• Pilot study
• Benchmark responses
• Anchor papers
• Faculty feedback, expert judge feedback
• Two field tests
• Scorer training
• Rubric language, response minimum
![Page 9: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/9.jpg)
Scorer Training
• Full-time and adjunct faculty
• Pre-training packet
• Open discussion and review of cases, questions, and rubrics
• Norming session with anchor papers
• Sample scoring session, 10% of total papers
![Page 10: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/10.jpg)
![Page 11: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/11.jpg)
![Page 12: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/12.jpg)
![Page 13: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/13.jpg)
![Page 14: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/14.jpg)
Validating the Rubrics
Validity: Can the rubrics detect the differences in students’ development of global awareness and global perspective?
Reliability: Can trained faculty raters agree on rubric scores 80% of the time or more?
![Page 15: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/15.jpg)
Validating the Rubrics
Pretest Treatment
Posttest
Global learning O1 X1 O1
- - - - - - - - - -
- - - - - - - - - - - - - - - - - - - -
Non-global learning O2 X2 O2
Research Design
![Page 16: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/16.jpg)
Data Collection
• Emails to chairs and potential faculty
• Pretest—within first two weeks of class
• Posttest—within last two weeks of class
• Trained faculty raters
• Two raters scored each question, third rater for discrepant scores
![Page 17: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/17.jpg)
ResultsInter-rater reliability
•Global awareness rubric
• Pretest (.89, p < .0001)
• Posttest (.95, p < .0001)
•Global perspective rubric
• Pretest (.92, p < .0001)
• Posttest (.91, p < .0001)
![Page 18: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/18.jpg)
Results
Validity – Global Awareness Rubric
•No significant main effects
•Post-hoc analysis: Groups differed significantly on pretest scores, p =.003
•Global learning (M = 1.51)
•Non-global learning (M = 1.85)
•Significant interaction between global awareness pretest score and the treatment in predicting global awareness posttest score, p = .005
![Page 19: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/19.jpg)
Global Awareness Rubric
![Page 20: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/20.jpg)
Results
Validity – Global Perspective Rubric
•No significant main effects
•Post-hoc analysis: Groups differed significantly on pretest scores, p =.003
•Global learning (M = .90)
•Non-global learning (M = 1.2)
•Significant interaction between global perspective pretest score and the treatment in predicting global perspective posttest score, p = .005
![Page 21: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/21.jpg)
Results
![Page 22: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/22.jpg)
Interpretation of Results
Reliability•Both rubrics highly reliable
•Consistent with literature on rubric development and rater training
•Empirical support for structural validity
![Page 23: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/23.jpg)
Interpretation of Results
Validity•Rubric detects differences between global learning and non-global learning students
•Rubric detects differences within the group of global learning students
![Page 24: Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education](https://reader033.vdocuments.net/reader033/viewer/2022051412/5514c27555034693478b4854/html5/thumbnails/24.jpg)
Thank You!
For more information, please contact me:
Stephanie Doscher, Office of Global Learning
Visit our web site:
goglobal.fiu.edu