developing valid rubrics for assessing global awareness and global perspective dr. stephanie doscher...

Developing Valid Rubrics for Assessing Global Awareness

and Global Perspective

Dr. Stephanie DoscherFlorida International University

AAC&U General Education and Assessment ConferenceBoston, Massachusetts, March 1st, 2013

Session Outline Global Learning @ FIU

Rubric anatomy

Developing the rubrics

Scorer training

“Rubric for rubrics”

Gathering evidence of validity and reliability

Text

FIU’s Global Learning SLOs

Global Awareness: knowledge of the interrelatedness of local, global, international and intercultural issues, trends, and systems

Global Perspective: ability to develop a multi-perspective analysis of local, global, international, and intercultural problems

Global Engagement: willingness to engage in local, global, international, and intercultural problem solving

Active Learning Strategies/ Performance Assessments

Developing the Rubrics• Faculty learning community

• Pilot case studies, questions, scoring criteria

• Revisions based on response trends

• Pilot study

• Benchmark responses

• Anchor papers

• Faculty feedback, expert judge feedback

• Two field tests

• Scorer training

• Rubric language, response minimum

Scorer Training

• Full-time and adjunct faculty

• Pre-training packet

• Open discussion and review of cases, questions, and rubrics

• Norming session with anchor papers

• Sample scoring session, 10% of total papers

Validating the Rubrics

Validity: Can the rubrics detect the differences in students’ development of global awareness and global perspective?

Reliability: Can trained faculty raters agree on rubric scores 80% of the time or more?

Validating the Rubrics

Pretest Treatment

Posttest

Global learning O1 X1 O1

- - - - - - - - - -

- - - - - - - - - - - - - - - - - - - -

Non-global learning O2 X2 O2

Research Design

Data Collection

• Emails to chairs and potential faculty

• Pretest—within first two weeks of class

• Posttest—within last two weeks of class

• Trained faculty raters

• Two raters scored each question, third rater for discrepant scores

ResultsInter-rater reliability

•Global awareness rubric

• Pretest (.89, p < .0001)

• Posttest (.95, p < .0001)

•Global perspective rubric

• Pretest (.92, p < .0001)

• Posttest (.91, p < .0001)

Results

Validity – Global Awareness Rubric

•No significant main effects

•Post-hoc analysis: Groups differed significantly on pretest scores, p =.003

•Global learning (M = 1.51)

•Non-global learning (M = 1.85)

•Significant interaction between global awareness pretest score and the treatment in predicting global awareness posttest score, p = .005

Global Awareness Rubric

Results

Validity – Global Perspective Rubric

•No significant main effects

•Post-hoc analysis: Groups differed significantly on pretest scores, p =.003

•Global learning (M = .90)

•Non-global learning (M = 1.2)

•Significant interaction between global perspective pretest score and the treatment in predicting global perspective posttest score, p = .005

Results

Interpretation of Results

Reliability•Both rubrics highly reliable

•Consistent with literature on rubric development and rater training

•Empirical support for structural validity

Interpretation of Results

Validity•Rubric detects differences between global learning and non-global learning students

•Rubric detects differences within the group of global learning students

Thank You!

For more information, please contact me:

Stephanie Doscher, Office of Global Learning

sdoscher@fiu.edu

Visit our web site:

goglobal.fiu.edu