-
INCREASING EQUITY USING CULTURALLY RELEVANT ASSESSMENT
Karen Singer-Freeman, Director of Academic Planning & Assessment
Christine Robinson, Executive Director
Harriet Hobbs, Director of Assessment Systems & University Accreditation
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
Office of Assessment and Accreditation
http://assessment.uncc.edu/
2018 Designee
-
OVERVIEW
•Learning Outcomes for the Workshop•Approach to analyzing equitability of assessments•Culturally Relevant Assessment
•Theoretical model•Student feedback
•Data dive•Engaging faculty in conversation
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
WORKSHOP LEARNING OUTCOMES
By the end of the workshop, participants will be prepared to:
1) Identify questions and practices about inclusive assessment and student learning that can be addressed using existing sources of campus data.
2) Identify strengths and weaknesses of assignment types.
3) Understand the importance of disaggregating data by student characteristics.
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Equity in Assessment and Data Sources• What questions could we ask about assessments and
equity?• Do different assignments measure similar abilities?• Do assignment formats influence students’ abilities to
demonstrate learning?• Do student learning outcome reports capture important
differences in student success?
• Utilize software systems’ data • Learning Management Systems (LMS)• Assessment Management Systems (AMS)• Student Information Systems (SIS)
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
STUDENT LEARNING OUTCOME (SLO)
REPORTS
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Do student learning outcomes reports capture important differences?
•Student Learning Outcomes Reporting can compress data•Full assignment grade
•5 point ratings on multiple rubric dimensions•Average across rubric grades
•Above or below a criterion cut off
•Using Learning Management Systems it is possible to disaggregate data to see a more complete picture
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
SLO Report ExampleITCS 2300 Web-Based Application Development-Computer Science• SLO- Students demonstrate ability to create static web pages…
• Assignments - 4-6 webpages using a common theme: • 4 Criteria: Understanding target demographic, Effective use of HTML and JavaScript,
Browser compatibility, and Aesthetics.• Target: 80% will receive average rubric score of 3 (acceptable) or higher• Result: 97% received average rubric score of 3 (acceptable) or higher
• What other questions do we have?
2 3 11
47
139
0
50
100
150
1 2 3 4 5
Number of Students Receiving Each Rubric Rating
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Data Disaggregation: Is Achievement Equitable?
0% 0%4%
25%
72%
2% 3%7%
22%
67%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5
% Receiving Each Rating Disaggregated by Ethnicity
Non-URM (n = 146) URM (n = 60)
5% of URM Don’t meet but all Non-URM meet7.04 (1), p < .01
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Does the difference in achievement matter?
3.76
3.15
3.66
3.01
0
0.5
1
1.5
2
2.5
3
3.5
4
Semester GPA Cummulative GPA
Average Semester and Cumulative GPAURM grades marginally lower than Non-URM, p < .10
Non-URM (n = 146) URM (n = 60)PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
STUDENT SUCCESS: ASSIGNMENT TYPES
-
Activity1) Please pick up the paper on your seat and circle the items
that would generally describe this sort of assignment.
2) Compare your assignment and responses with the person next to you
3) Total the number of odd items and the number of even items you selected
4) Discuss – What do you think these questions are trying to measure?
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Culturally Relevant Assessment(Singer-Freeman, Hobbs, & Robinson, 2019)
• Aligned with teaching and SLOs
• Clear instructions• Limits effects of prior knowledge and privilege
• Scaffolding• Limits effects of prior knowledge and creates opportunities for early
success
• Inclusive content
• High utility value• Work has meaning beyond the academic context
• Avoids stereotype threat
• Results disaggregated
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Predictions Based on Analysis of Assignment Types
Low Risk
• Reflective writing
Moderate Risk
• Open-ended tests
• Inclusive projects
• Writing in discipline
High Risk
• Multiple-choice tests
• Formal Essays
• Homework
High Utility Value Low Utility Value
High Inclusive Content
Reflective Writing Inclusive Projects
LowInclusive Content
Writing in DisciplineFormal Essays,
Tests, HomeworkPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
What do students think about different types of assignments?
•16 items assess utility value, inclusive content, & alignment
•Completed for extra credit (n = 162)•Lower level classes in
• Biology (2)• Theater (1)• Writing (2)
•Upper level class in • Public Health
42% 40%35%
0%10%20%30%40%50%
Non-White 1stGeneration
Male
19%10%
46%
24%
0%
10%
20%
30%
40%
50%
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
The Measure
Utility Value (Chronbach’s Alpha = .90)
• Result in something I will discuss with others.
• Help me understand myself better.
• Improve my understanding.
• Provide me with experience that will be professionally useful.
• Have personal value.
----------------------------------
• Are only completed to earn a grade.
Inclusive Content (Chronbach’s Alpha = .93)
• Allow me to express my learning in my own words.
• Make me feel confident I can succeed.
• Include examples and materials that are familiar to me.
• Measure my true understanding.
• Allow me to relate class materials to my own experiences.
• Include clear instructions.
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Assignments RatedPlease indicate the extent to which _______ generally:Formal Papers or Essays
Reflective Writing in which you relate class materials to your own experiences.
Inclusive Projects – Projects that allow you to write in different styles or using familiar content.
Writing in Discipline – Writing assignments in the format used in a career you are interested in pursuing (e.g., lab reports, progress reports, theater reviews).
Oral Presentations
Short Answer Test Questions
Multiple-Choice Test Questions
oNot at all, A little, To a moderate extent, Very muchPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
Student Responses – Utility Value
2.78 2.722.59 2.53 2.44
2.282.12.15
1.912.12 2.17
1.99 1.911.65
1
1.5
2
2.5
3
3.5
4
Writing inDiscipline
OralPresentation
InclusiveProject
ReflectiveWriting
FormalPapers
ShortAnswer Test
Mult. ChoiceTest
Average Rating on 4 point Likert ScaleCritical Value for Utility Value Construct = .11, p < .05
Utility Completed for more than a gradePLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Student Responses – Inclusive Content
2.81 2.77 2.77 2.75 2.742.49 2.49
1
1.5
2
2.5
3
3.5
4
Writing inDiscipline
OralPresentation
InclusiveProject
ReflectiveWriting
ShortAnswer Test
FormalPapers
Mult. ChoiceTest
Average Rating on 4 point Likert ScaleCritical Value = .11, p < .05
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Student Confirmation of Model
High/ModerateUtility Value
Low /Moderate Utility Value
High Inclusive Content
Reflective WritingOral Presentations
Inclusive Projects
LowInclusive Content
Writing in DisciplineFormal Essays
Tests
Possible Correction Confirmation Addition
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
Some Research Support(Singer-Freeman, Hobbs, & Robinson, 2019)
Theater Appreciation (n = 64)• 69% URM
• Achievement gap • Multiple-choice exams
• No gap • Inclusive projects
Child Development (n = 110)• 44% URM
• Achievement gap • In-class multiple choice quizzes
• No gap • Reflective writing • Online multiple choice quizzes
9073
9483
0
50
100
Inclusive Project Multiple-Choice*
Theater AppreciationURM Non-URM
9077 85
93 85 87
0
50
100
ReflectiveWriting
In-ClassQuizzes*
OnlineQuizzes
Child Development
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Experimental Psychology
• 5 semesters
• 135 students • 30% URM
• Achievement Gap• URM students receive
lower grades on exams than on lab reports
• Non-URM students receive similar grades on exams and lab reports
7973
80 76
0102030405060708090
100
Writing in theDiscipline
Open-Ended Exams
Experimental Psychology
URM* Non-URMPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Research Confirmation of Model
High/ModerateUtility Value
Low /Moderate Utility Value
High Inclusive Content
Reflective WritingOral Presentations
Inclusive Projects
LowInclusive Content
Writing in DisciplineFormal Essays
Tests
Possible Correction Confirmation Addition
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Discussion
•Do our measures capture utility value and inclusive content?
•Do the predictions in the model and findings make sense given your experience with classroom assessments?
•What else would you want to know about how different groups of students respond to different types of assignments?
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
BREAK TIME
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
DISAGGREGATING RESULTS BY ASSESSMENT TYPE
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Our Process
•Obtained a list of 88 classes with enrollments of over 50 students and DFW rates of over 30%
•Reviewed 36 classes • Identified 9 that included different forms of assessment and
reported grades through LMS• The majority of classes we rejected only included testing
•Of the 9 classes with differentiated assignments • 100% included Exams and Quizzes
• 2 Formal essays and 1 Writing in the Discipline (Papers)• 2 Projects and 1 Oral presentation (Projects)• 2 Homework problem sets and 3 Informal writing (Homework)
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Classes•Pre-Calculus MATH 1103•Organic Chemistry Lab CHEM 2131L•Introduction to Communication Theory COMM 2100•Network Theory ENGR 2112•Principles of Accounting ACCT 2122 (flipped)•Physiological Psychology PSYC 3113•Design and Implementation ITCS 3112•Sociology of Health & Illness SOC 4120•Conservation Biology BIOL 4244
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
How to disaggregate data•Download grades, assignment descriptions, and syllabi for desired course from LMS• Consider merging data from multiple semesters if enrollments
are low
•Decide whether to include or exclude• Missing assignments• Completion-based grades
•Aggregate grades for equivalent assignments•Convert points to percentages •Merge file with demographic data from Institutional Research
•Compare outcomes using independent groups t-testPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Introduction to the data tables
Partner with someone at your table and pick a dataset
p < .05 p < .10 p < .05 p < .10
Achievement Gaps Reverse Achievement Gaps
Title provides broad introduction to contents
N = Total number of students in the class
(URM, Transfer, or Aid-Based Loans) = Number of students who were members of disaggregated group
Columns labeled with groups contain average percentage scored on each type of assignment
sig. = Whether observed differences were statistically significantPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Assignment Class N (URM)
Non-URM
URM sig N (Aid Loans)
No Loan
Aid Loan
sig N (Trans)
Native
Trans sig
Reading Response 20% Sociology of Health & Illness SOCY 4130
118 (63)
78 73 .05 129 (33)
75 79 ns 130 (81)
82 72 .001 Exams 25% (open book, online) 77 71 .01 72 76 .09 77 71 .01 Paper 35% 82 78 .05 79 83 .09 84 77 .001 Final Grade 77 70 .01 71 78 .001 76 70 .02 Assignment Class N
(URM) Non-URM
URM sig N (Aid Loans)
No Loan
Aid Loan
sig N (Trans)
Native
Trans sig
Quizzes (20%) Physiological Psychology PSYC 3113
51 (18)
87 83 ns 55 (17)
86 84 ns 54 (37)
90 84 .01 Exams (45%) 80 79 ns 79 78 ns 84 77 .04 Oral (15%) 97 94 ns 94 98 ns 97 96 ns Final Grade 82 81 ns 80 80 ns 82 80 ns
Assignment Class N (URM) Non-URM URM sig Quizzes 20%
Network Theory II ENGR 2112
42 (8)
85 84 ns Tests 40% 66 66 ns Final Exam 30% 72 65 ns Homework 10% 74 65 .09 Final Grade 67 65 ns Assignment Class N (URM) Non-URM URM sig Pre-Lab Quiz
Organic Chemistry Lab CHEM 2131L
141 (43) 78 70 ns Lab reports 80 75 ns Final grade 81 76 ns Assignment Class N (URM) Non-URM URM sig Exams (70%)
Introduction to Communication Theory COMM 2100
136(54)
73 72 ns Application Paper (10%) 88 84 .09 Minute Responses (20%) 97 92 .07 Final Grade 72 70 ns
Assignment Class N (URM) Non-URM URM sig Quizzes (7%)
Principles of Accounting ACCT 2122 (Fli d Cl )
50 (18)
89 89 ns Exams (64%) 75 72 ns Shark Tank Group Project (14%) 85 88 .06 Homework (7%) 81 87 ns
Introductory Classes Disaggregated by URM status
Advanced Classes in Sociology and Psychology
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
Assignment
Class
N (URM)
Non-URM
URM
sig
Quizzes 20%
Network Theory II
ENGR 2112
42 (8)
85
84
ns
Tests 40%
66
66
ns
Final Exam 30%
72
65
ns
Homework 10%
74
65
.09
Final Grade
67
65
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Pre-Lab Quiz
Organic Chemistry Lab
CHEM 2131L
141 (43)
78
70
ns
Lab reports
80
75
ns
Final grade
81
76
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Exams (70%)
Introduction to Communication Theory
COMM 2100
136(54)
73
72
ns
Application Paper (10%)
88
84
.09
Minute Responses (20%)
97
92
.07
Final Grade
72
70
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Quizzes (7%)
Principles of Accounting
ACCT 2122 (Flipped Class)
50 (18)
89
89
ns
Exams (64%)
75
72
ns
Shark Tank Group Project (14%)
85
88
.06
Homework (7%)
81
87
ns
Final Grade
77
75
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Tests (80%)
Pre-Calculus
MATH 1103
101 (33)
79
78
ns
MyMathLab (20%)
79
80
ns
Final Grade
76
75
ns
-
Dive into some dataAnswer the following questions:1) For which classes do the final grades tell the same story as
the assignment grades (look at significant and non-significant differences)?
2) For which classes do different assignments result in different patterns of performance than final grades (look at significant and non-significant differences)?
3) Can you identify any particular type of assignment that might be showing evidence of achievement gaps?
4) What else would you want to know?
5) Pick a class from the handout and outline some questions and suggestions you might have for the faculty member who teaches the class.
6) What other observations did you make?PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
URM Results
-6
-4
-2
0
2
4
6
8
10
12
Tests Papers Projects Homework Final Grade
Points Separating non-URM from URM students
MATH CHEM COMM ENGR ACCT PSYC ITCS SOC BIOLPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
Transfer Student Results
-6
-4
-2
0
2
4
6
8
10
12
Tests Papers Projects Homework Final Grade
Points Separating Native fromTransfer Students
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Aid-Based Loan Student Results
-6
-4
-2
0
2
4
6
8
10
12
Tests Papers Projects Homework Final Grade
Points Separating non-Loan from Aid-Based Loan students
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Summary
FrequentAchievement Gaps
• Tests• Low stakes or high stakes• Multiple choice or open ended• Open-book and Online
• Low-Stakes Homework, Writing, and Projects
• Formal Writing
Fewer Achievement Gaps
• Writing• Writing in the discipline• Reflective writing• Inclusive projects
• Oral Report• Group Project
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
CONCLUSIONS &
NEXT STEPS
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Next Steps• Disaggregation of responses to different forms of assessment
reveals importance of• Aligning assessments with teaching and SLOs• Attending to context
• Faculty Workshops• Differentiated Assignments/Assessments
• Utility Value and Inclusive Content• Implicit Bias
• Scale project to other courses and programs
• Determine if achievement gaps reflect assessments or competence
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
DISCUSSION
If achievement gaps exist on your campus, after hearing this talk, what new ways might you use to investigate these gaps?
• Who would you include in the conversation?
• What are approaches you could take to determine whether gaps reflect assessments or competence?
• What would next steps include to address gaps that result from each?
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
SPECIAL THANKS
UNC Charlotte Academic Affairs
Arna Erega
Students and Faculty
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
REFERENCESAssociation of American Colleges & Universities (2018). A vision for
equity: results from AAC&U’s project committing to equity and inclusive excellence: Campus-based strategies for student success. Retrieved from https://aacu.org/publications/vision-equity.
Eccles J. S. et al. Expectations, values and academic behaviors. In: Spence JT, editor. Perspective on achievement and achievement motivation. San Francisco: W. H. Freeman; 1983. pp. 75–146.
Harackiewicz et al. (2016). Achievement gaps with a utility-value intervention: Disentangling race and social class. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4853302/
Montenegro, E., & Jankowski, N. A. (2017). Equity and assessment: Moving towards culturally responsive assessment (Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Singer-Freeman, K., Hobbs, H., & Robinson, C. (2019). Theoretical matrix of culturally relevant assessment. Assessment Update.
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
C o n t a c tInformation
Christine Robinson, [email protected] Harriet Hobbs, [email protected]
Karen Singer-Freeman, [email protected] DO NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
Increasing Equity using culturally Relevant AssessmentOverview Workshop Learning outcomesSlide Number 4Student Learning Outcome (SLO) ReportsDo student learning outcomes reports capture important differences?SLO Report ExampleData Disaggregation: �Is Achievement Equitable?Does the difference in achievement matter?Student Success: ASSIGNMENT TypesActivityCulturally Relevant Assessment�(Singer-Freeman, Hobbs, & Robinson, 2019) Predictions Based on Analysis of �Assignment TypesWhat do students think about �different types of assignments?The MeasureAssignments RatedStudent Responses – Utility ValueStudent Responses – Inclusive ContentStudent Confirmation of ModelSome Research Support�(Singer-Freeman, Hobbs, & Robinson, 2019)Experimental PsychologyResearch Confirmation of ModelDiscussionBreak timeDisaggregating Results by assessment typeOur ProcessClassesHow to disaggregate dataIntroduction to the data tablesSlide Number 30Dive into some dataURM ResultsTransfer Student ResultsAid-Based Loan Student ResultsSummaryConclusions �& �Next StepsNext StepsDiscussionSpecial thanksReferencesContact Information