jazzin’ up assessment: writing items that are relevant and sensitive scott strother, mike sieve,...
TRANSCRIPT
Jazzin’ Up Assessment:
Writing Items that are Relevant and Sensitive
Scott Strother, Mike Sieve, John Kellermeier
11/19/15
Quantway and Statway:Assessment Team (A-Team)
Summative AssessmentsItem BanksCurriculum Item Improvement
Assessment Expert Convenin
g
Assessment Team & PD
1. Diversity of Item Types
2. High Quality Assessments
3. High Quality Items
1. Improved Network Capacity2. Leadership within Colleges3. Support and
Collaborationwithin and between colleges
Assessment Expert Convenin
g
Assessment Team & PD
1. Diversity of Item Types
2. High Quality Assessments
3. High Quality Items
1. Improved Network Capacity2. Leadership within Colleges3. Support and
Collaborationwithin and between colleges
High Quality Assessments Items
▪Context
▪Bias and Sensitivity
▪Relevance
▪Goal of Item (e.g. productive struggle)
▪Clear and concise
▪Depth of Knowledge
▪Cognitive Load
▪Difficulty (ensuring the appropriate spread)
▪Discrimination (of various ability levels)
▪Design Specifications
High Quality Assessments Items
▪Context
▪Bias and Sensitivity
▪Relevance▪Goal of Item (e.g. productive struggle)
▪Clear and concise
▪Depth of Knowledge
▪Cognitive Load
▪Difficulty (ensuring the appropriate spread)
▪Discrimination (of various ability levels)
▪Design Specifications
Writing High Quality Items
6 week course focused on:
Diverse Item Types Characteristics of High Quality Assessment Items
Understanding Depth and Rigor of Items Purpose of Assessment and Alignment
Reliability and Validity Assessment Construction
7
Carnegie’s Network Improvement Community Assessment Literacy Training
The Challenge
8
How can we write items that are relevant for todays
students?
The Solution
9
Carnegie’s Quantway and Statway lessons revolve around 3
central topics:
Citizenship – information about your society, government and world that can impact many
decisions you make
Medical Literacy – information about health issues and medical treatments
Personal Finance – financial information and how to use it to make decisions in your life
The Solution
10
Replace problems like this….
The Solution
11
Replace problems like this….
With problems like this….
12
Learning Outcome: Compare two quantities using a percent
Traditional Assessment Item:
The number 36 is what percent of 56?
Making the item more relevant (Medical Literacy):
According to the CDC, there are an estimated 56 million deaths each year worldwide. Of these, 36 million deaths are caused by non-
communicable diseases, such as cardiovascular disease, cancer, diabetes and chronic lung diseases. What percent of deaths worldwide
are caused by non-communicable diseases?
http://www.globalissues.org/issue/587/health-issues
Making Items Relevant
Learning Outcome: Estimate ratios of large numbers
Traditional Assessment Item:
The number 124,358,000 is larger than the number 10,342.Approximately how many times larger is it?
Making the item more relevant (Citizenship):
Throughout the past several years, the growth of the internet has exploded. According to Netcraft, an internet services company that
provides web hosting market-share analysis, there were 23,603,183 active web sites in June of 2004. This number had increased to
181,414,052 active web sites by June of 2014. Estimate how many times larger the number of active websites was in 2014 compared to 2004.
Round your answer to the nearest whole number.
13
Making Items Relevant
14
Making Items Relevant
Learning Outcome: Identify and interpret the vertical/horizontal intercepts and the slope in context
Traditional Assessment Item: Use the graph pictured below, to answer the following questions:
What is the y-intercept of the graph?
What is the x-intercept of the graph?
What is the slope of the graph?
15
Making Items Relevant
Making the item more relevant (Personal Finance):
What is the vertical intercept? Interpret its meaning in the context of this problem. Be sure to use quantitative information along with the writing principle that was presented in class.
16
Making Items Relevant
What is the vertical intercept? Interpret its meaning in the context of this problem. Be sure to use quantitative information along with the writing principle that was presented in class.
What is the slope of the line? Be sure to include units in your answer. Write a sentence that describes the slope in the context of this problem.
Making the item more relevant (Personal Finance):
The Hidden Curriculum
▪All assessment items have an overt curriculum and a hidden curriculum
▪The overt curriculum is the goal of the item
▪The hidden curriculum is the context of the item
17
The Hidden Curriculum
▪Every context has a cultural location
▪Not every student (or instructor) understands that culture
▪Every context has implications for power, privilege and inequity.
18
The Basketball Problem
▪The National Basketball Association (NBA) announced that a new synthetic basketball would be used for the 2006–2007 season.
▪Players in the NBA complained about the new ball. The NBA announced that the traditional leather ball would be used again beginning January 1, 2007.
▪The players’ complaints suggested that they thought that the new ball impacted their game performance. If this is true, the new ball would have a measurable effect on the games. In this task, you will develop a plan to answer the question, “Did the synthetic ball affect game performance?”
▪What variables could you look at to answer the question? Be specific about what you would measure. Think about aspects of the basketball game that you could use as variables.
19
The Zodiac Problem
▪Can someone’s birth date influence his or her personality traits? If you wanted to gather information about birth dates and personality, what kind of information would you look for?
▪Your have a list of personality traits grouped by birthdays. Each group has three sets of personality traits. Only one set actually goes with that birthday group (according to the Zodiac calendar).
▪Look for your birthday. Read the three sets of personality traits for your birthday group. Choose the set of personality traits that you think matches your own personality. We want to investigate if the Zodiac’s personality predictions are likely to be correct.
20
Intentional Use of the Hidden Curriculum
▪Increase awareness and cultural competence
▪Address issues of equity and diversity
21
Sex and Gender
Members of an advocacy organization for transgender rights were asked what sex they were assigned at birth (female or male) and how the identify their gender presentation now (woman, man or other). The results are summarized in the table below
What percent of those who were assigned female at birth now present as other than man or woman?
22
Gender PresentationSex Assigned at Birth
Woman
Man Other Total
Female 10 62 9 81Male 75 13 11 99Total 85 75 20 180
Poverty Rates of Elderly Americans
Data from 2001 gives the poverty rates of elderly Americans (65 years and older) broken down by race (U.S. Census Bureau, 2002). The poverty rate is the percent of people in a group who have income below the poverty line.
What percent of poor elderly Americans are white?
23
Poverty rates of elderly Americans by race
Race Poverty rate Percent of total elderly population
Black 21.9% 8.4%
White 8.1% 82.8%
Other 17.7% 8.7%
Conclusions
24
Initially, students struggled with the format of these items compared to traditional assessment items – They needed to overcome their fear of word problems!
Soon, students began to see real value in developing their mathematical skills– The proverbial “lightbulb” came on when concepts were introduced in a relevant context!
Finally, students began using these skills across disciplines – They showed a detailed understanding of ideas by applying these skills in other classes!
25
Questions
