a quantitative literacy assessment rubric
Post on 23-Feb-2016
47 Views
Preview:
DESCRIPTION
TRANSCRIPT
A Quantitative Literacy Assessment Rubric
Development & Lessons Learned
- Stuart Boersma, Central Washington Univ.- Caren Diefenderfer, Hollins University- Shannon Dingman, Univ. of Arkansas- Bernie Madison, Univ. of Arkansas
Supported by the National Science Foundation DUE-0715039
Numeracy Article: Quantitative Reasoning in the Contemporary World 3: Assessing Student Learninghttp://services.bepress.com/numeracy/
Our Course
• Uses case studies centered on one or more media articles
• Each case study contains study questions focusing on the quantitative content of the article
• Student responses often involve critiquing, analyzing, decision making, and explaining.
Making Assessment Difficult!!QRCW Casebook: Case Studies for Quantitative Reasoning: A Casebook of Media Articles, 2nd edition, Pearson Custom Publishing. ISBN13: 978-0-558-19880-0.
Need: Instrument that we could apply to these non-standard student responses.
AAC&U VALUE Project:•Developed for institutional level assessment (not for grading student work)•15 areas, 10 of which are termed Intellectual and Practical skills:• Inquiry and analysis, critical thinking, written
communication, oral communication, reading, quantitative literacy, information literacy, teamwork, problem solving
A Little History
InterpretationRepresentationCalculation
Application/AnalysisAssumptionsCommunication
Alignment of QL VALUE Rubric w/ QRCW course
Missing Competencies?:– Critical Reading– Number sense– Synthesis
Scale– No “zero”– “4” designated capstone
achievement; not a consideration for student work in a QL course
Folded into “Interpretation”
Changed “application/analysis” to “analysis/synthesis”
New Scale: 0-3
Achievement Level
Quantitative Literacy Core Competency
3 2 1 0
Interpretation Ability to glean and explain
mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words)
Correctly identifies all relevant information.
Correctly identifies some, but not all, relevant
information.
Some relevant information is identified, but none is
correct.
No relevant information identified.
Representation Ability to convert information from
one mathematical form (e.g. equations, graphs, diagrams, tables,
words) into another.
All relevant conversions are present and correct.
Some correct and relevant conversions are present but
some conversions are incorrect or not present.
Some information is converted, but it is irrelevant
or incorrect.No conversion is attempted.
Calculation Ability to perform arithmetical and
mathematical calculations.
Calculations related to the problem are correct and lead to a successful completion
of the problem.
Calculations related to the problem are attempted but either contain errors or are
not complete enough to solve the problem.
Calculations related to the problem are attempted but contain errors and are not complete enough to solve
the problem.
Calculations given are not related to the problem, or no
work is present.
Analysis/Synthesis Ability to make and draw
conclusions based on quantitative analysis.
Uses correct and complete quantitative analysis to
make relevant and correct conclusions.
Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or
partially complete (e.g. there are logical errors or
unsubstantiated claims).
An incorrect quantitative analysis is given to support a
conclusion.
Either no reasonable conclusion is made or, if present, is not based on quantitative analysis.
Assumptions Ability to make and evaluate
important assumptions in estimation, modeling, and data
analysis.
All assumptions needed are present and justified when
necessary.
At least one correct and relevant assumption is given
(perhaps coupled with erroneous assumptions), yet some important assumptions
are not present.
Attempts to describe assumptions, but none of the assumptions described are
relevant.
No assumptions present.
Communication Ability to explain thoughts and
processes in terms of what evidence is used, how it is organized,
presented, and contextualized.
A correct and complete explanation is clearly
presented.
A partially correct relevant explanation is present, but
incomplete or poorly presented.
A relevant explanation is present, but is illogical, incorrect, illegible, or
incoherent.
No relevant explanation is provided.
Use of QL VALUE Rubric in QRCW course
Descriptions of achievement levels not objective enough for our uses:
Uses quantitative analysis of data as the basis for competent judgments, drawing reasonable conclusions…
Uses quantitative analysis of data as the basis for workmanlike judgments, drawing plausible conclusions…
v
3 2
v
Skillfully converts relevant information into insightful mathematical portrayal…
Competently converts relevant information into an appropriate and desired mathematical portrayal
4 3
Achievement Level
Quantitative Literacy Core Competency
3 2 1 0
Interpretation Ability to glean and explain
mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words)
Correctly identifies all relevant information.
Correctly identifies some, but not all, relevant
information.
Some relevant information is identified, but none is
correct.
No relevant information identified.
Representation Ability to convert information from
one mathematical form (e.g. equations, graphs, diagrams, tables,
words) into another.
All relevant conversions are present and correct.
Some correct and relevant conversions are present but
some conversions are incorrect or not present.
Some information is converted, but it is irrelevant
or incorrect.No conversion is attempted.
Calculation Ability to perform arithmetical and
mathematical calculations.
Calculations related to the problem are correct and lead to a successful completion
of the problem.
Calculations related to the problem are attempted but either contain errors or are
not complete enough to solve the problem.
Calculations related to the problem are attempted but contain errors and are not complete enough to solve
the problem.
Calculations given are not related to the problem, or no
work is present.
Analysis/Synthesis Ability to make and draw
conclusions based on quantitative analysis.
Uses correct and complete quantitative analysis to
make relevant and correct conclusions.
Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or
partially complete (e.g. there are logical errors or
unsubstantiated claims).
An incorrect quantitative analysis is given to support a
conclusion.
Either no reasonable conclusion is made or, if present, is not based on quantitative analysis.
Assumptions Ability to make and evaluate
important assumptions in estimation, modeling, and data
analysis.
All assumptions needed are present and justified when
necessary.
At least one correct and relevant assumption is given
(perhaps coupled with erroneous assumptions), yet some important assumptions
are not present.
Attempts to describe assumptions, but none of the assumptions described are
relevant.
No assumptions present.
Communication Ability to explain thoughts and
processes in terms of what evidence is used, how it is organized,
presented, and contextualized.
A correct and complete explanation is clearly
presented.
A partially correct relevant explanation is present, but
incomplete or poorly presented.
A relevant explanation is present, but is illogical, incorrect, illegible, or
incoherent.
No relevant explanation is provided.
CompetencyNumber
of questions
Percent of
questions
Number of case studies (N=24) which
have at least one question mapped to
competency
Number of case studies which have 50% or
more questions mapped to
competency
Interpretation 152 65% 24 18
Representation68 29% 19 6
Calculation 101 43% 21 9
Analysis/Synthesis
79 34% 20 7
Assumptions 14 6% 10 0
Communication
53 23% 18 4
Frequency and Prevalence of Core Competencies in QRCW Casebook
Reliability of QLAR
• Focus on Case Study 4.3• Readers A and B scored questions 3a, 3b, 3c,
and 3d on 24 student papers (section 1)• Readers C and D scored 2b, 2c, 2d, 3a, and 3d
from 24 papers (section 2)
Question Competencies 2b Assumptions 2c Assumptions 2d Assumptions 3a Representation, calculation,
analysis/synthesis, communication3b Interpretation, analysis/synthesis 3c Interpretation, analysis/synthesis 3d Representation, calculation
Core Competency for Study Questions
Reliability Test 1: Readers A and B
QL Competency # of disagreements
N % disagreement
Interpretation 1 48 2.08%Representation 1 48 2.08%Calculation 3 46 6.25%Analysis/Synthesis 2 72 2.78%
Communication 0 24 0%
Disagreement Level Per Core Competency
Overall: •Readers A and B disagreed on 7/238 = 2.94% of assessments.•Average of 7 differences: 1.23
Reliability Test 2: Readers C and DDisagreement Level Per Core Competency
QL Competency # of disagreements
N % disagreement
Assumptions 2 42 4.76%
Representation 2 28 10.71%
Calculation 6 28 21.43%
Analysis/Synthesis
0 14 0%
Communication 5 14 35.7%
Overall: •Readers C and D disagreed on 15/126 = 11.9% of assessments.•Average of 7 differences: (just over) 1
Added Value of QLAR
1. Allows one to align student prompts with an organized thought process as represented by the rubric
To solve a complex problem a student needs to read and glean information, convert the information into a usable form, calculate, make or draw a conclusion, and then communicate the result via explanation.
Scaffold student learning by sorting case studies by competency: Start with Interpretation & Representation and build up to more complex prompts.
Students often do not recognize these as components of a strong response. Guiding students through mastery of these competencies is crucial!
Added Value of QLAR
2. Improved student prompts.
Find out how the Standard & Poor’s 500 stock index is computed.
Describe how the Standard & Poor’s 500 stock index is computed.
Explain your reasoning!
Be sure to differentiate between quantitative information taken directly from the article and extra assumptions you made.
Justify any extra assumptions.
Use a quantitative argument to…[answer some question]
QLAR Achievement Level
Quantitative Literacy Core Competency
3 2 1 0
Interpretation Ability to glean and explain
mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words)
Correctly identifies all relevant information.
Correctly identifies some, but not all, relevant
information.
Some relevant information is identified, but none is
correct.
No relevant information identified.
Representation Ability to convert information from
one mathematical form (e.g. equations, graphs, diagrams, tables,
words) into another.
All relevant conversions are present and correct.
Some correct and relevant conversions are present but
some conversions are incorrect or not present.
Some information is converted, but it is irrelevant
or incorrect.No conversion is attempted.
Calculation Ability to perform arithmetical and
mathematical calculations.
Calculations related to the problem are correct and lead to a successful completion
of the problem.
Calculations related to the problem are attempted but either contain errors or are
not complete enough to solve the problem.
Calculations related to the problem are attempted but contain errors and are not complete enough to solve
the problem.
Calculations given are not related to the problem, or no
work is present.
Analysis/Synthesis Ability to make and draw
conclusions based on quantitative analysis.
Uses correct and complete quantitative analysis to
make relevant and correct conclusions.
Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or
partially complete (e.g. there are logical errors or
unsubstantiated claims).
An incorrect quantitative analysis is given to support a
conclusion.
Either no reasonable conclusion is made or, if present, is not based on quantitative analysis.
Assumptions Ability to make and evaluate
important assumptions in estimation, modeling, and data
analysis.
All assumptions needed are present and justified when
necessary.
At least one correct and relevant assumption is given
(perhaps coupled with erroneous assumptions), yet some important assumptions
are not present.
Attempts to describe assumptions, but none of the assumptions described are
relevant.
No assumptions present.
Communication Ability to explain thoughts and
processes in terms of what evidence is used, how it is organized,
presented, and contextualized.
A correct and complete explanation is clearly
presented.
A partially correct relevant explanation is present, but
incomplete or poorly presented.
A relevant explanation is present, but is illogical, incorrect, illegible, or
incoherent.
No relevant explanation is provided.
Interpretation Ability to glean and explain mathematical information presented in various forms
(e.g. equations, graphs, diagrams, tables, words)
3Correctly identifies all relevant information.
2Correctly identifies some, but not all, relevant information.
1Some relevant information is identified, but none is correct.
0No relevant information identified.
Representation Ability to convert
information from one mathematical form (e.g.
equations, graphs, diagrams, tables, words)
into another.
3All relevant
conversions are present and correct.
2Some correct and
relevant conversions are
present but some conversions are incorrect or not
present.
1Some information is converted, but it
is irrelevant or incorrect.
0No conversion is attempted
Calculation Ability to perform arithmetical and
mathematical calculations.
3Calculations related to the problem are correct and lead to a successful
completion of the problem.
2Calculations related to the problem are
attempted but either contain errors or are not complete enough
to solve the problem.
1Calculations related to
the problem are attempted but contain
errors and are not complete enough to solve the problem.
0Calculations given are not related to
the problem, or no work is present.
Analysis/Synthesis Ability to make and draw
conclusions based on quantitative analysis.
3Uses correct and complete
quantitative analysis to make relevant and correct conclusions.
2Quantitative
analysis is given to support a relevant conclusion but it is either only partially correct or partially complete (e.g. there are logical errors or
unsubstantiated claims).
1An incorrect
quantitative analysis is given to support a
conclusion.
0Either no
reasonable conclusion is made or, if present, is not
based on quantitative
analysis.
Assumptions Ability to make and evaluate important
assumptions in estimation, modeling, and data
analysis.
3All assumptions needed are present
and justified when necessary.
2At least one correct
and relevant assumption is given
(perhaps coupled with erroneous
assumptions), yet some important
assumptions are not present.
1Attempts to describe
assumptions, but none of the assumptions
described are relevant.
0No assumptions
present.
Communication Ability to explain
thoughts and processes in terms of what evidence is used, how it is organized,
presented, and contextualized.
3A correct and complete explanation
is clearly presented.
2A partially correct
relevant explanation is present, but
incomplete or poorly presented.
1A relevant explanation
is present, but is illogical, incorrect,
illegible, or incoherent.
0No relevant
explanation is provided.
top related