carol gittens, phd santa clara university wasc assessment leadership academy january 26, 2012
TRANSCRIPT
Critical Thinking: What It Is &
How To Assess It
Carol Gittens, PhDSanta Clara University
WASC Assessment Leadership Academy
January 26, 2012
Goals for the SessionAugment your understanding of the
definition of critical thinking
Expand your repertoire of classroom, program, and institution level critical thinking assessment strategies
Engage and Affirm your critical thinking skills and positive critical thinking habits of mind
Challenging a Few MythsCT is what you learn at school (but it
doesn’t apply to real life).
Critical thinking naturally improves just
from being in college.
Nobody knows what “critical thinking”
means.
Whatever it is, CT can’t be measured…
© 2011 Measured Reasons, Hermosa Beach, CA
Failures of critical thinking contribute to…patient deaths * lost revenue * ineffective
law enforcement * job loss * gullible voters * garbled communications * imprisonment * combat casualties * upside down mortgages * vehicular homicide * bad decisions * unplanned pregnancies * financial mismanagement * heart disease * family violence * repeated suicide attempts * divorce * drug addiction * academic failure * … * … *
© 2011 Peter A. & Noreen C. Facione and Measured Reasons, Hermosa Beach, CA
WHAT WERE WE THINKING?
Novel Question Human Reflective Response Time
Tick
TickTick
Tick
Tick
Tick
Humans need 11 - 16 secondsto process a novel question.
Tick
Tick
TickTick
TickTick
TickTick
Tick
What does this mean for a student faced with an novel question in a classroom, field, or testing
setting?
And what should the instructor, supervisor or assessor do?
in contexts of uncertainty, risk
“Critical Thinking” - Working Consensus
Purposeful, reflective judgment which manifests itself in reasoned consideration
of evidence, context, methods, standards, and conceptualizations
in deciding what to believe or what to
do.The Delphi Report: Executive Summary: (1990), The California Academic Press, or ERIC Doc ED315 423
Purposeful, reflective judgment which manifests itself in reasoned consideration of evidence, context, methods,
standards, and conceptualizations in deciding what to believe or what to do.
Three Ways Humans Reason Bottom-Up:
Observations Hypotheses Scientific Thinking
Top-Down: Beliefs & Values Applications Ideological Thinking
This Is Like That: Familiar Unfamiliar Analogical / Scripted Thinking
© 2011 Measured Reasons LLC, Hermosa Beach, CA. All rights reserved
Critical Thinking Habits of Mind
(
American Philosophical Association Delphi Report: Consensus Definition of Critical Thinking (1990) Download : CT Resources Link www.insightassessment.com
CCTDICalifornia Critical Thinking Dispositions InventoryAvailable in 20 culturally competent translations
Positive Habit Negative Habit
© 2011 Measured Reasons, Hermosa Beach, CA. All rights reserved.
© 2011 Measured Reasons, Hermosa Beach, CA
Planning Design Measurement Interpretation Value
Assessment Questions
1. How is assessment different than grading?
2. Is our main goal developmental or summative?
3. Which curricular level should we assess? lessoncoursegeneral education requirementmajordegree programtotal institutional experience
Assessment Planning Questions:
4. If we hope to show positive growth, do we have clean baseline / pretest data?
5. If we hope to show comparative strength, do we have relevant benchmark / norms?
6. If we hope to demonstrate accomplishing goals, do we a clear criteria of success?
Design Questions:
7. Do we have a focused, accurate, consensus definition of the construct we seek to measure?
8. Do we have one or more valid, reliable ways of gathering data about that construct?
9. Do we have a clear idea of which students we shall need to sample?
10. Must we gather data from them all, or can we use representative sampling?
Measurement Questions:
© 2012 Measured Reasons, Hermosa Beach, CA
Three Basic Options for Measuring Learning Outcomes
www.InsightAssessment.com Measuring Critical Thinking Worldwide
1. Rubrics and Rating ToolsQualitative Rating Forms, Typological Matches, ChecklistsRequire practiced judgment and inter-rater calibration
Adaptable to performance and written data
3. Testing InstrumentsBaseline / Cross-Sectional / Longitudinal
Time thrifty / Norm referencedPotential for comparisons & data integration
2. Self ReportsJournals, Self Critiques, Focus Groups, QuestionnairesInsights about personal progress and deficiency Require significant resources for data analysis
Are we consistently getting a valid and reliable measure of the phenomenon we intended to target?
11. When and where should we gather data?
12. How can we motivate student to give their best effort?
13. Who will collect and store the data?
14. Who will score, rate, compile and analyze the data?
Logistical & Tactical Questions:
© 2012 Measured Reasons, Hermosa Beach, CA
Teaching and Assessment Tool Scoring Rubrics Holistic Critical Thinking
Scoring Rubric (HCTSR)
Excellent/Strong = 4
Adequate/Satisfactory = 3
Deficient/Weak = 2
Truly Dreadful = 1
www.InsightAssessment.com Measuring Critical Thinking Worldwide
Describe three or four levels of performance
© 2010 Peter A. & Noreen C. Facione and Measured Reasons, Hermosa Beach, CA
Share a scoring rubric from day-1
to establish expectations
to make “critical thinking” operational for students
Inductive Reasoning Deductive Reasoning Analysis & Interpretation Inference Explanation & Evaluation
Total Percentile Demographics
Scale Scores on the CCTST
© 2011 Measured Reasons, Hermosa Beach, CA. All rights reserved.
ANYWHERE U., May 2010 – CCTST Raw Score – Qualitative Categories
© 2011 Measured Reasons, Hermosa Beach, CA. All rights reserved.
Weak
Moderate
Strong
Insight Assessment LLCMeasuring Critical Thinking Worldwide
Publishers of the California Critical Thinking Tests& other measures of reasoning and problem
solving: California Critical Thinking Skills Test (CCTST) Test of Everyday Reasoning (TER) California Critical Thinking Disposition Inventory (CCTDI) Health Sciences Reasoning Test (HSRT – Parts 1&2 ) California Measure of Mental Motivation (CM3) Business Critical Thinking Skills Test (BCTST) Business Reasoning Test (BRT) Business Attitude Inventory (BAI) Legal Studies Reasoning Profile (LSRP - Parts 1 & 2) Military and Defense Critical Thinking Inventory (MDCTI -Parts 1&2) Quant Q (Test of Mathematical Reasoning)
Willing and Able
Interpretation Question: 15. By exit, what percentage of students should be in a mid-range or higher category?
35 18
180 MBA Students
127
© 2012 Measured Reasons, Hermosa Beach, CA
Using Quantitative / Categorical
Scores
2005 2006 2007 2008 2009 20100
5
10
15
20
25
30
35
40
PretestPosttest
Interpretation Question 16: Should we benchmark internally to our own past performance or to norms of actual or aspirational peers?
17. Do cross-sectional comparisons mean anything, or are the differences explainable by student attrition, maturation, selection bias, etc.?
18. What level of achievement should we have expected of our students as a group?
19. If we see gains, are they educationally significant and for which subgroups of students?
More Interpretive Questions:
© 2012, and Measured Reasons, Hermosa Beach, CA
TRUTH-SEEKING:
the courageous desire for the best possible
knowledge in any given context,
the inclination to ask hard questions,
and to follow reason and evidencewherever they lead.
© 2011 Measured Reasons, Hermosa Beach, CA
Drs. Peter & Noreen Facione – © 2007 – www.insightassessment.com
Truth-seeking: A Profile of 155 Entering Freshmen 1992
10 15 20 25 30 35 40 45 50 55 60
Mean = 35.7s.d. = 5.6
Weak dispositionToward truth-seeking
Strong dispositionToward truth-seeking
A Look across Four Years at the Disposition toward Critical Thinking Among Undergraduate Students, C. Giancarlo and P. Facione, The Journal of General Education, (2001). Volume 50, number 1. 29-55.
Drs. Peter & Noreen Facione – © 2007 – www.insightassessment.com
Truth-seeking: Exiting Seniors 1996The same 155 students 4 years later
10 15 20 25 30 35 40 45 50 55 60
Positive movement across categories 38.8%
Negative movement across categories 10.9%
Mean = 38.5s.d. = 6.2
A Look across Four Years at the Disposition toward Critical Thinking Among Undergraduate Students, C. Giancarlo and P. Facione, The Journal of General Education, (2001). Volume 50, number 1. 29-55.
Strong dispositionToward truth-seeking
Weak dispositionToward truth-seeking
Gains: Critical Thinking Disposition
www.InsightAssessment.com Measuring Critical Thinking Worldwide
Poor CT Skills Moderate CT Skills Strong CT Skills
Critical Thinking Scores – Implications for Learning
Avers
e
N
eg
ati
ve
Am
biv
ale
nt
Posit
ive
Str
on
g
© 2008 Insight Assessment, The California Academic Press LLC: Millbrae, CA.
60
50
40
30
20
10
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
Name New Transfers Students Date ___Fall 2008 Pre-Test_____Skills Measure ___CCTST_- 2000_ Dispositions Measure ____CCTDI__
Strong Dispositions / Strong Skills
Negative Dispositions / Poor Skills
Neutral Dispositions / Moderate Skills
Strong Dispositions / Moderate Skills
Use fair-minded, reflective thinking when interpreting
results! Use Powerful Critical Thinking Skills:
Interpret the data display Analyze and explain what you find What can we infer from these data? Evaluate the inference we just drew Rethink a judgment in light of new facts
Call Forth Positive CT Habits of Mind: Go ahead, Ask. Have courage and seek truth Follow the data and reasons wherever they lead Keep an open-mind about what others have to say Proceed systematically, don’t jump to conclusions Don’t lock yourself in – be ready to reconsider when
conditions change
20. How do we fund the faculty time, staff positions, and operating budget, and how do we manage impact of outcomes assessment on other academic and institutional priorities?
21. Which unit has the expertise and objectivity to analyze, interpret and report on the data we gather?
22. How can we maximize the benefits of outcomes assessment to inform and support other functions?
Accreditation, Admissions, Student Success, Institutional Accountability, Curricular Development, Research, Grants, Fund-Raising
Value Questions:
© 2012, and Measured Reasons, Hermosa Beach, CA
Assessing Critical Thinking
Are there a variety of assessments that engage students in one or more critical thinking skills?
Are there assessments that elicit students’ critical thinking habits of mind?
Do the assessments provide opportunities to evaluate students’ independent critical thinking and their thinking in groups?
Are there a sufficient number of assessments that will be reviewed and returned to students so that they receive frequent feedback on their performances?
What benchmarking data will be used for CT assessments at the course, program, and institutional levels?
How will assessment results be shared to the campus community so that students are able to practice their critical thinking skills and dispositions and make appropriate modifications?
© 2011 Measured Reasons, Hermosa Beach, CA. All rights reserved.
Thoughts on valid and reliable methods to demonstrate gains on desired outcomes
Use local talent and good data
Student motivation and timing
Data analysts and true measures
www.InsightAssessment.com Measuring Critical Thinking Worldwide
Use the right tools
Attention to design is needed
Correct calibration and clear interpretation
Use thinking verbs to express outcomes and to ask questions in class and on assessment tools.
ParticipateListenKnowUnderstandRecognize
IdentifyCategorizeNameDiscuss
AnalyzeInterpretExplainInferEvaluate
To measure a phenomenon it must first be present
© 2011 Measured Reasons, Hermosa Beach, CA
Why Teach and Assess for Critical Thinking?
In education measure what you value because you get what you measure.
Critical thinking – purposeful reflective judgment - is the key to academic success, a necessary element in every professional endeavor, and a central factor in individual and communal adaptation and survival.
© 2011 Peter A. & Noreen C. Facione and Measured Reasons, Hermosa Beach, CA
