Keith Wurtz, Senior Research Analyst, Chaffey College

Giovanni Sosa, Research Analyst, Chaffey College

Jim Fillpot, Director of Institutional Research, Chaffey College

An Integrated Model for Measuring the Impact of Course, Program, and Institutional Learning Outcomes

Strengthening Student Success ConferenceOctober 9th, 2009

Measuring the Impact of Learning Outcomes

“Good evidence, then, is obviously related to the questions the college has investigated and it can be replicated, making it reliable. Good evidence is representative of what is, not just an isolated case, and it is information upon which an institution can take action to improve. It is, in short, relevant, verifiable, representative, and actionable.”

Guide to Evaluating InstitutionsACCJC, August 2008

Measuring the Impact of Learning Outcomes

How do we collectively examine disparate learning outcomes assessment efforts? (representativeness and replicability)

How do we provide evidence about these disparate efforts? (verifiable)

How can we use evidence to improve upon learning outcomes practices? Upon district practices? (actionable)

Measuring the Impact of Learning Outcomes

Assessment often stands alone; relationships are often not explored between:

Courses From Different Programs Diverse Programs (within Instruction and Between Instruction and Student Services) Course/Program Learning Outcomes and Their Contribution to Institutional Learning Outcomes Types of Learning Outcomes (SLOs & AUOs) Differing Methods of Assessment Learning Outcomes and College Goals

Measuring the Impact of Learning Outcomes: Statistical Analysis

How do we statistically assess impact? Tests of statistical significance are not

necessary Greatly influenced by sample size (mean =

10.0; 10.1) Do not speak to the magnitude of the

difference Not well understood – even by ‘experts’

Effect Size as measure of practical significance Unstandardized Standardized (d, r)

Cohen’s conventions: d = .20 – small; .50 – moderate; .80 – large

Statistical Significance N = 30

Question Item Statistically

Significant? Q1 - FAFSA Deadline No Q2 - Online FAFSA Application No Q3 - Pin Number No Q4 - Application Window No Q5 - Verification Process No Q6 - Checking Award Status No Q7 - Receipt of Funds No Q8 - Enrollment Requirements No Q9 - Progress Requirements No Overall (Avg.) No

N = 30 Question Item Number Correct

Pre-Test Number Correct

Post-Test Statistically Significant?

Q1 19 24 No Q2 12 16 No Q3 26 30 No Q4 7 10 No Q5 13 21 No Q6 5 13 No Q7 10 15 No Q8 6 16 No Q9 3 15 No Overall (Avg.) 7.0 9.0 No

Effect Size (unstandardized)

N = 30

Question Item Number Correct Pre-Test

Number Correct Post-Test

Statistically Significant? Effect Size (d)

Q1 19 24 No .37 Q2 12 16 No .27 Q3 26 30 No .75 Q4 7 10 No .22 Q5 13 21 No .55 Q6 5 13 No .60 Q7 10 15 No .34 Q8 6 16 No .71 Q9 3 15 No .93 Overall (Avg.) 5.0 7.0 No .61

Effect Size (standardized)

Meta-Analysis

Combines research results across all identified studies Combines results using Effect Size (i.e.,

d) - generates an Average d

Average d may vary across study characteristics (i.e., Moderators): Number of Assessment Cycles Type of Assessment Instrument

Moderators(Grouping Variables)

Term – term data collection occurred Academic Year (AcadYr) – Academic year

(SU, FA, SP) data collection occurred Learning Outcome (LearnOutcome)

AUO – Administrative Unit Outcome SLO – Student Learning Outome

Researcher – person who conducted research

Program – department Course

Effect Size by Academic Year

Number of Studies 1 9 15 44 48

Effect Size by Learning Outcome Type (AUO or SLO)

Number of Studies 2 115

Effect Size by Program

Number of Studies

3 1 14 1 2 11 10 5 11 59

Effect Size by Course

Number of Studies 1 21 12 12 14 3 1 4 2 4 2

Moderators (Continued)(Grouping Variables - Measure)

Criterion – How was the Criteria for Success set by the program? Population – A measure obtained from the population (e.g.: critical thinking, Nursing Licensing

Exam, etc.) Pre-assessment score – The pre-assessment score in a pre-post assessment Program – The program set a specific criterion (e.g.: PE, Communication Studies, etc.)

PrePost – The assessment was a post-assessment or pre-post Standardized – Measure used to assess outcome was standardized or unstandardized Instrument

Rubric True/False Multiple choice Matching Likert Scale Anchored Scale

Unduplicated – used to identify number of students impacted by learning outcomes processes. From 2004-005 to 2008-2009 6,714 students have been impacted by learning outcomes.

Assessor Self-Assessment (e.g.: student self-assessed) External Evaluation (e.g.: student evaluated by teacher)

Effect Size by Criterion Type

Number of Studies 1 31 85

Effect Size by Pre-Post or Post-Assessment

Number of Studies 85 32

Effect Size by whether the Instrument was Standardized or Unstandardized

Number of Studies 10 107

Effect Size by Measurement Type

Number of Studies 7 22 2 8 75 3

Effect Size by whether Student Self-Assessed Learning or Learning was Assessed by External Evaluator (e.g.: teacher)

Number of Studies 82 35

Moderators (Continued)(Grouping Variables - Planning)

EndsPolicy1 (1) Instructional and Student Services, (2) Comprehensive Education

Program, (3) Collaborative Partnerships, (4) Continuous Improvement, (5) Learning Outcomes, (6) Core Competencies

EndsPolicy2 (1) Foundation Skills Courses, (2) Occupational Programs, (3) Transfer

Level Programs, (4) Outreach, (5) Student Services, (6) Library and Learning Support Services

EndsPolicy3 (1) College Facilities, (2) Technology, (3) Human Resources

EndsPolicy4 (1) Balanced Budget, (2) Achievement of Planned Enrollment Growth,

(3) External Funding, (4) Bond Reserve EndsPolicy5

(1 Qualified Personnel and Professional Development, (2) Commitment to Diversity, (3) Employment Agreement, (4) Professional Ethics

Effect Size by Ends Policy 1: Learning Centered College

Number of Studies 1 7 23 86

Effect Size by Ends Policy 2: Institutional Effectiveness

Number of Studies 1 92 22 1

Moderators (Continued)(Grouping Variables – Learning Outcomes)

KSA – Knowledge, Skill, or Ability (e.g.: critical thinking, speech, confidence, self-efficacy, etc.)

KSACateg = KSA Categorized by Core Competency and Instruction1. Communication – Students will demonstrate effective communication and

comprehension skills.2. Critical Thinking – Students will demonstrate critical thinking skills in problem

solving across the disciplines and in daily life3. Community/Global Awareness and Responsibility – Students will demonstrate

knowledge of significant social, cultural, environmental and aesthetic perspectives

4. Personal, Academic, and Care Development – Students will assess their own knowledge, skills and abilities; set personal, educational, and career goals; work independently and in group settings; identify lifestyle choices that promote self reliance, financial literacy and physical, mental and social health

5. Instruction – Students will demonstrate KSA specific to course subject (e.g.: theory identification, research methods, etc.)

Cycles – Number of cycles program has assessed outcome Level

Course Program Institution

Effect Size by Knowledge, Skill, or Ability

Number of Studies 65 9 12 20 11

Effect Size by Number of Times Program has Assessed Outcome

Number of Studies 74 34 8 1

Effect Size by Course and Program Level Learning Outcomes

Number of Studies 77 40

Limitations

Database only includes results from outcomes evaluated by the Office of Institutional Research

How do we connect information to student success (e.g.: goals, course success, transfer, etc.)?

Resources

Microsoft Excel Spreadsheets that calculate effect size statistics: http://www.stat-help.com/spreadsheets.html

Effect Size Calculator with Confidence Intervals: http://www.cemcentre.org/renderpage.asp?linkID=30325017