using idea for assessment howard community college august 17, 2012 shelley a. chapman, phd

Using IDEA for AssessmentHoward Community CollegeAugust 17, 2012

Shelley A. Chapman, PhD

Plan for this Session

• Program Evaluation & Assessment of Student Learning• Group Summary Reports• Accreditation Guides• Benchmarking Reports• Aggregate Data File

What makes IDEA unique?

1. Focus on Student Learning

2. Focus on Instructor’s Purpose

3. Adjustments for Extraneous Influences

4. Validity and Reliability5. Comparison Data6. Flexibility

Student Learning Model: 2 Assumptions

Assumption 1:

Types of learning must reflect the instructor’s purpose.

Student Diagnostic Form

Assumption 2:

Effectiveness determined by students’ progress on objectives stressed by instructor

Diagnostic Report Overview Page 1 – Big Picture

How did I do?

Page 3 – Diagnostic What can I do differently?

Page 2 – Learning Details What did students learn?

Page 4 – Statistical Detail Any additional

insights?

Your Average (5-point Scale)

Raw Adj.

A. Progress on Relevant Objectives1

Four objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.

The Big Picture

ProgressOnRelevantObjectives

4

4.3 + 4.34.14.23.6

5

Summary Evaluation: Five-Point Scale Report Page 1

Your Average Score

(5-point scale)

Raw Adj.

A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

Overall Ratings B. Excellent Teacher 4.7 4.9

C. Excellent Course 4.1 4.4

D. Average of B & C 4.4 4.7

Summary Evaluation(Average of A & D) 4.3 4.5

50%

25%25%

Individual Reports to Group Reports

The Group Summary Report

How did we do?How might we improve?

Defining Group Summary Reports (GSRs)

• Institutional• Departmental• Service/Introductory Courses• Major Field Courses• General Education Program

Adding Questions

Up to 20 Questions can be added

• Institutional• Departmental• Course-based• All of the above

Local CodeUse this section of the FIF to code types of data.

Defining Group Summary Reports• Local Code

• 8 possible fields• Example: Column one – Delivery Format

• 1=Self-paced• 2=Lecture• 3=Studio• 4=Lab• 5=Seminar• 6=Online

Example from Benedictine University

Example Using Local code

Assign Local Code• 1=Day, Tenured• 2=Evening, Tenured• 3=Day, Tenure Track• 4=Evening, Tenure

Track• 5=Day, Adjunct• 6=Evening, Adjunct

Request Reports• All Day Classes

• Local Code=1, 3, & 5• All Evening Classes

• Local Code=2, 4, & 6• Courses Taught by

Adjuncts• Local Code=5 & 6

Description of Courses Included in this Report

Number of Classes IncludedDiagnostic From 42Short Form 27Total 69

Number of Excluded Classes 0

Response RateClasses below 65% Response Rate 2Average Response Rate 85%

Class SizeAverage Class Size 20

Page 1 of GSR

Assessment of Learning

What are our faculty emphasizing?How do students rate their learning?How do our classes compare with others?How do our students compare with others (self-rated characteristics)?What efforts can we make for improvement?

(How can we “close the loop”?)

What learning are we targeting?

HCC Gen Ed Courses 1 2 3 4 5 6 7 8 9 10 11 12

English 121 Composition X X X X

Biology 101 General Biology X X X X

Math 108 Business Math X X X

Art History 104 Art History 1 X X

History 121 The Ancient World

X X X X

Psychology 101 General Psychology

X X X

Anthropology 110 Global Archeology

X X X

IDEA Learning Objectives

What are We Emphasizing?Percent of Classes Selecting Obj. as

Important or EssentialThis Group Institution IDEA

SystemObjective 1 16% 70% 78%Objective 2 13% 59% 75%Objective 3 41% 58% 75%Objective 4 32% 35% 55%Objective 5 23% 19% 32%Objective 6 32% 14% 25%Objective 7 22% 27% 27%Objective 8 78% 43% 47%

Objective 9 19% 23% 41%Objective 10 7% 11% 23%Objective 11 68% 42% 49%Objective 12 20% 23% 41%Average # of Obj. Selected 3.7 4.2 5.7

Page 2

What are We Emphasizing?

Page 9 Section B

NumberRating

Percent indicating amount required

None or Little

Some Much

Writing 66 2% 17% 82%

Oral Communication 66 6% 42% 52%

Computer Application 66 50% 44% 6%

Group Work 66 27% 59% 14%

Mathematics/Quantitative Work

65 97% 3% 0%

Critical Thinking 66 0% 30% 70%

Creative/Artistic/Design 66 61% 33% 6%

How do students rate their learning?Page 3 Part 1: Distribution of Converted Scores

Compared to the IDEA Database

Overall Progress Ratings (Courses)

Page 3 Percent of Classes at or Above the IDEA database Average

PRO Excellent Teacher

Excellent Course

Summary0

102030405060708090

RawAdjusted

A. Progress of Relevant Objectives-Converted Score Category

Expected Distribution

Fall 2010 Raw

Spring 2011 Raw

Fall 2011 Raw

Much Higher (63 or higher) 10% 6% 7% 7%Higher (56-62) 20% 32% 34% 31%Similar (45-55) 40% 46% 46% 44%Lower (38-44) 20% 11% 9% 10%Much Lower (37 or lower) 10% 5% 5% 8%

B. Excellence of Teacher-Converted Score CategoryExpected Distribution

Fall 2010 Raw

Spring 2011 Raw

Fall 2011 Raw


C. Excellence of Course-Converted Score CategoryExpected

Distribution

Fall 2010 Raw

Spring 2011 Raw

Fall 2011 Raw


Summary Evaluation (Average of A, B, C)-Converted Score Category

Expected Distribution

Fall 2010

Spring 2011

Fall 2011


Primary Instructional Approaches

Fall 2010

Primary

Spring 2011

Primary

Fall 2011

PrimaryLecture 61% 60% 61%Discussion/Recitation 7% 8% 10%Seminar 7% 9% 9%Skill/Activity 7% 6% 7%Laboratory 8% 9% 9%Field Experience 1% 1% 1%Studio 2% 3% 2%Multi−Media 0% 0% 0%Practicum/Clinic 0% 0% 0%Other/Not Indicated 6% 4% 2%

Number Classes Rating 787 677 859

Secondary Instructional Approaches

Fall 2010

Secon-dary

Spring 2011

Secon-dary

Fall 2011

Secon-dary

Lecture 16% 13% 16%Discussion/Recitation 30% 27% 30%Seminar 2% 3% 4%Skill/Activity 11% 13% 13%Laboratory 3% 6% 4%Field Experience 2% 3% 2%Studio 0% 0% 1%Multi−Media 2% 3% 3%Practicum/Clinic 1% 1% 1%Other/Not Indicated 34% 31% 28%

Number Classes Rating 787 677 859

Course Emphases: Writing

Fall 2010

Spring 2011

Fall 2011

None or Little 25% 26% 22%Some 46% 45% 44%Much 29% 30% 34%

0%

20%

40%

60%

80%

100%

Fall 2010 Spring 2011 Fall 2011

None or Little

Some

Much

Course Emphases: Oral communication

Fall 2010

Spring 2011

Fall 2011


0%

20%

40%

60%

80%

100%


None or Little

Some

Much

Instructors’ Reports on Course Emphases: Selected Pairings-Writing and Oral Communication

Course Emphases: Critical thinking

Fall 2010

Spring 2011

Fall 2011


0%

20%

40%

60%

80%

100%


None or Little

Some

Much

Course Emphases: Writing

Fall 2010

Spring 2011

Fall 2011


0%

20%

40%

60%

80%

100%


None or Little

Some

Much

Instructors’ Reports on Course Emphases: Selected Pairings-Critical Thinking & Writing

Highlights for ABCU• Remarkably similar profiles

across terms

• Overall response rates ranged from 75% to 81%

• 1st term in which administration was primarily online achieved a 75% response rate

• Transition from paper to online (fall 2010 to fall 2011) does not show major differences in profiles

• ABCU faculty focus on 3-4 outcomes as essential/important

• IDEA scores for ABCU faculty are in the similar range on each of the 4 primary indices of teaching effectiveness

• Over the last 3 terms, a significant increase on several objectives has been observed: application of course material, oral and written communication skills, & analysis and critical thinking skills (objectives 3, 8, & 11, respectively)

Overall Progress Ratings (Courses)

PRO Excellent Teacher

Excellent Course

Summary0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

RawAdjusted

Part 3: Percent of Classes at or Above This Institution’s Average

Page 4

Do Students’ report of learning meet our expectations?

Pages 5 and 6

Raw Average

Adj.Average

# ofClasses

ThisReport

3.9 3.9 11

Institution 4.2 4.2 3,963IDEA System

4.0 4.0 31,991

Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends)

IDEA System

Institution

This report

0 10 20 30 40 50 60 70 80 90 100

Percent of Classes Rating

At least 4.0At least 3.75At least 3.5

Which teaching methods might we use to improve learning?

Page 7Teaching Methods and Styles

Stimulating Student Interest # Classes Av. s.d.

15. Inspired students to set and achieve goals which really challenged them

42 3.8 0.5

# 15

0 5 10 15 20 25 30 35

FrequentlyInfrequently

Relationship of Learning

Objectives to Teaching Methods

How do students view course work demands?

Page 8B Student Ratings of Course CharacteristicsDiagnostic Form Item # & Item Average % Classes

Below 3.0% Classes 4.0 or Above

33. Amount of Reading

This Report 3.4 21% 24%

Institution 3.3 31% 19%

IDEA System 3.2 33% 15%

34. Amount of work in other (non-reading) assignments




35. Difficulty of subject matter




Program EvaluationGSRs, Accreditation Guides, Benchmarking Reports, Aggregate Data Files, Raw Data Files

GSRs Help Address Questions• Longitudinal• Contextual• Curricular• Pedagogical• Student Learning-

focused

Aggregate Data File

Allows you to • Use Excel

Spreadsheet• Use with SAS or

SPSS• Ask other types of

questions• Display data in

different ways Cate

gory

1

Catego

ry 2

Catego

ry 3

00.5

11.5

22.5

33.5

44.5

Column1Column2

BenchmarkingInstitutional and Discipline Reports

Benchmarking Reports

Comparison to• 6-10 Peers• Same Carnegie

Classification• IDEA database

Benchmarking Reports

The student, rather than the class, is the unit of analysis

Percentage of positive ratings is given rather than averages

Comparison Groups

ABCU ---------- ----------

Peer* ------------------------------

Carnegie ---------- ----------

National ---------- ----------

* Peer group is based on 10 institutions identified by the Provost staff and Deans

Response Rates

ABCU student participation is similar to that of each comparison group

ABCU=79%Peer=77%

Carnegie=79%National=75%

Students’ Perceptions

Trends: Student Perceptions• Small, steady positive trend over last 5 years

• ABCU clusters with the other comparison groups

• Although trends for ABCU are higher than for other groups, these are not meaningful differences

• Of the 5 contexts identified on the FIF, the strongest positive perceptions at ABCU were for progress on objectives in the graduate education context; this represents a narrow, yet important, segment of ABCU programs (see next slide)

Graduate Students

Instructional Objectives Selected by Instructors

Instructors’ Intentions/ focus Students’ Self-Reported Progress on Learning

Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)

% of total classes where instructor selected objective as “Essential” or “Important”

IDEA Objective 3Learning to apply course material (to improve thinking, problem solving, and decisions)

IDEA Objective 8

Objective 8: Developing skill in expressing oneself orally or in writing


Developing skill in expressing oneself orally or in writing

IDEA Objective 11

Objective 11: Learning to analyze and critically evaluate ideas, arguments, and points of view


Learning to analyze and critically evaluate ideas, arguments, and points of view

Teaching Methods and Styles Reported by Students

(Diagnostic Forms Only)

• Fostering Student Collaboration

• Encouraging Student Involvement

Assessment of Learning in HCC’s General Education

Howard Community CollegeGeneral Education

Possible IDEA Learning Objectives1

English 121: Composition 8, 9

Biology 101: General Biology 1, 2

Math 108: Business Math 1, 2, 5

Art History 104: Art History 1 6, 4, 1, 2

History 121: The Ancient World 1, 11, 8

Psychology 101: General Psychology

1, 2, 3,

Anthropology 110: Global Archeology

7, 10,12

1Based partly on study of percent of classes selecting objectives by discipline

How Did Students Rate their Learning in General Education?

Benchmarking Look at “General Education”

Using Aggregate Data for Assessment

Howard Community

College Goals

Program Learning

OutcomesProgram Learning

OutcomesProgram Learning

OutcomesGroup Summary

Report

Group SummaryReport,

Include Extra Questions

Benchmarking: One Year or 3-5 Year Trend Report

Benchmarking: Discipline Report

Course Learning Outcomes





HCC General Education Goals and IDEAHoward Community College General Education Goals

IDEA Learning Goals IDEA Teaching Methods Associated with Progress on Learning Goal

Written Communication8 Developing skill in expressing myself orally or in writing

5, 4, 14 (real life), 15, 18

Oral Communication

7 Gaining a broader understanding and appreciation of intellectual/cultural activity

7, 8, 13, 19

Scientific and Quantitative Reasoning

2 Learning fundamental principles, generalizations, or theories11 Learning to analyze and critically evaluate ideas, arguments, and points of view

2, 7, 8, 13, 15, 16, 18, 19

Critical Thinking 9 Learning how to find and use resources for answering questions or solving problems

2, 7, 8, 9, 13, 14, 15, 18, 19

Global Competency

7. Gaining a broader understanding of intellectual/cultural activity

Extra Question

7, 8, 13,19

Example: Pre-PharmacyPre-Pharmacy Goals & Outcomes IDEA data

Mastery of fundamental biological, chemical, and physical principles underlying the living systems

Learning Objective 2,Teaching Methods 2, 4, 6, 8, 10, 12, 15, 16

Apply knowledge of scientific principles and concepts to clinical/scientific problems

Learning Objective 3Teaching Methods 1, 2, 3, 4, 6, 8, 10, 11, 15

Prepare effective written reports and oral presentations to communicate science to professionals and lay persons

Learning Objective 8Teaching Methods 7, 15, 16, 18, 19

Use mathematical concepts to analyze data Learning Objectives 2 & 3Teaching Methods 1, 2, 3, 4, 6, 8, 10, 15

Questions? www.theideacenter.org

using idea for assessment howard community college august 17, 2012 shelley a. chapman, phd

Documents