using idea for assessment, program review, and sacs
DESCRIPTION
Using IDEA for Assessment, Program Review, and SACS. University of Alabama Birmingham September 11, 2012. Shelley A. Chapman, PhD. Plan for this Session. Program Evaluation & Assessment of Student Learning Group Summary Reports Aggregate Data File Benchmarking Reports Accreditation Guides. - PowerPoint PPT PresentationTRANSCRIPT
Using IDEA for Assessment, Program Review, and SACSUniversity of Alabama BirminghamSeptember 11, 2012
Shelley A. Chapman, PhD
Plan for this Session
• Program Evaluation & Assessment of Student Learning• Group Summary Reports• Aggregate Data File• Benchmarking Reports• Accreditation Guides
What makes IDEA unique?
1. Focus on Student Learning
2. Focus on Instructor’s Purpose
3. Adjustments for Extraneous Influences
4. Validity and Reliability
5. Comparison Data
6. Flexibility
Student Learning Model: 2 Assumptions
Assumption 1:
Types of learning must reflect the instructor’s purpose.
Student Diagnostic Form
Assumption 2:
Effectiveness determined by students’ progress on objectives stressed by instructor
Diagnostic Report Overview Page 1 – Big Picture
How did I do?
Page 3 – Diagnostic What can I do differently?
Page 2 – Learning Details What did students learn?
Page 4 – Statistical Detail Any additional
insights?
Your Average (5-point Scale)
Raw Adj.
A. Progress on Relevant Objectives1
Four objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.
The Big Picture
ProgressOnRelevantObjectives
4
4.3 + 4.34.14.23.6
5
Summary Evaluation: Five-Point Scale
Report Page 1
Your Average Score
(5-point scale)
Raw Adj.
A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
Overall Ratings B. Excellent Teacher 4.7 4.9
C. Excellent Course 4.1 4.4
D. Average of B & C 4.4 4.7
Summary Evaluation(Average of A & D) 4.3 4.5
50%
25%
25%
Individual Reports to Group Reports
The Group Summary Report
How did we do?
How might we improve?
Defining Group Summary Reports (GSRs)
• Institutional
• Departmental
• Service/Introductory Courses
• Major Field Courses
• General Education Program
GSRs Help Address Questions
• Longitudinal• Contextual• Curricular• Pedagogical• Student Learning-
focused
Adding Questions
Up to 20 Questions can be added
• Institutional
• Departmental
• Course-based
• All of the above
Local CodeUse this section
of the FIF to code types of data.
Defining Group Summary Reports• Local Code
• 8 possible fields• Example: Column one – Delivery Format
• 1=Self-paced• 2=Lecture• 3=Studio• 4=Lab• 5=Seminar• 6=Online
Example from Benedictine University
Example Using Local code
Assign Local Code• 1=Day, Tenured• 2=Evening, Tenured• 3=Day, Tenure Track• 4=Evening, Tenure
Track• 5=Day, Adjunct• 6=Evening, Adjunct
Request Reports• All Day Classes
• Local Code=1, 3, & 5
• All Evening Classes• Local Code=2, 4, & 6
• Courses Taught by Adjuncts• Local Code=5 & 6
Description of Courses Included in this Report
Number of Classes IncludedDiagnostic From 42
Short Form 27
Total 69
Number of Excluded Classes 0
Response RateClasses below 65% Response Rate 2
Average Response Rate 85%
Class SizeAverage Class Size 20
Page 1 of GSR
UAB Spring 2012
Page 1 of GSR
Assessment of Learning
What are our faculty emphasizing?
How do students rate their learning?
How do our courses compare with others?
How do our students compare with others (self-rated characteristics)?
What efforts can we make for improvement?
(How can we “close the loop”?)
UAB Core Competencies
Shared Vision for a UAB Graduate
Possible IDEA Learning Objectives
Communication 8, 11
Knowledge 1, 2, 4, 7
Problem Solving 3, 9, 11
Citizenship 10, 12, extra questions
Are we targeting “Core Competencies” in the Core Curriculum?
UAB Core Curriculum Courses
1 2 3 4 5 6 7 8 9 10 11 12
EH 101 English Composition X X X X
BY 101 Topics in Contemporary Biology
X X X X
MA 105 Pre-Calculus Algebra X X X
ARH 101 The Art Experience X X X
HY 101 Western Civilization I X X X X
PY 101 Introduction to Psychology
X X X X
IDEA Learning Objectives
Are we targeting “Core Competencies” in the Core Curriculum?
UAB Core Curriculum Courses
1 2 3 4 5 6 7 8 9 10 11 12
EH 101 English Composition X X X X
BY 101 Topics in Contemporary Biology
X X X X
MA 105 Pre-Calculus Algebra X X X
ARH 101 The Art Experience X X
HY 101 Western Civilization I X X X X
PY 101 Introduction to Psychology
X X X
IDEA Learning Objectives
What are We Emphasizing?Percent of Classes Selecting Obj. as
Important or Essential
This Group Institution IDEASystem
Objective 1 16% 70% 78%
Objective 2 13% 59% 75%
Objective 3 41% 58% 75%
Objective 4 32% 35% 55%
Objective 5 23% 19% 32%
Objective 6 32% 14% 25%
Objective 7 22% 27% 27%
Objective 8 78% 43% 47%
Objective 9 19% 23% 41%
Objective 10 7% 11% 23%
Objective 11 68% 42% 49%
Objective 12 20% 23% 41%
Average # of Obj. Selected 3.7 4.2 5.7
Page 2
UAB Spring 2012
What are We Emphasizing?
Page 9 Section B
NumberRating
Percent indicating amount required
None or Little
Some Much
Writing 66 2% 17% 82%
Oral Communication 66 6% 42% 52%
Computer Application 66 50% 44% 6%
Group Work 66 27% 59% 14%
Mathematics/Quantitative Work
65 97% 3% 0%
Critical Thinking 66 0% 30% 70%
Creative/Artistic/Design 66 61% 33% 6%
How Did Students Rate their Learning on Core Competencies?
Do Students’ report of learning meet our expectations?
Pages 5 and 6
Raw Average
Adj.Average
# ofClasses
ThisReport
3.9 3.9 11
Institution 4.2 4.2 3,963
IDEA System
4.0 4.0 31,991
Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends)
How do students rate their learning?
Page 3 Part 1: Distribution of Converted Scores Compared to the IDEA Database
Overall Progress Ratings (Courses)
Page 3 Percent of Classes at or Above the IDEA database Average
Overall Progress Ratings (Courses)
Part 3: Percent of Classes at or Above This Institution’s Average
Page 4
Which teaching methods might we use to improve learning?
Page 7
Teaching Methods and Styles
Stimulating Student Interest # Classes Av. s.d.
15. Inspired students to set and achieve goals which really challenged them
42 3.8 0.5
Assessing the QEP for UABUAB’s Core Competencies
IDEA Learning Goals IDEA Teaching Methods Associated with Progress on Learning Goal
Communication8 Developing skill in expressing myself orally or in writing11 Learning to analyze and critically evaluate ideas, arguments, and points of view
2, 7, 8, 15, 16, 18, 19
Knowledge1 Gaining factual knowledge (terminology, classifications, methods, trends)2 Learning fundamental principles, generalizations, or theories4 Developing specific skills, competencies, and points of view needed by professionals in the field most closely related to this field7 Gaining a broader understanding and appreciation of intellectual and cultural activity (music, science, literature, etc.)
1, 2, 4, 6, 7, 8, 10, 12, 13, 15, 19
Problem solving3 Learning to apply course material (to improve thinking, problem solving, and decisions) 9 Learning how to find and use resources for answering questions or solving problems11 Learning to analyze and critically evaluate ideas, arguments, and points of view
2, 7, 8, 9, 13, 14, 15, 18, 19
Citizenship10 Developing a clearer understanding of, and commitment to, personal values12 Acquiring an interest in learning more by asking my own questions and seeking answers
1, 2, 4, 6, 7, 8, 11,13, 15, 16, 18,
Relationship
of Learning Objectives to
Teaching
Methods
How do students view course work demands?
Page 8B Student Ratings of Course Characteristics
Diagnostic Form Item # & Item Average % Classes Below 3.0
% Classes 4.0 or Above
33. Amount of Reading
This Report 3.4 21% 24%
Institution 3.3 31% 19%
IDEA System 3.2 33% 15%
34. Amount of work in other (non-reading) assignments
This Report 3.3 24% 10%
Institution 3.4 23% 20%
IDEA System 3.4 21% 18%
35. Difficulty of subject matter
This Report 3.2 19% 0%
Institution 3.5 13% 19%
IDEA System 3.4 20% 18%
Aggregate Data File
Allows you to •Use Excel Spreadsheet•Use with SAS or SPSS•Ask other types of questions•Display data in different ways
A. Progress of Relevant Objectives-Converted Score Category
Expected Distribution
Fall 2010 Raw
Spring 2011 Raw
Fall 2011 Raw
Much Higher (63 or higher) 10% 6% 7% 7%Higher (56-62) 20% 32% 34% 31%Similar (45-55) 40% 46% 46% 44%Lower (38-44) 20% 11% 9% 10%Much Lower (37 or lower) 10% 5% 5% 8%
B. Excellence of Teacher-Converted Score CategoryExpected Distribution
Fall 2010 Raw
Spring 2011 Raw
Fall 2011 Raw
Much Higher (63 or higher) 10% 0% 0% 0%Higher (56-62) 20% 35% 38% 34%Similar (45-55) 40% 42% 43% 40%Lower (38-44) 20% 12% 11% 12%Much Lower (37 or lower) 10% 11% 9% 14%
C. Excellence of Course-Converted Score CategoryExpected
Distribution
Fall 2010 Raw
Spring 2011 Raw
Fall 2011 Raw
Much Higher (63 or higher) 10% 9% 11% 8%Higher (56-62) 20% 26% 26% 29%Similar (45-55) 40% 41% 40% 40%Lower (38-44) 20% 14% 13% 13%Much Lower (37 or lower) 10% 10% 10% 10%
Summary Evaluation (Average of A, B, C)-Converted Score Category
Expected Distribution
Fall 2010
Spring 2011
Fall 2011
Much Higher (63 or higher) 10% 5% 4% 4%Higher (56-62) 20% 33% 35% 33%Similar (45-55) 40% 45% 45% 44%Lower (38-44) 20% 10% 10% 10%Much Lower (37 or lower) 10% 7% 6% 9%
Primary Instructional Approaches
Fall 2010
Primary
Spring 2011
Primary
Fall 2011
PrimaryLecture 61% 60% 61%Discussion/Recitation 7% 8% 10%Seminar 7% 9% 9%Skill/Activity 7% 6% 7%Laboratory 8% 9% 9%Field Experience 1% 1% 1%Studio 2% 3% 2%Multi−Media 0% 0% 0%Practicum/Clinic 0% 0% 0%Other/Not Indicated 6% 4% 2%
Number Classes Rating 787 677 859
Secondary Instructional Approaches
Fall 2010
Secon-dary
Spring 2011
Secon-dary
Fall 2011
Secon-dary
Lecture 16% 13% 16%Discussion/Recitation 30% 27% 30%Seminar 2% 3% 4%Skill/Activity 11% 13% 13%Laboratory 3% 6% 4%Field Experience 2% 3% 2%Studio 0% 0% 1%Multi−Media 2% 3% 3%Practicum/Clinic 1% 1% 1%Other/Not Indicated 34% 31% 28%
Number Classes Rating 787 677 859
Course Emphases: Writing
Fall 2010
Spring 2011
Fall 2011
None or Little 25% 26% 22%Some 46% 45% 44%Much 29% 30% 34%
0%
20%
40%
60%
80%
100%
Fall 2010 Spring 2011 Fall 2011
None or Little
Some
Much
Course Emphases: Oral communication
Fall 2010
Spring 2011
Fall 2011
None or Little 36% 35% 32%Some 47% 47% 44%Much 17% 18% 24%
0%
20%
40%
60%
80%
100%
Fall 2010 Spring 2011 Fall 2011
None or Little
Some
Much
Instructors’ Reports on Course Emphases: Selected Pairings-Writing and Oral Communication
Course Emphases: Critical thinking
Fall 2010
Spring 2011
Fall 2011
None or Little 19% 19% 18%Some 49% 44% 41%Much 32% 37% 41%
0%
20%
40%
60%
80%
100%
Fall 2010 Spring 2011 Fall 2011
None or Little
Some
Much
Course Emphases: Writing
Fall 2010
Spring 2011
Fall 2011
None or Little 25% 26% 22%Some 46% 45% 44%Much 29% 30% 34%
0%
20%
40%
60%
80%
100%
Fall 2010 Spring 2011 Fall 2011
None or Little
Some
Much
Instructors’ Reports on Course Emphases: Selected Pairings-Critical Thinking & Writing
Highlights for Sample U• Remarkably similar profiles
across terms
• Overall response rates ranged from 66% to 80%
• 1st term in which administration was primarily online achieved a 75% response rate
• Transition from paper to online (fall 2009 to fall 2010) does not show major differences in profiles
• Sample U faculty focus on 4-5 outcomes as essential/important
• Over the last 3 terms, a significant increase on several objectives has been observed: application of course material, oral and written communication skills, & analysis and critical thinking skills (objectives 3, 8, & 11, respectively)
BenchmarkingInstitutional and Discipline Reports
Benchmarking Reports
Comparison to•6-10 Peers•Same Carnegie Classification•IDEA database
Comparison Groups
Your University ---------- ----------
Peer* ----------------------------
Carnegie ---------- ----------
National ---------- ----------
* Peer group is based on 6-10 institutions identified by your institution
Benchmarking Reports
The student, rather than the class, is the unit of analysis
Percentage of positive ratings is given rather than averages
Response Rates
Your University: Student participation is similar to that of each comparison group
Your University=79%
Peer=77%
Carnegie=79%
National=75%
Students’ Perceptions
Graduate Students
Instructional Objectives Selected by Instructors
Instructors’ Intentions/ focus Students’ Self-Reported Progress on Learning
Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)
% of total classes where instructor selected objective as “Essential” or “Important”
IDEA Objective 3
Learning to apply course material (to improve thinking, problem solving, and decisions)
IDEA Objective 8
Objective 8: Developing skill in expressing oneself orally or in writing
% of total classes where instructor selected objective as “Essential” or “Important”
Developing skill in expressing oneself orally or in writing
Teaching Methods and Styles Reported by Students
(Diagnostic Forms Only)
• Fostering Student Collaboration
• Encouraging Student Involvement
Benchmarking Look at “General Education”
Benchmarking Look at “General Education”
Using Aggregate Data for Assessment
UAB
Core
Competencies
Program Learning
Outcomes
Program Learning
Outcomes
Program Learning
OutcomesGroup Summary
Report
Group SummaryReport,
Include Extra Questions
Benchmarking: One Year or 3-5 Year Trend Report
Benchmarking: Discipline Report
Course Learning Outcomes
Course Learning Outcomes
Course Learning
Outcomes
Course Learning Outcomes
Course Learning
Outcomes
Accreditation
Guides• SACS
• NCATE
• CACREP
Questions?