using idea for faculty evaluation
DESCRIPTION
Using IDEA for Faculty Evaluation. Shelley A. Chapman, PhD. Texas A & M University February 2013. Plan for this Session. “Teaching Effectiveness” What it is Uniqueness of IDEA Conditions for the Good Use of IDEA 3-Phase Process for Faculty Evaluation Using Reports to Improve Teaching. - PowerPoint PPT PresentationTRANSCRIPT
Using IDEA for Faculty Evaluation
Shelley A. Chapman, PhDTexas A & M University February 2013
Plan for this Session “Teaching Effectiveness” What it is Uniqueness of IDEA Conditions for the Good Use of IDEA 3-Phase Process for Faculty
Evaluation Using Reports to Improve Teaching
Most SurveysHow well do the instructor’s methods resemble those of a “model” teacher?
How well do students rate their progress on the types of learning the instructor targeted?
Teaching Effectiveness
What makes IDEA unique?1. Focus on Student
Learning 2. Focus on Instructor’s
Purpose3. Adjustments for
Extraneous Influences4. Validity and Reliability5. Comparison Data6. Flexibility
Conditions for Good Use
The instrument o Focuses on learning o Provides suggested
action steps
Conditions for Good Use
The Facultyo Trust the processo Value student
feedbacko Are motivated to
make improvements
Conditions for Good Use
Campus Cultureo Teaching excellence - high
priorityo Resources to improve -
providedo Student ratings -
appropriate weight
Conditions for Good Use
The Evaluation Processo 30-50% of evaluation of teachingo 6-8 classes, more if small (<10)oNot over-interpreted
(3-5 performance categories)
Underlying Philosophy of IDEA
Teaching effectiveness is determined primarily by students’ progress on the types of learning the instructor targets.
Faculty Information Form
Diagnostic Report Overview
1. How did students rate their learning experience?
2. What contextual factors impacted those ratings?
3. How do my scores compare to: IDEA, discipline, and institution?
4. What might I do to facilitate better learning for my students next time?
Wisdom
Knowledge
InformationData
What the Report Can Provide
Calculation of Scores
Context: Variables and Comparisons
Suggested Action Steps
Using IDEA
As Part of a Faculty Evaluation Process
Multiple forms of Assessment
Student Ratings
External PerspectiveArtifacts
BalancedPlan for SummativeEvaluation
Evidence of Good Teaching
Artifacts
• Syllabi• Graphic Organizers• Assignments and project
descriptions• Rubrics• Written Teaching
Philosophy/Reflections• Samples of Student Work• CATs and results
Evidence of Good Teaching
• Classroom Observation• Classroom Visitation• Invited Presentations• Alumni Surveys• Focus Groups of
Graduating Students
External Perspectiv
e
Classroom ObservationsTime What Happened What Was Said
Classroom ObservationsTime What Happened What Was Said
8:05
8:10
8:15
Instructor shut door
Students are shuffling papers, opening books.
Student comes in late
Several students raise hands
Female in first row is called on
Instructor (I): OK, Class. Let’s begin. Make sure you turned in your homework as you came in.
Today we will begin our discussion on the brain. Turn in your textbooks to chapter 5.
Is your brain more like a computer or a jungle? Who would like to respond first?
Student (S) My brain is a jungle! I am so unorganized! (class laughs)…
Flow of Communication MapInstructor
M
F
F
F
M M M M M
M M
M
F F F F
F
F
F
F F
Evidence of Good Teaching
• Administer Appropriately• Collect 6-8 Reports (more if class size is <10)• 30-50% of Overall Evaluation
Student
Ratings
Evidence of Good Teaching
• Student Comments-formative • Be mindful of standard error of
measurement (±.3)• Use 3-5 Performance Categories
Student
Ratings
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
I. Set ExpectationsWhat does this entail regarding IDEA?
Criterion-Referenced Standards
Criteriono Use averages on 5-
point scaleo Recognize that
some objectives are more difficult to achieve
o “Authenticate” objectives
Page 1
Norm-Referenced Standards
Use Converted Averages
o IDEAo Disciplineo Institutio
n
Comparison Information: Converted Averages
T-Score Distribution
40%
Similar
Lower20%10%
Much Lower
10%Much Higher
20%Higher
Gray Band
Comparison Scores
Create Categories of Performance
Below Acceptable Standards
Marginal, Needs Improvement
MeetsExpectations
ExceedsExpectations
Outstanding
Does Not Meet Expectations Meets Expectations Exceeds Expectations
Performance Categories: EXAMPLECriterionAverage Rating
Effectiveness Category
NormativeT-Score
Below 3.0 Below acceptable standards Below 38
3.0-3.4 Marginal, improvement needed
38-44
3.5-3.9 Meets expectations 45-54
4.0-4.4 Exceeds expectations 55-62
4.5 or higher Outstanding 63 or higher
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
II. Collect DataWhat do you look for regarding IDEA?
Improving Online Response Rates
• Create value for student feedback
• Monitor and Communicate through multiple modalities:• Twitter• Facebook• Other
• Prepare Students• Talk about it• Syllabus
Example: Course Syllabus
Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)
Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomenaStudents will think more critically about the earth and environment
Objective 8: Developing skill in expressing myself orally or in writing
Students will be able to present scientific results in written and oral forms
IDEA Center Learning Objective
Course Learning Outcomes
For Personnel Decisions
Pages 1 and 2 What were students’ perceptions of the
course and their learning?
Things toConsider…
Were the appropriate objectives selected?
• How many?• Do they match the course?
• How might you “authenticate” the objectives selected?
FIF: Selecting Objectives• 3-5 as “Essential” or
“Important”o Is it a significant part of the
course?o Do you do something
specific to help students accomplish the objective?
o Does the student’s progress on the objective influence his or her grade?Be true to your course.
Things to Consider…
What were the students’ perceptions of their course and their learning?
How Did Students Rate Their Learning?
Your Average (5-point Scale)
Raw Adj.
A. Progress on Relevant Objectives1
Four objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.
How Did Students Rate Their Learning?
ProgressOnRelevantObjectives
4
4.3 + 4.34.14.23.6 5
Summary Evaluation: Five-Point Scale
Report Page 1
Your Average Score
(5-point scale)
Raw Adj.
A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
Overall Ratings B. Excellent Teacher 4.7 4.9 C. Excellent Course 4.1 4.4
D. Average of B & C 4.4 4.7
Summary Evaluation(Average of A & D) 4.3 4.5
50%
25%25%
Understanding Adjusted Scores
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38 4.28 4.13 4.04
High Avg. 4.38 4.29 4.14 3.96 3.76
Average 4.28 4.14 4.01 3.83 3.64
Low Avg. 4.15 4.05 3.88 3.70 3.51
Low 4.11 3.96 3.78 3.58 3.38
Impact of Extraneous Factors
Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38
High Avg. 4.38 4.29
Average 4.01
Low Avg. 3.70 3.51
Low 3.58 3.38
Impact of Extraneous Factors
Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
Raw or Adjusted Scores
Purpose Raw or Adjusted?How much did students learn? Raw
What were the instructor’s contributions to learning?
Adjusted
How do faculty compare?
Adjusted
Doraw
scores meet or exceed
Expectations?*
Are adjusted
scores lower or higher than
rawscores?
Use adjusted
scoresUse
raw scores
Lower Yes
Higher
When to Use Adjusted Scores
*Expectations defined by your unit.
No
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
III. Use DataWhich data will you use and how?
.
IDEA Faculty Worksheet
Keep track of reports Look for longitudinal trends Use for promotion and tenure
Created by Pam Milloy, Grand View University
Available from The IDEA Center Website
Using the DataSummative (pp.1-
2) Criterion or Norm-
referenced Adjusted or raw Categories of
Performance 30-50% of Teaching
Evaluation 6-8 Classes (more if
small)
Formative (p.3) Identify areas to
improve Access applicable
resources from IDEA website
Read and have conversations
Implement new ideas
Reflective Practice
Collect Feedback
Interpret Results
Read & Learn
Reflect & Discuss
Improve
IDEA resources that are keyed to reports
Talk with colleagues
Try new ideas Online, Paper
What the reports sayand what they mean
Research
Relationship of Learning Objectives to Teaching
Methods
Information Improvement
Information Improvement
Questions?