Innovative Educators WebinarDecember 1, 2010

Jan Norton, Presenter

TerminologyThe Black BoxTimetablesData/MeasurementData/Measurement CautionsProgram Outcomes & ExamplesStudent Outcomes & ExamplesClosing & Questions

VisionsMissionsValuesGoalsObjectivesOutcomesProgram vs. Student OutcomesLearning Outcomes

.

Something

Happens

Actions leading to specific, observable results

What students can expect to learn How students or programs will be

different as a result of what happens in the box

What knowledge, skills, and abilities will be improved

What measurable changes occur in tutoring programs

Scope of assessment ≥ 6 years 2 years 1 year 1 semester 1 course 1 assignment 1 tutoring session

Planning for Assessment Continual processes Multiple times per semester Once or twice per semester Two or three times per year Once per year Every two years Every 4 years

Why Measure

What to measure Before (Benchmarks) During After (Outcomes)

How to measure Program-generated data Course-specific data Campus data Campus surveys Standardized tools Locally developed tools

More about locally developed tools Surveys Likert-type scales Explicated scales Behavioral rubrics Self assessment by student Tutor assessment Observer assessment

Direct vs. Indirect MeasuresData access and consistencyRealistic limitationsCooperationThe Baseline ChallengeTime & money

(and staff and comfort and ……)

Possible Areas of Assessment Focus Establishing baselines Usage Satisfaction or quality Level/variety of services Staff development Reaching certification or meeting

standards

Outcome Statement As a result of the new marketing strategy,

the number of students attending group tutoring sessions will increase by 10% in Spring 2011.

Timetable One semester

Starting Measure (Benchmark) Average of past attendance

Ending Measure Workshop attendance records

Outcome Statement During 2010-11, the number of tutors who

earn CRLA Level I certification will increase by 5%.

Timetable One academic year

Starting Measure (Benchmark) Number of tutors achieving Level I in previous

year(s) Ending Measure

Number of tutors achieving Level I in 2010-11

Outcome Statement Outside reviewers using the NADE self-

study process will score the tutoring program at 85% or above on all sections.

Timetable One year

Starting Measure (Benchmark) Internal review & self-study

Ending Measure External review

Outcome Statement As a result of tutor training sessions, tutors

will demonstrate an improved understanding of Bloom’s Taxonomy.

Timetable One semester or year, depending on

training schedule Starting Measure (Benchmark)

Pre-test Ending Measure

Post-test

TYPES

Affective Build confidence Improve attitude Reduce anxiety

Cognitive Mastery of material Conscious strategies

use

MEASURES

Qualitative Interviews & focus groups Opinion surveys Observations or self

reports

Quantitative Assignment & course

grades Instrument scores Observation rubrics

Outcome Statement After being tutored for at least one hour,

clients will be more confident in their ability to accomplish the academic task they focused on in tutoring.

Timetable One tutoring session

Starting Measure (Benchmark) Client self-report

Ending Measure Client self report

Outcome Statement Students who work with the study skills

tutors will demonstrate an increased ability to use memorization strategies.

Timetable One tutoring session

Starting Measure (Benchmark) Tutor or observer checklist

Ending Measure Tutor or observer checklist

Outcome Statement 75% of the students who work with a

reading tutor for at least two sessions will score at least 75% on the main ideas reading test.

Timetable Within three weeks

Starting Measure (Benchmark) Reading quiz at start of tutoring

Ending Measure Chapter reading test in class

Outcome Statement Students who attend at least 10 group

tutoring sessions for Algebra Review 090 will earn an average grade of B or above in the course.

Timetable One semester, length of course

Starting Measure (Benchmark) Placement score into course

Ending Measure Course grade

Outcome Statement After attending mandatory tutoring for all 5

of their developmental education courses, students will be retained until the start of their junior year at a rate at least 25% higher than developmental students for whom tutoring was not required.

Timetable Two years

Starting Measure (Benchmark) Developmental course placement

Ending Measure Enrollment level

Outcome Statement Students attending SI for World Religions

will “[apply] previously understood information, concepts, and experiences to a new situation or setting.” (CAS)

Timetable Within one semester

Starting Measure (Benchmark) Pre-survey

Ending Measure Post-survey

Check with your institutional research or assessment office

Review early drafts of outcomes statements with your supervisor

Pre-test any locally created measurement tool

Be prepared for outcomes that are not met or wildly exceeded

Remember the black box: we do not own or control the box, and our ability to see into it is very limited