1

Evaluation is a Evaluation is a PartnershipPartnership

Dr. Darla M. CooperDr. Darla M. CooperAssociate DirectorAssociate Director

Center for Student SuccessCenter for Student Success

Basic Skills Coordinators WorkshopBasic Skills Coordinators WorkshopSeptember 17, 2009September 17, 2009

2

OverviewOverview Importance of EvaluationImportance of Evaluation

Balance between Reality and RigorBalance between Reality and Rigor

Direct and Indirect MeasuresDirect and Indirect Measures

Relationship between Faculty and Relationship between Faculty and ResearchersResearchers How Researchers Can Help FacultyHow Researchers Can Help Faculty What Researchers Need from FacultyWhat Researchers Need from Faculty

Lessons LearnedLessons Learned

3

Importance of EvaluationImportance of Evaluation

Demonstrates whether program/project Demonstrates whether program/project is having desired impact on studentsis having desired impact on students

Identifies what is working and what Identifies what is working and what needs improvementneeds improvement

Measures effect of any changes made Measures effect of any changes made within program/projectwithin program/project

Enables ongoing internal and external Enables ongoing internal and external sharing/reporting of evaluative resultssharing/reporting of evaluative results

Helps justify continued support and Helps justify continued support and fundingfunding

4

Balance between Reality Balance between Reality and Rigorand Rigor

First identify data already being collectedFirst identify data already being collected Data collection should not place an undue Data collection should not place an undue

burden on the program/projectburden on the program/project Use direct measures whenever possible and Use direct measures whenever possible and

reasonablereasonable Need to ensure that data being collected are Need to ensure that data being collected are

actually measuring what you intended to actually measuring what you intended to assessassess

Requires conversation between Requires conversation between program/project leaders and researcher to program/project leaders and researcher to achieve a suitable balanceachieve a suitable balance

5

Direct and Indirect Direct and Indirect MeasuresMeasures

Direct MeasuresDirect Measures Provide evidence of cognitive (knowledge) or Provide evidence of cognitive (knowledge) or

behavioral (skills) learning that directly behavioral (skills) learning that directly corresponds to specific intended learning corresponds to specific intended learning outcomesoutcomes

Examples: exams, papers, grades, portfoliosExamples: exams, papers, grades, portfolios

Indirect MeasuresIndirect Measures Assess whether learning has been meaningful Assess whether learning has been meaningful

by gathering information related to by gathering information related to perceptions, opinions, experiences, and perceptions, opinions, experiences, and achievementsachievements

Examples: surveys, journals, graduation ratesExamples: surveys, journals, graduation rates

6

Direct and Indirect Direct and Indirect MeasuresMeasures

DirectDirect IndirectIndirect

Measures actual Measures actual learninglearning

YesYes NoNo

Gathering informationGathering informationMore More

challenginchallengingg

More More accessibleaccessible

ObjectivityObjectivity More More objectiveobjective

More More subjectivesubjective

Reflects factual Reflects factual informationinformation

More More factsfacts

More More beliefsbeliefs

Insight into learning Insight into learning experienceexperience

Less likelyLess likely More More likelylikely

7

Relationship between Relationship between Faculty and ResearchersFaculty and Researchers

Researchers need to understand the Researchers need to understand the program/projectprogram/project

Project/program leaders need to Project/program leaders need to understand demands on researcher understand demands on researcher

Develop Evaluation Plan togetherDevelop Evaluation Plan together Researcher is seen as a member of Researcher is seen as a member of

the teamthe team Ongoing relationship is key to ongoing Ongoing relationship is key to ongoing

success of the evaluationsuccess of the evaluation

8

How Researchers Can Help How Researchers Can Help FacultyFaculty

Provide options for assessment Provide options for assessment methodsmethods

Share knowledge of data already Share knowledge of data already availableavailable

Facilitate accurate data interpretation Facilitate accurate data interpretation ListenListen NOT dictate data to be usedNOT dictate data to be used NOT advocate changing NOT advocate changing

program/project to fit dataprogram/project to fit data

9

What Researchers Need What Researchers Need from Facultyfrom Faculty

To be invited and included in the To be invited and included in the conversation earlyconversation early

Share details of the project/program (e.g., Share details of the project/program (e.g., goals, data, interventions, intended goals, data, interventions, intended outcomes)outcomes)

Consider options provided by researcher and Consider options provided by researcher and provide constructive feedbackprovide constructive feedback

To be kept involved and informed about To be kept involved and informed about ongoing progress and changes in the ongoing progress and changes in the program/projectprogram/project

Work with researcher to make the data Work with researcher to make the data meaningful and usefulmeaningful and useful

Patience and understandingPatience and understanding

10

ExerciseExercise

Common types of Common types of programs/projects:programs/projects:

Learning CommunitiesLearning Communities

TutoringTutoring

Supplemental InstructionSupplemental Instruction

First-Year ExperienceFirst-Year Experience

11

ExerciseExercisePreparing to Meet with Preparing to Meet with

ResearcherResearcherHave answers to the following Have answers to the following

questions:questions:1.1. Goal(s)Goal(s) – What effect do you intend the – What effect do you intend the

program/project to have?program/project to have?2.2. OutcomesOutcomes – What tangible results do you – What tangible results do you

expect to see from students?expect to see from students?3.3. InterventionIntervention – What treatment(s) students – What treatment(s) students

will receive? Where and when are the points will receive? Where and when are the points of contact with students?of contact with students?

4.4. DataData – What data will demonstrate intended – What data will demonstrate intended outcomes? What data are the outcomes? What data are the program/project already collecting or program/project already collecting or planning to collect?planning to collect?

12

ExerciseExercisePreparing to Meet with Preparing to Meet with

ResearcherResearcherExamples:Examples:1.1. Goal(s)Goal(s) – Improve students’ success in – Improve students’ success in

basic skills courses through …?basic skills courses through …?2.2. OutcomesOutcomes –Users have higher success –Users have higher success

rates than non-users; Increased rate of rates than non-users; Increased rate of success to transfer over timesuccess to transfer over time

3.3. InterventionIntervention – one-on-one tutoring, – one-on-one tutoring, phone calls during 1phone calls during 1stst three weeks of three weeks of termterm

4.4. DataData – Course grades, test results, – Course grades, test results, portfolio assessments, surveys, ?portfolio assessments, surveys, ?

13

Case StudyCase Study

SBCC’s Partnership SBCC’s Partnership for Student Successfor Student Success

(What really happens (What really happens when faculty and when faculty and

researchers get together)researchers get together)

14

Lessons LearnedLessons Learned

Start discussion with what you want to Start discussion with what you want to know about program/projectknow about program/project

Decide on data collection BEFORE Decide on data collection BEFORE implementation, if possibleimplementation, if possible

Be flexible, open and availableBe flexible, open and available Be involved, invested and stay Be involved, invested and stay

informedinformed Work together as partnersWork together as partners Data and research are your friends!!!Data and research are your friends!!!

15

Questions?Questions?

16

Thank You!!Thank You!!