Evaluation:It’s Not Just ‘at the End’

Terry Uyeki, MSEdDirector of Evaluation & Community Services

Terry.Uyeki@humboldt.edu

Program Evaluation Symposium, Sept. 10, 2010

CCRP “Hats”

Program evaluation

Meeting design & facilitation (including graphic facilitation)

Qualitative data analysis

Community based participatory research (CBPR)

EvaluationEvaluationIt’s not just for the report for the funder …It’s not just for the report for the funder …

and it doesn’t just happen at the end…and it doesn’t just happen at the end…

Evaluation is the “systematic investigation of the worth Evaluation is the “systematic investigation of the worth or merit of an object”or merit of an object”

Why would you develop anWhy would you develop anevaluation plan when you design evaluation plan when you design

a program or intervention?a program or intervention?

(Who is evaluation for?)(Who is evaluation for?)

Why?Why?

• Formative/process evaluationFormative/process evaluation– How can we improve as we develop?How can we improve as we develop?

• Summative/outcome evaluationSummative/outcome evaluation– What happened? How effective was it?What happened? How effective was it?

Project / Program Evaluation

Thinking about evaluation (benchmarks) as you develop a project

Program / Project

Design & Delivery

Program / Project

Outcomes for

Participants

Program / Project

Dissemination

Process Evaluation (what did you do?)

Outcome Evaluation (what did they do?)

Impact Evaluation (how did it affect the problem?)

Some Models for Program Design• W.K. Kellogg Foundation Logic ModelW.K. Kellogg Foundation Logic Model

– A systematic and visual way to represent the resources and A systematic and visual way to represent the resources and activities of your program and proposed resultsactivities of your program and proposed results

• Aspen Institute Theory of Change Outcome Aspen Institute Theory of Change Outcome FrameworkFramework– Specifying outcomes and assumptions, and backwards-Specifying outcomes and assumptions, and backwards-

mapping to connect outcomes to your activitiesmapping to connect outcomes to your activities

• Intervention Mapping (Bartholomew et al., 2000)Intervention Mapping (Bartholomew et al., 2000)– Health Promotion program planning: Design, Health Promotion program planning: Design,

adoption, implementation & evaluationadoption, implementation & evaluation

Kellogg Foundation Logic Model

Logic Model: Family Visitation Program

Healthy weight in children & families; reduced incidence of Type 2 Dm.

40 families participate. Enrolled families complete 10 PACT visits.

Increased levels of physical activity & increased fruit & veg. consumption. Positive change in mediators.

PACT program publicized. Family advisors plan, conduct visits. Annual clinics held. Group activities held.

Family advisors hired, trained as coaches. Portfolio of fun activities compiled.

Making Project Objectives SMART

How can evaluation help shape project objectives?

• S Specific

• M Measurable

• A Attainable, Actionable

• R Relevant, Results-focused

• T Time framed

Making Project Objectives SMARTHow can evaluation help shape

project objectives?• S Specific • M Measurable • A Attainable,

Actionable• R Relevant, Results-

focused• T Time framed

From “PACT Program participants will eat healthier” to

At least 60% of PACT Program participants will increase consumption of fruits and vegetables.

Implications for Evaluation:•Self-reported meal recall•Weight of reported fruits and vegetables consumed•Perceived mediators in preparation of meals and consumption of fruits and vegetables•Participation rates of program families•Satisfaction with PACT family visits•Competence of family advisors (adherence to visit protocol; assessment of adults relative to stage of change; coaching skills)

Logic Model: Family Visitation Program

Healthy weight in children & families; reduced incidence of Type 2 Dm.

40 families participate. Enrolled families complete 10 PACT visits.

Increased levels of physical activity & increased fruit & veg. consumption. Positive change in mediators.

PACT program publicized. Family advisors plan, conduct visits. Annual clinics held. Group activities held.

Family advisors hired, trained as coaches. Portfolio of fun activities compiled.

Aspen Institute Theory of Change Aspen Institute Theory of Change Outcome FrameworkOutcome Framework

• A graphic representation of the change process, mapping the pathways of change that will be brought about by an intervention/program

• From Outcomes (long-term goals and assumptions behind them)

• Backwards mapping to pre-conditions required to cause the desired change.

• Connected outcomes = pathways of change

Project “Superwomen” as a Logic Model

Project “Superwomen” as a Theory of Change Outcome Framework

From “Intervention Mapping”: DefiningPerformance Objectives for Health Behavior

• Negotiate the use of a condom with a partner (based on negotiation theory, Fisher & Ury, 1991)– State mutual goals such as prevention of

pregnancy or AIDS– State clearly the intention of using a condom as a

prerequisite for intercourse– Listen to partner’s concerns– Pose solutions to partner’s concerns that reference

mutual goals & personal requirements

Why Build Evaluation Plan as Part of Program/Intervention Design?

• Not just for the funder…

• Keeps objectives realistic

• Ensures that objectives are measurable

• Helps with QA/QC

• Enables one to make program adjustments

• What happens if you did wait until the end?

• Other reasons?

Contact us at

California Center for Rural PolicyCalifornia Center for Rural Policy

Humboldt State UniversityHumboldt State University

707-826-3400 707-826-3400 ccrp@humboldt.edu

www.humboldt.edu/~ccrp/www.humboldt.edu/~ccrp/