Program Evaluation

Spero Manson PhD

Program evaluation is a careful investigation of a program’s characteristics and merits.

Its purpose is to provide information about the effectiveness of an activity or set of activities in order to optimize the outcomes, efficiency, and quality of health care.

Evaluation examines a program’s structure, activities, organization, as well as broader social environment.

Evaluation also appraises the achievement of a program’s goals and objectives, and extent of its impact as well as cost.

We study, we plan, we research. And yet, somehow evaluation remains as much an art as a science.

The Basic Steps in Program Evaluation

Describe the program or activities to evaluate Define goals and objectives Create a Logic Model Pose the evaluation question(s) Select the methods - information needed to answer

the question Gather the information Report on and learn from results

Describe the Program or Activities to Evaluate

Need clarity as to the program and activities that are the focus of evaluation

Must be able to distinguish the program from the broader environment in which it operates

Premium placed on specifying the nature of the activities: who, what, where, when, how

Define the Program’s Goals and Objectives

Goals: a broad general statement of what the program hopes to accomplish

Objectives: measurable, time-specific out-comes that are expected to be achieved as a result of the program or activities

Define the Program’s Goals and Objectives

Establishing measurable, time-specific outcomes is a major key to an evaluation’s credibility

The more specific they are, the easier they are to measure

The absence of uniformly accepted definitions or levels of performance introduces ambiguity – the bane of program evaluation!

Define the Program’s Goals and Objectives

Example: Goal: Reduce the risk of diabetes in youth by

increasing their participation in physical activity Objective: Increase the number of students

participating in physical activity in the wellness center by 50% during the next school year

Create a Logic Model

Overall description of program activities and potential outcomes, impacts

Useful to planning evaluation, defining questions, determining information that you can measure

Logic Model Components

Context Influences, circumstances, resources, stakeholders

Program goals, objectives

Outputs, Activities – what will be done or result

Outcomes –

Potential measurable data, information

Impacts –

Short and long term benefits

Logic Model ExampleContext Influences, circumstances, resources, stakeholders

Program goals, objectives

Outputs, Activities – what will be done

Outcomes –

Potential measurable data, information

Impacts –

Short and long term benefits

Tribe, school, health program,

Move it! Grant resources

Summer Camp

Provide Nutrition education

Increase physical activity

• Daily walks

• Introduce new activities

Prepare meals Classes on

nutrition

• Increase participation, distance, # steps per day

• Knowledge on nutrition, fitness

• Awareness of their risk for diabetes

• Weight loss

• Decreased obesity

• Prevention of diabetes

Evaluation

Art as well as Science

They’re harmless when they’re alone, but get a bunch of them together to evaluate a program … watch out!!

Pose the Evaluation Question(s)

Most important stepDetermines the methods of the evaluationCan have more than one questionPose after reviewing the logic model and

can see possible questions

Types of Evaluation Questions

Process evaluation questionsDid we do what we said we would do?Documentation of activities, review

Outcome evaluation questionsDid we achieve any short or long term outcomes as a result of our program?

Typical Evaluation Questions

To what extent did the program achieve its goals and objectives?

What are the characteristics of the individuals and groups who participated in the program?

For which individuals or groups was the program most effective?

How enduring were the effects?

Typical Evaluation Questions

Which features (e.g., activities, settings, care strategies) of the program were most effective?

How applicable are the program’s objectives and activities to other participants in other settings?

What are the relationships among the costs of the program and its effects?

To what extent did changes in social, political, and/or financial circumstances influences the program’s support and outcomes?

Evaluation Questions: ExamplesProcess evaluation:

Did the school implement the Move It! Campaign activities during the summer school session?

Outcome evaluation: Did student awareness of their risk for diabetes

increase after the summer school session? Did at least ½ of the student population participate in

physical activity each day? What percent of students experienced weight loss

during the summer school session?

Select the Methods What you need to do to answer your evaluation

questions – depends on the question! Various methods

Pre/post test on knowledge, attitudesTrack participation rates in activitiesTrack clinical parameters – weight, heart rate, blood

glucoseEvaluate satisfaction with activities

Always consider a comparison if possibleBefore vs. after comparisonsParticipants vs. non-participants

Evaluation Designs

Evaluations with concurrent controls in which participants are randomly assigned to groups

Benefits: If properly conducted, can establish the extent to which a program caused outcomes

Concerns: More difficult to implement, logistically and methodologically

Evaluation DesignsEvaluations with concurrent controls in which participants are not randomly assigned to groups

Benefits: Easier to implement

Concerns: A wide range of potential biases may occur because, without an equal chance of selection, participants in the program may be systematically different from those in the control. Also the 2 groups in the evaluation may be systematically different than other, nonparticipating groups.

Evaluation DesignsEvaluations with self-controls. Require pre-/post-measures and often are referred to as longitudinal evaluations or before-and-after designs

Benefits: Relatively easy to implement logistically. Provides data on change and improvement

Concerns: Must be certain that measurements are appropriately timed. Without a control group, cannot tell if seemingly program effects are also present in other, nonparticipants

Evaluation Designs

Evaluations with historical controls use data collected from participants in other evaluations

Benefits: Easy to implement, unobtrusive

Concerns: Must be certain that “normative” comparisons are applicable to participants in the evaluation.

Threats to Validity of Evaluation

Maturation – as a part of normal human development, individuals mature intellect-ually, emotionally, and socially. This new maturity may be as important as the program in producing change.

Threats to Validity of Evaluation

History – historical events may occur that can bias results or produce similar changes as those intended by the program, e.g., new educational campaigns that encourage the community at large to change its behavior, change in the structure and financing of health care, etc.

Threats to Validity of Evaluation

Instrumentation – unless the measures or tools used to collect the data are dependable, one cannot be confident that the data are accurate.

Threats to Validity of Evaluation

Attrition – the participants who remain in a program may be, indeed often are different from those who drop-out.

Gather information Set up a process for gathering the

information Create forms, tracking sheets, surveys Plan for how you will analyze or review the

information when it is doneHand counts, tallysSpreadsheets, Statistical DatabasesCommon themes

Questions to Ask in Choosing a Data Source

What variable, constructs, concepts need to be measured?

Are they sufficiently well defined to be measured? Can I borrow or adapt a currently available

measure, or must a new measure be created? If an available measure seems appropriate, has it

be used in circumstances similar to the current evaluation?

Questions to Ask in Choosing a Data Source

Do I have the technical skills, financial resources, and time to create a valid measure?

Do I have the technical skills, financial resources, and time to collect information with the chosen measure?

Are participants likely to be able to fill out the forms, answer the questions, and provide the information called for by the measure?

Questions to Ask in Choosing a Data Source

When medical and other confidential records are relevant data sources, are these records likely to:

Contain the necessary information?Be complete?Be timely?Be reliable?Be accessible?

Questions to Ask in Choosing a Data Source

To what extent will users of the evaluation’s results have confidence in the nature of the information, the manner in which it was collected, and its sources?

Report on and Learn from Results

Most commonly forgotten step Communicate results to stakeholders Review results – lessons learned Develop a plan to respond to results, future activities

to address Communicate with parents, community Include in grant progress report! Publish in appropriate scientific forums to established

best practices specific to our communties