program evaluation: simplifying the program evaluation process

Program EvaluationSimplifying The Program Evaluation Process

PhilipAdu,Ph.D.

Methodology Expert

National Center forAcademic & Dissertation Excellence (NCADE)

The Chicago School of Professional Psychology

[email protected]

@drphilipadu

mailto:[email protected]

Outline

What is program evaluation? Learning about the program, stakeholders and beneficiaries Involving stakeholders in the evaluation Examining the program using logic model Determining the evaluation purpose Creating evaluation questions Determining evaluation data sources Deciding on data collection strategies Choosing evaluation method

Impact evaluation Writing evaluation plan

What Is Program Evaluation?

To access the worth of a program by looking at:

1. How the program is being implemented

2. How the goals are being met

3. Whether the goals were met

4. The outcome of the program

5. Impact of the program

Inputs Activities Outputs Outcomes Impact

(Kettner, Moroney & Martin, 2017)

Understanding Program Evaluation:Program Monitoring and Evaluation

Monitoring: Keeping an eye on and documenting mainly program

activities to determine whether they are occurring as

planned

Program evaluation: Monitoring plus making decision about the outcome of the

monitoring (i.e. formative evaluation) Maintaining the current program activities if they are on course

Adjusting the current program activities if they are not happening

as expected Making sure that all program inputs and activities are leading to

meeting the expected outputs, outcome and impact

Evaluation could happen at the end of the program (i.e.

summative evaluation) Assessing whether the program met the set outcomes Determining the impact of the program

Monitoring

Making Decisions & Taking Actions

Evaluation

(Kettner, et. al., 2017)

Learning about the Program,

Stakeholders and Beneficiaries

What What are the program:

Objectives Resources

Activities Hypotheses (if any)

Who Who are the program Stakeholders?

Interests and expectations Organizational and political climate

Who are the program beneficiaries? Needs

When When will (did) the program start?

When will (did) the program end?

Where Where is the program taking place?

Sources of information• Program documents

• Stakeholders

• Beneficiaries

(Alkin & Vo, 2018)



What What are the program:

Goals

Resources

Activities

Hypotheses (if any)

When When will (did) the program start?

When will (did) the program end?

Where Where is the program taking

place?

Problem

Goals Actions

Outcomes

About the Program



Problem to be addressed: Increased incidence of domestic

violence

Problem

GoalsActions

Outcomes

About the ProgramExample




Program goals To reduce domestic violence

incidence by 50% Immediate goals

To attain financial independence

To increase victims’ self-esteem

To establish connections with family

members and community


Problem

GoalsActions

Outcomes


Learning more about the Program,


Actions/Activities/Services Counseling service for victims

Job and financial management

training for victims

Provision of food and housing

Engagement in community

volunteering


Problem

Goals

Actions

Outcomes




Outcomes Immediate

Increased self-esteem

Financial independence

Long-term Increased engagement in the

community Deduction of domestic violence

incidence by 50%


Problem

GoalsActions

Outcomes


Learning more about the Program,


Who

Who are the program

beneficiaries (participants)?

What

What are their needs?

What are their demographics?

About the Beneficiaries

Beneficiaries

Needs

Program GoalsProgram Activities

(Alkin & Vo, 2018)



Who

Who are the program stakeholders?

Those who have some connection with or

interest in the program

Program designers/developers

Program implementers

Program funders

Program custodians

What

What are their needs, concerns and expectations?

What are their demographics?

How How much influence do they have in the

program?

How much interest do they have in the program?

About the Stakeholders

Stakeholders’

Influence

Interests

Expectations

Needs

Pulse and Reflect

Think about designing an evaluation plan

with the USE in mind

Finding out who may use the evaluation

findings

Involving them in the evaluation process

(Patton, 2012)

Involving Stakeholders in the Evaluation

Let them know your background and credentials

Make an agreement with them

Deciding on them

Their roles in the evaluation process

Evaluation purpose, expectation and timeline

Be transparent

Involve them in the decisions of the evaluation

Provide internal report of the evaluation emphasizing on how the findings could be used

(Alkin & Vo, 2018)

Examining the Program Using Logic Model

Adopted from Kettner, et. al. (2017, p. 7)

Determining the Evaluation Purpose

Program

Component

Evaluation Types Purpose of the evaluation

Inputs

Formativeevaluation

Input evaluation To assess the resources needed to implement the program activities or meet the program goals

Activities Process evaluation

To assess the program activities – whether the program is being implemented according to plan

Outputs Output evaluation

To assess immediate results based on the

activities carried out (accounting for actions taken)

Outcomes Summativeevaluation

Outcome evaluation

To assess the benefits of the activities

Impacts Impact evaluation

To assess effect of the program on beneficiaries


Creating Evaluation QuestionsProgram

Component Purpose of the evaluation Research Questions

(Examples)

Input evaluation

To assess the resources needed

to implement the program

activities or meet the program goals

• Are the resources adequate to implement the

program?

• Are the program staff capable of implementing the

program or addressing the needs of the beneficiaries?

Process evaluation

To assess the program activities

– whether the program is being

implemented according to plan

• Are the program being implemented as planned?

Output evaluation

To assess immediate results

based on the activities carried

out (accounting for actions taken)

• How long did it take to provide support to each

beneficiary?

• How many counseling sessions were offered?• How many participants completed all the sessions?

Outcome evaluation

To assess the benefits of the activities

• What have program participants benefited?• What the immediate effect of the program on them?

Impact evaluation

To assess effect of the program on beneficiaries

• Did the program impact participants’ [state specific behavior your are interested in assessing]?

Determining Evaluation Data SourcesProgram

Component Research Questions

(Examples)Data Sources

Input evaluation

• Are the resources adequate to implement the program?

• Are the program staff capable of implementing the program or addressing the needs of the beneficiaries?

Program documents, staff, and stakeholders

Process evaluation

• Are the program being implemented as planned? Program monitoring documents, staff, and stakeholders

Output evaluation

• How long did it take to provide support to each beneficiary?


Program monitoring documents and staff

Outcome evaluation


Participants, program documents, and staff

Impact evaluation


Participants and program documents

(Alkin & Vo, 2018)

Deciding on Data Collection Strategies

Research Questions(Examples)

Data Sources Data Collection Strategies

• Are the resources adequate to implement the

program?

• Are the program staff capable of implementing the program or addressing the needs of the beneficiaries?

Program documents, staff, and stakeholders

Document analysis, interviews, focus groups and surveys

• Are the program being implemented as planned? Program monitoring documents, staff, and stakeholders

Document analysis, interviews, focus groups and observations

• How long did it take to provide support to each

beneficiary?


Program monitoring documents and staff

Document analysis, interviews, focus groups, observations and surveys


Participants, program documents, and staff

Document analysis, interviews, focus groups and surveys


Participants and program documents

Mainly surveys or usinginstrument to measure the variable of interest

Choosing an Appropriate Evaluation Method

Concurrent Mixed

Methods Design

Concurrent Triangulation

Concurrent Nested

Concurrent Transformative

Sequential Mixed

Methods Design

Sequential Explanatory

Sequential Exploratory

Sequential Transformative

Source: https://www.slideshare.net/kontorphilip/mixing-methodologies-ppt-2013-28526867

(Hanson et al., 2005)

https://www.slideshare.net/kontorphilip/mixing-methodologies-ppt-2013-28526867

Concurrent Mixed

Methods Design

Concurrent Triangulation

(Quan + Qual)

Concurrent Nested

(Quan + qual OR quan + Qual)

Concurrent Transformative

(Either Triangulation or Nested with

Advocacy Role)



Sequential Mixed

Methods Design

Sequential Explanatory

(Quan qual)

Sequential Exploratory

(Qual quan)

Sequential Transformative

(Either Explanatory or Exploratory with Advocacy Role)



Impact Evaluation

Program

(Cause)

Inputs

Activities

Outputs

Results

(Effect)

Low self-esteem

Counseling sessions

High self-esteem

Impact Evaluation

Determine “cause and effect relationship” (p. 243)

Research Questions

(Example)

Hypothesis

(Example)

Data Sources Data Collection

Strategy

Tool

Did the program impact participants’ self-esteem?

The program will have statistically significant effect on participants’ self-esteem

Participants Survey Coopersmith Self-Esteem Inventory (CSEI) http://www.mhhe.com/socscience/hhp/fahey7e/wellness_worksheets/wellness_worksheet_024.html

Low self-esteem

Counseling sessions

High self-esteem

http://www.mhhe.com/socscience/hhp/fahey7e/wellness_worksheets/wellness_worksheet_024.html

Quantitative Method for Impact

Evaluation

Three Main Types

Single-Group Pretest/Posttest

Nonequivalent Comparison Group

Randomized Experimental Design


Evaluation

Single-Group Pretest/Posttest

Focusing on program beneficiaries (participants)

Program

Measure Participants’ Self-esteem

Measure Participants’ Self-esteem

Research Questions(Example)

Hypothesis(Example)

Data Sources Data Collection Strategy

Tool

Did the program

impact

participants’ self-esteem?

The program will

have statistically

significant effect on

participants’ self-esteem

Participants Survey Coopersmith Self-Esteem Inventory

(CSEI)

http://www.mhhe.com/socscienc

e/hhp/fahey7e/wellness_worksheets/wellness_worksheet_024.html

The same participants

http://www.mhhe.com/socscience/hhp/fahey7e/wellness_worksheets/wellness_worksheet_024.html


Evaluation

Nonequivalent Comparison

Group

Two groups

Experimental group

Control groups

Compare

Program Participants’ Self-esteem

Self-esteem of

participants who were not

part of the program

Two groups


Evaluation Randomized Experimental

Design

Two groups – randomly

assign participants to:

Experimental group

Control groups

Compare

Program Participants’ Self-esteem

Self-esteem of

participants who were not

part of the program

Two groups

Randomly

assign

participants to

Writing Evaluation Plan• Program

• Staff

• Stakeholders

• Beneficiaries

• Logic Model

• Type of evaluation

• Evaluation purpose

• Evaluation question

• Evaluation method

• Source of data

• Data collection strategies

• Participants

• Recruitment

• Procedure

• Instrumentation

• Quality Assurance

• Data analyses

• Presentation of findings

• Utilization of finding

References

Alkin, M. C., & Vo, A. T. (2018). Evaluation essentials from A to Z. New

York: Guilford Press.

Hanson, W. E., Creswell, J. W., Plano Clark, V. L., Petska, K. S., &

Creswell, D. J. (2005). Mixed methods research designs in

counseling psychology. Journal of Counseling Psychology, 2(55),

224-235. doi:10.1037/0022-0167.52.2.224

Kettner, P. M., Moroney, R. M., & Martin, L. L. (2017). Designing and

managing programs: an effectiveness-based approach. Los

Angeles: SAGE.

Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Los

Angeles, CA: SAGE.

Philip Adu, PhD.

Methodology Expert

National Center forAcademic & Dissertation Excellence

(NCADE) The Chicago School of Professional Psychology

[email protected]

You could reach me at [email protected] and @drphilipadu on twitter.




program evaluation: simplifying the program evaluation process

Education