Introduction to Evaluation

Objectives

• Introduce the five categories of evaluation that can be used to plan and assess ACSM activities.

• Demonstrate how each evaluation category can be used to answer different evaluation questions.

• Review examples of each type of evaluation.

Evaluation is Part of the M&E Plan

Monitoring versus Evaluation

Monitoring

•Data collected on program activities

•Ongoing, routine

•Focus on process and output, compared to target

Are we doing the work we planned?

Evaluation

•Data collected to answer specific questions

•Periodic

•Focus on outcome, impact

How should we design our activities?

How effective were they?

Categories of Evaluation

1. Baseline evaluation (needs assessment)

2. Formative evaluation

3. Process evaluation

4. Outcome evaluation

5. Impact evaluation

Baseline Evaluation/Needs Assessment

• Identifies our ACSM starting point

Who do we need to reach and how?What activities are most important or most needed?Where do we need to go? What is the baseline value for key indicators?

Formative Evaluation

• Helps design activities for the target group

• Strengthens interventions by getting the right message to the right people

• Pre-testing

Do we have the right message?

Is this the right communication method?

Are we reaching the right people?

Do they understand the message?

Process Evaluation

• Documents what happens in a program

• What is going well, what is not, and why

• Often overlaps with monitoring

Did we do what we planned? Is the program being implemented effectively?What challenges do we need to address?What best practices can we share? Lessons learned?

Outcome Evaluation

Impact Evaluation

• Contribution of ACSM interventions to high-level goals and objectives

• Generates evidence about effectiveness of ACSM activities

• Very rigorous, time consuming, and expensive

Relies on high-quality baseline data collection.MUST be planned from the very beginning. In practice, rarely done.

Match the Question and Category!

Match the Question and Category!

BASELINE

FORMATIVE

PROCESS

OUTCOME

IMPACT

When do we evaluate?

E dW' ŽĂů͗� ZĞĚƵĐĞŵŽƌďŝĚŝƚLJĂŶĚŵŽƌƚĂů͗ŝƚLJƌĞů͗ĂƚĞĚƚŽd ͘

E dWK ďũĞĐƚŝǀ Ğ( /ŶĐƌĞĂƐĞĐĂƐĞĚĞƚĞĐƚŝŽŶƌĂƚĞĨƌŽŵϰϮй ƚŽϲϬй ďLJϮϬϭϱ ͘͘

^D K ďũĞĐƚŝǀ Ğ( ZĂŝƐĞŬŶŽǁ ů͗ĞĚŐĞŽĨdƐLJŵƉƚŽŵƐ ĂŶĚdƐĞƌǀ ŝĐĞƐƚŽŝŶĐƌĞĂƐĞƚŚĞŶƵŵďĞƌŽĨƉĞŽƉů͗ĞŝŶŝƚLJyƐĞĞŬŝŶŐĐĂƌĞĨŽƌdƐLJŵƉƚŽŵƐĂƚK d^ĐĞŶƚĞƌƐďLJϯϬй ďLJĞĐĞŵďĞƌϮϬϭ ϯ͘͘

/E Wh d^ • ^Ƶƌǀ ĞLJ • ^Ƶƌǀ ĞLJƐƚĂĨĨ • &ƵŶĚŝŶŐ • >ŝƐƚŽĨƐƵďǁ ĂLJ

ƌŽƵƚĞƐ • >ŝƐƚŽĨƌĂĚŝŽ

ƐƚĂƚŝŽŶƐ • D ĞĚŝĂďƵLJĞƌ

d/s /d/̂

• ŽŶĚƵĐƚ<WƐƵƌǀ ĞLJ

• Ğǀ Ğů͗ŽƉĂŶĚĂŝƌƌĂĚŝŽƐƉŽƚƐ

• Ğǀ Ğů͗ŽƉĂŶĚƉƌŽĚƵĐĞƐƵďǁ ĂLJĂĚƐ

K h dWh d^ • ηŽĨƉĞŽƉůĞ

ƐƵƌǀ ĞLJĞĚ

• ηŽĨƌĂĚŝŽƐƉŽƚƐĚĞǀ ĞůŽƉĞĚ

• # of times radio spots aired

• # of subway ads developed

• ηŽĨƐƵďǁ ĂLJƚƌĂŝŶƐǁ ŝƚŚĂĚƐ

K h dK D ^ • /ŶĐƌĞĂƐĞĚ

ŬŶŽǁ ůĞĚŐĞŽĨd ĂŶĚKd^ĐĞŶƚĞƌƐ

• /ŶĐƌĞĂƐĞĚηŽĨƉĞŽƉůĞƐĞĞŬŝŶŐĐĂƌĞĨŽƌdƐLJŵƉƚŽŵƐĂƚKd^ĐĞŶƚĞƌƐ

/D Wd • /ŶĐƌĞĂƐĞĚ

ŶƵŵďĞƌŽĨdĐĂƐĞƐ ĚĞƚĞĐƚĞĚ

RESULTS

Creating a Framework

BASELINE

FORMATIVE

PROCESS OUTCOME

IMPACT

Questions?