evaluating impact of ovc programs: standardizing our methods jenifer chapman, phd futures...
TRANSCRIPT
Evaluating impact of OVC programs: standardizing our
methods
Jenifer Chapman, PhDFutures Group/MEASURE Evaluation
Overview
Evaluation vs. monitoring vs. case management
Update on the MEASURE Evaluation OVC program evaluation tool
Key questions
Evaluation vs. monitoring vs. case management
Case management questions What are the child’s immediate priority needs?
Is child in school now?
Have earlier priority needs abated?
Monitoring questions How many children are receiving services?
How many social workers have been trained?
Is the program being implemented as planned?
3
Evaluation vs. monitoring vs. case management
Evaluation questions: Did program beneficiaries show improved well-
being over time? What proportion of households are food secure,
compared to 5 years ago?
Can improvements in well-being be attributed to the program?
4
Tools for answering these questions We have case management tools, and
monitoring tools… We lack a standardized approach and tool for
evaluating the impact of OVC programs globally
An evaluation tool, in context
Purpose: To assess the impact of a program (on child/household wellbeing) over time
Who uses it?: Trained data collectors / researchers
Among whom is it applied? A sample of beneficiaries
Sometimes a matched group of potential beneficiaries
What is an evaluation tool?
What information is collected? Client contact information / demographics
Wellbeing information that is changeable over time
Services provided
Information must be useful at program/national level (different information than needed for case management/care planning)…
Evaluation protocols
Evaluation requires a protocol detailing: Sampling and recruitment strategy
Data collection procedures
Data management and data analysis
Dissemination
Ethical approval is required. Evaluation is not part of routine QI involving service providers.
8
MEASURE Evaluation OVC Program evaluation toolkit
9
The problem
Lack of standardized, objectively verifiable, impact indicators of child wellbeing
10
The vision Quantitative child outcomes and household
outcomes measurement tools, valid across countries at the population level With survey protocol, analysis plan and report
template, and training materials
Progress: final shortlist of 12 child & 3 caregiver/HH impact
indicators
Draft tools in discussion
Piloting in Nigeria, Mozambique, Zambia & Ethiopia
11
Methods
Extensive research (lit reviews, international/national child wellbeing tools and indicators, OVC program evaluation tools, national OVC M&E plans, DHS, etc.)
Result: 100s of child & household wellbeing indicators
Indicator analysis against defined criteria & wide stakeholder discussions
Result: shortlist of indicators
Inclusion criteria
1. Measures impact/outcomes
2. Amenable to change from program interventions
3. Relevant across a wide range of interventions
4. Contributes to a holistic vision of child wellbeing
5. Objectively verifiable
6. Easy to implement
7. Relevant across different regions / countries
8. Relevant or easily adapted across age and sex
13
12 Child wellbeing indicators
1. Percent of children malnourished 6-59 months old: measurement of middle-upper
arm circumference (MUAC)
>60 months old: measurement of weight and height
2. Percent of children with recent diarrhea Have you / has the child had diarrhea in the two
weeks preceding the survey?
14
12 Child wellbeing indicators
3. Percent of children with recent fever Have you / has the child had a fever in the two
weeks preceding the survey?
4. Percent of children who are too sick to participate in daily activities In the last 2 weeks, have you / has the child been
too sick to participate in daily activities?
15
12 Child wellbeing indicators
5. Percent of children reporting irregular food intake Have you / has the child gone a whole day or night
without eating in the last 4 weeks?
6. Percent of children fully immunized Has child received all age-appropriate
immunizations?
16
12 Child wellbeing indicators
7. Percent of children with basic shelter Is the place that you / the child slept last night
protected from the weather?
8. Percent of children with basic social support Is there someone you / the child can go to, to help
solve a problem?
17
12 Child wellbeing indicators
9. Percent of children who have a birth certificate / identification card Does the child have a birth certificate or
registration / ID card?
10. Percent of children currently enrolled in school Are you / is the child currently enrolled in school?
18
12 Child wellbeing indicators
11. Percent of children regularly attending school During the last school week, did you / the child
miss any school days for any reason?
12. Percent of children who progressed in school over time What is the highest level of education you have /
the child has completed?
19
3 Household wellbeing indicators
1. Percent of households in which caregiver reports basic social support Is there someone you can go to, to help solve a
problem?
2. Percent of households that are food insecure due to lack of resources In the past 4 weeks, was there ever no food to eat
of any kind in your household because of a lack of resources to get food?
20
3 Household wellbeing indicators
3. Percent of households able to access money to meet important family needs If you needed money to meet an important family
need, such as to pay for school fees, pay for transportation, or purchase for food, how would you pay? (smaller expenses)
If you needed money to meet an important family need, such as to pay for a family emergency, pay for a house repair, or pay for medical treatment, how would you pay? (larger expenses)
21
And the tool?
Indicators form the basis of two tools:
1. Child questionnaire (ages 10-17)
2. Caregiver questionnaire (including questions on caregiver wellbeing, household, child wellbeing)
Draft tools available Autumn 2012
Tools will be piloted in Autumn/Winter 2012
Final tools & guidance documents available mid-2013
For discussion PSS and HES indicators – your experience,
please!
Methodological fine points (or not so fine points)…what works, what does not?
How is your organization currently evaluating programs? How will this tool support you in evaluation?
What more is needed?
23
The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00-08-00003-00. Views expressed are not necessarily those of PEPFAR, USAID or the United States government.
MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.
Round-up
Our Community of Practice
There is lots going on in OVC M&E but cross-country dialogue is lacking
ChildStatusNet is being revamped as an OVC M&E Community of Practice (or even M&E of community-based interventions) Forum to exchange ideas & share tools /
indicators / methods
Regular webinars with guest speakers
For more information, contact:
Jenifer Chapman: [email protected]
Karen Foreit: [email protected]
http://www.cpc.unc.edu/measure/