beyond logical frameworks to program impact pathways ciifad m&e workshop 5 november 2011 135...
TRANSCRIPT
Beyond logical frameworks to program impact pathways
CIIFAD M&E Workshop5 November 2011 135 Emerson Hall
Sunny S. Kim, MPH PhD candidate
Division of Nutritional Sciences
Role of research in evaluation
• Evaluation usually focuses on an internal situation, such as collecting data about specific programs, with no intent to generalize the results to other settings and situations. Evaluation determines the merit, worth, or value of things.
• Evaluation asks what should be, which makes reference to the standards and criteria of evaluations.
• Program-related research usually conducted with to the intent to generalize the findings from a sample to a larger population.
• Research asks why, which indicates a theoretical basis.
Basic steps of evaluation
• Identifying program goals and objectives
• Mapping the program
• Selecting an evaluation design– Sample size, control group, sampling frame
• Selecting indicators
• Determining data collection strategy
• Developing data collection instruments
• Analyzing the data
• Communicating the results
Program theory vs. logic model
• Program theory explains why a program is expected to work, and a logic model illustrates a program theory.
• Program logic model is a picture of how your organization does its work… it links the theoretical assumptions of the program and program activities/processes and outcomes.
Translating program theory…
Program theory has also been referred to as:
• program logic or intervention logic• theory-based or theory-driven evaluation • theory of change• theory-of-action• impact pathway analysis
all ways of developing a causal modal, linking program inputs and activities to intended or observed outcomes
Translating program theory…
• Program theories are often captured in a series of “if-then” statements.
• And between the “if” and “then,” there should be “how” and “why”, i.e. evidence or well-established connections (established theory, previous results, published research, feedback from participants).
… into logic models• a commonly-used tool for illustrating program theory. • a picture of describing the sequence of activities to
bring about change and how these activities are linked to the results the program is expected to achieve.
• often presented in a flow chart that illustrates the linkages between program components and outcomes:
Inputs Resources, supplies, staffActivities Processes, techniques, tools, events, technology, actions Outputs Direct results of activities (size and/or scope) Outcomes Changes in attitudes, behaviors, knowledge, skills, status,
or level of functioningImpact Organizational, population, community, and/or system
level changes (improved conditions, increased capacity, and/or policy changes)
… into logic models • “Logic models come in as many sizes and shapes as the
programs they represent.” [W.F. Kellogg Foundation, 2001, p.7]
• No best logic model, only the model that fits your program best and provides the information you need in the format that is most helpful.
Why use a logic model? • Design and planning helps to explain and illustrate program
concepts and approach for key stakeholders (advocate and train); finds “gaps” in the theory or logic of a program and work to resolve them .
• Implementation and management focuses attention on the most important connections between actions and results for monitoring, improved programming, and reporting.
• Evaluation, communication, and marketing presents progress toward goals; provides a way to involve and engage stakeholders in the design, processes, and use of evaluation.
Why use a logic model? • Donor requirements:
Indicators
Means of verification
Assumptions
Goals
Objectives
Outputs
Activities
Resources
Goal: Health status improved
Strategic Objective: Use of key services and behaviors improved
Intermediate Result 1: Access and availability of services and supplies increased
Intermediate Result 2: Quality of services improved
Intermediate Result 3: Demand for key services and behaviors increased
Intermediate Result 4: Social and policy environment enabled
Log frame (OECD/DAC, USAID, CIDA…)
Results framework(USAID…)
Intervention model of Programa de Nutrición Infantil (PNI)
Maternal and child health and nutrition
Small-scale economic activities
Strengthening of civil society
Basic sanitation and sanitary education
Improved health and nutrition knowledge and practices
Improved development plans of CBOs in health and sanitation
Improved basic sanitation
Increased access to health services
Improved health and nutritional status of children <3 years
Improved health and nutritional status of pregnant women
Strategies: Information Education Social communication
FINAL OUTCOMES
INTERMEDIATE OUTCOMES
COMPONENTS
Intermediate Result 1: Improved nutrition, health knowledge and practices of those attended in the target communities
Strategic Objective: Improved health and nutritional status of children <3 years and pregnant women
Intermediate Result 3: Improved basic sanitation in the target communities
Intermediate Result 4: Community-based organizations applied their improved knowledge on health and sanitation in the formulation of their development plans
GOAL: Reduced rates of child malnutrition
Intermediate Result 2: Increased access to health services of the target population
Results framework of Programa de Nutrición Infantil (PNI)
From boxes to arrows…
• While “filling in” the logic models with the program components , we must not neglect the mechanisms and linkages… don’t forget about your program theory or impact pathways.
“The theme/issue is the training that you give through the participatory methdology to the mothers. If you just give the theory, it is the theme. The workshop is the practice. So if that day, for example, the mothers didn’t bring food to prepare their workshop, then the [health] promoter only gives the theory. They discuss, socialize, and receive only the theories in the small groups… Then the workshop returns another day to learn about doing. For example, the health themes, they make their workshop do dramatizations. The mothers do their dramatizations, it tells the priority, what happens and everything, and so they are going to learn.”
[Regional level, ex-ADRA adviser]
Description of mothers workshops
Impact pathway of mothers workshops
Husbands permit/ support women to participate
CHAs raise topic of health and care for children (i.e., diarrhea and infections; respiratory illness; early stimulation) and reproductive health and prenatal care
CHAs raise topic of self esteem and family violence
CHA raise topic of hygiene and sanitation
CHAs convene pregnant women and mothers of children <3 years to participate in workshops
CHAs and mothers discuss ideas and experiences
CHAs and mothers demonstrate recipes
Mothers understand nutritional needs of children and pregnant women and know feeding and cooking practices
Mothers recall nutritional needs and feeding and cooking practices
Mothers implement feeding practices and recipes
Nutritional status of pregnant women improved
CHAs raise topic of child feeding (i.e., BF and CF; balanced diets; feeding sick children) and nutrition for pregnant women
Children receive appropriate care and health attention in a timely manner
CHAs and mothers discuss ideas and experiences
Pregnant women consume adequate and appropriate foods
CHAs and mothers perform dramatizations
Mothers understand child health, reproductive health and prenatal care issues and know appropriate practices
Mothers recall child health, reproductive health and prenatal care issues and appropriate practices
Mothers implement care practices and seek health services
Pregnant women receive appropriate care and health attention in a timely manner
CHAs and mothers discuss ideas and experiences
CHAs and mothers perform dramatizations
Mothers understand issues of self esteem and violence and know how to address them
Mothers recall information on self esteem and violence
Mothers implement practices related to self esteem and violence
Children exposed to safe and caring home environments
CHAs and mothers discuss ideas and experiences
CHAs and mothers demonstrate practices
Mothers understand issues of hygiene and sanitation and know appropriate practices
Mothers recall information on hygiene and sanitation
Mothers implement hygiene and sanitation practices
Children exposed to safe and clean home environments and less exposure to contamination
Child nutrition and growth improved
Household food availability and access
Quality health service access and availability
Influencing factors –facilitators
Children consume adequate and appropriate foods
Household access to clean water
Women see incentive to participate (e.g. food baskets or small economic activities)
Use of local language and readily accessible foods
Husbands are sensitized about domestic violence
Repeated reminders or internal/external stimulus (e.g. home visits, concern for child’s future)
?
Main references• Kellogg Foundation. (2004). Logic model development guide: Using
logic models to bring together planning, evaluation, and action. Battle Creek, MI: W.K. Kellogg Foundation.
• Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (Eds), Evaluation models, 209-232. Boston, MA: Kluwer Academic Publishers.
• Rogers, P.J. (2008). Using program theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1): 29-48.
• Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: teaching and training guide. Madison, WI: University of Wisconsin-Extension.