but we will always be here! assuring sustainable benefits after food aid programs_beatrice...
TRANSCRIPT
Beatrice Rogers and Jennifer Coateswith:
Johanna Andrews, Alexander Blau, Ameya Bondre, Jamie Fierstein, Kathryn Houk, Tina Galante, Carisa
Klemeyer, Elizabeth Kegode, Leslie Sanchez
“BUT WE WILL ALWAYS BE HERE!”ASSURING SUSTAINABLE BENEFITS
AFTER FOOD AID PROGRAMS
Exit Strategy Study Overview2
Study Rationale3
Title II programs closing in non-priority countries
Little systematic knowledge of whether program impacts are maintained and how to maintain them
Effectiveness of Title II programs depends on both short-term impact and long-term sustainability
Immediate and long-term impact may be trade-offs
Study Objectives4
1. Determine the extent to which activities, outcomes, and impacts of Title II programs were sustained.
2. Identify program characteristics that make it possible to sustain program activities and effects after the program shuts down.
3. Assess how the process of “exiting” affects sustainability.
4. Provide guidance to future programs on how to exit while ensuring sustainability.
Key Concepts5
Sustainability Sustainability of
impacts among program participants
Diffusion of impacts to new populations
Sustainability/ expansion of activities
Exit From specific
activities By the Title II
Awardee from the area
Key Concepts: Exit Strategy/Sustainability Plan
6
Explicit plan for transition from program support to exit
Often called “sustainability plan” in agency documents
Specifies approach to exit: phase over (to whom), phase out (of resources)
Should include timeline and allocation of responsibility for ensuring progress
Study Methods Overview7
1. Qualitative picture of exit strategy implementation around time of agency exit
2. Qualitative inquiry one year after exit: to explore the success of exit strategy in leading to sustainable service provision (where applicable) and beneficiary uptake
3. Qualitative and quantitative data two years after exit to assess sustainability of impacts and outcomes
4. Compare quantitative follow-on data to agency endline survey data to determine sustainability and link to exit strategies
Analytic Strategy: Associating Exit Strategy with Sustainability of Impacts
8
Agency evaluations did not include control groups for attribution of impact
Associations based on: Testing Implementation Pathways
Sustainability of desired behaviors and impacts Continuation of service provision and utilization Implementation of an exit strategy
Use of Secondary Data To serve as a point of comparison for survey results
Study Locations• Kenya• Bolivia• Honduras• India
Selection Criteria• Programs achieved key
impacts on food security and malnutrition
• Programs implemented exit strategies
• Close-out coincided with study time frame9
Conceptualizing Sustainability and Exit
10
Sustained Impact
Sustained Behaviors and/or Service Utilization
SUSTAINED
SERVICE DELIVERY
SUSTAINED
ACCESS
SUSTAINED
DEMAND
SUSTAINED
RESOURCES
SUSTAINED
CAPACITY
SUSTAINEDMOTIVATIO
N
SUSTAINED
LINKAGES
Program Exit StrategiesProgram Exit StrategiesEXTE
RN
AL F
ACTO
RS
11
Critical combination of resources, motivation, and technical and managerial/administrative capacity
Programs should be designed with exit in mind Gradual transition to independence with a period of
independent operation before exit Importance of linkages - variable Impact at exit is not the same as sustained benefit later
on Provision of free resources poses challenges for
sustainability Depending on government linkages is risky, and
depending on newly entering NGOs poses questions Different technical sectors face different challenges
applying these principles
Emerging Conclusions
RESULTS - BOLIVIA12
13
Title II Program Areas in Bolivia
14
2002 20112008-9
Agencies collect baseline
quant data
Tufts collects
post-program
qualitative data
2010
Tufts collects post-program qualitative and
quantitative datafollow-up
quantitative survey replicates
agencies’ endline surveys
2004
Agencies collect mid-term quant and qual
data
Agencies collect endline
qual and quant data
Study Methods Overview
15
1. Maternal and Children Health and Nutrition (MCHN)
2. Water and Sanitation (WatSan)
3. Agriculture/Rural Income Generation (RIG) (after mid-term evaluation change in focus from production to commercialization)
4. Natural Resource Management (NRM)
Program Technical Sectors
16
Motivate CHWs to continue providing GMP, health promotion and home visits through social prestige and respect; credentialing.
Link CHWs to decentralized health services to assure supervision, training, resources
Teach mothers to substitute local foods for Title II provided foods
MCHN Exit StrategyMCHN Exit Strategy
17
Prevalence of stunting declined substantially during Title II and generally remained low after exit: ADRA, CARE, SC: No significant
change in stunting since exit FH: Stunting continued to decline
significantly after exit While overall stunting in the
Departments declined during the period of Title II implementation, the Awardee declines were larger than regional declines and Awardees worked in areas with much higher initial rates of stunting
MCHN Results: Stunting
18
Departmental statistics (< 5y)Title II Communities (3–35 mo)
Percentage of children stunted (HAZ)
Stunting
19
GMP participation and prenatal care visits remained high Success of linkages to national government programs:
Zero Malnutrition (ZM): nutritional supplements for children under 2; Comprehensive Nutrition Units (UNIs) at healthcare centers
Bono Juana Azurduy (BJA) conditional cash transfers for completing prenatal/postpartum care and growth monitoring visits
Awardees with both a significant increase in GMP and the strongest malnutrition impacts, CARE and FH, continue to operate after exit
GMP participation declined in SC communities, where fewer governmental and NGO resources may be available
MCHN Results: Demand for ServicesGrowth Monitoring and Prenatal Care
20
Percentage of households taking child < 5y to GMP
Growth Monitoring Participation
21
Percentage of mothers reporting any prenatal care during last pregnancy
Prenatal Care
22
Percentage of mothers reporting prenatal care in first 5 months during last pregnancy
Prenatal Care (< 5mo)
23
Presence of a community health worker remains high
Coherent governmental and NGO initiatives likely sustaining CHW presence: Intercultural Family and Community
Health Program (SAFCI) Tarija’s “Health Guards” NGOs (Esperanza Bolivia, Plan
International) adapt CHWs for their own program activities
MCHN Results: Service Delivery
24
FH collaborates with UNICEF in some former Title II communities in malnutrition prevention and GMP, with funding from Spain and USAID (Proyecto Integrado de Seguridad Alimentaria, PISA)
In some cases local incentives may sustain CHWs Incorporation into water committee;
profit-generating women’s groups
MCHN Results: Service Delivery
25
Percentage of communities reporting CHW presence
Community Health Workers (2011)
26
EBF increased during the program and has been sustained No cost; reinforced bygov’tprograms
(BJA, ZM) All other practices have declined since exit
Proper treatment of diarrhea, hygiene Importance of reinforcing behavior change
messages: Behaviors not reinforced through home
visits/trainings since exit Not prioritized since health system targets
services like prenatal visits/GMP
Results: MCHN Practices
27
Title II Communities National Statistics
Percentage of mothers reporting exclusive breastfeeding for child < 6mo
Exclusive Breastfeeding
28
Diarrhea Treatment (food and liquid)
Percentage of mothers offering same/more food to child during diarrhea
Percentage of mothers offering same/more liquids to child during diarrhea
29
Percentage of mothers offering same/more food , liquid, or ORT to child during last diarrhea episode
Diarrhea Treatment (food, liquid, and ORT)
30
Percentage of children 12–23 months of age receiving the third dose of DPT or pentavalente vaccine
Note: No data available for ADRA in 2002 and 2004
Vaccinations
31
Sustainability of impacts and CHW service use due to alignment with government health programs and presence of NGOs for continued support ANC and GMP remain high since exit
Behavior change harder to sustain once food rations removed Rations had been provided children < 35 months and
pregnant women participating in GMP/health talks Use of health services remains high since exit, but the
lack of resources for community training and CHW supervision appears to limit the quality of information provided or the incentive to continue household-level behaviors requiring time/resources
MCHN Summary
32
Piped water and latrinesAwardee provided inputs for construction of piped water and latrinesElected water committees trained in system maintenance and repair, and in financial management and administration.Beneficiaries provided labor for construction, pay a connection fee and a monthly fee for water use; water cut off for lack of payment;User fees cover maintenance and repairsSome water committees were operating prior to Title II but received training
WatSan Exit Strategy
33
Vertical linkages to municipal government and horizontal linkages among water committees were part of sustainability plan, not implemented
Water quality testing Water committees would take over water quality testing
once awardees left
Hand washing and latrine use Hygiene behaviors promoted by community health
workers in home visits would continue
WatSan Exit Strategy
34
Infrastructure continues to be available in Awardee areas Percentage of communities reporting piped water and
latrine availability generally maintained Immediate tangible benefit; responds to priority need;
successful water committee model; infrastructure coverage is government priority
Sanitation and handwashing practices were not sustained after exit Water quality testing also not maintained
Results: WatSan
35
Percentage of households with piped water
Title II Communities Departmental Statistics
Piped Water
36
Percentage of communities maintaining their own water system (2011)
Water System Maintenance
37
Percentage of households with latrine
Latrine Access
38
Percentage of population using hygienic sanitation facilities (with signs of use, Awardee criteria)
Latrine Use
39
Percentage of caregivers (responsible for food preparation) with proper handwashing practices (awardee criteria)
Handwashing Practices
40
Percentage of caregivers using soap/detergent
Note: No data available for CARE
Soap Use
41
Percentage of households using any water treatment (boiling, chlorination, SODIS)
Note: No data available for FH
Water Treatment
42
Water and sanitation infrastructure demonstrated greater sustainability than hygiene behaviors and water purification Infrastructure: Tangible benefit, self-financing user
fees, community capacity to operate system and maintain accountable administration; not dependent on linkages
Behaviors: No immediate, tangible benefit, no financial incentives to motivate implementation, no national program focused on such behaviors, since national priorities are more concerned with visible infrastructure coverage
Water quality: no independent operation before exit
WatSan Summary
WatSan Sustainability!
43
44
Rural Income Generation (RIG) Exit Strategy
Model farmers were given free agricultural inputs and training in return for training other farmers
Model farmers would continue to serve as technical resources
Focus on improved production and diversification shifted to commercialization after MTE
Farmers associations formed, trained in production and in management, contracting
Profits from increased sale would motivate sustained use of practices taught in the program, and profits would cover purchase of inputs
45
Results Rural Income Generation (RIG)
Agricultural income of farmers increased throughout the program, but fell substantially at follow-up, though remaining substantially higher than baseline Inputs provided free during program now must be
purchased
Farmers who are members of producer associations (PA) have maintained substantially higher incomes than those who received training but are not PA members
National GDP per capita continues to rise according to World Bank indicators
46
Mean Annual Income from Agricultural Sales by Farmers Trained in the Program
(adjusted for inflation, 2011=100)
Agricultural Income
47
Mean Annual Agricultural Income: Producer Association Members vs. Non-Members
(2011)
48
Percentage of trained farmers belonging to a producer association
Note: No endline data available for ADRA and FHCARE participation data are only for farmers in promoted value chains
Producer Association Membership
49
Percentage of trained farmers adopting improved agricultural practices (Awardee criteria)
Improved Agricultural Practices
50
Percentage of farmers in producer associations is declining
Members of producer associations have substantially higher incomes than non-members
Qualitative evidence indicates that farmers may drop out of associations because they are unable to meet quality standards
More successful farmers may be those able to afford inputs through the profits from marketing
Commercialization model is successful for those who can take advantage of it
RIG Results Summary
51
Use of inputs and promoted practices declining Inputs provided free during life of program now
must be purchased; farmers may have become accustomed to receiving these inputs without charge
Practices sustained by trained farmers are those returning noticeable benefit and low cost (organic fertilizer, crop rotation)
Proportion of farmers adopting improved practices similar among trained and un-trained farmers Individual producers likely copying improved
production methods
RIG Results Summary
52
Sales through an association more likely where associations continue to receive support from external sources (NGOs, foundations, government)
Government partners have high rates of turnover, changing priorities, stretched budgets, and shortages of technical staff; partnerships with buyers are more reliable
Sustained operation of PAs is more likely when responsibility of negotiating contracts is transitioned gradually, with a period of independent operation
RIG Results Summary
53
Practices have declined significantly since exit Food for work and free inputs no longer available Motivation is low if tangible benefit is not perceived
Continued practices are those that produce tangible benefit and do not require purchased inputs
Municipal Natural Resource units are underfunded and understaffed
Some activities produced lasting change during the program, but were not continued (forestation, terracing)- phase out
Results: Natural Resource Management
54
Trained Farmers Adopting at Least Three NRM Practices
55
NRM activities largely not sustained after exit
Technical capacity was strong; resources and motivation lacking
Municipal support was weak or lacking
Withdrawal of food and free inputs jeopardized sustainability of activities
NRM Results Summary
Conclusions56
Conclusions57
Impact at exit does not consistently predict sustained impact two years later.
There are specific ways to increase the likelihood of sustainability.
Provision of free resources poses risks to sustainability.
Impact Assessment at Exit Can Be Misleading
58
Impact assessment at exit does not consistently predict impact two years later.
Many activities, practices, and impacts across sectors declined over the two years after exit.
These declines are related to inadequate design and implementation of sustainability strategies and exit processes.
Sustaining service provision and beneficiary utilization of services and practices depends on three critical factors:1. Resources 2. Technical and Management Capacity3. Motivation
There are often synergies among these three elements.
Best practice models have emerged for each – and they are often sector-specific.
The relevance of linkages is sector-specific. Gradual exit and independent operation are
necessary (though not sufficient) for sustainable results.
Specific Ways to Increase the Likelihood of Sustainability
59
Provision of Free Resources Poses Risks to Sustainability
60
Withdrawal of food rations or any other free input (as incentive) jeopardizes sustainability without consideration of substitute incentives.
Provision of free food rations risks creating unsustainable expectations.
Consideration of alternative incentive structure must be incorporated into program design.
Beneficiaries receiving free inputs to support program activities may not be willing or able to replace them once project ends.
MCHN: Exit Strategy Models Lacked Sustained Sources of Resources, Capacity, and Motivation61
Motivation: Withdrawal of food was a disincentive for participation in and provision of growth monitoring. In Kenya, withdrawal of food resulted in
reduced participation in growth monitoring. NGO return also reinstated the incentive.
In Honduras, withdrawal of food reduced demand participation in growth monitoring, while in Bolivia, the government implemented a cash incentive for growth monitoring and promotion.
Overall community health worker (CHW) service provision declined in Kenya, Bolivia, and Honduras with decline in material incentives and demand.
MCHN: At least four types of resources must be considered in order to sustain CHW service
delivery62
1. Resources that helped CHWs do their job, such as weighing scales, report forms, and bicycles, as well as training and supervision from the health sector
2. Resources that they offered the community as an incentive to participate in activities
3. Benefits accrued to the CHWs that incentivized their service, such as access to goats distributed to women’s goat groups
4. CHW time and its converse, the opportunity cost of time spent on CHW activities rather than on other productive labor
• No fee for service models were observed in the health sector. • Linkages to government did not fulfill all resource needs..
MCHN: Success of linking to government depended on government capacity and resources63
Bolivia: Some CHWs continued linkages to health system; public health system provided services.
In Honduras, linkages with health system worked until government resources ran out.
In Kenya, government did not/could not take the responsibility for supervision, training, or provision of resources resulting in declines in CHW service quality, motivation, and capacity when they were not being reinforced.
Water system exit strategy demonstrated that motivation, capacity, and resources are all critical
to sustainability64
Beneficiaries are motivated to pay for water services that are reliable, convenient, and abundant.
Technical and management capacity of water committees permits system to continue operation post-project.
Fees paid by consumers provide ensured resources for system maintenance.
WatSan: Linkages to Government Not Always Needed
65
Linkages were actively avoided by most water committees; independence from government entities that could be unreliable sources of support
Sustainability of Water Qua lity Provides a Counterexample….
66
Motivation for water quality testing is low because benefits are not visible.
Therefore willingness to pay was low.
Capacity-building activities during the DAPs did not emphasize water quality.
Water quality testing was not operating independently at exit.
No linkages to support water quality testing were in place at exit.
Agriculture Sector Exit Strategies Yielded a Mixed Sustainability
Story67
Model farmers (extension farmers) lacked motivation, resources, and reinforcement of capacity to continue providing technical assistance after incentives were withdrawn. Service provision largely declined.
Producer associations participation rates were variably low/declined. PAs provided concrete benefits in some countries but not all.
Commercialization activities produced sustainable benefits for those engaged in them.
Improved agricultural practices generally declined when requiring resources provided during the
DAP68
Free inputs were provided during program life
and
Use of these inputs declined when they needed to be purchased after the program ended
Engagement in NRM activities and practices declined dramatically after
program exit69
Food as pay (FFW) was withdrawn.
Resources in the form of inputs (seedlings, etc.) was no longer provided free of charge.
Motivation was lacking unless costs were low and direct benefits to farmers were high.
Linkages to local government environmental groups were ineffective as those groups lack resources.
Positive changes (reforestation, terracing, etc.) were maintained, though activities to expand these changes were not.
COSAMO in Kenya Was a Model of Sustainability
70
Sustained capacity, motivation, and resources. Sustained beneficiary utilization and impact.
Self-financing: No outside seed money and no other external resources needed.
Modular training program built solid technical and managerial capacity.
There was a gradual withdrawal of NGO after a period of COSAMO group independent operation
Linkages were not necessary – could pay for technical assistance of trained resource persons.
Recommendations71
Program Design72
Sustainability should be built into the design of programs from the beginning.
Plans must include: decision about approach (phase out, phase over); explicit benchmarks for progress; timeline; clear allocation of responsibility, graduation criteria
Each element — capacity, motivation, and resources — must be considered, with redundancies and contingencies for external shocks built into the plan.
Project Cycles Should Be Longer and Incorporate Sustainability Benchmarks
73
Phased Program Implementation
74
Sustainable design and initial service delivery, demand creation, and partnership formation
Strengthening capacity, developing ensured resources, ensuring appropriate linkages, promoting independent operation
Independent operation of program activities and gradual agency withdrawal
Program Monitoring and Evaluation
75
Emphasis on assessment of impacts (e.g., reduction in stunting) can undermine focus on sustainability.
Progress toward sustained capacity, motivation, and resources should be measured as process indicators during monitoring.
Evaluations should be done at each phase; criteria for assessment will differ by the objectives of the phase.
Program Withdrawal (Exit)76
Phase-over of responsibility must be gradual.
Groups should be operating independently (with agency backstopping) before full withdrawal.
Community organizations and individuals should be aware of their post-exit roles and responsibilities from the outset.
Philo s o phy of sustainability should be embedded throughout implementation to withdrawal.
Be aware that communities may not commit to independence, in the expectation of new sources of external support.
This study is made possible by the generous support of the American people through the support of the Office of Health, Infectious Diseases, and Nutrition, Bureau for Global Health, and the Office of Food for Peace, Bureau for Democracy, Conflict and Humanitarian Assistance, U.S. Agency for International Development (USAID), under terms of Cooperative Agreements GHN-A-00-08-00001-00, AID-OAA-A-11-00014, and AID-OAA-A-12-00005 through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360.
The contents are the responsibility of Tufts University and do not necessarily reflect the views of USAID or the United States Government.
77
Acknowledgments and Disclaimer
THANK YOU!