The National Evaluation Platform

Approach

Robert E Black MD, MPHInstitute for International ProgramsBloomberg School of Public Health

Johns Hopkins University

Outline

1. Why a new approach is needed

2. National Evaluation Platforms (NEPs): The basics

3. Country example: Malawi

4. Practicalities and costs

Most current evaluations of large-scale programs aim to use designs like this

Impact

Coverage

Program

No impact

No coverage

No program

But reality is much more complex

General socioeconomic and other contextual factors

Impact

Coverage

Routine health services

Interventions in other sectors

Other healthprograms

Program

Other nutrition and health

programs

Mozambique

• How to evaluate the impact of USAID-supported programs?

• Traditional approach: intervention versus comparison areasSource:

Hilde De Graeve, Bert Schreuder.

Mozambique

• Simultaneous implementation of multiple programs

• Separate, uncoordinated, inefficient evaluations– if any

• Inability to compare different programs due to differences in methodological approaches and indicatorsSource:

Hilde De Graeve, Bert Schreuder.

New evaluation designs are needed

• Large-scale programs

• Evaluators do not control timetable or strength of implementation

• Multiple simultaneous programs with overlapping interventions and aims

• Contextual factors that cannot be anticipated

• Need for country capacity and local evidence to guide programming

Lancet, 2007

Bulletin of WHO, 2009

Sources: Victora CG, Bryce JB, Black RE. Learning from new initiatives in maternal and child health. Lancet 2007; 370 (9593): 1113-4. Victora CG, Black RE, Bryce J. Evaluating child survival programs. Bull World Health Organ 2009; 87: 83.

NATIONAL EVALUATION PLATFORMS: THE BASICS

Lancet, 2011

Builds on a common evaluation framework, adapted at country level

Common principles (with IHP+, Countdown, etc.)

Standard indicators Broad acceptance

Evaluation databases with districts as the units

• District-level databases covering the entire country• Containing standard information on:

Inputs (partners, programs, budget allocations, infrastructure)Processes/outputs (DHMT plans, ongoing training, supervision,

campaigns, community participation, financing schemes such as conditional cash transfers)

Outcomes (availability of commodities, quality of care measures, human resources, coverage)

Impact (mortality, nutritional status)Contextual factors (demographics, poverty, migration)

Permits national-level evaluations of multiple simultaneous programs

A single data base with districts as the rows

District … …

Chitipa

Karonga

….

Core Data Points from Health Sector

Core Data Points from Other Sectors

HMIS

DHS

National Stocks data base

Nutrition Surveillance System

Rainfall patterns

Women’s education

Quality Checking & Feedback to Source

Types of comparisons supported by the platform approach

• Areas with or without a given program – Traditional before-and-after analysis with a

comparison group• Dose response analyses

– Regression analyses of outcome variables according to dose of implementation

• Stepped wedge analyses– In case program is implemented sequentially

Evaluation platform Interim (formative) data analyses

• Are programs being deployed where need is greatest?– Correlate baseline characteristics (mortality, coverage, SES, health

systems strength, etc) with implementation strength– Allows assessment of placement bias

• Is implementation strong enough to have an impact?– Document implementation strength and run simulations for likely

impact (e.g., LiST)

• How to best increase coverage?– Correlate implementation strength/approaches with achieved

coverage (measured in midline surveys)

• How can programs be improved?– Disseminate preliminary findings with feedback to government and

partners

(All analyses at district level)

Evaluation platform Summative data analyses

• Did programs increase coverage?– Comparison of areas with and without each program over time– Dose-response time-series analyses correlating strength of program

implementation to achieved coverage

• Was coverage associated with impact?– Dose-response time-series analyses of coverage and impact indicators– Simulation models (e.g. LiST) to corroborate results

• Did programs have an impact on mortality and nutritional status?– Comparison of areas with and without each program over time– Dose-response time-series analyses correlating strength of program

implementation with impact measures

The platform approach can contribute to all types of designs

• Having baseline information on all districts allows researchers to measure and control placement bias

• In real life one cannot predict which districts will have strong implementation and which ones will not

• In intervention/comparison designs, it is important to document that comparison districts are free of the intervention

• Collecting information on several outcomes allows assessment of side-effects of the program on other health indicators

COUNTRY EXAMPLE: CCM IN MALAWI

Simultaneous implementation of multiple programs

Separate, uncoordinated, inefficient evaluations (if any)

Inability to compare different programs due to differences in methodological approaches and indicators

Malawi CCM scale-up limits use of intervention-comparison design

CCM supported in all districts beginning in 2009…

… and implemented in Hard-to-Reach Areas! (March 2011)

DistrictsUNICEF /

WHO/UNFPA

Save the Children

DELIVER/ BASICS

PSI SC4CCM Concern At risk

Balaka

BlantyreChikhwawaChiradzuluChitipaDedzaDowaKarongaKasunguLikomaLilongweMachingaMangochiMchinjiMulanjeMwanzaMzimba North Mzimba South NenoNkhata BayNkhotakotaNsanjeNtcheuNtchisiPhalombeRumphiSalimaThyoloZomba

BALAKABLANTYRE

CHIKWAWACHIRADZULU

CHITIPADEDZADOWA

KARONGAKASUNGU

LIKOMALILONGWEMACHINGAMANGOCHI

MCHINJIMULANJEMWANZA

MZIMBA (north)MZIMBA (south)

NENONKHATABAY

NKHOTAKOTANSANJE

NTCHEUNTCHISI

PHALOMBERUMPHISALIMATHYOLOZOMBA

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

81%100%

83%90%

77%53%

65%60%

48%100%

56%66%

39%85%

38%87%

50%30%

83%100%

88%89%

59%100%

78%38%

57%51%51%

Proportion of Hard-to-Reach Areas with ≥1 Functional Village Clinic, March 2011

Malawi adaptation of National Evaluation Platform approach

National Evaluation Platform design using dose-response analysis, with

DOSE = PROGRAM IMPLEMENTATION STRENGTHRESPONSE = INCREASES IN COVERAGE;

DECREASES IN MORTALITY

Evaluation Question:Are increases in coverage and reductions in mortality greater in districts with stronger MNCH program implementation?

Platform design overviewDesign element Data sources (sample = 29 districts) Documentation of program implementation and contextual factors

Full documentation every 6 months through systematic engagement of DHMTs

Quality of care survey at 1st-level health facilities

Existing 2009 data to be used for 18 districts; repeat survey in 2011

Quality of care at community level (HSAs)

Desirable to conduct in all 28 districts (Not included in this budget proposal)

Intervention coverage DHS 2010, with samples of 1,000 households representative at district level in all 28 districts

DHS/MICS 2014 with samples representative at district level in all 28 districts

Costs Costing exercises in ≈ 1/3 of districts distributed by region and chosen systematically to reflect differences in implementation strategy or health system context

Impact (under-five mortality and nutritional status)

End-line household survey (MICS or DHS?) in 2014

Modeled estimates of impact based on measured changes in coverage using LiST

National Evaluation Platform: Progress in Malawi - 1

Continued district level documentation in 16 districts Pilot of cellphone interviews for community-level

documentation Stakeholder meeting in April 2011

− Full endorsement by the MOH− Partners urged to coordinate around developing a common

approach for assessment of CCM and non-CCM program implementation strength

− Need to allow sufficient implementation time to increase likelihood of impact

− MOH addressed letter to donors requesting support for platform

Partners’ meetings in September and December 2011 to agree on plans for measuring implementation strength

National Evaluation Platform: Progress in Malawi - 2

All partners (SCF, PSI, WHO, UNICEF) actively monitoring CCM implementation in their districts

Funding secured for 16 of 28 districts; additional funding for remaining districts seems probable

Discussions under way about broadening platform to cover nutrition programs

Other countries expressing interest! Mozambique, Bangladesh, Burkina Faso, …

Analysis Plan

“Dose” CCM implementation

strength (per 1,000 pop): CHWs CHWs trained in CCM CHWs supervised CHWs with essential

commodities available Financial inputs

“Response” Change in Tx rates for

childhood illnesses Change in U5M

Contextual FactorsCategories Indicators

ENVIRONMENTAL, DEMOGRAPHIC AND SOCIOECONOMICRainfall patterns Average annual rainfall; seasonal rain patterns Altitude Height above sea levelEpidemics QualitativeHumanitarian crises Qualitative

Socio-economic factors Women’s education & literacy; household assets; ethnicity, religion and occupation of head of household

Demographic Population; population density; urbanization; total fertility rate; family size

Fuel costs! Added as this slowed program implementation in 2010-11

HEALTH SYSTEMS AND PROGRAMSUser fees Changes in user fees for IMCI drugsOther MNCH Health Programs

The presence of other programs or partners working in MNCH

Advantageous context for NEP

strong network of MNCH partners implementing CCM

administrative structure decentralized to districts SWAp II in development now district-level data bases (2006 MICS, 2010 DHS,

Malawi Socio-Economic Database (MASEDA)) DHS includes approx. 1,000 households in each

district

PRACTICALITIES AND LIMITATIONS

Sample sizes must be calculatedon a country-by-country basis

Statistical power (likelihood of detecting an effect) will depend on:– Number of districts in country (fixed; e.g. 28 in Malawi)

– How strongly the program is implemented, and by how much implementation affects coverage and mortality

– How much implementation varies from district to district

– Baseline coverage levels

– Presence of other programs throughout the districts

– How many households are included in surveys in each district

• May require oversampling

Practical arrangements

Platform should be led by credible independent partner (e.g. University or Statistical Office) Supported by an external academic group if

necessary Steering committee with MOH and other relevant

government units (Finance, Planning), Statistical Office, international and bilateral organizations, NGOs, etc.

Main costs of the platform approach

Building and maintaining database with secondary information already collected by others Requires database manager and statistician/epidemiologist for supervision May require reanalysis of existing surveys, censuses, etc

Keeping track of implementation of different programs at district level Requires hiring local informants, training them and supervising their work

Adding special assessments (costs, quality of care, etc) May require substantial investments in facility or CHW surveys

Oversampling household surveys May require substantial investments But this will not be required in all countries

Summary: Evaluation platform

Advantages– Adapted to current reality of

multiple simultaneous programs/interventions

– Identification of selection biases– Promotes country ownership

and donor coordination– Evaluation as a continuous

process– Flexible design allows for

changes in implementation

Limitations– Observational design (but no

other alternative is possible)– High cost particularly due to

large size of surveys• But cheaper than standalone

surveys– Requires transparency and

collaboration by multiple programs and agencies

Thank you