national evaluation platforms: a solution that serves...
Post on 16-Mar-2020
1 Views
Preview:
TRANSCRIPT
National Evaluation Platforms: A Solution that Serves
Governments and their Partners
Robert Black
Institute for International Programs Johns Hopkins Bloomberg School of
Public Health
FIRST GLOBAL SYMPOSIUM ON HEALTH SYSTEMS RESEARCH
Outline
1. Why a new approach is needed
2. National Evaluation Platforms (NEPs):
The basics
3. Country example: Malawi
4. Practicalities and costs
Most current evaluations of large-scale
programs aim to use designs like this
Impact
Coverage
Program
No impact
No coverage
No program
But reality is much more complex
General socioeconomic and other contextual factors
Impact
Coverage
Routine health
services Interventions in
other sectors
Other health
programs
Program
Other health
programs
New evaluation designs are needed
Large-scale programs
Evaluators do not control
timetable or strength of
implementation
Multiple simultaneous
programs with overlapping
interventions and aims
Contextual factors that cannot
be anticipated
Need for country capacity and
local evidence to guide
programming
Lancet, 2007
Bulletin of WHO, 2009
Sources: Victora CG, Bryce JB, Black RE. Learning from new initiatives in maternal and child health. Lancet 2007; 370 (9593): 1113-4.
Victora CG, Black RE, Bryce J. Evaluating child survival programs. Bull World Health Organ 2009; 87: 83.
NATIONAL EVALUATION
PLATFORMS: THE BASICS
Lancet, 2010
Source: Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the MDG era and beyond: A new
approach to large-scale effectiveness evaluations. Lancet, published on line 9 July 2010.
Builds on a common evaluation
framework, adapted at country level
Common principles (with IHP+, Countdown, etc.)
Standard indicators
Broad acceptance
Evaluation databases
with districts as the units
District-level databases covering the entire country
Data for standard impact pathway:
Inputs (partners, programs, budget allocations, infrastructure)
Processes/outputs (DHMT plans, ongoing training,
supervision, campaigns, community participation, financing
schemes such as conditional cash transfers)
Outcomes (availability of commodities, quality of care
measures, human resources, coverage)
Impact (mortality, nutritional status)
Contextual factors (demographics, poverty, migration)
Permits national-level evaluations
of multiple simultaneous programs
Types of comparisons
supported by the platform approach
Areas with or without a given program
Traditional before-and-after analysis with a
comparison group
Dose response analyses
Regression analyses of outcome variables according
to dose of implementation
Stepped wedge analyses
In case program is implemented sequentially
Evaluation platform Interim (formative) data analyses
• Are programs being deployed where need is greatest?
– Correlate baseline characteristics (mortality, coverage, SES,
health systems strength, etc) with implementation strength
– Allows assessment of placement bias
• Is implementation strong enough to have an impact?
– Document implementation strength and run simulations for likely
impact (e.g., LiST)
• How best to increase coverage?
– Correlate implementation strength/approaches with achieved
coverage (measured in midline surveys)
• How can programs be improved?
– Disseminate preliminary findings with feedback to government
and partners
(All analyses at district level)
Evaluation platform Summative data analyses
Did programs increase coverage?
– Comparison of areas with and without each program over time
– Dose-response time-series analyses correlating strength of
program implementation to achieved coverage
Was coverage associated with impact?
– Dose-response time-series analyses of coverage and impact
indicators
– Simulation models (e.g. LiST) to corroborate results
Did programs have an impact on mortality and nutritional
status?
– Comparison of areas with and without each program over time
– Dose-response time-series analyses correlating strength of
program implementation with impact measures
COUNTRY EXAMPLE
MALAWI
Simultaneous
implementation of multiple
programs
Separate, uncoordinated,
inefficient evaluations (if
any)
Inability to compare
different programs due to
differences in
methodological
approaches and indicators
Malawi
Community case-management
(CCM) for childhood illness
Region and districts
CCM Partners
PMNCH MSH/
BASICS SAVE PSI UNICEF # CCM
partners NORTHERN REGION Chitipa 1 Karonga 1 Mzimba 1 Nkhata Bay 1 Rumphi 1 Likoma 1
CENTRAL REGION Dedza 1 Dowa 1 Kasungu 2 Lilongwe 1 Mchinji 1 Nkhotakota 2 Ntcheu 1 Ntchisi 1 Salima 1
SOUTHERN REGION Balaka 2 Blantyre 1 Chikwawa 1 Chiradzulu 1 Machinga 1 Mangochi 1 Mulanje 1 Mwanza 1 Nsanje 2 Phalombe 2 Thyolo 1 Zomba 2 Neno 1 Total districts
supported 10 8 6 5 4
Policy = rapid scale-up
of CCM
MOH identified ≥ 1
partner to support in
each district
“Comparison” districts
therefore not available
But implementation is
likely to be uneven,
allowing dose-response
analyses
Advantageous context for NEP
strong network of MNCH partners implementing
CCM
administrative structure decentralized to 28
districts
SWAp
district-level data bases (2006 MICS, 2010 DHS,
Malawi Socio-Economic Database (MASEDA))
DHS includes approx. 1,000 households in each
district
Analysis Plan
“Dose”
CCM implementation
strength (per 1,000 pop):
+ CHWs
+ CHWs trained in CCM
+ CHWs supervised
+ CHWs with essential
commodities available
Financial inputs
“Response”
Change in Tx rates for
childhood illnesses
Change in U5M
PRACTICALITIES
AND LIMITATIONS
Sample sizes must be calculated
on a country-by-country basis
Statistical power (likelihood of detecting an effect) will
depend on:
Number of districts in country (fixed; e.g. 28 in Malawi)
How strongly the program is implemented, and by how much
implementation affects coverage and mortality
How much implementation varies from district to district
Baseline coverage levels
Presence of other programs throughout the districts
How many households are included in surveys in each district
• May require oversampling
Main costs of the platform approach
Building and maintaining database with secondary information
already collected by others
Requires database manager and statistician/epidemiologist for
supervision
May require reanalysis of existing surveys, censuses, etc
Keeping track of implementation of different programs at
district level
Requires hiring local informants, training them and supervising their
work
Adding special assessments (costs, quality of care, etc)
May require substantial investments in facility or CHW surveys
Oversampling household surveys
May require substantial investments
But this will not be required in all countries
Evaluation platform
Advantages – Adapted to current reality of
multiple simultaneous
programs/interventions
– Identification of selection
biases
– Promotes country ownership
and donor coordination
– Evaluation as a continuous
process
– Flexible design allows for
changes in implementation
Limitations
– Observational design (but
no other alternative is
possible)
– High cost particularly due
to large size of surveys
• But may be less than
several standalone surveys
– Requires transparency and
collaboration by multiple
programs and agencies
Platform design overview
Design element Data sources (sample = 28 districts)
Documentation of program implementation and contextual factors
Full documentation every 6 months through systematic engagement of DHMTs
Quality of care survey at 1st-level health facilities
Existing 2009 data to be used for 18 districts; repeat survey in 2011
Quality of care at community level (HSAs)
Desirable to conduct in all 28 districts (Not included in this budget proposal)
Intervention coverage DHS 2010, with samples of 1,000 households representative at district level in all 28 districts
DHS/MICS 2014 with samples representative at district level in all 28 districts
Costs Costing exercises in ≈ 1/3 of districts distributed by region and chosen systematically to reflect differences in implementation strategy or health system context
Impact (under-five mortality and nutritional status)
End-line household survey (MICS or DHS?) in 2014
Modeled estimates of impact based on measured changes in coverage using LiST
Average baseline coverage level (% of children with
suspected pneumonia treated with antibiotics) 30%*
Standard deviation of baseline average coverage 13 pp*
Coefficient of variation of baseline average coverage 0.42*
Average endline coverage (assumed based on target set by
country) 67%
Standard deviation of endline coverage (assuming same
coefficient of variation as in baseline) 28 pp
Standard deviation of change (Y variable) 20 Pp
Assumed 10 percentage point increase in implementation
strength score leads to 7% change in coverage 1 = 0.7
Standard deviation of implementation strength score 20 pp
ASSUMPTIONS
* Based on the latest DHS results
Sample size calculations
(dose-response analysis)
26 districts required if 10 pp increase in
implementation leads to 5 pp coverage increase
Number Districts Slope
(N) (B) 191 0.2
82 0.3 44 0.4
26 0.5 17 0.6 11 0.7
7 0.8 5 0.9
Alpha = 5%; power = 80%
SD(y) = SD(x) = 20 pp
N vs B with SX=20.00 SY=20.00 Alpha=0.05 Power=1.00T-Test
N
B
0
50
100
150
200
0.2 0.4 0.6 0.8 1.0 1.2
Contextual Factors
Categories Indicators ENVIRONMENTAL, DEMOGRAPHIC AND SOCIOECONOMIC
Rainfall patterns Average annual rainfall; seasonal rain patterns
Altitude Height above sea level
Epidemics Qualitative
Humanitarian crises Qualitative
Socio-economic
factors
Women’s education & literacy; household assets;
ethnicity, religion and occupation of head of household
Demographic Population; population density; urbanization; total
fertility rate; family size
HEALTH SYSTEMS AND PROGRAMS
User fees Changes in user fees for IMCI drugs
Other MNCH Health
Programs
The presence of other programs or partners working in
MNCH
Practical arrangements
Platform should be led by national academic
institution (e.g. University or Statistical Office)
Supported by an external academic group if
necessary
Steering committee with MOH, Statistical Office,
international and bilateral organizations, NGOs, etc
top related