1

Indicators for Malaria Impact Evaluation

Impact Evaluation TeamImpact Evaluation Team

2

Malaria Working Groups

1. Biometrics Working Group2. Cognitive and Educational Working Group3. Socio-Economic Working Group (+ KAP)4. Cost Effectiveness

3

Malaria Impact Evaluation Team

Cluster Coordinating

Team

PROGRAM COORDINATION & HARMONIZATION

Country 1 Project

Coordinator

MA

LA

RIA

& IE

EX

PE

RT

S

Technical Advisory Group

OPERATIONAL RESEARCH

Associate

Researcher

CASE COUNTRY I : OR IMPLEMENTATION - FIELD WORK Local Research Partner (Government Agency, Academia, NGO)

Embedded Field Research Coordinator (Liaison)

FIE

LD

OP

ER

AT

ION

IN C

AS

E C

OU

NT

RY

PROJECT MANAGEMENT

CLIENT / POLICY LINK & PROJECT IMPLEMENTATION

Working Group 1:

Biometrics

Working Group 2: Cognitive

Working Group 3: Socio-

Economic

Working Group 4:

KAP

Working Group 5:

Cost Effectiveness

s

CLIENT DEVELOPED IE PROGRAM

GOVERNMENT PROGRAM TEAM MIEP RESEARCH TEAM

IE TEAM WORKING WITH CLIENT METHODOLOGY / IE QUALITY

COUNTRY-SPECIFIC TEAM

4

Developing a Common Approach to Measuring the Biometric Impact

of Malaria Control InterventionsBiometrics Working GroupBiometrics Working GroupMalaria Impact Evaluation

Joseph KeatingJoseph Keating (Tulane University) Simon BrookerSimon Brooker (LSHTM)

5

Malaria Impact Indicators I: Parasitaemia and DiseaseMalaria Impact Indicators I: Parasitaemia and DiseasePopulations: -Children < 5 years old

-Pregnant women

-Population in malarious areas

– Data source: population based household survey, HMIS – high versus low transmission seasons; stable versus unstable transmission areas

– Diagnostic method: finger-prick, thick and thin blood smear for microscopy (Gold Standard) or Rapid Diagnostic Test (RDT) kit

Indicators: -Prevalence of malaria parasite infection (< 5 years old/all ages)

-All cause mortality in children < 5 years old

-Laboratory confirmed malaria death rate (< 5 years old/all ages)

-Malaria incidence

Costs: -RDT: USD $1-3 plus cost of training personnel; Microscopy (Gold Standard): varies as a function of existing equipment, reagents, and trained personnel

6

Malaria Impact Indicators II: AnemiaMalaria Impact Indicators II: Anemia

Populations: Children 6-59 months

Pregnant women

Schoolchildren

– Data source: population based household survey, clinic based survey, school survey

– Diagnostic method: finger-prick blood sample, portable Hemocue machine

Costs: $0.5/sample

Accuracy: 0.1 g/L

Anaemia definition: age specific, e.g. 110g/L (under 5s); 115-120 g/L (school-age children)

Alternative methods: Haemoglobin Colour Scale

finger-prick blood sample, special chromatography paper ($0.05/sample but accuracy only to 10 g/L

therefore unsuitable for impact evaluation)

7

Developing a Common Approach for Cognitive and Educational Assessments

Cognitive and Educational Working GroupCognitive and Educational Working Group Malaria Impact Evaluation

Matthew JukesMatthew Jukes (Harvard University) Don BundyDon Bundy (World Bank)

8

0

0.1

0.2

0.3

0.4

0.5

0.6

3 yrs in program 4 yrs in programImp

rov

em

en

t in

Co

gn

itiv

e F

un

cti

on

(S

Ds

)

p=.08

p=.01

Impact of Early Childhood Malaria Prevention on Global Cognitive Function

Jukes et al PLOS clinical trials 2006

9

Can IPT in schoolsreduce parasitaemia

and anaemia and improve school

performance?

A randomised controlled trial of IPT using SP+AQ in 30 primary schools in

western Kenya

Malaria Infection

in Semi-immune Schoolchildren

Clinical Attack

Asymptomatic Parasitaemia

Anaemia

Absent from School

Reduced Attention

During Lessons

Educational Achievement

Most common Less common

IPT

10

Impact of IPT on sustained attention and education?

Clinical AttackAnaemia

Absent from School

Reduced Attention

During Lessons

Educational Achievement

Outcome n Mean difference

95% CI p-value Effect size

Counting sounds (max score=20)

481 2.12 (-0.17, 4.42) 0.07 0.65

Code transmission (max score=40)

469 7.74 (2.83, 10.65) 0.005 1.01

Exam score 6 286 0.55 (-2.26, 3.36) 0.35 0.15

Exam score 7 266 0.69 (-0.93, 2.15) 0.21 0.30

Clarke et al. forthcoming

11

Language Differences in Cognitive Tests Performance

0%

10%

20%

30%

40%

50%

60%

Mandinka Wollof

Dig

it S

pan

% C

orr

ect

Digits 1 to 5

Digits 1 to 9

Jukes et al. forthcoming

12

13

Developing a Common Approach to Measuring the Socio-Economic Impact of

Malaria Control Interventions

Socio-Economic Working GroupSocio-Economic Working Group Malaria Impact Evaluation

Jed FriedmanJed Friedman (World Bank) Edit V. VelenyiEdit V. Velenyi (World Bank)

14

From Data ….. To Impact

Savigni and Binka (2004)

DataData

ActionAction

InformationInformation

EvidenceEvidence

KnowledgeKnowledge

ImpactImpactPathway

for

Evidence-based

Planning

Organize Integrate Analyze

(MIS)

Package & Communicate to

Planners & Stakeholders (MIS)

Package (MIS)

Influence the Plan (Planners)

Implement the Plan (System)

Monitor Change in Indicators

and Forecast (M&E)

15

But what data for IE?

• “The data we have are not the data we want.”

• “The data we want are not the data we need.”

• “The data we need are not available.”

• How do we then measure impact?

• What impact do we measure?

• How precise is what we measure?

Quotes: Savigni and Binka (2004)

16

Factors Influencing Malaria Burden

Jone and Williams (2004)

Knowledge Attitude and Practice (KAP)Knowledge Attitude and Practice (KAP)

Underlying Health Status

Endemicity Immunological Status

Socio-Economic Socio-Economic StatusStatus Social

OrganizationCultural Roles

Cultural Beliefs

Observed Disease Burden

17

Berman, Alilio, and Mills (2004)

Health Links to GDPMacro Economic Impact: Poor health reduces GDP per capita by reducing both labor productivity and the relative size of the labor force.

Higher Fertility andChild Mortality

Child Illness

Reduced Investment in Physical Capital

Reduced Schooling & Impaired Cognitive

Capacity

Labor Force Reduced byEarly Mortality

Higher Dependency Ratio

Adult Illness & Malnutrition

Child Malnutrition Reduced Labor Productivity

Reduced Access to Resources & Economy

Lower GDP perLower GDP perCapitaCapita

18

Data Sources

Savigni and Binka (2004)

Type

Level Cross-Sectional Retrospective Longitudinal Prospective

Individual and HH

Population Survey

(Census, DHS, MICS)

Prospective Surveillance

(Vital events and DSS)

Health Facility

Routine Reporting

(HMIS, IDS, DHS) HF Survey

Modeling Risk Mapping (GIS)

Remote Sensing and Early Warning Systems

DHS = Demographic and health Survey, MICS = Multi-Indicators and Cluster Survey, DSS = Demographic Surveillance System, HMIS = Health Management Information System, IDS = Integrated Disease Surveillance, HF = Health Facility, GIS = Geographic Information System

Types and Levels of Data for Health Information Systems Important for Malaria Control Programs

19

Conceptual FrameworkEconomic Burden of Illness for HHs

Russell (2004)

Health Health SystemSystem

Box 2:Treatment Behavior

Box 3a: Direct Costs

Box 3b: Indirect Costs

Box 4: Coping Strategies(Risky, less risky)

Box 1:Reported Illness

Social Social ResourcesResources

Box 6:

Access,fees, quality ofcare, insurance

Box 7:

SocialNetworks

Box 5: Impact on Livelihood(Assets, income, food security)

Individual & HouseholdIndividual & Household

20

Weak / Missing Link …Biomedical & Socio-Economic

• Asset v. Consumption Module• Health Care Seeking & Expenditures• Copying Mechanisms & Poverty • Labor Market / School Participation• KAP

– Community Effects– Social Norms (Gender, Vulnerability)

21

Weak / Missing Link … (2)Biomedical & Socio-Economic

Some Operational / Technical Issues• Are the questions tailoredtailored to capture the intervention? Is our approach parsimonious?

• Should the samplesample be expandedexpanded?

• What is our knowledge gainknowledge gain, and the marginal cost of the informationcost of the information?

• Are we gaining predictive powerpredictive power and making a good Biomedical-SE linklink?