Better nutrition and healththroughimpact evaluation!?

Response to “How to evaluate nutrition and health impacts of agricultural innovations?”

Linxiu Zhang

Center for Chinese Agricultural Policy

Chinese Academy of Sciences

Contents

• Why we want evaluate impacts

• Managing impact evaluations

• Nutrition and health – more than ag. Innovations

• Communicating findings to policy makers –scaling up, REAP example

Challenge/ Excuses

• We don’t have time

• It costs too much to do rigorous impact evaluation

• It is unethical

• Project implementation is completely site and context specific

• We already know!

• So many of the innovations/programs that we heard about on the news …

… how many of the new technologies/programs that we have become excited about …

… how many have been rigorously evaluated?

• Do we have empirical evidence, based on a carefully constructed counterfactuals, that these breakthroughs/programs work, can positively affect the lives of the poor and do so in a cost effective way? Yes but not many

KEY to IE: the attribution problem:factual and counterfactual

Impact varies over time

Impacts also are defined over time … Little attention has been given to the dynamics over time … though people think about this …

With project

Withoutproject

Project impact

Change in the CAL program effect on the standardized math test scores over time

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Baseline (Sept. 1) Midterm (early Nov) Final (late Dec)

Time

Stan

dard

ized

mat

h te

st s

core

The CAL program effect occurred by the midterm evaluation, less than two months after the start of the program.

0

0.120.11

Change in the CAL program effect on the standardized math test scores over time

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Baseline (Sept. 1) Midterm (early Nov) Final (late Dec)

Time

Stan

dard

ized

mat

h te

st s

core

There is no improvement between month two and month three..

0

0.120.11

Impact of nutrition intervention at infancy in Guatemala

• After 2 years � greater BMI

• After 10 years � higher grades in school

• After 15 years � higher school attainment

• After 40 years � higher wages / income

Source: Maluccio et al. 2009

When to use what methods?• RCTs Gold standard

– Where random assignment is possible

• Matching methods– Quasi-experimental methods (construct a

comparison group)– Propensity score matching (PSM)– Regression discontinuity design (RDD)

– ‘Intuitive matching’

• Regression-based/Modeling– Instrumental variables: need to be well-

motivated

Managing Impact Evaluation

• The importance of ex ante designs with baseline (building evaluation into design)- A theory-based approach (using causal chain) allowing analysis of causal pathways

• The importance of a credible design with a strong team.– Clear objectives, simple interventions

• Ensure management feedback loops

Budget and timeline

• Ex post or ex ante?• Existing data or new data?• Qualitative and quantitative?• How many rounds of data collection?• How large is sample?• When is it sensible to estimate impact?

Nutrition and health: morethanag. innovations

Example of China:

•Annual growth rate at 8-10%

•Agricultural growth rate at 3-4%

•High speed of urbanization andindustrialization

•Income for both rural and urban increasedrapidly

•Yet, China’s challenge in nutrition and healthis HUGE

Between 2008 and 2012 we tested nearly 60,000 children across

China for iron-deficiency anemia

In fact, anemia is all over China

Luo, R., X. Wang, C. Liu, et al. (2011) “Alarmingly High Anemia Prevalence in Western China.” Southeast Asian Journal of Tropical Medicine and Public Health Vol. 42 No. 5

Total

Total 33.7Shaanxi—2008 (Dataset 1) 37.5Shanxi—2009a (Dataset 2) 31.6Gansu—2010 (Dataset 3) 31.2Qinghai—2009 (Dataset 4) 51.1Ningxia—2009 (Dataset 5) 25.4Sichuan—2010 (Dataset 6) 24.8Guizhou—2010 (Dataset 7) 33.1

Poor areas of China

Children with anemia (≈ 33%)

Children with out ≈ 30 to 35 million school aged

children are estimated to be suffering from malnutrition!

Non-poor areas

of China

Children with anemia (≈ 8%)

Children with out (92%) < 5 million school aged children in

all of the rest of China

THE SCOURGE WITHIN:INTESTINAL WORMS IN RURAL CHINA

We have tested more than 4000 children for:

Incidences of Intestinal Worms, Guizhou Province, 2010& 2013

3 to 5 year olds 8 to 10 year olds

33.9% with worms

40.1% with worms

Without Without

Zhang et al., 2013… millions of children are infested with these …

Even earlier(malnutrition during the first 1000 days)

• Testing ≈1000 babies and their Mom’s in Southern Shaanxi (11 counties)

(these areas are 2 to 3 hour drive from Xi’an – one of China’s fastest growing cities)

Main Outcomes of InterestAt the time of enrollment and at each follow-up survey, we test babies for:

• Hemoglobin (anemia)

• Length & weight

• Gross & fine motor skills

• Cognitive development

• Social-Emotional development

One of these test is the Bayley Scales of Infant Development

Like an IQ test for babies

Some Initial Results from the Baseline Survey

Surveyed about 1,000 mothers and babies in rural Shaanxi Province

Of the nearly 1,000 babies tested….

55% were anemic� But less than 20% were stunted/wasted, indicating that this

is a micronutrient problem—the babies are getting enough calories, but not enough nutrients

Of the nearly 1,000 babies tested….

35% were cognitively delayed – scored outside of the “normal” range for other babies their age around

the world

Of the nearly 1,000 babies tested….

57% were significantly delayed in their motor development

Ultimate Consequences:

If the micronutrient deficiencies of infants / toddlers are not corrected before baby is 30 months old, there will be permanent and irreversible effects on:

– IQ

– Mental health

– Height

– Weight

– Health

We hope that we can help to address these issues through our intervention project, and ultimately bring national policy attention

to this important problem!

What does this mean?

In harshest terms:

Between 20 to 30 percent of China’s future population are in danger of becoming PERMANENTLY physically and mentally HANDICAPPED

What’s the challenge?1. Rural China’s missing

babies –broken chain – During our survey we

found that many families with babies had moved out of the villages.

– Where did they go?– What is the nutritional

status of “migrant babies”?

– If you cannot find them, how to help?

To sum up the “China story”

• Nutrition and health improvement does notnecessary go hand in hand with income and productivity growth

• We need to identify and map complete causal-chain between agriculture/income –nutrition/health, that would include care,knowledge, health systemreform, genderand governance and MORE

Communicating findings to policy makers- scalingup

Lessdiscussedtopicat thisforum

Find solutions by helping 100s and 1000s …

… and work with the government to help 1,000,000s …

Our experience of doingproject and its evaluationis to:

REAP Experiments (Projects) in China’’’’s Poor Rural Areas (and

Migrant Communities)

>30

REAP has touched the lives of thousands of China's poorest and most needy children

REAPhastestedmorethan schoolkidsonanemia

Passedoutmorethan multivitamins（toaddressanemia）

Testedmorethan schoolkidsforvisionproblems

Passedoutmorethan pairsofglasses

REAPhastestedmorethan ruralkidsonworms

Introducedcomputerassistedlearning(CAL)tomorethan schoolkids

Sofarhasgivenout millionUSDscholarshipstokeepkidsinschool

From Results to Policy Action

Approximately Millionchildreneatdailyegg

Over millionchildreneatafreeheathermealaday

Over kidshavebeenpreventedfromdroopingoutofschoo

schools have introduced CAL into regular clas

Inthepast years,REAPsubmitted policybriefstoGovernmen

Thank you!

Please visit us / contact us:• http://www.reapchina.com

• http://reap.stanford.edu