Gender and Impact Evaluation

Urmy Shukla

Capacity Building Manager

CLEAR South Asia / J-PAL South Asia at IFMR

Overview

What is impact evaluation? What is a randomized evaluation?

How can we use impact evaluation as a tool for gender policy and advocacy?

How can gender be successfully integrated into impact evaluation?

Examples from J-PAL

J-PAL’s Approach

J-PAL is a Centre at MIT’s Department of Economics, with Regional Offices Worldwide

Impact is defined as a comparison between:

• The outcome some time after the program has been introduced

• The outcome at that same point in time had the program not been introduced

This is know as the “Counterfactual”

How to Measure Impact?

Impact: What is it?

Time

Pri

mary

Ou

tcom

e Intervention

Impact: What is it?

Time

Pri

mary

Ou

tcom

e

ImpactCounterfactual

Intervention

Impact: What is it?

Time

Pri

mary

Ou

tcom

e

ImpactCounterfactual

Intervention

Randomised Evaluations

Individuals, villages, or districts are randomly selected to receive the treatment, while other villages serve as a comparison

Treatment Group

Comparison Group

Village 1Village 2

=

Groups are Statistically Identical before the Program

Any Difference at the Endline can be Attributed to the Program

Two groups continue to be identical, except for treatment. Later, compare outcomes (health, test scores) between the two groups. Any differences between the groups can be attributed to the program.

How can we use impact evaluation as a tool for gender policy and advocacy?

Surprisingly little hard evidence on what works, coupled with popular notions on how to combat gender issues:

• Example: Toilets and gender-based violence

The growing role of impact evaluation in government and international development

• Positive evidence encourages funding, advocacy, and programme expansion

How can gender be successfully integrated into impact evaluation?

Interventions targeting women and girls with easily quantifiable outcomes (education levels, employment, fertility rates, etc.)

Gender-based or gender-defined outcomes

• More difficult, but possible

Examples of J-PAL Research: Targeted Interventions

Effect of a merit-based girls' scholarship program on attendance and academic achievement in Kenya

• Objective: Increase girls’ enrolment and reduce drop-out in middle school

• Program: Girls’ Scholarship Program carried out in randomly selected schools. High ranking girls receive scholarship for school fees, grant to the family for school supplies, public recognition

• Findings: Improvements in performance, attendance and parental involvement. Positive externalities also for girls with low chance of winning the award. High ranking girls won the scholarship, but effects were also seen in non-winners.

• Program by International Child Support Africa.

Examples of J-PAL Research: Targeted Interventions

Menstruation and Education in Nepal

• Objective: Increase girls’ enrolment and reduce drop-out in 7th and 8th standard.

• Program: Girls’ randomly selected to receive menstrual products and instructions.

• Findings: No impact on school attendance, no impact on test scores. Positive impact on time-saved (20 minutes/day).

• Project Partner: Institute for Social and Environmental Research (ISER-Nepal)

Example of J-PAL Research: Gender-based outcomes

Empowerment of Adolescent Girls in Rural Bangladesh

Objective: Empowerment and holistic development of adolescent girls, increased age of marriage, changes in attitudes and aspirations

Program:

• Literacy and numeracy training

• Financial skills: entrepreneurship, budgeting, etc.

• Food incentive to delay marriage

Outcomes of interest: education, health, nutrition, income generation, savings/investment, decision-making, attitudes, awareness, marital and reproductive outcomes.

Empowerment of Adolescent Girls in Rural Bangladesh

Evaluation Approach – How to measure empowerment?

1) In-depth situation analysis by Save the Children

• A review of government policies

• Qualitative assessment of adolescent girls: What is required for ‘empowerment’ Participation in community & household

decision-making Mobility Education Skills for earning an income

Empowerment of Adolescent Girls in Rural Bangladesh

Evaluation Approach – How to measure empowerment?

2) Development of survey instruments by J-PAL affiliate and researchers

• Comprehensive questions on decision making: Ability to travel to local event Medical decisions Mental health indices AspirationsEtc.

Gender and Impact Evaluation: Conclusions

Impact Evaluation can be a powerful tool for planning and advocacy

Impact Evaluation does not negate the need for more in-depth gender analyses

It is important to foster cross-disciplinary and interdisciplinary collaborations