development impact evaluation in finance and private sector 1
TRANSCRIPT
![Page 1: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/1.jpg)
• Development Impact Evaluation • in Finance and Private Sector
•1
![Page 2: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/2.jpg)
Workshop Development Impact Evaluation in Finance and Private SectorDakar February 2010With generous support from Gender Action Plan
Steps in Implementing an Impact Evaluation
Arianna LegoviniHead, Development Impact Evaluation InitiativeThe World Bank
![Page 3: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/3.jpg)
Steps
![Page 4: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/4.jpg)
Step 1. Build capacity for IE
Objectives: Become informed consumers of impact
evaluation Set the learning agenda Use it as an internal management tool to
improve program over time
How Training Learning by doing
![Page 5: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/5.jpg)
Step 2: Set learning agenda
Objective: Get answers to relevant policy and operational
questions How?
Dialectic discussion involving key policy makers and program managers
Technical facilitation to structure framework of analysis
Focus on few critical policy (what) and operational (how to) questions
Discuss agenda with authorizing environment and constituencies
![Page 6: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/6.jpg)
Cont. 2: Questions Operational: design-choices of program
▪ Institutional arrangements, Delivery mechanisms, Packages, Pricing/incentive
Management purpose Use random trials to test alternatives Measure effects on short term outcomes (months)▪ take up rates, use, adoption
Scale up better implementation modalities
Policy: effectiveness of program Accountability purpose Use random assignment or next best method Measure effects medium to long term Scale up/down, negotiate budget, inform
![Page 7: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/7.jpg)
Step 3: Design IE
Exploit opportunities: Will roll-out take time? Is the budget allocated insufficient to cover everyone? Are there quantitative eligibility rules? If the program has universal access, does it have imperfect
take-up? Set scale:
Pilot to try out an intervention Large scale w. representative sample: more costly, externally
valid Large scale with purposeful sample: less costly, indicative
Do power calculation to determine minimum sample size
![Page 8: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/8.jpg)
Cont. Step 3Select “best” method for each of your
questions Feasible Requires least assumptions
Ethics No to deny access to something for which
there is irrefutable evidence Test interventions before scale up when
you have no solid evidence
![Page 9: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/9.jpg)
Step 4: Planning implementation
Budget cost items▪ Staff time (PROJECT FUNDS) and training (DIME)▪ Analytical services and field coordination (DIME)▪ Data collection (PROJECT FUNDS)▪ Discussions and dissemination (shared)
Timeline▪ Use it to organize activities, responsibilities and work
backwards to know when to start Team
▪ Government (program manager, economist/statistician); WB Project team (Task manager or substitute); Research team (Lead researcher, co-researchers, field coordinator); Data collection agency
![Page 10: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/10.jpg)
Step 5: Assignment to treatment and control The smallest unit of assignment is the unit
of intervention Credit: individual or group
SME services: enterprise
Municipal registration system: municipality
Create listing of treatment units assigned to the intervention and control units that are not
Explain assignment to responsible parties to avoid contamination
![Page 11: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/11.jpg)
Step 6: Baseline data
Quality assurance : IE team (not data collection agency) to Design questionnaire and sample Define terms of reference for data
collection agency Train enumerators Conduct pilot Supervise data collection
Do not collect data before your design is ready and agreed
![Page 12: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/12.jpg)
Cont. Step 6: Baseline data
Contract data collection agency Bureau of Statistics: Integrate with existing
data Ministry concerned: Ministry of
Agriculture/Water Resources/Rural Development Private agency
Analyze baseline data a feed back into program and evaluation design if needed
Check for balance between treatment and control group: do they have similar average characteristics?
![Page 13: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/13.jpg)
Step 7: Roll out intervention
Conduct intensive monitoring of roll-out to ensure evaluation is not compromised
What if treatment and control receive the intervention?
What if all the control group receive some other intervention?
![Page 14: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/14.jpg)
Step 8: Follow-up dataCollect follow-up data with the same
sample and questionnaire as baseline data
At appropriate intervals
![Page 15: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/15.jpg)
Step 9: Estimate program effects
Randomization: compare average outcomes for treatment and control group
Other methods: Use relevant econometric analysis , test assumptions, check robustness
Are the effects statistically significant? Basic statistical test tells whether differences are due to
the program or to noisy data Are they significant in real terms?
If a program is costly and its effects are small, may not be worthwhile
Are they sustainable? Is the trajectory of results sustained?
![Page 16: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/16.jpg)
Step 10: Discuss, Disseminate and Feedback into policy Are you thinking about this only now? Discuss what are the policy implications of the results What actions should be taken How to present them to higher ups to justify
changes/budget/scale up? Talk to policy-maker and disseminate to wider audience If no one knows about it, it won’t make a difference Make sure the information gets into the right policy
discussions Real time discussions Workshops Reports Policy briefs
![Page 17: Development Impact Evaluation in Finance and Private Sector 1](https://reader036.vdocuments.net/reader036/viewer/2022062809/5697bf891a28abf838c89c99/html5/thumbnails/17.jpg)
Final step: Iterate
What do you need to learn next?