how (does) the world bank learn from its operations? john heath independent evaluation group...

How (Does) the World Bank Learn from its Operations?

John HeathIndependent Evaluation Group

Washington, DCJuly 2015

Connect with us online: ieg.worldbank.org

@WorldBank_IEG#WBLearns #WhatWorks

/IndependentEvaluationGroup

2

Overview of the

PresentationI. Learning>Results: Evidence

II. Evaluation Approach

III.Findings

IV.Recommendations

V.Questions

How (Does) the World Bank

Learn from its Operations?

I. Learning>Results: Evidence Are there credible proxies?

Denizer et al, 2013 & Geli et al, 2014 examine effect of TTL Quality on IEG Outcome rating of project. QUERY: Is TTL Quality a proxy for learning?

Haas, 2006 examines effect of project team member’s N of Years the World Bank on the QAG Quality at Entry rating of project. QUERY: Is N of Years a proxy for tacit knowledge?

IEG, 2014 content analyzes IEG project assessments to assess link between use of lessons learned during preparation and IEG Quality at Entry rating. QUERY: Is Use of Lessons a proxy for tacit

knowledge?



I. Learning>Results: EvidenceTTL Quality Drives the IEG “Outcome Rating

Source: Denizer et al, 2013 (N=6,569 World Bank projects)+/- (Positive or Negative Correlation/Magnitude) * (Statistical significance)

STRONGER PREDICTORS WEAKER PREDICTORS

TTL Quality +++*** Real GDP per capita growth +*CPIA Rating ++*** Multisector +* Repeater project + Size of loan/credit -* Preparation cost - Supervision cost -* Lag between approval and effectiveness -* Length of implementation - Problem flags -* TTL Turnover -* Restructuring +*



I. Learning>Results: EvidenceYears of WB Staff Experience Influences Quality at Entry

6



I. Learning>Results: EvidenceUse of Lessons Influences Quality at Entry

Source: IEG, Results and Performance of the World Bank Group 2014

QAE (MU- Projects)

N=32

QAE(MS+

Projects)N=100

Top 3 positive attributes

1. Active use of lessons 25% 50%

2. Adequate technical analysis 22% 50%

3. Good relations with stakeholders 25% 37%



I. Learning>Results: Evidence

Lessons are Context Specific

“Most of the well-designed projects that were found to use past lessons more frequently draw on lessons embedded in specific country context and less frequently on generalized (i.e., regional and global) lessons.”

“Lessons from experience or analytical work that are specific and easier to operationalize appear to be most relevant.”

(IEG, Results and Performance of the World Bank Group 2014)



I. Learning>Results: EvidenceDrivers of Quality at Entry

(IEG, Results and Performance of the World Bank Group 2014)

0.15 0.2 0.3 0.4 0.5 0.7 1 1.5 2 3 4 5 7 10Propensity to Satisfactory Quality at Entry

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Frequency of factor

Risks - POS

Analysis - Political Economy - POS

Analysis - Institutional - NEG

Analysis - Institutional - POS

Analysis - Political Economy - NEG

Analysis - Technical - NEG

Analysis - Technical - POS

Bank Team Support - NEG

Bank Team Support - POS

Government Commitment - NEG Government Commitment - POSLessons - NEG

Lessons - POS

Relationships - NEG

Relationships - POS

Risks - Mitigating Measures - NEG

Risks - Mitigating Measures - POSRisks - NEG

Technical - M&E Design - NEG

Technical - M&E Design - POS

Technical - Objectives & Results Framework - NEG

Technical - Objectives & Results Framework - POS

More frequentfailure factors

More frequentsuccess factors

Less frequentsuccess factors

Less frequent failure factors



II. Evaluation ApproachThe Influence Nexus

Thank you!

Don’t forget to connect with us online for more information: Website: ieg.worldbank.org

@WorldBank_IEG#WBLearns #WhatWorks

/IndependentEvaluationGroup

how (does) the world bank learn from its operations? john heath independent evaluation group...

Documents

world bank group

ieg quality

entry ieg

ieg outcome rating of

qag quality

evidencedrivers of quality

effect of ttl quality

ieg project assessments