A Premier Evaluation Agency: USAID Progress in Capacity

Building

Cindy Clapp-WincekDirector, Office of Learning, Evaluation & ResearchBureau for Policy, Planning and LearningUSAIDJune 19, 2013

1

Guidance

2

Technical Support

• TA and advice from LER staff • Program Cycle Service Center • ProgramNet and Learning Lab

http://programnet.usaid.gov/ www.usaidlearninglab.org

3

• Special Collection on Evaluation http://dec.usaid.gov

• Evaluation Showcasewww.usaid.gov/evaluation

• Transparency – Evaluation Registry tracks evaluations and budgets– Waivers for public posting on DEC

Systems

                                

4

3

Year Number of DEC

Documents Verified as Evaluations

Number of Verified Evaluations Coded

Included in the Meta-Evaluation

Statistical Characteristics of Samples Drawn

Separately for Each Year of the Meta Evaluation

2009 112 73

For each sample year: Confidence Level: 85% Margin of Error: +/- 5%

2010 142 85

2011 154 89 2012 165 93

Combined 573 340 Confidence Level: > 99% Margin of Error: +/- 5%

Progress

5

3

The Long March Progress

6

3

7

3

 CollectionMethods

Evaluation Described Plans to

Use the Method

Report Review Found Evidence

of Use of the Method

Difference between Plan to Use and

Actual Use

Percentage of Evaluations that

Demonstrated Use of the Method

USAID Performance Data 243 285 +42 (117%) 84%

Document Review 252 274 +22 (109%) 81%

Key Informant Interviews 261 245 -16 (94%) 72%

Individual Interviews 187 185 -2 (99%) 54%

Unstructured Observation 156 152 -4 (97%) 45%

Survey 143 118 -25 (83%) 35%

Focus Group 147 100 -47 (68%) 29%

Structured Observation 24 26 +2 (108%) 8%

Group Interview 64 32 -32 (50%) 9%

Instruments (e.g., scale) 9 11 +2 (122%) 3%

Community Interview 5 3 -2 (60%) 1%

Planned and Actual Use of Evaluation Data Collection Methods

8

3

Elements an Evaluation Client Determines Elements an Evaluation Team Provides

Scope of the evaluation – single or multiple projects or programs

Timing of the evaluation – during implementation, towards the end of a project, and evaluation schedule

Management purpose – improve performance, generate lessons

Type of evaluation sought – performance, impact

Evaluation questions – number and types Team composition – external evaluation

team leader, evaluation specialist, local evaluators

Identification of deliverables, and the transmission of Agency evaluation quality standards

Duration – number of weeks or months Evaluation budget Evaluation quality control activities,

including evaluation product reviews

Executive Summary – degree to which it accurately mirrors most critical elements of the report

Presentation of Project or Program Background – completeness from a reader’s perspective

Description of the Project or Program’s “Theory of Change” – development hypotheses

Presentation of the Evaluation Questions – consistency with SOW, completeness

Description of the Data Collection and Analysis Methods Used – specificity, links to questions

Description of the Study Limitation Findings, Conclusions and

Recommendations – clear distinctions among them, logical flow

Annexes -- completeness

9