2012 ees peace precarious workshop full slides
TRANSCRIPT
M&E and evaluation for development
in peace precarious situations
Catherine Elkins, PhD, MALD Belling the Cat LLC–Duke University EES Workshop 01 October 2012
Assisting development: Is our territory cognita?
2
Our day’s agenda
• Sessions and schedule in handouts – Strategic contingency approaches – Case studies over time – Six-dimensional analytical model – Lessons for peace precarious situations and lessons
from peace precarious situations
What is ‘normal’ development?
• “Think–pair–share” • Define
– elements of typical development context – typical development goals – typical development stakeholders
– typical development implementers / partners
Peace Precarious Situations
photo © Joe Holmes 5
What is ‘peace precarious’ development?
• Differences we experience in
– context – goals – stakeholders
– implementers / partners
Peace precarious situations
7
• May overlap circumstances labeled in other discussions as fragile, low-level violence, conflict, post-conflict, failed states
Peace precarious situations
8
• Power disputes • recent, ongoing, or chronic • violent at least sometimes
• Destabilized but not obliterated core institutions
• No directional assumptions • No ‘state’ assumptions
Introductions: Catherine Elkins
• http://linkedin.com/in/seaelkins • Professor, Duke University; Consultant • About 20 years’ experience working in
developing countries • Afghanistan, Cambodia, Eritrea, Guinea,
Guyana, Iraq, Kazakhstan, Lebanon, Mauritania, Pakistan, Rwanda, South Africa, Uganda, West Bank, Zimbabwe
• Belize, Jordan, Senegal, Tanzania, Thailand
Introductions: Colleagues
• Name, main occupation • Evaluation (or M&E) experience • Development experience • Peace-precarious experience
Introductions: Cases 1, 2
11
Introductions: Case 3
Development in peace precarious situations
USAID 2006, Provincial Reconstruction Teams in Afghanistan: an Interagency Assessment (PN-ADG-252) . 13
• Extreme cases: International development efforts ongoing despite widespread internal or regional areas characterized by a “mid-range of violence”
• “Instability still precludes heavy NGO involvement, but […] is not so acute that combat operations predominate”
Critical dimensions
14
Key aspects for M&E/Evaluation • Volatility • Internal and external ‘constants’ are
potentially/probably variable • And especially: Multiple parameters can
change at once
Peace precarious situations
Activity
• Afghanistan – external evaluation – What do we know about the 6 dimensions? – What else do we need to know?
– Ideas for evaluation plan • In principle • 20 questions
Focus on stakeholders
17
Stakeholder complexity & sensitivities
Sources for volatility • Who has a stake in project failure?
• Who has mixed interests, hedged bets?
• What additional political or other concerns do those supporting success need to consider?
18
Lunch
19
Theories of change
• Hypothesized relationships among influential factors shaping conditions or behavior of interest
• Probabilistic causal expectations • Observable, measurable, falsifiable model • Mechanisms expected to produce results
20
Implementation – or so we like to think
21
Theories of change and peace precarious development
• “Normal” development interventions – Complicated – Complex
• “Peace precarious” interventions – Complicated, complex, and volatile – Relationships between good program design &
good M&E/evaluation design in peace-precarious situations
22
Peace precarious stakeholders and planning
• From initial exercise • From Afghan case exercise • From own experiences
• Relationship dynamics and contingencies
Activity
• Guinea governance M&E – Dynamics over time
– http://www.reuters.com/article/2010/06/27/us-guinea-events-idUSTRE65Q0ZP20100627
Project spr’08
Army mutiny sum‘08
Coup Dec’08
Massacre Sep’09
Coup attempt Dec’09
2 intra-coup
transitions Jan-Feb’10
Pres election June’10
Run-off results final
Nov’10
Legislative elections fall ’11** (not yet!)
Activity (cont’d)
• Guinea – Initial design – Big disruption – Ongoing disruptions
M&E strategy
• Guinea governance M&E – Model and measurement
– Elasticities
Afghanistan and Guinea
• M&E and evaluation dynamics – Same or different challenges?
– Documentation
Model (theory of change) and measurement
28
‘Normal’ vs peace precarious challenges • If–then hypotheses of predicted
relationships – Falsifiable using empirical evidence – Discriminating paths among multiple influences – Quasi-experimental counterfactual; may be very
loose “other things being equal”
• Baseline, implementation, results
Fidelity and slippage
• Contextual constants and variables • Cost and implementation elasticities • Articulation of expectations, execution, M&E,
and evaluation including changes from design through close-out
• Learning systems
29
Egypt, girls’ education, and “revolution”?
globalgiving.org
31
Punctuated equilibria • Full or partial disruptions, interruptions,
unpredictable timing • Expansions and curtailments of components,
flexible scope • Nature and timing of specific shocks or factor
changes (context, institutions, stakeholder/counterpart composition)
Recap: Exploring the cases
Exploring the cases – coping under stress
32
Cost factors • Fluctuation in budget • Fluctuation in security expenses • Staffing through interruptions, evacuations • Recruiting, screening, incentives, and turnover
Measurement risk factors
33
Data quality and precision • Human capital and stress effects on timeliness,
completeness, accuracy • Perceived risk and risk attitudes
– Risks to participants, beneficiaries, enumerators
– Transparency and accountability risks • Existence, accessibility, reliability of secondary
data sources
Information uses
Lower confidence in data and…
• Potential for political or other unintended uses of project results and evaluation findings?
• Likelihood that certain actors and audiences will misunderstand or misuse findings?
• Potential damage to any stakeholders from use or misuse of results or findings?
34
Information and stakeholders in context
• International context and global dynamics – Goals, uses, audiences, information/security
dimensions affecting development results information
– Six dimensions across our cases
– Appropriate, useful M&E/evaluation? • Possible? • Likely? • Under what conditions?
35
Cross-cutting conclusions and challenges
36
Gaps analysis and theory building
• What are we missing?
• What else can we bring to bear?
37
Strategy
38
Where program and evaluation concerns meet • Design evaluation around challenges to project
and ways challenges are most likely to affect evaluation
• Identify specific decision points – More than one theory of change? – Client understanding or learning curve? – Variance across stakeholder interests? – Data needs for methodology options? – Budget requirements for strategic options?
Design for development
39 handout
Useful M&E/evaluation (peace precarious)
• Questions: – Adequacy of development models for (some)
peace-precarious situations – Adequacy of M&E/evaluation approaches for
(some) peace-precarious situations – Managing expectations and validating deliverables – Ethics and other issues
40
Final Questions?
41
Thanks!