11

Emerging Agenda and Emerging Agenda and Challenges for Evaluation in Challenges for Evaluation in

the 21the 21stst Century CenturyBy By

Susan D. TamondongSusan D. TamondongVice-President, IDEASVice-President, IDEASsusan.tamondong@yahoo.comsusan.tamondong@yahoo.com

www.ideas-int.orgwww.ideas-int.org

Presentation at the APEA WorkshopPresentation at the APEA WorkshopManila, PhilippinesManila, Philippines27 November 201227 November 2012

Sponsored by ODA , MOFA, JAPANSponsored by ODA , MOFA, JAPAN

22

BackgroundBackground

• Once upon a time, the role of evaluator was simple—as long as development was defined in terms of aid and projects, the evaluator’s focus was on well demarcated and constructed set of factors.

• But as understanding of development has been broader and more sophisticated, this task became much more complex.

• Changes in development thinking and practice led to shift in evaluation approach the last 15 years- from an

assessment of outputs to evaluation of RESULTS

33

Early YearsEarly Years

• Project Model – dominated development thinking and practice until 1990s

• Donor driven

• Deliverables set in logical framework

• Little attention to impact assessment

• Focus on whether deliverables were met and assumptions held

44

Emerging Agenda begun at the end Emerging Agenda begun at the end of Cold Warof Cold War

• Complex ways of conceptualizing the development process demands sophisticated and interpretive approaches to evaluation

• New trends• New methodological challenges• Partnerships• Sector wide approaches (SWAPs)• General Budget Support• Policy Coherence

55

New TrendsNew Trends

• Impact more than meeting deliverables

• Attribution –influence of other factors

• Effective use of funds (taxpayer’s money)

• Broader issues of development effectiveness

• Learning functions replacing older principles of accountability and conditionality

• Empowerment• Harmonization• Changing relationship

between aid givers and receivers

66

Focus on SWAPS (sector wide Focus on SWAPS (sector wide approaches)approaches)

• Increasing recognition of the importance of trade, investment, and other economic and social activities in the development process.

• Development assistance is only one of many ways in which the developed and underdeveloped world relate to each other; and trade and private investment are in quantifiable terms, and much more important strands in this relationship.

77

Issues on Changing trendsIssues on Changing trends

• Shift to SWAP exacerbated the problem of attribution

• Increasingly difficult to disentangle the results of donor assistance from the overall processes in any particular sector.

• The importance of long-term development factors such as security, migration, and climate change- also need attention.

• Any realistic attempt to evaluate the effectiveness of aid has to be seen in this wider context

88

Situation facing development today is Situation facing development today is much more complex than it was 15 or much more complex than it was 15 or

even 10 years ago. even 10 years ago.

Evaluation form based on models derived Evaluation form based on models derived from project logical frameworks are no from project logical frameworks are no

longer appropriate to the fluid and longer appropriate to the fluid and complex forms of development complex forms of development

assistance in the 21st century,assistance in the 21st century,

99

New ChallengeNew Challenge

For the evaluator, reaching a firm conclusion on attribution results, to inputs and the chain of causation, becomes more challenging and difficult.

1010

Methodological IssuesMethodological Issues

• RCTs – randomized control trials

Although the use of such techniques works well in relatively simple contexts (e.g. impact of drugs on health, or impact on literacy of increased educational expenditure), the range and variety of factors in the development world today, which influence development trajectories are simply too complex to be dealt with in terms of simple cause and effect models.

1111

Methodological IssuesMethodological Issues

• Theory-Based Evaluation

Only a minority of development interventions can be dealt with simply through recourse to the original project plans (even revised or modified). In the complex world of SWAPs, donor harmonization, and budget support, there is less room for the effective use of logical frameworks or other project planning tools and a need for more flexible and less constrictive methods.

1212

Institutional IssuesInstitutional Issues

• Processes of Evaluation

• Empowerment

Accountability has changed, not only referring to the donors, but downward accountability to beneficiaries.

• Market of Evaluation• Partnership• Independence• Policy Impact

1313

Policy ImpactPolicy Impact

• Evaluation can provide “evidence” on which to base policies, when they are directed towards issues that are of direct relevance to policy makers.

• Unless it feeds into policy-making and has a positive impact on the development process, it becomes meaningless and merely an academic exercise.

1414

Ethical Issues and EquityEthical Issues and Equity

• The use and abuse of power in evaluation. Code of Conduct, Ethical Guidelines common among various evaluation associations

• Why Evaluate and For Whom?

Confidentiality vs Public Accountability

1515

Meeting the Evaluation ChallengesMeeting the Evaluation Challenges

• Joint Evaluations• Social Network Analysis• Outcome Mapping• Evaluation of Agencies• Country Programme Evaluation• Ethical Standards• Capacity Building• Public Awareness

1616

Complex Evaluation ExamplesComplex Evaluation Examples

1. Uganda’s Poverty Reduction Strategy- addressing the multi-dimensional nature of poverty with

31 indicators of output and impact; and another set of indicators to assess inputs and value for money

2. Enhanced Evaluation Framework- ambitious approach developed in the context of

evaluating general budget support across a range of countries, which seeks to establish a broad framework and sought to integrate in the analysis a wide range of factors.

1717

The Future of EvaluationThe Future of Evaluation

• The primary objective of the evaluation is not accountability but LEARNING in order to increase future effectiveness

• Focus of development evaluation should not be restricted to development assistance but to a wider set of forces.

1818

Words of WisdomWords of Wisdom

• The time has come to recognize that interpretive approaches which owe more to history than to experimental science are better suited to the world of development (Van Der Berg, 2005)

• The time has come to recognize the importance of narratives which make sense of what is observed, narratives which are never final products, but always, like development, in a state of “becoming” (Dart and Davies, 2003; Hulme, 1997)

1919

Words of WisdomWords of Wisdom

• Evaluation power guided by ethical standards play dominantly in complex evaluations in our contemporary times. Ethics and Equity Panel at the European Evaluation Conference (Tamondong, EES Conference, Helsinki, 2012).

2020

Words of WisdomWords of Wisdom

• A new set of priorities needs to be developed, centered on policy coherence, and how these different set of policies in developing world impinge on development process, covering not just aid, but trade, investment and migration, and the nature of global public goods. Renewed attention should be given on the evaluation at the country level focusing on a whole range of factors (Picciotto, 2005, 2006).

2121

CONCLUSIONCONCLUSION

• The problematic nature of producing convincing attributional arguments,

• persuading policy-makers of the validity of methods increasingly used in evaluation which recognizes the complexity of contemporary situation, and

• practicing ethical evaluations following the guidelines on code of conduct-- are the the major challenges for evaluation in the 21st century.

2222

Finally…Finally…

Different solutions and approaches will work in different contexts.

2323

Enduring Evaluation QuestionsEnduring Evaluation Questions

• 1. What should be the subject of evaluation in the development context?

• 2. Is there a continuing role for the “scientific” model of evaluation through randomized control trials (RCTs)?

• 3. How far should evaluators restrict their activities to the activities and impacts of development agencies?

• 4. Does a positivist approach make sense?

2424

QuestionsQuestions

• 5. How far can the existing techniques of evaluators be suitable for new tasks in the contemporary world?

• 6. How can evaluators produce reports which meet the differing demands of various stakeholders?

• 7. How about ethics? Should ethical standards in evaluation be enforced or sanctioned? And if so, how and by whom?

• 8. Why evaluate and for whom?

2525

QuestionsQuestions

• 9. How far can evaluation be seen as a discipline, with its own academic standards, body of theoretical knowledge, and methodological approaches? Or shall evaluation be treated as a profession, with people abiding by a code of conduct but not necessarily sharing a common theoretical outlook?

2626

THANK YOU THANK YOU

Feedback is the Fuel of Champions

- STamondong 2012

2727

ReferencesReferences

• Back, L. (2006)”Challenges to Evaluation Capacity: Development through Country Level Evaluations, URL (Nov 7,2007): http://www.sussex.ac.uk/development/1-4-9-1-1.html

• Conlin, Sean and Roderick L. Stirrat, “Current Challenges in Development Evaluation,” in Evaluation, 2008, Vol 14(2): 193-208.

• Dart, J. and R. Davies (2003) “A Dialogical, Story-Based Evaluation Tool: The Most Significant Change Technique,” American Journal of Evaluation 24(2): 137-55.

• Davies, R. (2006) Networks and Partnerships: Some Tools for Representing and Analyzing Networks, URL (Nov 7, 2007): http://www.sussex.ac.uk/development/1-4-9-1-1.html

• Fetterman, D. and A.Wandersman, eds(2004) Empowerment Evaluation Principles in Practice, New York: Guilford Press.

• Fetterman, D.M.,S.Kaftarian and A. Wandersman, eds (1996) Empowerment Evaluation: Knowledge and Tools for Self Assessment and Accountability. Thousand Oaks, CA: SAGE.

• Freeman, T.(2005) Country Programme Evaluation in an Era of Change. Evaluation Working Papers. New York: UNICEF.

2828

ReferencesReferences

• Gariba, S.(2006) For Whom the Bell Tolls: Improving Accountability for Policy through Evaluation, URL(Nov 7, 2007): http://www.sussex.ac.uk/development/1-4-9-1-1.html

• Holte-McKenzie, M., M.,S., Forde and S. Theobald(2006) “Development of a Participatory Monitoring and Evaluation Strategy, “ Evaluation and Program Planning 29(4):365-76.

• Holvoet, N. and R.Renard(2007) “Monitoring and Evaluation under the PRSP:Solid Rock or Quicksand?”, Evaluation and Program Planning 30(1):66-81.

• Hume, D., (1997) “Impact Assessment Methodologies for Microfinance: A Review, paper prepared for a meeting of the CGIAP working group on Impact Assessment Methodologies, URL (Nov 7,2007): http://www.ids.ac.uk/impact/resources/introduction/Hulme_IA_meth_review.pdf

2929

References References

• Jacobs, C.(2005)”Evaluating the Comprehensive Development Framework in Kyrgyz Republic, Central Asia: Magic Bullet or White Elephant?”, Evaluation 11(4):480-95.

• Kakande, M., (2006) “Evaluating Poverty Reduction Strategies: Uganda’s Experience”, URL (Nov 7, 2007) http://www.sussex.ac.uk/development/1-4-9-1-1.html

• Lister, S.(2006) “Evaluating General Budget Support,” URL(Nov 7, 2007) http://www.sussex.ac.uk/development/1-4-9-1-1.html

• Liverani, A. and H.E.Lundgren(2007) “Evaluation Systemsin Development Aid Agencies:An Analysis of DAC Peer Reviews 1996-2004,” Evaluation 13(2):241-56.

• Picciotto, R.(2005) “The Evaluation of Policy Coherence for Development,” Evaluation 11(3):311-30.

• Picciotto, R. (2006) The Global Dimension of Development Evaluation, URL (Nov7, 2007) http://www.sussex.ac.uk/development/1-4-9-1-1.html

3030

References References

• Schnoes, C.J.,V.Murphy-Berman and J.M.Chambers(2000) “Empowerment Evaluation Applied:Experiences, Analysis, and Recommendations from a Case Study,” American Journal of Evaluation 21(1):53-64.

• Tamondong, Susan D., “Why Evaluate and For Whom?” Ethics and Equity Panel, European Evaluation Society Biennial Conference, Helsinki, Finland, October 3-6,2012.

• Van derBerg(2005) “Results Evaluation and Impact Assessment in Development Co-operation,” Evaluation 11(1):27-36.

• World Commission on Dams (2000) Dams and Development: A New Framework for Decision-Making. London: Earthscan, URL (Nov 7,2007): http://www.dams.org//docs/report/wedreport.pdf