Evaluation amidst complexityEight questions evaluators should ask

Originally presented at Australasian Evaluation Society Annual Meeting, Melbourne, September 2015.Revised November 2015.

Ann Larson, PhDwww.socialdimensions.com.au

Complexity is often invoked as a reason for a project’s failures. My key message is that evaluators can use a complexity lens to

understand and facilitate success – if we know what to look for.

Overview of presentation

• Framing the problem• Explaining characteristics of complex

adaptive systems• Eight questions to ask in an evaluation• Closing remarks

Framing the problem

Characteristics of projects introducing an evidence-informed intervention which are affected by complexity

Signs of resistance or lack of support among some stakeholders or intended beneficiaries

Slow progress in starting implementation

Small or lack of adoption of new practices despite different strategies to change behaviour

Not clear how to sustain gains when the project finishes

Context for evaluatorsInvolved only at the end of the project, not at the beginning or middle.

The task is to make a narrative to explain success or failure that is convincing to implementers and donors.

We also must make recommendations that are grounded in evidence.

There is rarely a logic model or an M&E system that is particularly helpful for the evaluation.

Explaining complex adaptive systems

Properties and behaviours of complex adaptive systems

• Path dependency• Diversity• Multiple interdependent relationships • Self-organising emergent behaviours • External shocks• Non-linear outcomes

Path dependency

Diversity

External shocks & outside influences

Interdependent relationships

Self-organising and emergent behaviour

Non-linear outcomes

Harnessing complexity

An eclectic list of strategies for projects to harness rather than control complexity

• Flexible, long term funding• New behaviour grounded in relevant history and

saliency• Build coalitions around a vision for change• Understand motivations for behaviour change:

introduce accountability and incentives• Start small, be flexible and experiment• Balance local initiative with quality standards• Monitor, review and act in a timely manner

Evaluation strategies

Evaluation designs for coping with CAS

Suggestions from evaluatorsDevelopmental evaluations

Push-back from clientsEvaluation only budgeted at end, take too long, too expensive, too much like internal evaluation

Intensive case studies to elucidate challenges and successes

Lacks generalisability, doesn’t address original project objectives and logic models

8 complexity-sensitive evaluation questions

Summary of complexity-sensitive evaluation questions

Is there evidence that project designers and implementers ….

1. Were sensitive to history and current priorities?2. Accommodate diversity in its design and implementation by employing

different approaches depending on capacity and circumstance?3. Understood dynamics of relevant behaviour?4. Effectively influenced those dynamics?5. Monitored, reviewed and took action based on regular information?6. Recognised and embraced emergent behaviours that supported the

intervention?7. Responded to external change?8. Were focused on what happens after the project?

1) Has the project aligned itself with history and current priorities? Has that had an effect on its acceptability?

Shape of the Busselton Sheds, planks from the old

jetty

2) Did the project take into account the differences in regions, local context, workforce or beneficiaries? How could it have been more effective if it had?

States -2Districts -5Facilities-5

States -19Districts -35Facilities-53

States -19Districts- 40Facilities - 81

States -19Districts – 206Facilities - 371

States -19Districts – 236Facilities - 458

Different strategies employed in different states and facilities

In this sanitation project, households build their own latrines using standardized construction methods but individualised shelters.

3) Did project designers and implementers understand relationships and dynamics in the workplace, families or agencies?

This can be done by being part of the culture, involving the group whose behaviour you want to change, or running small scale pilots.

Projects within complex adaptive systems need to identify and work ‘with’ or work ‘around’ all of the important components, such as supply chains.

4) What did they put into place to change those dynamics to change behaviour?

Training alone is rarely effective in changing behaviour because there are so many other influences reinforcing the old behaviour.

Among strategies to change behaviour affected by local norms and reinforced by feedback loops are coaching, measures to increase accountability, rewarding performance and removing obstacles.

Encourage or enable communities to demand the new service

Provide incentives to local leaders

5) Did the project routinely collect, review and respond to information about activities and behaviour change?

6) Did the project recognise emergent behaviours and, if so, how did they respond?

Local co-option of men’s health strategies

7) How did the project respond and adapt in face of external shocks and changes? Why were they able to do this?

Projects need good relationships and the capacity to be flexible to respond to changes in policies, funding or security issues through advocacy or problem solving at the appropriate level.

8) How is the project preparing for what will happen when it ends?

What aspects will government or local NGOs retain? Will the principal goal continue to inspire action? How is the implementing agency altering their approach?

Summary of complexity-sensitive evaluation questions

Is there evidence that project designers and implementers ….

1. Were sensitive to history and current priorities?2. Accommodate diversity in its design and implementation by employing

different approaches depending on capacity and circumstance?3. Understood dynamics of relevant behaviour?4. Effectively influenced those dynamics?5. Monitored, reviewed and took action based on regular information?6. Recognised and embraced emergent behaviours that supported the

intervention?7. Responded to external change?8. Were focused on what happens after the project?

Closing remarks

These questions can be used in an interview guide with stakeholders or as a framework for analysis.

We need more evidence on how to influence behaviour in complex adaptive systems. Evaluators

can contribute to the evidence base.

A personal reading listPaina, L. and D. H. Peters (2012). "Understanding pathways for scaling up health services through the lens of complex adaptive systems." Health Policy and Planning 27(5): 365-373.Preskill H et al. Evaluating Complexity: Propositions for improving practice. FSG working paper, 2014.Everything by Lant Prichett but especially Pritchett, L. and F. de Weijer (2010). Fragile States: Stuck in a Capability Trap? World Development Report 2011 Background Paper. Washington DC, World Bank.Everything by Trish Greenhalgh but especially Greenhalgh, T., J. Russell, R. E. Ashcroft and W. Parsons (2011). "Why National eHealth Programs Need Dead Philosophers: Wittgensteinian Reflections on Policymakers’ Reluctance to Learn from History." Milbank Quarterly 89(4): 533-563.Axelrod, R., & Cohen, M. D. (1999). Harnessing Complexity: Organizational Implications of a Scientific Frontier Chandy, L., A. Hosono, H. Kharas and J. Linn, Eds. (2013). Getting to Scale: How to Bring Development Solutions to Millions of Poor People. Washington DC, Brookings Institutions Press.Sutton, R. I. and H. Rao (2014). Scaling Up Excellence: Getting to More without Settling for Less. New York, Random House Business Books.