unpacking the notion of program theory from a complexity lense: can indigenous and cross-cultural...

Unpacking the notion of program theory from a complexity lense: Can

indigenous and cross-cultural perspectives help?

Sanjeev Sridharan and Janet SmylieCentre for Research on Inner City Health

University of Toronto

Benita van Wyk, Feedback Research & Analytics

Outline

• 1. Why is program theory important and useful?

• 2. What are the limitations of the standard approach to program theory? The view from evaluation

• 3. Ideas for a potential research project

Conceptualizing and Critiquing Programs:

Program Planning and Evaluation

Evaluation Literature

Indigenous/ Cross-Cultural

Literature

Identify concepts to enhance program planning and

evaluation

Implement Concepts in

Two Programs

Evaluate Programs

Basis for dialogue

Sustainability

Connectivity

Dynamics

Heterogeneity

The important questions

What is a program?

Does the nature of the causal assumptions make a difference

in the way we plan programs and evaluations?

Dealing with complexity

An Example of a complex intervenion

Dealing with complexity

An Example: Primary Prevention Have a Heart Paisley

The nature of the causal problem

• Note that the causal assumptions are not clearly specified

• Most assumptions are implied and of the form:

• If (A and B and C and D and E and at time T) then good outcomes might result

• The problem:– Causal contingencies not clearly specified or

understood– Dynamic complexity not well understood

Another example: The Smoking Ban in Scotland

The incompleteness of program theories The logic model while complex is still incomplete

A policy such as the smoking ban might lead to proximal impacts through perhaps an uncomplicated pathway; however the logic of the causal chain for intermediate and long-term outcomes might be incomplete

Additional “programmatic” inputs might be needed.

Information on the nature of such additional interventions can perhaps come from a synthesis of literature plus additional dialogue with stakeholders with an intimate knowledge of the processes by which programmes works.

Defining a complex intervention (MRC, 2000)

“The greater the difficulty in defining precisely what exactly are the ‘active ingredients’ of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention.”

System Dynamic Approaches (Sterman, 2006)

Constantly changing; Governed by feedback; Non-linear, History-dependent; Adaptive and evolving; Characterized by trade-offs; Policy resistance: “The result is policy resistance, the

tendency for interventions to be defeated by the system’s response to the intervention itself.”

Features of complex health interventions (Pawson et al., 2004)

The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not linear,

and involve negotiation and feedback at each stage. Interventions are embedded in social systems and how

they work is shaped by this context. Interventions are prone to modification as they are

implemented. Interventions are open systems and change through

learning as stakeholders come to understand them.

An additional purpose of evaluation for complex interventions

Explication

Why?

Multiple components– be explicit about the “active ingredients”

Timeline of impact unclear

Non-uniformity– Contextualizing

Stakeholder involvement: Co-construction of intervention

On causality and complexity

Does a cause operate the same way in simple systems as compared to complex systems?

Are there different kinds of causes? E.g. Famines

Precipitating causes Amplifying causes Causes of vulnerability

Innovations in evaluation design and approaches

So what do programmes do?

A need for a richer vocabulary on programmes as causes– can we learn from other sciences such as evolution or chemical kinetics?

Different metaphors: Programmes as catalysts, programmes as foundations, programmes as operating systems

Understanding complexity in multiple light– The system-as-cause– Understanding dynamic complexity

System-as-Cause

Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.

Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>.

Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991.

Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

“Solutions” Can Also Create New Problems

Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.

Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.

Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.

Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.

-- Meadows, Richardson, Bruckman

Unpacking the causal contingencies of change

•If (A AND B AND C AND D AND E and at time T) then good outcomes might result

•What does this really mean? Unpacking the complexity of the change process

•What is missing in our understanding of how programs work?

Seeing and understanding connection“You think you understand two because you understand one and one. You think you understand two because you understand one and one.

But you must also understand ‘and’.”

You think you understand two because you understand one and one.

But you must also understand ‘and’.”

-- Sufi Saying

1. The role of sustainability

Why sustaining programs might be key to success

Going further – sustainability of programs at the individual level – the critical importance of

Continuity of Care

2. The limited understanding of heterogeneity

Heterogeneity of needs

Heterogeneity of mechanisms of change

Heterogeneity of interventions

3. The role of connectivity between interventions

Far too much focus of most evaluations on a single intervention as the unit of analysis

Understanding connectivity between programs

Better understanding of ecology of interventions

4. Understanding the relationship between multiple levels of interventions

Many complex interventions require programming at the community, neighborhood, school and individual level

Multilevel alignment across the interventions

4. Multilevel alignment complexity

5. Understanding Dynamic Complexity

Timeline of impact

Developmental Synergies

Anticipated timeline of impact

Dynamic complexity (1)

Dynamic complexity (2)

5. Coordination complexity

The advantages of partnerships are poorly understood

More importantly, collaborative advantages of individual recipients rarely made

Coordination complexity

Cracks in a network complexity

6. Sequential Complexity

Relationship between sequences of interventions poorly understood

Most of our methodologies are extremely limited in understanding such relationships

6. Sequential intervention complexity

7. Structural complexity

Policy resistance

Translating programs to fit within the structures

The system-as-cause

Next Steps

Identify partners

Identify concepts from Evaluation and Indigenous/Cross-Cultural Field/Literature that can help enhance the concepts

Translate concepts into practical testable experiments

Other concepts and ideas?

sridharans@smh.toronto.on.ca