Towards a Transformative View of Evaluation: Evaluating Global

Health Interventions

Presentation at the International Food Policy Research Institute

Sanjeev Sridharan The Evaluation Centre for Complex Health

InterventionsUniversity of Toronto &St. Michael’s Hospital

November 19, 2013

What is evaluation?

How do we make

evaluations matter?

TwoExamples

Summing up

Evaluating Research: Pathways

of Influence

3

What is evaluation? A useful

but perhaps incomplete

definition

• Evaluation is defined both as a means of assessing performance and to identify alternative ways to deliver

• “evaluation is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.”

…..what role can evaluation/ evaluative thinking play in navigating interventions?

4

Purpose of evaluation (Mark, Henry and Julnes, 2000

• Assessing merit and worth• Causal questions, RCT, observational studies

• Programme and organizational improvemento Formative evaluation

• Oversight and compliance

• Knowledge developmento Neglected purpose of many evaluations

An Example: Primary Prevention Have a Heart Paisley

Features of complex interventions (Pawson et al.,

2004) The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not

linear, and involve negotiation and feedback at each stage.

Interventions are embedded in social systems and how they work is shaped by this context.

Interventions are prone to modification as they are implemented.

Interventions are open systems and change through learning as stakeholders come to understand them.

System Dynamic Approaches (Sterman,

2006)

• Constantly changing;• Governed by feedback;• Non-linear, History-dependent;• Adaptive and evolving;• Characterized by trade-offs;• Policy resistance: “The result is policy

resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.”

“Solutions” Can Also Create New Problems

Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.

Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.

Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.

Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention

itself. -- Meadows, Richardson, Bruckman

System-as-Cause

Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.

Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>.

Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991.

Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

So why are evaluations so often

not very useful?

UN Office of the Internal

Oversight Services, 2008• A Critique of Results-Based Management (2008).• “Results-based management at the United

Nations has been an administrative chore of little value to accountability and decision-making.”

The UN Critique of performance

management and evaluation• Lack of strategic direction and cross-

organizational performance incentives

• Problems of attribution and trivializing innovation

• Trivializing outcomes

• The practice of lacks rigor

• A lack of purpose

The UN Critique (2)• Lack of clarity on the consequences of good

and poor performance

• Lack of clarity on the capacity needed to build a results-based management system

• Technical solutions are not a substitute for substantive clarity

14

The logic of an evolutionary strategy

Box et al (1978, p. 303):

... the best time to design an experiment is after it is finished, the converse is that the worst time is the beginning, when least is known. If the entire experiment was designed at the outset, the following would have to be assumed as known: (1) which variables were the most important, (2) over what ranges the variables should be studied... The experimenter is least able to answer such questions at the outset of an investigation but gradually becomes more able to do so as a program evolves. (p. 303)

15

What kind of evaluation will you be doing?

Formative

Developmental

Summative

16

INTERVENTION THEORY AND DEVELOPING EXPECTATIONS OF IMPACTS OVER TIME

The key components of the complex intervention

The program theory of the complex intervention

Learning from the Evidence Base

The anticipated timeline of impact

LEARNING FRAMEWORKS AND PATHWAYS OF INFLUENCE

The pathways of influence of an evaluation

Learning framework for the evaluation

IMPACTS AND LEARNINGAssessing the impact of the intervention: DESIGN

Learning about the intervention over time

SPREAD AND SUSTAINABILITY

Spreading learning from an evaluation

Reflections on performance and sustainability

A B

C D

A Ten Step approach to Evualation

An Example: Primary Prevention Have a Heart Paisley

What are the key

LEARNINGS from the

evaluation?

On what basis do we make a

decision to SUSTAIN the

program?

What gets SPREAD as a result of the evaluation?

Design, methods,

approaches

ACTION

PROGRAMMES

Interpersonal,

Individual & Collective

Mechanisms

EVALUATION

THEORY

EVIDENCE

POLICY LANDSCAPE

(dynamic, changes over time)

Hypothesised Pathways

Evidence Base Linked to Pathways

Areas of Uncertainty

A

B

CE

D

Evaluator

ALIGNMENT(moving beyond programmes at

level of analysis)

POLICY IMPACT

INDIVIDUAL IMPACTS

ORGANISATIONAL LEARNING

PROCESS LEARNING

RISK LANDSCAP

E

19

Example 1

20

Background• Project in Collaboration with the China National

Health Development Research Centre to build Evaluation Capacity for evaluating health system reform efforts

• Focus on Health Inequities• Developing guidelines to evaluate health inequity

initiatives• Relationship to complexity

Evaluate 3 projects:

Test and Refine the Guidelines

Levels of Evaluation Influence

Individual

Inter-personal

Collective

Create Guidelines

• Literature Reviews

• Surveys of Innovations

• Determine Evaluation Assets and Needs

Revise Guidelines

Knowledge Translation

Evaluation Capacity Building

21

22

Complexity of Intervention

Eva

luati

on

Ap

pro

ach

23

Complexity of Intervention

Ap

pro

ach

to E

valu

ati

on

Clin

ical

Int

erve

ntio

n

Com

mun

ity

Inte

rven

tion

Sys

tem

-leve

l Int

erve

ntio

n

24

Review of articles

ContextExpectations and

timelines

Understanding and changing

understanding

Organizational structures for

learning

Multiple understandings of

impacts

Sustainability and spread

25

Describe the

Intervention

What was the Setting of the intervention? What was the

Context?

Was there a discussion of the evidence informing the program? Was the evidence from multiple disciplines?

Is the program a

pilot?

Challenges of

adaptation to specific setting?

What was the

duration of the

intervention?

Discussion on

timelines and

trajectories of impacts?

Were there changes in

the intervention over time?

How did the evaluation

explore this?

Was the theory of change

described? Did it

change over time?

26

How were impacts studied;

what design were

implemented?

Was there a Formal

process of ruling out threats to

internal and external validity?

Was the program a success?

Unintended outcomes? Differential impacts for

different groups?

Did the evaluation help with decisions

about sustainability

?

Was the organizatio

nal structure of

the interventio

n described?

What was the

Intervention Planners’

view of success?

Were there formal

structures to learn and modify the

program over time? Was there a

discussion of what can be

spread as part of

learnings from the

evaluation?

27

Example 2

Goals“To contribute to improving health and strengthening health systems in low and middle income countries (LMICs), by supporting innovative international approaches to integrating health knowledge generation and synthesis (including consideration of environmental, economic, socio-cultural, and public policy factors) through research, health research capacity development, and the use of research evidence for health policy and practice.”

Objectives• Foster international partnerships and collaboration

to promote the generation and effective communication and use of relevant health research (including consideration of environmental, economic, socio-cultural, and public policy factors) in, for and by low- and middle-income countries (LMICs);

• Train and support researchers responsive to policy and practice priorities of LMICs relating to or influencing health; and

• Support active collaboration between researchers and research users (e.g. policymakers, practitioners, civil society organizations, and community members) to support health priorities of LMICs.

Basic QuestionsHow complete is our knowledge of how these initiatives are intended to work?Do we have clarity on the timeline of impact of such initiatives? How do we ensure that we don’t get caught up in the ‘activity space’ of such initiatives? How do we align metrics to provide incentives to focus on the outcomes?

Key Concepts• Limited sphere of control

• Flexible sphere of control

• Timeline of impact

• Metrics and incentives

• Integrated planning and evaluation

The Problem of Insufficient Imagination

Granting Mechanisms

Research, Capacity

Building and Knowledge Translation activities

Health Impacts

Direct Control

7. R

ESEA

RCH

,CAP

ACIT

Y BU

ILD

ING

AN

D K

NO

WLE

DG

E TR

ANSL

ATIO

N P

ROCE

SSES

and

OU

TPU

TS

Direct Influence

16. P

LAN

NIN

G F

OR

IMPL

EMEN

TATI

ON

: ALI

GN

MEN

T O

F RE

SEAR

CH A

ND

CAP

ACIT

Y BU

ILD

ING

WIT

H L

OCA

L PO

LICY

AN

D

PRAC

TICE

NEE

DS

Indirect Influence

8. KNOWLEDGE GENERATION:

FOCUSSED AND CROSS-CUTTING

THEMES

1. DRIVERS FOR THE

DEVELOPMENT OF TEASDALE-

CORTI

4. MERIT REVIEW

9. ENHANCED RESEARCH

CAPACITY AND CAPACITY TO

USE EVIDENCE

10.STRENGTHENE

D RELATIONSHIP

S BETWEEN NORTH AND

SOUTH RESEARCHERS

12. ENABLED RESEARCHERS

13. RESEARCH

DISSEMINATION

14. GREATER AWARENESS OF SALIENCE

OF RESEARCH

AND CAPACITY BUILDING

FOR LOCAL POLICY AND

PRACTICE CONCERNS

15. ENHANCED LOCAL

FOUNDATIONS FOR FUTURE

RESEARCH AND CAPACITY

BUILDING

18. L

OCA

L U

SE A

ND

INFL

UEN

CE O

F RE

SEAR

CH A

ND

CAP

ACIT

Y BU

ILD

ING

19. S

TREN

GTH

ENED

HEA

LTH

SYS

TEM

S A

ND

EN

HA

NCE

D

HEA

LTH

EQ

UIT

Y

20. I

MPR

OVE

D H

EALT

H

17. MECHANISMS

TO SUSTAIN RELATIONSHI

PS AND FOUNDATION

S

2. COMMISSIONING RESEARCH AND CAPACITY BUILDING AND KNOWLEDGE

TRANSLATION GRANTS : TEAM AND

LEADERSHIP GRANTS, KTE

GRANTS

3. NORTH-SOUTH RESEARCHERS

AND KNOWLEDGE USERS

RESPOND TO REQUEST FOR PROPOSALS

11. KNOWLEDGE

USER ENABLED TO IMPLEMENT LEARNINGS

5. IMPLEMENTATI

ON OF PROJECTS

6. BUILDING

TEAMS: INITIAL RELATIONSHIP

BUILDING

A

B

C

D

EF

G

H

I

J

K

L

M

N

O

P

Q

RS

T

U

V

W

X

Z AA BB

CC

DataInterviews with Teasdale-Corti planners in multiple funding organizations including CIHR and IDRC

Formal analysis of 8 final reports

Formal analysis of 8 proposals

Surveys of Teasdale-Corti grantees—separate surveys were conducted with Canadian researchers, Southern researchers and knowledge users

Interviews with grantees at the October 2012 GHRI Ottawa meeting—this includes video interviews with grantees

Brief case studies of three grantees including Skype interviews with Southern partners

Bibliometrics analysis

Data collection to support theory of change work.

Support from evidence• There was strong support from the

evidence for the overall goal of Teasdale-Corti: there was support for how Teasdale-Corti contributed to building research and practice that can contribute to health and strengthening health systems.

• There was limited evidence that the Teasdale-Corti initiative directly impacted health outcomes (but need to reflect on the timeline of impact)

Some learnings (1)• Greater upfront clarity of the anticipated

timeline of impact • Greater upfront thinking about planning for

sustainability • Greater clarity around who counts as a

knowledge user • Focused monitoring on knowledge-user

engagement: possible metrics• Greater clarity on what equity in partnerships

means • Boundaries and contexts of the Teasdale-Corti

model

Some learnings (2)• Limited focus on health equity • Greater clarify on amount and types of support

grantees need • Focus on the end-user and not just the

knowledge user• Greater focus on the benefits of the North-

South relationships • Greater clarity on what progress and

accountability means given the complexity of Teasdale-Corti • Incentives in the system of metrics to

encourage collaboration and co-creation

Refining the Theory of Change:

examples of learnings

Greater clarity (by providing models, exemplars, narratives of successful implementation, etc.) on how best to create synergy between knowledge generation, knowledge translation and capacity building

More explicitly clarify the role of the knowledge user in the implementation of Teasdale-Corti

Explicitly recognize the ‘values’ that guide the implementation of Teasdale-Corti

The implementation of the theory of change needs to be supported by a M&E system that pays attention to the complexities and heterogeneities of the implementation of Teasdale-Corti

Towards a more imagined plan

Problem Definition

1

LMIC Priorities:

2

Designing Locally-Useful Research

3

Promoting Research Use and Influence

4

Potential for Useful Capacity Building:

5

Collaboration

6

Equity

7

Sustainability

8

Clarity and coherence of outcomes and specific theory of change

9

Timeline of Impact:

10

Monitoring/Evaluation Plan:

11

Gender and Ethical Considerations:

12

Ideas for improvement

Recommendations based on proposal analysis

• More explicit focus on barriers

• Have a clearer focus on the diversity of knowledge users

• Equity should have a more explicit focus

• Encourage thinking about sustainability upfront

• Planning for relationship-building over time

• Monitoring and evaluation of complex interventions

Synergizing activities

The ‘black-box’ of capacity-building

Consistency in values and metrics

Non-linear models of the research/policy interface

Long timeline of impact that may exist for research to influence health outcomes

Unpacking the Plan

Theory of Change

Developmental - what are capacities needed?

Sphere of Control

Timeline of Impact

Metrics and incentives

Phased approach to commissioning

Evaluating research• What are pathways of influence by which

research impacts policies? • Can a results based culture help with greater

understanding of the pathways of influence?• Approaches:

o Conceptualizing research as an intervention with short and long term goals. One of the intermediate goals is to influence policy. The long term goal is to improve individual lives

? Policy Influence

Social Betterme

nt

Research as Intervention

Individual Research Project

Multiple research projects

Monitoring and

Evaluation

Evaluative

Thinking

Monitoring and

Evaluation

Evaluative

Thinking

Monitoring and

Evaluation

Evaluative

Thinking

Monitoring and

Evaluation

Evaluative

Thinking

Monitoring and

Evaluation

System

Research Quality

Pathways of Policy Influence

Research as

Intervention

ContextMechanis

msOutcomes

MODEL 1From

Knowledge to Policy

See Knowledge to Policy: Making the most of Development Research, Fred

Carden, 2009

Research as

Intervention

Overall Context

Mechanism: So,

what can you do?

Outcomes

Stability of Decision Making Institutions

Policy Absorption Capacity

Nature of Governance

Opportunities for countries in transition

Economic crisis and pressures on government

Relationship Building

 Communication Networks of Influence

 Institutional Mechanisms

Policy Influence

“Better” Decisions

Social Betterment

Government Context

Clear Government Demand

 Government interest but leadership absent

Interest but no capacity

 Interesting research but research uninteresting for Government

 Government disinterest or hostility

Research Quality

Research Networks

Action Research

New tools

Process Outcome

sPolicy Capacity

Broader Policy Horizons

Policy Regimes

Model 2: Pathways of

InfluenceMel Mark and Gary Henry, Evaluation,

2003

Research Initiatives in specific places

• Local research for lasting solutions

• Some key questions:

o What is local about your research? How does your research respond to local needs? What is different about your local networks? What is the value added of the local research?

o Are there local pathways of influence? o How different are the local pathways of influence from other more

general pathways?

Research as

Intervention

Global Context

Global Mechanis

ms

Outcomes

Local Context

Local Mechanis

ms

Local Context• Are your theories locally generated? Are the

needs local? • Examples of local context that have interfered

with standard influence processes• Does your research address local needs in a

timely manner?• Are the local networks different from other more

general networks of influence?• Example: What are some of the challenges of

policy influence in specific contexts in Africa?

58

Summing up and looking ahead

59

Models of Causation

(Successionist vs. Generative Models of Causation)

Ecology of Evidence

Integrating Knowledge Translation

with evaluation

Capacity Building

Development al evaluation in Complex

Dynamic Settings

Portfolio of designs

and approaches

Program Theory and

Incompleteness

Time Horizons and

Functional forms

Spread, Scaling up

and Generalization