1

International Perspectives

Making Research Work for Decision Makers: Innovations and Priorities for Action Sydney, Australia

John N. Lavis, MD, PhD

Professor and Director, McMaster Health ForumMcMaster University

Adjunct Professor of Global HealthHarvard School of Public Health

I am or was involved in Creation and operation of the McMaster Health Forum (MHF) Development and maintenance of Health Systems Evidence (HSE) Development of (soon-to-launch) Health Systems Learning (HSL) Work of WHO-sponsored Evidence-Informed Policy Networks

(EVIPNet) Development of the SUPPORT tools

Conflict of Interest Statement

2

Rationales for and a definition of evidence-informed decision-making Promising practices in addressing the challenges to finding and using

research evidence What is needed to do this work, with a detour about

Clarifying a problem Framing options Identifying implementation considerations

My top three priorities for action

Overview

3

Which would be considered the most compelling in NSW / Australia? Better process / more efficient process? Better decisions / more acceptance of decisions? Better communication of decisions / less embarrassment? Better outcomes / better value for money?

Note that research evidence can be used in many ways Solve particular problems at hand (instrumental) Think about problems/options differently (conceptual) Justify a decision made for other reasons (political)

Possible Rationales for Evidence-Informed Decision-making

4

Using the best available* data and research evidence – systematically and transparently – in the time available in each of Agenda setting (esp. clarifying the problem iteratively, while being

attentive to policy and politics) Policy or program development (esp. framing options iteratively) Implementation (esp. identifying barriers / facilitators iteratively and

strategies to address them)

* Best available research evidence = highest quality, most locally applicable, synthesized research evidence (looking first for a perfect match to support an instrumental use and then looking more broadly to support a conceptual use)

A Possible Definition ofEvidence-Informed Decision-making

5

1. Research evidence competes with many other factors in the decision-making process

2. Research evidence isn’t valued as an information input

3. Research evidence isn’t relevant

4. Research evidence isn’t easy to use

Caveat The approaches I’ll describe are promising (based on a

synthesis of findings from 124 observational studies) but not yet tested rigorously in terms of effectiveness

Addressing the Challenges in Using Research Evidence

6

Challenge 1 Research evidence competes with many other factors in the

decision-making process• Institutional constraints• Interest group pressure• Other ideas such as citizens’ values, tacit knowledge, real-world

views and experiences• External events (e.g., global recession)

One option (among many) for addressing challenge 1 Improve decision-making processes (but this is beyond the scope of

many of us) or create “routine” processes (as many countries have done for new technologies) (e.g., NICE)

Addressing Challenge 1

7

Challenge 2 Research evidence isn’t valued as an information input

Two options (among many) for addressing challenge 2 Convince decision-makers and stakeholders to place value on

the use of research evidence by highlighting examples from the past or from other jurisdictions where research evidence made the difference between policy/program success and failure (or between communication success and failure) (e.g., WHO)

Work with journalists to highlight statements that are and aren’t based on the best available research evidence (e.g., Science-ish)

Addressing Challenge 2

8

Challenge 3 Research evidence isn’t relevant

One option (among many) for addressing challenge 3 Engage decision-makers and stakeholders periodically in priority-

setting processes and communicate the priorities to researchers• Long-term requirements for data-collection systems and for

new primary research (e.g., Listening for Direction)• Medium-term term requirements for systematic reviews (e.g.,

3ie)• Short-term requirements for ‘evidence briefs’ (e.g., EVIPNet

Cameroon)

Addressing Challenge 3

9

Challenge 4 Research evidence isn’t easy to use

Challenge 4a Research evidence isn’t communicated effectively

(i.e., decision-makers and stakeholders hear noise instead of music)

Two options (among many) for addressing challenge 4a Identify a high-priority issue, identify data and systematic

reviews that address different facets of the issue (problem, options and implementation considerations), and prepare an ‘evidence brief’ to summarize the findings (e.g., EVIPNet, MHF)

Stop (incentivizing) researchers and university media offices from publicizing the results of single studies without putting them in the context of a systematic review (and clarifying the many other questions that need to be answered before action is taken)

Addressing Challenge 4a

10

11

Addressing Challenge 4a (2)

Challenge 4 Research evidence isn’t easy to use

Challenge 4b Research evidence isn’t available when decision-makers and

stakeholders need it and in a form that they can use

Three options (among many) for addressing challenge 4b Promote the use of ‘one-stop shops’ for synthesized research

evidence• e.g., Cochrane Library and PubMed ‘hedges’ for ‘my health’ • e.g., Health Evidence and PubMed ‘hedges’ for ‘our health’• e.g., Health Systems Evidence for ‘our system’

Staff a rapid-response unit (e.g., REACH Uganda, MHF) Provide training (and tools) for decision-makers and stakeholders

about how to find and use research evidence (e.g., MHF’s Health Systems Learning)

Addressing Challenge 4b

12

Challenge 4 Research evidence isn’t easy to use

Challenge 4c Decision-makers lack mechanisms to prompt them to use

research evidence in decision-making

One option (among many) for addressing challenge 4c Propose changes to cabinet submissions and senior

management committee briefings (and to periodic program reviews) to prompt analysts to summarize whether and how research evidence informed the clarification of a system problem, the framing of options to address the problem, and the proposed approach to implementation (e.g., Ontario’s Research Evidence Tool)

Addressing Challenge 4c

13

Challenge 4 Research evidence isn’t easy to use

Challenge 4d Decision-makers lack forums where challenges can be

discussed with stakeholders and researchers

One option (among many) for addressing challenge 4d Plan deliberative dialogues at which pre-circulated evidence

briefs serve as the starting point for off-the-record deliberations involving decision-makers, stakeholders and researchers (stakeholder dialogues) or exclusively citizens (citizen panels) (e.g., EVIPNet, MHF)... A big step beyond GOBSATT

Addressing Challenge 4d

14

A Potential Two-Pronged Approach

Approach 1 – What researchers and ‘knowledge brokers’ can doPrepare evidence / citizen briefs and convene stakeholder dialogues / citizen panels over time frames of weeks and monthsPrepare rapid syntheses over time frames of days and weeksPromote one-stop shops and provide training so decision-makers and stakeholders can find research evidence on their own over time frames of hours and days

Approach 2 – What decision-makers can doSend clear signals that using research evidence is a high priorityEstablish a performance requirement for staffEstablish an internal rapid-response unitEstablish a requirement to summarize whether and how research evidence informed understanding of the problem, options and implementation considerations before decisions are taken

15

What Is Needed To Do This Work

Knowledge (see the ‘summary sheet’)Questions to ask about a problem, options and implementation considerationsTypes of research evidence needed to answer these questionsAppropriate sources of key types of research evidenceWhat an AMSTAR score meansQuestions to ask about local applicability considerationsDifference between knowing what other states or countries are doing (jurisdictional scans) versus knowing the results of research conducted in other countries (research evidence)

16

What Is Needed To Do This Work (2)

AttitudesWorking iteratively to understand a problem, options and implementation considerations in light of institutional constraints, interest group pressure, values and many other types of information, as well as ‘external’ factors such as the state of the economyBeing systematic and transparent in finding and using research evidence as one input to the decision-making processFinding and using the best available (i.e., highest quality, most locally applicable, synthesized) research evidence in the time you’ve been givenLooking first for a perfect match in the available research evidence (to support an instrumental use) and then looking more broadly (to support a conceptual use)

17

What Is Needed To Do This Work (3)Skills (for the ‘doers’)Clarifying a problem (and its causes), framing options to address the problem, and identifying implementation considerationsSearching appropriate sources of research evidenceUsing AMSTAR to assess the quality of a systematic reviewConducting a local applicability assessmentCommissioning research to fill gaps in research evidence

18

1. What is the problem (and its causes)?2. How did the problem come to attention and has this process influenced

the prospect of it being addressed?3. What indicators can be used, or collected, to establish the magnitude

of the problem and to measure progress in addressing it?4. What comparisons can be made to establish the magnitude of the

problem and to measure progress in addressing it?5. How can a problem be framed (or described) in a way that will motivate

different groups?

Clarifying the Problem - Questions

19

Low rates of childhood immunization A risk factor, disease or condition – Re-emergence of some

preventable childhood diseases, such as measles A program, service or drug currently being used – Program coverage

rate dropped 5% in the past year and the program has not corrected a celebrity’s assertions about safety and effectiveness

Current health system arrangements within which programs, services and drugs are provided

• Governance arrangements – Fragmented accountability• Financial arrangements – Refugees have to pay out-of-pocket for

immunizations for their children• Delivery arrangements – Many children are not registered with a

primary care team that can proactively reach them Current degree of implementation of an agreed course of action (e.g., a

policy) – Some parents are not aware of new schedule

Q1: What Is the Problem(and its Causes)? Example 1

20

Many citizens do not have a primary care physician A risk factor, disease or condition – Rising rates of chronic diseases A program, service or drug currently being used – Many citizens do not

benefit from coordinated prevention and proactive management of chronic diseases

Current health system arrangements within which programs, services and drugs are provided

• Governance arrangements – No mechanism to introduce foreign physicians without them ‘competing’ with local physicians

• Financial arrangements – Most physicians are paid on a fee-for-service basis and can thrive even in well-served communities

• Delivery arrangements – Many physicians work in solo practices Current degree of implementation of an agreed course of action (e.g., a

policy) – Policymakers do not enforce return-of-service agreements signed by physicians

Q1: What Is the Problem(and its Causes)? Example 2

21

Problem clarification involves Brainstorming about a problem and its potential causes (question 1) Searching for relevant data (questions 2 and 3) and research

evidence (questions 4 and 5) in appropriate sources (PubMed for single studies or Health Systems Evidence for systematic reviews related to health system arrangements)• See the summary sheet

Iteratively clarifying the problem (and its causes) in light of the data and research evidence found

Clarifying the Problem - Summing Up

22

1. What is an appropriate set of options to address the problem?

2. What benefits are important to those who will be affected and which benefits are likely to be achieved with each option?

3. What harms are important to those who will be affected and which harms are likely to arise with each option?

4. What are the local costs of each option and is there local evidence about their cost-effectiveness?

5. What adaptations might be made to any given option and might they alter its benefits, harms and costs?

6. Which stakeholders’ views and experiences might influence the acceptability of an option and its benefits, harms, and costs?

Framing Options - Questions

23

To address the problem of low rates of childhood immunization

1. Establish accountability among primary care practices for registering all children in their catchment area and for achieving a target immunization coverage rate (governance arrangement)

2. Remove all out-of-pocket charges for childhood immunization (financial arrangement)

3. Undertake a mass-media campaign to correct a celebrity’s assertions about the safety and effectiveness of childhood immunization (program change) and to raise awareness about a new immunization schedule (implementation strategy)

Q1: What is an Appropriate Set of Options? Example 1

24

To address the problem of many citizens not having a primary care physician

1. Increase the supply of physicians by raising medical school enrolment (delivery arrangement)

2. Improve the distribution of physicians by enforcing return-of-service agreements signed by physicians (implementation strategy)

3. Improve the supply and distribution of physicians by introducing geographically restricted licences (governance arrangement)

4. Change the dominant physician-remuneration mechanism from fee-for-service payment to capitation (financial arrangement)

Q1: What is an Appropriate Set of Options? Example 2

25

What benefits are important to those who will be affected and which benefits are likely to be achieved with each option? (Question 2)

Systematic reviews of effectiveness studies (e.g., randomized trials) What harms are important to those who will be affected and which harms are likely

to arise with each option? (Question 3) Systematic reviews of effectiveness studies or observational studies

What are the local costs of each option and is there local evidence about their cost-effectiveness? (Question 4)

Local cost-effectiveness analysis What adaptations might be made to any given option and might they alter its

benefits, harms and costs? (Question 5) Systematic reviews of qualitative studies (process evaluations)

Which stakeholders’ views and experiences might influence the acceptability of an option and its benefits, harms, and costs? (Question 6)

Systematic reviews of qualitative studies (e.g., acceptability studies)

Research Evidence Can Help to Respond to the Additional Questions

26

Options framing involves Brainstorming about an appropriate set of options to address a

problem (question 1) Search for research evidence (in the Cochrane Library + PubMed,

Health Evidence + PubMed, or Health Systems Evidence) about each option in turn• See the summary sheet

Iteratively framing the options in light of the research evidence found

Framing Options - Summing Up

27

28

Identifying Implementation Considerations - Questions to Consider

1. Where are the potential barriers to (and facilitators of) the successful implementation of an option? Are they at the level ofo Patients/citizens (see question 2)o Health workers (see question 3)o Organizations (see question 4)o System (see question 5)

2. What strategies should be considered in order to facilitate the necessary behavioural changes among patients/citizens?

3. What strategies should be considered in order to facilitate the necessary behavioural changes among health workers?

4. What strategies should be considered in order to facilitate the necessary organizational changes?

5. What strategies should be considered in order to facilitate the necessary system changes?

Identifying implementation considerations involves Brainstorming about (and searching for local data and research

evidence about) potential barriers to (and facilitators of) the successful implementation of an option (question 1)

Searching Health Systems Evidence for research evidence about each potential strategy in turn• Benefits (question 2) • Harms (question 3) • Cost-effectiveness (question 4) • Adaptations (question 5) • Stakeholders’ views and experiences (question 6)

Iteratively calibrating the strategies in light of the research evidence found

Identifying Implementation Considerations - Summing Up

29

My Top Three Priorities for Action

Establish a requirement to summarize whether and how research evidence informed understanding of the problem, options and implementation considerations before decisions are taken

Promote training for (and the use of tools and resources by) decision-makers and stakeholders so they can find and use the best available data and research evidence – systematically and transparently – in the time available

Create incentives for and communicate priorities to those who have the knowledge, attitudes and skills to

Prepare rapid syntheses in days and weeks Prepare evidence briefs (or citizen briefs) and convene stakeholder

dialogues (or citizen panels) in weeks and months(and stop incentivizing researchers and university media offices to

publicize the results of single studies without clear justification)

30

Resources

Available on Health Systems Evidence (www.healthsystemsevidence.org) Summary sheet on ‘finding and using research evidence’ (one page) Hyperlinked list of SUPPORT tools (two pages) Health Systems Evidence (four pages)

31