The ISOR Project

No health without research.

And no research without evaluation

Panel Session - HTAi 2012, Bilbao

Maite Solans-DomènechCatalan Agency for Health Information, Assessment and Quality

(CAHIAQ)

Outline of the presentation

1. CAHIAQ call in clinical and health

services research

2. The ISOR project

3. Impact on informing decision-making

Outline of the presentation

1. CAHIAQ call in clinical and health

services research

2. The ISOR project

3. Impact on informing decision-making

• Mission: to fill local knowledge gaps and inform clinical and healthcare decision-makers

• Research topic identification and priority-setting designed and implemented to ensure that the knowledge gaps of decision makers are addressed

• Ex-ante, ongoing and ex-post evaluations (13 years experience)

• There is a scientific committee that monitors the whole process.

• Predominance of oriented research with local relevance and informing local decision-making

• 7 calls from 1996 to 2008

Call in clinical and health services research

Call in clinical and health services research

CallPrioritised

topicsFunded projects

NAmount granted

(€)Average Euros/

project

1996 15 18 510,860.0 28,381.1

1998 20 19 540,911.0 28,469.0

2000 30 25 841,417.0 33,656.7

2002 35 25 841,417.0 33,656.7

2004 30 22 1.021,721.5 46,441.9

2006 29 18 1.021,721.5 56,762.3

2008 20 14 1,021,214.0 72,943.9

TOTAL 179 141 5,779,261.0 37,622.4

Outline of the presentation

1. CAHIAQ call in clinical and health

services research

2. The ISOR project

3. Impact on informing decision-making

Closing the assessment cycle: The ISOR Project

Objectives:

To carry on the evaluation of the impact of research

funded by CAHIAQ (accountability)

To develop a methodology and/or tool for the

assessment of research projects (transfer methods and

capacity-building)

To contribute to the spreading of these assessment

practices beyond CAHIAQ (advocacy)

Adoption of the Canadian CAHS-ROI model:

Useful common framework for any different type of health research (from basic research to health services)

Useful for any stakeholder’s view (from funder to evaluator)

Provides a set of generic indicators in the five impact levels: −Advancing knowledge−Capacity building − Informing decision-making −Health benefits−Broad economic and social impacts

Closing the assessment cycle: The ISOR Project

Catalan Health Research

Economic and Social Prosperity

Determinants of Health

Improvements in Health and Well-being

Healthcare Appropriateness,

Access, etc.

Prevention and Treatment

Health Status, Function, Well-being, Economic conditions

Initiation and Diffusion of Health Research Impact

Global Research

Research capacity Impacts feed back into inputs for future research

Government

Research Agenda

Interaccions

/Feedback

Other industries

Health industry

Advancing Knowledge

Capacity Building

Informing Decision Making

Health Benefits

Economic Benefits

Rese

arch

Res

ults

Know

ledg

e P

ool

Inte

racti

ons/C

olla

bora

tions

• Clinical

• Health Services

• Population and Public Health

Public Information, Groups

Canadian Academy of Health Science.

Return on Investment framework 2009

Closing the assessment cycle: The ISOR Project

PAYBACK

FRAMEWORK

Topic identificationInputsProcess

PrimmaryOutputs

Dissemination

SecondaryOutputs

Adoption Final Outcomes

Outline of the presentation

1. CAHIAQ call in clinical and health

services research

2. The ISOR project

3. Impact on informing decision-making

Bottom-up assessment of CAHIAQ Calls

Descriptive

[based on questionnaires to 70 PI]

Case study (respiratory disease research)

[based on semi-structured interviews to 8 decision-

makers and 15 researchers related to 6 projects]

Impact on informing decision-making

Questions to PI (n=70) n (%)Types of changes carried out in practice based on the research outcomes Clinical 29 (58.0) Organizational -

centre/institution12 (24.0)

Patient´s behaviour 5 (10.0) Public Health management 2 (4.0) Legal/regulations 2 (4.0)Unknown / No changes in practice. 28 (40.0)

Stakeholders or recipients who have used or taken into account the research outcomes Managers and planners 24 (41.4) Clinical professionals 17 (29.3) Investigators/researchers 12 (20.7) Scientific Societies 5 (8.6)

Not taken into account/ unknown 13 (18.6)

70% (n=46)

•Stakeholders in clinical and health services

•Scientific community

•Scientific societies.

60% PI: (n=40)

Real changes

Clinical and/or organizational setting.

Impact on informing decision-making

Almost all projects have induced or may have induced changes in clinical practice or in the organization of health services

Changes

Direct / indirect

Contribute to a complex change in clinical practice or healthcare organization

“After two years ... it [home-based hospitalization] grew as a real service and an integrated care cross-sectional unit was created within the hospital”

“Yes, it has all generated a culture... We are essentially a clean unit…”

… some “good stories”:

Impact on informing decision-making

Scientific publications: scientific articles, communications in congresses

Direct transmission to potential users: clinical sessions, …

Unconventional or poorly studied ways for the knowledge transfer

Scientific societies - as authoritative sources

Connection between managers and research teams – direct/indirect participation and collaboration

The channels used to transfer new knowledge into clinical practice are complex

“... the participation of hospital and primary care specialists in charge was beneficial [to change]”

… dissemination….

Impact on informing decision-making

Structural barriers

Lack of channels for the translation of research

Organizational dynamics

Individual barriers

Personal factors: reluctance to change

Nature of research: gap between researchers and practitioners

“Overall, I would say that researchers are forced to take on many roles. I am ready to be a researcher, but not to make an electoral pamphlet. In this regard, I think the relationship with healthcare policies and health plans is not as it should be.”

“Because hospital directors change frequently, or because this issue is not deemed to be as important as it should be...”

...and translation barriers:

Impact on informing decision-making

Project: Risk factors for relapse in COPD

Capacity Building

Informed decision-making

Health benefits

Overall economic benefit

Input Primary outcome

Secondary outcome

Adoption Final outcome

Relevant contribution (few

prior studies) Dissemination in

Scientific Societies

Contribution to the design care to chronic patients and

hospital management*

Cost reduction

Prevention in

excerbations

Identification of modificable

risk factors

New lines of research

Forecast of health care services to reduce

admissions

Difficulty to cultural change due to the patients (learning with the disease)l

IMPACT

PHASE

Impact found mainly in intermediate levels

Gaps or distances between potential and real impacts

Other impacts emerged implicitly and not intentionally

Impact on informing decision-making

ISOR project: Challenges

‘Ongoing project’

SIRECS

Comparability (across countries, research programmes,

disciplines, …)

Attribution

Time-lag

Interviewees (researchers, health policy or healthcare

decision-makers) interpret narrowly the terms (‘impact’)

General context: incipient culture of assessment

Thanks to Paula Adam, Marta Aymerich, Silvina Berra, Imma Guillamón, Gaietà Permanyer-Miralda, Joan MV Pons and Emilia Sánchez for their contribution to CAHIAQ strategy and the ISOR project