Assessing the impact of research and KTE activities

Sandra Nutley

Agenda for this session

1. Key approaches to assessing the use and impact of research and knowledge mobilisation activities

2. Common challenges and methodological issues

3. Summary of findings from existing assessment activities

Why assess research impact?

• Addressing accountability• Assuring value for money• Setting priorities• Assisting learning• Improving outcomes

Summative or formative purposes?

Impact of research

Impact of initiatives to increase research use

Research use in ‘user’ communities

Systematic reviews

Centres promoting research use

1. Key approaches to assessing research impact

Target KTE interventions

Organisational research use

Single studies

Research programmes

Policymakers

PractitionersMedia

Forward tracking

Tracking back

Evaluation of initiatives

Forward tracking from research projects and programmes to use and impact

Two common methods:

• Describe and quantify impacts of research - E.g. Payback model 5 categories of possible impact:– Knowledge production– Research capacity building– Policy or product development– Sector benefits– Wider societal benefits

• Describe and map networks and flows of knowledge and the effects of any interactions

Tracking back from decisions or behaviours to research influences on these

• Bibliometric studies of policy documents and practice guidelines

• User panels – surveys and workshops• Ethnographic studies of communities of

practice• Economic impact evaluations – estimate

economic impact of a policy and estimate extent of research influence on that policy

Figure 5: Approaches and methodologies

0

10

20

30

40

50

60To

tal n

um

ber

of

pap

ers

Discussed

Used

What methods are used in practice?Systematic review by Boaz et al 2008

Evaluations of initiatives to increase research impact

• Standard programme evaluation strategies: – Qualitative investigation of processes and

perceived effects– Action research– Experimental and quasi-experimental methods

• Importance of strong theories of change & models of KTE to design and test interventions

• Need shared taxonomy of KTE strategies and interventions

KTE models

Ward et al 2009:• 63 different theories or models of KTE• Distilled down to 5 components of the KTE

process• Connection between them is seen as

interactive and multi-directional• Aim of framework is to provide a foundation

for gathering evidence about knowledge into action processes

Knowledge/Research

Context Barriers/Supports

Problem

Interventions Utilisation

Attributes of knowledge (relative advantage, complexity and compatibility)

How it is synthesised and adapted

Recognition of a need for action

Organisational, individual, environmental and structural

Distribution-type interventions Linkage-type interventions

Process of using intervention: cycles of selection, tailoring, implementation & evaluation

Conceptual, instrumental, political, procedural

Taxonomies of KTE strategiesE.g. Mitton et al 2007 – 8 key strategies• Face-to face exchange between researchers and

‘users’• Education sessions for users• Networks and communities of practice• Facilitated meetings between researchers and users• Interactive, multidisciplinary workshops• Capacity building in user organisations• Web-based information, electronic communications• Steering committees for research

projects/programmes

RURU taxonomy

• Intervention types and underlying mechanisms, with focus on the latter

• Five key mechanisms:– Dissemination– Interaction– Social influence– Facilitation– Incentives and reinforcement

2. Common challenges and methodological issues

• What types of use/impact are of interest?• When to assess impact?• Importance of context – assessing actual or

potential impacts?• Dealing with attribution and additionality –

constructing a convincing impact narrative• Getting away from linear models research

use/impact

What uses/impacts are of interest?

• Instrumental, conceptual or symbolic research use?

• Reach/awareness, impact on behaviour & service delivery, outcomes for service users?

• Positive and dysfunctional consequences?

When to assess impact?

Impact often occurs far down the line, but impact trails grow fainter over time

Tools for analysing context

• Accounting for the difference between actual and potential impacts

• E.g. Conducive policy environments - research more likely to be used when information, ideology and interests coincide and institutions can support developments (Weiss)

Constructing a convincing impact narrative

• Dealing with attribution – the potential of contribution analysis

Slide source: Wimbush 2010

Slide source: Wimbush 2010

Slide source: Wimbush 2010

Conclusions on methods• Case study methods to take account of differing

types of research and contexts for impact• Combination of quantitative and qualitative

methods and indicators within case study approach

• Need for research impact theory/model to guide data collection and analysis

• Limited benefits from studying the impacts of individual research projects

• Dangers of generalising from case sampling – because distribution of research impact is likely to be uneven

3. Summary of findings from the UK ESRC’s impact assessment activities

• The most important drivers of impact are:– Established relationships and networks with user

communities– Involving users at all stages with research– Well-planned user-engagement and KTE strategies– Portfolios of research activity that build reputations with

research users– Good infrastructure and management support– The involvement of intermediaries and knowledge brokers

as translators, amplifiers, network providers

RURU’s conclusions on generic features of effective KTE practices

Research must be translated - adaptation of findings to specific policy and practice contexts

Enthusiasm- of key individuals - personal contact is most effective

Contextual analysis - understanding and targeting specific barriers to, and enablers of, change

Credibility - strong evidence from trusted source, inc. endorsement from opinion leaders

Leadership - within research impact settings Support - ongoing financial, technical & emotional

support Integration - of new activities with existing systems

and activities

Implications for future research on research use and impact

• Much that we still need to know• We need to move away from:

– Poor documentation and under-evaluated KTE activities– Studies that focus only on the instrumental use of research– An assumption that research is used and applied mainly by

individual practitioners– Studies that result only in a now familiar listing of barriers

and enablers, especially where these are the barriers/enablers experienced by individual practitioners

Key messages• Need to understand more about how research-based

knowledge enters and flows through diverse policy, organisational and practice settings

• Need realistic assumptions about the nature and processes of research use and impact– these are many and complex

• No single model of research use is likely to be sufficient for all situations when impact is to be assessed

• Need to make choices about where and how to look for use and impact based on the purpose of the assessment

ReferencesMain reference• Davies HTO and Nutley SM (2008) ‘Learning More about How Research-

Based Knowledge Gets Used Guidance in the Development of New Empirical Research’, Working Paper for the WT Grant Foundation, New York

Other useful references• Boaz et al (2008) ‘Assessing the impact of research on policy: A review of

the literature’, Kings College London/ PSI• ESRC (2009) Taking Stock: A summary of ESRC’s work to evaluate the

impact of research on policy and practice, http://www.esrc.ac.uk/_images/Taking%20Stock_tcm8-4545.pdf

• ESRC (2010) Branching out: New direction in impact evaluation from the ESRC’s Evaluation Committee http://www.esrc.ac.uk/_images/Branching%20Out_tcm8-14881.pdf

• Meagher L, Lyall C and Nutley S (2008) ‘Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research’ Research Evaluation 17(3): 163-173

• Ward et al (2009) ‘Developing a framework for transferring knowledge into action: a thematic analysis of the literature’ Journal of Health Services Research and Policy 14(3): 156-164

For further information

The following three slides summarise the main research questions identified by Davies and Nutley (2008) when they were asked to define the emerging research agenda on research use and impact for the WT Grant Foundation. These questions are categorised under the three headings of:

• Knowledge source, presentation and integration

• Context and connections

• Strategies and process

Knowledge source, presentation and integration

• What models of research supply and synthesis might better support knowledge integration by potential users?

• How do different kinds of messaging and messengers affect the use of research knowledge?

• What is the role of the web in providing access to existing research?

• To what extent are policymakers and service managers conducting their own in-house research using administrative and local data?

• How is new knowledge integrated into current ways of thinking and models of practice?

Context and connections• What communities are involved in the field of interest?

How are these connected?• Are policy and practice communities networked and

does research-based knowledge flow across these networks?

• How, where and under what circumstances do practitioners source new knowledge?

• What models of research brokerage or intermediary activities have the best potential for fostering research use?

• How can lay people, service users, and others contribute more fully to evidence-informed discussions?

• How are education and CPD connected to and supportive of knowledge accumulation and integration?

Strategies and processes• Do different models of research use co-exist in

different practice settings?• What knowledge management strategies are used

across the sector?• What models of push, pull and linkage-exchange are

in place? How are these evolving and connecting?• What strategies for increasing research use and

impact have the best evidence in support of them, and how might they be tested to consolidate our knowledge base in this area?

• How can we get new knowledge on the research-action process to influence the future actions of researchers, funders, intermediaries, policy/decision-makers, and practitioners/end users?