tracking the external impacts of academic research in long-term ways and for hefce purposes

31
Tracking the external impacts of academic research in long-term ways and for HEFCE purposes Patrick Dunleavy London School of Economics and Political Science See our ‘Impact of Social Sciences’ blog at: http://blogs.lse.ac.uk/impactofsocialsciences/

Upload: patrick-boyer

Post on 04-Jan-2016

29 views

Category:

Documents


0 download

DESCRIPTION

Patrick Dunleavy London School of Economics and Political Science See our ‘ Impact of Social Sciences ’ blog at: http://blogs.lse.ac.uk/impactofsocialsciences/. Tracking the external impacts of academic research in long-term ways and for HEFCE purposes. Structure of this talk - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Tracking the external impacts of academic research  

in long-term ways and for HEFCE purposes

Patrick Dunleavy

London School of Economics and Political Science

See our ‘Impact of Social Sciences’ blog at:

http://blogs.lse.ac.uk/impactofsocialsciences/

Page 2: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Structure of this talk

1. Defining external research impacts and exploring how they operate

2. Capturing useful evidence of external impacts (starting now for the longer term)

3. HEFCE’s maximalist view of external “impacts”, and its case study method

Page 3: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Defining research impacts in an evidence-based way

A research impact is: “a recorded or otherwise auditable occasion of influence from university research upon another actor or organization”.a. Academic impacts from research are influences upon

other actors in academia or universities, e.g. as measured by citations.

b. External impacts are influences on actors outside higher education, that is, in business, government or civil society, e.g. as measured by references in the trade press or in government documents, or by coverage in mass media.

Page 4: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Primary impacts – observable

occasions of influence

Academic

work

Academic impacts

e.g. citations

External impacts

e.g. influencecounts

Academic/universityoutputs

Economic/public policy/civil society

outputs

Academic work and external impacts: a simple view

Page 5: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Primary impacts – observable

occasions of influence

Academic

work

Academic impacts

e.g. citations

External impacts

e.g. influencecounts

Dynamicknowledgeinventory

Academic/universityoutputs

Economic/public policy/civil society

outputs

Academic work and external impacts: allowing for time lags and cumulation/ delay effects

Page 6: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Media, cultural and civil society systems

Economicand

businesssystems

Publicpolicy

systems

Discovery

Application

Single discipline processesWidersociety

A naïve (science-based) view of how an academic discipline achieves external impacts

Page 7: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Media, cultural and civil society systems

Economicand

businesssystems

Publicpolicy

systems

Discovery

Integration

Application

Renewal

Single discipline processesImpactsinterface

Widersociety

How key forms of scholarship within each academic discipline begin to achieve external

impacts

Page 8: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Media, cultural and civil society systems

Economicand

businesssystems

Publicpolicy

systems

Discovery

Integration

Application

Renewal

Single discipline processesJoined-up

scholarshipImpactsinterface

Widersociety

AcademicService

Bridging

University- local

integration

How cross-disciplinary influences add to and mediate external impacts

Page 9: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Media, cultural and civil society systems

Economicand

businesssystems

Publicpolicy

systems

Discovery

Integration

Application

Renewal

Think tanks

SpecialistMedia

Professions

Entrepreneurs

Consultants

Policycommunities

NGOs

Single discipline processesJoined-up

scholarshipImpactsinterface

Widersociety

AcademicService

Bridging

University- local

integration

Media

Corporations

How the impacts interface shapes external impacts

Page 10: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

2. Capturing useful evidence of external impacts (starting now for the longer term)

Page 11: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Six metrics of academic/university involvements with external impact

• External funding linkages, especially business-to-sciences/ technology departments

• Media and specialist media presence

• Government website presence

• Academic service in government

• Reported ‘knowledge transfer’ activities

• Academics’ perceptions of impacts

Page 12: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

2007-08 2008-09 Change % Change

Collaborative research (£000s) 697,030 731,734 34,704 5

Contract research

Total number of contracts 27,051 28,111 1,060 4

Total value of contracts (£000s) 834,627 937,373 102,746 11

Consultancy

Total number of contracts 64,292 64,025 -267 0

Total income (£000s) 334,768 331,541 -3,227 -1

of which, number with SMEs 22,802 20,596 -2,206 -11

number with large business 10,499 10,360 -139 -1

Patents

Number of new patent applications 1,898 2,097 199 9

Number of patents granted in year 590 653 63 10

Intellectual property income

Total revenues (£000s) 66,271 124,368 58,097 47

Total costs (£000s) 21,003 27,794 6,791 24

Spin-off companies

Number created 2,223 2,289 66 3

Estimated external investment received (£000s) 89,497 154,451 64,954 42

Universities’ key interactions with business in the UK

Page 13: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Proportion of research found in UK national pressfrom different disciplines (2008)

0 10 20 30 40 50 60

Geography

Languages

Anthropology

Computer science

Humanities*

Arts

Education

Architecture

Communication and Media

English

Philosophy

Sociology

History

Law

Psychology

Business and Finance

Economics

Politics

Science and Technology

Health and Medicine

Number of items

Page 14: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Creative and arts

Science and engineering

Medical sciences

Mixed

HSS

0 5 10 15 20 25

Archeology

Design

Geography

Economics

Technology

Management and media studies

Politics and democracy

Transport policy

Public policy and policy studies

Education

Development

Psychology and psychiatry

Employment and work

Criminology and criminal policy

Health policy and public health

Law and legal

Medicine

Social policy and anthropology

Percentage of all research projects found

Proportion of research found on UK government websites from different disciplines

Page 15: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

University funding bodies

Wider government

bodies

All bodies

Top academics (managing universities) 28 21 24 Heads of department 15 12 13 Professors 58 57 57 Lecturers 0 11 7 Total 101% 100% 100% Number of academics 80 131 232

Academic Service: University staff involved in UK central government quasi-government agencies

Source: Griffiths, 2010

Page 16: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Knowledge transfer practice % academics involved 2009

% academics involved 2008

type of academic

activity

Attending conferences 87 56 general

Informal advice to business

57 35 application

External lectures 65 34 application

Networks 67 32 integration

Joint publication 46 26 application

Advisory boards 38 22 service

Student projects/placements

33 20 renewal

External visits 19 application

Formed/run consultancy 14 18 application

Contract research 37 18 application

Undertaken consultancy 43 17 application

Academics’ reporting of ‘knowledgetransfer’ activities - top 11 activities

Page 17: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Knowledge transfer practice

% academics involved 2009

% academics involved 2008

type of academic activity

Been involved in consortia

35 17 application

Joint research 49 17 application

Post-course placements na 14 renewal

Prototyping and testing 10 na application

Patenting 7 12 application

Licensed research 5 10 application

Standards forum 31 10 application

Spin out companies 4 7 discoveryapplication

Enterprise education 6 4 renewal/ application

External secondment 10 3 application

Academics’ reporting of ‘knowledgetransfer’ activities – next 10 activities

Page 18: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Area of external impacts Level of impact

Social scientists only

Academics spanning social sciences and humanities

All respondents (including academics across all social science and humanities)

Public policy Actual 4.6 3.6 3.4Potential 6.0 5.5 5.1

Civil society Actual 4.5 4.1 4.1Potential 5.6 5.3 5.1

Public debates and culture

Actual 3.9 5.0 4.6Potential 5.4 5.9 5.6Actual 3.6 3.1 3.0

Potential 4.7 4.0 3.8Science and technology

Actual 3.1 3.4 2.9Potential 4.5 4.1 3.9

Social science and humanities academics’ perceptions of external impacts (2008)

Page 19: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Eight longer-term steps towards universities boosting their external impacts

• Clean up ‘multiple branding’ problems• Reappraise events programmes• Develop an impacts file for all individual academics,

and then for departments• Pull together databases and services to give improved

‘customer relationship management’ of research clients – e.g. bring in CRM systems

• Start (and fill up) an online depository• For all research on the closed-web, publish a useful

open-web version (see session D)• Improve professional communication – starting with

multi-author blogs (see session D)• Work better in networks (see Panel 2)

Page 20: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

3. HEFCE’s expansive view of external impacts, and case study method

Page 21: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Primary impacts – observable

occasions of influence

Academic

work

Academic impacts

e.g. citations

External impacts

e.g. influencecounts

Dynamicknowledgeinventory

Academic/universityoutputs

Economic/public policy/civil society

outputs

PRIMARY EXTERNAL IMPACTS OF RESEARCH

Page 22: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Primary impacts – observable

occasions of influence

Changes in organizational

activities or outputs

Academic

work

Academic impacts

e.g. citations

External impacts

e.g. influencecounts

Dynamicknowledgeinventory

Economic/public policy/civil society

outputs

Multiple otherfactors

WHAT MORE DOES HEFCE NEED? – 1. EVIDENCE OF CHANGED OUTPUTS

DEMONSTRATED CETERIS PARIBUS,

CONTROLLING FOR ALL OTHER

INFLUENCES

Page 23: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Primary impacts – observable

occasions of influence

Changes in organizational

activities or outputs

Changes in societal

outcomes

Academic

work

Academic impacts

e.g. citations

External impacts

e.g. influencecounts

Dynamicknowledgeinventory

Economic/public policy/civil society

outputs

Economic/public policy/

civil societyoutputs

Multiple otherfactors

WHAT MORE DOES HEFCE NEED? – 2. EVIDENCE OF CHANGED OUTCOMES

AGAIN CETERIS

PARIBUS, CONTROLLING

FOR ALL OTHER

INFLUENCES

Page 24: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Primary impacts – observable

occasions of influence

Changes in organizational

activities or outputs

Evaluations of outcome changes

– against societal values

Changes in societal

outcomes

Academic

work

Academic impacts

e.g. citations

External impacts

e.g. influencecounts

Dynamicknowledgeinventory

Economic/public policy/civil society

outputs

Economic/public policy/civil

societyoutputs

Multiple otherfactors

+ ? -

WHAT MORE DOES HEFCE NEED? – 3. PROOF OF POSITIVE SOCIAL BENEFITS

Page 25: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Four star Exceptional: Ground-breaking or transformative impacts of major value or significance with wide-ranging relevance have been demonstrated

Three star Excellent: Highly significant or innovative (but not quite ground-breaking) impacts relevant to several situations have been demonstrated

Two star Very good: Substantial impacts of more than incremental significance or incremental improvements that are wide-ranging have been demonstrated

One star Good: Impacts in the form of incremental improvements or process innovation of modest range have been demonstrated

Unclassified The impacts are of little or no significance or reach; or the underpinning research was not of high quality; or research-based activity within the submitted unit did not make a significant contribution to the impact.

HEFCE’S SCORING BANDS FOR IMPACT CASE STUDIES’ ‘SIGNIFICANCE’/ VALUE

AND ‘REACH’/RELEVANCE

Page 26: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

low high medium

low

m

ediu

mhi

gh

Reach (or relevance)

Sig

nif

ica

nc

e (o

r v

alu

e)

CHARTING ALL HEFCE’S CRITERIA

Page 27: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

low high medium

low

m

ediu

mhi

gh 4*

exceptional

3*excellent

Reach (or relevance)

Sig

nif

ica

nc

e (o

r v

alu

e)

CHARTING ALL HEFCE’S CRITERIA

Page 28: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

low high medium

low

m

ediu

mhi

gh 4*

exceptional

3*excellent

2*very good

good1*

Reach (or relevance)

Sig

nif

ica

nc

e (o

r v

alu

e)

CHARTING ALL HEFCE’S CRITERIA

Page 29: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

low high medium

low

m

ediu

mhi

gh

Uunclassed

4*exceptional

3*excellent

2*very good

good1*

Reach (or relevance)

Sig

nif

ica

nc

e (o

r v

alu

e)

CHARTING ALL HEFCE’S CRITERIA

Additional criteria:

▪ Impact occurring in the

last 15 years

▪ Quality of Underlying

Research

▪ Distinctiveness of Research

Page 30: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Short-run steps to prepare for HEFCE case studies

• Develop a large long list of possible case studies• Collate grants, publications, contacts and

testimonials evidence from (senior) academics on all potential cases, especially for 2005-11

• For ‘probable’ cases - firm up outside evidence of outputs, activity

and outcome changes; - push back pre-history to 1997 if needed• Enlarge single-person cases wherever feasible• Strengthen ‘sub-group’ case studies, by adding

members (if feasible, without diluting ‘quality’), and re-branding for greater coherence by 2013

• Monitor potential cases closely from now to 2013, and discuss with uninvolved senior case-writers

Page 31: Tracking the external impacts  of academic research in long-term ways  and for HEFCE purposes

Thank you for listening

See our ‘Impact of Social Sciences’ blog at: http://blogs.lse.ac.uk/impactofsocialsciences/

From there, download the free Handbook: Maximizing the Impacts of Social Science

Research