Bioethics research and the ERA:How can we make our research

count in academia and in practice

Wendy Rogers, CAVE, Mq UniCatriona Mackenzie, CAVE, Mq Uni Katrina Hutchison, CAVE, Mq Uni

Ainsley Newson, VELIM, USyd

What is the ERA?: An exercise to identify and promote excellence in research in Australian higher education institutions (HEIs)

1. To establish an evaluation framework providing assurance of the excellence of research in HEIs

2. To provide a national stocktake of discipline-level areas of strength and areas with opportunity for development

3. To identify excellence across the full spectrum of research performance

4. To identify emerging research areas and opportunities for development

5. To allow for national and international comparisons of Australia’s research for all disciplines

Excellence in Research for Australia 2012, National Report, ARC

What is the ERA used for?

ERA is a key performance indicator in the 2011-13 mission-based compact between the Australian gov’t and the HEIs. The ERA has or will inform:

• the Research Workforce Strategy• the 2014-16 mission-based compacts• the allocation of some funding through the

Sustainable Research Excellence in Universities initiative

• the new minimum standards for higher ed research and training administered by TEQSA

Commonwealth research objectives

“To progressively increase the number of research groups performing at world class levels as measured by international performance benchmarks”

“To promote collaboration, including collaboration between researchers within Australia and internationally”

Mq Uni mission compact 2011-13, p 26

Mq Uni mission compact

“We will consider the ERA results in the context of our research and research training strategies to determine where additional resources should be made available to ensure we achieve world standing in all of our areas of research”

Mq Uni mission compact 2011-13, p 28

2012 ERA indicators1. Indicators of research quality: publishing profile,

citations, ERA peer review, peer-reviewed research income

2. Indicators of research volume and activity: total research outputs, research income, other research items

3. Indicators of research application: research commercialisation income and other applied measures

4. Indicators of recognition: based on range of esteem measures

Discipline specific indicators: quality and volume

FoR# Citations Peer r/v of 30% of publications*

Outlet frequency Research income

11 Yes - Yes Yes

1801 - Yes Yes Yes

2201 - Yes Yes Yes

2203 - Yes Yes Yes

# Field of Research11: Medical and health sciences1801: Law2201: Applied ethics2203: Philosophy* Books, book chapters, journal articles, conference publications, non-traditional research outputs

Discipline specific indicators: esteem

FoR Prestigious editing

Learned academy

Research fellowships

Statutory committee membership

11 - Yes Yes Yes

1801 Yes Yes Yes -

2201 Yes Yes Yes -

2203 Yes Yes Yes -

11: Medical and health sciences1801: Law2201: Applied ethics2203: Philosophy

Discipline specific indicators: applied measures

FoR Patents NHMRC-endorsed guidelines

Research commercialisation income

11 Yes Yes Yes

1801 - - -

2201 - - Yes

2203 - - Yes

11: Medical and health sciences1801: Law2201: Applied ethics2203: Philosophy

2012 ERA results: FoR 22 Philosophy and Religious Studies

FoR 11 and 18 contribution to the national landscape

Medical and health sciences Law

2012 ERA results: FoR 2201 Applied ethics

2012 ERA results: FoR 18 Law

ERA-related challenges for bioethics research1. Neither fish nor flesh nor good red herring*

• Not accepted/counted for home department FoR• Not considered scholarly/of high quality by

discipline specific peer review/panel

2. Tensions between:• Performing relevant and collaborative research • Communicating with appropriate audiences • Turning inward to FOR-specific journals

3. Mismatch between esteem measures for philosophy and law and likely achievements of bioethicists

4. Low overall rating of Applied Ethics cf Law and Philosophy

*The Proverbs of John Heywood, 1546

Case Study: ARC Linkage Project

On the Cutting Edge - Promoting Best Practice in Surgical Innovation

(ARC LP110200217)

Project Team by discipline(at Macquarie University unless specified)

Chief Investigators• Prof Wendy Rogers (bioethics:

philosophy and ASAM)• Dr Jane Johnson (philosophy)• Dr Mianna Lotz (philosophy)• Prof Denise Meyerson (law)• Mr George Tomossy (law)• Prof Guy Maddern (surgery,

University of Adelaide)• Prof Tony Eyers (surgery,

ASAM)

Partner Investigators• Dr Angela Ballantyne

(bioethics, University of Otago)

• Prof Colin Thomson (Law, UoW and Houston Thomson)

Postdoc + RA• Dr Katrina Hutchison

(philosophy)• Ms Swantje Lorrimer-Mohr

(philosophy)

Ethics of surgical innovation

Law and regulation of surgical innovation

Partner Relations

Taxonomy: what is innovation?

Qualitative research: views of stakeholders

Admin and logistics

Weaving a research thread: multiple strands and multiple audiences for the outputs of each strand

Publications 2012/13

Innovative surgery: the ethical challenges J Johnson & W Rogers, Journal of Medical Ethics 2012; 38: 9-12.Controversy Over Vertebroplasty J Johnson, W Rogers & L Jeffree, Journal of Medical Imaging and Radiation Oncology 2012; 56: 449–451Innovative Surgery and the Precautionary Principle D Meyerson, Journal of Medicine and Philosophy (accepted 8/3/12)Addressing within-role conflicts of interest in surgery W Rogers and J Johnson, Journal of Bioethical Inquiry 2013; 10 (2): 219-225.Identifying surgical innovation: A qualitative study of surgeons’ views Rogers WA, Lotz M, Hutchison K, Pourmoslemi A, Eyers A. Annals of Surgery (in press) What Chappel v Hart really stands for and some ramifications for innovative surgery. D Meyerson. ANZ Journal of Surgery (in press)Reconceiving early-stage surgical innovation as sui generis surgical research M Lotz Theoretical Medicine and Bioethics (accepted)

Other related papers

The ethical challenges of innovative surgery - a response to the IDEAL recommendations. J Johnson, W Rogers, M Lotz, C Townley, D Meyerson, G Tomossy. Lancet 2010, 376 (9746): 1113-1115.What Can Feminist Epistemology do for Surgery? M Walker, W Rogers. Hypatia (accepted).Defining surgical innovation: a "family resemblance" concept? K Hutchison, W Rogers, A Eyers, M Lotz. Submitted to Journal of the American College of Surgeons (under review).

Journals and FoRs in which Linkage outputs are publishedJournal title ERA 2012 FoR codes

Lancet 11Annals of Surgery 11Journal of the American College of Surgeons

1103

ANZ Journal of Surgery 1103Journal of Medical Imaging and Radiation Oncology

1103/1112

Theoretical Medicine and Bioethics 1103/2201Journal of Medical Ethics 1199/2201Journal of Bioethical Inquiry 1801/2201Journal of Medicine and Philosophy 2201/2203Hypatia 2203/1606

Journal ranking of Linkage outputs by IF and journal rankings

Journal title JIF Journal Rank

Lancet 39.060 2/151 (Gen and int med)

Annals of Surgery 6.329 1/198 (Surgery)

Journal of the American College of Surgeons 4.500 7/198 (Surgery)

ANZ Journal of Surgery 1.500 91/198 (Surgery)

Journal of Medical Ethics 1.419 4/18 (Medical ethics)9/48 (Ethics)

Journal of Medical Imaging and Radiation Oncology

0.981 97/120 (Radiology)

Theoretical Medicine and Bioethics 0.962 17/48 (Ethics)

Journal of Medicine and Philosophy 0.812 21/48 (Ethics)

Journal of Bioethical Inquiry 0.594 16/18 (Medical ethics)28/48 (Ethics)

Hypatia 0.338 25/38 (Women’s Studies)

Journal ranking of Linkage outputs by ERA 2010 listRank by ERA 2010 list 2010

rankRanked by IF IF

Lancet A* Lancet 39.060

Annals of Surgery A* Annals of Surgery 6.329Hypatia A* Journal of the American College

of Surgeons4.500

Journal of Medicine and Philosophy

A* ANZ Journal of Surgery 1.500

Journal of Medical Ethics A Journal of Medical Ethics 1.419

Journal of Bioethical Inquiry A Journal of Medical Imaging and Radiation Oncology

0.981

Journal of the American College of Surgeons

B Theoretical Medicine and Bioethics

0.962

Theoretical Medicine and Bioethics

B Journal of Medicine and Philosophy

0.812

ANZ Journal of Surgery C Journal of Bioethical Inquiry 0.594

Journal of Medical Imaging and Radiation Oncology

C Hypatia 0.338

The RAE/REF in the United KingdomWhat has been the experience for academics in bioethics?

Centre for Values, Ethics and the Law in Medicine | The University of SydneyDr Ainsley Newson | Senior Lecturer in Bioethics

The UK ‘Research Excellence Framework’ (REF)

› Assesses quality of research in HEIs in UK via expert review- Professional judgement of discipline-based expert panels (36 Units of

Assessment (UoA) divided into 4 main Panels), each referencing a common framework

- 2008-2013; census date 31 Oct 2013. Results 2014.

› Three purposes to the REF:I. Assessment outcome will inform selective allocation of research funding to

HEIs from 2015-16;

II. Provide accountability for public investment in research and produces evidence of the benefits of this investment; and

III. Provide benchmarking information and establish reputational yardsticks

24

The UK ‘Research Excellence Framework’ (REF)

› Three elements to a REF submission (result per panel, per institution):1. Outputs (65%): assess the “originality, rigour and significance” of submitted

research outputs, with reference to international quality standards. Four submitted outputs per staff member.- Some panels use citation data in quality assessment- Journals not ranked, but individual outputs graded on a 5-point scale of

quality

2. Impact (20%): the “reach and significance” of impacts on economy, society and/or culture that were underpinned by excellent research undertaken by the UoA since 1993

3. Environment quality (15%): its “vitality and sustainability”; includes equality and staff development

25

How is bioethics assessed in the REF?

› Several places where bioethics could go: - Panel A, UoA 2: Public Health, Health Services and Primary Care*

- Panel C, UoA 20: Law

- Panel C, UoA 22: Social work and social policy

- Panel C, UoA 23: Sociology

- Panel D, UoA 32: Philosophy (which has at least some members who may be ‘sympathetic’ to bioethics research)

- Panel D, UoA 33: Theology and Religious Studies

* = this sub-panel will make use of citation data

› Problem: bioethics research does not have its own sub-panel

› Problem: bioethics research gets rejected (internally) for most panels

26

Has anyone pointed this issue out?

› Kong et al (Institute of Medical Ethics): BMJ Letter (2011;342:d3968 )

- “Conceptual error” to exclude theoretical medical ethics

- Given central part of ethics education in medical schools, poor that it is not reflected in valuing research excellence

- Preventing integration of sciences and humanities

› Result: no change / no response

› Concern lack of REF recognition will deter researchers from entering field

27

Example

› Can affect job prospects:- Recent Senior Lecturer/Lecturer in Global Ethics job at the University of

Birmingham, UK:

- “Candidates should be able to demonstrate achievement against the following criteria:

- a strong research record including evidence of independent research in high quality (strong REF) publications in philosophy and support and recognition as appropriate to level is already clearly evident.

- (Please note that while global ethics is multidisciplinary the successful candidate will need to be eligible for inclusion in the Philosophy REF and is likely to have expertise in applied philosophy, moral philosophy and/or political philosophy.)”

‘Straw poll’ of UK colleagues’ thoughts on REF

› Many academics in bioethics experience problems in being returned- Medical Schools are rejecting bioethics research that is not empirical

- Philosophy departments are rejecting bioethics research that is not advancing theory in philosophy

- Some (senior) academics are attempting to get submitted to different panels

- Some (senior) academics are attempting to get submitted under Impact alone

29

Twitter discussion, early June 2013

› “applied ethics is rumoured to get a better hearing in philosophy since Heather [Widdows] is on the panel - that said everyone seems to be doing their own thing and still avoiding that panel - I heard Lancaster… is going in Social Policy instead…” – David Hunter

› “some bioethics goes into law. But - as usual - it will likely suffer _wherever_ it goes” – Richard Ashcroft

› “and some in education, but share Richard’s concerns about subject's consideration” – Deborah Bowman

› “wherever we're put, the dominant view is "all very interesting, but not Philosophy"... Or law, social policy, etc. The publications are too short, too long, wrong method. Too many authors, too few, too discursive, not enough data etc. etc. etc. So in sum: I hate the whole charade.” – Richard Ashcroft

30

Some UK Colleagues’ views:

Personal experience

› Was returned under 2008 RAE, albeit with 2 publications due to ‘early career’ status, despite having 4 eligible publications

› 2012: - Not submittable for the Health Services Research Panel. Informed that the

publications lacked sufficient empirical work

- Not submittable to the Philosophy Panel. Informed that the publications do not advance philosophical theory

- Rather they apply it to examples in medicine and health care

- Bristol academic is chairing the Philosophy UoA…

› Affects morale – i.e. working in a discipline that is not recognised, nor understood by those outside of it?

31

Participated in RAE (2008) and internal mock REF (2012)

The difficulties of ‘Impact’

› Definition of Impact (20%): “An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”

› Includes (but not limited to) effect, change or benefit to:- The activity, attitude, awareness, behaviour, capacity, opportunity, performance,

policy, practice, process or understanding

- Of an audience, beneficiary, community, constituency, organisation or individuals

- In any geographic location whether locally, regionally, nationally or internationally

› Impact has to:- Occur during the relevant time period (5-year window)

- Be underpinned by your own published research (20 year time window)

- Underpinning research needs to be 2* or higher in quality (i.e. recognised internationally, internationally excellent or world leading)

› Demonstrated via statement and submission of case studies32

REF: ‘Impact’

› NICE guideline citations – demonstrate change as a result of research

› Evidence of changes in clinical or policy practice, especially where there can be quantified

› Check the NHS Map of Medicine (http://eng.mapofmedicine.com/evidence/map/index.html) to see if your research is referenced.

› Department of Health reports, policy statements and even minutes of meetings which cite your research.

› Check guidelines of other counties (e.g. US, Netherlands).

› Press and media publications

33

How we were told to try to impact:

Can bioethics make an ‘Impact’??

- X is renowned in bioethics for her work on A. This leads her to being appointed to a relevant commission examining policy/law on A. That commission recommends a particular policy response, which is taken up by the government.

› How can X demonstrate this impact in the REF?- X’s research needs to be linked explicitly to the policy/law in question

- X’s research needs to be 2* quality or above (internationally recognised)

› A lot of the ‘engagement’ work we do in bioethics may be too general to meet these more measurable/quantifiable/tangible standards

Thanks to Dr Richard Huxtable, University of Bristol

34

Case study:

Lingering questions…

› How should ‘quality’ in research be defined?

› Is one approach to fit all disciplines appropriate?

› Should impact only be that which is quantifiable?

› What should be done about institutional game playing? - ‘On paper’ bioethics has plenty of places to ‘count’, but institutions don’t follow

this locally

› What should count as a citation? And who should measure it? Be mindful of biases in citation- E.g http://kieranhealy.org/blog/archives/2013/06/19/lewis-and-the-women/

35

Measuring impact (Australian Government discussion paper June 2013)

• Impact understood as ‘benefit’: ‘positive economic, social and environmental changes that can be attributed to university research’

• Engagement: ‘the pathways from university research activities to uptake and adoption of research outputs by research users and the realisation of subsequent economic, social and environmental benefits. These pathways may encompass activities (such as knowledge transfer and dissemination), policy frameworks, governance arrangements and skill development’

Measuring impact. Aims?

1. ‘Demonstrate the public benefits attributable to university research

2. Identify the successful pathways to benefit3. Support the development of a culture and practices within

universities that encourage and value research collaboration and engagement

4. Further develop the evidence base upon which to facilitate future engagement between the research sector and research users, as well as future policy and strategy.’

Measuring impact: 1) Metrics

1. Category 2, 3, 4 research income 2. Patents3. Licenses4. # staff in research commercialisation5. # staff & postgrads in start up

companies6. Engagement: online publications,

professional & applied research publications

7. $$s and numbers from research engagement events

Measuring impact: 2) Case studies

• Minimum of 5 per institution, set maximum• Reported using 4 digit FoR codes plus SEO (socio-

economic objective) codes• Assessed by panels consisting of at least 70% research

users• Assessment in terms of reach (spread or depth of

benefit); significance (intensity); contribution (of research to reported benefit); validation (corroboration of impact claims)

• Selected case studies reported in dedicated report, MyUniversity website, mission-based compacts.

Case study on impact: Submission for Linkage to NHMRC 2011 consultation on “Identifying and Managing Conflicts of Interest: Draft Policies …”Five points from our submission are reflected in the 2012 final version of these guidelines:1. Include within role conflicts: 1.4.2(f) new section on within role

interests2. Try to find experts without conflicts: 1.2.2 acknowledges that

technical knowledge should not override all other considerations; and 1.2.4: make efforts to find experts without conflicts

3. Provide education on CoI: 3.1.2 mentions education of appointees on CoI

4. Criticism of $$ thresholds: thresholds now omitted5. Appoint chair with no (cf“no major”) conflict: 1.5.1(d) chair to have

no CoI ….

What can we do about it?

1. Engage:• Know your FoRs, journal codes and the

rules for reassigning these (Exemptions under 5.4.3.1)

• Be strategic (double publish)• Plan strategy with department head and

ERA leaders

2. Catalogue potential measures of impact• Committees, meetings• Commissioned reports• Submissions, eg to legislation reviews• Track any influence/mention

Thank you