demonstrating real -world impact of research

Demonstrating Real-world Impact of Research2021 Clarivate Publisher Forum

Joshua Schnell, PhD

2 March 2021

My view of the last 25 years in research assessment

90’s – Where researchers published (Cell, Science or Nature) and annual journal impact factors scores were the hot topics.

00’s – The h-index emerged (now it has dozens of variants) alongside a rapid proliferation of journals, an increase in publications and additional data sources for broader citation analysis released (Google Scholar).

An acceleration of a measurement philosophy in science and concerns about administrative burden for researchers lead to the proliferation of Current Research Information Systems (CRIS).

10’s – Broadening of the definition of impact, emergence of alternative metrics in response to social media, and a recognition of technology as a key solution to administrative burden and data quality issues. 2021 Clarivate Publisher Forum 2

A story of real-world impact

Screenshots from the webpage of the US National Institutes of Health Office of the Director Scientific Workforce Diversity: https://diversity.nih.gov/

2021 Clarivate Publisher Forum 3

https://diversity.nih.gov/

4

Why should we assess research impact?

• Advocacy: to demonstrate the benefits of supporting research, enhance understanding of research and its processes among policymakers and the public; to make the case for policy and practice change

• Accountability: to show that money has been used efficiently and effectively, and hold researchers to account

• Analysis: to understand how and why research is effective and how it can be better supported, feeding into research strategy and decision-making by providing a stronger evidence base

• Allocation: to determine where best to allocate funds in the future, making the best use possible of a limited funding pot

Taken from:Guthrie, Susan, Watu Wamae, Stephanie Diepeveen, Steven Wooding, and Jonathan Grant. Measuring research: A guide to research evaluation frameworks and tools. Santa Monica, CA: RAND Corporation, 2013. https://www.rand.org/pubs/monographs/MG1217.html.

2021 Clarivate Publisher Forum

https://www.rand.org/pubs/monographs/MG1217.html


Audience PollHow does your organization primarily use evaluation/assessment (choose one)?

1) advocacy2) accountability3) analysis4) allocation5) we don’t do regular evaluation/assessment

• Submit your answers in the WebEx window on the right hand side

6

Some of the challenges in demonstrating real-world impact

Publications are an early step in the path to real-world impact, and timelines to impact can be decades.

Not all indicators point to actions that an organization can take to increase or enhance their impact.

Many different actors with varied goals must contribute to the realization of real-world impact of research.

Not all data exist in readily accessible databases, and it’s not sufficient to rely on what’s readily available.

Golden nuggets and anecdotes are precious for demonstrating real-world impact, but they are rare.


Evaluation frameworks aid measuring real-world impact

• To unite the fields of physical science with cancer biology and oncology

• To develop trans-disciplinary teams and infrastructure

• To generate new knowledge and catalyze new fields of study

Program Goals

• $150M, $10M per center• Twelve centers were funded

2009-2014• 150 main investigators from the

fields of physics, mathematics, chemistry, engineering, cancer biology and clinical oncology

• 110 Institutions involved across the US

Network Participants Inputs

Activities

Outputs

Outcomes & Impact


Individual-Level• Publications• Patents• Grants (NIH, other)• Science Awards (innovative, translational, training)• Clinical Trials• Conference presentations• Courses and workshops taught• Trainee disciplinesCenter-Level• Cost, content and people involved in research projects, pilot projects and cores• Stage, content and people involved in collaborations• Datasets, techniques, technologies and bio-specimens generated and utilized• Enumeration and content of transdisciplinary team science activities

Network-Level• Cost, content and people involved in trans-network projects and outside network pilot projects• Stage, content and people involved in collaborations• Datasets, techniques, technologies and bio-specimens generated and utilized• People and centers involved in trainee exchanges• Location and content of outreach activities

Inputs and Activities Outputs Outcomes Relative to Comparison Groups

Generated Robust Collaborations that Resulted in Significant Transdisciplinary Research • Accelerated the formation of a greater quantity of transdisciplinary

collaborations• Accelerated the creation of a greater quantity of field convergent research• Communicated effectively across disciplines to form optimal team sizes• Effectively contributed to team based activities and outreach

Connected Physical Sciences Perspectives with Clinical Research• Accelerated the formation of a greater quantity of collaborations among physical

and physician scientists• Reduced the time between the appearance of a physical sciences perspective or

technology to its application in translational research• Acted as key investigators leading a convergence of physical sciences perspectives

within translational research and motivating transdisciplinary translational research

Bridged Oncology Research Gaps• Accelerated the generation of innovative and impactful transdisciplinary solutions

to outstanding questions in oncology (e.g. integrated transdisciplinary datasets, technologies and bio-specimens, prominently positioned in citation networks and commercialized cancer-relevant patented technology)

Trained a New Generation of Transdisciplinary Scientists• Conducted a greater quantity of transdisciplinary training activities• Attracted a greater volume of training grant applications to the PS-OC program• Graduated a greater quantity of transdisciplinary scientists• Accelerated the trainee development path toward a career in physical sciences-

oncology

Generated a Sustainable Transdisciplinary Infrastructure• PS-OC alumni sustained a transdisciplinary perspective by integrating team science

into their infrastructure and attracting new investigators to the field • Motivated the formation of other inter-/intra- national programs promoting

physical sciences perspectives in cancer research

Network-Level• Coordinate Expertise Trans-network Projects Physical or virtual infrastructure Integrative training Data Coordinating Center Research Contracts to further

support clinical translation, cross-validation and integration of datasets, techniques, technologies, bio-specimens

• Communicate with PS-OC and Broader Research Community

Center-Level• Primary leading physical scientist and cancer researcher• Research framework: 3-5 projects• Shared Resources: 1-3 non-redundant core facilities• Pilot Projects • Transdisciplinary lectures, workshops, working groups, courses

Individual-Level• Research findings: pre-award publications, grants, patents, clinical trials and business development• Research discipline• Organization associations (location, Title/Rank, department)• Degrees received• Other demographics


9

Spotlight on Outcomes/Impact

Connected Physical Sciences Perspectives with Clinical Research• Accelerated the formation of a greater quantity of

collaborations among physical and physician scientists• Reduced the time between the appearance of a physical

sciences perspective or technology to its application in translational research

• Acted as key investigators leading a convergence of physical sciences perspectives within translational research and motivating transdisciplinary translational research

Bridged Oncology Research Gaps• Accelerated the generation of innovative and impactful transdisciplinary solutions to

outstanding questions in oncology (e.g. integrated transdisciplinary datasets, technologies and bio-specimens, prominently positioned in citation networks and commercialized cancer-relevant patented technology)

Generated Robust Collaborations that Resulted in Significant Transdisciplinary Research • Accelerated the formation of a greater quantity of

transdisciplinary collaborations• Accelerated the creation of a greater quantity of

field convergent research• Communicated effectively across disciplines to form

optimal team sizes• Effectively contributed to team-based activities and

outreach


Where to start with measuring your journal’s impact?

• What is/are your journal function(s)?• Who is/are your target audience(s)? • What is unique/different about your

authors?• What is your subject focus? Single

discipline or varied? • What kinds of documents do you

publish? • What are the Open Access features of

your journal? • What is your editorial process?

10

Example


Registration - journals serve to register scientific reports and to establish priority (time) and authority (people)

Curation - journals serve to bring together related and complementary content that establishes important connections between reports

Evaluating - journals serve to manage peer-review evaluation of the science and will correct the record after publishing in the event of a retraction or corrigendum

Dissemination - journals serve to disseminate the report to readers

Archiving - journals are responsible for providing metadata and information that is used to archive the report and its findings for the scholarly record.

Nature comment 28 May 2019

Findings from a 2017 workshop

https://www.nature.com/articles/d41586-019-01643-3

Functions of journals


12Insert footer

Contributions to real-world impactSustainable Development Goals (SDG)

Download

Case study of SDG research impact: national activity


https://clarivate.com/webofsciencegroup/campaigns/sustainable-development-goals/

13

Enhancing real-world impact: HIBAR and Convergence

From Seidel, Marc-David L., Lyle H. Schwartz, and Gretchen B. Jordan. “HIBAR Research: The Frontier Is Actually an Ecosystem.” Issues in Science and Technology (August 14, 2020).

Highly Integrative Basic and Responsive (HIBAR) Research

“HIBAR research is focused around solving problems, not advancing disciplinary agendas, although disciplinary breakthroughs are often stimulated by the work.”


14

Exploring Convergence Research: An Initial Examination of What It Means and What It Hopes to Accomplish

Project PIs Joy Frechtling (Westat)

Gary Silverstein (Westat)

Project team membersAnand Desai (Clarivate)

Dean Evasius (Univ of Virginia)Lisa Gajary (Clarivate)

Shalini Misra (Virginia Tech)Josh Schnell (Clarivate)

John Wells (Westat)

The purpose of this exploratory research is to understand how “Convergence Research” has been conceptualized and implemented. The goal of the proposed research for EAGER funding is to build on the established literature and leverage existing NSF work to develop a deeper understanding of the conceptual foundations of Convergence Research, how it is practiced and its focus, and lay the groundwork for creating an assessment framework for supporting, documenting and enhancing NSF’s programs that invest in the research area.

Facilitating Transdisciplinary Integration of Life Sciences, Physical Science, Engineering, and Beyond

2014 US NASEM2021 Clarivate Publisher Forum

© 2020 Clarivate. All rights reserved. Republication or redistribution of Clarivate content, including by framing or similar means, is prohibited without the prior written consent of Clarivate. Clarivate and its logo, as well as all other trademarks used herein are trademarks of their respective owners and used under license.

Thank you for your attention!

Joshua [email protected]+1.301.448.5562

demonstrating real -world impact of research

Documents