1

Clinical Citations:Telling the Storyof Clinical ImpactMay 26, 2016

Andrea MichalekPresident and Founder, Plum Analytics

andrea@plumanalytics.com@amichalek | @plumanalytics

2

Agenda• Our approach to analytics• Clinical Citations• Q&A

TELL THE STORY.

4Over 250 PlumX Customers Worldwide

5

6

?

7

What impact has our research had in the past 12 months?

8

How do we help our early career researchers secure grants.

9

How do we increase the visibility of our research?

10

How do we assess and evaluate our research?

11

What clinical impact does our research have?

12

Our Approach• Categorize

– Define the unique properties to measure research output by

– Group research outputs your way• Visualize

– See impact at a glance• Analyze

– Discover your research stories

13

The universe of metrics is expanding

14

15

Journal citations are the traditional form of

Scholarly Measure15

JournalImpact Factor

3.161

16

17

Researchers have Moved Online

18

Examples: Scholarly Data Exhaust

TECHNOLOGY HAS FINALLY CAUGHT UP

Measuring data exhaust to tell the story…

Make visible

what was invisible

Makemeasurablewhat was

immeasurable

22

Defining altmetrics• Altmetrics = Alternative ways of measuring

the use and impact of scholarship• Altmetrics combine traditional impact

measures (citation counts) with non-traditional measures

• Altmetrics = ALL METRICS

“Altmetrics are measures of scholarly impact mined from activity in online tools and environments” – Jason Priem, Co-Founder, ImpactStory

Using Altmetrics to Illustrate the Impact of Open Access on Graduate Student Research (Barnett, Collister, Chan 2014)

23

Benefits of Altmetrics• Broadness

– Measure the broader impact of research—beyond science

• Diversity– Measure the impact of scholarly products beyond just

the journal article. • Speed

– Outputs can be measured immediately.• Openness

– Obtaining and viewing the data is easier.

Woulters & Costas, 2012

24

MetricsCategories

USAGE(clicks, downloads, views,

library holdings, video plays)

CAPTURES(bookmarks, code forks, favorites,

readers, watchers)

MENTIONS(blog posts, comments, reviews,

Wikipedia links)

SOCIAL MEDIA(+1s, likes, shares, tweets)

CITATIONS(citation indexes, patent

citations, clinical citations)

25

USAGE

• Is anyone reading our work?

• Did anyone watch our videos?

• Usage is the #1 stat researchers want to know after Citation counts

• PlumX is the only product that includes Usage

clicks, downloads, views, library holdings,

video plays

26

• Captures indicate that someone wants to come back to the work

• Early indicator of future citations

bookmarks, favorites, readers, watchers

CAPTURES

27

• This category measures people truly engaging with your research

• Automatically uncover the conversations about your research

• Discover feedback, opinions, etc.

blog posts, comments, reviews,

Wikipedia linksMENTIONS

28

• Social media measures how well a researcher is promoting their work

• This is especially important for early career researchers to measure and understand

• Track the buzz and attention around your research output

+1s, likes, shares, tweets

SOCIAL MEDIA

29

• Citations still a standard of long-term impact

• Including citations enables side-by-side analysis with other metrics categories

• PlumX is expanding the definition of citations

– Patent citations, Clinical Citations, etc.

Citation Indexes,Patent Citations, Clinical Citations

CITATIONS

30

5 Categories in ActionArticle level

metrics are the building blocks

for PlumX

31

The Plum Print• Includes 5 categories

of metrics

• Designed to communicate engagement without a score

32

33

LIST OF GRANTS

for the Institution Example

of Analytics Social Media

by Grant

34

ClinicalCitations

35

Clinical Citations• Showcase your researchers and institution• Help your researchers

– Get credit for their work– Tell their stories for funding

• Understand the value of clinical and translational research

36

…clinical medicine draws heavily on basic science, but not vice versa. The result is that basic medicine is cited three to five times more than clinical medicine, and this is reflected in journal impact factors.”

Clinical Research & Citations

“

37

Researchers have difficulty getting funding.

38

Early-career researchers opt out of translational medicine.

39

Hospitals cannot showcase their talents.

40

41

Clinical Citations

Clinical Alerting Services

Clinical Guidelines

Systematic Reviews Clinical Trials

42

DynaMed Plus Topics• Clinical reference tool• Designed for point-of-care• Content: Extensive network of clinical

experts– Review 500 journals and 120 guideline

organizations– select the best and most appropriate evidence– confirm the clinical applicability of content– peer-review topics

43Examples of guideline organizations include:

• American Academy of Neurology (AAN)• American Academy of Pediatrics (AAP)• American Academy of Clinical Endocrinologists

(AACE)• American Association for the Study of Liver

Diseases (AASLD)• American College of Cardiology (ACC)• American College of Chest Physicians (ACCP)• American College of Obstetricians and

Gynecologists (ACOG)• American College of Physicians (ACP)• American Diabetes Association (ADA)• British Thoracic Society (BTS)• Canadian Cardiovascular Society (CCS)• Canadian Medical Association (CMA)• Canadian Task Force of Preventive Health

(CTFPHC)• Centers for Disease Control and Prevention

(CDC)

• European Respiratory Society (ERS)• European Society for Medical Oncology

(ESMO)• European Society of Cardiology (ESC)• Global Initiative for Chronic Obstructive Lung

Disease (GOLD)• Infectious Diseases Society of America (IDSA)• International Society on Thrombosis and

Haemostasis (ISTH)• National Comprehensive Cancer Network

(NCCN)• National Health & Medical Research Council

(NHMRC) [Australian guidelines]• National Institute for Health and Care

Excellence (NICE)• Pan American Health Organization (PAHO)• Royal College of Physicians (RCP)• Scottish Intercollegiate Guidelines Network

(SIGN)• World Gastroenterology Organisation (WGO)• World Health Organization (WHO)

44Showing when research makes its way into clinical practice

45

Different artifact types reveal different impacts

46A research study used for clinical advice

47Meta analysis of studies used to determine clinical practice

48Researchers contributing to clinical guidelines

49

Compare like with like

50

ANALYZE

• By metrics category• By type of output• By person• By grant• By department• By institution

51

PlumX: Built for Big Data• Built for Scale – Our goal is to track all the

world’s researchers, all of their output, and metrics for each.

51

>18 Million Researchers Worldwide

>120 Million Scholarly Articles

Map artifacts people groups

52

People not Papers

52

53

PlumX Benchmarks• Gather research output associated with

NIH grants• Calculate metrics across all 5 categories• Compare institutions using altmetrics

2012 - 2015

54

PlumX Benchmarks Features

• Select specific institutions to benchmark• Filter

– By Spending Category– By Granting Institute and Center– By Geography

• Rank institutions– For each category of metrics

• Analyze

55See Your Clinical Impact Compared to Peer Institutions

56

57

Online users expect metrics

58

59

60

Plum Printshelp

showcaseresearchers

61

62

Using metrics to gain an advantage in finding grants.

63

Big Swings in NIH Funding• $3 billion

no longer being spent on HIV research

• $65M refocused on ending AIDS epidemic

64

State of the Union Address: January 12, 2016“Let’s make America the country that cures cancer once and for all.”

65

Vice President Biden’s Memo“Over the next year, I will lead a dedicated, combined effort by governments, private industry, researchers, physicians, patients, and philanthropies to target investment, coordinate across silos, and increase access to information for everyone in the cancer community,”

66

http://report.nih.gov/NIHDatabook/Charts/Default.aspx?showm=Y&chartId=20&catId=2

4 in 5people doNOT

get funded51,073 Applications 9,241 Awards

18% success rate

NIH: The Competition for Research $$$ in 2014

67

Finding Funding Opportunities can leverage metrics

68

69

Search for HIV FundingOpportunities

in Health

70

Link directly to thegrant detail

71

FundingOpportunity

In PlumXDashboard

Funding Opps Integrates with PlumX Dashboards

72

Where we’re heading…Automatically

TargetingFunding

Opportunities

XXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

73

Feedback changes behavior

74

Metrics that update in real time provide a Feedback Loop

Feedback loops help

change behavior.

75

Tools to analyze your stars• Established researchers

– What are your key researchers doing well?– How do you measure that?

• Early career researchers– How are they successfully establishing

themselves?• Build archetypes by discipline

– What types of outputs are important?– How should these be measured?

76

Summary• Altmetrics can be used to uncover and tell

the story of your research.• We are defining (and expanding)

– Traditional categories of metrics• Usage• Citations

– As well as modern metrics• Captures• Social Media• Mentions

77

Summary• Clinical citation counts are the latest

innovation by Plum Analytics.• They can help to tell the story about

clinical and translational research.

78

Answer the questions and tell the stories about your research

Find out more information at:http://www.plumanalytics.com

79

The PlumX SuitePlumX MetricsGet more value from your institutional repository.

PlumX DashboardsKnow what is going on with your research. Without waiting years.

PlumX +GrantsGain insight about your grants.

PlumX BenchmarksSee how you compare. Now.

PlumX Funding OpportunitiesFind the right funding opportunities.