roi and beyond - king

The library’s (potential) role in creating research metrics for their

organization

June 15, 2011

Special Libraries Association Conference

Philadelphia, PA

James King, NIH Library

Office of Research Services

THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION


http://echosphere.net/star_trek_insp/star_trek_insp.html


What does this mean?

• Snap out of it!• We live in a time when:

• Change is constant• Flexibility matters• Attitude is everything

• We need to be:• Positive• Interesting• Inventive• Persistent


http://collections.nlm.nih.gov/


http://www.morganstanley.com/views/perspectives/

tablets_demand.pdf


http://techcrunch.com/2010/01/05/quantcast-mobile-web-apple-android/


http://becker.wustl.edu/impact/assessment/index.html


Trends in scholarly research Competition for government research funding increasing

Available funding decreasing

Competition for top research faculty is on the rise

Accountability:– Research spending – Demonstrating return on investment (ROI)

Proving the institution’s quality of research to: – Prospective students – Prospective faculty members/research staff– Investors/donors

Result: Institutions seek objective data on research performance, for data-based decision making

Thanks to Thomson Reuters for this and the next couple slides


Different stakeholders in research evaluation

Government agencies/funding organizations /accreditors/state government

External Entities

IndividualsFaculty,

staff

Institutional Depts/DivisionsDeans; Department Heads

Institutional assessment, academic affairs

Institutional Management

Management, including committees, provost, vice

provosts, rectors, chancellors

Library


Relative Impact/Bench-marking

Journal actual/expected

1.14

Category actual/expected

1.48

Mean percentile 33.91

% papers in top 10% of their field

19%

% papers in top 1% of their field

5%

Special-ization

Disciplinarity index 0.97

Productivity # papers 149

Total influence

# citations 1,136

Self-citing papers

57

H-index 22

Efficiency Avg. citation rate

7.62

Percent of papers cited

43

What is a good citation count?

15

Journal and category performance ratios compare this researcher’s citation counts to the norms or “expected” rates in his discipline. Boonen is performing above average on both the journal and category levels (1.14 ,1.48).

Mean percentile reflects the percentile performance of the researcher’s work, on average. e.g. in top 33.91 percent


Capturing corporate knowledge


People

Opportunities

AwardedGrants

Publications ANALYTICS


Performance &

Evaluation

Performance &

Evaluation

Demonstration of

Capabilities

Demonstration of

Capabilities Research EnhancementResearch Enhancement

Institution/CountryInstitution/Country Researcher/GroupResearcher/Group InstitutionInstitution ResearcherResearcher

“Showcas

e”

Q4 2011

“Showcas

e”

Q4 2011

SciVerse

Scopus

SciVerse

Scopus

Office of Research

DeanDept. Head

ResearchersStudents

ResearchersResearchersAdministrato

rs

Researchers

What are my interdisciplinar

y strengths?

Who are the high

performers in my strategic priorities?

How can I act upon my

competitive advantage?

How are my researchers performing

relative to their peers?

I want to evaluate

researchers with

the criteria that

are important to me.

I want to demonstrate

my institution’s research

expertise in the areas that are

of strategic importance to

us.

I want to find researchers

within my institution with

whom I can collaborate.

I want to understand ‘who knows what’ in my institution.

I want to find funding

opportunities

I need to determine the right funding

strategy

I want to increase

success rates

What are the most highly-cited articles

and authors within

a research area?

What is the h-index of an

author?

Overview of all of an author’s or institution’s publications

Overview of SciVal Solutions

Aggregation

Primary User

Customer

Challenges

The Solution

Addresses


Who can use VIVO?Who can use VIVO?

…and many more!

http://www.vivoweb.orghttp://www.vivoweb.org


Electronic Scientific Portfolio Assistant

Multi-Dimensional Science Management Tool

eSPA supports the NIH mission by providing a comprehensive solution for program planning and evaluation staff and NIH

leadership to search, build, and collaboratively review and analyze portfolios of NIH

research projects.


Information Wants to Be Free

“On the one hand information wants to be expensive, because it's so valuable. The right information in the right place just changes your life. On the other hand, information wants to be free, because the cost of getting it out is getting lower and lower all the time. So you have these

two fighting against each other.”

Stewart Brand

First Hackers' Conference in 1984

Thank You!

James King

NIH Library, Information Architect

[email protected]


Types of citation metrics and what they measure

24

Metric Calculation Evaluator questions

Productivity -# papers-share of papers in field

-# papers-# papers in field/ papers in field

What is the research output of X? (a country, institution, researcher, etc.)

Total influence

# citations # citations What is the overall impact of a body of work?

H-index Number of papers (N) with at least N citations each.

What is the impact and productivity of a body of work?

Efficiency

Average citation rate Total citations/Total papers

What is the rate at which a body of work is cited?

Percent of papers cited

# papers with at least one citation/ Total # papers in population

How many papers get cited? Never get cited?

THE LIBRARY’S (POTENTIAL) ROLE IN CREATING RESEARCH METRICS FOR THEIR ORGANIZATION25


Relative Impact/Benchmarking

Journal performance ratio (actual/expected)

Sum of citations/sum of journal expected citation rates

Expected citation rate calculated for a journal, for each year and document type combination (e.g. JAMA, 2001, review)

Has this body of work performed better than average vis-à-vis the journals represented?

Category performance ratio(actual/expected)

Sum of citations/sum of category expected citation rates

Expected citation rate calculated for a journal category, for each year and document type combination (e.g. Physics, 1995, article)

Has this body of work performed better than average vis-à-vis the specific disciplines represented?





Aggregate Performance Indicator

Impact of an institution or country relative to an expected citation rate for the entity. The indicator is normalized for field differences in citation rates as well as size differencesamong entities and time periods.

Where ci is the total citations for institution (or country) i, Pift is the number of publications from institution i in field f and year t, cft/pft is the average citation rate of papers in field f and year t. Sums are over all years and all fields.

What is the overall impact of my institution’s or country’s research amongst the fields we publish in?





% papers in top x% of their field

e.g. 10% of Dr. Lopez’ papers are in the top 1% of their fields

What proportion of a body of work achieves a specific level of performance?

Percentile in category and mean percentile

Percentile placement of article within a journal category (e.g. oncology, 2002)

How has this body of work performed compared to the disciplines represented?




Specialization Disciplinarity index

where s is the share of papers in category i and n is the number of categories

How multidisciplinary is a body of work?

Interdisciplinarity index

where p is the share of papers in category i and n is the number of categories

How dispersed is a body of work across disciplines?

Emerging areas of research

Research Fronts Clusters of highly cited papers identified via co-citation analysis

What are the emerging areas of research in chemistry?

Indirect impact Second generation citations

Citations received by a paper’s citing papers

Did the papers citing a body of work go on to have impact?


roi and beyond - king

Education

field papers

research performance

percent of papers

research faculty

research metrics

citing papers

research spending

research area