Upload File

the next era of research productivity evaluation:...

  • Home
  • Documents
  • The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity
25
ACS 243rd National Meeting San Diego, CA Mar 28, 2012 Atyab Tahir SciVal Consultant A & G Institutions www.elsevier.com The Next Era of Research Productivity Evaluation: The Multi-Dimensional Research Assessment Framework

Upload: doanhanh

Post on 30-Mar-2018

218 views

Category:

Documents


1 download

Report

TRANSCRIPT

Page 1: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

ACS  243rd  National  Meeting  

San  Diego,  CA  

 Mar  28,  2012    

 

Atyab  Tahir  

SciVal  Consultant  

A  &  G  Institutions  

 

 

www.elsevier.com  

 

 The Next Era of Research Productivity Evaluation:

The Multi-Dimensional Research Assessment

Framework      

Page 2: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Who  we  are:  A  global  company  based  in  Amsterdam,  employing  7,000+  people  in    24  countries    Partners:  A  global  community  of  7,000  journal  editors,  70,000  editorial  board  members,  300,000  reviewers  and  600,000  authors    In  the  community:  We  are  a  founding  publisher  of  global  programs  that    provide  free  or    low-­‐cost  access  to  science  and  health  information  in  the  developing  world    History:  Elsevier's  roots  are  in  journal  and  book  publishing,  where  we  have  fostered  the    peer-­‐review  process  for  more  than  125  years.      Products:  Elsevier  publishes  around  2,000  journals  and  close  to  20,000  books  and  major  reference  works  as  well  as  a  variety  of  scholarly  databases  and  other  solutions.  

 

Elsevier  serves  more  than  30  million  scientists,  students  and  health  and  information  professionals  worldwide.    

Elsevier at a glance

2  

Page 3: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

A framework to characterize and position bibliometric indicators and methods

3  

The multi-dimensional research assessment framework Top-down versus bottom-up delimitation methods Three generations of bibliometric indicators

The future of research assessment exercises lies in the intelligent combination of metrics and peer review

Page 4: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

THE MULTI-DIMENSIONAL RESEARCH ASSESSMENT FRAMEWORK

Source: Expert Group on the Assessment of University-Based Research (AUBR, 2010)  

Page 5: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Multi dimensional Research Assessment Framework Unit of assessment

Purpose Output dimensions

Bibliometric indicators

Other indicators

Individual Allocate resources

Research productivity

Publications Peer review

Research group

Improve performance

Quality, scholarly impact

Journal citation impact

Patents, licences, spin offs

Department Increase multi-discipl. research

Innovation and social benefit

Actual citation impact

Invitations for conferences

Institution Increase regional engagement

Sustainabi-lity & Scale

Internat. co-authorship

External research income

Research field

Promotion, hiring

Research infrastruct.

citation

PhD com-pletion rates

READ COLUMN

WISE

Page 6: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Multi dimensional Research Assessment Framework Unit of assessment

Purpose Output dimensions

Bibliometric indicators

Other indicators

Individual Allocate resources

Research productivity

Publications Peer review

Research group

Improve performance

Quality, scholarly impact

Journal citation impact

Patents, licences, spin offs

Department Increase multi-discipl. research

Innovation and social benefit

Actual citation impact

Invitations for conferences

Institution Increase regional engagement

Sustainabi-lity & Scale

Internat. co-authorship

External research income

Research field

Promotion, hiring

Research infrastruct.

citation

PhD com-pletion rates

Page 7: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

7

Types of Research Output

Impacts Publication/text Non-publication

Scientific-scholarly Journal paper; book chapter; monograph

Research data file; video of experiment

Educational Teaching course book; syllabus Skilled researchers

Economic Patent Product; process; device; design; image; spin off

Cultural Newspaper article Interviews; events; performances; exhibits

Page 8: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Example 1

Individual Hiring/promotion Productivity &

impact PhD date, place,

supervisor; Invitations for

conferences

Publications in international jrnls;

Actual citation impact

Page 9: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

9

SCIVAL STRATA DEMO

http://strata.scival.com/
Page 10: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Example 2

Research group Allocate resources Res. productivity &

impact Competitive

research income; Ratio research

active/total staff

Publications in international jrnls;

Actual citation impact

Page 11: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

11

SCIVAL EXPERTS DEMO

http://experts.scival.com/
Page 12: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Example 3

Department Increase regional

engagement Social benefits &

innovation % Funding from

endusers; Licenses, spin

offs

% Industry co-authored papers;

Patents Res. reports

Page 13: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

SCIVAL SPOTLIGHT DEMO

13

http://spotlight.scival.com/
Page 14: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Example 4

Institution Increase multi-

disciplinary research Res. productivity &

impact Inter-discipl.

teaching progr; Funding from

multi-disc. res. progr

% Papers in multi-disciplinary fields;

Co-authorships from different discipl

Page 15: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Assessment Solutions Unit of assessment

Purpose Aspect SciVal Solutions

Research group

Improve performance; Allocate resources

Productivity/ impact

Strata

Department; institution

Improve performance; Allocate resources;

Research infrastructure and potential

Experts

Institution Stimulate multi-disciplinary research; Improve performance;

Social benefits; sustainability; Prod./impact

Spotlight

Group, Dept, Institution

Improve performance & regional engagemt; Allocate resources;

Productivity/ impact; social impact; infrastr.

Custom Analytics

Page 16: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

TOP-DOWN VS. BOTTOM-UP INSTITUTIONAL DELIMITATION

Page 17: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Top-down institutional analysis  

author  affiliations  (incl.  verification)  

Categorize  articles  into  research  fields  

Calculate  indicators  

Compare  with  benchmarks    

Page 18: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Bottom-up institutional analysis

Compile  a  list  of  researchers  

Compile  a  list  of  publications  per    researcher  (incl.  verification)  

Aggregate  researchers  into  groups,    departments,  fields,  etc.  

Calculate  indicators;    compare  with  benchmarks    

Page 19: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

TOP-DOWN VS. BOTTOM-UP SUBJECT FIELD DELIMITATION

Page 20: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Top-down field analysis

(Pre-­‐)define  a  field  categorization  

(Pre-­‐)allocate  papers  (journals)    to  categories  

Calculate indicators per category

Page 21: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Bottom-up field analysis

Create  a  (large)  dataset    of  publications  

Cluster  papers  on  the  basis  of    

(co-­‐)  citation  or  word  analysis  

Name  the  clusters  

Calculate indicators by cluster

Page 22: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Chemistry Competency Map of US

22  

Page 23: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

THREE GENERATIONS OF BIBLIOMETRIC INDICATORS

Page 24: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

Three generations of bibliometric indicators

1 Basic No. publications No. Citations Cites per paper (e.g., Jrnl Impact Factor)

2 Normalized Relative Citation Rate No. publ. in top citation percentiles

3 Contextual Entire citation network based (e.g., Eigenfactor, SJR, Jrnl Prestige); Citation context based

Page 25: The Next Era of Research Productivity Evaluation: …bulletin.acscinf.org/PDFs/243nm_124.pdfAtyab$Tahir$ SciVal$Consultant$ A&G Institutions$ $ $ $ $! The Next Era of Research Productivity

ACS  243rd  National  Meeting  

San  Diego,  CA  

 Mar  28,  2012    

Atyab  Tahir  

SciVal  Consultant  

A  &  G  Institutional  Markets  

[email protected]    

 

www.elsevier.com  

 

 The Next Era of Research Productivity Evaluation:

The Multi-Dimensional Research Assessment Matrix    

 

Thank  You!  


The chemical most common denominator: Use of chemical ...bulletin.acscinf.org/PDFs/247nmACS69.pdf• Goal: providing quick information on chemical compounds featured in an article,

Depression Era Productivity Growth in U.S. Railroadsdata show that the decade long IT productivity boom ran out of steam in 2005. Although TFP for the private nonfarm economy grew

Labor Productivity. Definition Labor productivity : Productivity per hour- worked:

a new era in high-productivity qPCR - Thermo Fisher Scientific · high-productivity qPCR. Building on the gold-standard design elements of six generations of real-time PCR systems,

Database on ionic liquids solubilities in molecular ...bulletin.acscinf.org/PDFs/247nm109.pdf · Database on ionic liquids solubilities in molecular solvents: progress and ... UNIFAC,

REASSESSING THE PRODUCTIVITY GAINS FROM TRADE AND …€¦ · REASSESSING THE PRODUCTIVITY GAINS FROM TRADE AND FDI LIBERALIZATION: AN INDUSTRY-LEVEL APPROACH: JaeBin Ahn, Era Dabla-Norris,

An Organic Chemistry Laboratory Exercise In Information ...bulletin.acscinf.org/PDFs/BCCE_2012_P364.pdf · o A Very Brief Introduction to the Scientific Literature 1. Locate examples

A new era of productivity - B&R: Perfection in Automation

"Ecological Footprint" Sustainability Net Primary Productivity 20 – 31% 60% of freshwater runoff Anthropocene era

ASIAN PRODUCTIVITY ORGANIZATION...PRODUCTIVITY (Continuation). Relationship of Productivity to Profitability, Quality and the Environment. Roles of Stakeholders in Productivity (Productivity

Productivity, Productivity, Productivity

Representation of Markush structures — from molecules ...bulletin.acscinf.org/PDFs/240nm76.pdf · ChemAxon Markush project Goals: – Extend structural search capabilities to combinatorial

A new era of low-cost colour, high productivity, and ......Product guide | Transactional products A new era of low-cost colour, high productivity, and strong security HP PageWide Pro

South Africa National Day Special · 2019. 4. 27. · productivity and sustainable development. In celebrating the new era of Japan, the Reiwa Era, which illuminates Japan’s long

Reassessing the Productivity Gains from Trade Liberalization ...WP/16/77 Reassessing the Productivity Gains from Trade Liberalization By JaeBin Ahn, Era Dabla-Norris, Romain Duval,

EXAMPLES OF COMMUNITY-BASED DATA REPORTING …bulletin.acscinf.org/PDFs/251nm/2016_spring_CINF_047.pdf · 2016-06-06 · Examples taken from Peter Halling et al., Poster „Standards

Firm Productivity, Innovation, and Financial Development · Firm Productivity, Innovation and Financial Development. Prepared by Era Dabla-Norris, Erasmus Kersting, and Geneviève

Productivity Growth, Market Power, and Monetary Policy · 2019. 7. 6. · Productivity Growth, Market Power, and Monetary Policy Challenges in the Digital Era ECB Conference on Digitalization

Smartphone Era Productivity for Entrepreneurs

Industrial-era decline in subarctic Atlantic productivity

Open PHACTS: Semantic interoperability for drug …bulletin.acscinf.org/PDFs/251nm/2016_spring_CINF_141.pdfACS National Meeting, San Diego Herman van Vlijmen 16 Mar 2016 Open PHACTS:

Linking Resource Description Framework to …bulletin.acscinf.org/PDFs/240nm03.pdfLinking Resource Description Framework to Cheminformatics and Proteochemometrics EgonWillighagen

Microsoft Vision and Roadmap - The New Era of Productivity from Atidan

PROMOTING DIGITAL PRODUCTIVITY AND SECURITY IN THE ERA … · promoting digital productivity and security in the era of the fourth industrial revolution hopeton s. dunn d i rector,

Complementary responses to the scientific data problem ACS …bulletin.acscinf.org/PDFs/247nmACS46.pdf · 2014-03-26 · Complementary responses to the scientific data problem ACS

Avoiding the Middle Income Trap - GRIPS · 2014-09-26 · 2 Vietnam’s New Era Vietnam is entering a new era where productivity breakthrough and value creation are needed. Opening

The NCI/CADD Group's InChI Usage and Analysis of ...bulletin.acscinf.org/PDFs/247nm56.pdf · Chemical Biology Laboratory Center for Cancer Research National Cancer Institute National

a new era in high-productivity qPCR

Cheminformatics for Dye Chemistry Research: Bringing ...bulletin.acscinf.org/PDFs/247nmACS74.pdfTextile Engineering, Chemistry & Science Faculty 31 interdisciplinary faculty Chemists

FirstCuts 34 - Smart Phone Era Productivity

Intel - ECTC - Keynote - June-11-rev4 (2) Intel - ECTC... · 1970 20001980 1990 2010 2020 Productivity Era TeleComm Era Internet Era “Smart” Era er Source Wearable computing lab

Data-intensive drug design - Chemical Information …bulletin.acscinf.org/PDFs/240nm18.pdfData-intensive drug design ... make decisions. •Today, in Big Pharma, biotechs, ... examples

MODERN ERA: 1750 - 1914 IMPACT OF TECHNOLOGY. INDUSTRIAL REVOLUTION 1780s Breakthrough in productivity –Mechanization of factory, labor –Production of

Productivity in Australian Manufacturing · Productivity Commission Paula Barnes . Productivity Analysis Branch . Productivity Commission . PC-ABS Productivity Perspectives Conference,

Structure Clipper an interactive tool for extracting ...bulletin.acscinf.org/PDFs/247nm51.pdf · Structure Clipper – an interactive tool for extracting chemical structures from