abbott et al. - 2010 - do metrics matter

8/2/2019 Abbott Et Al. - 2010 - Do Metrics Matter

1/3

80706050403020100

Hiring decisions

Tenure decisions

Promotion

Salary decisions/

bonuses

Performance reviews/

appraisals

Allocation of

research resources

Yes No

Percentage of respondents

METRICS PERCEPTIONSAt your institution or department are metrics of scientific

performance used to any degree in any of the following?

Q:

No scientists career can be summarizedby a number. He or she spends count-less hours troubleshooting experi-ments, guiding students and postdocs,

writing or reviewing grants and papers, teach-ing, preparing for and organizing meetings,participating in collaborations, advisingcolleagues, serving on editorial boards andmore none of which is easily quantified.

But when that scientist is seeking a job,promotion or even tenure, which of thoseduties will be rewarded? Many scientists

are concerned that decision-makers puttoo much weight on the handful of thingsthat can be measured easily the numberof papers they have published, for example,the impact factor of the journals they havepublished in, how often their papers havebeen cited, the amount of grant money theyhave earned, or measures of published out-put such as the h-index.

Last month, 150 readers respondedto a Nature poll designed to gauge howresearchers believe such metrics are beingused at their institutions, and whetherthey approve of the practice. Nature also

contacted provosts, department heads andother administrators at nearly 30 research

institutions around the world to see what met-rics are being used, and how heavily they arerelied on. The results suggest that there maybe a disconnect between the way researchersand administrators see the value of metrics.

Three-quarters of those polled believe that

metrics are being used in hiring decisionsand promotion, and almost 70% believe thatthey are being used in tenure decisions andperformance review (see Metrics percep-tions). When asked to rate how much theythought administrators were relying on spe-

cific criteria for evaluation, poll respondentsindicated that the most important measureswere grants and income, number of publi-cations, publication in high impact journalsand citations of published research. And amajority (63%) are unhappy about the way

in which some of these measures are used(see No satisfaction). Too much emphasisis paid to these flawed, seemingly objectivemeasures to assess productivity, wrote abiologist from the United States. Respond-ents doubted that traditional, qualitativereview counts for much. From a field of34 criteria, Review of your work by peersoutside your department or institution andLetters of recommendation from people inyour field were tenth and twelfth, respec-tively with 2030% of the respondentsstating that their institutions placed noemphasis on these factors at all.

Yet inNature

s interviews, most admin-istrators insisted that metrics dont matter

Many researchers believe that quantitative metrics determine who gets hired and who gets

promoted at their institutions. With an exclusive poll and interviews,Nature probes to what extent

metrics are really used that way.

Do metrics matter?

d.

parkins

860

Vol 465|17 June 2010

860

NATURE|Vol 465|17 June 2010NEWS FEATURE

20 Macmillan Publishers Limited. All rights reserved10


2/3

Very

satisfied

Quite

satisfied

Not very

satisfied

Not satisfied

at all

Not applicable/

dont know

NO SATISFACTIONIn general, how satisfied are you with the way

metrics are used?

Q:

It discourages me from doingimportant research workthat may be of null association.

I am more likelyto accept an article for review if I wantto verify that it is citing a paper of mine that is near

the cusp of being counted for myh-factor.

51% of respondents said that they have changedtheir behaviour because of the way they are evaluated.

71% of respondents said that they are concernedtheir colleagues can game or cheat the systems

for evaluation in their institutions.

A great deal of politicsare involved and a focus on

These metrics can beskewedby people if they know thattheir performancewill be evaluated on metrics alone.

numbers over qualitywith regard to publications.

nearly as much for hiring, promotion andtenure as the poll respondents seem to think.Some administrators said that they ignore

citation-based metrics altogether when mak-ing such decisions, and instead rely largelyon letters of recommendation solicited fromoutside experts in a candidates field. Outsideletters basically trump everything, says RobertSimoni, chairman of the biology department atStanford University in California.

That sentiment was echoed by academicadministrators worldwide. Metrics are notused a great deal, says Alex Halliday, head ofthe Mathematical, Physical and Life SciencesDivision at the University of Oxford, UK. Themost important things are the letters, the inter-

view and the CV, and our opinions of the papers

published, he says.I dont look at impact factors of the jour-

nals a candidate publishes in, says KenichiYoshikawa, dean of the Graduate School of Sci-ence at Japans Kyoto University. These usu-ally highlight trendy papers, boom fields andrecently highlighted topics. We at Kyoto dontwant to follow boom.

Metrics are not wholly excluded, of course.Those qualitative letters of recommendationsometimes bring in quantitative metrics bythe back door. We do not look at publicationrecords or tell the reviewers to, says YigongShi, dean of the School of Life Sciences at Tsin-

ghua University in Beijing. But in reality, theydo have an impact, because the reviewers willlook at them.

Mixed messagesAdministrators may also send mixed signals:metrics dont matter, except that they do. Eachyear we collect the average performances ofpeople across various different things: studentevaluations of lectures, teaching loads, researchincome, paper output, h-indices, says TomWelton, head of the chemistry department atImperial College London. Welton insists thatthis information is reported back to research-

ers as a guideline, not a hurdle that has to beleapt over to get a promotion. Nevertheless,the fact that such measures are being madecould give the impression that they are beingrelied on heavily.

At the Massachusetts Institute of Technologyin Cambridge, Claude Canizares, vice-president for research and associate provost,says that we pay very little attention, almostzero, to citation indices and counting num-bers of publications. But, he says, if someonehas multiple publications in a higher-impactjournal, its like getting another set of letters the peers that reviewed that paper gave it

high marks.A separate reason for the disparity is that

the use of metrics can vary markedly betweencountries (see Around the world with metrics,page 862) or even between disciplines.

Poll respondents and administrators agree

that metrics have potential pitfalls. For exam-ple, 71% of respondents said that they wereconcerned that individuals at their institutionscould manipulate the metrics, for example bypublishing several papers on the same basicwork. Most deans and provosts seemed lessconcerned about that possibility, arguingthat such practices were unlikely to slip past

reviewers. But they were wary of the moreinsidious effects of using metrics.

If you decide that publishing a large numberof papers is important, then youve decided

thats what quality is, says Gregory Taylor,dean of the Science Faculty at the University ofAlberta in Edmonton, Canada. Thats alwaysa very dangerous route to go down, becausethen you get people working to achieve bythe formulae, which isnt a very good way toencourage people to use their imagination.Indeed, half the poll respondents said that theyshaped their research behaviours on the basisof the metrics being used at their university.Although many of the altered behaviours givenwere fairly innocuous for example, workharder some had the potential to compro-mise scientific ideals. It discourages me from

doing important research work that may be ofnull association, said one respondent, a USpostdoctoral fellow.

Breaking the old-boys networksDespite general dissatisfaction with the wayin which metrics are being used, some pollrespondents welcome them. Many said thatthey appreciated the transparency and objec-tivity that quantitative metrics could provide.I prefer this to qualitative metrics, wrote one,a department head in chemistry and engineer-ing from Europe. Others who were dissatisfiedwith the use of metrics at their institution said

they felt that the metrics werent being usedenough or werent being used consistently.

861

Vol 465|17 June 2010

861

NATURE|Vol 465|17 June 2010 NEWS FEATURE



3/3

The metrics can be nullified at the collegeor provost level, complained a US professor

of neuroscience. If nothing else, says Welton,the use of quantitative measures can reassureyoung researchers that the institution is notperpetuating an old-boys network, in whichpersonal connections are valued over actualachievement. Administrators who say thatthey do consider metrics in the decision-making process stress that they recognize thelimitations of such measures in defining thecareer of an individual. Researchers in differ-ent fields and different specialities publish andcite at different rates. An intimate understand-ing of the fields and more importantly theindividuals being assessed is crucial, they

say. This ultimately makes the use of metricsmore subjective by necessity.

Surprisingly, if poll respondents desirechange, its not necessarily away from quanti-

tative metrics. WhenNature

gave respondentsa list and asked them to choose the five criteriathat they thought should be used to evaluateresearchers, the most frequently chosen wasPublication in high-impact journals, followedby Grants earned, Training and mentoringstudents and Number of citations on pub-lished research. In other words, what respond-ents think they are being measured on roughlymatches what they want to be measured on.

The challenge for administrators, it seems,is not to reduce their reliance on metrics, butto apply them with more clarity, consistencyand transparency. The citation index is one of

those things that is interesting to look at, but ifyou use it to make hiring decisions or use it as a

sole or main criterion, youre simply abrogatinga responsibility to some arbitrary assessment,

says Jack Dixon, vice-president and chief sci-entific officer of the Howard Hughes MedicalInstitute in Chevy Chase, Maryland. While hesays that the institute eschews such metrics, herecognizes that they will continue to be used.All decisions are based on various criteria. Thething you hope for is that the decisions are fair,and are based upon criteria that the reviewersknow and understand. Alison Abbott, David Cyranoski, Nicola Jones,

Brendan Maher, Quirin Schiermeier and

Richard Van Noorden all contributed to this

article.

See Editorial, page 845, and metrics special at

www.nature.com/metrics. Full results of the

survey are available at go.nature.com/em7auj.

North America

acdemc ttuto the Uted stte

come comlex ytem of tte-uoted

d vte uvete, ll lgely deedet

of oe othe. although they ue ml

ytem fo evlutg eeche fo hg,

omoto d teue ( mot ce, lette

fom exet the feld hold the mot wy)

Nature tevew wth ely doze de,

ovot d detmet hed howed

codeble vto.

Fo exmle, my uvete y cloe

tteto to cddte blty to e gt

moey. ad fo ome the ouce cout: t

azo stte Uvety Teme, ovot

Betty Cld y tht fudg fom the Usntol scece Foudto egded

moe etgou th fudg fom the Us

ntol ittute of Helth, becue the

fome codeed moe comettve. By

cott, Mc kte, de of the school

of scece t Mchuett ittute of

Techology Cmbdge, y tht the

mout of fudg lmot eve come to

codeto fo eeche omoto.

if omeoe c hve tog mct o

tetol commuty wthout y fudg,

moe owe to them, he y.

My uvety offcl Cd y tht

they evlute cett o the b of 40%

eech, 40% techg d 20% evce tothe ttuto d the cetfc commuty.

But mot tht Natureoe to dmt to looe

teetto of th bedow tcully

whe t come to techg. although we

techg c be ovelooed fo omeoe wth

fluetl eech ogmme, evel

noth amec dmtto d tht

omeoe who excellet teche but oo

eeche wll td lttle chce of eg

teue.

Europe

i the Uted kgdom, college d

uvete e beholde to the couty

reech aemet Exece (raE), whch

govemet fudg bed o the qulty of

eech t detmet level. Thu ttuto

e ytemtc bout collectg metc,

cludg tttc o e ublhed d

tudet-evluto meue. it ot ug

tht ocedue tht uvete ue telly

ted to m o to d emulte the judgemet

tht we ow e beg mde by the raE, y

rod Coomb, vce-edet fo ovto

d ecoomc develomet t the Uvety

of Mchete, Uk. Qutttve metc

my become eve moe mott ude ooed ew ytem, the reech Excellece

Fmewo, but detl e tll beg debted.

itl uvete hve vey dffeet

ytem: ethe detmet o uvety

c chooe t ow tff. a d hoc commttee

et u fo evey ot the couty, wth jut

oe eeettve fom the uvety volved.

Fo ety-level oto, electo commttee

mut be guded by tetolly ecogzed

metc, uch totl umbe of ctto,

vege umbe of ctto e ublcto

d the mct fcto of joul ublhed .

The lw doe ot ecbe how much weght

thee metc hould hve deco-mg.

some itl eech ttuto hve moedeedece. pe polo d Foe, fome decto

of the FirC ittute of Molecul Ocology

(iFOM), fo exmle, y tht cte uch

mct fcto eve oly oetto.

i Gemy, the 80 bc eech ttute u

by the Mx plc socety hve well-defed

evluto ytem bed tly o metc d

tly o lette of evluto. Elewhee

Gemy, thee lttle coeu o how to

meue cdemc efomce objectvely.

Asia

The om thoughout much of Ch, south

koe d J fo hot-tem cdemc

otmet, commoly fve ye. But

becue eewl e motly utomtc,

emloymet effectvely fo lfe. The fl de

tht thee o et chedule fo omoto;

cett h the chce to move to the ext

tge h o he cee oly whe omeoe

ele leve o de.

Metc e ued hevly, lthough ome

Chee uvete hve begu to lg the

evluto method wth noth amec,

d to de-emhze metc. Y ro, de

of the school of Lfe scece t peg

Uvety, d Ygog sh, de of TghuUvety school of Lfe scece, both

y tht the ttuto ow be much of

the emet o ecommedto fom

tetol exet cddte feld.

itetol exeece lo looed o

vey fvoubly, y Moto O, de of

the school of scece t the Toyo ittute of

Techology.

Othe a ttuto hve moved towd

moe metc. Be Cho, de of the Eth

cece college t Tw ntol Cetl

Uvety Jhogl Cty, y tht the ytem

h become moe gd d qutttve

the t fve ye. Cddte fo hg o

omoto e judged 50% o eech, 30%o techg d 20% o commuty evce.

Ech ctegoy h coe, y Cho, d

mmum coe e equed fo dvcemet

fom tt to octe ofeo. i the

evluto, the mct fcto of joul

codeed, e metc bout the dvdul

to ome degee. it Ok, he y, but we

lwy fd ce whee umbe dot eflect

how hd they wo d how much they

cotbute.

around the world with metrics

862

Vol 465|17 June 2010

862

NATURE|Vol 465|17 June 2010NEWS FEATURE


abbott et al. - 2010 - do metrics matter

Documents