daniele checchi, irene mazzo1a and sandro momigliano · science, technology and innovaon indicators...

Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France

DanieleChecchi,IreneMazzo1aandSandroMomigliano

Na#onalAgencyfortheEvalua#onofUniversi#esandResearchIns#tu#ons(ANVUR)ROMA,Italy

www.anvur.it


Aimofthestudyandresultsinanutshell

•  Performance based funding are oEen subject to the cri2cismthat they may produce cumulaAve vicious cycles, since worseperforming ins2tu2ons receive lessmoney,making itmore andmoredifficulttocatchupbeLerperformingones.

•  In thisworkweprovidefirst evidenceon this issue, comparingthe results achieved by Italian universi2es in the two na2onalevalua2onexercises,whichtookplaceinItalyin2012and2016.

•  AEer making the two exercises comparable, we find that thedispersion in researchqualityof the ItalianuniversiAes falls inthe second exercise. Convergence is largely due to changes inrela2ve performance of researchers par2cipa2ng to bothexercisesandtothehiringdecisionsofuniversi2es.

2


Context:asimplifiedoverviewofthetwoItalianresearchassessmentexercises(1)

3

Bothexercises:

-  followedthesameapproachbasedonacombina2onofpeerreviewandbibliometricmethods.Theyreferred,respec2vely, totheyears2004-2010and2011-14.

-  Evalua2onwascarriedoutbypanelsofexperts(about450expertsin eachexercise,with61whopar2cipated toboth)who recruitedabout15,000externalrefereesineachexercise

- were highly publicized, making a noteworthy impact on thereputa2onofins2tu2onsinvolved

-  hadadirectimpactonthepublicfundingoftheuniversi2es,basedontheaverageperformanceoftheirresearchers’staff.Comparedtoapropor2onaldistribu2on,theschememodifiedalloca2onoftotalpublicfundstoindividualins2tu2onsbyupto4.5%(onaverageby2%)inthesecondexercise.


Context:asimplifiedoverviewofthetwoItalianresearchassessmentexercises(2)

4

• 


Basicdata

5

QuarAlesVQR2004-2010 VQR2011-2014

#universiAes #researchers #universiAes #researchers

1stquarAle 22 35,415 22 33,188

2ndquarAle 23 14,075 23 13,540

3rdquarAle 23 4,645 23 4,753

4thquarAle 23 823 23 1,095

Total 91 54,958 91 52,576

Theuniversi2esthatpar2cipatedtotheexerciseshadverydifferentsizes,intermsofresearchersstaff.


HarmonizaAonofgrades

6

ResearchproductallocaAon VQR2004-2010 VQR2011-2014 HarmonisaAon

Fraud (self)plagiarism:-2 notassessable:0

limited:0

Wrongdeliverables notassessable:-1 notassessable:0

Absenceofdeliverables missing:-0.5 notassessable:0

Decile*1

limited:0

limited:0Decile*2

Decile*3

acceptable:0.1Decile*4

Decile*5

Decile*6 acceptable:0.5fair:0.4 fair:0.4

Decile*7good:0.8

Decile*8high:0.7 high:0.7

Decile*9excellent:1

Decile*10 excellent:1 excellent:1

Ex-antemeanscore 0.41 0.35 0.32

Ex-antestd.dev.ofscores 0.43 0.33 0.36

*Decileintheworlddistribu2onofquality


DistribuAonofuniversiAes’scoresinthetworesearchassessmentexercises(1)

Fortes2ngtheconvergence/divergenceofuniversi2esintermsofqualityofresearch,wecomputethefollowingsta2s2cs

7

Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France8

DistribuAonofuniversiAes’scoresinthetworesearchassessmentexercises(2)

0,0

0,5

1,0

1,5

2,0

2,5

3,0

-1,0 -0,8 -0,6 -0,4 -0,2 0,0 0,2 0,4 0,6 0,8 1,0 1,2

VQR2011-2014H VQR2004-2010 VQR2004-2010HStd.dev. 0,190,27 0,25WeightedStd.dev. 0,11 0,16 0,14

•  “VQR2011-14H”isvisiblymoreconcentratedaroundthemeanthanVQR2004-10.

•  Theharmoniza2onreducesthedeclineinthedispersionofthescoresbetweenthetwoexercises.

•  Convergencebetweenthetwoharmonizedexercisesissta2s2callysignificantwithFvalueequalto1.77(p-value<1%).


AmodelforexaminingthedynamicsofuniversiAes’grades

9

whereγmeasuresthedependencefromini2alcondi2ons:•  𝛾<0impliesconvergence(i.e.regressiontothemean);thecloserthe<0impliesconvergence(i.e.regressiontothemean);thecloserthe

es2mated𝛾to-1thequickeristheconvergenceto-1thequickeristheconvergence•  𝛾>0(oritisbelow-2)impliesdivergence>0(oritisbelow-2)impliesdivergence

For beLer understanding the causes of convergence we split theresearchers involved in the two exercises (R1 and R2, respec2vely) intosubgroups,accordingtothefollowingdecomposi2on:

“perm”!whopar2cipatedinbothexerciseswiththesameacademicposi2on“1only”!whopar2cipatedonlytothefirstexercise“2only”!whopar2cipatedonlytothesecondorwerepromotedaEerthefirst

exercise.

Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France10

Applyingthemodelto4differentpopulaAons(1)

• Highlysignificantnega2veregressioncoefficients

=>CONVERGENCE

• Coefficientofthe4thregressionisthelargest(inabsolutevalue)=>relaAvelylargecontribuAontoconvergenceofrecruitedandpromoted(R2only).


R1+R2:theregressioncoefficient(-0.383)showsconvergence.Onaverage,inthesecondexerciseuniversiAesreducebymorethanathirdtheiriniAaldistancefromthemeangrading.

Rperm:theregressioncoefficient(-0,405)showsafasterconvergencetothemeanifweconsideronlythoseresearchersparAcipaAngtobothexerciseswiththesameacademicposi2on.

Rperm+R1only:includingtheeffectscomingfromreArement(largelydeterminedbyage),theregressioncoefficient(-0,372)getsclosertozero.

Rperm+R2only:theregressioncoefficient(-0.413)isthehighest(inabsolutevalue),sugges2ngarelaAvelylargecontribuAontotheconvergencecomingfromuniversiAesdecisionsonrecruitmentandpromoAons.Astheperformancerelatedschemeislargelytargetedtouniversi2es,itisherethatweshouldseethelargestimpactofthereputa2onandmonetaryincen2ves.

11

Applyingthemodelto4differentpopulaAons(2)


Conclusionsandfuturework(1)

In this paper we look at the performance of individualItalianuniversi2es in the twona2onalevalua2onexercisesof research quality, which took place in Italy in 2012 and2016.

We find that, contrary to what was expected by cri2cs ofperformancebasedfunding,intheItaliancontexttherehasbeenasignificantconvergenceinthequalityofresearchofinvolvedinsAtuAons.

12


Conclusionsandfuturework(2)

Our results suggest that this convergence was induced by themonetary and reputaAon incenAves. Apparently, the fact thatworse performing ins2tu2ons received less money did nothampertheireffortstocatchupbeLerperformingones.

Thesearepreliminaryresultsandshouldbetakenwithcau2on.Theycouldbetheeffectoftherela2velysmallamountoffundsshigedbythescheme;atmost4.5%ofpublicfundsreceivedbythe individual universi2es. Also, the convergence may be aniniAal(notpermanent)effectofthescheme.

Weplanto furtherstudythis issue,examiningthedeterminantsof the convergence and refining our harmoniza2on of the twoexercises.

13


Thanksforyoura1enAon!

14

daniele checchi, irene mazzo1a and sandro momigliano · science, technology and innovaon indicators...

Documents