researchinformaonsystems research+performance ... · researchinformaonsystems...

Research Informa.on Systems

www.avedas.com

Research performance assessment in interdisciplinary research: two case studies for Germany and the UK Bonn, 14/05/2013

Wolfram Schüssler | Mar@n Jägerhorn

2 © AVEDAS AG

Topics for today

• Challenges for repor.ng on different classifica.ons

• Case study: KIT disciplinary repor.ng

• Case study: Research Excellence Framework repor.ng

3 © AVEDAS AG

Challenges for repor@ng on different classifica@ons

• Results from each project may be in different disciplines

• Reports desired out of several perspec.ves: organisa.on, discipline, theme, … and other classifica.ons at the same .me!

• Evalua.on schemes / classifica.ons changing over .me

4 © AVEDAS AG

Example for KIT based on Renewable Energy

• Result: 1 publica.on on public acceptance of nanomaterials for energy storage

• Views: • Org unit: ins.tute of nano-‐toxicology

• Discipline: social science

• Theme: energy storage

•  Evalua@on: according to different classifica.ons (DFG, THE, Wissenscha\srat, …)

5 © AVEDAS AG

CRIS data on the organisa.on: Org structure vs. Thema@c matrix due to merger

Execu.ve Board

disciplinary/hierarchical Thema.c matrix

Rectorate Senate

Supervisory Board

F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11

V4I1

I2

I3

I4

I5

I6

I7

I8

I9

I10

I11

I12

I13

I14

I15

I16

I17

I18

I19

I20

I21

P1P1P1P1P1P1P1P1P1P1P1

V1 V2 V3

Supervisory Board Scien.fic technical Council

University of Karlsruhe

Research Centre Karlsruhe (FZK)

Solu@on at the KIT

6 © AVEDAS AG

CRIS data on the organisa.on: Org structure vs. Thema@c matrix due to merger

Execu.ve Board

disciplinary/hierarchical Thema.c matrix

Rectorate Senate

Supervisory Board

F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11

V4I1

I2

I3

I4

I5

I6

I7

I8

I9

I10

I11

I12

I13

I14

I15

I16

I17

I18

I19

I20

I21

P1P1P1P1P1P1P1P1P1P1P1

V1 V2 V3

Supervisory Board Scien.fic technical Council

University of Karlsruhe

Research Centre Karlsruhe (FZK)

Solu@on at the KIT

Problem: organisa@onal structure supports either diciplinary or thema@c view (or none)

7 © AVEDAS AG

Intelligent mapping from organisa@onal structure

to disciplinary view!

8 © AVEDAS AG

Mixed organiza@onal structure at KIT (university & research centre)

Classifica.on = facul.es directly from

organiza.onal structure

Disciplinary view!

9 © AVEDAS AG




Disciplinary view!

Rest: interfaculta.ve university ins.tutes

& interdisciplinary ins.tutes from the research center

10 © AVEDAS AG




Disciplinary view!

Classifica.on?

Classifica.on?

11 © AVEDAS AG

Institut

Geographie Geowissen schaften Biologie Chemie Physik Mathematik Verfahrens

technik MaschinenbauElektrotechnik

und Informations technik

Bauingenieurwesen Architektur Informatik Wirtschafts

informatikWirtschafts

ingenieur wesenWirtschafts

wissen schaften BWL VWL Zwischen-SUMME Geschichte Germanistik Romanistik Anglistik /

AmerikanistikErziehungs

wissen schaft

Soziologie/ Sozial

wissenschaft

Kommunika tionswiss. / Journalistik

(Uni)

Medien wissen schaften (Uni)

Politik wissenschaft /

Sozialwiss.Jura

Sport/ Sportwissen

schaftPsychologie Pflegewissensc

haft Human medizin Zahnmedizin Pharmazie SUMME

IAI Angewandte Informatik 60% 10% 30% 100% 100%IAM-AWP Angewandte Materialien -

Angewandte Werkstoffphysik 10% 20% 70% 100% 100%

IAM-WBM Angewandte Materialien - Werkstoff- und Biomechanik 100% 100% 100%

IAM-WPT Angewandte Materialien - Werkstoffprozesstechnik 20% 40% 20% 20% 100% 100%

IBG-I Biologische Grenzflächen 50% 50% 100% 100%IBG-II Biologische Grenzflächen 30% 40% 30% 100% 100%IFG Funktionelle Grenzflächen 25% 35% 5% 15% 20% 100% 100%IFP Festkörperphysik 100% 100% 100%IHM Hochleistungsimpuls- u.

Mikrowellentechnik 10% 90% 100% 100%IK Kernphysik 100% 100% 100%IKFT Katalyseforschung u. -

technologie 80% 20% 100% 100%

IMK Meteorologie und Klimaforschung 15% 20% 60% 5% 100% 100%

IMK-AAF - Atmosphärische Aerosolforschung 25% 75% 100% 100%

IMK-ASF - Atmosphärische Spurenstoffe und Fernerkundung

15% 20% 60% 5% 100% 100%

IMK-IFU - Atmosphärische Umweltforschung

35% 20% 5% 30% 5% 5% 100% 100%

IMK-TRO - Forschungsbereich Troposphäre

20% 80% 100% 100%IMT Mikrostrukturtechnik 20% 10% 20% 10% 20% 20% 100% 100%IMVT Mikroverfahrenstechnik 100% 100% 100%INE Nukleare Entsorgung 35% 35% 10% 10% 10% 100% 100%INR Neutronenphysik und

Reaktortechnik 30% 5% 60% 5% 100% 100%INT Nanotechnologie 50% 50% 100% 100%IPE Prozessdatenverarbeitung und

Elektronik 40% 40% 20% 100% 100%ISS Syncrotronstrahlung 10% 10% 15% 60% 5% 100% 100%ITAS Technikfolgenabschätzung und

Systemanalyse10% 5% 10% 10% 10% 5% 5% 5% 5% 10% 75% 10% 15% 100%

ITC-TAB Technische Chemie - Bereich Thermische Abfallbehandlung

10% 20% 50% 20% 100% 100%ITEP Technische Physik 15% 40% 15% 25% 5% 100% 100%ITEP-TLK Technische Physik - Tritiumlabor 20% 30% 30% 15% 5% 100% 100%ITG Toxikologie und Genetik 100% 100% 100%SCC Steinbuch Centre for Computing 10% 10% 80% 100% 100%FUSION 10% 10% 20% 40% 20% 100% 100%NANOMIKRO 45% 45% 5% 5% 100% 100%NUCLEAR 5% 5% 25% 25% 5% 5% 20% 5% 5% 100% 100%EE 30% 50% 15% 95% 5% 100%ATMO 70% 20% 5% 95% 5% 100%REUN 20% 60% 15% 95% 5% 100%

ITC-WGT

Wasser- u. Geotechnologie jetzt: IFG altes Institut - Daten aus 2005/06 noch vorhanden

30% 20% 20% 30% 100% 100%

ITC-ZTS

Zentralabteilung Technikbedingte Stoffströme mittlerweile in ITAS integriert - Daten aus 05/06 noch vorhanden

10% 15% 15% 25% 15% 10% 90% 10% 100%

HVT-TLK

Hauptabt. Versuchstechnik - Tritiumlabor - alte Daten vorhanden übergegangen zu ITEP(-TLK) geänderte Aufgaben

20% 10% 50% 15% 5% 100% 100%

20% 180% 270% 670% 935% 35% 505% 600% 275% 35% 0% 145% 0% 45% 10% 15% 10% 3750% 0% 0% 0% 0% 0% 10% 0% 0% 15% 0% 0% 0% 0% 0% 0% 0% 3800%

Fächergruppen zuordnung:ZUK MNW MNW MNW MNW MNW MNW INW INW INW INW INW MNW RWS RWS RWS RWS SKW SKW SKW SKW SKW RWS RWS RWS RWS RWS SPO HMA HMA HMA HMATHE SOW NAW BIW NAW NAW NAW IWT IWT IWT IWT KGW IWT SOW SOW SOW SOW KGW KGW KGW KGW SOW SOW SOW SOW SOW SOW GHM SOW GHM GHM GHMDFG NAW NAW LEW NAW NAW NAW INW INW INW INW INW INW GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO LEW / 2xx LEW / 2xx LEW / 2xx

stat. LA MNW MNW MNW MNW MNW MNW INW INW INW INW INW MNW RWS RWS RWS RWS SKW SKW SKW SKW SKW RWS RWS RWS SPO SKW HuM HuM

G e s u n d h e i t & P s y c h o l o g i eN a t u r w i s s e n s c h a f t e n I n g e n i e u r w e s e n W i r t s c h a f t & T e c h n i k S p r a c h e & G e s c h i c h t e G e s e l l s c h a f t, B i l d u n g, R e c h tFac

hbereich

Disciplines

Ins@tutes

Prozentuale Zuordnung Percentage share

Mapping org structure à discipline by expert judgement

12 © AVEDAS AG

Institut









wissen schaft

Soziologie/ Sozial

wissenschaft


(Uni)



Sozialwiss.Jura

Sport/ Sportwissen









technologie 80% 20% 100% 100%




15% 20% 60% 5% 100% 100%


35% 20% 5% 30% 5% 5% 100% 100%





Systemanalyse10% 5% 10% 10% 10% 5% 5% 5% 5% 10% 75% 10% 15% 100%



ITC-WGT


30% 20% 20% 30% 100% 100%

ITC-ZTS


10% 15% 15% 25% 15% 10% 90% 10% 100%

HVT-TLK


20% 10% 50% 15% 5% 100% 100%

20% 180% 270% 670% 935% 35% 505% 600% 275% 35% 0% 145% 0% 45% 10% 15% 10% 3750% 0% 0% 0% 0% 0% 10% 0% 0% 15% 0% 0% 0% 0% 0% 0% 0% 3800%




hbereich

Disciplines

Ins@tutes



Fachberei

ch

N a t u r w i s s e n s c h a f t e n

Institut

Geographie Geowissen schaften Biologie Chemie Physik Mathematik

IAIIAM-AWP 10% 20%IAM-WBMIAM-WPT 20%IBG-I 50%IBG-II 30% 40% 30%IFG 25% 35% 5%IFP 100%IHM

13 © AVEDAS AG

Institut









wissen schaft

Soziologie/ Sozial

wissenschaft


(Uni)



Sozialwiss.Jura

Sport/ Sportwissen









technologie 80% 20% 100% 100%




15% 20% 60% 5% 100% 100%


35% 20% 5% 30% 5% 5% 100% 100%





Systemanalyse10% 5% 10% 10% 10% 5% 5% 5% 5% 10% 75% 10% 15% 100%



ITC-WGT


30% 20% 20% 30% 100% 100%

ITC-ZTS


10% 15% 15% 25% 15% 10% 90% 10% 100%

HVT-TLK


20% 10% 50% 15% 5% 100% 100%

20% 180% 270% 670% 935% 35% 505% 600% 275% 35% 0% 145% 0% 45% 10% 15% 10% 3750% 0% 0% 0% 0% 0% 10% 0% 0% 15% 0% 0% 0% 0% 0% 0% 0% 3800%




hbereich

Disciplines

Ins@tutes



Institut







wissen schaften BWL VWL Geschichte Germanistik Romanistik Anglistik / Amerikanistik

Erziehungs wissen schaft

Soziologie/ Sozial

wissenschaft


(Uni)



Sozialwiss.Jura

Sport/ Sportwissen


haft Human medizin Zahnmedizin Pharmazie

IAI Angewandte Informatik 60% 10% 30%IAM-AWP Angewandte Materialien -

Angewandte Werkstoffphysik 10% 20% 70%

IAM-WPT Angewandte Materialien - Werkstoffprozesstechnik 20% 40% 20% 20%

IBG-I Biologische Grenzflächen 50% 50%IBG-II Biologische Grenzflächen 30% 40% 30%

Fächergruppenzuordnung:ZUK MNW MNW MNW MNW MNW MNW INW INW INW INW INW MNW RWS RWS RWS RWS SKW SKW SKW SKW SKW RWS RWS RWS RWS RWS SPO HMA HMA HMA HMATHE SOW NAW BIW NAW NAW NAW IWT IWT IWT IWT KGW IWT SOW SOW SOW SOW KGW KGW KGW KGW SOW SOW SOW SOW SOW SOW GHM SOW GHM GHM GHMDFG NAW NAW LEW NAW NAW NAW INW INW INW INW INW INW GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO GSO LEW / 2xx LEW / 2xx LEW / 2xx



hbereich

THE ZUK

DFG Stat. LA

14 © AVEDAS AG

Result: Example Disciplinary view on personnel

Natural Sciences share of university ins.tutes and research centre ins.tutes

University Ins@tutes Research Centre Ins@tutes

Sum of scien@sts

762

610

46 104

96 96

199 225

257 228

101

Biology Earth Sciences

Chemistry

Physics

Geography

Mathema@cs

15 © AVEDAS AG


Disciplinary view!



Classifica.on via discipline matrix

Classifica.on via discipline matrix

16 © AVEDAS AG

Ini.al KIT mapping fits quite well to REF, but different aggrega.ons!

Discipline Mapping KIT-‐to-‐REF & REF-‐to-‐KIT

17 © AVEDAS AG

Ini.al KIT mapping fits quite well to REF, but different aggrega.ons!


20 © AVEDAS AG

Objec@ves of the CONVERIS REF Module

Focus on the Output Quality dimension: • Facilitate for the ins.tu.on to iden.fy and select:

• the (usually) 4 outputs per researcher to submit to the REF • the staff to submit • …and assigned to the most suitable Unit of Assessment

• Enable peer-‐review (and journal) ra.ngs per output

• Produce REF reports with an indica.ve Quality Profile to guide the selec.on process

• Export the data in HEFCE-‐specified Excel template format

21 © AVEDAS AG

CONVERIS REF Management module data model

ORGANISATION

UNIT OF ASSESSMENT

CARD PERSON

PUBLICATION (1.1.2008-‐31.12.2013)

JOURNAL

PEER-‐REVIEW

CITATION

Output quality

REF CLAIM

Roles §  Researcher

§  Peer-‐reviewer

§  Unit Lead

§  REF Manager HEFCE REF Excel import template

CONVERIS REF repor@ng

22 © AVEDAS AG

CONVERIS REF Management module REF Report REF Repor@ng delivered by the new CONVERIS

Analy@cs repor@ng tool

The REF report can be delivered in PDF, Word and Excel and more for easy circula.on and review

23 © AVEDAS AG

CONVERIS REF Management module REF export

REF export func@onality

• Output submission data is exported in HEFCE-‐prescribed Excel import template format, enabling easy review prior to import into the HEFCE submission system

Staff template (REF1a)

Output template (REF2)

25 © AVEDAS AG

Conclusions: using a CRIS to support performance assessments

A CRIS ensures: • A bener data quality (e.g. through reuse of data from internal and external sources, automa.c deduplica.on of outputs, valida.on workflows, …)

• More efficient workflows with electronic approvals (peer-‐reviews, selec.on, evalua.on, expor.ng and submiqng)

• A generic approach to support performance assessments, not only the REF, for common elements like ra.ngs (peer-‐reviews, journal ra.ngs, bibliometrics, ranking, repor.ng and selec.on)

researchinformaonsystems research+performance ... · researchinformaonsystems...

Documents