towards indicators for ‘opening up’ science and technology policy ismael rafols, tommaso ciarli,...

Towards indicators for ‘opening up’ science and technology policy

Ismael Rafols, Tommaso Ciarli,

Patrick van Zwanenberg and Andy Stirling

Ingenio (CSIC-UPV), Universitat Politècnica de València &

SPRU —Science and Technology Policy Research, University of Sussex

Stockholm, October 2013

Building on work with Loet Leydesdorff and Alan Porter

Paper born out of the reflection on the contrast between interdisciplinary research and journal rankings

Interdisciplinary maps

versus rankings

On the role of scientific advice in policy(scientometric is the science of science –hence scientific advice)

The linearity-autonomy model of scientific advice (Jasanoff, 2011) Scientific knowledge is the best possible foundation for public

decisions Scientists should establish the facts that matter independently.

– S&T indicators produce evidence of these facts.

However, this (enlightenment) model has been challenged The mechanisms to establish facts and make decisions is a social

process – “knowledge enables power, but power structures knowledge” (Stirling, 2012)

Modes of advice: The pure scientist vs. honest broker (Pielke, 2007)

What is (should be) the role of STI indicators in policy advice? Closing down vs. Opening up

The challengeProblems with current use of S&T indicators

Use of conventional S&T indicators is en *problematic*

(as many technologies, in particular those closely associated with power, e.g. nuclear)

Narrow inputs (only pubs!) Scalar outputs (rankings!) Aggregated solutions --missing variation Opaque selections and classifications (privately owned databases) Large, leading scientometric groups embedded in government /

consultancy, with limited possibility of public scrutiny Sometimes even mathematically debatable

Impact Factor of journals (only 2 years, ambiguity in document types) Average number of citations (pubs) in skewed distributions

From S&T indicators for justification and disciplining…

Justification in decision-making• Weak justification, “Give me a number, any number!”• Strong justification, “Show in numberrs that X is the best choice!”

S&T Indicators have a performative role: They don’t just measure. Not ‘just happen to be used’ in science

policy (neutral) Constitutive part incentive structure for “disciplining” (loaded) They signal to stakeholders what is important.

Institutions use these techniques to discipline subjects Articulate framings, goals and narratives on performance,

collaboration, interdisciplinarity…

… towards S&T indicators as tools for deliberation

Yet is possible to design indicators that foster plural reflection rather than justifying or reinforcing dominant perspectives

This shift is facilitated by trends pushed by ICT and visualisation tools

More inputs (pubs, pats, but also news, webs, etc.) Multidimensional outputs (interactive maps) Multiple solutions -- highlighting variation, confidence intervals More inclusive and contrasting classifications (by-passing

private data ownership? Pubmed, Arxiv) More possibilities for open scrutiny (new research groups)

1. Conceptual framework:

“broadening out” vs. “opening up” policy appraisal

Policy use of S&T indicators: Appraisal

Appraisal:

‘the ensemble of processes through which knowledges are gathered and produced in order to inform decision-making and wider institutional commitments’ Leach et al. (2008)

Breadth: extent to which appraisal covers diverse dimensions of knowledge

Openness: degree to which outputs provide an array of options for policies.

Policy use of S&T indicators: Appraisal

Appraisal:

‘the ensemble of processes through which knowledges are gathered and produced in order to inform decision-making and wider institutional commitments’ Leach et al. (2010)

Example: Allocation of resources based on research “excellence”

Breadth: extent to which appraisal covers diverse dimensions of knowledgeNarrow: citations/paper

Broad: citations, peer interview, stakeholder view, media coverage, altmetrics

Openness: degree to which outputs provide an array of options for policies. Closed: fixed composite measure of variables unitary and prescriptive

Open: consideration of various dimensions plural and conditional

narrow

broad

closing-down opening-up

range of appraisals inputs(issues, perspectives, scenarios, methods)

effect of appraisal ‘outputs’ on decision-making

Leach et al. 2010

Appraisal methods: broad vs. narrow & closing vs. opening

narrow

broad




Appraisal methods: broad vs. narrow & close vs. open

cost-benefit analysis

open hearings

consensusconference

scenarioworkshops

citizens’ juries

multi-criteria mapping

q-method

sensitivityanalysis

narrative-based participant observation

decision analysis

risk assessment structured interviews

Stirling et al. (2007)

narrow

broad





Most conventionalS&T indicators??

narrow

broad




Broadening out S&T Indicators

ConventionalS&T indicators??

Broadening out

Incorporation plural analytical dimensions:

global & local networkshybrid lexical-actor netsetc.

New analytical inputs: media, blogsphere.

narrow

broad





Journal rankings

University rankings Unitary measuresthat are opaque, tendency to favour the established perspectives

… and easily translated into prescription

European InnovationScoreboard

narrow

broad




Opening up S&T Indicators

ConventionalS&T Indicators??

opening-up

Making explicit underlying conceptualisations and creating heuristic tools to facilitate exploration

NOT about the uniquely best methodOr about the unitary best explanationOr the single best prediction

2. Examples of Opening Up

a. Broadening out AND Opening up

b. Opening up WITH NARROW inputs

narrow

broad




1. Preserving multiple dimensions in broad appraisals

ConventionalS&T indicators??

Leach et al. 2010

Broadening out opening-up

Composite Innovation Indicators (25-30 indicators)

European (Union) Innovation Scoreboard

Grupp and Schubert (2010) show that order is highly dependent on indicators weightings. Sensitivity analysis

Solution: representing multiple dimensions(critique by Grupp and Schubert, 2010)

Use of spider diagramsallows comparing like with like

U-rank, University performance Comparison tools(Univ. Twente)

5.4 Community trademarks indicator

U-Map: Comparison of Universities in Multiple Dimensions

2. Examples of Opening Up

b. Opening up WITH NARROW inputs

narrow

broad




Opening up S&T Indicators

ConventionalS&T Indicators??

Leach et al. 2010

opening-up

Making explicit underlying conceptualisations and creating heuristic tools to facilitate exploration

NOT about the uniquely best methodOr about the unitary best explanationOr the single best prediction

Manchester Inn Inst

WarwickBusinessSchool

Disciplinary Diversity of Publications Variety 19 20Shannon Entropy 2.966 3.078

Disciplinary Diversity of References Variety 17 20Shannon Entropy 3.378 3.153

Disciplinary diversity of Citations Variety 22 24Shannon Entropy 3.415 3.503

Interdisciplinarity as diversity

Rafols, Porter and Leydesdorff (2010)

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer SciBusiness & MGT

Geosciences

Ecol Sci

Econ Polit. & Geography

Health & Social Issues

A Global Map of Science222 SCI-SSCI Subject Categories

Warwick Business SchoolSubject Categories of publications

Nodes labelled if >0.5% publications

Manchester MIoIRSubject Categories of publications

Nodes labelled if >0.5% publications

Heuristics of diversity

(Stirling, 1998; 2007)

Diversity:‘attribute of a system whose elements may be apportioned into categories’

Characteristics: Variety: Number of distinctive categoriesBalance: Evenness of the distribution Disparity: Degree to which the categories

are different.

Variety

Balance Disparity

Herfindahl (concentration): i pi2

Shannon (Entropy): i pi ln pi

Dissimilarity: i di

Generalised Diversity (Stirling) ij(ij) (pipj)a (dij)

b

Manchester Innov Inst

WarwickBusiness S

Diversity of Publications Variety 19 20Balance 0.543 0.460Disparity 0.817 0.770Shannon Entropy 2.966 3.078Rao-Stirling Diversity 0.726 0.680Diversity of References Variety 17 20Balance 0.415 0.325Disparity 0.846 0.780Shannon Entropy 3.378 3.153Rao-Stirling Diversity 0.729 0.689Diversity of Citations Variety 22 24Balance 0.505 0.454Disparity 0.836 0.801Shannon Entropy 3.415 3.503Rao-Stirling Diversity 0.771 0.736

Comparing degree of interdisciplinarity of two university units: Manchester is more??

Multiple concepts of interdisciplinarity:

Conspicuous lack of consensus but most indicators aim to capture the following concepts

Integration (diversity & coherence)• Research that draws on

diverse bodies of knowledge • Research that links different

disciplines

Intermediation• Research that lies between or

outside the dominant disciplines

Coherence

Low High

Diversity

Low

High

InterdisciplinaryMultidisciplinary

Monodisciplinary

Intermediation

Low High

Monodisciplinary Interdisciplinary

Diversity

ISSTI Edinburgh WoS Cats of references

Assessing interdisciplinarity

ISSTI EdinburghObserved/ExpectedCross-citations

CoherenceAssessing interdisciplinarity

RiskAnal

PsycholBull

PhilosTRSocA

Organization

JPersSocPsychol

JLawEconOrgan

JIntEcon

Interfaces

EnvironSciPolicy

CanJEcon

ApplEcon

AnnuRevPsychol

RandJEcon

JPublicEcon

JManage

JLawEcon

HumRelat

BiomassBioenerg

AtmosEnviron

PolicySci

JIntBusStud

JApplPsychol

Econometrica

PublicUnderstSci

PsycholRev

JFinancEcon

JApplEcolJAgrarChangeClimaticChange

AcadManageJ

JRiskRes

JDevStud

Scientometrics

HarvardBusRev

IntJMedInform

GlobalEnvironChang

EconJ

JFinanc

StudHistPhilosSci

DrugInfJ

Futures

WorldDev

StrategicManageJ

SciTechnolHumVal

EconSoc

PublicAdmin

Lancet

IndCorpChange

AccountOrgSoc

EnergPolicy

Nature

AmJSociol

ResPolicy

TechnolAnalStrateg SocStudSciBritMedJ

ISSTI EdinburghReferences

IntermediationAssessing interdisciplinarity

Summary: IS (blue) units are more interdisciplinary than BMS (orange)

More DiverseRao-Stirling Diversity

More CoherentObserved/Expected

Cross-Citation Distance

More InterstitialAverage Similarity

0.02

0.0300000000000001

0.0400000000000001

0.0500000000000001

0.0600000000000001

0.0700000000000002

2. Excellence: Opening Up Perspectives

Provide different perspectives of performance(alternative measures of the same type of indicator)

Are measures of “excellence” consistent and robust?

Good

Average

Bad

Van Eck, Waltman et al. (2013)

More basic

More applied

Clinical neurologyIs basic always better than applied?

Citations: not stable to changes in classification and granularity (Zitt et al., 2005; Adams et al., 2008).

Measures of “excellence”

ISSTI SPRU MIoIR Imperial WBS LBS0

0.5

1

1.5

2

2.5

3

3.5

4A

BS R

ank


1

2

3

4

Cita

tions

/pub

Jo

urna

l-fiel

d N

orm

alis

ed


0.05

0.1

0.15

0.2

Cita

tions

/pub

Citi

ng-p

aper

Nor

mal

ised

Which one is more meaningful??


1

2

3

4

Jour

nal I

mpa

ct F

acto

r

The new Leiden ranking (2011-12) • Different measures of performance

• MNC, MNCS, MNCJ, Top 10%, • Under different conditions (fractional, language)• Include confidence interval (bootstrapping)

3. Summary and conclusions

S&T indicator as a tools to open up the debate

• ‘Conventional’ use of indicators (‘Pure scientist ‘--Pielke) Purely analytical character (i.e. free of normative assumptions) Instruments of objectification of dominant perspectives Aimed at legitimising /justifying decisions (e.g. excellence) Unitary and prescriptive advice

• Opening up scientometrics (‘Honest broker’ --Pielke) Aimed at locating the actors in their context and dynamics

Not predictive, or explanatory, but exploratory Construction of indicators is based on choice of perspectives

Make explicit the possible choices on what matters Supporting debate

Making science policy more ‘socially robust’ Plural and conditional advice

Barré (2001, 2004, 2010), Stirling (2008)

Strategies for opening up or how to “keep it complex” yet “manageable”

• Presenting contrasting perspectives At least TWO, in order to give a taste of choice

• Simultaneous visualisation of multiple properties / dimensions Allowing the user take its own perspective

• Interactivity Allowing the user give its own weigh to criteria / factors Allowing the user manipulate visuals

.

Is ‘opening up’ worth the effort? (1)

Sustaining diversity in S&T system

Decrease in diversity.

Potential unintended consequence of the evaluation machine:

Why diversity matters

Systemic (‘ecological’) understanding of the S&T S&T outcomes depend on synergistic interactions between

disparate elements.

Dynamic understanding of excellence and relevance New social needs, challenges, expectations from S&T

Manage diverse portfolios to hedge against uncertainty in research Office of Portfolio Analysis (National Institutes of Health)

http://dpcpsi.nih.gov/opa/

Open possibility for S&T to work for the disenfranchised Topics outside dominant science (e.g. neglected diseases)

http://dpcpsi.nih.gov/opa/

Is ‘opening up’ worth the effort? (2)

Building robustness against bias

Do conventional indicators tend to favour incumbents?

Hypothesis:

Elites and incumbents (directly or not) influence choice of indicators, which tend to benefit them…

“knowledge enables power, but power structures knowledge” (Stirling, 2012)

Crown indicator –Standard measure of performance (~1990-2010)– ‘systematic underrating of low-ranked scientists’ (Opthof and

Leydesdorff, 2010) (Not spotted for 15 years!) Journal rankings in Business and Management.

– systematic underrating of interdisciplinary (heterodox) depts. (Rafols et al., 2012).

Others?? H-index??– favours established academics over younger.

‘lock-in’ to policy favoured by incumbent

power structures

multiple practices, and processes, for informing social agency (emergent and unstructured as well as deliberately designed )

complex, dynamic, inter-coupled and mutually-

reinforcing socio-technical configurations

in science

narrow scope of attention

Conventional Policy Dynamics

SOCIAL APPRAISAL

GOVERNANCE COMMITMENTS

simple ‘unitary’ prescriptions

POSSIBLE FUTURES

expert judgements /

‘evidence base’

“best / optimal /legitimate”

S&T indicators

risk assessment

cost-benefit analysis

also: restricted options, knowledges, uncertainties in participation

incomplete knowledges

Res. Excellence

$IIIIII

GUIDANCE / NARRATIVE

Stirling (2010)

POSSIBLE PATHWAYSMULTIPLE

TRAJECTORIES

SOCIAL APPRAISAL

GOVERNANCE COMMITMENTS

broad-based processes of ‘precautionary appraisal’

‘opening up’ with ‘plural conditional’

outputs to policymaking

dynamic portfolios pursuing diverse

trajectories

viable options under: conditions, dissonant views,

sensitivities, scenarios, maps, equilibria, pathways, discourses

multiple: methods, criteria, options, frames, uncertainties, contexts, properties, perspectives

Breadth, Plurality and Diversity

Sustainability

$

Stirling (2010)

S&T indicator as a tools to open up the debate

• ‘conventional’ use of indicators Instruments of objectification Analytical character (i.e. free of normative assumptions) Aimed at making decisions (e.g. excellence) Unitary and prescritive advice

• Opening up scientometrics Construction of indicators is based on choice of perspectives

implicit normative choice on what matters Aimed at locating the actors in their context and dynamics

Not predictive, or explanatory, but exploratory Supporting debate

making science policy more ‘socially robust’ Plural and conditional advice

Barré (2001, 2004, 2010), Stirling (2008)

Heuristics of diversity

(Stirling, 1998; 2007)

Diversity:‘attribute of a system whose elements may be apportioned into categories’

Characteristics: Variety: Number of distinctive categoriesBalance: Evenness of the distribution Disparity: Degree to which the categories

are different.

Variety

Balance Disparity

Herfindahl (concentration): i pi2

Shannon (Entropy): i pi ln pi

Dissimilarity: i di

Generalised Diversity (Stirling) ij(ij) (pipj)a (dij)

b

Rafols, Porter and Leydesdorff (2010)

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer SciBusiness & MGT

Geosciences

Ecol Sci

Econ Polit. & Geography

Health & Social Issues

A Global Map of Science222 SCI-SSCI Subject Categories

• CD-ROM version of the JCR of SCI and SSCI of 2009.• Matrix of cross-citations between journals (9,000 x

9,000)• Collapse to ISI Subject Category matrix (222 x 222)• Create similarity matrix using Salton’s cosine

Diversity indexes

Stirling Generalised Diversity

Diversity indexes

Stirling Generalised Diversity

a=0, b=0

Number of disciplines

Diversity indexes

:Stirling Generalised Diversity

a=0, b=1Simpson(Herfindahl) Index

Diversity indexes

Generalised Stirling Diversity

a=1, b=1quadratic entropy

Different aspects of diversity are uncorrelated

rvar,bal = 0.18, p < .001rvar,dis = 0.32, p < .001

rbal,dis = -0.20, p < .001Yegros et al. (2010)

Which diversity measureshould we choose?

Leiden Ranking: high correlation but important individual differences

mcs_rank mncs_rank mncs_frac_rank mncs_doc_rank mncs_SM_rank mncs_LW_rank

mcs_rank 1.000 0.911 0.815 0.915 0.904 0.937

mncs_rank 0.911 1.000 0.959 0.995 0.998 0.987

mncs_frac_rank 0.815 0.959 1.000 0.953 0.958 0.927

mncs_doc_rank 0.915 0.995 0.953 1.000 0.993 0.983

mncs_SM_rank 0.904 0.998 0.958 0.993 1.000 0.982

mncs_LW_rank 0.937 0.987 0.927 0.983 0.982 1.000

Spearman‘s rank correlation coefficient matrix.

(Thanks to Daniel Sirtes and Ludo Waltman for sharing this data)

towards indicators for ‘opening up’ science and technology policy ismael rafols, tommaso ciarli,...

Documents

science of science

role of sti indicators

policy advice

role of scientific advice

scientific knowledge

dynamic sustainabilities

technology policy research

modes of advice