joint research centre - europa · 2013 •september: the global slavery index 2013 •october: the...
TRANSCRIPT
![Page 1: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/1.jpg)
The European Commission’s science and knowledge service
Joint Research Centre
![Page 2: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/2.jpg)
Ten Steps Guide for Indices & Scoreboards
Michaela Saisana
COIN 2018 - 16th JRC Annual Training on Composite Indicators & Scoreboards 05-07/11/2018, Ispra (IT)
![Page 3: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/3.jpg)
3 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• 3 dimensions & 12 areas • 14 headline indicators • 21 secondary indicators • 93 indicators in total (gender/age)
https://composite-indicators.jrc.ec.europa.eu/social-scoreboard/
Example of a scoreboard Social Pillar Scoreboard for the European Pillar of Social Rights
![Page 4: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/4.jpg)
4 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• 3 dimensions & 12 areas • 14 headline indicators • 21 secondary indicators • 93 indicators in total (gender/age)
Example of a scoreboard Social Pillar Scoreboard for the European Pillar of Social Rights
![Page 5: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/5.jpg)
5 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• 1 index • 2 sub-indices • 7 pillars • 21 sub-pillars • 80+ indicators
Example of a composite indicator Global Innovation Index
![Page 6: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/6.jpg)
6 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• 1 index • 2 sub-indices • 7 pillars • 21 sub-pillars • 80+ indicators
Example of a composite indicator Global Innovation Index
![Page 7: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/7.jpg)
7 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• 1 index • 2 sub-indices • 7 pillars • 21 sub-pillars • 80+ indicators
Example of a composite indicator Global Innovation Index
![Page 8: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/8.jpg)
8 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Why are they needed?
Globalization + Complexity + Data Revolution
Who needs them?
International organizations, European Commission, governments, public
What are they?
com’-po-site: made of various parts or elements
in’-di-ca-tor: a device providing information on the state or condition
Need for indices and scoreboards
![Page 9: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/9.jpg)
9 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
1987 First CI proposed by the EC Proposal for a Council Regulation establishing a Community system of aids to agricultural income [COM(87) 166 final] 1985 Second CI developed by a government German gov. to select areas eligible for investment aid in the North Rhine-Westphalia [85/12/EEC] 1982 First CI developed by a government Danish gov. to select areas eligible for regional aid [82/691/EEC] 1973 First reference within an EC document but … 14th rep. on the activities of the Monetary Committee [OJ 9.11.73 No C94]
Uptake of indices in the EC policies
![Page 10: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/10.jpg)
10 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Search for «composite indicator» OR «synthetic indicator» OR «aggregate indicator»
118 documents (most of them by the EC)
Uptake of indices in the EC policies
![Page 11: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/11.jpg)
11 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Last night: 14000
65 381
2630
5760
12400
17300
1950 1960 1970 1980 1990 2000 2010 2020
Directional radio receiver
Uptake of indices in the academia
More than ten-fold increase since 2000!
![Page 12: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/12.jpg)
12 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Uptake of indices in the media
![Page 13: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/13.jpg)
13 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Enthusiastic supporters, mostly
from advocacy groups developing their own indices to advance a
cause
Skeptical economists and official statisticians
concerned by the subjective nature of the selection of variables, weights and
aggregation
on the other skeptical economists and
official statisticians concerned by the
subjective nature of the variables’ and
weights’ selection and aggregation
procedure.
Polarized audience
![Page 14: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/14.jpg)
14 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Enthusiastic supporters, mostly
from advocacy groups developing their own indices to advance a
cause
Skeptical economists and official statisticians
concerned by the subjective nature of the selection of variables, weights and
aggregation
on the other skeptical economists and
official statisticians concerned by the
subjective nature of the variables’ and
weights’ selection and aggregation
procedure.
Polarized audience
![Page 15: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/15.jpg)
15 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Indicators are powerful advocacy tools
2010
• A. Forrest decided to take up the cause of combatting human trafficking he established the Walk Free Foundation
• Bill Gates gave him some advice: “use a quantifiable metric. […] if you can’t measure it, it doesn’t exist.”
2012
• A. Forrest got Richard Branson on board, and by Dec. 2012, they had appealed to 25 large companies and governments to ban the use of forced labor
2013
• One year later, one activity got more attention than all of Walk Free’s efforts combined…
Andrew Forrest
![Page 16: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/16.jpg)
16 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Indicators are powerful advocacy tools
2013 • September: The Global Slavery Index
2013
• October: The index was covered in >100 newspaper stories around the world
2014
• July: In a global survey of NGOs that work on trafficking issues, over 40% had already heard of this new index
Andrew Forrest
![Page 17: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/17.jpg)
17 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Enthusiastic supporters, mostly
from advocacy groups developing their own indices to advance a
cause
Skeptical economists and official statisticians
concerned by the subjective nature of the selection of variables, weights and
aggregation
Polarized audience
![Page 18: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/18.jpg)
18 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Conceptual Framework – World Health Organization
Unlike the Alcohol Policy Index, most composite indicators cannot be validated versus a ground truth [Alcohol Policy Index, 2007, PLoS Medicine, 4(4):752-759]
«Measurements without theory…»
![Page 19: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/19.jpg)
19 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
…and on top of that indices have …
strong Political and Policy Implications
The Malaysian Industrial Development Authority insisted that Malaysia aims to
move from the 24th to top 10 on in the World Bank's `Doing Business' ranking list. “We continue to ask ourselves what it will take to reach the top 10, and are we willing to do what it takes to get there." [Asia in Focus, Jan. 8 2007]
The Minister of the Economic Development in Kyrgyzstan, expressed a hope (in 2008) that his country shall rank among top 20 countries in the Doing Business rating in three years.
[The WB Doing Business Report has long been credited with bringing about reforms in countries – as many as 2000 distinct reforms since its 2003 launch (Source: The Economist 2013)]
![Page 20: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/20.jpg)
20 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
This is WHY composite indicators should be …
developed sensibly and used responsibly ‘Sensible development’ of a composite indicator implies a quality control process based on both
conceptual and statistical considerations.
‘Responsible use’ calls for care in drawing conclusions and recommendations without taking into
account the conceptual context in which composite indicators were developed.
Indices are powerful advocacy tools
![Page 21: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/21.jpg)
21 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Mean Median
[…Refreshing key statistical terms…]
![Page 22: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/22.jpg)
22 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Variance
[…Refreshing key statistical terms…]
![Page 23: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/23.jpg)
23 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Percentiles
[…Refreshing key statistical terms…]
![Page 24: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/24.jpg)
24 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
[…Refreshing key statistical terms…]
The correlation coefficient only
assesses the strength of a linear
relationship between two
variables.
The correlation coefficient does
not assess causation – that one
event causes the other.
![Page 25: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/25.jpg)
25 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
[…Refreshing key statistical terms…]
![Page 26: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/26.jpg)
26 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
[…Refreshing key statistical terms…]
![Page 27: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/27.jpg)
27 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
[…Refreshing key statistical terms…]
![Page 28: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/28.jpg)
28 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
[…Refreshing key statistical terms…]
KPI-1 with KPI-2
r = 0. 768
(high positive
correlation)
![Page 29: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/29.jpg)
29 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
[…Refreshing key statistical terms…]
KPI-1 with KPI-2
r = 0. 213
(negligible
correlation)
![Page 30: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/30.jpg)
30 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Step 10. Presentation & dissemination
Step 9. Association with other variables
Step 8. Back to the indicators
Step 7. Robustness & sensitivity
Step 6. Weighting & aggregation
Step 5. Normalisation of data
Step 4. Multivariate analysis
Step 3. Data treatment
Step 2. Selection of indicators
Step 1. Developing the framework
Finally endorsed (after 2 rounds of consultation) by the OECD high level statistical committee in March 2008
Ten Steps Guide for Composite Indicators
![Page 31: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/31.jpg)
31 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Step 10. Visualisation & Communication
Step 9. Back to the data
Step 8. Robustness & Sensitivity
Step 7. Statistical coherence
Step 6. Aggregation
Step 5. Weighting
Step 4. Normalisation
Step 3. Data treatment
Step 2. Selection of indicators
Step 1. Developing the framework
New version to be released 2019-2020 10+ year anniversary
Ten Steps Guide for Composite Indicators & Scoreboards
Next version
![Page 32: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/32.jpg)
32 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Definition of the phenomenon
Added-value of index/scoreboard
Nested structure of the framework
Involve experts and stakeholders
Step 1 Theoretical/Conceptual framework
![Page 33: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/33.jpg)
33 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Selection criteria for indicators input, output, outcome, process
salience, credibility, legitimacy
data coverage
consider using proxy variables when official statistics are scarce
Summary statistics Source & data availability (countries, time)
type (hard, soft or input, output, process),
descriptive statistics (mean, median, skewness, kurtosis, min, max, variance, histogram)
Step 2
Data selection
![Page 34: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/34.jpg)
34 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
…
Discuss strengths and limitations of each selected indicator
Involve experts and stakeholders
Step 2
Data selection
![Page 35: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/35.jpg)
35 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Make scale adjustments if necessary (e.g. divide by population, GDP, other)
Check for missing data and outliers
Treat outliers, if needed (so as to avoid that they become unintended benchmarks)
Estimate missing data, if appropriate (and estimate confidence interval for each imputed value to assess the impact of
imputation on the results)
Step 3
Data treatment
![Page 36: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/36.jpg)
36 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Make directional adjustments (so that higher scores correspond to better performance in all indicators or vice versa)
Select a suitable normalisation method that respects the conceptual framework and the data properties
Step 4
Normalisation
Does a large group of foreign-born population decrease social cohesion? (yes/no direction)
5 seconds rule!
![Page 37: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/37.jpg)
37 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Make directional adjustments (so that higher scores correspond to better performance in all indicators or vice versa)
Select a suitable normalisation method that respects the conceptual framework and the data properties
Step 4
Normalisation
ranking z-score Min-max
Distance to reference country
Categorical scales
Adjust for measurement unit
Y Y Y Y Y
Adjust for variance
Y Y N N N
Adjust for range
Y N Y N Y
Adjust for extreme values
Y N N N Y
Y/N
0.0
0.2
0.4
0.6
0.8
1.0
1.2
7.0 8.0 9.0 10.0 11.0 12.0 13.0
Raw indicator: Mean years of schooling (years)
No
rmali
sed
: m
in-m
ax
0
5
10
15
20
25
7.0 8.0 9.0 10.0 11.0 12.0 13.0
Raw indicator: Mean years of schooling (years)
No
rmali
sed
: R
an
k
![Page 38: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/38.jpg)
38 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Discuss if correlation among indicators should be accounted for in the weights If yes, then how (more correlated indicators more
or less weight?)
Select a suitable weighting method that respects the conceptual framework and the data properties
Step 5
Weighting
X2
X1 0
![Page 39: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/39.jpg)
39 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Discuss if compensability among indicators should be allowed (fully, partially, not at all)
Discuss up to which level to aggregate
Select a suitable aggregation method that respects the conceptual framework and the data properties
Step 6
Aggregation
Question 8+2 < = > 5+5
A B C D E
A 0 0.2 0.4 0.2 0.2
B 0.8 0 0.8 1 0.2
C 0.6 0.2 0 0.4 0.4
D 0.8 0 0.6 0 0.2
E 0.8 0.8 0.6 0.8 0
Outranking matrix
![Page 40: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/40.jpg)
40 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
17.1) During the last 12 months, for how many months was your household’s main source
of water sufficient to meet your household’s drinking, cooking, bathing and cleaning
needs?
Months: Don’t remember (-1)
17.2) How often do you worry there will not be enough water from your household’s main
water source to satisfy your household’s drinking, cooking, bathing and cleaning needs?
Never (1) Rarely (2) Sometimes (3) Often (4) Always (5)
Example: Multidimensional Poverty Assessment Tool
Component: Domestic Water Supply, Subcomponent: Availability
104 HHs
8
3 HHs
4
Too worried? Careless?
Suggestion: With survey data, given some unavoidable inconsistencies (in part due to the way the human mind works), use a (weighted) arithmetic average (rule of thumb: 5-10 indicators) within a subcomponent.
![Page 41: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/41.jpg)
41 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Assess if few indicators dominate the framework
(rindicator,index>0.95)
Assess if indicators behave as “noise” in the framework
(-0.3<rindicator,index<0.3)
Assess if indicators are negatively related to the index
(rindicator,index<-0.3)
Step 7
Statistical coherence
Pearson Correlations
Ecosystem Vitality
Environmental Health
EPI 2010 0.29 0.90
Environmental Health
-0.08*
Variance 117 615
![Page 42: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/42.jpg)
42 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
…
Assess if indicators statistically fit better under different dimensions than those in the framework
Assess if dimensions should be merged or split
Assess if bias has been introduced in the index (e.g., due to population size, population density, GDP)
Step 7
Statistical coherence
![Page 43: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/43.jpg)
43 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Identify the sources of uncertainty in the index development
Assess the impact of the uncertainties to the index scores or ranks complement scores/ranks with confidence
intervals
Step 8
Robustness & Sensitivity Including/
excluding variables Normalisation
Missing data Weights
Aggregation
Country 1
10
20
30
40
50
60
Country 2 Country 3
![Page 44: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/44.jpg)
44 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Identify the sources of uncertainty in the index development
Assess the impact of the uncertainties to the index scores or ranks complement scores/ranks with confidence
intervals
Step 8
Robustness & Sensitivity
![Page 45: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/45.jpg)
45 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
…
Identify which uncertainties are more crucial in determining the final classification
• Explain why certain countries notably improve or deteriorate their relative position given the assumptions
Robustness ≠Quality
Step 8
Robustness & Sensitivity
![Page 46: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/46.jpg)
46 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
1
51
101
151
201
251
301
351
401
451
501Med
ian
rank
(and
99%
con
fiden
ce in
terv
al) a
ccou
ntin
g fo
r
met
hodo
logi
cal u
ncer
tain
ties
Seoul National University
University of Frankfurt
University of Hamburg
University of California-Davis
University of Alaska-
Fairbanks
Hanyang University
54 universities outside the interval (total of 503)
[43 universities in the Top 100]
1
51
101
151
201
251
301
351
401
Med
ian
rank
(and
99%
con
fiden
ce in
terv
al) a
ccou
ntin
g fo
r
met
hodo
logi
cal u
ncer
tain
ties
250 universities outside the interval (total of 400)
[61 universities in the Top 100]
University of California, Santa
Barbara
Stockholm School of Economics
University of st.
Gallen
University of Tokyo
University of
LeichesterUniversity La Sapienza,
Roma
ARWU 2008 ranking (more robust, higher correlations)
THES 2008 ranking
(less robust, lower correlations)
Robustness ≠Quality
![Page 47: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/47.jpg)
47 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Decompose performance at the indicator level (to reveal strengths and limitations for each country)
Correlate the index with relevant measurable phenomena and explain similarities or differences
Develop evidence based narratives
Perform causality tests (if time series data or microdata are available)
Step 9
Look back into the data & vis-à-vis other measures
![Page 48: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/48.jpg)
48 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Decompose performance at the indicator level (to reveal strengths and limitations for each country)
Correlate the index with relevant measurable phenomena and explain similarities or differences
Develop data-driven narratives on the results.
Perform causality tests (if time series data or microdata are available)
Step 9
Look back into the data & vis-à-vis other measures
![Page 49: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/49.jpg)
49 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Identify suitable visualisation tools for the targeted audience
Select the visualisation technique which communicates the most information without hiding vital information
Make your index/scoreboard EAST Easy, Attractive, Social and Timely
Step 10
Visualisation & Communication
![Page 50: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/50.jpg)
50 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Sound framework
Carefully selected indicators
Sound model
can help to depict reasonably reality
Prerequisites for any index
![Page 51: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/51.jpg)
51 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Sound framework
Carefully selected indicators
Sound model
can only offer an imperfect mirror of reality
Prerequisites for any index
![Page 52: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/52.jpg)
52 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
Powerful evidence based narratives supported by good statistical measures and good analytic work are a possibility which should not be left untried
We need relevant and sound…
(Composite) Indicators
![Page 53: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/53.jpg)
53 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• https://ec.europa.eu/jrc/en/coin
• https://composite-indicators.jrc.ec.europa.eu/
References and related reading
![Page 54: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/54.jpg)
54 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• Becker, W., M. Saisana, P. Paruolo, and I. Vandecasteele. 2017. ‘Weights and Importance in Composite
Indicators: Closing the Gap. Ecological Indicators 80: 12–22.
• Cohen, A., Saisana, M., 2014, Quantifying the Qualitative: Eliciting Expert Input to Develop the Multidimensional
Poverty Assessment Tool, Journal of Development Studies 50(1): 35-50.
• OECD/JRC, 2008, Handbook on Constructing Composite Indicators. Methodology and user Guide, OECD
Publishing, ISBN 978-92-64-04345-9.
• Paruolo P., Saisana M., Saltelli A., 2013, Ratings and Rankings: voodoo or science?. J Royal Statistical Society A
176(3), 609-634.
• Saisana, M., and Saltelli, A., 2011, Rankings and Ratings: Instructions for use, Hague Journal on the Rule of Law
3(2), 247-268.
• Saisana M., D’Hombres B., Saltelli A., 2011, Rickety Numbers: Volatility of university rankings and policy
implications. Research Policy 40, 165–177.
• Saisana, M., Saltelli, A., 2014, JRC Statistical Audit of the WJP Rule of Law Index® 2014 (p.188-198). In the
World Justice Project Rule of Law Index® Washington, D.C.: The World Justice Project.
References and related reading
![Page 55: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/55.jpg)
55 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards
• Saisana, M., Weziak-Bialowolska, D., 2013, JRC Statistical Audit on the Environment and Gender Index (p.143-
153), in IUCN The Environment and Gender Index (EGI) 2013 Pilot. Washington, D.C.: IUCN.
• Saisana, M., Saltelli A., 2012, Corruption Perceptions Index 2012. Statistical Assessment, EUR 25623, European
Commission, JRC-IPSC, Italy.
• Saisana M., 2010, ELLI-Index: a sound measure for lifelong learning in the EU, EUR 24529, European
Commission, JRC-IPSC, Italy.
• Saisana M., Saltelli A., 2010, The Multidimensional Poverty Assessment Tool (MPAT): Robustness issues and
Critical assessment, EUR 24310, European Commission, JRC-IPSC, Italy.
• Saisana M., 2008, The 2007 Composite Learning Index: Robustness Issues and Critical Assessment, EUR
23274, European Commission, JRC-IPSC, Italy.
• Saisana M., D’Hombres B., 2008, Higher Education Rankings: Robustness Issues and Critical Assessment, EUR
23487, European Commission, JRC-IPSC, Italy.
References and related reading
![Page 56: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/56.jpg)
Welcome to email us at: [email protected]
THANK YOU
COIN in the EU Science Hub https://ec.europa.eu/jrc/en/coin
COIN tools are available at: https://composite-indicators.jrc.ec.europa.eu/
The European Commission’s
Competence Centre on Composite
Indicators and Scoreboards
![Page 57: Joint Research Centre - Europa · 2013 •September: The Global Slavery Index 2013 •October: The index was covered in >100 newspaper stories around the world 2014 •July: In a](https://reader033.vdocuments.net/reader033/viewer/2022042307/5ed2f43e3e92a05e0d15d691/html5/thumbnails/57.jpg)
Questionnaire – Post Day 1:
https://goo.gl/qy19yD