ec research on composite indicators with special focus on

OutlineOverview to the KEI Project

Missing Values and Multiple ImputationA Case Study on a Composite Indicator

Summary and Outlook

Eberhard Karls

UniversitatTubingen

EC Research on Composite Indicators with SpecialFocus on Missing Values

Ralf Munnich

Department of Statistics, Econometrics, and Operations ResearchUniversity of Tubingen

SECOND WORKSHOP ON COMPOSITE INDICATORS OFCOUNTRY PERFORMANCE

OECD Paris, 26. February 2004

Paris, 26. February 2004 Ralf Munnich EC Research on Composite Indicators



Summary and Outlook

Eberhard Karls

UniversitatTubingen

EC Research on Composite Indicatorswith Special Focus on Missing Values

Outline of the PresentationOverview to the KEI ProjectMissing Values and Multiple ImputationA Case Study on a Composite IndicatorSummary and Outlook




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Knowledge Economy Indicators:Development of Innovative and Reliable Indicator Systems

KEI will focus on indicators and composite indicatorsReview of state-of-the-art methodologyMain thematic areas in relation to Lisbon and BarcelonaobjectivesIdentification of gaps

Development of innovative approachesAppropriate statistical methodologyMulti-criteria methodsAggregation and weighting techniquesElaboration of sensitivity analysisEvaluation of analytical properties of indicatorsInvestigation of adequate presentational techniquesSupport by large-scale simulation studyScenario analysis in co-operation with Commission services

−→ Policy-orientated research (FP6)




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Some Important Tasks of the Project

Five workshops with special focuses

Special topicsInvited and contributed paper presentations

Interest in co-operation with other projects / researchers inthe KEI area

Exchange of information via KEInews and WEB page

Contribution to journals and Commission publication(e.g. Statistics in Focus)

Final reports

Overview to KEI achievementsDetailed workpackage reportsPolicy analysis of knowledge economy indicators




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Structure of the Project

Defining the Knowledge-Based Economy

Indicators for the KBE

Statistical Analysis for KBE Indicators:

Mathematical and statistical properties of indicatorsData quality including missing value analysis

The Way Forward: Innovative Use of KBE Indicators

Composite Indicators for the KBE

Role of Multinationals for Information on R & D

Simulation Study:

Investigation of outcomes via simulation studyScenario analysis




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Partners of KEI

Eberhard Karls University of Tubingen: Ralf Munnich (CO)

Joint Research Center, Ispra: Andrea Saltelli

Katholieke Universiteit Leuven: Tom van Puyenbroeck

University of Maastricht, MERIT: Anthony Arundel

Statistics Finland: Mikael Akerblom




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Some aspects of Tubingen work

7 aspects of data quality (report) from Eurostat:relevance, accuracy, timeliness, accessability and clarity ofinformation, comparability, coherence, and completeness

Elaboration of data sourcesAnalysis based on Eurostat data quality workInvestigation of peculiarities and their influence on indicatorsRecommendations for choice of data sources

Recommended practices of methodology

Tools for indicator values computationOpen Source: R, cf. http://www.r-project.orgMicrosoft Excel c©Mathworks MatLab c©

Missing value analysis and their compensation−→ Imputation methodology


http://www.r-project.org



Summary and Outlook

Eberhard Karls

UniversitatTubingen

Example for Missing Values in IndicatorsGERD PhD FTE GFCF EGov TEE LLL POP POP1

BDKDELEFIRLILNLAPFINSUK

BGCYCZEEHULTLVMTPLROSISKTR

USJP

0 1 2 3 4 5 6 7 8

Indicators for theknowledge-based economy

Number of missing values between1995 to 2002 with respect tocountry and indicator:

cf. DG RTD:Key Figures 2003 – 2004




Summary and Outlook

Eberhard Karls

UniversitatTubingen

The Multiple Imputation Principle (1)

NA

NA

NANA

Y1 Y2 Y3

Y1 Y2 Y31

1

11

Y1 Y2 Y32

2

22

Y1 Y2 Y33

3

33

Estimate 3

Estimate 2

Estimate 1

MI estimate

MI inference




Summary and Outlook

Eberhard Karls

UniversitatTubingen


Completedata

Incompletedata

Imputeddata set 1 θ(1) , var

(θ(1)

)

Imputeddata set 2 θ(2) , var

(θ(2)

)

Imputeddata set m θ(m) , var

(θ(m)

)

θ , var(θ)

missingvalues

θMI

varMI

(θ)




Summary and Outlook

Eberhard Karls

UniversitatTubingen


MI Estimates: Rubin (1978, 1987), or Barnard and Rubin (1999)

(θ − θ)/√

var(θ) ∼ N(0, 1)

Produce m completed data sets and calculate θ(j), var(θ(j))

θMI =1

m

m∑j=1

θ(j)

MI inference: Its estimated total variance is

var(θ) = varW(θ) +(1 +

1

m

)· varB(θ) with

varW(θ) =1

m

m∑j=1

var(θ(j)) and varB(θ) =

m∑j=1

(θ(j) − θMI )2

m − 1




Summary and Outlook

Eberhard Karls

UniversitatTubingen

First KEI-Imputation Based on PAN (1)

Research by Rassler and Munnich

Assume indicators are missing at random (MAR), Rubin andLittle (1987, 2002)

Fit univariate mixed-effects model for each KEI indicatorseparately (S-PLUS library PAN by Schafer 1997):

yc = Xcβ + Zcbc + εc , c = 1, 2, . . . ,C

Indicators yc = (yc1, yc2, . . . , ycT )′ for country cTime Xc with time interceptIntercept Zc

Fixed effects β0, β1

Random effects bc ∼ N(0, ψ) random effect for country cRandom errors εc ∼ NT (0, σ2 · IT )




Summary and Outlook

Eberhard Karls

UniversitatTubingen

First KEI-Imputation Based on PAN (2)

Generate m = 30 imputations after a burn-in period of 2000Gibbs cycles.

ACFs of ψ, σ2 and β suggest quick convergence

Lags of 1000 between each imputation are used

To do:

allow correlation between indicators ⇒ PAN for KEI accordingto Schafer & Yucel (2002)heteroscedasticity ⇒ possibly with Schafer & Yucel (2002)flexible serial correlation ⇒ future researchspacial autocorrelation ⇒ future research

Implementation in KEI with recommendations on modelling

Elaboration of accuracy with different models




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Composite Indicator on the Knowledge-based Economy (1)

Countries: EU15 + accession countries + USA + Japan

Time period: 1995 . . . 2002, early estimates for 2003

Indicators:

GERD Gross domestic expenditure for R & D per capita (POP)PhD Total new science and technology PhDs per capitaFTE Total researchers (FTE) per capitaGFCF Total gross fixed capital formation (excl. building) per

capitaEGov E-governmentTEE Total education expenditure per capitaLLL Life-long learning (per population aged 25-64 years par-

ticipating in education and training; POP1)




Summary and Outlook

Eberhard Karls

UniversitatTubingen


The seven single indicators will be denoted by y ti ;c with i for

indicator i , c for country c and t for year t.

Computation of z-scores for country c with given indicator iand time t with respect to t0 = 1995:

y ti ;c =

y ti ;c −

115

∑j∈EU15

y ti ;j√

115

∑j∈EU15

(y t0i ;j −

115

∑k∈EU15

y t0i ;k

)2=

y ti ;c −mean y1995

EU15

stdv y1995EU15

Special weighted average used to calculate z-scores

Translation term ignored

EU-14 instead of EU-15 (NA problem in Luxembourg)




Summary and Outlook

Eberhard Karls

UniversitatTubingen


The composite indicator I tc for country c and time t is then

I tc =

7∑i=1

λi · y ti ;c

with∑

i λi = 1 and

ytc =

(GERD

POP;PhD

POP;FTE

POP;GFCF

POP;EGov

1;TEE

POP;

LLL

POP1

),

λ =1

24·(2; 4; 2; 3; 3; 7; 3

).




Summary and Outlook

Eberhard Karls

UniversitatTubingen


1995 1996 1997 1998 1999 2000 2001 2002 2003B 3.7229 4.2197 4.2558 3.8697 4.1502 3.9287 5.0848 6.5412 7.8810DK 4.4973 4.5743 4.9003 5.3064 5.4270 5.8365 5.5205 4.8756 4.9856D 3.4558 3.3426 3.4559 3.5769 3.6986 3.8722 3.8699 4.1903 4.9375EL 1.1455 1.4450 1.4231 1.7580 1.7208 2.7858 2.7703 4.0024 4.8640E 2.0512 2.2117 2.3207 2.4624 2.6718 2.9120 3.3364 4.2854 4.8839F 3.5089 3.6056 3.6565 3.7244 3.9126 4.2079 4.3223 4.1323 4.9318IRL 2.3519 2.6215 2.8568 3.5979 3.5963 3.9702 4.4014 4.2331 4.8635I 2.5876 2.6998 2.7050 2.8407 3.0141 3.1834 4.2111 4.4280 4.8621L 4.1828 4.2132 4.7945 4.6289 5.1160 5.0595 4.7066 5.2216 5.8232NL 3.1769 3.3517 3.4954 3.6411 3.8147 3.9463 4.1874 4.2134 4.9699A 3.4867 3.6121 3.8066 4.0851 4.1375 4.2995 4.2735 4.4929 4.7863P 1.9244 2.1549 2.2246 2.6384 2.6827 2.8818 3.3156 3.5184 4.7724FIN 3.7844 3.7742 4.0505 4.5224 4.6914 5.0878 5.0697 4.7786 4.8476S 4.3808 4.4973 4.9553 5.1871 5.5521 5.6386 6.4295 4.7105 4.8236UK 3.1845 3.5367 3.4323 3.4701 3.5333 3.7311 4.1013 4.4949 4.9030

US 3.4160 3.4985 4.0556 3.9666 4.1419 4.3980 4.5686 4.8137 5.3452JP 3.5608 3.8332 3.9618 4.1311 4.0200 4.1406 4.3014 4.0665 4.8702




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Composite Indicator (1995)

●

●

●

●●●●

●

●●

● ●

●

●

●

●

●

●

●

●

B DK D EL E F IRL I L NL A P FIN S UK US JP

02

46

810

Com

posi

te In

dica

tor

Val

ue




Summary and Outlook

Eberhard Karls

UniversitatTubingen


●●●

●

●

●


02

46

810

Com

posi

te In

dica

tor

Val

ue




Summary and Outlook

Eberhard Karls

UniversitatTubingen


●

●●

●

●

●

●

●


02

46

810

Com

posi

te In

dica

tor

Val

ue




Summary and Outlook

Eberhard Karls

UniversitatTubingen


●

●●●


02

46

810

Com

posi

te In

dica

tor

Val

ue




Summary and Outlook

Eberhard Karls

UniversitatTubingen


●

●

●

●

●

●


02

46

810

Com

posi

te In

dica

tor

Val

ue




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Rank of Indicator Value (1995)R

ank


15

1015

17

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●




Summary and Outlook

Eberhard Karls

UniversitatTubingen


ank


15

1015

17

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●




Summary and Outlook

Eberhard Karls

UniversitatTubingen


ank


15

1015

17

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●




Summary and Outlook

Eberhard Karls

UniversitatTubingen


ank


15

1015

17

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Comparison of EU14 and EU15 ScalingEU15 EU14 EU15 EU14 EU15 EU14

I 1995c rk I 1995

c rk I 2000c rk I 2000

c rk I 2003c rk I 2003

c rkB 3.7229 13 3.8248 13 3.9287 7 4.0301 7 7.8810 17 8.0079 17DK 4.4973 17 4.6388 17 5.8365 17 6.0338 17 4.9856 14 5.0793 13D 3.4558 9 3.5238 9 3.8722 6 3.9398 6 4.9375 12 5.0295 12EL 1.1455 1 1.1699 1 2.7858 1 2.7988 1 4.8640 7 4.9714 8E 2.0512 3 2.1025 3 2.9120 3 2.9792 3 4.8839 9 4.9805 9F 3.5089 11 3.5826 10 4.2079 11 4.2958 11 4.9318 11 5.0171 11IRL 2.3519 4 2.4124 4 3.9702 9 4.0690 9 4.8635 6 4.9635 7I 2.5876 5 2.6701 5 3.1834 4 3.2862 4 4.8621 5 4.9551 6L 4.1828 15 4.2689 15 5.0595 14 5.1589 14 5.8232 16 5.9719 16NL 3.1769 6 3.2622 7 3.9463 8 4.0602 8 4.9699 13 5.0833 14A 3.4867 10 3.5892 11 4.2995 12 4.4129 12 4.7863 2 4.8877 2P 1.9244 2 1.9793 2 2.8818 2 2.9630 2 4.7724 1 4.8661 1FIN 3.7844 14 3.8869 14 5.0878 15 5.1752 15 4.8476 4 4.9511 5S 4.3808 16 4.4769 16 5.6386 16 5.7561 16 4.8236 3 4.9257 3UK 3.1845 7 3.2479 6 3.7311 5 3.8083 5 4.9030 10 4.9860 10US 3.4160 8 3.4648 8 4.3980 13 4.4432 13 5.3452 15 5.4021 15JP 3.5608 12 3.6345 12 4.1406 10 4.2288 10 4.8702 8 4.9475 4

σEU15 = (0.153; 0.034; 0.753; 0.288; 0.158; 0.308; 0.068)

σEU14 = (0.151; 0.036; 0.782; 0.285; 0.154; 0.288; 0.064)




Summary and Outlook

Eberhard Karls

UniversitatTubingen

Summary and Outlook

Missing values cause many problems

Possible exclusion of important indicatorsDelayed publication of composite indicators

Multiple imputation

Gain stable estimatesInferenceAchieve early estimates for indicators (timeliness)

Data quality has to be considered

Accuracy of estimates (sample based data)Coherence of indicator valuesAcceptance in politics

http://kei.publicstatistics.net


http://kei.publicstatistics.net

ec research on composite indicators with special focus on

Documents