Overview of European Microdata from Official Statistics

Roxane SilbermanCNRS/Réseau Quetelet and DwB Principal Investigator

4th DwB Training CourseWorking with data from Official Statistics particularly the Labor Force Surveys

Ljubljana, September 17-19, 2014

Aim of this presentation 4 objectives for this introductionProvide a comprehensive overview of what is available for research both at European and at national level regarding official microdata which are still underused

• Provide an overview of European microdata An increasing rich resources as the number of official microdata provided at European level is

increasing• Provide a general understanding of their construction

Eurostat microdata completely based on national microdata Using the European microdata requires understanding these links as they raise important

methodological issues• Provide a broad overview of official microdata available at national level

Though increasing links between national and European level Related microdata available at national level may be more detailed, other variables, and

sometimes on a longer period (ex. LFS) Other microdata available at national levels

• How to locate and access these microdata and new tools DwB will offer

Outline• National and European official microdata: terminology and historical backgrounds• Overview of official microdata in Europe

Integrated European microdata at Eurostat Other European official microdata National official microdata collected and harmonized in other European databases Other national microdata

• How to locate and access official microdata within Europe? General principles DwB support and new tools

I. National and European official microdata Terminology and historical backgrounds

- Official microdata ? A vast perimeter, moving and covering different types of microdata

- Historical backgrounds for European microdata- Differences in national systems for official microdata- Consequences for working with data

Official microdata ?A vast perimeter …

• Different words (official, government, national etc..) In any cases, statistics provided by government bodies A larger perimeter than NSIs and Eurostat

• Microdata provided by : National level

National Statistical Institute National Statistical Administrations (more or less coordinated by the NSI) Central banks Government agencies particularly those in charge of Social security, health, pensions … Local authorities increasinglyNumber of government bodies producing and providing data vary according to the organization of

the each statistical system and degree of centralization

European level Eurostat European Central Bank European Commission and agencies

… moving• Perimeter may change according to the decision of governments/NSIs

Ex.1. France: Cereq microdata (ex Transition from school to work) went out of the official perimeter Ex.2. Household finance and consumption surveys (HFCS) move to central banks coordinated by the BCE

• Mixed status for some government agencies (social security, unemployment …) New kind of data

o Data certified by NSIso Big data such as data from mobile phones may be used by NSIs and European bodies

Under different legal framework (surveys, administrative data, business data, fiscal data, health data, financial data, big data …) with consequences for access

• Perimeter does not necessarily cover similar data in the different countries depending on the role of other producers (universities …) and historical changes (Eastern countries recently joining the EU)

… and covering different types of microdata

• Censuses and/or registers (including longitudinal samples from censuses)

• Surveys (including some panels) Individual and households surveys Business surveys

• Administrative data (frequently longitudinal databases) Individuals Business

• Combined datasets Administrative datasets from different sources

o More common in countries based on registerso Now increasingly the case in all countries (yet requiring a common identifier)

Surveys also using administrative data more and more common (particularly for longitudinal data)

Historical background for European microdata

• Increasing harmonization process at international level since WW2 led by international organisations (UN, OECD…) mostly by encouragement and persuasion

• European Union framework specific as there is a political and legal framework European Statistical System development started with the CECA during the 50ies Progressively developed during 30 years within a somewhat unclear framework (Commission also relying on

other sources) New start and developments since the 90ies towards more autonomy and integration A legal framework establishing the European Statistical System in 2009 (Regulation N° 223/2009)

o Rather recentlyo As a partnership between the Commission statistical authority (Eurostat), the NSIs and other national

statistical authorities, with cooperation with the ECBo In compliance with the principle of subsidiarity

• Main focus on indicators, less on microdata for research analysis • Subsidiarity principle at the core of the ESS • Harmonization and integration process growing: 80% of the national data linked to the European

requests

… relying on national official statistical systems built in different ways• National statistical systems pre-existed to the ESS

• Built in quite different ways through history from diverse sources and bodies, piece by piece, Toward an increasing coordination role of the NSI Yet keeping traces from the construction even in centralized statistical system where some bodies still remain apart Central banks in general apart (also at European level)

• Important differences Surveys and registers countries Centralized vs decentralized Level of coordination differ

• Political systems Regional autonomy (Spain, UK and Scotland) Federal system (Germany and the landers) France and overseas departments National and local authorities gaining importance Recent political changes (Eastern countries)

• Resulting in differences also in type, number and variety of datasets and access conditions

Some consequences for working with data

• Rich and still underused resources

• However a complex landscape Information and access points highly fragmented both at national and at European level Different legal frameworks for access both at national and European level

• European microdata strongly relying on this complex landscape requiring agreements and harmonisation

• National data remain an important resource, particularly for administrative data However also raising issues for transnational access The harmonisation process may impact the series

• Issues for European research : information, access, good metadata, language issues

II. European and national microdata

Three subsets from a European perspective

European integrated microdataNational microdata harmonized at European level Other national microdata for comparative research

European integrated microdataProduced and provided at national level and integrated and provided at European level by European government bodies Reminder: national part may differ at national level and at European level

A part are pre-existing surveys integrating harmonization requirements

Eurostat European Central Bank European Commission and other European government bodiesOthers under EU regulations or recommendations of Eurostat

European Integrated microdata Eurostat microdata

A growing number of datasets European Union Labour Force Survey (LFS)

European Community Household Panel (ECHP)Statistics on Income and Living Conditions (SILC)Adult Education Survey (AES)Community Innovation Survey (CIS)Structure of Earnings Survey (SES)European Road Freight Transport Survey (ERFT)European Health Interview Survey (EHIS)Continuous Vocational Training Survey (CVTS) Community Statistics on Information Society (CSIS)

However …• Date of inclusion for countries differ

National microdata may offer a wider historical perspective in some cases

• Implementation in national instruments differ: different surveys, variables from administrative data Good metadata important (ex. questionnaires)

• More questions in several cases at national level

• Also more detailed microdata at national level in some countries for equivalent type of files

More questions in some national surveys: ex. SILC

France

•The current French SRCV system took over from the former permanent survey of living conditions (EPCV) system (EPCV 1996-2004 and Permanent Living conditions Survey 1996-2004) in 2004

It reprises some questions from the EPCV and includes some other questions not required at Eurostat level

Persons being in the panel for 9 years /vs 4 years required by Eurostat

Estonia

•ESS is the Estonian branch of the EU-SILC. •Statistics Estonia, however, has added questions which are of interest to the domestic consumers of Estonia

In 2004, four modules were added commissioned by Estonian domestic consumers. The topics of the four modules concerned social contacts; family attitudes and political views; crime, violence and feeling of security; and ethnic integration.

In 2005, there were two modules were added at Estonian level, entitled “From school to work” and “Trade unions and collective agreements”.

More details for same type of files: ex. LFS

• Recent progress in terminology regarding the different type of files

• Public Use Files (PUF)

• Scientific Use Files (SUF) or de facto anonymized files

• Secure Use Files (ScF) or de-identified files

• However differences remain in the practices Ex. LFS: Variables such as age, family status, nationality less aggregated Scientific Use

Files (de facto anonymized files) provided by INSEE in France than by Eurostat for the same type of files

Countries Years CUF PUF SUF On-site access

Remote Execution

Remote Access

Czech 1995-2011

Estonia 1995-2011 X X X X

Finland 1969-2011 X X

France 1962-2010 X X X

Hungary 1992-2011 X X

Italy 2004-2011 X

Latvia 1996-2011 X

Lithuania 1998-2011 X X

Norway 1972-2011 X X

Poland 2000-2011

Portugal 1992-2011

Slovakia 2002-2011 X

Slovenia 1997-2011 X X

Spain 1964-2011 X X

Switzerland 1991-2011 X

UK 2002-2010 X X X X

And a wider historical perspective for detailed microdata at national levelEx: LFS

European integrated microdata European Central Bank microdata

• Household Finance and Consumption Survey

• Every 3 years

• First deliverable in 2013

• No pre-existing national survey in some countries while older waves in others France:

o Every 6 years and oldest waveso Questionnaire 112 pages in France vs 65 pages for the European survey, yet some variables

collected at European level not in the French surveyo Adaptation of the survey was needed (break in the national serie)o Moved from the NSI to the Central Bank (with consequence for researcher access)

European integrated microdata Other European government microdata

• The European Commission The Eurobarometers since 1973 The Business and Consumer Surveys (The Directorate-General for Economic and

Financial Affairs /DG ECFIN)

• EUROFOUND Surveys on working conditions

The Eurobarometers

•Since 1973•European opinion•Twice a year

DG ECFIN and the Business and Consumer Surveys

•The European Commission, the Directorate-General for Economic and Financial Affairs (DG ECFIN) manages a network of national institutes to conduct a harmonised EU programme of 6 business and consumer tendency surveys (quarterly or monthly from 1985, 1995 for services).

Industry Services Consumers Retail trade Building Investment

and some others

EUROFOUND and the surveys on working conditions The European Foundation for the Improvement of Living and Working Conditions

• An autonomous EU agency• Set up by the Council Regulation (EEC) No. 1365/75 of 26 May 1975• Contributes to the planning and design of better living and working conditions``

3 surveys combining companies and employees surveysThe European Working Conditions Survey (EWCS) : 1990, 1995, 2000, 2005, 2010 Working conditions and the quality of work and employmentThe European Quality of Life Survey (EQLS) : 2003, 2007, 2010 A broad range of indicators of quality of life, both objective and subjectiveThe European Company Survey (ECS) : 2004, 2009, 2013 Workplace practices based on the views of both managers and employee representatives

• Datasets available for download at the ESDS (UKDA)

European Integrated microdata Other microdata under EU regulations

• Information and Communication technologies surveys (TIC)` Regulation n°808/2004 and updated regulation 1006/2009 linked with the European

roadmap for the TIC

• Household budget surveys (HBS) Eurostat recommendations about methodology and harmonization

• Censuses Successive regulations (2008, 2009, 2010…) to achieve more comparability, output

oriented

National official microdata post-harmonized at European level

• By non government bodies

• Collected and a posteriori harmonized by universities, archives IECM (IPUMs international)

o Censuses

• May gather official microdata and academic datasets LIS (Luxembourg Income Study)

o Household Budget Survey

MTUSo Time Use Survey

More detailed microdata may be accessed at national level

IECM + national disseminationIECM in progress + national dissemination

Only national disseminationNo information

IECM/IPUMS and national dissemination of European censuses

However, more may be available at national level Ex: United Kingdom and France

Several Public Use Files (PUF) available for dwelling, individuals, residential mobility at different geographical levels on INSEE website

Several Scientific Use Files (SUF) more detailed available for the researchers via Archives

Access to highly detailed microdata available via Secure access for approved research projects (ONS in UK, CASD/GENES in France)

LIS and MTUS

• LIS and MTUS are examples of post harmonized microdata at European level from both government sources and non government (universities) sources

• More detailed microdata at national level in several countries

Country Survey Year Income Unit Data Collection

Cyprus LWS 2001 Primary Economy Unit Central Bank of Cyprus

and University of Cyprus

Finland LWS 1998 Household Wealth Survey Statistics Finland

Germany LWS 2001 German Socio Economic

Panel

German Institute for

Economic Research, DIW

Italy LWS 2002 Survey of Household

Income and Wealth

Bank of Italy

UK LWS 2000 British Household Panel

Survey + cross national

equivalent files

Institute for Social and

Economics Research

LIS 1999 Family Resource Survey Department for Work and

Pension , ONS, National

Centre for Social Research

LIS is a cross-national data center, located in Luxembourg. LIS is home to the Luxembourg Income Study Database (LIS) and the Luxembourg Wealth Study (LWS)

France Time Use Survey INSEE

BELGIUM 1966 The Multinational Comparative Time-Budget

Research Project

Pierre Feldheim and Claude Javeau, Sociological Institute,

Free University of Brussels

BULGARIA 1988 The 1988 Bulgarian National Time Use Survey Central Statistical Office, Institute of Sociology at the

Bulgarian Academy of Sciences

FINLAND 1987-88 Time Use Survey Statistics Finland

HUNGARY 1965 The Multinational Comparative Time-Budget

Research Project

Sociological Research Group, Hungarian Academy of Sciences

ITALY 1979/80 Il Tempo della Citta. Una Ricerca Sull'uso del

Tempo Quotidiano in una Metropoli

University of Turin

UK 1961 The People's Activities BBC

UK 2005 Omnibus Survey, One Day Diary of Time Use

Module

The Office for National Statistics coordinated the study and

collected the data. The Institute for Social and Economic

Research at the University of Essex transferred the diaries

into coded electronic data.

The Centre for Time Use Research collects Time Use Surveys

Other national rich resources from official statistics, non integrated nor post-harmonized at European level

• Example for surveys on living conditions (other than those related to the SILC)May include some from academic institutions

• Administrative microdata Social security, pensions. The Impact of Social Security Contributions on Earnings: Evidence from administrative data in France, Germany, Netherlands an UK ( Antoine Bozio, Research proposal submitted in September 2011 to the Open Research Area (ORA) call)Tax data

• Administrative and surveys combined Ex: Employers and employees A guide to Linked Employer-Employee Data Sources in the EU and Beyond (Tanvi Desai, London School of Economics, 2008)

Other national surveys on living conditionsCountry

Denmark

The register for health and social conditions 1977-2012

Family allowance and child benefits 1957-2012

Estonia

Estonian Social Survey 2004-2010

Household Budget Survey 2010

France Living conditions 78-79; 86-87; 93-94

Permanent Living Conditions Survey (EPCV) 1996-2004

The Statistical survey on income and living conditions (SRCV)

2004-2009

Survey of users of accommodation and hot meal distribution services (Homeless people)

2001

United Kingdom

General Lifestyle Survey (GLF) 2000-2008

Living Standards During Unemployment 1983-1984

English Housing Survey 2008-2011

Norway

Norwegian Level of Living Study 1973-2007

Study on housing conditions among low-income families

1995

Welfare and level of living among the very frail elderly

2000

Linked Employer-Employee Data sources, examples of national surveysCountry National surveys Data producer Access notesCzech Republic

Information System on Average Earnings (ISAE)

The Czech Ministry of Labour commissions the private agency TREXIMA

Direct access is only available on-site at TREXIMA

The Structure of Earnings Survey for the Czech Republic is derived from the ISAE there is no access to the Czech SES data at the national level.

Spain INE does not provide access to any other linked employer-employee or firm panel data resources than SES. The Banco de España conducts an annual survey of non-financial firms, the Central Balance Sheet Data. However, the microdata for this survey are only available to researchers affiliated with the Banco de España

Germany LIAB (Linked Employer-Employee Data of the IAB)

IAB (Establishment Panel )

Data access is possible via on-site use and afterwards via remote data access also.

The L-IAB data is a linked employer-employee dataset constructed from the IAB Establishment Panel and the Federal Employment Agencies employment statistics.

DESTATIS (employment statistics)

France REPONSE (Relations Professionnelles et Negociations d’Entreprise)

DARES DARES

COI (Changement Organisationnels et l’Informatisation)

DARES Réseau Quetelet ,(SUF) The COI is used for France’s contribution to Eurostat’s ICT survey

Enquête Familles at Employeurs

INED

DADS INSEE Réseau Quetelet, (SUF)Labour cost and structure of earnings survey (Ecmoss)

INSEE Réseau Quételet, (SUF) The Structure of Earning Survey for France is a part of this survey

Country Data sources Years Data collection ProviderFrance DADS 1993- Fiscal and social

administration, INSEEReseau Quételet (CMH, GENES)

Germany Sample of Integrated Labour Market Biographies (SIAB)

1975-2008 IAB IAB

Lohn- und Einkommensteuerstatstik – faktisch anonymisierte Daten (FAST)

1992-2004 Fiscal administration; DESTATIS

DESTATIS

Verdienststrukturerhebung (VSE) 1990-2006 DESTATIS DESTATIS

United Kingdom New Earnings Survey (NES) 1975-2003 ONS Secure data service, UKDA

Annual Survey of Hours and Earnings (ASHE) 2004-

Annual Business Inquiry

Bozio Antoine. The Impact of Social Security Contributions on Earnings: Evidence from administrative data in France, Germany, Netherlands an UK. Research proposal submitted in September 2011 to the Open Research Area (ORA) call

III. How to locate and access official microdata within Europe?

MetadataTransnational access

DwB support and new tools

Information, metadata dissemination at national level

• Still highly fragmented NSIs generally provide information, however metadata not always available on websites

Other government departments/bodies much more uneven

Some countries are opening a portal for official data (open data initiatives) with a larger perimeter yet mostly for aggregate data

In a few countries, data archives for social sciences provide information and metadata for official microdata

Metadata dissemination at national level for official microdata: NSIs and Archives

Information and metadata dissemination at European level

• Each European body (Eurostat, ECB …)

• The CESSDA (the European research data infrastructure for social sciences) catalogue vurrently includes only few metadata from official microdata

AccessTerminology issues

Type of files ranging from fully anonymized to highly detailed microdata Campus files (CUF) Public use files (PUF) Scientific Use files (SUF) or de facto anonymized microdata Secure Use Files (ScF) or highly detailed, de-identified microdata

Mode of Access via CDRom/FTP, on site, remote execution, remote access

Progress towards a common terminology, however differences remain in terms and in practices

Access to Eurostat and other integrated European microdata

Eurostat A new framework for access (see Michel Isnard presentation) A 2 steps procedures (institution and researcher/ research project that may be time

consuming, particularly for access to SUF For ScF, currently only in Luxembourg A Remote access network in project (DARA ESSnet)

European Central Bank in progress

LIS: remote execution for countries paying fees as members

IPUMS/ IECM free and easy, yet currently highly anonymized

Access to national OS in Europe• Highly fragmented: Different types of accreditation procedures, application forms,

criteria

However transnational access to national microdata increasingly possible

Even for highly ScF

For the national components of Eurostat microdata may sometimes be easier or …. more difficult

For all kind of data, may be time consuming in case you want to access data provided by different/countries/ institutions

Combining national data from different countries for running a single analysis still problematic for ScF (highly detailed microdata)

Important for future comparative research based on administrative data difficult to integrate at European level

Transnational access to Public Use Files

In some countries the number of PUF is (very) limited.

Transnational access to Scientific Use Files

Data archives providing access to SUF for OS

Transnational access to highly detailed microdata

Countries Years CUF PUF SUF On-site access

Remote Execution

Remote Access

Austria 2003-2007   X X X  

Czech 2005-2010          

Estonia 2004-2010 X X X   X

Finland 1967-2011   X X    

France 2004-2009   X     X 

Germany 2005-2008     X X  

Ireland          

Italy 2005-2010   X X    

Latvia 2005-2011   X     X

Lithuania 2005-2010 X X      

Poland 2005-2010          

Portugal 2004-2009          

Slovakia 2006-2011   X      

Slovenia     X    

Spain 2004-2011   X X    

Switzerland 2007-2009   X      

Ex. Type of access at national level for the national components related to the SILC

III. DwB support

Current activities and future perspectives CIMES and MISSY

DwB support • Current activities

Support for transnational access to ScF (highly detailed microdata) from 4 countries (Germany, France, UK, Netherlands) o DwB regular calls, support for accreditation, financial support for travel and fees for RDCs

(last call: ultimate deadline is 15 Oct. 2014)

Metadatao MISSY Microdata Information SystemAn online information system with metadata for all integrated European microdata from

official statistics held by Eurostat and Integrated European Census Microdata (IECM)

o CIMES Centralising and Integrating Metadata from European StatisticsCurrently 1,796 datasets from 22 European countries including information on access

conditions

• Future mid-term and long-term perspective A set of Services for Official Statistics ideally linked to the CESSDA Portal in cooperation with the

European Statistical That could offer a range of services: metadata, training, support for accreditation, a European

Remote Access Network for access to confidential OS

Information on the accreditation procedures

• Currently factsheets See DwB website at: http://dwbproject.org/service/accreditation_db.html

• To be soon integrated with CIMES

Thanks for Listening

Contact: roxane.silberman@ens.fr

Website: http://www.dwbproject.org/