in partnership with

Overall handbook to set up a S-DWH

WP: 4 – Dissemination Deliverable: 4.6

Version: 0.4 (draft) Date: 19 September 2013

Authors: Lars Goran Lundell,Harry Goossens, Michel LindelaufMaia Ennok

NSI: Statistics SwedenCBS (NL)Statistics Estonia

ESS - NET

ON MICRO DATA LINKING AND DATA WAREHOUSING

IN PRODUCTION OF BUSINESS STATISTICS

Handbook to set up a S-DWH 1version 0.4 / 19 September 2013

Content

1. Introduction.......................................................................................................................3

2. The Statistical Data Warehouse.........................................................................................4

3. The main phases for setting up a S-DWH...........................................................................6

4. The 3 tracks within the S-DWH process:............................................................................7

1.1 Metadata....................................................................................................................7

1.2 Methodological aspects..............................................................................................8

1.3 Technical aspects........................................................................................................9

5. The Road Map for setting up a S-DWH............................................................................13

5.1 Roadmap S-DWH: General overview........................................................................14

5.2 Roadmap S-DWH: Design phase...............................................................................15

5.3 Roadmap S-DWH: Build phase..................................................................................16

5.4 Roadmap S-DWH: Implementation phase................................................................17

5.5 Roadmap S-DWH: Total map....................................................................................18

Handbook to set up a S-DWH 2version 0.4 / 19 September 2013

1. Introduction

In October 2010, the ‘ESSnet on micro data linking and data warehousing in statistical production' was established to provide assistance in the development of more integrated databases and data production systems for (business) statistics. In order to improve and optimise statistical production, ESS Member States are searching for ways to make optimal use of all available data sources, existing and new. In daily statistical practice this means supporting and assisting statistical institutes to increase the efficiency of data processing in statistical production systems and to maximize the reuse of already collected data in the statistical system.

This modernisation implicates an important organisational impact. First there is the need to develop and implement a complete new way of organising and operating the statistical production processes. Second, it also comes with higher and stricter demands for the data and metadata management. Both activities are often decentralised and implemented in various ways, depending on the needs of specific statistical systems (stove-pipes), whereas realising maximum re-use of available statistical data just demands the opposite: a centralised and standardised set of (generic) systems with a flexible and transparent metadata catalogue that gives insight in and easy access to all available statistical data.

To reach these goals, building a statistical data warehouse (S-DWH) is considered to be a crucial instrument. The S-DWH approach enables NSIs to identify the particular phases and data elements in the various statistical production processes that need to be common and reusable.

Main focus of the ESSnet was on issues that are common for the majority of the NSIs within the ESS when applying a data warehousing approach for statistics. A thorough enquiry among the ESS Member States resulted in a set of deliverables divided over 3 main topics:

1. Metadata

2. Methodological aspects

3. Technical aspects

In the various workshops, held to interactively exchange information and receive feedback, MS expressed great demand for a practical handbook that helps and guides in the process of developing and implementing a S-DWH.

This handbook answers the following questions: What is a Statistical Data Warehouse (S-DWH) ? How does a S-DWH differ from a traditional = 'commercial' DWH ? Why should we build a S-DWH ? Who are the envisaged users of a S-DWH ? Give a road map for designing, building and implementing the S-DWH:

- What are the prerequisites for implementing a S-DWH ?- What are the phases/steps to take ?- How to prepare for an implementation ?

The handbook is set up as a lean quick reference guide around the S-DWH roadmap. Goal is to guide users through the process of setting up and implementing a S-DWH by indicating what deliverables of the ESSnet (recommendations, guidelines etc.) to use at which phase in the development process.

Handbook to set up a S-DWH 3version 0.4 / 19 September 2013

2. The Statistical Data Warehouse

This chapter gives a short explanation on most common terminology to explain the statistical data warehouse. Deliverable 1.1, ‘the Metadata Framework’1 gives more detailed explanation and information on the terminology used in the project.

Data Warehouse (DWH)

The generic definition of a Data Warehouse (DWH) says that it is “a central repository of data which is created by integrating data from one or more disparate sources”2. In the DWH current and historical data are stored and organised in ways that facilitate combining data to, e.g., to perform analyses and to create reports.According to broader and perhaps more useful definitions the term DWH should not only be understood as a way of storing data, but it must also include all the functions and tools necessary to extract, transform and load data (ETL tools), to maintain the data structure, and to make data available to end users in ways that suit their tools.According to the role and function, a commercial (or traditional) DWH mostly is set up as a supportive system to the primary process of an organisation, with as main goal to produce and deliver management information that is used to manage and improve the primary process.

Statistical Data Warehouse (S-DWH)

This project uses the term Statistical Data Warehouse (S-DWH) to refer to a DWH that is purpose-built specifically to support the production of national and international statistics. Thus the S-DWH is defined as a central store of statistical data, regardless of their sources, for managing all available data of interest, thereby improving the NSI’s ability to:- use and reuse data in order to create new data or new outputs;- create reports;- execute analyses;- produce any required information.According to the role and function, a statistical data warehouse is developed as a crucial element in the primary process, which simply is: to produce statistics.

Business Intelligence (BI)

The general DWH is often considered part of a Business Intelligence (BI) system. BI technology can handle large amounts of historical and current data stored in a DWH. Specialised BI tools let the users analyse the information in the DWH and even make predictions in order to make better business decisions. Many BI tasks, such as decision support, include quick creation and immediate analysis of statistics based on data from the DWH. Supporting creation and analysis of statistics is the main purpose of the S-DWH, but the demands for quality are generally higher, while the analysis may follow immediately on creation or later.

1 Framework2 Wikipedia: http://en.wikipedia.org/wiki/Data_warehouse Handbook to set up a S-DWH 4version 0.4 / 19 September 2013

Metadata

The S-DWH contains only statistical data and is dedicated to supporting efficient production of statistics. Data in the S-DWH may be atomic, micro data, or aggregated, macro data. All data must always be defined and described in accompanying metadata.Since the data warehouse is not only one single data store, but consists of several parts, or layers3, metadata must also describe the processes that move the data through the layers from source to presentation and dissemination (process metadata).

Standards

There are several formal and industry standards that should be considered when building a DWH. The architecture should be supported by well-established data modelling standards. In addition to the standards and rules that support the design of any DWH, the S -DWH should also be designed and built in accordance with the standards that are used in the statistics society. The process model GSBPM, the information model GSIM, the metadata registry standard ISO/IEC 11179 and the classification model (Neuchâtel model) are examples of important and widely accepted standards that should be taken into account when designing a S-DWH4.

Why build and use a S-DWH ?

There are several alternative models that can be used to describe and build statistics production systems, e.g., the traditional stovepipe model and several versions of integrated models5. The S-DWH model is generally considered as being the most advantageous one compared to the other models. Some arguments that speak in favour of using a S-DWH include: Easier to reuse data, “collect once, use many times”; Facilitates cross-domain analysis; Well suited for process oriented production systems (even though its data model is not

specifically designed for that purpose); Supports standardisation of tools and methods; Enables efficient governance and maintenance.

3 S-DWH Business Architecture (http://www.cros-portal.eu/sites/default/files//DWH-SGA2-WP3%20-%203.1%20S-DWH%20Business%20Architecture_V3.0_Feb2013_0.doc )4 Overview and recommendations on the use of metadata models (http://www.cros-portal.eu/sites/default/files//DWH-SGA2-WP1%20-%201.3%20Overview%20of%20and%20recommendations%20on%20the%20use%20of%20metadata%20models_v0.5.doc )5 S-DWH Modular Workflow (http://www.cros-portal.eu/sites/default/files//DWH-SGA2-WP3%20-%203.2%20S-DWH%20Modular%20Workflow_V4.1_2.doc) Handbook to set up a S-DWH 5version 0.4 / 19 September 2013

3. The main phases for setting up a S-DWH

From a project management view, the process of setting up and implementing a S-DWH does not essentially differ from other major projects that involve organisational changes in combination with new processes and (IT) systems. Basically 5 more or less generic phases can be distinguished:

Business Case

As for all projects it is an essential and required precondition to compose a solid business case that needs to be approved the responsible authority/management/sponsors.The business case must clearly state the aimed goals, describe and explain the expected benefits and of course give a sound cost – benefit analysis. The framework [deliverable 1.1] can be used as a good fundament when writing the business case.

Design

The first phase in the actual development is the design of the S-DWH, with all elements and aspects. This should cover various aspects:

What type of S-DWH, active or passive ? The architectural framework for the S-DWH. A clear description of the functions of the S-DWH. The necessary metadata designs (metadata model, meta system etc.) Methodological concepts (role BR etc.).

All designs must be approved by the responsible managerial body (steering group, program management e.g.)

Build

In the ‘build’ phase the various elements of the S-DWH need to be realised. For the most part these are strongly IT related components: databases, repository, ETL processes etc. Main milestones in this phase are tool selection, translating design to business rules, testing and documentation. As the development of a S-DWH mostly consists of a complex set of systems, it is recommended to work in small incremental steps.

Implement

The implementation phase means actually putting the S-DWH to work. After defining a sound implementation strategy, most important milestones in this phase are setting up the governance of the (,eta)data management, ensuring confidentiality and training users.

Use & Maintain

After implementing the S-DWH the phase of operational use starts. The feedback from daily statistical use requires also a steady process of maintaining 2 main aspects of the S-DWH:

1. The content of the S-DWH (metadata and statistical data)2. The functional and technical systems

The focus of ESSnet was on the elements of the phases design, build and implementation. Therefore the roadmap of this handbook concentrates on and describes these 3 phases.

Handbook to set up a S-DWH 6version 0.4 / 19 September 2013

4. The 3 tracks within the S-DWH process:The goal of the statistical data warehouse is to enable NSIs to produce flexible outputs, in an efficient way, with maximum re-use of data that is already available in the statistical system. Therefore the ESSnets needs to focus on issues that are common for the majority of the NSIs when applying a data warehousing approach for statistics, resulting in 3 main tracks (or work packages):

1.1 Metadata

One of the key factors and drivers in a S-DWH is the information about one or more aspects of the data itself, usually referred to as "metadata".

‘Metadata is the DNA of the data warehouse, defining its elements and how they work together. [...] Metadata plays such a critical role in the architecture that it makes sense to describe the architecture as being metadata driven’.

The metadata provides the access to the data and must enable a clear and unambiguous description of the data and its elements. All data in the S-DWH must have corresponding metadata: ‘no data without metadata’. Users must be able to search the entire metadata layer and, if permitted, to access the physical statistical data via the metadata. Thus, metadata plays a vital role in the S-DWH, satisfying 2 essential needs:

1. to guide statisticians in processing and controlling the statistical production

2. to inform end users by giving them insight in the exact meaning of statistical data

In order to meet these 2 essential functions, the statistical metadata must be:

correct and reliable (the metadata must give a correct picture of the statistical data),

consistent and coherent (the metadata driving the statistical processes and the reporting metadata presented to the end users must be compatible with each other),

standardised and coordinated (the data of different statistics are described and documented in the same standardised way).

Finally, since the different users of the (meta)data have diverse needs, it is essential to ensure an effective management of the statistical metadata in the S-DWH.

In the metadata track (or work package), the first focus was on the identification of the various kinds of essential metadata and recommendations and guidelines on their use. Further focus was on the use of metadata models, the required functions of a metadata system and the governance of metadata in the S-DWH.

Deliverables of the metadata work package:1.1 Framework of metadata requirements and roles in the S-DWH.

This deliverable gives definitions and background information on the roles and purposes of metadata in the S-DWH in generic terms. It destined to provide a common language.

1.2 Recommendations on the impact of (meta)data quality in the S-DWH.This deliverable is about monitoring the quality of (meta)data in a S-DWH. For data exchange, it is more or less common to use indicators to measure data quality. The advice is to also define a set of indicators for metadata quality, following and using the data quality systems.

Handbook to set up a S-DWH 7version 0.4 / 19 September 2013

1.3 Overview of and recommendations on the use of metadata models.This deliverable gives an overview of metadata models and recommendations on their use. The use of a metadata model is a key element in structuring and standardising the statistical metadata within a NSI in a generic way. In the context of the S-DWH, a metadata model is a standardized representation used to define all necessary metadata elements of statistical information systems.

1.4 Definition of the functionalities of a metadata system to facilitate and support the operation of the S-DWH.This deliverable gives a detailed description of the functionalities that are necessary to facilitate and support the operation of the S DWH. In order to meet these diverse needs of different users of the (meta)data, the statistical metadata must be managed and maintained in a metadata system that covers these functionalities.

1.5 Recommendations and guidelines on governance of metadata management in the S-DWH.This deliverable explains the importance of reliable governance of metadata management in a statistical organisation when operating a S-DWH. It focuses on the main issues to consider when establishing, running and maintaining metadata management in a S-DWH. Implementing good governance for metadata management is highly important for a S-DWH.

1.6 Documentation of the mapping of deliverable 1.4 on the ‘ideal architecture’ framework.The detailed metadata system functionalities are mapped on the layered S-DWH architecture and the GSBPM workflow.

1.2 Methodological aspects

A key challenge in the process of designing and implementing a Statistical Data Warehouse is to match the various statistical requirements that are set by the statistical users of the S-DWH. The indicated methodological challenges that need to be covered and ensured are about:

Impacts on statistical methods

Which are the methodological advantages and drawbacks ?

Which considerations as to statistical methods are needed ?

How to handle confidentiality issues ?

How to deal with data linking ?

Also this work package provided input to actions/deliverables of the other 2 tracks, by reviewing deliverables and advising from the methodological perspective.

Deliverables of the methodological work package:

2.1 Methodological evaluation of metadata framework (deliverable 1.1, WP1)This is an Internal review of the metadata framework from a methodological stand point.

2.2 Guidelines (including options) on how the BR interacts with the S-DWH.This document describes an essential part of the S-DWH: the role and position of the statistical business register. The Business Register holds a central role in the S-DWH in order to link different units from different data sources and to act as a population frame.

Handbook to set up a S-DWH 8version 0.4 / 19 September 2013

2.3 Methodological evaluation of the S-DWH Business Architecture (deliverable 3.1, WP 3).This deliverable documents the analysis of the evaluation of the S-DWH business architecture, indicating methodological preconditions and issues that need to be considered before designing, building and implementing the S-DWH.

2.4 Guidelines/recommendations for application within the S-DWH of the data linking aspects.This deliverables gives an overview on data linking aspects in a S-DWH. It provides information about data linking methods, about useful links, and it mentions possible problems that can occur when linking data from multiple sources. Finally it presents guidelines about the methodological challenges on data linking.

2.5 Guidelines/recommendations for application in the S-DWH of the confidentiality aspects.This report outlines the options for understanding and dealing with the confidentiality aspects of combining and re-using data from a Statistical Data Warehouse that comes with an increased risk for compromising the confidentiality of the data.

2.6 Guidelines on selective editing options for the S-DWH.This report examines options for efficient editing in a Statistical Data Warehouse, specifically exploring how selective editing may be used in this context. Focus is on two widely available selective editing tools, to consider if they could be used for efficient editing in a S-DWH.

2.7 Mapping the coverage of ESSnet projects relevant to WP2 of the ESSnet on DWH. The purpose of this document is to identify any potential areas of overlap, or any potential gaps, between the deliverables of Work Package 2 of the ESSnet on Data Warehousing and the results of other ESSnet projects. It identifies which of the completed and on-going ESSnet projects are of direct relevance to WP2 of the ESSnet DWH.

2.8 Guidelines on detecting and treating outliers for the S-DWH.This deliverable explains the distinction between outliers and errors, the three possible types of outliers in a S-DWH and gives recommendation on how to deal with them.

1.3 Technical aspects

This track covers all essential architectural and technical elements for designing, building and implementing the statistical data warehouse. Focus of this work package is to provide a generic model of the statistical data warehouse:

The S-DWH business architecture.

A generic fully active S-DWH system consists of four functional layers:

I. Source layer

II. Integration layer

III. Interpretation and data analysis layer

IV. Access layer

Handbook to set up a S-DWH 9version 0.4 / 19 September 2013

The Source layer is the gathering point for all data that are going to be stored in the S-DWH. The input consists of data from both internal and external sources. Internal data mainly originate from surveys carried out by the NSI, but may also be data from maintenance programmes used to manipulate data in the DWH. External data refer to administrative data, normally collected by other organisations for other purposes than statistics production.

From the Source layer, data is loaded into the Integration layer. The process of extracting data from source systems and transform them into useful content in the data warehouse is commonly called ETL (Extract-Transform-Load). The Integration layer is used for all integration and reconciliation activities of data sources in order to become a first integrated staging area, independent from sources.

The Interpretation layer is designed for the needs of statistical experts. It is built to support data manipulation and complex operations such as hypothesis testing, data mining and design of new statistical strategies. The Interpretation layer is also where the designs of data cubes for use in Access layer are defined.

The Interpretation layer will contain sums, aggregates and calculated values, but it will also keep all data at the finest granular level. This is necessary to be able to handle all possible queries, and to manage changes of required output over time.

The Access layer is the layer for the final presentation, dissemination and delivery of information. This layer is used by a wide range of users and computer instruments. The data storage is optimized to effectively present and compile data.

Handbook to set up a S-DWH 10version 0.4 / 19 September 2013

Management processes to govern S-DWH operations

In the S-DWH are fourteen over-arching statistical processes needed to support the statistics production processes, nine of them are those found in the GSBPM, while the remaining five are a consequence of a fully active S-DWH approach; they are:

1. S-DWH Management

2. Data Capturing Management

3. Output Management

4. Web Communication Management This includes for example management of a thematic web portal.

5. (Business) Register Management (or for institutions or civil registers)

The WP produced a functional diagram for operational over-arching processes, contextualizing the nine phases of the GSBPM as arrows between modules in an S-DWH functional diagram.

Models & Tools

There is a great variety of models and tools that can be used to support the creation of a S-DWH:

Generic Statistical Business Process Model (GSBPM)In order to treat and manage all stages of a generic production process it is useful to identify and locate the different phases of a generic statistics production process by using the Generic Statistical Business Process Model (GSBPM).

Generic Statistical Information Model (GSIM)Another model used for describing statistical processes is the Generic Statistical Information Model (GSIM), a reference framework providing a set of standardized, consistently described information objects, which are the inputs and outputs in the design and production of statistics. GSIM is intended to support a common representation of information concepts at a “conceptual” level.

CORE There are many software models and approaches available to build modular flows between layers. One of the approaches is CORE (Common Reference Environment), which is an environment supporting the definition of statistical processes and their automated execution. CORE services can be used to move data between S-DWH layers and also inside the layers between different sub-tasks.

The Integrated Warehouse model The Integrated Warehouse model combines technical and process integration with the warehouse approach into one model. To have an integrated warehouse centric statistical production system, different statistical domains should use a common methodology, share common tools and have a distributed architecture. Decisions in the design phase, like questionnaire design, sample selection, imputation method, etc., are made “globally”. This way, integration of processes provides reusable data in the warehouse. The warehouse contains each variable only once, making it easier to reuse and manage valuable data.

Handbook to set up a S-DWH 11version 0.4 / 19 September 2013

There is also a big variety of software tools used for statistics production. Which tool to choose mainly depends on the NSI’s possibilities to adopt a particular technology, what tools are already used, which skills and experiences are available, as well as other considerations and available resources. In the interpretation and source layers standard tools can be used out-of-the-box, even though they are not generally very customizable to adapt to statistical processes. In the Integration layer, where all operational activities needed for the statistical elaboration processes are carried out, mainly in-house developed software is used. This is because the needs are very specific and cannot be covered by standard applications. In these cases sharing of experience between NSIs is very desirable as it avoids unwanted duplication of work and allows using the experiences already acquired.

Deliverables of the technical work package:

3.1 Business Architecture of the S-DWH

3.2 Modular Workflow of the S-DWH

3.3 Functional Architecture of the S-DWH

3.4 Overview of various technical aspects

3.5 Relate the 'ideal' architectural scheme (from 3.3) into an actual developmentand implementation strategy.

Handbook to set up a S-DWH 12version 0.4 / 19 September 2013

5. The Road Map for setting up a S-DWHAfter illustrating and explaining the 5 phases and the 3 tracks for setting up a S-DWH, in this chapter a roadmap is given, explaining which general steps to take and what deliverables of the ESSnet on DWH to use in which step(s). The results and deliverables of the ESSnet mainly focus on the design, build and implement phases, as these are the essential phases in developing a S-DWH. The (approved) ‘business case’ is seen as a required precondition for even starting the actual process whereas the ‘use and maintain’ phase is the actual operational phase.

For this purpose we use a graphical representation based comparable to an underground map.

The first map gives a general overview from start to end. The S-DWH development process is represented by 1 single line with the most essential ‘stops’:

1. The approved business case, the official ‘GO’ to start the S-DWH project.;

2. The approved designs of the various components of the S-DWH(business architecture, meta model, etc.);

3. A set of tested and approved systems, representing the working S-DWH(but not yet implemented);

4. The operational S-DWH, in use to produce statistics.

The 3 main phases are then worked out in detailed maps that show the essential milestones/steps, represented as a ‘station or stop’. The with stops are specific for the S-DWH development process. The grey stops are generic stops, like ‘testing’, ‘training users’ etc.

All the each specific S-DWH stops are linked to the deliverables to be used in that stage of the S-DWH development process.

In these detailed sub maps the 3 tracks are represented by collared lines:

the green line represents WP1 – Metadata

the blue line represents WP2 – Methodology

the red line represents WP3 – Technical aspects

Furthermore there is a continuously - - - grey line - - - running through each phase and emphasising the importance of good documentation, not only during the development process, also in the operational phase.

Finally, the 3 detailed maps are combined into 1 total map. Though this map is not so easy to read at first sight, it clearly illustrates the complexity of S-DWH development process.

Handbook to set up a S-DWH 13version 0.4 / 19 September 2013

5.1 Roadmap S-DWH: General overview

Handbook to set up a S-DWH 14version 0.4 / 19 September 2013

5.2 Roadmap S-DWH: Design phase

Handbook to set up a S-DWH 15version 0.4 / 19 September 2013

5.3 Roadmap S-DWH: Build phase

Handbook to set up a S-DWH 16version 0.4 / 19 September 2013

5.4 Roadmap S-DWH: Implementation phase

Handbook to set up a S-DWH 17version 0.4 / 19 September 2013

5.5 Roadmap S-DWH: Total map

Handbook to set up a S-DWH 18version 0.4 / 19 September 2013