Agenda Item 3.3

SDMX reference architecture for NSIs

Francesco Rizzo

24th Meeting of the STNE Working Group

“Statistics, Telematic Network & EDI”

16-17 June 2009

Eurostat Unit B5 – Statistical Information TechnologiesSTNE 24th Meeting – 16-17 June 2009

1

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 2

Presentation summary The NSI perspective on SDMX: benefits The NSI perspective on SDMX: how to reduce costs The NSI perspective on SDMX: three important aspects Data Repository (Warehousing) Architecture Data Hub Architecture SDMX reference metadata architecture The Mapping Process Codes mapping: example for frequency The NSI perspective on SDMX: where to start from First of all a quick simple analysis Disseminating and reporting data in SDMX: different scenarios Free/open SDMX software and tools inventory SDMX Reference architecture for MSs The toolkit for participating to an Eurostat SDMX project

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 3

The NSI perspective on SDMX: benefits

Reduce reporting burden to National, European and International institutions

Can improve harmonisation, standardisation and integration processes inside a NSI

be part of an international “community” where NSIs can:– share experiences and best practices– share freely software and tools

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 4

The NSI perspective on SDMX: how to reduce costs

SDMX in general follows the open source culture and, as such, tools used within the SDMX initiative can be made publicly available

Eurostat has been designing a SDMX reference architecture for MSs and developing some building blocks in order to facilitate SDMX implementations

Several NSIs and International Organizations have been producing case studies from their direct experience in implementing SDMX

Eurostat from 2007 has launched a training plan, oriented principally to MSs

Eurostat, upon request, provides technical advice to NSIs interested in starting some SDMX projects

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 5

The NSI perspective on SDMX: analysis factors

Data Repository (Warehousing) Architecture

Data Hub architecture

The Mapping Process

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 6

Data Repository (Warehousing) Architecture

NSI

EurostatPull Requestor

eDAMIS

Data Input

SDMX Registry

Intermediatestorage

Verification /ConversionTo SDMX

Receiveddata in

SDMX-MLLoader

register

Warehousestorage

Eurobase

query

Dissemination

XSL forSDMX-ML

PULL

PUSH

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 7

Data Hub Architecture

NSI

NSI

NSI

NSI

SDMX RegistryRSS / data registration

Dissemination

XSL forSDMX-ML

Data PortalQuery

Data queryResponse

Retrievedataset

cache

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 8

The Mapping Process

Data sets within data producers’ Information System are described using “local” structural metadata (concepts, code lists, formats)

SDMX standards harmonize structural metadata within a statistical community, and describe data sets by DSDs (concepts, code lists, dimensions, attributes, measures, etc.)– SDMX-ML structure files

“local” structural metadata and SDMX structure metadata must be mapped(*):– concepts mapping– codes mapping

(*) see SDMX User Guide page 73

Concepts mapping one concept of the DSD corresponds to a single “local” concept. A

typical example is the measured value in the data provider database that corresponds to the Primary measure in the DSD;

one “local” concept corresponds two or more concepts within the SDMX Structure file. For example in the “local” concept, named Um, there is an element as follows: “one million of Euro”. In the related SDMX structure file it corresponds to two concepts: Unit (Euro) and Unit multiple (one million);

one concept within the SDMX structure file doesn’t correspond to any “local” concept. For example the concept Reference area, in fact that concept is generally not used in a National Organization because is the default;

one concept within the SDMX Structure file corresponds to two or more “local” concepts. For example the Adjustment concept could corresponds to two concepts named DAYADJ (working day adjusted) and SEASADJ (Seasonally adjusted), see Model version 2.1 of pc-axis

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 10

Codes mapping: example for frequency

CODE DESCRIPTION

1 Annual

12 Monthly

365 Daily

4 quarterly

52 weekly

CODE DESCRIPTION

A Annual

M Monthly

D Daily

Q quarterly

W Weekly

H Half-yearly

B Business

SDMX CODE

Proprietary CODE

DESCRIPTION

A 1 Annual

M 12 Monthly

D 365 Daily

Q 4 quarterly

W 52 weekly

H Half-yearly

B Business

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 11

The NSI perspective on SDMX: where to start from

Decide to start in using SDMX autonomously– Design and build “unilaterally” DSDs and/or reuse those already

available at European and International level– Decide which part of the Information System will be affected

(collection, processing, analysis, dissemination) and which kind of SDMX architecture would be more suitable

Join SDMX projects launched by International organizations– Several pilot project launched by Eurostat within ESS (SODI,

Census Hub, EuroGroup Register, etc.)– DSDs defined centrally by Eurostat after agreements taken within

WG and TF– The SDMX architecture implemented in the NSI must be

compatible with the reference architecture of the whole project

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 12

First of all a quick simple analysis

Which statistical domains are involved Where data and structural metadata are currently stored How the involved data are currently disseminated,

exchanged or reported? What is the business process behind the exercise? Will the new SDMX architecture be part of a data

warehouse, a Hub or of both reference architectures?

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 13

Disseminating and reporting data in SDMX: different scenarios

Starting point Action Comments

1 Files: Excel, CSV, Gesmes, etc.

Convert in SDMX-ML data files and:a. push the files to the data collectorb.store those files on a web server and notify to the data collector the URL to pull the files

Low development costHigh production costLow mapping process cost

2 Existing databases

a.extract SDMX-ML data files and:- push the files to the data collector- store those files on a web server and notify to the data collector the URL to pull the files

b.data will be available upon request directly from the database on Internet

High development costLow production costPartial re-using in different databasesHigh mapping process cost

3 Ad-hoc database

As in point 2 High development costLow production costRe-usable for each statistical domainLow mapping cost

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 14

Free/open SDMX software and tools inventory 1/2

Data/Metadata Structure Definition and transformations– SDMX Converter (Eurostat)– Data Structure Wizard (Eurostat)– SDMX Transformation Package (Metadata Technology)– SDMX Authoring Tool (Metadata Technology)– Data Structure Definition Tool (Metadata Technology)– Metadata Structure definition Editor (Metadata Technology)

Implementation of SDMX registry specifications– SDMX Registry (Eurostat) (Metadata Technology) (UNSD)– KeyMaster (Metadata Technology)– Data provisioning (Metadata Technology)– Data Set registration (Metadata Technology)– SDMX Query Tool (Metadata Technology)– SDMX Query Client (Metadata Technology)

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 15

Free/open SDMX software and tools inventory 2/2

Presentation of SDMX-ML data files to users– Business Cycle clock (Eurostat)– SDMX Visualization Tools (Eurostat)– Visual framework (ECB)

Frameworks and toolkits for working with SDMX– SDMX Framework (Istat)– SDMX framework (Ole Sørensen)– The NSI Web Service Prototype (Eurostat)– Data Retriever Building Block (Eurostat)

Mapping tools– SDMX Mapping assistant (Eurostat)

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 16

SDMX Reference architecture for MSs

The architecture represents the syntheses of several experiences worldwide and can be considered not a strict specification rather than a guide or “best practice” document

The main objective is to provide a description/specification of a generalized architecture to be used partially or as whole by MSs interesting in starting SDMX projects

In 2009 Eurostat will develop and sharing two building blocks detailed in the architecture as open source

In 2010 more building block will be developed

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 17

Eurostat Unit B5 – Statistical Information Technologies STNE 24th Meeting – 16-17 June 2009 18

The toolkit for participating to an Eurostat SDMX project

SDMX Structure file XML schema SDMX Generic data file example SDMX Compact or Cross-sectional data file

example Message Implementation Guide RSS2 feed file example Atom feed example Practical Guidelines for the Implementation of web feeds Requirements for pull data transmission approach

Eurostat SDMX Technical Workshop Title

– From the SDMX Information Model to the development of reusable software components

Purpose– The workshop is aimed at software Designers and

Developers and will be organised in several technical sessions, with the main goal of providing the knowhow for starting to design and develop SDMX architectures for data exchange.

When and where– The workshop duration is two full days, beginning Tuesday

22 September 2009 at 09:00 and ending Wednesday 23 September at 17:00, in the Instituto Nacional de Estadistica, Paseo de la Castellana 183, Room number 118 ( First floor), 28046 MADRID