sdmx interface for ilostat
DESCRIPTION
ILOSTAT, the new database of labour statistics, has been designed based on a number of key ideas among which the aim to reduce the overburden to data providers by supporting as many data channels as possible, to be metadata driven, and to adopt every possible standard, played a fundamental role. With these in mind, we developed a bi-directional interface to allow the dissemination and collection of data and metadata from and to ILOSTAT through SDMX datasets and related artefacts. The implementation project had to get over several issues, especially on the conceptual side. In this presentation we are going to see how the Software architecture for the interface was defined, the concepts that conforms the ILOSTAT concepts scheme, how it deals with the Descriptive metadata, a crucial resource in ILOSTAT, the definition of the scope of the DSD, with its pros and cons, and the implementation of a virtual registry and versioning system. (Presented at SDMX Global Conference 2013, Paris)TRANSCRIPT
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
SDMX INTERFACE FOR
ILOSTATILO Department of Statistics
Edgardo [email protected]
1
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
I. IntroductionII. DesignIII. Software ArchitectureIV. Data Collection &
Dissemination V. Next Steps
Agenda
2
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
SDMX has been “around” since 2002LABORSTA’s information model drawbacks
Lack of resourcesWaiting for the standard to mature
ILOSTAT design in 2010New information model following SDMX COG
SDMX included as part of ILOSTAT project
ILOSTAT development in 2011SDMX interface for data collection and dissemination
Introduction
3
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
ILOSTAT modulesData collectionData cleaning processData disseminationWorkflow controlMetadata
ILOSTAT
4
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
ILOSTAT concepts schemeDimensions
CollectionCountryFrequencySurveyRepresented Variable
(OBS_VALUE)Classification Type (1..6)Time
AttributesNote TypesValue StatusUnit of measureUnit multiplierTime format
Design
5
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
Data Structure DefinitionScope of the DSD
1. One general DSD?Easy to maintain but huge and volatile
2. One DSD per topic? (~20 topics)Still too big and volatile.
3. One DSD per indicator? (~100 ind - i.e. Employment by sex and age)OK for disseminationToo many useless entries in country specific code lists
4. One DSD per Questionnaire table (indicator + country)?OK. But …
How to maintain ~100 ind x ~200 cou =
Solution: Virtual Registry & Versioning module
Design
20.000 DSD’s
6
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
Virtual RegistryKey factors:
1. ILOSTAT is metadata driven2. ILOSTAT Information Model is very similar to
SDMX Information Model
All SDMX artifacts considered as «virtually» existing.
The SDMX connector creates and delivers «on-the-fly» any requested artifact
Design
7
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
VersioningAutomatic for data structures and related data
flowsVersion increases with any change in the
structural metadata (code lists, classification versions, required notes, etc.)
Process: The data structure is generated with the default 1.0 version and full
references The result is serialized to an in memory buffer and a SHA1 hash
computed The hash result is compared to the data stored in the database:
• If no existing hash exists, the new hash is stored and the version initialized at 1
• If the hashes are equal, the current version is returned• If the hashes differ, the version is incremented and the new
hash stored The generated version number is passed to the actual structure
generation process, to be included in the returning flow
Design
8
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
Descriptive metadata (metacontent)ILOSTAT includes many notes at different
levelsAll the notes are coded and classified by
Note_TypeAvoided MSD usage for simplificationNotes are included in the DSD/DF as coded
attributesAttachment level:
Currently: All notes attached at Observation_value level. Actual level determined by attribute name.
Future: Notes attached at the proper level (format change req.)
Only for collection: Special “Free_Text” note type allow for capturing non-
coded annotations
Design
9
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
Software ArchitectureJava EE application based on the following components:
SDMXsource Oracle Application Development Framework (ADF) ILOSTAT Taskflow Library (also used for the ILOSTAT
Website)
10
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
Dissemination Standard SDMX RESTful
API (partial)http://www.ilo.org/ilostat/sdmx/ws/rest/...
Collection Triggered by an APEX
interface for a given file
Data Collection & Dissemination
11
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
Set up provision agreementsInternational OrganizationsCountries’ NSO & MoL
Develop new interfacesJSONSDMX 2.1SDMX-RI Gateway
End-user access toolsILO Information & Knowledge Management
GatewayILOSTAT country profile reportGrapher toolMobileExcel add-in
Next steps
Capacity building + Tools
12
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
E-mail:[email protected]
Skype:egreising
Twitter: egreising
LinkedIn: http://www.linkedin.com/in/egreising
Questions?
13
SDMX Global Conference - Paris, September 2013.SDMX Global Conference - Paris, September 2013.
E-mail:[email protected]
Skype:egreising
Twitter: egreising
LinkedIn: http://www.linkedin.com/in/egreising
Thank you!
14