International HouseholdSurvey Network (IHSN)

Microdata Management Toolkit

Trevor Crofttcroft@blancroft.com

MICS3 Data Archiving, Disseminationand Further Analysis WorkshopGeneva - November 6th, 2006

Original presentation by Pascal Heus, pascal.heus@gmail.com

IHSN Background International Household Survey Network

Marrakech Action Plan for Statistics (MAPS) (Feb 2004) Established in September 2004 International organizations actively sponsoring household surveys +

NSO Coordination: ILO, DFID, Paris 21, UNICEF, UNSD, WHO, World Bank http://www.surveynetwork.org

Survey under-use: limited access, poor return on investment, limited impact, difficulties in policy making

Common obstacles: quality, technical capacity, management / political / legal / ethical issues

Common problems: Accessibility, Timeliness, Coherence, Lack of metadata / documentation / data, poorly organized archives

IHSN Activities

Coordinating survey programs Web based information on planned surveys

Harmonizing concepts & methods Planning, sampling, questionnaire design,

processing, data disclosure & confidentiality, etc.

Maintaining a survey catalog Web based central catalog

Developing data dissemination tools Microdata Management Toolkit

IHSN Website

www.internationalsurveynetwork.org

Microdata Management ToolkitTools to facilitate archive and dissemination of surveys

Toolkit Requirements User friendly software for microdata Facilitate metadata exchange (DDI, Dublin Core) Facilitate archiving (metadata and data, quality

control) Facilitate preservation/dissemination: network,

CD / DVD, web sites Works with common data formats Multilingual support Free or Inexpensive Availability of technical support and training Supported by national, international and research

communities

DESCRIPTIVEINFORMATION

RESOURCE DESCRIPTIONFRAMEWORK

W3C CONSORTIUM

RSS NEWS

XML TAGS

MARKUP DOCUMENT

What is metadata?

eXtensible Markup Language XML, XSL, DTD, XSD,

SOAP, XQUERY,…

METADATAEXCHANGE

SEMANTIC WEBWEB 2.0

STANDARDSPECIFICATIONS

EASY TOTRANSFORM

HUMANREADABLE

SEARCHABLE

MACHINEPROCESSABLE

WEB SERVICES

ONTOLOGIES

What is metadata?

“DATA ABOUT DATA”

What is metadata?

Unlabeled stuff Labeled stuff

The bean example is taken from: A Manager’sIntroduction to Adobe eXtensible Metadata Platform, http://www.adobe.com/products/xmp/pdfs/whitepaper.pdf

Source: Courtesy of Jostein Ryssevik, Nesstar Inc.

Metadata – user view

HTML

PDF

International Standards Data Documentation Initiative (DDI)

Surveys (description, files, variables) Microdata Management Toolkit, DevInfo

Statistical Data and Metadata Exchange (SDMX) Time series, indicators DevInfo5

Dublin Core (DC) Documents, images, maps, other resources

Microdata Management Toolkit, DevInfo5, CDS/WinISIS

ISO 19115 Geographical Information Systems (GIS) DevInfo5

ISO 11179 Registries, classifications, life cycle future catalog tools

Important to adopt and to promote use

Generate HTML based CD-ROM

Import metadata and prepare CD-ROM

Import data and compile metadata

The Toolkit Process1

2

3

Toolkit Components Archiving: Metadata Editor (World Bank / Nesstar Ltd.)

Compile survey data, documentation and metadata in a standard format (Nesstar/DDI). Free data reader for users.

Built on Nesstar Publisher

Dissemination: CD Builder (World Bank / Mark Diggory) Facilitate the publication of survey data, documentation and metadata

on CD-ROM and on the web (transforms DDI into HTML based navigation)

Based on Eclipse Platform, open source

Nesstar Explorer Free software Access metadata & data and export to common statistical formats

IHSN Tools Reporting, diagnostics and other utilities

What is the Metadata Editor?

DDI / Dublin Core specialized editor Template driven Import/Export common data files Integrated interface, multilingual support Metadata and data in single file Export to DDI / DC Licensing agreement

for developing countriesand IHSN members

What is the Metadata Editor?

What is the CD Builder? Publish survey

metadata, documents and data on a CD-Rom or web site)

Transforms DDI into an HTML based interface

User can customize the layout (branding) and content of the CD (single or multi-surveys)

Based on DDI / Dublin Core Integrates with Metadata Editor

CD Builder ProcessCreate new CD-ROM

Project

Create new CD-ROM Project

Add a survey to the project and select its

type and branding

Add a survey to the project and select its

type and branding1

2• Selecting a survey consist in opening the DDI-XML or Nesstar file• The survey “branding” determines the overall look and feel of the CD• The survey “type” determines the default metadata content

• Selecting a survey consist in opening the DDI-XML or Nesstar file• The survey “branding” determines the overall look and feel of the CD• The survey “type” determines the default metadata content

Click the “Save” button to generate the HTML

interface

Click the “Save” button to generate the HTML

interface

3After a few minutes,

your CD Project is ready for publishing!

After a few minutes, your CD Project is ready

for publishing!

4

Sample output

Nesstar Explorer

Free software PDF philosophy Access to survey metadata Access to data (no need for specialized

software) Export to common formats Single file holds data and metadata

IHSN Tools

Free / Open Source Utilities to

complement existing software

Integrates in Editor Diagnostics Reporting Metadata sharing

Benefits and Users Improves and facilitates documentation

preservation, cataloguing, metadata exchange, dissemination, quality, etc.

Users: survey producers, survey sponsors, data archives

Benefits: data producers, national & international survey sponsors, survey data repositories, data analysts, policy makers, DDI Community

Status & Availability Metadata Editor:

Available in English, Commercial release in Nesstar 4.0 (end 2006?), testing French, Spanish & Russian

CD Builder Available in English & French, testing Spanish & Russian

IHSN Tools Diagnostics reports available, Survey Documentation in

PDF Other

I18n: working on French, Spanish, Bosnian, Russian, Portuguese,…

User’s Guide in English, draft in French, Spanish, Russian Open source project components hosted on SourceForge

at http://www.sourceforge.net/projects/toolkit

Roll-out Roll-out in in 2006/2007 (global) Completed training / pilot in about 20 countries,

mainly in the Africa region Use by UNICEF for Multiple Indicators Cluster

Surveys (MICS) Asia: Partnership with United Nations Economic and

Social Commission for Asia and the Pacific (ESCAP) Latin America: partnership with Inter-American

Development Bank (IADB) and Pan-American health Organization (PAHO)

Used by IHSN member agencies (WHO, ILO, etc.) Component of World Bank Accelerated Data

Program (ADP)

Thank you!

QUESTION / ANSWER