data warehouse federal department of home affairs fdha federal office of meteorology and climatology...

Data Warehouse

Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss

The MeteoSwiss Data Warehouse

Real-time access to data and metadata – basis for the WIS-implementation in Switzerland

WMO RA VI Workshop on the implementation of WISSofia, 1-3 November 2011

Stephan Vogt, Meteorological Data Coordination, MeteoSwiss

[email protected]

2 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt


Agenda

• Situation in 1999• Towards the MeteoSwiss Data Warehouse• Metadata in the DWH• Ongoing improvements• Considerations on the WIS-implementation



Situation in 1999 – and the visionYear 2000 Problem: maximum credible data accident to be expected?



Objectives of the MeteoSwiss Data Warehouse

• Implementation of a central, integrated data platform for the collection, storage and processing of meteorological data and its metadata.

• Storage of metadata in one central metadata repository

• Subdivision in a processing and an application area

• Creating a basis for a customized access to data respecting official standards for internal and external clients



2000

2001

2003 Phenology & Pollen

2002 Surface data (rain gauge measurements, AWS data, observations)

2004 Aviation messages

2005 Lightning data

2006 International surface data (SYNOP)

2007 National sounding measurements

2008 Homogenized Data

2009 National & International Upper Air Data

2010 Gridded Datasets

Start of the MeteoSwiss Data Warehouse Project

Model Data2011

Metadata



Conceptual Architecture

Dat

a so

urc

esS

tagi

ng

area

D

ata

stor

age

Database for point data analysis

Calc, Check, Data Transformation & Integration

Data delivery and products (on-/offline)

Weather Forecast

Data for the Public Internet

Durable data archive

Database for point data processing

(incl. raw data repository)

Light-ningdata

RadarClimaAnalysis Tools

Model Validation Research ClimatologyClimate Information Service

Dat

a an

alys

is

Autom.Observ.systems

UpperAir data

Aviationdata

Geodatabase(raster data, ...)

Data analysis & Interfaces (API, DB Tools, Web access, GIS, R …)

VERACMSAFNWP

TL

PartnerObserv.systems

QC

Met

a D

ata

Obser-vations

Met

adat

a M

anag

emen

t Too

ls



Metadata in the MeteoSwiss DWH

…stored in more than 200 tables in the DWH



Enhanced DataClean Data

customers

Raw Data

level 2data

level 1data

level 3data

level 5data

level 4data

quality control II calculation of

derived quantities

data collection quality control I

measurement

simulation

metadata

Raw data repositor

y

Data repositor

y for external

data

applications for data distribution

data

sou

rce

quality control III errorr correction

extreme value messages

imm

ed

iate

ly

days

years

exte

rnal

data

sou

rce

applications for data processing

immediately

durable data

archive

MeteoSwiss data archives

durable data archives outside MeteoSwiss

MeteoSwiss data sources

data sources outside MeteoSwiss

data transmission and distribution

Legend

Level 1 measurement data

Level 2 raw data

Level 3 cleaned data

Level 4 corrected data

Level 5 homogenized data

homogenization

Instrument Lifecycle

Spatial reference

Data import

AggregationcalculationContract

data

Station history

Homo-genization

Quality control

Parameters



Development cycle & metadata management

Development

Deployment

Production

Failover Production

Master Metadaten Repository



How are the metadata accessed?

• DWH-Webtool with user guidance for all important use cases

• MS ACCESS for all other casesKontext-

datenbankDWHPROD

Kontext-datenbankDWHFOPR

Kontext-datenbank

PMETAREP

Intranet

WebLogic

KDB-Tool 2

SecPro

Client Internet Browser

Benutzerinteraktion

Zugriff Security Provider

Zugriff Kontextdaten

Active Directory(LDAP)



Look and Feel of DWH-Webtools



Advantages of the Data Warehouse

• measurement data• metadata • control data• quality tests

Allows an efficient and economic maintenance

company-wide data integration platform for the long term

independant from technical realisation

specified in a uniform way, centrally administrated and configurable



Lessons learned

Do‘s Dont‘s

Listen to the data; conduct broad-based reviews

Listen too much to single data users

Follow a spiral development model, gain experience in operation, grow the infrastructure accordingly

Do everything in one step waterfall procedure

Follow the golden rules of database design

Fix technology too early

Understand the business process Don‘t let consultants talk you into a solution

Provide one attractive application for data analysis from the very beginning

Segregated architectures



Do things the same way all over the organisation data governance

• Data architecture documents• Gremium with representatives from all MeteoSwiss

departements



Not everything is solved...

• RENAISSANCE = Renewal and ENhAncement of the servIceS baSed oN the mEteoswiss data warehouse

• Subprojects:• Data supply of COSMO from DWH• Generic delivery mechanism for data marts• Station history• Consolidation of production tools



Generic delivery mechanism for data marts• Existing delivery mechanisms have been generalized• Towards event-based triggering of processes• Generic delivery mechanism includes

• Measurement data• Metadatabased on XML standard

Primary purpose: Data delivery to GIN (Swiss joint Information Platform for Natural Hazards)



Lots of tools for lots of products…



…and what about WIS?

• All remarks so far refer to our status as NC• MeteoSwiss has been asked to develop GAWSIS as a

DCPC for WIS • Today GAWSIS is not yet integrated into the DWH• Two options for the next steps:

• solution on the basis of the current GAWSIS Provide the data with an OAI-Service ready for collection by GISC

• Integration of GAWSIS into the DWH, make use of the planned Web Service-infrastructure



Vision regarding WIS



Vision regarding WIS

NC

MeteoSwissData Warehouse

Internet

GISC

XMLMetadata



Questions?

Thanks for your attention!

[email protected]

data warehouse federal department of home affairs fdha federal office of meteorology and climatology...

Documents