Deutscher Wetterdienst
DWD GISC ImplementationGISC Development Team
GISC Audit 17.09.2010 Page: 2
Agenda Welcome
Architecture– High level Architecture
– Hardware
Performance Tests
International Collaborations and Co-operations
Contributions to WIS
Q&A
GISC Audit 17.09.2010 Page: 3
Design Principles of DWD GISC
Reuse of existing well established software and implementation methodology
Use of established Interfaces – in particular to „critical components“ to allow later exchange if necessary
– Metadata Providing
– Authentication and Authorization
Modular concept to address performance and scalability requirements
Clear separation of responsibilities
Keep things simple
Describe external interfaces
GISC Audit 17.09.2010 Page: 4
DWD GISC - Development In house development team and support of business units
Professional Services – Fraunhofer – research facility
– Con terra – experts on Metadata
Co-operation with companies– ask
• Contributes to the development with 2-3 persons
• Supports WMO events (EC Demo)
– IBL
• Long term partner
• Supports WMO events (EC Demo)
• Face lift of DWD GISC
• Offers professional services (integration and maintenance) for DWD GISC
GISC Audit 17.09.2010 Page: 5
High Level Architecture: Functional Blocks
Security Layer
Monitoring and Reporting Layer
MetadataDatabase
SubscriptionManagement
Web Interfaces– Customer– Operator– Administrator
Data & ProductImport Management
&GISC Synchroniser
DAR Components– Metadata Harvesting– Metadata Providing
Data & Product AccessManagement
24 hCache
WIS Global Ex.
File pools,Archives,Databases
DAR Generator/Importer
SRUSearch and Retrieval by URL
GISC Audit 17.09.2010 Page: 6
Timeline
Project Start
Concept Phase
11/08 01/09 03/09 05/09 07/09 09/09 11/09 01/10 03/10 05/10
VGISC Workshop
G2G com. Workshop
Impl. Start:Web InterfaceSubscription
DAR MetadataGenerator
DAR CataloguePerf. Study
G2G com. Tests
G2G com. Workshop
MoU CMA
VGISC Backup WS
OAI Infrastructure
SRU
OAI & WIS Workshop
Data & ProductManagement
Test
Test
06/10
EC Demo
08/10
OAI Tests
09/10
MetadataStudy
GISC Audit 17.09.2010 Page: 7
High Level Architecture: Functional Blocks
Security Layer
Monitoring and Reporting Layer
MetadataDatabase
SubscriptionManagement
Data & ProductImport Management
&GISC Synchroniser
DAR Components– Metadata Harvesting– Metadata Providing
Data & Product AccessManagement
24 hCache
WIS Global Ex.
File pools,Archives,Databases
DAR Generator/Importer
SRUSearch and Retrieval by URL
Web Interfaces– Customer– Operator– Administrator
GISC Audit 17.09.2010 Page: 8
High Level Architecture: Web Interfaces
GISC– Redesign of WebWerdis
– Navigation structure
– Search functions
• Simple Search
• Extended Search (e.g. Full Text Search)
• Expert Search
– Internationalization (I18N)
– Integrate logging framework (necessary for monitoring)
– Code clean-up
– Entry points for External Application (e.g. Online Resource)
AUST– Redesign of AUST
GISC Audit 17.09.2010 Page: 9
High Level Architecture: Functional Blocks
Security Layer
Monitoring and Reporting Layer
MetadataDatabase
SubscriptionManagement
Data & ProductImport Management
&GISC Synchroniser
Data & Product AccessManagement
24 hCache
WIS Global Ex.
File pools,Archives,Databases
SRUSearch and Retrieval by URL
Web Interfaces– Customer– Operator– Administrator
DAR Components– Metadata Harvesting– Metadata Providing
DAR Generator/Importer
GISC Audit 17.09.2010 Page: 10
High Level Architecture: DAR Generator, Importer and Components DAR Metadata Provider
– jOAI (also used by NCAR/UCAR – reference site)
DAR Metadata Harvester– OAIHarvester2
– Modified Source Code with CMA together
– In house development of a scheduler and workflow component (done by ask)
DAR Metadata Generator– In house development by DWD shared with CMA, Hong Kong, WMO, Brazil, Norway, Australia, South Africa, …
DAR Metadata Importer– In house development by DWD
GISC Audit 17.09.2010 Page: 11
High Level Architecture: Functional Blocks
Security Layer
Monitoring and Reporting Layer
MetadataDatabase
DAR Components– Metadata Harvesting– Metadata Providing
DAR Generator/Importer
SRUSearch and Retrieval by URL
Web Interfaces– Customer– Operator– Administrator
SubscriptionManagement
Data & ProductImport Management
&GISC Synchroniser
Data & Product AccessManagement
24 hCache
WIS Global Ex.
File pools,Archives,Databases
GISC Audit 17.09.2010 Page: 12
High Level Architecture: Delivery Components
Subscription Management– Controller: time and event triggered management
AFD– Workhorse in the background
– GISC-to-GISC communication
– File distribution
– White List
Message Switching System– REST Web-Service to manage and control Moving Weather
– First proposal for generic Interface
GISC Audit 17.09.2010 Page: 13
High Level Architecture: Functional Blocks
Security Layer
Monitoring and Reporting Layer
MetadataDatabase
SubscriptionManagement
Data & ProductImport Management
&GISC Synchroniser
DAR Components– Metadata Harvesting– Metadata Providing
Data & Product AccessManagement
24 hCache
WIS Global Ex.
File pools,Archives,Databases
DAR Generator/Importer
Web Interfaces– Customer– Operator– Administrator
SRUSearch and Retrieval by URL
GISC Audit 17.09.2010 Page: 14
High Level Architecture: SRU
SRU– WMO SRU2JDBC implementation
Cooperation between WMO and DWD available as Open Source
– In house development by DWD: Mapping of CQL (Contextual Query Language) to SQL (done by ask)
– External configuration file for Mapping
• It is easy to add new indices
• Database specific feature (e.g. functions, special operators) are kept outside of the code
– All Queries and Responses are encodedaccording to the SRU-Standard allowingfor both interactive as well as automatedusage – e.g. by catalogue systems
GISC Audit 17.09.2010 Page: 15
Hardware Architecture –Infrastructure
DMZ
INTRANET
Web Cache OAI Infrastructure
Application Server
Adapter &Scheduler
Operations
Application Server
Adapter &Scheduler
OAI Infrastructure
Integration System
GISC Audit 17.09.2010 Page: 16
Performance 05/2009 Performance Study of DAR Metadata Catalogues – Both Systems terraCatalog 2.3
and Geonetworks 2.2 do not fulfill all Performance Requirements
09/2009 CMA and DWD did performance study of GISC-to-GISC communication solutions – JMA Blog solution and AFD
03/2010 CMA and DWD did performance tests of OAI Provider solution
04/2010 INMET and DWD did performance tests of GISC-to-GISC communication solutions – AFD and Moving Weather (IBL)
07/2010 Internal performance tests of DAR Metadata Catalogue and SRU – Lessons learned– Database is hot spot, especially when we talk about Full Text Search
– Oracle standard reports
– Limit the result set – configuration parameter
– Optimize SQL Statements
• Explain plan
• Rewrite SQL Statements
• Indices
– Improved responses for most critical queries by an order of magnitude
GISC Audit 17.09.2010 Page: 17
International collaboration: CMA (GISC) GISC-to-GISC communication
– Synchronization of 24 hour cache
• Extensively tests of JMAs Blog solution
• Extensively tests of AFD
GISC-to-GISC communication– Description for GISC-to-GISC backup based on XML Schema
GISC Audit 17.09.2010 Page: 18
International collaboration: CMA (GISC) DAR Metadata
– DAR Metadata Generator
– Metadata Template
– Request for Clarification paper 11/2009 (CMA, JMA and DWD)
Metadata Providing / Harvesting– Functional tests of standard OAI provider
– Common development of OAI Harvester
– Extensively performance tests for DAR Metadata synchronization
SRU– Mapping indices to DAR Metadata elements
Workshops:– GISC-to-GISC communication workshop 04/2009 at DWD
– GISC-to-GISC communication workshop 09/2009 at CMA
– OAI workshop 03/2010 at CMA
GISC Audit 17.09.2010 Page: 19
International co-operation: JMA (GISC) DAR Metadata
– Design of DAR Metadata
– Request for Clarification paper 11/2009 (CMA, JMA and DWD)
Metadata Providing / Harvesting– Synchronization tests based on OAI
GISC-to-GISC communication– Knowledge exchange JMA Blog solution
– JMA will take XML Schema (based on a proposal from CMA and DWD)as working draft for GISC-to-GISC backup
Workshop– WIS Workshop at JMA 03/2010
GISC Audit 17.09.2010 Page: 20
International co-operation: Brazil (GISC) DAR Metadata
– DAR Metadata Generator
Metadata Providing / Harvesting– Synchronization tests based on OAI
GISC-to-GISC communication– Moving Weather
– AFD
GISC Audit 17.09.2010 Page: 21
International collaboration: EUMETSAT (DCPC) DAR Metadata
– Conceptual design of DAR Metadata
– Knowledge exchange
DAR catalogue– Extensively knowledge exchange
VGISC ITT– Attended as expert
EUMETSAT is connected as DCPC since 23th of August 2010– Test Phase 06/07 2010
GISC Audit 17.09.2010 Page: 22
International co-operation: Hong Kong (DCPC) DAR Metadata
– DAR Metadata Generator
Metadata Providing / Harvesting– Synchronization tests based on OAI
International co-operation: NCAR/UCAR (DCPC)
DAR Metadata– Knowledge exchange
Metadata Providing / Harvesting– Synchronization tests based on OAI
GISC Audit 17.09.2010 Page: 23
International co-operation: VGISC Since 2002 VGISC – Météo France, UK MetOffice, DWD, met.no, ECWMF and EUMETSAT
SIMDAT
VGISC ITT
Several workshops regarding Backup and Recovery
GISC Audit 17.09.2010 Page: 24
Contributions to WIS DWD participates several ETs, ICG, …
Released Metadata catalogue performance study to community Geonetwork improvements
Shared results of DAR Metadata analysis with WIS community
DAR Metadata Generator
Released SRU2JDBC in co-operation with WMO as Open Source http://sru2jdbc.svn.sourceforge.net/
Active member of jOAI community
Contribute experts for WIS Jump Start Offer
GISC Audit 17.09.2010 Page: 25
Q&A
GISC Audit 17.09.2010 Page: 26
Live Demo
http://gisc.dwd.de/