may 2003national coastal data development center brief introduction two components data exchange...

May 2003National Coastal Data Development Center Brief

Introduction

Two components•Data Exchange Infrastructure (DEI)•Spatial Data Model (SDM)

Together, they make up “NCDDC Middleware”


Introduction to NCDDC Middleware - what it does and why

Richard Owens


What Is DEI (High-level overview)

DEI is a client/server framework by which applications and data repositories can interact. The goal of the framework is to solve the problems of Data Access and Data Representation. DEI is a CORBA-based set of interfaces which are used to locate and transfer geospatial data. It is a client/server architecture which uses CORBA's IIOP (uses TCP) as its transport protocol.


DEI Concepts

Gateway• Server component that adapts DEI interfaces to a specific

data source Data source dependent

DEI Client• Any software that uses DEI CORBA interfaces to

communicate with gateways

Client Gateway DataCORBA


DEI Concepts (continued)

• Data Flavor (formats) Like a MIME type


What DEI Is Not

Complete

Web services

• Lacks generic query processing• Poor support for very large datasets

Data Archive/StorageData Format

• Supports them, though


HistoryOriginally Navy NRL funded projectNow being developed by NCDDC

Current StatusJava based gateways via Java GDK

Current focus

C++ gatewaysOriginal focus, now lagging behind Java

FutureMature Java GDK, then C++ GDKAdd generic query processingAdd support for very large datasetsOpen source


What is the SDM

• Object-oriented library that is used to represent and manipulate spatial data objects.• 3D

Some other dimensional capabilities, too

Structured Grids Regular, Rectilinear, Curvilinear, Projected

Unstructured GridsCoordinate Systems

Cartographic, Cartesian, Cylindrical, Spherical GIS-like structures

Points, Lines, Polygons, etc.


What is the SDM (continued)

• Java~160 classes~1050 Unit tests


Key Features

• Called the “Field Spatial Data Model” by its developer because it uses a ‘field’ concept, which relates spatial objects to data.

• Fundamental storage is multi-dimensional array.

• Decoupling of data storage from its geometry and topology.

• Requires basic metadata (quantity, units, validity, accuracy, etc.) for all data.

Aids in interoperability


Small, consistent API for data access.

• The SDM hides all implementation details of data representation behind its abstract interfaces.• Decouples many issues regarding storage and internal representation of the data, while providing a consistent interface to it.• Data may be logically represented one way and physically

stored another way.• Many manipulations may be done without actually having to have the data physically in memory.

Key Features (continued)


Externalized format for representing datasets

• XML based (FDML)• Can be generated without the use of the SDM.

Key Features (continued)


Field Concept

ARRAY

MESH FIELD DATA

ARRAY

DATA


What SDM Is Not

Transport mechanismStorage/ArchivalMagic. Won’t solve all problems.Just a data formatComplete


History

EPA projectPorted from Eiffel to Java in 2002


FutureDocumentationFY03

•Text data (done)•Generalize dimensional support beyond existing 3-D

(to 2-D, 1-D or N-D)•Investigate Time dimension support (complete in FY04)•Application Layer Services

Common Query mechanismSubsetting (parameters, spatial)Format support (NetCDF, Shapefile)

FY04Aggregation of data typesTime dimensions support - including subsetting in

Application LayerOpen Source


NCDDC Middleware Objective

Develop a middleware architecture that can be used to tie data and applications together across a local or wide-area network in a way that is independent of the data and application platforms, operating systems, and formats.


NCDDC Middleware Defined

“NCDDC Middleware” is the fusion of DEI with SDM

DEIData Transport

SDMData Representation

DEI + SDM (“NCDDC Middleware”)• Spatial data representation distributed over a network• Common form for data, and the means for transporting it


DEI/SDM, HTTP/HTML Analogy

TransportHTTP, DEI

ContentHTML, SDM


Benefits• Gateways can be created that can transport descriptions of

the data with embedded links to the actual data – the actual transport of the data is done piecewise, “on demand”.

Can have datasets that are larger than memory. Caching mechanism improves performance.

• Data is transported in binary form Sun’s XDR format

Foundation for netCDF Fast and efficient

• Since DEI is integrated within the SDM, clients can use SDM interfaces to access gateways without the need to code to DEI’s CORBA interfaces.

may 2003national coastal data development center brief introduction two components data exchange...

Documents