copyright jet propulsion laboratory1 object oriented data technology dan crichton steve hughes...

Copyright Jet Propulsion Laboratory 1

Object Oriented Data Object Oriented Data TechnologyTechnology

Dan Crichton

Steve Hughes

JPL/Caltech

CCSDS XML Workshop

Oxford, England

April 3-5, 2002


Overview

Funded in 1998 by the NASA Office of Space Science to research technology in middleware and distributed object computing architectures to enable interoperability of data management systems


OODT Focus Areas Information Architectures

– Metadata Management– Data Languages (i.e. XML)

Distributed Architectures– Messaging– Distributed Object Computing– P2P Architectures

Development of an architecture for distributed information management and retrieval

Application to the Planetary Data System to support Mars Odyssey Data Distribution


Middleware

Web, GUI, console,2D/3D visualization,immersionproduct data,

real-time data

Applications

s/w, h/w,models

Middleware

Dat

aU

ser Interface

N. Lamarra


Middleware Review History

– Pre-web: IDE (dev), client/server (apps)– Post-web: n-tier c/s, thin/thick, frameworks, …– Communications: RPC, MOM, Web , …– Applications: Desktop and Remote, …– Architecture: Open Reference Model, HLA, TOGAF, …

Standards– O/S: POSIX, Unix, Windows– Communications: TCP/IP, OSI, DCE/ONC, HLA/RTI, HTTP– DOC: CORBA/COSS, DCOM/COM+, Java RMI– Data access: O/JDBC, EJB persistence, STEP, HTML, XML– Security: SSL, SSH, SHTTP, IPSec, PKI, CDSA, BioAPI, etc.– Directory: X.500, LDAP– Systems Management: CIM, SNMP, CMIS/CMIP– GUI: X11, CDE, Java-Swing


OODT Principles

Location Independence Information Hiding Model driven architecture Scalable, Extensible Client APIs for locating and accessing

distributed data products


Model Driven Architecture

Decouple solutions from platform specific technology

Allows data architecture and technology architecture to evolve independently

The technology architecture supports the data architecture (i.e. it supports the model)

Apply architecture to different problem spaces


OODT Architecture

Address two key architectural focuses– Technology Architecture

• Basic communication between geographically distributed data systems

• Framework for plugging in new data systems

– Data Architecture• Standard method for describing data (profile)

• Standard method for exchanging data (XMLQuery)


OODT Standards The architecture for accessing distributed resources is based on

the following standards:– W3C Extensible Markup Language (XML) 1.0 (Second Edition)

(Recommendation 6 October 2000)– W3C HTML 4.01 Specification (Recommendation 24 December

1999)– W3C XHTMLT 1.0: The Extensible Hypertext Markup

Language (Recommendation 26 January 2000)– ISO/IEC 11179 - Specification and Standardization of Data

Elements, Parts 1-6, ISO/IEC specification, http://www.iso.ch/iso.

– Dublin Core Metadata Initiative. The Dublin Core Element Set Version 1.1, Dublin: DMCI, July 1999.


OODT Technology Implementation

Implemented in Java Data layer implemented with XML Uses CORBA for remote object invocation

– We use JacORB SSL for data encryption Uses a standardized XML DTD (XMLQuery) as the

messaging format (See http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd)

Supports a variety of client access methods– Java API– HTTP– C++ API


Why XML? XML doesn’t provide a “silver bullet,” but it does allow us

to refocus the problem on metadata– Metadata is a key to interoperability

XML is language neutral Allows the designer to separate the data and the transport

(re: CORBA vs XML-over-CORBA)– Transport mechanism and data are not tied together

• Could be XML/HTTP• Simpler deployments• Simpler interfaces• Allows technologies to grow and change independently

Real value of XML is providing a content language for documenting the information architecture


XML and CORBA XML over CORBA/IIOP module jpl {

module user { interface UserManager { string do(string xml);

}; };};

<transaction> <findUser> <user> <surname>Doe</surname> </user> </findUser></transaction>

CORBA method

module jpl { module user { interface UserManager { User findUser(string

name); }; interface User { String getName(); }; };};


OODT Data Architecture

Use Profiles to describe data resources– System Profiles– Dataset Profiles– Product Profiles

A profile is just a metadata description of the resource defined by an XML DTD (see oodt/dtd/prof.xml)– Uses Dublin Core – Uses ISO 11179


What is a profile?

• Profiles describe and provide the location for anything of interest– Things of interest (data and system resources)

• Data Systems, Data Sets, Web Sites, etc

– Written as XML documents– Grouped

• Data Product profiles grouped by data set• All others grouped into single database

– Provide sufficient information to determine if the resource can resolve a user query


Profile Overview Implementation

A profile includes a pointer to the resources

A profile has three main sections. The profAttributes (Profile Attributes) section describes the

profile. The resAttributes (Resource Attributes) section describes

the resource by using a common set of keywords (Dublin Core)

The profElement (Profile Element) section describes the resource by using domain specific attributes (PDS keywords)


Profile DTD<!ELEMENT profiles (profile+)>

<!ELEMENT profile (profAttributes, resAttributes, profElement*)>

<!ELEMENT profAttributes (profId, profVersion*, profTitle*, profDesc*, profType*, profStatusId*, profSecurityType*, profParentId*, profChildId*, profRegAuthority*, profRevisionNote*, profDataDictId*)>

<!ELEMENT resAttributes (Identifier, Title*, Format*, Description*, Creator*, Subject*, Publisher*, Contributor*, Date*, Type*, Source*, Language*, Relation*, Coverage*, Rights*, resContext*, resAggregation*, resClass*, resLocation*)>

<!ELEMENT profElement (elemId*, elemName, elemDesc*, elemType*, elemUnit*, elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*, elemSynonym*, elemObligation*, elemMaxOccurrence*, elemComment*)>


?xml version="1.0"?>

..

<!DOCTYPE profiles SYSTEM “http://ginger.fhcrc.org/dtd/prof.dtd">

<profiles> <profile>

<profAttributes>

<profId>1.1.1.1.102</profId>

<profType>profile</profType>

<profStatusId>ACTIVE </profStatusId>

</profAttributes>

<resAttributes>

<Identifier>EDRN_MOFFITT_PRODUCT_SERVER</Identifier>

<Title>EDRN H. Lee Moffitt Cancer Center Product Server</Title>

<Format>text/html</Format>

<Description>A product server providing access to a database of clinical sample information for the Early Detection Research Network (EDRN) at the H. Lee Moffitt Cancer Center.</Description>

<Language>en</Language>

<resContext>NIH.NCI.EDRN.MOFFITT</resContext>

<resAggregation>granule</resAggregation>

<resClass>system.productServer</resClass>

<resLocation>iiop://oodt.jpl.nasa.gov:10000/NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER</resLocation>

</resAttributes>

<profElement>

<elemId>STUDY_PARTICIPANT_ID</elemId>

<elemName>STUDY_PARTICIPANT_ID</elemName>

<elemDesc>EDRN Participant ID</elemDesc>

<elemType>CHARACTER</elemType>

<elemEnumFlag>F</elemEnumFlag>

</profElement>

<…>


Adopt an XML Query Structure for Messaging

Support queries across distributed resources

Provide a standard for inter-communication between heterogeneous applications and computing environments

Describe both queries and results


XMLQueryImplemented in XML at http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd

Defined as followsThe query description

The resultsResult 1

Result Header

Result Data

Result 2Result Header

Result Data

...

Result NResult Header

Result Data

Final results are aggregated from all profile/product servers


XML Query Example (1 of 2)<query> <queryAttributes> <queryId>1.2.3.4…</queryId> <queryTitle>OODT_XML_QUERY Query Example</queryTitle> <queryDesc>Query for TARGET_NAME = MARS</queryDesc> <queryType>QUERY</queryType> <queryStatusId>ACTIVE</queryStatusId> <querySecurityType>UNKNOWN</querySecurityType> <queryRevisionNote>2000-05-12 JSH V1.2 Updated for new prof.dtd</queryRevisionNote> <queryDataDictId>1.2.3.5…</queryDataDictId> </queryAttributes> <queryResultModeId>ATTRIBUTE</queryResultModeId> <queryPropogationType>BROADCAST</queryPropogationType> <queryPropogationLevels>N/A</queryPropogationLevels> <queryMaxResults>100</queryMaxResults> <queryResults>0</queryResults> <queryKWQString>TARGET_NAME = MARS</queryKWQString>


XML Query Example (2 of 2)<querySelectSet></querySelectSet> <queryFromSet></queryFromSet> <queryWhereSet> <queryElement> <tokenRole>elemName</tokenRole> <tokenValue>TARGET_NAME</tokenValue> </queryElement> <queryElement> <tokenRole>LITERAL</tokenRole> <tokenValue>MARS</tokenValue> </queryElement> <queryElement> <tokenRole>RELOP</tokenRole> <tokenValue>EQ</tokenValue> </queryElement> </queryWhereSet> <queryResultSet></queryResultSet></query>


OODT Components to Support the Architecture

Name Server Query Server Archive Server Profile Server Product Server Server Manager


OODT Software Components

Name Server– CORBA name server that maps an object to a location (i.e.

ern:NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER can be communicated at edrn.moffitt.edu:7877)

– All implementations of CORBA, DCE, DCOM, SOAP, etc use a name server

– Only needs to be run once at a central site

Query Server– Implemented by OODT – Multi-threaded server for managing queries to distributed servers– Aggregates results from multiple servers– Only needs to be run once at a central site


Archive Server

Provides a server framework for– Adding products remotely to a generic archive– Retrieving products remotely from a generic archive

A generic data model that is driven by the metadata model– Captures a catalog of archived products using well

defined data elements Uses XML to label data products Extract Profiles of archived products


Profile Server Profile Server

– Used to identify the existence and location of resources in a distributed data environment

– Two types of profile servers• Dynamic Profile Server

– Front ends an existing database catalog to extract the resources and format them as profiles

– Implemented as a Java Interface class

• Static Profile Server– Manages its own local resources

– XMLQuery is the messaging structure to query and return profiles

– As many profile servers can be run as needed to scale the system. Typically one per catalog.


Profile Server Overview

• Profile Servers serve profiles– Allow search and retrieval of profiles– Retrieves from profile databases

• Static profiles stored as XML documents• Dynamic profiles generated from information stored in

databases

– Can be distributed for local management and scalability at a site


Profile Server Requirements

1) A profile server shall search and retrieve profiles from a profile database

2) For search, a profile server shall allow any profile attribute as a query constraint. Profile attributes include those from the profile element, resource attribute, and profile attribute sections of the profile document.

3) For retrieval, a profile server shall return matching profiles. The user can request the complete profile or any subset of the profile.


Profile ServerXML Interface

1) Input: XMLQUERY – A profile server accepts an XMLQUERY XML document as input. User query constraints have been encoded into the WHERE clause of the XMLQUERY. The requested profile attribute results have been encoded into the SELECT clause.

2) Output: XMLQUERY – A profile server inserts the set of matching profiles in the RESULTS clause of the XMLQUERY.


Profile ServerServlet Interface

• User calls optional servlet using “DIS” query• Servlet creates XMLQUERY and calls predetermined

profile server• Results returned in XMLQUERY• Query Server passes parallel query to each product

server that has a match

htpp://ginger.fhcrc.org/servlet/jpl.oodt.servlets.Profile_Servlet?

keywordQuery={CANCER_SPECIMEN_TYPE = BLOOD

&object= NIH.NCI.EDRN.DMCC.PROFILE_SERVER

&mimeType=XML

&mimeType=*/*}


Profile Server Distributed Architecture


Product Server

Product Server– Used to return a product using the XMLQuery

messaging structure– Implemented as a Java Server– Contains one or more query handlers

• Query handler implemented as a Java interface class

• Query handlers loaded at server start time

• Defined in a local configuration file

– Can use any method to access and return data


Product Server Implementation The Product Server plugs into the OODT framework and

manages the “handshake” between the data system and the OODT system.

Extensible by dynamically loading objects at runtime which are specific to the data system model

Queries and results are passed using an OODT XML Query structure

Product Server

Flat file accessor

Unusual accessor

Query

Result

Generic Server Implementation Class

FileSys

Database


Profile Server

DistributedProfileServers

Product Servers for Planetary Data System

HTTP, IIOP, Java, C++

APIs

ASU ThemisProducts

jdbc:// jdbc://

Data Source InterfaceFor

Dynamically Loaded Query Handlers

Java Server Framework

PDS.ASU.THEMIS.PRODUCT_SERVER

Mars OdysseyImaging Products

Java Server Framework

PDS.IMAGING.PRODUCT_SERVER

XMLQUERY

Data Source InterfaceFor

Dynamically Loaded Query Handlers

HTTP, IIOP, Java, C++

APIs

XMLQUERY


Layered OODT Product Server Architecture

PC/Sun,etc Hardware

Unix/Windows operating system

Java

OODT

CORBA/SSL

Local Product serversProvided by JPL

Provided by hosting site

Stability


Server Manager

Server Manager– Distributes software– Manages distributed software– Remote Management and Debugging – Automatic process management– APIs to allow tools to inquire about server

status (i.e PDS Themis Node at ASU is up or down)


Challenges with Distributed System Integration

Off site development– Debugging by telephone

New databases and database drivers– Inexperience led to programming errors

Institutional Challenges– Finicky configurations– Mysterious network issues


Possible Solutions

Remote PC control– Unacceptable security risk at some sites

Mandate Unix– Lack of IT experience at member sites– Retooling time too great

Put remote management into the product server itself


Server Manager

Manages servers– Server manager runs at each site

Client/server implementation– XML-RPC with HTTP authentication

One-click installation– Well, almost: install Java first

Capture Log Files Perform restarts


Remote Control Application

Authenticates with a server manager Enables remote administration of server

processes– View log files and process output– Set Java runtime parameters– Update files– Define, start, stop processes– Execute upgrade scripts


Remote Control User Interface


Software Distribution Environment

Installation– First, you install Java– Windows users download and run one self-

installer•pds-setup.exe• Displays typical Windows-style wizard installer

– Unix/Mac users download a tarball and follow instructions in a README file


OODT Configurations

oodtrc.xml– Local configuration file that contains

• Server names

• List of profile servers

• Logging methods

• Timeout methods

• Database configuration parameters

• …


OODT Applied to PDS

Provides a common messaging infrastructure for PDS

Provides data abstraction and location independence for PDS Nodes– Common tools can look across the PDS sites – Heterogeneous implementations are standardized

Uses the PDS Data Dictionary Uses existing PDS databases and catalogs


Generic Architecture

Query Server

Profile Server

Site 3Product Server

XMLQuery

XMLQuery

XMLQuery

XM

LQ

ue

ry

XM

LQ

ue

ry

ProfileServer

Que

ryC

lien

tWeb

se

rver

Plu

gin

s

Web

Ser

ver

XMLQuery

Site 1 DB

Site 2 DB

Site 3 DB

Web I/F

Desktop I/F

XMLQuery

XMLQuery

XMLQuery

XMLQuery

Name Server

ClientEnvironment Distributed Repositories




Profile/Location Server

Profile DB

Default DataSet Browsers

Custom DataSet Browsers

Middleware

PDS-DistributionDelivery - D01Architecture

Single Point of EntryData Set View

Data Set SelectionPage

Data SetCatalog

Other NodeResources

ProductServer

GRS GRS RS

ProfileServer

RS

ProductServer

THEMIS THEMIS

ProductServer

SPICE

MARIE

Custom DataSet BrowsersCustom Data

Set Browsers


OODT Projects

PDS D01 will use OODT to support Mars Odyssey Distribution (October 2002)

PDS D02 will rollout OODT to all of PDS OODT building a data architecture for the

National Cancer Institute’s (NCI) Early Detection Research Network integrating up to 40 sites by 2005

OODT Supporting development of a JPL Institutional Data Architecture with Data Management Services


OODT References

OODT Overview– http://oodt.jpl.nasa.gov

OODT Papers– http://oodt.jpl.nasa.gov/doc/papers

http://oodt.jpl.nasa.gov/

http://oodt.jpl.nasa.gov/doc/papers

copyright jet propulsion laboratory1 object oriented data technology dan crichton steve hughes...

Documents

data encryptionuses

data profilestandard

planetary data system

modelapply architecture

distributed resources

specification recommendation

research technology

isoiec specification