2/15/20052/15/2005 Denise MartinDenise Martin 11

The USACE The USACE DataNetDataNet

Providing efficient data access Providing efficient data access through a netthrough a net--centric frameworkcentric framework

2/15/20052/15/2005 22

AuthorsAuthors

Denise MartinDenise MartinJames StinsonJames StinsonDavid StuartDavid StuartCary ButlerCary Butler

2/15/20052/15/2005 33

AbstractAbstract

The U.S. Army Corps of Engineers (USACE) relies on interactive The U.S. Army Corps of Engineers (USACE) relies on interactive computercomputer--based systems to identify and assess alternatives, make based systems to identify and assess alternatives, make decisions, and solve problems. Data is the principle component decisions, and solve problems. Data is the principle component that drives the decision making process. Much of the data needethat drives the decision making process. Much of the data needed d to support USACE are available from other federal agencies. to support USACE are available from other federal agencies. Acquisition and use of these data is often inefficient and Acquisition and use of these data is often inefficient and inconsistent. Moreover, data are provided in a myriad of disparinconsistent. Moreover, data are provided in a myriad of disparate ate formats and structures while the applications that consume theseformats and structures while the applications that consume thesedata require differing formats as well. The goal of the USACE data require differing formats as well. The goal of the USACE DataNetDataNet is to streamline the acquisition and dissemination of data is to streamline the acquisition and dissemination of data across all USACE business areas. This presentation describes 1)across all USACE business areas. This presentation describes 1)the development and implementation of a common netthe development and implementation of a common net--centric centric framework that provides a consistent interface to data sources framework that provides a consistent interface to data sources internal and external to USACE, and 2) client applications that internal and external to USACE, and 2) client applications that use use the the DataNetDataNet for data acquisition and delivery, including ArcGIS for data acquisition and delivery, including ArcGIS extensions.extensions.

2/15/20052/15/2005 44

ProblemProblem

USACE uses a multitude of heterogeneous USACE uses a multitude of heterogeneous computerized systems to assist in decisioncomputerized systems to assist in decision--making. making. Data is the principle component that drives Data is the principle component that drives the decisionthe decision--making process.making process.The efficient handling of data is critical in The efficient handling of data is critical in making timely, costmaking timely, cost--effective decisions.effective decisions.S&E spend more time acquiring, S&E spend more time acquiring, manipulating, and organizing data than manipulating, and organizing data than performing analyses to support decisions.performing analyses to support decisions.

2/15/20052/15/2005 55

Scientific work flow Scientific work flow

~70%time

~30%time

Distributed Heterogeneous Data Sources

Locate, Assemble and

Load Data

Scientific Simulations and Experiments

Scientific Analysis and Discovery

Data Preprocessing

Locate, Assemble and

Load Data

Scientific Simulations and Experiments

Scientific Analysis and Discovery

~70%time

~15%time

Distributed Heterogeneous Data Sources

Data Preprocessing

~15%time

70-15-15 Rule70-15-15 Rule

2/15/20052/15/2005 66

Problem Problem (continued)(continued)

Decentralized computing across Decentralized computing across USACE resulted in vertically focused USACE resulted in vertically focused technology/information.technology/information.How can technology assist in How can technology assist in integrating disparate heterogeneous integrating disparate heterogeneous information and making it available to information and making it available to employees, customers, decision employees, customers, decision makers?makers?

2/15/20052/15/2005 77

Guiding principles for managing dataGuiding principles for managing data

Avoid duplication in data acquisition. Share data Avoid duplication in data acquisition. Share data wherever possible via networks and partnerships.wherever possible via networks and partnerships.Look for existing datasets before performing data Look for existing datasets before performing data collection.collection.Adhere to government and industry data content, Adhere to government and industry data content, access, and delivery standards.access, and delivery standards.Manage data to maximize its use by multiple Manage data to maximize its use by multiple processes.processes.Manage data at the owner level and negotiate access Manage data at the owner level and negotiate access arrangements.arrangements.Require the use of metadata for every dataset.Require the use of metadata for every dataset.

- Natural Resources Information Mgmt ToolkitConcise Guide for Technical Managers

2/15/20052/15/2005 88

RequirementsRequirements

Develop, promote, and deploy a Develop, promote, and deploy a common netcommon net--centric framework that centric framework that provides a consistent interface to data provides a consistent interface to data sources internal and external to sources internal and external to USACEUSACEFormalize connectivity to Formalize connectivity to USACEUSACE’’ss data data sharing partnerssharing partners

2/15/20052/15/2005 99

SolutionSolution

The The DataNetDataNet employs a netemploys a net--centric centric approach to streamline and approach to streamline and standardize the acquisition, standardize the acquisition, dissemination, and management of dissemination, and management of data across all USACE business areas.data across all USACE business areas.

2/15/20052/15/2005 1010

DataNetDataNet FrameworkFrameworkInteragency

ProvisioningReplicate Warehouse Proxy

Convert TransformIntegration

USACE applications

Sec

urity

Man

agem

ent

Infr

astr

uctu

re

Sec

urity

Man

agem

ent

Infr

astr

uctu

re

Data Sources

Databases

Accessibility

Fuse/Aggregate

Intranet Internet Extranet LAN

Standard API

GIS

tableImages

Intraagency

Met

adat

a

2/15/20052/15/2005 1111

ServiceService--oriented oriented architecturearchitecture

ServiceProvider

ServiceRequestor

ServiceRegistry

Publish Interact

FindClient

SOAPService

WSDL

UDDISLA

2/15/20052/15/2005 1212

Our Web Service VisionOur Web Service Vision

Enable the Enable the use of distributed resourcesuse of distributed resources. . –– From this we expect to see fundamental changes in From this we expect to see fundamental changes in

how scientists and engineers access powerful how scientists and engineers access powerful computing systems, multicomputing systems, multi--scale data archives, scale data archives, scientific instruments, and collaboration tools.scientific instruments, and collaboration tools.

Provide services that are Provide services that are integrated with the integrated with the useruser’’s work environments work environment–– and that provide uniform and highly capable access and that provide uniform and highly capable access

to computers, data, and instruments, regardless of to computers, data, and instruments, regardless of the locations or exact nature of these resourcesthe locations or exact nature of these resources

2/15/20052/15/2005 1313

WebWeb--Based Integrated Based Integrated Development Environment Development Environment (WIDE)(WIDE)

Business Partners

Gov’t/Industry Services

Gov’t/IndustryStandards

Developers

CDFTeam

Configuration Management

DevelopmentEnvironment

S&E Requirements

RequirementsTechnology &

Services

Repository(Operational)

Web Farm MonitoringHosting

Monitor

Guidance/Feedback Transition

Web Framework External Forces

Operations

Users

ApplicationBuild

Leverage

2/15/20052/15/2005 1414

WebWeb--centric Implementation centric Implementation

Web services technology offers a platform neutral approach for integrating applications across four domains:

• Internal to USACE• Government to

Government• Government to

Business• Government to

Citizen

ArcInfoArcInfo XMSXMSOKCOKC

Tools and Models

USACE

Industry Federal and State PartnersFederal and State Partners

USGSUSGS

Geospatial One-Stop

Geospatial One-Stop

NOAANOAA

USGS LULC dataUSGS LULC dataLandUseLandUse

USDA STATSGOUSDA STATSGOSoilsSoils

USACE NIDUSACE NIDInfrastructureInfrastructure

CWMS CWMS dssdss datadata

CWMS CWMS RealtimeRealtime Gage dataGage data

USGS Stream FlowUSGS Stream FlowHydroHydro

NCDC PrecipitationNCDC Precipitation

METAR current surface METAR current surface condcond..

Univ. of Utah Univ. of Utah MesoMeso West West Precipitation/WeatherPrecipitation/Weather

WeatherWeather

NOAA Estuarine BathymetryNOAA Estuarine Bathymetry

USGS NEDUSGS NEDElevationElevation

Data SourceData SourceCategoryCategory

2/15/20052/15/2005 1515

AccessibilityAccessibility

•. Centralized and secure Web development and Web hosting environment that serves as the operational platform for all local DataNet services.

Web Farm Approach•Centralizes interactions with DREN and CEEIS•Provides 24x7x365 operational capabilities –higher availability•Conducts backups•Provides connectivity to Internet, Intranet, and Extranet•Supports both Windows and Unix•Monitors resource access•Provides DoD certified computing environment•Shares cost among all users•Recycles hardware every 3 years

2/15/20052/15/2005 1616

MetadataMetadata

Technical Technical documentationdocumentationService RegistryService Registry

2/15/20052/15/2005 1717

Accessibility

DataNetDataNet Framework Framework --ApplicationsApplications

COTSCOTS

desktopdesktop

WebWeb--basedbased

HPC Systems

The challenge is to develop a framework that will support data accessibility by all of these applications.

A common approach toaccessing

and sharingdata – “One Data World”

2/15/20052/15/2005 1818

Locating and assembling dataLocating and assembling dataLocating and assembling data

USGSUSGS

EPAEPA

USDAUSDA

NationalNationalWeatherWeatherServiceService

Browse1

Assemble2

ESRIESRI

Download3

SEDB is a common gateway interface that serves data from a variety of information sources and scientific formats. SEDB accesses these sources via the DataNet.

2/15/20052/15/2005 1919

CDF Connection to ArcGISCDF Connection to ArcGIS

ArcGIS Application

S&E Toolbar provides simple interface to DataNet Services

2/15/20052/15/2005 2020

CDF Connection to ArcGISCDF Connection to ArcGISCDF Web ServiceTo EPA STORETdata

2/15/20052/15/2005 2121

Web Applications

connect to data via CDF Services

2/15/20052/15/2005 2222

Legacy applications

connect to data via CDF Services

2/15/20052/15/2005 2323

SummarySummary

The The DataNetDataNet framework:framework:–– avoids duplication in data acquisitionavoids duplication in data acquisition–– facilitates the sharing of data, both internal and facilitates the sharing of data, both internal and

external, via networks and partnershipsexternal, via networks and partnerships–– adheres to standardsadheres to standards–– promotes ownerpromotes owner--level management and servicelevel management and service--

level agreementslevel agreements–– requires metadata for data and servicesrequires metadata for data and services–– is accessible by distributed, heterogeneous is accessible by distributed, heterogeneous

applicationsapplications

Consistent with the basic guiding principles Consistent with the basic guiding principles of data managementof data management

2/15/20052/15/2005 2424

SummarySummary

The The DataNetDataNet provides a oneprovides a one--stopstop--shop for shop for data acquisition from within USACE, from data acquisition from within USACE, from other federal and state agencies, and from other federal and state agencies, and from industry.industry.S&E applications can access heterogeneous S&E applications can access heterogeneous data sources via the data sources via the DataNetDataNet in a timely and in a timely and secure manner. secure manner. The The DataNetDataNet framework truly provides a framework truly provides a standardsstandards--based realization of the servicebased realization of the service--oriented computing paradigm. oriented computing paradigm.

2/15/20052/15/2005 2525

Scientific work flow Scientific work flow improvementimprovement

Locate, Assemble and

Load Data

Scientific Simulations and Experiments

Scientific Analysis and Discovery

~70%time

~15%time

Distributed Heterogeneous Data Sources

Data Preprocessing

~15%time

~43%time

~42%time

DataNet

Locate, Assemble and

Load Data

Scientific Simulations and Experiments

Scientific Analysis and Discovery

Data Preprocessing ~15%

time

2/15/20052/15/2005 2626

QuestionsQuestions

Denise MartinDenise MartinAssociate Technical DirectorAssociate Technical DirectorInformation Technology LaboratoryInformation Technology LaboratoryEngineer Research & Development CenterEngineer Research & Development CenterU.S. Army Corps of EngineersU.S. Army Corps of Engineers3909 Halls Ferry Rd.3909 Halls Ferry Rd.Vicksburg, MS 39180Vicksburg, MS 39180--61996199Phone: (601) 634Phone: (601) 634--45744574Fax: (601) 634Fax: (601) 634--26382638Email: Email: denise.b.martin@erdc.usace.army.mildenise.b.martin@erdc.usace.army.mil