Web Services Implementation Case Study:DataFed Air Quality Data & Services

Project Coordinators:Software Architecture: R. Husar

Software Implementation: K. HöijärviData and Applications: S. Falke, R. Husar

Center for Air Pollution Impact and Trend Analysis (CAPITA)Washington University, St. Louis, MO 63130

April 2005, rhusar@me.wustl.edu

Agenda

• Project Overview• Motivation• Approach• Web Services Provided• WS Implementation Challenges • WS Benefits• Conclusion

DataFed Overview

• Vision– Better air quality management and science through effective use of

relevant data

• Goals– Facilitate flow of atmospheric data from provider to users– Support development of user-driven data processing value chains– Participate in specific application projects

• Approach– Building WS-based data gateway & source wrapper utilities– Building WS-based browsers and analysis tools    

• Data Provided– Air quality

• Services Provided– Subsetting, fusion/layering

• URL– www.datafed.net

Motivation: Air Quality Management Requires a More Agile Information System

• Challenges– Shift from primary to secondary pollutants

– New regulatory approach based on ‘weight of evidence’

– Shift from command & control to participatory management

– Broader user community

– A richer set of data and analysis

• Opportunities– Rich AQ data availability from EPA, NASA, NOAA available.

– New information technologies

Monitoring

Controls

Set Policy

Set Goals

Assess-ment

Air Quality

Approach: Giving Users Control

• Data are supplied by the provider and exposed on the ‘smorgasbord’

• Data selection and tailoring choices are made by the user

Provider Push User Pull

Flow of DataFlow of Control

AQ DATA

METEOROLOGY

EMISSIONS DATA

Informing Public

AQ Compliance

Status and Trends

Network Assess.

Tracking Progress

Data to Knowledge Transformation

Data Refining & Mediation

Services

DataFed Services

• Software for the User– Data catalog– Dataset Viewer/Editor– Web page collection of views (“console”)– Predefined web service chains (“mini-apps”)

• Software for the Developer– *Dataset registration– *Data access– *Data processing– *Data rendering tasks– Web service chaining facility– Dataset wrapper generator

* Available as a web service

DataFed Services Example: Subsets/Views

• Views are slices through a cube of data organized by lat, lon, altitude, and time (X,Y,Z,T)

WS Implementation Challenges

• Providing access via web services to data on web pages

– Useful data is often embedded in web pages

– Service interface to data source would be helpful

– Developed wrapper to clip data from web pages

• {List others here}

• Need client/GUI to demonstrate WS benefits?

• Reliability: Distributed computing issues: network reliability, bandwidth, etc

• Chaining: Orchestrating distributed web services to act as a single application

• Links: Linking users to providers and other federations (e.g. OGC, OPenDAP)

Implementation Challenge: Providing Access via Web Services to Data on Web Pages

• Short descriptor defines exact location of data on page• Wrapper software uses descriptor to present data on

web page as a web service

src_img_width

src_

img_

heig

h t

src_margin_rightsrc_margin_left

src_margin_top

src_margin_bottom

src_lon_min src_lat_max

src_lat_min src_lon_max

Image Description for Data Access:

src_image_width=502 src_image_height=329

src_margin_bottom=105 src_margin_left=69 src_margin_right=69 src_margin_top=46

src_lat_min=-70 src_lat_max=70 src_lon_min=-180 src_lon_max=180

Transparent colors for overlays

RGB(89,140,255) RGB(41,117,41) RGB(23,23,23) RGB(0,0,0)

Reflectance (SeaWiFS)

Organics (IMPROVE)

Air Trajectory (ATAD)

Map Border (OGC)

Web Service Benefits

• Data from web services can be easily tailored and combined with other data

• <ServiceFlow>• <View>• <MapImageMargin ref="ws_margins" />• <AnnotateImage ref="ws_title" icon_text="T" label="Title" execute="true" />• <StockAnnotation ref="DATAFED" />• <StockAnnotation ref="CIRA" />• <AnnotateImage ref="ws_cursor_annotation" icon_text="C" label="Cursor" execute="true" />• </View>• <Layers current="Traj_Grid" order="Traj_Grid Traj_Point Traj_Line MapImage_1">• <Layer id="Traj_Line" dataset_abbr="VIEWS_OL" visible="false" visibility="1" click_target="true">• <DataMapTimePoint ref="ws_data" label="Query Filter" />• <DataMapTrajectory ref="ws_traj_render" show_data="false" icon_text="TR" />• <AnnotateImage ref="ws_network_legend" icon_text="N" label="Network Legend" execute="true"

execute_always="true" />• </Layer>• <Layer id="Traj_Point" dataset_abbr="VIEWS_OL" visible="true" visibility="1" click_target="true">• <DataMapPoint ref="ws_point_data" show_data="false" />• <DataMapPoint ref="ws_store_cursor_value" />• <RenderMapPoint ref="ws_point_render" icon_text="RP" label="point rendering" />• <MapParamLocationAccessRender ref="ws_loc" execute_always="true" />• <RenderLegend ref="ws_pntleg" icon_text="L" label="Point Legend" execute="true"

execute_always="true" />• </Layer>• <Layer id="Traj_Grid" dataset_abbr="VIEWS_OL" visible="true" visibility="1">• <DataMapTimePoint ref="ws_ref_data" label="Reference Query Filter" />• <DataMapTrajectory ref="ws_ref_traj_grid" icon_text="TG" />• <DataMapTimePoint ref="ws_data" label="Query Filter" />• <DataMapTrajectory ref="ws_traj_grid" icon_text="TG" />• <MapGridOperator ref="ws_mgo" />• <RenderMapGrid ref="ws_rg" />• <RenderLegend ref="ws_rainbow" icon_text="L" label="Rainbow Legend" execute="true"

execute_always="true" />• </Layer>• <Layer id="MapImage_1" dataset_abbr="MapImage" visible="true" visibility="1">• <MapImageAccessRender ref="ws_map_image" />• </Layer>• </Layers>• </ServiceFlow>

Web Service Benefits: Creating a Custom View From Multiple Layers in a Page of XML

Name & title the new view

Identify each of 4 layers

Implementation Challenge: Providing a GUI for WS Information

• Core maintains the state and executes the data selection, access and render services

• Wrappers encapsulate the heterogeneous external data sources

• Device Drivers translate generic, abstract graphic objects to specific devices and formats

• Ports connect the internal parameters of the program to external controls

• WDSL web service description documents

Data Sources

Controls

Displays

I/O Layer

Dev

ice

Dri

vers

Wra

pp

ers App State Data

Flow Interpreter

Core

Web Services

WSDL

Ports

Conclusions

• DatFed is an infrastructure for non-intrusively federating autonomous, distributed, heterogeneous datasets

• Data are ‘wrapped’ for access through SOAP web services

• Service chaining performs filtering, aggregation, fusion, overlay, rendering and other operations on data.

• Several successful web-applications were built from the federated data and service components.

• A key challenge is to link DataFed to other open data systems (e.g. OpenDAP, OGC) and to interoperate with other processing services.