Utilizing Web 2.0 in SOA for Enhanced Utilizing Web 2.0 in SOA for Enhanced Data Sharing Among Federal AgenciesData Sharing Among Federal Agencies

Nick Duan, Ph.D.Sr. Software ArchitectMcDonald Bradley, Inc.

Oct. 2, 2007

MBI At a GlanceMBI At a Glance

• Information technology firm, founded in 1985

• Headquartered in Herndon, VA• Focus on DoD, IC, and Federal Civil• About 300 associates

– Experts and Industry leaders in enterprise technology and subject matter expertise

• Top Secret Facility clearance with many cleared professionals

CONTRACTOR OF THE YEAR

MBI Core Competency AreasMBI Core Competency Areas

Service-Oriented Enterprise

Data Interoperability

Information Assurance

Enterprise Governance and Management

Overview

• The Ultimate Quest of Sharing Data and Services• SOA Implementation – The Scalability Problem• SOA Enablement via Web 2.0

– Simple, Agile, Efficient

• Current Web 2.0 Technologies• A Web 2.0-based Framework for Enhanced Data

Sharing• A Sample Case Study• Future Tasks• Summary and Conclusions

The Ultimate Quest: Sharing Data and Services

• Millions of data records are published on the Internet by the government– Census data, weather data, housing data, map

data, budget data, airport data, etc..• Data utilization is limited

– Manual search and retrieval, lack of standards and quality control, limited corporation among agencies, long development cycles

• Unleash the potential– A robust and scalable SOA framework to add

values beyond search

General Concepts in SOA for Data and Service Interoperability

• Data Structuralization and Standardization– Data format, metadata standards, authoring tools

• Service Standardization– Protocol standard at transport level and message level

• Infrastructure Services– Search and Discovery– Service Registration– Metadata Registry and Discovery

• Security– Transport level and message/content level

• Governance Rules and Policies– Getting the right data at the right time

The SOA Scalability Problem

• Most SOA implementations today are based on Web Services with complex WS-* stack– Out-of-the-box adoption of the commercial SOA

models and products– Complexity: Distributed Object Computing (DOC)

model does not scale across enterprises– Centralized service registry and repository with

complex governance policies suitable for developing monolethic, closely-coupled systems

– Topdown, ”Creational” development – Lack of overall and consistent conceptual data models

across multiple domains• $Millions spent with limited results

Using Web 2.0 to Achieve Simplicity, Agility and Efficiency

• What is Web 2.0?– Collection of web technologies that leverage existing

web tools and infrastructure• Asynchronous JavaScript and XML (AJAX)• REpresentational State Transfer (REST)• Really Simple Syndication (RSS)• Rich Client Applications, Semantic Web, Wiki, Blogs, etc

– Client-side and Server-side Data Mash-ups• Examples: GoogleMaps, Yahoo Maps, Yahoo!Pipes,

Amazon, eBay, MyFaceBook• Key Advantages of Web 2.0

– Data and service mash-ups at various levels, enabling non-transactional data retrieval

– De-centralized, bottom-up, evolutional approach to enable rapid service offering

Web 2.0 in ActionWeb 2.0 in Action

Asynchronous JavaScript and XML

• Microsoft introduced XMLHttpRequest in IE in late 90’s– Allows JavaScript object to connect with back-end web

servers directly– Enables partial update of web pages from multiple web sites – Simple, efficient object serialization via JSON

<script type=”text/javascript”>

........

</script>

XMLHttpRequestXMLHttpRequest

Web ServerWeb Browser

Server-sideComponent

Server-sideComponent

Server-sideComponent

Full Update

Partial Update

REpresentational State Transfer

• As an architecture style first introduced by Roy Fielding in 2000

• Basic design concept of a good protocol: – Separating transport control from content!– Utilizing the full spectrum of HTTP methods:

• POST, GET, PUT, DELETE– Representing states via resource URLs

• No need for cookies

• Resource-Oriented Computing (ROC): a virtual system comprised of multiple, interconnected web resources

REST Examples

• Retrieving part list info using GET (http://www.parts-depot.com/parts)

<?xml version="1.0"?><p:Parts xmlns:p="http://www.parts-depot.com" xmlns:xlink="http://www.w3.org/1999/xlink"> <Part id="00345" xlink:href="http://www.parts-depot.com/parts/00345"/> <Part id="00346" xlink:href="http://www.parts-depot.com/parts/00346"/> <Part id="00347" xlink:href="http://www.parts-depot.com/parts/00347"/> <Part id="00348" xlink:href="http://www.parts-depot.com/parts/00348"/></p:Parts>

• Retrieving individual part (http://www.parts-depot.com/parts/000345) <?xml version="1.0"?>

<p:Part xmlns:p="http://www.parts-depot.com" xmlns:xlink="http://www.w3.org/1999/xlink"> <Part-ID>00345</Part-ID> <Name>Widget-A</Name> <Description>This part is used within the frap assembly</Description> <Specification xlink:href="http://www.parts-depot.com/parts/00345/specification"/> <UnitCost currency="USD">0.10</UnitCost> <Quantity>10</Quantity></p:Part>

Benefits of Using REST

• Statelessness: Clients are completely de-coupled from the Server

• Uniform Interface: HTTP GET, POST, PUT and DELETE• Uniform State Representation: Define a virtual machine

whose states are represented by interconnected resources via URL

• Unrestricted Scalability: The Web is the limit

Data and Service MashupsData and Service Mashups

• Wikipedia: A mashup is a web application that combines data from more than one source into a single integrated tool

• In general, mashup is the concept of integrating data from multiple data sources to create presentation in a new dimension– Data mashup for presentation– Service mashup for reuse

• Client-side mashup– JavaScript, HTML/DHTML, AJAX, AdobeFlash, Customized APIs

• Server-side mashup– Proxy or gateway services providing data mapping, conversion,

and mediation functions

It's all about Mashups

Server-site MashupsServer-site Mashups

Client-side MashupsClient-side Mashups

Federal Data ProvidersFederal Data Providers

Client vs. Server-side Mash-ups

• Client-side mash-ups: data mash-ups within a browser– Simple, easy, cost-effective, end user driven– Limited functionality, primarily presentation use only,

constraint by security

• Server-side mash-ups: value-added, aggregated services for data integration– Feature rich, sophisticated functionality, less security

constraints– Complex, expensive, business driven

• Create the best combination to achieve maximum effectiveness and reuse

• Server-centric• Thin/simple client• SOAP, WSDL, UDDI• Heavy WS-* stack • Standard-committee

driven• Top-down, creational• Long dev cycle

Moving SOA to a New Dimension

• User/business-centric• Rich client• REST, AJAX, RSS,

REST Registry • Leveraging existing

HTTP/HTTPS • User and community

driven• Bottom-up, evolutional• Short development

cycle

From Web 1.0/SOA 1.0 From Web 1.0/SOA 1.0 to Web 2.0/SOA 2.0to Web 2.0/SOA 2.0

SOA Landscape with Web 2.0

FAA NOAA Census Bureau

USGS States Local

End Users

Aggregated Content and Service Providers

Open Infrastructure Services

YahooPipes

GoogleMaps

Amazon S3

Programmable Web

Yahoo News

Google RSSInternet

Data Providers

Open Source

A Case Study

• Case Scenario – Emergency Planning – In case of a category 5 hurricane, which area

(states/counties) may be affected– Which hospitals are to be evacuated?– Which airports may be used for delivering emergency

supplies?

• Data Contents and Providers– Real-time forecast data in RSS from NOAA (NWS, NHC)– County data from US. Census Bureau in CVS– Airport information from FAA in Excel/CVS– Hospital information from AHD in Excel

Plant 1 Plant 2 Plant 3 Plant 4

0

2

4

6

8

10

12

Estimated Generators Needed

Category 3

Category 4

Category 5

Interaction of Services

Aggregation & Mediation Service

Data MashupsService Registry and Discovery

21

Rich User Functions beyond Search

Geo-spatial Analysis: Locate the hospitals in the impact region

Plant 1 Plant 2 Plant 3 Plant 4

0

2

4

6

8

10

12

Estimated Reduction in Electricity

Category 3

Category 4

Category 5

Converting GIS data intoGoogle-compliant lat/long

Converting/Correlating hospital data AJAX-specific format

Statistical Analysis: Estimated number of evacuees per hospital

Analytics capabilities provided by service aggregation

Aggregated Service

Future TasksWhat are needed for SOA 2.0

Future TasksWhat are needed for SOA 2.0

• Aggregated services to enable rapid data and service integration– Data mapping, mediation,

• Government-sponsored infrastructure services to meet specific need of agencies– Service Registry and Discovery– Standard Metadata Registry– Standard cross domain taxonomies– Security if needed

• Government standard for data quality– Tolerable error rate, availability, security, etc.

Summary and Conclusions

• The majority of data published by the federal government have not been fully utilized

• Vast potential of data and service sharing can be unleashed via SOA

• A simple, agile and efficient SOA framework can be achieved using Web 2.0 technologies for enhancing data and service interoperability

• Data and service mashups are the key towards rapid development and service offerings

Q & AQ & A

??