anabas
DESCRIPTION
ANABAS. Use of Grids in DoD Applications. Geoffrey Fox, Alex Ho SAB Briefing November 16, 2005. The Ten areas covered by the 60 core WS-* Specifications. \ NCOW (and RTI) needs all of these?. Activities in Global Grid Forum Working Groups . NCOW needs all of these?. - PowerPoint PPT PresentationTRANSCRIPT
1
ANABAS
Use of Grids in DoD Applications
Geoffrey Fox, Alex HoSAB Briefing November 16, 2005
2
The Ten areas covered by the 60 core WS-* Specifications
WS-* Specification Area Examples1: Core Service Model XML, WSDL, SOAP
2: Service Internet WS-Addressing, WS-MessageDelivery; Reliable Messaging WSRM; Efficient Messaging MOTM
3: Notification WS-Notification, WS-Eventing (Publish-Subscribe)
4: Workflow and Transactions BPEL, WS-Choreography, WS-Coordination
5: Security WS-Security, WS-Trust, WS-Federation, SAML, WS-SecureConversation
6: Service Discovery UDDI, WS-Discovery
7: System Metadata and State WSRF, WS-MetadataExchange, WS-Context
8: Management WSDM, WS-Management, WS-Transfer
9: Policy and Agreements WS-Policy, WS-Agreement
10: Portals and User Interfaces WSRP (Remote Portlets)
\ NCOW (and RTI) needs all of these?
3
Activities in Global Grid Forum Working Groups GGF Area GS-* and OGSA Standards Activities1: Architecture High Level Resource/Service Naming (level 2 of fig. 1),
Integrated Grid Architecture
2: Applications Software Interfaces to Grid, Grid Remote Procedure Call, Checkpointing and Recovery, Interoperability to Job Submittal services, Information Retrieval,
3: Compute Job Submission, Basic Execution Services, Service Level Agreements for Resource use and reservation, Distributed Scheduling
4: Data Database and File Grid access, Grid FTP, Storage Management, Data replication, Binary data specification and interface, High-level publish/subscribe, Transaction management
5: Infrastructure Network measurements, Role of IPv6 and high performance networking, Data transport
6: Management Resource/Service configuration, deployment and lifetime, Usage records and access, Grid economy model
7: Security Authorization, P2P and Firewall Issues, Trusted Computing
NCOW needs all of these?
4
The Global Information Grid Core Enterprise Services Core Enterprise Services Service FunctionalityCES1: Enterprise Services Management (ESM)
including life-cycle management
CES2: Information Assurance (IA)/Security
Supports confidentiality, integrity and availability. Implies reliability and autonomic features
CES3: Messaging Synchronous or asynchronous cases
CES4: Discovery Searching data and services
CES5: Mediation Includes translation, aggregation, integration, correlation, fusion, brokering publication, and other transformations for services and data. Possibly agents
CES6: Collaboration Provision and control of sharing with emphasis on synchronous real-time services
CES7: User Assistance Includes automated and manual methods of optimizing the user GiG experience (user agent)
CES8: Storage Retention, organization and disposition of all forms of data
CES9: Application Provisioning, operations and maintenance of applications.
55
Major Conclusions I One can map 7.5 out of 9 NCOW and GiG core
capabilities into Web Service (WS-*) and Grid (GS-*) architecture and core services• Analysis of Grids in NCOW document inaccurate
(confuse Grids and Globus and only consider early activities)
Some “mismatches” on both NCOW and Grid sides GS-*/WS-* do not have collaboration and miss some
messaging NCOW does not have at core level system metadata
and resource/service scheduling and matching Higher level services of importance include GIS
(Geographical Information Systems), Sensors and data-mining
66
Major Conclusions II Criticisms of Web services in a recent paper by
Birman seem to be addressed by Grids or reflect immaturity of initial technology implementations
NCOW does not seem to have any analysis of how to build their systems on WS-*/GS-* technologies in a layered fashion; they do have a layered service architecture so this can be done• They agree with service oriented architecture• They seem to have no process for agreeing to WS-*
GS-* or setting other standards for CES Grid of Grids allows modular architectures and
natural treatment of legacy systems
7
DoD Core Services and WS-* plus GS-* INCOW Service or Feature WS-* Service area GGF Others
A: General Principles
Use Service Oriented Architecture WS-1: Core Service Model
Build Grids on Web Services
Industry Best Practice (IBM, Microsoft …)
Grid of Grids Composition Legacy subsystems and modular architecture
B: NCOW Core Services (to be continued)
CES 1: Enterprise Services Management
WS-8 Management GS-6: Management CIM
CES 2: Information Assurance(IA)/Security
WS-5WS-Security
GS-7 Security(Authorization)
Grid-Shib, Permis Liberty Alliance etc.
CES 3: Messaging WS-2, WS-3Service InternetNotification
NaradaBrokering, Streaming/Sensor Technologies
CES 4: Discovery WS-6 UDDI Extended UDDI
CES 5: Mediation WS-4 Workflow Treatment of Legacy systems. Data Transformations
CES 6: Collaboration Shared Web Resources Asynchronous Virtual Organizations
XGSP, Shared Web Service ports, Anabas
CES 7: User assistance WS-10 Portlets GridSphere NCOW Capability Interfaces, JSR168
8
DoD Core Services and WS-* and GS-* IINCOW Service or Feature WS-* Service area GGF Others
B: NCOW Core Services Continued
CES 8: Storage (not real-time streams)
GS-4 Data NCOW Data Strategy
CES 9: Application GS-2; invoke GS-3 Best Practice in building Grid/Web services (proxy or direct)
Environmental Control Services ECS
WS-9 Policy
C: Key NCOW Capabilities not directly in CES
System Meta-data WS-7 Semantic GridGlobus MDS
C2IEDM, XBML, DDMS, WFS
Resource/Service Matching/Scheduling
Distributed Scheduling and SLA’s (GS-3)
Extend computer scheduling to networks and data flow
Sensors (real-time data) Work starting OGC Sensor standards
Geographical Information Systems GIS
OGC GIS standards
NAME | SAB2003
Information Exploitation | Information Fusion & Understanding | Information Management | Cyber Operations | Command & Control | Connectivity | Advanced Computing Architectures
McQuay | SAB2005
Technologies:• Business Process Execution Language• Global Multimedia Collaboration System• Narada Brokering• Web Service Standards • Open Grid Computing Environment
Problem: Unable to satisfy interoperability, scalability, and security information management requirements for Net-Centric Operations without an advanced grid-based scalable service oriented framework
Grid of Grids Service Architecture for Net-Centric Operations and Warfare
Team: Indiana University/Anabas (Dr. Geoffrey Fox), Ball Aerospace, AFRL/IFSD (Michelle Cheatham)
Objective: Integrate Global Grid Forum (GGF) information management research to provide a Grid of Grids Middleware for Net-Centric Operations consistent with the Global Information Grid (GIG).
Approach:
• Analyze Net-Centric Operations and Warfare (NCOW) service specifications and relate core enterprise services in Net-Centric Enterprise Services (NCES) to core GGF and Web Services (WS-*) standards
• Develop the Grid of Grids architecture and information management middleware to address federation of legacy and new DoD enterprise systems with service oriented mediation between component Grids
• Develop prototype for NCES capabilities (Collaboration, Messaging, Management, Security/Information Assurance, Discovery, Mediation, User Assistance, Storage, Applications) with advanced Grid and Web Service standards support
• Demonstrate for Geographical Information System
• Tech transfer Anabas & Ball Aerospace, Global Grid Forum
Challenges:• Combining scalability and interoperability of
Grid and Web Services with DoD real time and security requirements
• Supporting legacy systems and standards while maximizing value of new commercial service oriented architectures
Impact:• A scalable managed interoperable secure
architecture and middleware for NCOW• Rapid implementation strategy leveraging
commercially pervasive technology• Brought GIG and GGF communities together
Results:• Phase I middleware addresses 6 of 9
NCOW Core Enterprise Services• Demonstrated GIS application in
collaboration grid• Middleware demonstrated with 500 users
and 20,000 messages/second
1010
General Message I Our proof of concept demonstrates many of the NCOW
core enterprise services (CES) implemented using Grid services built on top of the WS-* Web service industry specifications.
We will illustrate the use of the Grid of Grids architecture to integrate heterogeneous systems. The papers describe how all CES can be implemented using Grid technology and this is proposed in phase II SBIR.
Note the adherence to standards with a common line protocol SOAP implies that all service implementations are interoperable and one takes services from multiple sources. Anabas/Indiana University only has to implement some of the key Grid services.
1111
General Message II: Why Grids Web services gives us interoperability but Grids are
essential as we aim at Information Management Grids are the key idea to manage complexity but
applying uniform policies and building managed systems
Grids of Grids allows one to build out the management in a modular fashion
Uniform Grid messaging handles complex networks with managed QoS such as real-time constraints
Managed Services and Messaging gives scalability and performance (later slide)
1212
Scenario Laptop 8 Windows Client (small Sony) Laptop 2 Old Windows Client and PPT machine Laptop 3 Windows Server Laptop 4 Linux Assume no Internet access List of CES and GS-* WS-* mapping on Laptop 2
followed by complete stream of PPT screen shots if failure
On mapping slide highlight services
1313
Script I: Data Mining and GIS Grid This will show a set of Open Geospatial Consortium (OGC)
compatible services implementing a GIS (Geographical Information System) grid supporting streaming of feature and map data.
Intrinsic features of a region are supplemented here by features coming from a data-mining code that is filtering data to predict likely earthquake positions.
This uses discovery, metadata, database, workflow, messaging, data transformation, simulation (data-mining) services.
Note the OGC compatible WFS (Web Feature Service) plays role as a domain specific service interface to a database
This used by Los Alamos for DHS simulations replacing data mining by critical infrastructure simulations
1414
Script I: Google Map Grid Service This first demo also illustrates how the Google map
system can be wrapped as a Grid itself front-ended by a OGC Web Map Service.
This is used in a Grid of Grids fashion with Google linked with traditional (NASA) Web Map services.
Illustrates how linking NCOW to commodity Grid technology allows access to major IT resources• Google’s 100,000 computers• DoD DoE NSF Supercomputers
1515
Script II: Collaborative Grid Service This demonstrates how streams can be formed from
messages and managed in a uniform way whether maps or video. Collaboration is achieved by multicasting of the input or output streams to Grid services.
Our messaging infrastructure handles all multicasting (using software) transparently to services
First we demonstrate collaborative maps using “shared input ports” on web service
1616
Script III: Collaboration Grid Collaborative Audio-video conferencing on Laptop 3 and 8
with services running on 3 Collaboration uses basic Grid services – metadata,
discovery, workflow, security plus the XGSP stream management services.
Complex collaboration scenarios correspond to additional services for particular shared applications and to gateways in Grid of Grids fashion to H323, SIP and other protocols. Annotation, record, replay, whiteboards, codec conversion, audio and video mixing become services.
We demonstrate MPEG4 transcoding and video mixing services
Only Grid Web service based collaboration environment Use of Grids ensures scalability and performance (next slide)
1717
Performance Reduction of message delay jitter to a millisecond. Dynamic meta-data access latency reduced from seconds to
milliseconds using web service context service. The messaging is distributed with each low end Linux node
capable of supporting 500 users at a total bandwidth of 140 Mbits/sec with over 20,000 messages per second.
Systematic use of redundant fault tolerance services supports strict user QoS requirements and fault tolerant Grid enterprise bus supports collaboration and information sharing at a cost that scales logarithmically with number of simultaneous users and resources.
Supporting N users at the 0.5 Mbits/sec level each would require roughly (N/500)log(N/500) messaging servers to achieve full capability.
18
ANABAS
I: Data Mining and GIS Grid
SAB Briefing November 16, 2005
19
I: Data Mining and GIS Grid
WMS handlingClient requests
WMS Client
UDDI
WFS2
Databases withNASA, USGS features
SERVOGrid Faults
WFS1 NASA WMS
HTTPSOAP
WFS3
Data Mining Grid
WMS Client
20
California fault data from Quake Tables fault database via Web Feature Service.
Standard Open Geospatial Consortium WMS Clients
21
Get Feature Info allows users to get map information. This can also be used to read feature info off the map when creating input data for applications
Standard Open Geospatial Consortium WMS Clients
22
I: Data Mining Grid
HPSearchWorkflow
UDDI
Databases withNASA,USGS features
SERVOGrid FaultsWFS4
SOAP
WS-Context
WFS3
PI Data Mining
Filter
GIS Grid
Filter
NaradaBrokering
Pipeline
System Services
23
PI demo combines WFS, WMS, and HPSearch for service orchestration.
Tool bar items allows you to adjust maps
Users set up problems by adding filtered seismic archives from WFS as map layers.
24
Hot spots calculations--areas of increased earthquake probability in the forecast time-- calculations are re-plotted on the map as features.
Electric Power and Natural Gas data from LANL Interdependent Critical Infrastructure Simulations
Zoom-in
Zoom-out
FeatureInfo mode
Measure distance mode
Clear Distance
Drag and Drop mode
Refresh to initial map
26
ANABAS
I(contd): Google Grid Service
SAB Briefing November 16, 2005
27
Google Maps as Service
accessed from our WMS
Client
28
ANABAS
II: Sensor Grid(available on movie)
SAB Briefing November 16, 2005
29
Typical use of Grid Messaging in NASA
Datamining Grid
Sensor Grid
Grid Eventing GIS Grid
30
Typical use of Grid Messaging
HPSearchManages
NaradaBrokering
Sensor Grid
WS-ContextStores dynamic data
Filter orDatamining
WFS (GIS data)
Post beforeProcessing
Post afterProcessing
Notify
SubscribeGrid DatabaseArchives
Web Feature Service
GIS GridGeographical
Information System
31
Real Time GPS and Google MapsSubscribe to live GPS station. Position data from SOPAC is combined with Google map clients.
Select and zoom to GPS station location, click icons for more information.
32
Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.
Integrating Archived Web
Feature Services and Google Maps
33
ANABAS
III: Collaborative Google Grid Service
SAB Briefing November 16, 2005
34
Collaborative Google Mapswith faults from WFS
35
ANABAS
IV: Collaboration Grid
SAB Briefing November 16, 2005
3636
GlobalMMCS Web Service Architecture
SIP H323 Access Grid Native XGSPAdmire
Gateways convert to uniform XGSP Messaging
High Performance (RTP)and XML/SOAP and ..
Media ServersFilters
Session ServerXGSP-based Control
NaradaBrokeringAll Messaging
Use Multiple Media servers to scale to many codecs and manyversions of audio/video mixing
NB Scales asdistributed
WebServices
NaradaBrokering
37
Average Video Delays for one broker – Performance scales proportional to number of brokers
Latency ms
# Receivers
One sessionMultiple sessions
30 frames/sec
38
Collaboration Grid
UDDI NaradaBroker
HPSearch
WS-Context
Gateway
WS-Security
NaradaBroker
NaradaBroker
Gateway
Gateway
Gateway
XGSP MediaService
Video Mixer
Transcoder
Audio Mixer
Replay
Record
Annotate
Thumbnail
WhiteBoard
SharedDisplaySharedWS
39GlobalMMCS SWT Client
Chat
TV
WebcamVideo Mixer
GIS
40
ANABAS
V: Annotation Service
SAB Briefing November 16, 2005
4141
e - Annotation Player Archived stream
player Annotation / WB player
Archieved stream list Real time
stream list
e - Annotation Whiteboard
Real time stream player Archived Real Time Real Time
Stream List Stream List Player
e-Annotation Archived Stream Annotated e-Annotation Player Player Stream Player Whiteboard
42
ANABAS
Messaging
SAB Briefing November 16, 2005
4343
NaradaBrokering 2003-2006 Messaging infrastructure for collaboration, peer-to-peer and Grids
Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop)
Order-preserving message transport with QoS and security profiles Support for different underlying transport such as TCP, UDP, Multicast,
RTP SOAP message support and WS-Eventing, WS-RM and WS-Reliability.
• WS-Notification when specification agreed Active replay support: Pause and Replay live streams. Stream Linkage: can link permanently multiple streams – using in
annotation of real-time video streams Replicated storage support for fault tolerance and resiliency to storage
failures. Management: HPSearch Scripting Interface to streams and brokers (uses
WS-Management) Broker Topics and Message Discovery: Locate appropriate Integration with Axis2 Web Service Container (?) High Performance Transport supporting SOAP Infoset