Chuck MosherDirector, Federal Architecture &
Solutions
Abstraction As an Organizing Principle for Agile Enterprise Architectures
3rd SOA for E-Government ConferenceMay 2, 2007
2
Agility is Key
• Agility is the right precept
• Change is a constant– Technologies– Requirements– Standards– Models– Processes– ???
3
The Business Inflexibility Trap
• Inflexibility is the Mother of All Business Problems
• If you’re flexible enough, you can solve all the other problems
Ron Schmelzer, ZapThink
4
Multiple Dimensions to Address
• Governance– Leadership and oversight
• Policy– Rules for sharing, regulations, authority
• Technology– Systems, protocols, standards, security
• Culture– Potentially the most challenging issue
• Economics– Incenting the correct behavior
5
Focus of this talk
• Governance– Leadership and oversight
• Policy– Rules for sharing, regulations, authority
• Technology– Systems, protocols, standards, security
• Culture– Potentially the most challenging issue
• Economics– Incenting the correct behavior
6
Progressive Evolution of Solutions
• IT was supposed to help! And it has:– Assembly Language– Programming Languages, Procedural– Relational Databases– Programming Languages, OO– Messaging Frameworks– CORBA, IDL, RMI, UML– Java, J2EE, .NET, Application Servers– HTTP, XML, SOA, ESB, BPEL
Magic Bullets, All!
7
Magic Bullets, Current Crop
• XML
• Service Oriented Architecture (SOA)
• Enterprise Service Bus (ESB)
• Standardized Data Interchange Models
The Evolution of Computer Science has been as a result of more and more sophisticated thinking about appropriate levels of abstraction in how to solve problems in information management, and the technologies that result.
8
XML
• XML provides a transport-independent abstraction to interchange data between applications and across the enterprise
• Self-describing data• Human and machine-readable• Easily extendable• No more proprietary data formats
• It does not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them
9
SOA
• SOA provides an effective abstraction to encapsulate business functionality & provide interoperability across the enterprise
• Decouples data from applications & user views• Standardized interfaces improves info sharing• Leverage existing investments
• It does not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them
10
Enterprise Service Buses
• ESBs define an abstraction for machine and OS-independent messaging
• Pub/Sub, Point-to-Point, Content-based routing• Highly fault/failure-tolerant, assured delivery• Extensive logging/auditing capability• Complex event processing
• They do not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them
11
Standardized Data Models
• Enterprise-wide or COI-specific data abstraction definitions
• Simplifies information sharing between communities
• Many examples: GJXDM, NIEM, C2IEDM, JC3IEDM
• Concept of COI very powerful
• They do not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them
12
Information Sharing – THE Problem
• “It’s all about the data” – Hon. John Grimes, ASD(NII)/CIO
• “It’s all about the data” – Mr. Dave Wennergren, Deputy CIO, DoD
• “The biggest challenge is the data” – BGN Gary Salisbury, US European Command (ret)
• “It’s not the chain of command, it’s the chain of information” – Gen Cartwright, US STRATCOM
• The central concern of NCES• Thesis: If we can solve the data problem with
agility, adaptability, and cost-effectively, we have solved the most difficult (technical) problem
13
We Need One More Abstraction!
• A virtual data repository layer focused on turning the data we have into the data we want
• Data as an abstract service– Accessible in multiple formats (XML, relational)
• An abstraction that provides access to diverse distributed resource stores– Don’t move the data, virtualize it
• Middleware handles federation, structural, and semantic reconciliation to a reusable data abstraction
• Create agility precisely where it is needed, at the data layer
14
Information Virtualization
Information Resources
Information Consumers
Abstract Data TierInformation Virtualization Layer
15
Information Virtualization Layers
Unified Semantic Layer
Information Virtualization
Data Federation Layer
Data Access/Connectivity Layer
Enterprise Data Sources
Unification of different system conceptsSingle-query access to heterogeneous systemsUniform, standardized access to any system
16
Data Service Layer in SOAClient Process & Applications
Data Sources
Data Services Layer
Message Services (ESB)
Business Services
Business Process Services
App App App App App App
Data Service Data Service Data Service Data Service Data Service
17
Designing data services
Creating a Virtual Data Layer
xml
databases
warehouses
spreadsheets
services
<sale/> <value/></ sale >
geo-spatial
rich media
…Enterprise Enterprise Information Information
Sources (EIS)Sources (EIS)
Information Information ConsumersConsumers
Reusable,Reusable,Integrated Data Integrated Data
ObjectsObjects
ExposedExposedDataData
ServicesServices
<WSDL><WSDL>(contract)
<WSDL><WSDL>(contract)
<WSDL><WSDL>(contract)
Custom Apps
Web Services,Business Processes
Packaged Apps
Reporting, Analytics
EAI, Data warehouses
OD
BC
JDB
CS
OA
P
Intelligence
CRM, Logistics
18
Multiple layers of abstraction
19
Not Just a Notional Abstraction
• Multiple vendors provide a data management layer that efficiently and reliably enables federated access to a wide range of heterogeneous data sources
• Sophisticated security controls• Standards-based• Model-driven development
– Model, not code, your way to agility– Rich metadata management
• Data abstraction layer can concretize enterprise or COI-specific data models
20
MetaMatrix Integration Server
Information Consumers
Web Svc XML RDBMSPackaged Connectors
Siebel,SAP
OracleApps
CICSVSAM
MetaMatrix Catalog
MetaMatrix Designer- Design and deploy data services
MetaMatrix Product Architecture
JMSODBC JDBC SOAP
QueryProcessor
ProcessorProcessorOptimizerOptimizer
Integration ServerVirtualDataBases
VDBVDBVDBVDB
IntegratedSecurity
UsersUsers
RolesRoles
EntitlementsEntitlements
AccessModels
ViewsViews XMLDocsXMLDocs
<a>
</a>
<b>
</b>…
ServicesServices
in outproc
MetaMatrix Connector Framework
MetaMatrixServer
21
T
Authoritative Sources:• Mapped to logical
Multiple Internal/External Information Sources
Application views of information:
• Relational, XML
T T
XML Document<a>
</a>
<b>
</b>…
T
TT
ODBC/JDBC JDBC SOAP
WebServices
WebServices
Search Applications
Search Applications
BusinessIntelligence
Applications
BusinessIntelligence
Applications
Logical Data Model:• Agency or COI-specific• Rationalize, harmonize,
mediate
C2, Logistics, Intelligence, …
Data Abstractions and Data Models
bldg_id SITENUM Facility_ID
Location_ID
bldg_type Depot_Number
Location_Type
Semantic Data Services!
22
23
24
25
Metadata-aware Applications
Metadata management & federation
XML Models
RDBMS
MetaMatrix Repository
Metadata Sources
• Establish data dictionary• Define integration paths• Documentation• Version Control
Impact Analysis
Virtual Metadata
Metadata Search& Reporting
ModelingTools
MetadataRepositories
MetaMatrixServer
MetaMatrix Designer
26
Impact Analysis
27
Search Engine Index / Metadata Catalog
Master Data Person / Facility / Vehicle
MetaMatrix Enterprise
Stage SOAApp’s
Federal Agencies
Data Access Services• SQL, Web Service/XML• Staged Data (optional)
OntologyMgmt /Reasoning
Enterprise Service Bus
MetaMatrix Dimension
Local Commands
State/Local• Security/Authentication• Operations Management • Error / Exception Management
• Orchestration• Encryption• High Availability
MediationXSLT, Multi-source
Information Exchange Topology
28
US Defense Logistics Solution
• Global Combat Support System (GCSS)– Mission: supply the war-fighter with access to
accurate and timely logistics information
• Focused Logistics– Fusion of information technologies to enable forces of
the future to be more mobile and versatile – Provides the joint war fighter with a single capability to
manage and monitor units, personnel, and equipment– Deployed at 23 sites around the world– Networked environment allows DoD users to access
shared data & applications, regardless of location
29
DISA GCSS-J IDE = Abstract Data Layer
30
CFDB
CSDS DMDC
GSORTSIDE/AVNGA
FLIS
CSDS_PL
CSDS_VBL
Facilities_VMLMaterial_VML
Facilities_VQLMaterial_VQL
GDSS
Plans_VQL
Pri
vate
Dat
a an
d M
etad
ata
Pu
bli
c D
ata
Virtual Mid Layer (VML)
Virtual Query Layer (VQL)
(Exposed Views)
JOPESClassic
JOPES4.0
Virtual Base Layer
(VBL)
Physical Layer
(PL)GTN
GCSS Modeling Approach
31
MetaBase
• Data dictionary• Integration paths• Portal metadata
Intelligence Agency: Information Portal
Data Host(s)
Managed DataManaged
DataManaged Data
MissionDataMission
DataMissionData
MetaMatrixQuery Engine
Information IntegrationHost(s)
Application ServerHost(s)
Query Service(Business Layer)
Presentation Layer
Query Page
Control
ResultsPage
Control
FormPage
Control
Browser
Browser
Client(s)
T
Bottom-up(harmonization)
<X>
</X>
<X>
</X>
<X><X>
Top-down (mapping)
Org, Person, Image,
Location
Portal:• Entities• Components• RelationshipsMetadata:• Name• InformationContext• Usage• Description• Display Name• Default Value• Attribute Units• Presentation Type• Sort Order• Visible
ExternalFeeds
…
32
DCGS-A Services Network
En
terp
rise S
erv
ice B
usMetadata
DiscoveryService
SIGINTGateway
Service
MetadataPublishing
Service
MetadataCatalog
AlertSubscription
Service
EventAssessment
Service
WeatherEffect
Service
IMETS(IWEDA)
E-SpaceServices
WeatherEffects
EWData
AlertCriteria
Alerts /EventsMetadataMetadata
Searches
InfrastructureInfrastructure ServicesServices
ISR DataListenerService
Async“Callbacks”
Filters
WorkflowEngine
ServiceManagement
HUMINT Data
Service(s)
HDWS (CHAMS)
Map /Coverage
Google EarthRich Client
Handheld
NCES Service
Discovery
TransformationEngine
BC GatewayService
ForceTracking
MIP Blue Force Tracking
Intelligence Architecture: MetaMatrix & ESB
Google EarthRich Client
33
Long Term Software Cost Comparison
Competitive (it’s the business model!)
MediumHigh
ZeroMediumHigh
0++
++++
Competitive (it’s the business model!)
MediumHigh
ZeroMediumHigh
0++
++++
SI’s COTS Fully Supported Open Source
Code Quality
Profit Margin/Cost
Support
“Proprietary-ness”
34
Summary
• Leverage the power of abstraction to simplify, clarify, and realize your EA vision– Abstraction is how you design for change– Be very aware of the level you are working at
• Leverage the increasing power that COTS and open source tools can provide
• The rubber meets the road at the data layer– That you have, not what you want– Abstract it, don’t move it!
• Model, not code your way to agility! -RS
• Metadata management is a must-have– For repositories too
35
Unacceptable Risk???
36
Not if you know the lay of the land