metadata driven data services for soa

Metadata Driven Data Services for SOA

Paul Anderson

Technical Sales DirectorApril 20, 2023

2

Agenda

• SOA Review

• Data Integration

• Data Services

• Use Cases

• Wrap-Up

3

Overview

Data architecture and its management are evolving toward an data services management system that delivers information as a service.

4

SOA Timeline

• Personal involvement since 1998– “Candle's Roma product of 1998 is the ESB's most

direct ancestor" (Roy Schulte, Gartner)

• SOA concepts have existed for a long time– Business needs did not drive SOA adoption– Fragmented technology slowed adoption

• Gained widespread acceptance in last 5 years• Web Service Technology has enabled broad

adoption of the concept

5

What is SOA?

• Gartner– "Web services are about technology specifications, whereas SOA is a

software design principle.” (Yefim V. Natis)

• SOA Concepts– Decoupling of service production from consumption– Service Interchange– Service Discovery

• SOA is an architectural pattern– The architectural pattern has/does appeared in many guises– ESB tools represents the most widely accepted toolset for SOA

implementation– Just because you use an ESB does not mean you have an SOA

architecture• SOA benefits from the platform neutrality of Web Services

– Enables interchange of services– Requires decoupling– Provides discovery

• The changing business environment is driving SOA– The scope of business functionality is changing

6

SOA Business Drivers

• Business Process enhancements– Migration of customer management to relationship management– Risk Management across/between organizations– Order/Shipment tracking across/between organizations– Tighter Partner Management

• Cross-Sell / Up-sell– Based on history/demographics/relationship/value

• Business Agility– Services are re-usable building blocks– Cataloging of available services– Enhanced time to market– Focus on information rather than data

• Integrated Businesses– Transactions span multiple organizations

7

Characteristics of SOA

• Requires Horizontal integration– Scope crosses lines of business boundaries– Scope crosses corporate boundaries

• Requires Horizontal understanding– Requires understanding of multiple lines of business– Requires development of common vocabulary

• Require concept generalization– Exploitation of commonality– Drilldown into specifics

• Requires easy access to information– Need to remove the complexity of differing semantics,

location, and access methods

8

SOA often slowed by

• Vertical Fragmentation– Differing technologies across verticals– Differing semantics across verticals

• Infrastructure Fragmentation– Application fragmentation – Hardware fragmentation– Data fragmentation

• LOB focus• Multiple technologies• Replicated data

• Organizational fragmentation– Ownership Issues– Organizational issues– Governance issues

9

Common Misconceptions #1

SOA is an technology issue:

• Web Services are a suite technology specifications

• You can implement a web service based stove pipe

• SOA does not have to be web service based

• Looks like an arcitectural diagram but is just a list of standards

10


SOA is an application issue:

• Focuses on services but ignores the underlying data

• Characteristics of an application layer are very different from the characteristics of a data layer

• Focuses on reuse of functionality and ignores reuse of data

11


An ESB tool is all you need:

12

Common Misconceptions #3 (Continued)

• Fragile implementation– Data access is embodied in the business process– Adding an additional data source requires business process

change– The data services are source specific

• Poor performance– Data reduction is likely happening in the Business Process layer– XML well suited for business process layer but not as well suited

to the data services layer

• Unmanageable– Difficult to audit data access– Difficult to secure data access

13

SOA reality

• SOA is fueled by data– SOA combines People, Machines – SOA embodies Business Process– SOA is based on data

• SOA is fueled by metadata– You can only reuse what you know exists– You can only reuse what you understand

• SOA based applications need to co-exist with existing applications

14

DataServices

BusinessServices

Applications Processes Users

Get customer Close account

Archive history Get single view of product

Calc lifetime value

Request for data operations:(Type, Format, Latency, Quality)

Response: (Data or

Metadata)

DataDataData

Metadata Sync

Access Profile Move

TransformAggregate

Data Integration Implementation Evolves Toward Data Services – Gartners View

15

Deriving Value From Enterprise Information

Leading organizations incorporate information assets into their portfolio management process. – Treat information as an organizational asset

• Optimize availability and quality

• Balance accessibility vs. security

• Improve information supply chain performance and integration

– Squeeze more value out of information• Optimize and eliminate overlapping processes

• Improve enterprise performance

• Trade information for goods and services

• Strengthen enterprise relationships (partners, employees, customers/constituents, suppliers)

Information must be managed as a currency — save it, invest it, spend it, manage it and account for it.

16

Agenda

• SOA Review


• Data Services

• Use Cases

• Wrap-Up

17

??

Enterprise Information Problem

Enterprise InformationEnterprise InformationSourcesSources

Enterprise InformationEnterprise InformationConsumersConsumers

customerscustomers

salessales

supplier datasupplier data

billingbilling financialfinancial

accountsaccountsshippingshipping

customerscustomers

Data ServicesData Services

The current state of corporate data

Disparate Information

Silos

18

Data Integration Challenges

• Organizations cannot address basic questions– What information do I have?– Where is it?– How do I access the most current information?– How do I manage it?– What is the impact of change during and after

implementation of my system?

• Information is stored in myriad data stores across the Enterprise. – Different formats/data types– Different structures/semantics– Different access methods (API’s)

19

Data Integration Landscape

EAI

CRM ERP

EAI - Target is the application Even Driven data movement between apps Data movement pre-wired Message based. Requires application coordination / workflow. Requires complex translation / programming.

CRM ERP ETL

ETL - Target is the database Scheduled extraction of data Bulk loading of data warehouses. Batch driven, Large Volumes. Inflexible, unable to meet business agility needs Historical data analysis. One-way data movement, read-only access.

EAIDW /ODS

Reporting /Dashboard

Data Services - Target is the end-user

On-demand delivery of information Real-time integration of disparate data. Universal data access layer. Integrates multiple data sources. Push / Pull any data across the enterprise. Single View of Customer. Supports dynamic / evolving reporting. Supports / Enables SOA / Web Services.EAI

CRM ERP ETL

DW /ODS

DS

Web Services

BI BAM Apps

20

Integration Technologies

ETL

EAI

DataService

s

Data ProcessIntegration Style

Data

Inte

gra

tion T

imelin

ess

Real Time

Batch

21

What if……….

CustomApplication

s

Web Services,Business Processes

Packaged Applications

Reporting, Analytics

Each client has their own desired “view”

Clients issue queries against their “view”

EAI, ESB, BPM

Client sees single database containing enterprise data

Clients are freed from the specifics of underlying sources

Enterprise Data Enterprise Data ServicesServices

<sale/> <value/></ sale >

Virtual Data

22

Enterprise Data Services

XML DWDBMS FILESAPPS

Applications /Dashboards

ESB/Portals

Reporting / Analytics

• High Performance Infrastructure

• Universal data access layer.

• Integration of structured/unstructured data.

• Data access security enforcement.

• Insulates data consumers from sources.

• Real-time integration of disparate data.

• Federation of operational & historical data.

• Provides a virtual representation of data.

• Extract & Combine information on-demand.

• Supports dynamic reporting / dashboards.

• Read & Write to any data source.

Web Service RelationalWeb Service/

Relational

23

model, describe, relate

share, version, manage, discover

package, deploy, version

model, describe, relate

MetadataCatalog

ModelerModeler

access & updatereal-time

Custom Apps

Web Services,Business Processes

Packaged Apps

Reporting, Analytics

EAI, Data warehouses

xml

databases

warehouses

spreadsheets

services


geo-spatial

Enterprise InformationEnterprise Information

rich media

…

Model-driven Data servicesUnderstand, relate, harmonize, rationalize, and use

Data ServerData Server


Virtual Data Server

real-timediscover,access,update

24

Agenda

• SOA Review


• Data Services

• Use Cases

• Wrap-Up

25

SOA Challenges

• Majority of data is held in relational sources– Relational structure optimized for LOB usage– Probably do not want the existing data structure to drive data

service structures– Performance concerns may rule out XML federation

• Granularity Issues– Granularity of data needed to support SOA may be different

than current use cases– Typically there will be data reduction moving from existing data

sources to data service presentation• Web-Service data is XML based

– Data is organized hierarchically– Need to map from relational to Hierarchical

• Scope of application typically wider that traditional applications– Spanning lines of business and/or corporate boundaries

26

SOA Challenges (continued)

• Performance– XML based data is exceptionally verbose– Exposing existing data in XML clearly not viable

• Security– Broader access to data drives security concerns– Applications that span political boundaries may be

subject to local confidentiality regulations– May need to integrate data security with application

security infrastructure

• Auditability– Regulations demand access audit trails

28

What is a Data Service?

MasterData

SAPDivisionalOperationalApplication

Data Service

SQL SQLAPICall

SOAP/XML

• Decouple data sources from application– Data implementation shielded

from application

• Semantic/Format Mediation– Standard vocabulary

• Single access point– Web Service/XML– SQL

• Federation• Scalability

– Security, performance

Bridge theGap

29

Data Service Layer in SOAClient Process & Applications

Data Sources

Data Services Layer

Message Services (ESB)

Business Services

Business Process Services

App App App App App App

Data Service Data Service Data Service Data Service Data ServiceData Service

30

Obtain XML From Non-XML Sources

?

«XML»

<customerPositions> <accounts> <account ID=…>

…

</account> </accounts></customerPositions>

«Application»

«Relational»

<<XML Doc>>

NEED:Mapping from Data to XML

GIVEN: Data Sources containingInformation to integrate

GIVEN: Fixed XML SchemaWANT:Data complying to schema

31

XML Modeling to achieve business agility

The Problem:The Problem:

TT

<X>

</X>

<X>

</X>

…

XML Document

EIS Data

XML XML SchemaSchema

TT

<X>

</X>

<X>

</X>

…

XML Document

EIS Data

XML XML SchemaSchema

TT

<X>

</X>

<X>

</X>

…

XML Document

EIS DataThe Solution:The Solution:

32

Data Service DesignThree ‘typical’ use scenarios:• Bottom-up: expose relational-style sources in ‘table-like’ form• Business view: starting from XSD/XML-based business views• Top-down: starting from WSDL definition of Web service operations

Web SvcOperatio

n

XSDIN

XSDOUT

<<X>>

Import sources

Source models

Relational views

XML views

XSDs - in/out

Web service operation

VD

B C

onta

iner

VD

B C

onta

inerWSDL

WSDL descriptor

BottomUp

Start

33


Web SvcOperatio

n

XSDIN

XSDOUT

<<X>>

Import sources

Source models

Relational views

XML views

XSDs - in/out


VD

B C

onta

iner

VD

B C

onta

inerWSDL

WSDL descriptor

BusinessView

Start

34


Web SvcOperatio

n

XSDIN

XSDOUT

<<X>>

Import sources

Source models

Relational views

XML views

XSDs - in/out


VD

B C

onta

iner

VD

B C

onta

inerWSDL

WSDL descriptor

TopDown

Start

35


Web SvcOperatio

n

XSDIN

XSDOUT

<<X>>

Import sources

Source models

Relational views

XML views

XSDs - in/out


VD

B C

onta

iner

VD

B C

onta

inerWSDL

WSDL descriptor

BottomUp

BusinessView

TopDown

Start

Start

Start

36

Data Services – Design Time

XML DWDBMSAPPS

• Model and Manage metadata.

• Combines technical & business metadata

• Enrich models with custom metadata.• Infer relationships.• Infer ownership.

• Maintain consistent data dictionary

• Understand overlaps in business areas

• Reduce time to perform impact analysis.

• Metadata open standards

• MOF• XMI

• Import metadata from :

• Rational Rose• ERWIN• System ArchitectUML

MetadataModeler

MetadataRepository

RuntimeMetadata

Virtual Databases

Security

MetadataReports

Metadata Relational View

Design timeMetadata

37

Data Services – Run Time

XML DWDBMS FILESAPPS

MetaDataModeler

QueryBuilder

MetadataRepository

Connector Framework

Query Processing / Optimise Engine

Data Cache

Virtual Databases

Security/Audit

Console(Admin)

• Robust security and access control• Row & column level• Supports existing authentication solutions

• Connectivity to• Databases• ERP• MOM• Data Warehouses• Apps / Legacy• Files

• Distributed query optimization & processing

• Data Caching vs. Real Time

• Model driven Integration• GUI based modeler• Metadata Repository

• Data Connectivity via• SQL-92• JDBC• ODBC• SOAP/HTTP• SOAP/JMS

38

• On-demand information– Real time data integration– Information sharing between business units

• Federation of disparate Information– Structured, unstructured– Relational + XML + Enterprise Apps + Legacy

• Faster time to market– Integrated information in days, weeks– Tight coupling of design & implementation phases– Leveraging the skill-set of the data architects for integration

• Costs across application lifecycle reduced– Model-driven abstraction layer between information sources and

applications eases development and maintenance

Data Services benefitsData Services benefits

39

Agenda

• SOA Review


• Data Services

• Use Cases

• Wrap-Up

40

Typical Use Cases

• Financial Services – Market Reference Data

– Risk Management: BIS - Basel II

– Transaction Monitoring: Anti-money Laundering, Patriot Act

• Supply Chain– Visibility

– Reporting

• Customer Service– Single View of Customer

– Account Aggregation & Cross-marketing

• Financial Reporting– Corporate Governance & Compliance

– Sarbanes-Oxley, Executive Dashboard

• Homeland Security– Watch Lists

41

Securities Reference Data

MetaMatrix Server

Challenge• Securities data is inconsistent• Consistent pricing & other data needed for

better trades & accounting• “Hard-coding” was too expensive and inflexible

Solution• Models define common “view” of securities data• Transform data to XML formats• Publish to pub/sub system• Support over 50 front-end applications

ROI• Consistent data = better trades, better

accounting, better risk management• New data sources online faster (weeks)• $3 million in IT savings

SecurityPrice

SecurityPrice

SecurityDescription

IBM MQ Series

Front-End Applications

42

Implementing Web Services & Service Oriented Architecture

• Business Drivers – Information sources available and discoverable as location-

independent Services on the CSFB Network

– CSFB Lines of business have access to information when and where it is needed

– Web Services Loosely Coupled IT Architecture

• Technology Use Case – Universal Data Services– Web Service consumers integrated to discoverable Services

– Encapsulate existing/legacy functionality

• Solution Benefits – ROI – 70% of IT Budget spent on integrating CSFB systems

– Found new uses for old data

– Squeezed more value out of legacy systems

– Embrace heterogeneity

– Increase business agility

43

Single View of Customer

Challenge• Call center application• Full view of customer interaction• Data distributed across multiple systems• Data inconsistent• Need a enterprise-wide data layer for more

applications

MetaMatrix Solution• Real-time access• Virtual views of data• Federated queries across systems• Common re-usable data definitions• Enterprise-wide data layer for re-use

ROI• Better service of customers• Lower data integration costs

MetaMatrix

Call Center Application

Data Sources

44

Integration

XML XML / Binary

Application

XML XML / BinaryXML /Binary

Da

ta Q

ua

lity

Rules

Enterprise Information Integration

Information Bus

Extract, Transform and Load

Information Transport & Protocols

ODS(structured)

Near Real Time

Content(unstructured)

LOBApp(s)

LOBData

DataStaging

ExternalData

DataWarehouse

Met

adat

a M

anag

emen

t

MetaData

Info

rma

tion

Ass

ets

Info

rma

tion

Ser

vice

sIn

form

atio

n C

onsu

mpt

ion

DataMarts

Conceptual Architecture

45

Agenda

• SOA Review


• Data Services

• Use Cases

• Wrap-Up

46

Recommendations

• Start small, incrementally grow toward information as a service

• Build the infrastructure on a project by project basis

• Data security must be a top priority• Ensure performance as you go• Keep it in SOA context — information fabric is a

means to an end, promoting information as a service

• But ability to access and use information is a longer-term vision

MetaMatrix

Paul Anderson

Technical Director

508 720 9266

[email protected]

Thank You

metadata driven data services for soa

Documents

soa implementationjust

technology issue

yearsweb service technology

soa architecturesoa

stove pipe soa

gartnerweb services

data layer

metadata driven data