andre tost e s b ref arch

1

Founding Sponsors

This Presentation Courtesy of the

International SOA Symposium

October 7-8, 2008 Amsterdam Arena

www.soasymposium.com

[email protected]

Gold Sponsors

Platinum Sponsors

Silver Sponsors

© 2008 IBM Corporation

SOA Symposium

October 7-8, 2008, Amsterdam

ESB Reference Architecture and Product Selection Guidelines

André TostSenior Technical Staff Member, SOA TechnologyIBM Software Services for WebSphere

2

SOA Symposium

ESB Reference Architecture and Product Selection Guidelines © 2008 IBM Corporation3

Agenda

Introduction

Definition / Reference Architecture

Patterns

Federation

Product Selection Criteria

Sample Scenarios

SOA Symposium


Introduction

3

SOA Symposium


SOA Addresses Many Perspectives

SOA Entry PointsBusiness Services

Business Process Management

SOA Governance andService Lifecycle Management

Infrastructure & Management

in Support of SOA

ServiceVirtualization

ServiceManagement

ServiceSecurity

SOA Symposium


Greater innovation and flexibility through faster deployment and modification of business processes

Achieve business process innovation through treating tasks as modular services

Improved productivity and flexibility by enabling targeted user interactions for improved business operations and collaborationDeliver role-based interaction and collaboration

through services

Lower risk and faster time to market by leveraging proven, time-tested functionality

Service-enable existing assets and fill portfolio gaps with new reusable services

Reduced maintenance costs and greater reliability and consistency through flexible, any-to-any linkages

Connect systems, users, and business channels based on open standards

Provide trusted information in business context by treating it as a service

Better business operations, more informed decisions and reduced risk with information delivered in-line and in-context

Process

People

Reuse

Connectivity

Information

ValueWhat is it?

SOA Entry Points Help You Get StartedBoth Business Centric and IT Focused

4

SOA Symposium


What are some of today's business challenges that SOA Connectivity can address?

How do you…?

Ensure seamless flow of information from anywhere at anytime using anything

Execute broad business processes that span your company and business partners?

Build trusted relationships with your partners?

Scale your business to grow smoothly?

Deliver a consistent user experience regardless of channel or device?

SOA Symposium


Connectivity needs

How do you connect the assets and services that support your business process?

Enable "any-to-any" linkage and communication inside and beyond your company

Simplify connectivity by ensuring secure, reliable, and scaleable pipeline of information

5

SOA Symposium


Business Value…

*Software Strategies

“Enterprise Integration Challenge” 2005

Deliver a secure, consistent user experience…

…by exposing the same process through new business channels and devices

Strengthen trading partner relationships…

…with managed Service-based connections

Potential savings of 2X-4X…… over custom-built or traditional connectivity*

Minimize complexity in your organization…

… through any-to-every connectivity

Increased business flexibility and responsiveness…

…through scaleable access to information in the right format and at the right time

Eliminate redundancy……by extending IT assets rather than duplicating them

Decreased development and maintenance costs…

… through decoupling of function and interface

…Through SOA Connectivity

SOA Symposium


Connect business systems

Deliver a process across multiple

business channels

1. Internal connectivity based on open standards

2. Deliver an existing process through new business channels

3. Securely connect to external 3rd parties and trading partners

Three examples of SOA connectivity

Connect to external service

providers

6

SOA Symposium


ESB Definition

SOA Symposium


Core Principles of the ESB Architectural Pattern

ESB inter-connects requester and provider– Interactions are decoupled

– Supports key SOA principle – separation of concerns

ESB provides Service Virtualization of– Identity via routing

– Interaction via conversion

– Interface via transformation

ESB also enables Aspect Oriented Connectivity– Security

– Management

– Logging

– Auditing

– …

Service

Requestor

Service

ProviderESB

7

SOA Symposium


An ESB-centric view of the Logical Model

Outside ESB

– Business Logic (Business Services)– ESB does contain integration logic or connectivity logic

– Criteria: semantics versus syntax; aspects

Loosely coupled to ESB

– Security and Management– Policy Decision Point outside the ESB

– ESB can be Policy Enforcement Point

Tightly coupled to ESB

– Service Registry– Registry is a Policy Decision Point for ESB

– ESB is a Policy Enforcement Point for Registry

– But, Registry has a broader scope in SOA

Tooling required for ESB

– Development

– Administration

– Configuring ESB via Service Registry

IT M

anagem

ent

Serv

ices

Se

cu

rity

Ma

na

ge

me

nt

Business Services

InformationProcessInteraction

Business Services

AccessBusiness AppPartner

Develo

pm

ent

Serv

ices

ESB Registry

SOA Symposium


Message Models

Mediation Flows

Communication Protocols

Enterprise Service Bus

Interaction Patterns

Mediation Patterns

Meta models

Expanded view of the ESB

Communication Protocols– Supply basic connectivity to requesters and providers

– Impact QoS (e.g., reliable delivery, transactions)

– Supply inherent Interaction Patterns (e.g., request/reply, one-way, pub/sub)

An ESB leverages underlying communication fabrics of SOA infrastructure– ESB provides on-ramps and off-ramps

Standards are important Typical requirements

– HTTP (SOAP/HTTP, XML/HTTP)

– MQ (SOAP/JMS/MQ, XML/MQ, text/MQ, …)

– Adapters (legacy, EIS)

– WS-I, WS-Security

– RAMP

8

SOA Symposium


Message Models

Mediation Flows




Mediation Patterns

Meta models


Message Models– Describe messages exchanged with requesters and providers

– Based on Meta-models

– Fundamental means of describing messages

– For example, XML Schema language

– Content models define specific messages

– For example, XML schema

An ESB supports one or more message meta-models

An ESB supports multiple message content models– Can include industry standard models as well as enterprise specific models

– Can include weakly-typed models

Typical requirements

– XML schema definition

– Industry specific content models

SOA Symposium


Message Models

Mediation Flows




Mediation Patterns

Meta models


Mediation Flows

– Process messages exchanged between requester and provider via ESB

– Large grained

– Moderately reusable

– Constructed from Mediation Patterns

– Mediation Patterns define processing “steps” of a mediation flow

– Small to middle grained

– Highly reusable

– ESB products include pre-built “mediation primitives”

Typical requirements

– Dynamic routing

– Logging

9

SOA Symposium


ESB Mediation Flows and Mediation Patterns

Mediation enables Service Virtualization of

– Identity via routing

– Using basic mediation patterns (context, content, contract)

– Using composed mediation patterns (retry, failover, Distribution/aggregation, …)

– Dynamic, driven by metadata in registry

– Impact aspects of QoS (e.g., SLA, failover)

– Interaction (of protocol and pattern) via conversion

– Protocol conversion inherent with support for more than one transport protocol

– Impact aspects of QoS (e.g., reliable delivery, transactions)

– Interface via transformation

– Using specific mediation patterns

– Using adapters

– Impact aspects of QoS (e.g., performance)

– NOTE: other forms of mediation should be agnostic to interface (weakly-typed processing)

Mediation enables Aspect Oriented Connectivity

– Security & Management

– Logging

– Auditing

– …

Provided by a mediation framework

– Offering pre-built mediation pattern (primitives) support

– Enabling mediation pattern composition

SOA Symposium


Mediation Patterns - Examples

Monitor CorrelateDistributeRouteTransformProtocol Switch Enrich

Request / Response

Event Propagation

Request / Multi Response

Canonical Adapter

+ + +

Transform – Log - Route Gateway

10

SOA Symposium


Adapters for Existing Systems Integration

An ESB must allow access to existing Enterprise Information Systems

Adapters typically used, and may or may not be part of the ESB

– Technology

– Application

– Legacy

The following are the placement options for adapters, based on domain where adapter configuration managed:

– Outside of the ESB, and inside the EIS domain

– Outside the ESB, and the EIS domain

– On the boundary of the ESB

– Inside the ESB

SOA Symposium


ESB Patterns

11

SOA Symposium


ESB Patterns

All services in one namespace

All services have visibility to each other

Centrally Administered

Used by departments or small enterprises (or for development & test environments)

Global ESB

ESBProviderRequester

Requester Provider

SOA Symposium


ESB Patterns

Controlled and Secure service interactions at internal or external domain boundaries

ESB Gateway

ESB ESBESBProvider

Requester

Requester

Provider

12

SOA Symposium


ESB Patterns

Bridge services that selectively expose requesters or providers

Each ESB manages its own namespace.

Service interactions between ESBs facilitated through a common broker that implements the bridge services.

Used by departments that develop and manage their own services, but share a few of them

Brokered ESB

ES

B

ES

B

Requester ES

B Provider

ES

B

Requester

Provider

Provider

Provider

SOA Symposium


ESB Patterns

One master ESB with several dependent ESBs

Service requestors and providers connect to master ESB or one of the dependent ESBs

Used to federate a set of autonomous organizations

Federated ESB

ES

BRequester

ProviderESB

Requester

ProviderProvider

ProviderESB

ESB

Provider

Provider

Provider

13

SOA Symposium


ESB Federation

SOA Symposium


Observation

Federation is really about service visibility

– One domain

– Several domains

– All domains

A service registry defines service visibility “policy”

– In the desired domain(s)

– With the desired monitoring and management

– With the desired security

A service bus simply enforces “policy”

We use this to analyze approaches to federation

14

SOA Symposium


Some analysis – Service Virtualization

… implemented

by Service Bus

mediation…

Service requester

invokes a service…

… which is a

Virtual service …

1

2

3

… delegating to the

Actual service

4

Pattern: Service Virtualization

SOA Symposium


Service Virtualization with Service Registry

A‟ A

Virtual Service

Implemented by

Service Bus

Actual Service

the virtual service

delegates to

Services in Registry

advertised for use by

Requesters

A‟ A

Metadata about services

in Registry for use by

Service Bus

Advertise

virtual servicesUse Registry content to

implement smart mediations0

3.a

Pattern: DynamicService Virtualization

15

SOA Symposium


Federated Service Virtualization

A‟‟ A

Actual Service

the virtual service

delegates to

A‟‟ A‟

A‟

A‟ A

Virtual

Service

Virtual Service

the virtual service

delegates to

Pattern: RecursiveService Virtualization

SOA Symposium


Federated ESB interaction patterns

B

A

C

B‟B B‟‟

C‟C

A A‟

Remote

virtualization

Recursive

virtualization

Local

virtualizationDomain

ESB X

Domain

ESB YD

D‟

Remoted

virtualization

A‟

B‟‟C‟B‟

C

DA

BPrivate in X

Private in YPublic in X

Public in Y

D‟

D

Patterns:“Federated”

16

SOA Symposium


Federated ESB interaction patterns

In reactive situations, with “black box” service buses, addition of a „backbone‟ may be necessary

“Introspection” of existing service

registries required to populate

„global‟ service registry

SOA Symposium


Some analysis – Service Management

A

Virtual

Service

Observes

endpoints

Service Monitor &

Manager

B

Change & Configuration

Management DBService Registry

Endpoint Status

Offers Interaction

Status Information

A B

Uses Endpoint

Status Information 12

Metadata

Federation

17

SOA Symposium


Some analysis – Service Security

Identity

A‟‟ A

Actual Service

the virtual service

delegates to

A A‟

Virtual

Service

Virtual Service

the virtual service

delegates to

Identity

Identity

Mapping

SOA Symposium


Some analysis – Service Security

In reactive situations, with “black box” service buses, addition of a „backbone‟ may be necessary

Identity Identity

Identity

Mapping

18

SOA Symposium


ESB Product Selection Criteria

SOA Symposium


Key Criteria for ESB selection

Communication Protocols & Interaction Patterns

– Critical (e.g., MQ, SOAP/HTTP)

– Accommodated

– “Associated” standards (e.g., WS-Security)

– APIs (e.g., JMS)

– Adapters

Message Models

– Meta-models (e.g., XML)

– Content models (e.g., ACORD), including Weak-typing

– “Associated standards” (e.g., SOAP headers)

19

SOA Symposium


Key Criteria for ESB selection (cont.)

Mediation Flows and Mediation Patterns

– Conversion

– Transformation engine(s)

– Routing

– Security and Management integration

– Logging and auditing integration

– Registry integration

– Breadth of pre-built mediation primitives

– Custom mediation capability and Programming model

– Weakly-typed processing

SOA Symposium


Key Criteria for ESB selection (cont.)

Qualities of service

– Heterogeneous transaction coordination

– Reliable/assured delivery

– Performance

– Scalability

– Reliability

Non-functional

– Affinity to SOA environment (e.g., WebSphere Process Server)

– Affinity to IT environment (e.g. J2EE application server)

– Development tooling capabilities and affinity to current tools

– Configuration and administration tooling capabilities

– Existing and required skill set (e.g., J2EE skills)

– Ease of integration with management (e.g., monitoring, security) environment

– Product maturity and comfort level with leading edge products

– Price and total cost of ownership

20

SOA Symposium


Sample ESB Scenarios

SOA Symposium


Web Services

requester

EIS

Backend

SAP Adapter

ESB

Tooling

Case 1 – Connectivity between web service requester and an EIS system

Domain of interest -

Intranet

Registry

21

SOA Symposium


Case 1 Description

Customer environment

– This customer is leading adopter of technology. Comfortable with sophisticated solutions

– WebSphere Application Server customer

Business Requirements

– The customer wants to provide web service access to functionality in an Enterprise Information System such as SAP R/3, PeopleSoft, or Oracle Financials

– Intranet environment doesn‟t require complex security considerations

– The integration is based on message exchange/data replication scenarios - there is no business process or data synchronization between clients and EIS systems

– Support moderate volume of requests

Technical Requirements

– The targeted integration is one-to-one, i.e., one ESB virtual service maps to one EIS application, although multiple EISs can be exposed as web services at the same time

– Data transformation should use XSLT; development tooling important

– Log the messages as they flow through the hub – asynchronously to a file

Architecture Decisions

– J2EE environment preferred

– Use available adapter product to simplify development

SOA Symposium


Case 1 Analysis

Infrastructure Pattern

– Service virtualization

– Domain isolation unimportant

ESB Product selection

– Transport Protocols

– JCA adapter supported

– SOAP/HTTP

– Synchronous Request/response sufficient

– Message models

– XML metamodel desired

– Content model supported

– Mediation flows

– XLST transformation and supporting tooling

– QoS and Non-functional requirements

– J2EE foundation (WebSphere Application Server)

– XSLT transformation skills available

– Adequate throughput and/or response time

– Leading edge adopter

22

SOA Symposium


ESB

.NET Client

Portal

Tooling

SOAP/JMS

XML/MQ Client

XML/MQ

SOAP/HTTP

Text/MQ Client

Text/MQ

Some Client

XML/HTTP

SOAP/JMS

COBOL Copybook/MQ

SOAP/HTTP

XML/HTTP

.NET provider

J2EE provider

CICS

Some provider

Case 2: Connectivity between heterogeneous requesters and providers (Multi-protocol Exchange)

Domain of interest -

Intranet

Registry

SOA Symposium


Case 2 Description

Customer environment

– Customer is a leading adopter of technology. Comfortable with sophisticated solutions

Business Requirements

– Any provider must be accessible via multiple heterogeneous requestors

– Support moderate volume of requests

– Intranet environment does not require complex security considerations

– Global transactions across multiple heterogeneous transaction managers

Technical Requirements

– ESB must support

– Communication protocol conversion, but not adapters

– Flexible data model conversion, with acceptable performance and adequate tooling

– Enterprise class persistent messaging backbone

Architecture Decisions

– Canonical data model(s) used in ESB

– Consumers and providers must adapt to the service definition supported by the ESB

23

SOA Symposium


Case 2 Analysis

Infrastructure Pattern

– Significant service virtualization

ESB Product selection

– Transport Protocols

– Multiple protocols

– Synchronous and asynchronous Request/response

– MQ provides enterprise class persistent messaging backbone

– Message models

– Multiple built-in message metamodels

– Content models supported

– Mediation patterns

– Significant protocol conversion capability

– Significant transformation capability with supporting tooling

– Supports advanced correlation of asynchronous request/response

– Extensive runtime configuration options

– QoS and Non-functional requirements

– Global transactions across multiple heterogeneous transaction managers

– Mature product which supports throughput and/or response time

– Sophisticated transformation skills available

SOA Symposium


Headquarters

Retail Store

Example of Federated ESB (1)

Full-featured

ESB with

richer

mediation &

connectivity

Point of

Sale

Terminal

Mainframe

DBMS

SOAP/HTTP TP

Monitor

POS/Store

Applications

MQ

MQLightweight

commercial

Java-based

ESB

J2EE ServerEnterprise

Applications

MQ

JMS

Point of

Sale

Terminal

MQ

MQ

SOAP/HTTP MQ

Retail Store

Point of

Sale

Terminal

POS/Store

Applications

Lightweight

commercial

Java-based

ESBPoint of

Sale

Terminal

24

SOA Symposium


Enterprise Secure Zone

Demilitarized Zone


ESB 2

(Dynamically

Route and log

Web services)

PartnerJ2EE

Web services

SOAP/ HTTP

Web

Application

Registry

SOAPSOA Appliance

Web services

for CICS

Federated IM

ESB 1

(distribute

events)

PublishersSubscribers

SOAP/ HTTP

MQ

MQMQ

SOA Symposium



Departmental

Service Bus

Backbone Service Bus

Services Registry/Repository

Legacy

Application

.NET

ApplicationDepartmental

Application

WAS

Applicatio

n

External

Services/Consumer

s

.NET

Application

Legacy

Application

FTP/Files

SOAP/HTTP

MQ and Message Broker

MQ

MQSockets

SOAP/

HTTPXML/MQXML/HTTP

MQ

SOAP/HTTP/

WS-Security

BPEL Process

Service

Process Server

WebSphere ESB

SCA

BindingSOAP/

JMS

Business

PartnerBusiness

PartnerBusiness

Partners

EDI / AS2 /

SOAP/HTTP

Red – service requestors/providers

Blue – part of the ESB

Message Broker

MQ

BPEL Process

Process Server

MQ

XML/MQ

MQ

Adapter

Partner

Gateway

PM4Data

WAS

Applicatio

n

DataPowerDevice

Departmental

Service Bus

25

SOA Symposium


Summary

ESB provides Service Virtualization

ESB also enables Aspect Oriented Connectivity

– in support of an SOA Connectivity Entry Point

ESB is an architectural pattern

– various types

– federated ESBs become more wide-spread

We can identify a number of federated ESB patterns

The ESB ref arch provides the basis for product selection

SOA Symposium


Thank YouMerci

Grazie

Gracias

Obrigado

Danke

Japanese

English

French

Russian

GermanItalian

Spanish

Brazilian PortugueseArabic

Traditional Chinese

Simplified Chinese

Thai